[ESDB-106-15] Always replicate the chunk header #4211
Conversation
shaan1337
force-pushed
the
db-719-replicate-chunk-header
branch
9 times, most recently
from
898b1cb
to
b5278aa
Compare
shaan1337
force-pushed
the
db-719-replicate-chunk-header
branch
2 times, most recently
from
95e86ca
to
3461abd
Compare
|
Please don't merge yet, as some additional tests are probably needed. |
src/EventStore.Core/Services/Replication/LeaderReplicationService.cs
Outdated
Show resolved
Hide resolved
hayley-jean
changed the title
shaan1337
changed the title
shaan1337
force-pushed
the
db-719-replicate-chunk-header
branch
from
56ec6f1
to
c1893f9
Compare
shaan1337
force-pushed
the
db-719-replicate-chunk-header
branch
2 times, most recently
from
8637e02
to
c55c0ed
Compare
nice find! I believe that this bug already existed prior to this PR due to this call in SwitchChunk: it could easily be fixed by specifying a minimum value of 1 for the chunk version (although the versions wouldn't necessarily match with the leader) it's quite important to fix it as some places in the code assume that a chunk with version 0 hasn't been scavenged: |
sorry, my bad, it is actually a bug introduced in this PR, i'll push a fix for it (EDIT: pushed) |
shaan1337
force-pushed
the
db-719-replicate-chunk-header
branch
2 times, most recently
from
d8874a6
to
0f61015
Compare
This commit brings the following changes: * Always replicate the chunk header to followers - prior to this commit, chunk headers were replicated only for scavenged chunks * Doing the above implies that we can no longer create new chunks in advance: * on followers, when completing a chunk * when a new database is spinned up. the chunk is now created when a node becomes a leader and is writing an epoch. * when an existing database is spinned up but the writer checkpoint was at a chunk boundary * when truncating to a chunk boundary, we also need to delete the chunk at the boundary to re-replicate the chunk (incl. the header) from the leader Backward/Forward compatibility is preserved. Other changes: * cluster.proto: For clarity, rename `is_completed_chunk` to `is_scavenged_chunk` because that's what it really means (in the early days, completed chunks were replicated raw)
shaan1337
force-pushed
the
db-719-replicate-chunk-header
branch
from
0f61015
to
c607652
Compare
timothycoleman
dismissed
hayley-jean’s stale review
Hayley on leave, her review points have been addressed
Changed: Always replicate the chunk header
This PR causes followers to always replicate the chunk header from the leader. It is a pre-requisite for the encryption at rest plugin. Prior to this PR, the chunk header was replicated only for scavenged chunks.
This change has an additional benefit: it's easy for sysadmins to compare files on a leader and a follower by using standard tools like
md5sum,sha1sum, etc. without needing to exclude the first 128 bytes from the comparison (as previously, the chunk IDs were different on the leader and followers)