CEP-45: Commit log and mutation journal positions conflict with each other - #5105
Open
bdeggleston wants to merge 8 commits into
Open
CEP-45: Commit log and mutation journal positions conflict with each other#5105bdeggleston wants to merge 8 commits into
bdeggleston wants to merge 8 commits into
Conversation
The _build-accord target otherwise runs a full gradle clean build and publishToMavenLocal on every ant invocation. It also takes a lock on the shared gradle artifact cache, so a second checkout building concurrently blocks until the first finishes.
A replayed mutation's domain follows the log it was read out of, not the keyspace's current routing state. Commit log replay takes the untracked apply path and skips the routing precondition, because the commit log may be replayed after a migration to tracked has already completed. Journal replay stops re-deriving its route and uses the unified entry point. Adds CommitLogReplayRoutingTest, and SSTableProvenance to classify an sstable's origin from its coordinator log offsets and commit log intervals.
Offsets were being dropped piecemeal as compaction merged sstables, which left no single point where an sstable became reconciled. Promotion is now the one place offsets are cleared: a sweep promotes fully reconciled sstables, and write-time promotion clears offsets when it sets repairedAt. Only an incremental repair may advance mutation tracking migration, since a full repair does not establish the reconciliation barrier promotion needs. Adds a metric for unrepaired sstables carrying no offsets, which are otherwise invisible.
Compacting an sstable that carries coordinator log offsets together with one that does not produces an sstable whose offsets no longer describe its contents, so it can never be promoted. Tracked sstables are now keyed by their transfer set and each key gets its own CompactionStrategyHolder, so no strategy can select across the boundary. Silos holding no transfers share one key, and empty transfer silos are pruned. CompactionGroup replaces the ad hoc repaired/pending checks and routes any sstable carrying offsets to UNRECONCILED. PromoteReconciledTask promotes a silo once its offsets are fully reconciled, replacing the standalone promotion sweep. Offsets are cleared whenever repairedAt is set, which stops a repaired sstable retaining offsets it can no longer act on.
Marking a corrupt sstable unrepaired exists so incremental repair notices it and repairs it. On a tracked table that would produce an unreconcilable sstable, so repairedAt is left unchanged and the operator is told why. Full repair adds any missing data but cannot remove the corruption: read resolution is last-write-wins on timestamp, so a corrupt row with a mangled high timestamp still beats the rows repair streams in. The message says to scrub or replace the node rather than implying a full repair is sufficient.
The commit log and the mutation journal produce positions that cannot be compared, so a memtable's bound is only meaningful against one log. A memtable now carries the LogDomain it accepts and refuses a write from the other, and LogDomainBounds holds a separate bound per domain. Sealing a bound re-reads until it stops moving, so a write racing the seal cannot land outside it. During migration a table takes writes from both logs at once. SplitDomainMemtable holds one internal memtable per domain and is substitutable for a plain memtable on reads, writes and flush. It is installed lazily, on the first write whose domain the current memtable does not hold, and reuses the existing internal rather than rebuilding one. A retired generation refuses to split. Flushing a split generation writes one sstable per domain, which keeps each sstable's commit log interval comparable against the log it came from. The generation boundary lives on the View, which is what knows when a generation is retired.
Streaming reads a memtable's commit log bound to decide what the stream covers. A split generation has two bounds that cannot be reduced to one, so the flush-before-stream path asserts the memtable holds a single domain rather than picking one silently.
Covers the change end to end: a keyspace altered between untracked and tracked keeps each sstable's commit log interval comparable against the log it came from, and a table taking writes from both logs at once flushes one sstable per domain rather than one spanning both. Adds TrackedIndexFlushTest, since an index of a tracked table writes through the base table's journal and must be bounded in the same log. The index write path says none() rather than a placeholder id.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Writes to a table go to either the commit log or the mutation journal, depending on their replication type and migration status. Both record their position in the same CommitLogPosition type, but positions from the two logs are not comparable. Memtables and sstables use these positions to decide what has been persisted and what needs replay. Anything holding writes from both logs therefore records a bound that does not describe its contents.
This is reachable in normal operation. Migrating a keyspace between untracked and tracked replication routes writes to both logs at once, and a commit log can be replayed after that migration has already finished.
The change makes the two domains explicit: