fix(storage): drop the five retired share-batch-chain tables (#585) - #828
Merged
Conversation
SBC was deleted in #703, `share_batch_shadow` was turned off fleet-wide and the flag removed from config, and #692 closed on 2026-08-22. What survived the deletion was the DATA. Migrations v50/v51/v52 created these tables, nothing has read or written them since, and every backup and page-cache read has carried them ever since. Measured on ghost-vm5, 2026-09-02: sbc_batches 1390 rows 133 MB sbc_certs 1389 rows 2 MB sbc_watermarks 6 rows ~0 MB sbc_balances 5 rows ~0 MB sbc_quarantine 1 row ~0 MB ~135 MB per node, roughly 1.1 GB across the fleet. `sbc_batches` dominates because each row held a verbatim JSON payload, ~95 KB apiece. Verified before writing it: outside `migrations.rs`, the only mentions of any of these tables are two doc comments in `shard_store.rs` comparing the shard's design to SBC's. No read path, no write path, no flag left to turn it back on. Unlike v57 this does NOT refuse a non-empty table. That guard exists on `wraith_bonds` because every row was escrowed VALUE and dropping it would release or strand sats. These rows are retired accounting history for a mechanism the shard replaced. ⚠ The verification step originally read `WHERE name LIKE 'sbc_%'`, and a test caught that it was blind to exactly what it claimed to check: the indexes are called `idx_sbc_*` and `sqlite_autoindex_sbc_*`, so a name-prefix test matches none of them and would report success with orphaned indexes still in the schema. It now checks `tbl_name` as well, and escapes the `_` wildcard so it asserts the real prefix rather than something close to it. Control: with the old name-only check and a deliberately undropped index, the test PASSES; with the current check it fails. Two existing tests asserted these tables were PRESENT at head — v50's schema guard and v53's "additive, leaves sbc alone" check. Both are inverted rather than deleted: at head the tables must be gone, and a future migration recreating them should have to argue for it. v50's replacement carries a control asserting the shard tables that replaced it DO exist, so it cannot pass against a database that simply failed to migrate. ⚠ This frees pages; it does NOT shrink the file. `auto_vacuum=0` and the freelist is essentially empty, so the space is reused internally and growth slows rather than the file getting smaller. Returning it to the OS needs `VACUUM`, which needs twice the database free and took ghost-vm6 down once — deliberately not done here and tracked separately on #585. Claude-Session: https://claude.ai/code/session_01Td1vvfowptTTnu88qG2iym
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First substantive step on #585 after the backup pruning in #826.
What this reclaims
SBC was deleted in #703, the
share_batch_shadowflag is gone from config, and #692 closed on 2026-08-22. The data outlived the feature. Measured on ghost-vm5 today:sbc_batchessbc_certssbc_watermarkssbc_balancessbc_quarantine~135 MB per node, ~1.1 GB fleet-wide.
sbc_batchesdominates because each row stored verbatim JSON — roughly 95 KB apiece.Verified first: outside
migrations.rs, the only mentions of any of these tables are two doc comments inshard_store.rscomparing the shard's design to SBC's. No read path, no write path, no flag to re-enable it.Why this doesn't use v57's "must be empty" guard
migrate_v57refuses to drop a non-emptywraith_bondsbecause every row was escrowed value — dropping it would silently release or strand sats. That reasoning is about money, not a blanket rule. These rows are retired accounting history for a mechanism the shard replaced.⚠ The bug a test caught in my own verification
The migration verifies its work rather than assuming it. My first version read:
That is blind to what it claims to check — the indexes are
idx_sbc_*andsqlite_autoindex_sbc_*, so a name-prefix match catches none of them, and the check would report success with orphaned indexes still in the schema. It now also matches ontbl_name(the table an index belongs to) and escapes the_wildcard so it asserts the real prefix.Two existing tests inverted, not deleted
v50_stores_balances_...andv53_applies_on_v52...both asserted these tables were present at head. Both are flipped to assert they're gone, so a future migration recreating them has to argue for it. The v50 replacement carries a control asserting the shard tables that replaced SBC do exist — otherwise it would pass just as happily against a database that failed to migrate at all.⚠ Frees pages, does not shrink the file
auto_vacuum=0and the freelist is essentially empty, so the space is reused internally and growth slows — the file does not get smaller. Returning it to the OS needsVACUUM, which wants twice the DB free and took ghost-vm6 down once. Deliberately not done here; tracked separately on #585, where it is now affordable (23–25 GB free vs a 3.7 GB DB).cargo test -p ghost-storage --lib: 234 passed, 0 failed. Clippy and fmt clean.Deploy note
Bumps
SCHEMA_VERSIONto 59, so a migration runs on each node at next roll. v58 took ~81s and left a WAL high-water mark; this is a set ofDROP TABLEs rather than a column rewrite, so it should be cheaper, but expect the same WAL behaviour.https://claude.ai/code/session_01Td1vvfowptTTnu88qG2iym