fix(deploy): prune stale binary backups instead of accumulating them forever (#585) - #826
Merged
Merged
Conversation
…forever (#585) Every deploy copies the live binary to `$BINARY.bak.$TS` (~25 MB) and nothing has ever removed one. #585 step 0 freed 16.3 GB fleet-wide by hand in July and said to make it automatic; it was not, so it came straight back. Measured 2026-09-01 after the v1.11.33 roll: vm1 39 files 1.4 GB vm4 38 files 765 MB vm5 33 files 712 MB A manual step that must run after every deploy is a step that does not run. The deploy now keeps the two most recent backups per binary and deletes the rest. TWO, not one: rollback restores the most recent, so keeping only that leaves nothing to fall back on if the rollback itself has to be undone. It runs after the soak record and the smoke test, and is best-effort — it must never be what fails a deploy that otherwise succeeded. ⚠ The dangerous part is the offset, and it is computed LOCALLY. Written as `\$((PRUNE_KEEP + 1))` inside the remote payload the arithmetic evaluates on the far side, where `PRUNE_KEEP` does not exist: it expands to 1, `tail -n +1` selects EVERY backup including the one the deploy just made, and the only thing a rollback has to restore is deleted. I wrote it that way first. `S=$SUDO` follows the existing swap payload's pattern — assigned locally, used remotely. Five cases in the deploy-gate self-test, against a temp dir rather than through ssh, because the risk is in WHICH files the selection picks, not in the plumbing. They assert the count AND, separately, that the newest survived — losing the newest is the failure that matters and a count alone can hide it. The `offset 1` control reproduces the bug above. ⚠ The cases were initially appended at the END of the file, after the summary and its `exit 1`. They ran, incremented `fail`, and nothing looked at it again — the script would have exited 0 with a failing test. Moved above the summary, and the success line now names what it covers so it cannot under-report. Claude-Session: https://claude.ai/code/session_01Td1vvfowptTTnu88qG2iym
defenwycke
added a commit
that referenced
this pull request
Sep 1, 2026
The arming release. Two consensus-visible gates fire from this build, and both require every node to be running it beforehand: * ADDRESS_PROOF_HEIGHT 966_000 (#824, H-7, #605) * MESH_ENVELOPE_V2_HEIGHT 966_400 (#825, H-11, #604) Deliberately ~400 blocks apart rather than sharing a height, so a mesh problem in that window is attributable to one gate rather than two. MESH_ENVELOPE_V2 is the reason that matters: it does not degrade, it PARTITIONS — a node left on an older binary rejects every mesh message from every upgraded peer, and the only way back is upgrading it. ⛔ ALL EIGHT must carry this before 966_400. At 965,085 that is ~8 days. Also carries #826: the deploy now prunes stale `$BINARY.bak.*` to the two most recent instead of accumulating them forever. Measured before the fix, after the v1.11.33 roll: vm1 39 files / 1.4 GB, vm4 38 / 765 MB, vm5 33 / 712 MB. Both lockfiles synced; `tests/integration-sv2`, `crates/stratum-apps` and `prototypes/ghast-bench` carry no workspace version and still resolve under `--locked`. Checked with `grep -F`, since an unescaped `1.11.33` matches digits inside dependency checksums. Claude-Session: https://claude.ai/code/session_01Td1vvfowptTTnu88qG2iym
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
#585 step 0, finally automated.
The leak
Every deploy copies the live binary to
$BINARY.bak.$TS(~25 MB). Nothing has ever deleted one. July's cleanup freed 16.3 GB fleet-wide by hand, and the plan said to make it automatic. It wasn't, so it came back — measured today after the v1.11.33 roll:A manual step that must run after every deploy is a step that does not run.
The fix
Keep the two most recent per binary, delete the rest. Two, not one: rollback restores the most recent, so keeping only that leaves nothing to fall back on if the rollback itself has to be undone. Runs after the soak record and smoke test, and is best-effort — it must never fail a deploy that otherwise succeeded.
⚠ The bug I wrote first
The offset is computed locally, and that is the whole point:
Written as
\$((PRUNE_KEEP + 1))inside the remote payload, the arithmetic evaluates on the far side wherePRUNE_KEEPdoes not exist. It expands to 1,tail -n +1selects every backup including the one the deploy just made, and the only thing a rollback has to restore is deleted. That was my first version; it is now the control case.S=$SUDOfollows the existing swap payload's pattern — assigned locally, used remotely as\$S.Tests
Five cases in
test-deploy-gate.sh, run against a temp dir rather than through ssh, because the risk is in which files the selection picks, not the plumbing. Each asserts the surviving count and, separately, that the newest survived — losing the newest is the failure that matters and a count alone can hide it.The control asserts the newest was destroyed, rather than surviving. Asserting "newest survived" unconditionally would have failed the control for doing its job — and a control indistinguishable from a regression is not a control.
⚠ A second trap, caught before pushing
The cases were first appended at the end of the file, after the summary and its
exit 1. They ran, incrementedfail, and nothing looked at it again — the script would have exited 0 with a failing test. Moved above the summary; the success line now names what it covers so it cannot under-report.All 96 deploy-gate checks pass.
check-inlined-copies,check-workflow-scalars,check-stratum-config-agreementall clean.Not the whole of #585
This is step 0 only. Still open there:
auto_vacuumis still 0 (verified today — deletes return nothing to the filesystem),/var/logis at 2.4 GB with no cap, and the dead pre-v41 share rows. Scoping those separately on the issue.https://claude.ai/code/session_01Td1vvfowptTTnu88qG2iym