fix(sdk): a bundle nobody stored is not a bundle you may bind against - #779
Merged
Conversation
`put_bundle` called `put_immutable_to_all_members(...).map(|_| ())`. The inner
call returns `Ok(KeyedPutFanout)` whenever the member SDK could be CONSTRUCTED;
per-member acceptance lives in `fanout.accepted`, which the `map` discarded. The
outer `Result` was `Err` only when `member_sdk_with_auth` failed.
So the caller erased the difference between two unrelated facts:
the SDK invocation succeeded
enough committed members actually accepted B
Zero of three members accepting — a 413, a 401, an unreachable fleet — still
returned Ok. `bind_settlement` then placed the trader-parent fence and drove the
register against a `value_addr` no member holds, after which the bundle is
unfetchable, `settlement_resume` can never complete, and the DLV parent stays
fenced forever. Requirement 6.15's ORDERING was enforced; its SUCCESS was not.
The gate is the vault's committed quorum, not a hardcoded majority and not the
locally configured fleet size:
accepted attributable members >= q, for the exact committed storage set
otherwise refuse BEFORE any fence row and before any binding round
`accepted` is already the ATTRIBUTABLE count on both paths — the production
fan-out counts a 2xx only when the member echoes its own configured id (Req
15.8), and the fleet double applies the same echo rule — so no new attribution
logic is introduced. `artifact_republish` already gated on `set.quorum()` in
both of its fan-out paths; this brings the settle path in line with it. A sweep
of every other caller found no third instance: the remaining hits are inside
`#[cfg(test)]`.
Five boundary cases pinned at beta n=3, q=2:
0/3 accepted -> refuse; no fence; no bind round
1/3 accepted -> refuse; no fence; no bind round
2/3 accepted -> proceed
3/3 accepted -> proceed
transport succeeds, none attributable -> refuse; no fence; no bind round
The last is the case the discarded result hid: every HTTP call returns normally
and every member stores the bytes, but no acceptance is attributable, so the
publication is worth nothing and must read exactly like 0/3.
Each refusal test asserts the absence of the forbidden STATE before it asserts
the error value, so removing the threshold fails on "a fence row was written
despite a non-durable publication" rather than merely on a changed return.
Verified: the mutation reddens all three refusal tests at that assertion.
Also resets `storage_io::fake_fleet` alongside the binding double in this
module's tests. Only the register double was being reset, so injected put
failures and echo overrides were leaking between tests under --test-threads=1.
Found while proving a 50,330-byte owner-close SettlementBundle clears every
configured transport path (it does, 10.4x against the binding 512 KiB
device_auth body buffer). That margin was only load-bearing because the client
could not observe the put failing at all.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The defect
put_bundlecalledput_immutable_to_all_members(...).map(|_| ()). That inner call returnsOk(KeyedPutFanout)whenever the member SDK could be constructed; per-member acceptance livesin
fanout.accepted, which themapdiscarded. The outerResultwasErronly whenmember_sdk_with_authfailed.So the caller erased the difference between two unrelated facts:
Zero of three members accepting — a 413, a 401, an unreachable fleet — still returned
Ok.bind_settlementthen placed the trader-parent fence and drove the register against avalue_addrno member holds. The bundle is unfetchable,
settlement_resumecan never complete, and the DLVparent stays fenced forever. Requirement 6.15's ordering was enforced; its success was not.
The gate
qis the vault's committed quorum viaset.quorum()— never a hardcoded majority, never thelocally configured fleet size.
Two things this deliberately does not add.
acceptedis already the attributable count onboth paths: the production fan-out counts a 2xx only when the member echoes its own configured id
(Req 15.8), and the fleet double applies the same echo rule — so no new attribution logic. And
artifact_republishalready gated both of its fan-out paths onset.quorum(); this brings thesettle path in line with an existing in-repo pattern rather than inventing one. A sweep of every
other caller found no third production instance — the remaining hits are inside
#[cfg(test)].Ordering holds by construction, not convention:
put_bundleis called atbind_settlement:148,and both the fence and the register are driven inside
run_fencedat:185.Tests
Five boundary cases at beta
n=3, q=2:The last is the case the discarded result hid: every HTTP call returns normally and every member
stores the bytes, but no acceptance is attributable, so the publication is worth nothing and must
read exactly like 0/3.
Each refusal test asserts the absence of the forbidden state before it asserts the error value.
That ordering is the point: written the other way round, removing the gate reddens the test at
.unwrap_err(), which proves only that a return value changed. As written, the mutation fails ona fence row was written despite a non-durable publication— verified, all three refusal tests.Also resets
storage_io::fake_fleetalongside the binding double in this module. Only the registerdouble was being reset, so injected put failures and echo overrides leaked between tests under
--test-threads=1; two pre-existing tests failed once the gate landed, and this is the true cause.Verification
17 ignored. The 17 ignored are pre-existing (bluetooth, bitcoin_query, core_bridge_adapters,
storage_routes); none is new here, and all seven new tests report
ok.make lint(pinned 1.98.0,fmt --check+clippy --all-targets): exit 0.ci/production_safety_checks.sh: exit 0.Context
Found while proving a 50,330-byte owner-close
SettlementBundleclears every configured transportpath. It does — 10.4x against the binding 512 KiB
device_authbody buffer, with the body moving asraw octet-stream and no reverse proxy in front of the nodes. That margin was only load-bearing
because the client could not observe the put failing at all.