Take the aggregate apart: the join tree is the floor, not resolution - #200
Open
defenwycke wants to merge 6 commits into
Open
Take the aggregate apart: the join tree is the floor, not resolution#200defenwycke wants to merge 6 commits into
defenwycke wants to merge 6 commits into
Conversation
… explained The distributed aggregate was claimed by #153/#157/#161, ranked as the blocking measurement, and had never been exercised. It has now been, on two L40S in one datacentre with identical METHOD_IDs verified first, because the coordinator refuses foreign receipts by name. workers po2 22 po2 20 1 118 s 163 s 2 65 s 1.81x 90 s 1.81x 4 — 89 s 1.83x <- no gain Scenario (c), where ten minutes is unreachable at any fleet size, is dead: the aggregate is not serial, and at two workers it distributes at 91% of ideal. But it saturates at N=2 and the cause matters enormously. A two-point Amdahl fit predicted 54 s at N=4; it measured 89 s. That is the N^1.79 mistake repeating — a curve extrapolated from two points that cannot constrain it — and the third point is the only reason it was caught. Two candidates with opposite consequences. Join-tree width: four chunks give four leaves, two joins, one root, so only about two-way parallelism exists, and a sixteen-chunk aggregate would be eight wide. Or coordinator-bound: single-threaded seg-serve cannot feed four workers, in which case more chunks change nothing and the aggregate caps near 1.8x, which puts 1,575/1.8 = 865 s above the 600 s budget at any N. The GPU-utilisation diagnostic is inconclusive but not empty: box1's local workers ran at 82% with 7% idle while box2's remote workers ran at 59% with 35% idle. That rules out the clean "no work exists" reading, which would idle all four equally, and points at delivery rather than availability. It does not settle it. A sixteen-chunk aggregate does, and needs about 100 minutes of chunk proving. Also records the #139 GPU result in the reference sheet: 8.00x middle path, 9.10x wholesale, so a near-tip block needs about 7-9 cards rather than 32 — with both caveats attached rather than buried. Claude-Session: https://claude.ai/code/session_015nyynoX7kXHe7zCF3j6Jbf
#139 measured at 8.00x (middle path) and 9.10x (wholesale) on GPU proving wall, and the aggregate measured at 1.81x on two workers. Together those take a near-tip block from 32 cards to about 7-9. Both caveats are stated where the number is, not further down. The aggregate saturates at N=2 on a four-chunk test and nobody yet knows whether that is join-tree width, which is harmless at sixteen chunks, or a coordinator bottleneck, which would cap the aggregate near 1.8x and put it at 865 s — above the whole budget at any fleet size. And #190 has to land or the post-#139 straggler goes to 2.45x. Claude-Session: https://claude.ai/code/session_015nyynoX7kXHe7zCF3j6Jbf
…ion is the middle path
The 16-chunk test settles what the 4-chunk one could not. A 16-chunk
aggregate has an 8-wide join tree, so if width were the limit it would have
scaled. It did not move:
workers 4 chunks 16 chunks
1 163 s 165 s
2 90 s 1.81x 94 s 1.76x
4 89 s no gain 95 s no gain
Both saturate at exactly N=2, and GPU utilisation says the same thing from an
independent angle: the remote box starves worse as workers are added, 26% to
36% idle, while the local box stays busy at 13% to 11%. The parallel work
exists and is not being delivered. seg_serve_cmd is single-threaded — it runs
ExecutorImpl, dispatches segments and collects receipts on one thread — and
caps the aggregate near 1.76x whatever the fleet size.
That floors the aggregate at 897 s, above the whole 600 s budget, so ten
minutes is unreachable at any fleet size as things stand. Neither fix is
architectural. seg-serve is HOST-side: no METHOD_ID move, no re-baseline, no
board reset, and it alone takes the target from impossible to 7 cards. The
aggregate's witness read is guest source and rides the re-baseline already
queued with #139. Both together give 5-6.
Records the operator's decision to take the MIDDLE path rather than
wholesale. Wholesale is 15% faster but buys at most one card, and only in the
pessimistic aggregate case; one card is a cheap price for keeping Core's DER
parsing, low-S handling, r/s checks, inversion and final comparison as
literal libsecp code with only the group arithmetic moved.
That decision likely retires Helix. Helix exists to run wholesale for backfill
and Core at the tip; if the middle path runs everywhere there is no second
backend, no height gate, no cutover constant, no doubled audit surface, and
none of the silent-divergence risk. The middle path is not zero-surface —
double_scalar_mul is Shamir's trick and does not preserve wNAF/GLV — so
differential testing across chain history remains the load-bearing work.
Also records that a two-point Amdahl fit predicted 54 s at N=4 and measured
89 s. Two points cannot constrain a curve. That is the second instance of the
same error in that document, and it was caught only because a third point was
asked for.
Claude-Session: https://claude.ai/code/session_015nyynoX7kXHe7zCF3j6Jbf
The operator has explicitly not committed to it. The previous wording read as settled, which would let downstream work assume a choice that has not been made. Claude-Session: https://claude.ai/code/session_015nyynoX7kXHe7zCF3j6Jbf
The N=4 arm ran two worker processes on box1 and two on box2 — the same two physical cards as N=2. Each seg-connect is a separate process on one GPU, and GPU concurrency has been measured and rejected three times at 0.95-1.03x. So N=2 and N=4 had identical compute, the flat result is exactly what that predicts, and it says nothing about the segment coordinator, the join tree, or any ceiling. I also read the utilisation backwards. Box2 going from 26% to 36% idle when its second worker was added is two processes contending for one card, not a coordinator failing to feed them. I reported it as evidence of dispatch starvation. What was actually measured is that the aggregate scales 1.76-1.81x on two cards, which is 88-91% efficiency and good. Scaling beyond two cards is unmeasured and needs a third box, not a fourth worker process. That restores the fleet arithmetic: 7-9 cards stands on the two-card evidence, "ten minutes is unreachable at any fleet size" was never established, and whether seg_serve_cmd is a bottleneck is simply unknown. Method note, and it is the third instance in that document: section 8.12's N^1.79 artefact moved the block as well as N; section 8.14's Amdahl fit extrapolated from two points; this one changed the worker count without changing the hardware. Every one produced a confident wrong conclusion from a variable that was not the one under test. The superseded reasoning is kept, folded, as the record of the error. Claude-Session: https://claude.ai/code/session_015nyynoX7kXHe7zCF3j6Jbf
Three findings from reading the code against the day's coordinator logs. tx_prevouts cannot be dropped. The aggregate recomputes every accumulator leaf even though the chunks supply chunk_leaves, which looks like waste and is not: recomputing the binding digest is the check that a chunk verified THIS input, so a chunk cannot substitute a different valid spend of the coin or validate it under weaker flags. Sending less is not available. The resolution figures are stale. seg_serve_cmd's own comment estimates resolves at about 175 s for sixteen; measured with resolves pushed to workers they are 0.28 s each — 1.0-1.2 s for four, 4.4 s for sixteen, cleanly linear. Section 8.10 records 16.3 s for the same block, same po2, same chunk count, which is 14x this measurement and almost certainly predates #153 moving resolves off the coordinator. Its companion 196 s figure inherits the doubt. So resolution is not the aggregate's floor, and an earlier claim in this session that it becomes 26% of the aggregate was wrong. The join tree is. It is level-synchronous — every level waits for all of its joins before the next begins — and for a 116-segment aggregate the tree is seven levels of 58, 29, 14, 7, 4, 2, 1. Efficiency falls from 93% at two cards to 40% at thirty-two, which is precisely why the two-card measurement looked healthy: N=2 is the one regime where the problem does not show. Two separable parts. The barriers are fixable and host-side, so pipelining joins ships without a re-baseline. The narrow tail is structural, but free under the bounded-lag framing because one block's tail overlaps the next block's wide segment phase — a performance argument for that framing on top of the cost one. Ranked: the witness read at 2.05x, guest-side, riding the #139 re-baseline; then join-tree pipelining at up to 1.4x at fleet scale, host-side, alone. Resolution and the coordinator's 3 s execute phase are not worth attacking. Claude-Session: https://claude.ai/code/session_015nyynoX7kXHe7zCF3j6Jbf
defenwycke
added a commit
that referenced
this pull request
Aug 28, 2026
…g its removal The previous 'check whether txids is needed at all' bullet invited the wrong change. It is the merkle preimage checked against the header AND the binding target every computed txid is held to -- together what makes 'the raw bytes ARE the block's txs' true, which the output-leaf reconstruction rests on. Same shape as the tx_prevouts finding in #200. The win is unaffected: packing changes the encoding, not whether the field is sent, so every binding is untouched and the 2.09x stands as priced.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #199 (which it merges), so review that first. Three findings from reading the code against the day's coordinator logs — two correct published figures, one identifies the real parallelism floor.
1.
tx_prevoutscannot be dropped — it is a security invariantThe aggregate recomputes every accumulator leaf via
coin_leaves_batcheven though the chunks supplychunk_leaves. That looks like removable waste. It is not:⇒ The recomputation is the anti-substitution check. Sending less is not on the table.
2. ⛔ The resolution figures are stale — resolution is ~40x cheaper than believed
seg_serve_cmd's own comment estimates resolves at "roughly 11.35 M cycles each — about 175 s for sixteen". Measured, with resolves pushed to workers (#153):Cleanly linear at 0.28 s — the chain model is right, the constant was not.
⛔ §8.10 records 16.3 s for the same block, same po2, same 4 chunks — 14x this measurement. It almost certainly predates #153 moving resolves off the coordinator, which is exactly the win that issue describes. Its companion 196 s figure for block 962,000 inherits the doubt, and both are now flagged at source.
⇒ Resolution is not the floor. It also corrects a claim made earlier in this session that resolution becomes 26% of the aggregate after the read fix.
⚠ Not fully settled: block 962,000 has 376 segments against 741,000's 28, so a conditional-size dependence cannot be excluded from within-block evidence. Re-measure on a tip block.
⚠ It is linear in chunk count, so fan-out is not free forever — 64 chunks ≈ 18 s serial, 128 ≈ 36 s. Worth knowing before picking a chunk count for a large fleet.
3. ⏰ The join tree is level-synchronous — that is the real floor
A 116-segment aggregate is 7 levels of
[58, 29, 14, 7, 4, 2, 1], and the tail cannot fill a fleet:⛔ This is why #199's two-card measurement looked healthy — N=2 is the one regime where the problem is invisible.
Two separable problems:
METHOD_IDmove, ships alone.⚠ The projection calibrates one unit cost from a single N=2 point and assumes
t_join ≈ t_seg, unverified. The shape follows from the tree arithmetic alone.Ranked levers
METHOD_ID?write_sliceresolutiontx_prevoutsseg-servedispatchDocs only.
https://claude.ai/code/session_015nyynoX7kXHe7zCF3j6Jbf