The aggregate distributes: 1.81x on two cards, and #139 proves out at 8.00x - #199
Closed
defenwycke wants to merge 5 commits into
Closed
The aggregate distributes: 1.81x on two cards, and #139 proves out at 8.00x#199defenwycke wants to merge 5 commits into
defenwycke wants to merge 5 commits into
Conversation
… explained The distributed aggregate was claimed by #153/#157/#161, ranked as the blocking measurement, and had never been exercised. It has now been, on two L40S in one datacentre with identical METHOD_IDs verified first, because the coordinator refuses foreign receipts by name. workers po2 22 po2 20 1 118 s 163 s 2 65 s 1.81x 90 s 1.81x 4 — 89 s 1.83x <- no gain Scenario (c), where ten minutes is unreachable at any fleet size, is dead: the aggregate is not serial, and at two workers it distributes at 91% of ideal. But it saturates at N=2 and the cause matters enormously. A two-point Amdahl fit predicted 54 s at N=4; it measured 89 s. That is the N^1.79 mistake repeating — a curve extrapolated from two points that cannot constrain it — and the third point is the only reason it was caught. Two candidates with opposite consequences. Join-tree width: four chunks give four leaves, two joins, one root, so only about two-way parallelism exists, and a sixteen-chunk aggregate would be eight wide. Or coordinator-bound: single-threaded seg-serve cannot feed four workers, in which case more chunks change nothing and the aggregate caps near 1.8x, which puts 1,575/1.8 = 865 s above the 600 s budget at any N. The GPU-utilisation diagnostic is inconclusive but not empty: box1's local workers ran at 82% with 7% idle while box2's remote workers ran at 59% with 35% idle. That rules out the clean "no work exists" reading, which would idle all four equally, and points at delivery rather than availability. It does not settle it. A sixteen-chunk aggregate does, and needs about 100 minutes of chunk proving. Also records the #139 GPU result in the reference sheet: 8.00x middle path, 9.10x wholesale, so a near-tip block needs about 7-9 cards rather than 32 — with both caveats attached rather than buried. Claude-Session: https://claude.ai/code/session_015nyynoX7kXHe7zCF3j6Jbf
#139 measured at 8.00x (middle path) and 9.10x (wholesale) on GPU proving wall, and the aggregate measured at 1.81x on two workers. Together those take a near-tip block from 32 cards to about 7-9. Both caveats are stated where the number is, not further down. The aggregate saturates at N=2 on a four-chunk test and nobody yet knows whether that is join-tree width, which is harmless at sixteen chunks, or a coordinator bottleneck, which would cap the aggregate near 1.8x and put it at 865 s — above the whole budget at any fleet size. And #190 has to land or the post-#139 straggler goes to 2.45x. Claude-Session: https://claude.ai/code/session_015nyynoX7kXHe7zCF3j6Jbf
…ion is the middle path
The 16-chunk test settles what the 4-chunk one could not. A 16-chunk
aggregate has an 8-wide join tree, so if width were the limit it would have
scaled. It did not move:
workers 4 chunks 16 chunks
1 163 s 165 s
2 90 s 1.81x 94 s 1.76x
4 89 s no gain 95 s no gain
Both saturate at exactly N=2, and GPU utilisation says the same thing from an
independent angle: the remote box starves worse as workers are added, 26% to
36% idle, while the local box stays busy at 13% to 11%. The parallel work
exists and is not being delivered. seg_serve_cmd is single-threaded — it runs
ExecutorImpl, dispatches segments and collects receipts on one thread — and
caps the aggregate near 1.76x whatever the fleet size.
That floors the aggregate at 897 s, above the whole 600 s budget, so ten
minutes is unreachable at any fleet size as things stand. Neither fix is
architectural. seg-serve is HOST-side: no METHOD_ID move, no re-baseline, no
board reset, and it alone takes the target from impossible to 7 cards. The
aggregate's witness read is guest source and rides the re-baseline already
queued with #139. Both together give 5-6.
Records the operator's decision to take the MIDDLE path rather than
wholesale. Wholesale is 15% faster but buys at most one card, and only in the
pessimistic aggregate case; one card is a cheap price for keeping Core's DER
parsing, low-S handling, r/s checks, inversion and final comparison as
literal libsecp code with only the group arithmetic moved.
That decision likely retires Helix. Helix exists to run wholesale for backfill
and Core at the tip; if the middle path runs everywhere there is no second
backend, no height gate, no cutover constant, no doubled audit surface, and
none of the silent-divergence risk. The middle path is not zero-surface —
double_scalar_mul is Shamir's trick and does not preserve wNAF/GLV — so
differential testing across chain history remains the load-bearing work.
Also records that a two-point Amdahl fit predicted 54 s at N=4 and measured
89 s. Two points cannot constrain a curve. That is the second instance of the
same error in that document, and it was caught only because a third point was
asked for.
Claude-Session: https://claude.ai/code/session_015nyynoX7kXHe7zCF3j6Jbf
defenwycke
changed the title
The operator has explicitly not committed to it. The previous wording read as settled, which would let downstream work assume a choice that has not been made. Claude-Session: https://claude.ai/code/session_015nyynoX7kXHe7zCF3j6Jbf
The N=4 arm ran two worker processes on box1 and two on box2 — the same two physical cards as N=2. Each seg-connect is a separate process on one GPU, and GPU concurrency has been measured and rejected three times at 0.95-1.03x. So N=2 and N=4 had identical compute, the flat result is exactly what that predicts, and it says nothing about the segment coordinator, the join tree, or any ceiling. I also read the utilisation backwards. Box2 going from 26% to 36% idle when its second worker was added is two processes contending for one card, not a coordinator failing to feed them. I reported it as evidence of dispatch starvation. What was actually measured is that the aggregate scales 1.76-1.81x on two cards, which is 88-91% efficiency and good. Scaling beyond two cards is unmeasured and needs a third box, not a fourth worker process. That restores the fleet arithmetic: 7-9 cards stands on the two-card evidence, "ten minutes is unreachable at any fleet size" was never established, and whether seg_serve_cmd is a bottleneck is simply unknown. Method note, and it is the third instance in that document: section 8.12's N^1.79 artefact moved the block as well as N; section 8.14's Amdahl fit extrapolated from two points; this one changed the worker count without changing the hardware. Every one produced a confident wrong conclusion from a variable that was not the one under test. The superseded reasoning is kept, folded, as the record of the error. Claude-Session: https://claude.ai/code/session_015nyynoX7kXHe7zCF3j6Jbf
defenwycke
changed the title
Contributor
Author
|
Superseded by #200 — closing.
Merging this PR would land the superseded conclusion. #200 has the corrected story and is the one to take. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two GPU results from 2026-08-28, on two L40S in one datacentre.
1. The aggregate distributes — scenario (c) is dead
Claimed by #153/#157/#161, ranked as the blocking measurement, and never exercised. Now it has been, with identical
METHOD_IDs verified on both boxes first (the coordinator refuses foreign receipts by name).Block 741,000, 4 chunk receipts, aggregate only:
⇒ The aggregate is not serial, and at N=2 it distributes at 91% of ideal.
⛔ But it saturates at N=2, and the cause decides the project
A two-point Amdahl fit on N=1,2 predicted 54 s at N=4. It measured 89 s.
⚠ That is the
N^1.79mistake repeating — a curve extrapolated from two points that cannot constrain it. §8.12 has the post-mortem on the first instance. The third point is the only reason it was caught, and it was run only because someone asked for it.seg-servecannot feed 4 workersDiagnostic — GPU utilisation, 4 workers, both boxes sampled:
⇒ Inconclusive, but it rules out the clean "no work exists" reading — that would idle all four roughly equally. The remote box starves while the local one does not, pointing at work delivery rather than availability. Not decisive.
⏰ A 16-chunk aggregate settles it (~100 min of chunk proving). Until then "the aggregate distributes" is true at N=2 and unproven above it.
2. #139 proves out: 8.00x
9.19x in cycles → 8.00x in proving wall, so the separate coprocessor circuit takes ~13% rather than the whole win — which was #139's central unknown. Segment count corroborates independently: 189 → 21 is 9.0x.
⇒ Per-verify ECDSA 1,723,407 → 140,044 cycles. Block 962,000 is 1.8% taproot by input, so this accelerates essentially the whole block.
⇒ A near-tip block needs ~7-9 cards rather than 32.
⚠ Also requires hazync#190, or the post-#139 straggler goes to 2.45x and roughly halves the win.
Docs only — the evidence in
TEN_MINUTE_BLOCK.md§8.14, the conclusions inTOPOLOGY_AND_SETTINGS.md§0.5, with both caveats stated where the numbers are rather than further down.https://claude.ai/code/session_015nyynoX7kXHe7zCF3j6Jbf