The CUDA default is po2 21, not 22 — and it costs ~11% - #198
Open
defenwycke wants to merge 1 commit into
Open
Conversation
SEGMENT_DISTRIBUTION said "22 is the default on CUDA and peaks at ~40.6 GB of
VRAM". Both halves are wrong. main.rs is unambiguous:
.unwrap_or(if cfg!(feature = "cuda") { 21 } else { 20 })
po2 21 peaks at ~22 GB, measured today at 22,521 and 22,009 MiB on an L40S —
half the 40.6 GB the docs implied, and 40.6 GB is what po2 22 costs, which
you only get by setting the variable.
Two consequences. Any run that did not set HAZYNC_SEG_PO2 was at po2 21, so
figures labelled "at po2 22" elsewhere were only that if someone set it
explicitly. And a fleet's VRAM headroom was being sized against a number
twice the real default.
Measured on one L40S, block 140,000, two runs per cell:
stock libsecp 446 s (189 seg) -> 396 s (93 seg) 1.126x
bigint2 48 s ( 18 seg) -> 43 s ( 9 seg) 1.111x
The gain is consistent across two independent arms, which is what licenses
treating ~11% as a property of the setting rather than of one workload.
ACCELERATION's sweep found ~8% on a 415-segment chunk; this confirms it
transfers down to 9 segments, which that file's own caveat said could not be
assumed in the other direction.
Setting po2 22 carries an obligation the docs did not state: serialise GPU
work. At 88% of the card two concurrent proves OOM. `hazync run` serialises
through a lock, a direct `host prove-*` does not (#97) — reproduced today by
running three proves back to back.
Also corrects TOPOLOGY_AND_SETTINGS, which repeated the same wrong default. I
introduced that yesterday by consolidating the claim without checking it
against the code, which is the exact failure the docs audit was about.
And adds the #139 fleet numbers at po2 22: chunk work 14,926 -> 2,310
card-seconds, so 7 cards where it is 32 today. The important part is not the
number but what it exposes — the aggregate is untouched by #139 and becomes
39% of the work, the entire remaining chunk-side headroom is 7 to 3 cards,
and under a serial aggregate ten minutes is unreachable on any hardware.
Claude-Session: https://claude.ai/code/session_015nyynoX7kXHe7zCF3j6Jbf
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
SEGMENT_DISTRIBUTION.mdsaid "22 is the default on CUDA and peaks at ~40.6 GB of VRAM". Both halves are wrong.main.rsis unambiguous:po2 21 peaks at ~22 GB — measured today at 22,521 and 22,009 MiB on an L40S. That is half the 40.6 GB the docs implied, and 40.6 GB is what po2 22 costs, which you only get by setting the variable explicitly.
Why it matters
1. It silently split the project's measurements. Any run that did not set
HAZYNC_SEG_PO2was at po2 21. Figures labelled "at po2 22" elsewhere were only that if someone set it.2. A fleet's VRAM headroom was being sized against a number twice the real default.
3. The default leaves ~11% on the table. Measured on one L40S, block 140,000, two runs per cell:
The gain is consistent across two independent arms, which is what licenses treating ~11% as a property of the setting rather than of one workload.
ACCELERATION.md's sweep found ~8% on a 415-segment chunk; this confirms it transfers down to 9 segments — which that file's own caveat said could not be assumed in the other direction.An obligation the docs did not state
⛔ Setting po2 22 means serialising GPU work. At 88% of a 46 GB card, two concurrent proves OOM.
hazync runserialises through a lock; a directhost prove-*does not (#97). Reproduced today by running three proves back to back.Also corrects a doc I wrote yesterday
TOPOLOGY_AND_SETTINGS.mdrepeated the same wrong default. I introduced that in #196 by consolidating the claim without checking it against the code — the exact failure that PR was about.And adds the #139 fleet numbers at po2 22
Per-verify ECDSA drops 1,723,407 → 140,044 cycles (12.31x). Chunk work on block 962,000: 14,926 → 2,310 card-seconds.
⛔ The important part is not the number — it is what it exposes. #139 does not touch the aggregate (no EC verification happens there), so it goes from ~10% of one-card cost to 39%. Under a serial aggregate its 1,575 s alone exceeds the 600 s budget on any hardware, and the entire remaining chunk-side headroom is 7 → 3 cards even with infinitely fast chunks.
⇒ Whether the aggregate distributes is now worth more than every card, po2 and guest-codegen decision combined.
⚠ Assumes hazync#190 has landed — post-#139 ECDSA and Schnorr diverge, and #190's simulation puts an unaware packer's straggler at 2.45x, which would roughly halve the win. Block 962,000 is also only 2.7% Schnorr, which #190 calls "the mildest case available".
⚠ Middle-path @ po2 22 is inferred (scaled by the measured po2 factor), not directly measured; it is labelled as such.
https://claude.ai/code/session_015nyynoX7kXHe7zCF3j6Jbf