🚧 Vendor risc0-sys for upstream #3781's u32 overflow fix (needs B200 validation) - #182
Draft
defenwycke wants to merge 1 commit into
Draft
🚧 Vendor risc0-sys for upstream #3781's u32 overflow fix (needs B200 validation)#182defenwycke wants to merge 1 commit into
defenwycke wants to merge 1 commit into
Conversation
…ERGE YET)
Prepared for testing on a B200. NOT verified on hardware -- see the
warning in prover/Cargo.toml and the last section below.
THE BUG, confirmed present in OUR version (risc0-sys 1.5.0, not just
the 5.0.0-rc.1 upstream filed against):
kernels/zkp/cuda/supra/poseidon2.cuh:151
void _poseidon2_rows(..., uint32_t dim_x, uint32_t dim_y) {
uint32_t i = 0;
for (uint32_t j = 0; j < CELLS_RATE; j++, i++)
cells[j] = i < dim_y ? matrix[i * dim_x] : fr_t{0};
`i` is the column index and `dim_x` the domain, both uint32_t, so
`i * dim_x` runs in 32 bits and wraps once column*domain crosses 2^32.
The kernel then reads the wrong cell and commits a hash over the wrong
data -- with no panic and no error. The GPU returns a valid-looking seal
that fails verification.
WHY WE CARE. The domain is 2^po2 * 4, so the column threshold HALVES
with every po2 step: ~256 columns at po2 22, ~128 at po2 23. On
2026-08-25 a stock rv32im aggregate ran fine at po2 22 (~1 fault in 750
segments, a separate issue) and failed DETERMINISTICALLY at po2 23 --
segment 0, three attempts of three, with exactly the symptom above.
That also explains why #3773 was previously refuted as the cause of
hazync#119: at po2 22 it genuinely does not trigger.
Reported upstream on risc0#3773 with the arithmetic and a reproducer.
WHY THIS IS NOT MERGED. Widening a multiply cannot change behaviour
below the threshold -- the product already fit -- so po2 <= 22 is
unaffected by construction. But "by construction" is not "measured",
and this sits in the path that produces SHIPPED proofs. It needs a po2
23 run on a real card to show the fix works, and a po2 22 run to show
nothing else moved. Both boxes were deleted; a B200 is being arranged.
This was referenced Aug 25, 2026
defenwycke
added a commit
that referenced
this pull request
Aug 26, 2026
…ade to (#188) E6 was specified as a 1-D sweep of worker processes at fixed po2. That cannot answer the question, because processes and po2 trade against each other through VRAM: L40S at po2 22 peaks at ~41 GB of a 46 GB card -- 89% -- so there is very likely no room for a second process at po2 22 at all, on the exact card a fleet would be built from. The removal note also carries a skimmed detail: dropping the in-process pipeline "also unblocked po2 22, worth 1.15x on chunks and 1.42x on the aggregate". po2 22 is worth more than either scheduling fix, so anything that spends po2 22 to buy parallelism is probably a bad trade. And the 1.20x worker-process figure has NO recorded conditions -- neither the board nor prover_impl.rs says at what po2 or on which card. If it was taken at po2 20 it is not comparable to a po2 22 baseline at all. E6 now sweeps processes x po2 and records peak VRAM for every cell INCLUDING cells that OOM, because the failures are what separate contention from memory. This also re-gates E9. If the optimum is 3 processes at po2 20, processes already fill the idle time and E9 is dead. If it is 1 process at po2 22 -- VRAM-bound, GPU still 65% idle -- pipelining is the only thing that can fill it, because it shares one context and one allocation. E10 (a newer risc0) is CLOSED, checked rather than assumed: - latest stable is 3.0.6, whose entire content is "Repair Rust 1.97.0 guest build with heap-embedded-alloc". We do not enable that feature and we pin Rust 1.94.1, so 3.0.6 is a literal no-op here. - 5.0.0-rc.1 looks newer and is not. crates.io publish order is 3.0.4, 5.0.0-rc.1, 3.0.5, 3.0.6 -- the RC PREDATES our own pin. - upstream is dormant: five commits in three months, all housekeeping. risc0#3781 (the CUDA overflow fix blocking po2 23) is still open. Our #3798 and #3799 have zero comments. Three consequences, two of which reverse advice given elsewhere in this repo: #182's vendored fix is permanent rather than a stopgap; the vendor tree's "keep it re-appliable because risc0 moves" premise no longer holds; and E9 would be a fork rather than an upgrade, which raises its cost.
This was referenced Aug 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🚧 DRAFT — do not merge until a B200 run demonstrates it. Opened for visibility and to get CI on the vendored crate, not for review-to-merge.
Carries upstream risc0#3781's one-line fix for risc0#3773, which we have confirmed is present in our version.
The bug, in risc0-sys 1.5.0
Not just the 5.0.0-rc.1 upstream filed against —
kernels/zkp/cuda/supra/poseidon2.cuh:151:iis the column index,dim_xthe domain — bothuint32_t, so the product wraps oncecolumn × domaincrosses 2³². The kernel reads the wrong cell and commits a hash over the wrong data, with no panic and no error: the GPU returns a valid-looking seal that fails verification.Why it matters to us
The domain is
2^po2 × 4, so the column threshold halves with every po2 step:Measured 2026-08-25 on a B200, stock rv32im, same aggregate:
Exactly the symptom above. It also explains why #3773 was previously refuted as the cause of #119: at po2 22 it genuinely does not trigger.
Reported upstream with the arithmetic and reproducer.
Why this is a draft
Widening a multiply cannot change behaviour below the threshold — the product already fit — so po2 ≤ 22 is unaffected by construction.
But "by construction" is not "measured", and this sits in the path that produces shipped proofs. Two runs on a real card are wanted before this is mergeable:
306fc568…, total still ~1700 s? (nothing else moved)A B200 is being arranged. Until both pass, this stays draft.
Scope
risc0-sys1.5.0 vendored verbatim (720 K, 52 files) except the one cast, wired through the[patch.crates-io]block that already carriesrisc0-zkvm. The warning is repeated inprover/Cargo.tomlso it is visible at the wiring, not only here.