Skip to content

🚧 Vendor risc0-sys for upstream #3781's u32 overflow fix (needs B200 validation) - #182

Draft
defenwycke wants to merge 1 commit into
mainfrom
fix/risc0-3773-u32-overflow
Draft

🚧 Vendor risc0-sys for upstream #3781's u32 overflow fix (needs B200 validation)#182
defenwycke wants to merge 1 commit into
mainfrom
fix/risc0-3773-u32-overflow

Conversation

@defenwycke

Copy link
Copy Markdown
Contributor

🚧 DRAFT — do not merge until a B200 run demonstrates it. Opened for visibility and to get CI on the vendored crate, not for review-to-merge.

Carries upstream risc0#3781's one-line fix for risc0#3773, which we have confirmed is present in our version.

The bug, in risc0-sys 1.5.0

Not just the 5.0.0-rc.1 upstream filed against — kernels/zkp/cuda/supra/poseidon2.cuh:151:

void _poseidon2_rows(poseidon_out_t* out, const fr_t* matrix, uint32_t dim_x, uint32_t dim_y) {
  uint32_t i = 0;
  for (uint32_t j = 0; j < CELLS_RATE; j++, i++)
    cells[j] = i < dim_y ? matrix[i * dim_x] : fr_t{0};   // 32-bit multiply

i is the column index, dim_x the domain — both uint32_t, so the product wraps once column × domain crosses 2³². The kernel reads the wrong cell and commits a hash over the wrong data, with no panic and no error: the GPU returns a valid-looking seal that fails verification.

Why it matters to us

The domain is 2^po2 × 4, so the column threshold halves with every po2 step:

po2 domain overflow above
22 16.78 M ~256 columns
23 33.55 M ~128 columns

Measured 2026-08-25 on a B200, stock rv32im, same aggregate:

po2 result
22 completes, digest correct (~1 fault in 750 segments — separate, #119)
23 fails segment 0, three attempts of three

Exactly the symptom above. It also explains why #3773 was previously refuted as the cause of #119: at po2 22 it genuinely does not trigger.

Reported upstream with the arithmetic and reproducer.

Why this is a draft

Widening a multiply cannot change behaviour below the threshold — the product already fit — so po2 ≤ 22 is unaffected by construction.

But "by construction" is not "measured", and this sits in the path that produces shipped proofs. Two runs on a real card are wanted before this is mergeable:

  1. po2 23 with the patch — does it now produce a valid proof? (the fix works)
  2. po2 22 with the patch — digest still 306fc568…, total still ~1700 s? (nothing else moved)

A B200 is being arranged. Until both pass, this stays draft.

Scope

risc0-sys 1.5.0 vendored verbatim (720 K, 52 files) except the one cast, wired through the [patch.crates-io] block that already carries risc0-zkvm. The warning is repeated in prover/Cargo.toml so it is visible at the wiring, not only here.

…ERGE YET)

Prepared for testing on a B200. NOT verified on hardware -- see the
warning in prover/Cargo.toml and the last section below.

THE BUG, confirmed present in OUR version (risc0-sys 1.5.0, not just
the 5.0.0-rc.1 upstream filed against):

    kernels/zkp/cuda/supra/poseidon2.cuh:151
    void _poseidon2_rows(..., uint32_t dim_x, uint32_t dim_y) {
      uint32_t i = 0;
      for (uint32_t j = 0; j < CELLS_RATE; j++, i++)
        cells[j] = i < dim_y ? matrix[i * dim_x] : fr_t{0};

`i` is the column index and `dim_x` the domain, both uint32_t, so
`i * dim_x` runs in 32 bits and wraps once column*domain crosses 2^32.
The kernel then reads the wrong cell and commits a hash over the wrong
data -- with no panic and no error. The GPU returns a valid-looking seal
that fails verification.

WHY WE CARE. The domain is 2^po2 * 4, so the column threshold HALVES
with every po2 step: ~256 columns at po2 22, ~128 at po2 23. On
2026-08-25 a stock rv32im aggregate ran fine at po2 22 (~1 fault in 750
segments, a separate issue) and failed DETERMINISTICALLY at po2 23 --
segment 0, three attempts of three, with exactly the symptom above.

That also explains why #3773 was previously refuted as the cause of
hazync#119: at po2 22 it genuinely does not trigger.

Reported upstream on risc0#3773 with the arithmetic and a reproducer.

WHY THIS IS NOT MERGED. Widening a multiply cannot change behaviour
below the threshold -- the product already fit -- so po2 <= 22 is
unaffected by construction. But "by construction" is not "measured",
and this sits in the path that produces SHIPPED proofs. It needs a po2
23 run on a real card to show the fix works, and a po2 22 run to show
nothing else moved. Both boxes were deleted; a B200 is being arranged.
defenwycke added a commit that referenced this pull request Aug 26, 2026
…ade to (#188)

E6 was specified as a 1-D sweep of worker processes at fixed po2. That cannot
answer the question, because processes and po2 trade against each other through
VRAM: L40S at po2 22 peaks at ~41 GB of a 46 GB card -- 89% -- so there is very
likely no room for a second process at po2 22 at all, on the exact card a fleet
would be built from.

The removal note also carries a skimmed detail: dropping the in-process pipeline
"also unblocked po2 22, worth 1.15x on chunks and 1.42x on the aggregate". po2 22
is worth more than either scheduling fix, so anything that spends po2 22 to buy
parallelism is probably a bad trade.

And the 1.20x worker-process figure has NO recorded conditions -- neither the
board nor prover_impl.rs says at what po2 or on which card. If it was taken at
po2 20 it is not comparable to a po2 22 baseline at all. E6 now sweeps
processes x po2 and records peak VRAM for every cell INCLUDING cells that OOM,
because the failures are what separate contention from memory.

This also re-gates E9. If the optimum is 3 processes at po2 20, processes already
fill the idle time and E9 is dead. If it is 1 process at po2 22 -- VRAM-bound,
GPU still 65% idle -- pipelining is the only thing that can fill it, because it
shares one context and one allocation.

E10 (a newer risc0) is CLOSED, checked rather than assumed:

- latest stable is 3.0.6, whose entire content is "Repair Rust 1.97.0 guest build
  with heap-embedded-alloc". We do not enable that feature and we pin Rust
  1.94.1, so 3.0.6 is a literal no-op here.
- 5.0.0-rc.1 looks newer and is not. crates.io publish order is
  3.0.4, 5.0.0-rc.1, 3.0.5, 3.0.6 -- the RC PREDATES our own pin.
- upstream is dormant: five commits in three months, all housekeeping. risc0#3781
  (the CUDA overflow fix blocking po2 23) is still open. Our #3798 and #3799 have
  zero comments.

Three consequences, two of which reverse advice given elsewhere in this repo:
#182's vendored fix is permanent rather than a stopgap; the vendor tree's
"keep it re-appliable because risc0 moves" premise no longer holds; and E9 would
be a fork rather than an upgrade, which raises its cost.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant