Skip to content

GPU experiment runbook: everything the free tier could not settle - #191

Merged
defenwycke merged 2 commits into
mainfrom
docs/gpu-runbook
Aug 26, 2026
Merged

GPU experiment runbook: everything the free tier could not settle#191
defenwycke merged 2 commits into
mainfrom
docs/gpu-runbook

Conversation

@defenwycke

Copy link
Copy Markdown
Contributor

Everything achievable without a card is now done — Tier 0 (E1–E4) measured, E5's chunk half
measured, E10 closed, E11's free half measured
. This collects what's left, ordered so nothing
gets wasted.

The ordering is the point

  1. Prove 16 chunks first. Three later experiments consume the receipts, and regenerating them
    is the expensive part.
  2. E6 (processes × po2) before E9. E6 decides whether E9 is worth attempting: if worker
    processes fill the 65% GPU idle, E9 is dead; if VRAM caps them at one process at po2 22, only
    pipelining can fill it.
  3. E7/E8 — po2 23 with 🚧 Vendor risc0-sys for upstream #3781's u32 overflow fix (needs B200 validation) #182's patch, newer driver. Take a po2 22 control on the same binary
    first, or the result means nothing.
  4. E5 remainder, E11 GPU half, then E9 — in that order of cost.

E11's free half came back clean, and isolates the GPU question

quantity N=4 N=16 invariant?
total execute cycles 14,341,307,904 14,341,307,904 exactly (also N=128)
total segments 3,411 ~3,404 yes (0.2%)
total wire bytes 949.9 MB ~949 MB yes
straggler 1.00× 1.00× 1.00× through N=64

Chunking is free all the way to the aggregate. Segments per chunk halve almost exactly on every
doubling (805 → 426 → 213 → 104 → 49), and HAZYNC_CHUNKS=16 is confirmed as a default nobody
revisited — the code says so outright.

That leaves E11's GPU half with a single unknown: does assumption resolution cost grow
sublinearly in N? If so, more chunks is close to free throughput and 16 is leaving parallelism
unused.

⚠ The N=4 total is measured across all four chunks. Extrapolating from chunk 0 gives 3,220 and
is wrong — chunk 0 is the smallest (805 against 864/871/871).

Guards, because each caught a real false result today

  • Read METHOD_ID with host method-id, never by scraping the binary —
    strings | grep -oE '[0-9a-f]{64}' returns 000000000019d668…, the Bitcoin genesis hash,
    which is 64 hex characters and entirely plausible.
  • Record METHOD_ID per arm. A build that silently didn't happen returns identical numbers and
    reads as "no effect".
  • Run check-versions.sh only after git add — it enumerates with git ls-files and cannot
    see an unstaged file, so it passes by not looking.
  • Never quote a local build id in a doc. A local build never matches the canonical id anyway
    (hazync#88 — the build path is baked into panic metadata).

What NOT to spend the card on

Faster/bigger cards, cheap tiers, wire compression, NDEBUG, ECMULT_WINDOW, C/C++ LTO, a newer
risc0 — each closed, each with the number that closed it.

And one item that may outrank E9

#139's packer refit — worth 2.36×, host-only, no METHOD_ID, no fidelity cost, and a
prerequisite for #139 rather than a follow-up. That's now built in #190.

Docs only.

Everything achievable without a card is now done -- Tier 0 (E1-E4) measured, E5's
chunk half measured, E10 closed, E11's segment half measured. This collects what
is left, in an order that respects what produces what.

The ordering is the point. Three later experiments consume chunk RECEIPTS, which
only exist after proving, so §1 produces them first and says to keep them --
regenerating is the expensive part. §2 (E6) comes before §6 (E9) because E6
DECIDES whether E9 is worth attempting: if worker processes fill the 65% GPU idle
then E9 is dead, and if VRAM caps them at one process at po2 22 then only
pipelining can fill it.

Records the guards that made the free tier trustworthy, because each one caught a
real false result today:

- read METHOD_ID with `host method-id`, never by scraping the binary --
  `strings | grep -oE '[0-9a-f]{64}'` returns the BITCOIN GENESIS HASH, which is
  64 hex characters and entirely plausible
- record METHOD_ID per arm: a build that silently did not happen returns
  identical numbers and reads as "no effect"
- run check-versions.sh only AFTER `git add` -- it enumerates with `git ls-files`
  and cannot see an unstaged file, so it passes by not looking

Also names what NOT to spend the card on, with the reason each is closed: faster
cards, cheap-card tiers, wire compression, NDEBUG, ECMULT_WINDOW, C/C++ LTO, and
a newer risc0.

And flags that #139's packer refit is arguably ahead of E9 in value: worth 2.36x,
host-only, no METHOD_ID, no fidelity cost, and a prerequisite for #139 rather than
a follow-up -- without it #139 delivers 2.95x instead of 6.95x and the fidelity is
spent either way.
Chunking is free ALL THE WAY to the aggregate, not merely at the execute level.
Total segments (3,411 vs ~3,404) and total wire bytes (949.9 MB vs ~949 MB) are
invariant between N=4 and N=16, alongside the exactly-invariant cycle count and a
straggler that holds at 1.00x through N=64.

The N=4 total is measured across all four chunks. Extrapolating from chunk 0
alone gives 3,220 and is WRONG -- chunk 0 is the smallest of the four (805
against 864/871/871) -- which is the second time today that extrapolating a block
total from chunk 0 produced a trend that was not there.

That isolates E11's GPU half to a single unknown: whether assumption resolution
cost grows sublinearly in N.
@defenwycke
defenwycke merged commit e530d4f into main Aug 26, 2026
6 checks passed
@defenwycke
defenwycke deleted the docs/gpu-runbook branch August 26, 2026 16:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant