chore: Concurrent state load - #2466
Conversation
| // because loading has long synchronous sections (RocksDB opens, MMR hashing, SMT top | ||
| // reconstruction) that would serialize if polled from a single task. | ||
| let (blockchain, account_tree, nullifier_tree, forest) = tokio::try_join!( |
There was a problem hiding this comment.
Can we not use our task set for this? This is really difficult to read.
There was a problem hiding this comment.
The Tasks struct is designed around long-running service tasks. It should probably be renamed to reflect its specific use case.
The main issue is that Tasks only allows for futures returning anyhow::Result<()> which means we can't coalesce results via join. Even if it was Result<T> it wouldn't help here because every future returns a different type.
|
Could we also focus on getting the changelog entries more user friendly? At the moment they're written as for us - overly technical and specific. No operator/user knows (or cares) about the store, forests, trees, or concurrency etc; as an example this could just be: |
Summary
Relates to #1697.
Why:
verify_account_state_forest_consistency(6.9s, ~42%), which checked every public accountserially. The remaining structures (MMR, account tree, nullifier tree, forest) also loaded
strictly one after another despite living in independent storages, and the three blocking
RocksDB opens ran directly on the async runtime.
How:
but each page's per-account checks (storage-header decode + forest lookups) now run through
rayon. The forest parameter gained a
Syncbound.&Dbinstead of&mut Db(every underlying query is
&self; the&mutthreading was an API artifact), theDbisArc-wrapped up front, and the MMR, account tree, nullifier tree, and forest (+ its verify)load in four spawned tasks joined with
try_join!. Spawned tasks rather than joined futuresbecause loading has long synchronous sections (RocksDB opens, MMR hashing, SMT top
reconstruction) that would serialize if polled from a single task.
spawn_blocking_in_current_span, so they executeconcurrently, off the async workers, and remain traced. Panics from load tasks are resumed
on the caller so they propagate as panics rather than being masked as errors.
is safe because
load_mmr's existing consistency check pins the MMR to that same header.This is fine during startup (the write worker and gRPC servers are not up yet); if loading
ever runs concurrently with live traffic, those sections should move to the blocking pool.
open) needs a flush API upstream in miden-crypto, which does not currently expose one.
Results:
Local benchmark (
benchmark-store load-state, toy-scale store): the three RocksDB opens nowstart simultaneously (~22ms of summed open time → ~7.5ms wall clock), warm loads dropped from
~18–20ms to ~12–13ms.
Warm-iteration trace — note the identical start timestamps on the three RocksDB opens,
which previously ran strictly one after another:
Against the serial baseline on the same machine: warm loads dropped from ~18–20ms to
~12–13ms, with ~22ms of summed open time (7.1 + 7.4 + 7.5ms) collapsing to ~7.5ms of wall
clock. The tree/MMR loads and the forest verify also overlap with each other now. The
absolute numbers are toy-scale (200 accounts); on testnet data the same structure applies to
the 6.9s forest scan (rayon-parallel per page) and the three multi-second opens.
Changelog