Skip to content

dash(replay): incremental SML merkleRootMNList in the fold self-check - #1290

Draft
frstrtr wants to merge 1 commit into
integrator/dashd-cut-simd-sha256from
integrator/dashd-cut-incremental-sml-root
Draft

dash(replay): incremental SML merkleRootMNList in the fold self-check#1290
frstrtr wants to merge 1 commit into
integrator/dashd-cut-simd-sha256from
integrator/dashd-cut-incremental-sml-root

Conversation

@frstrtr

@frstrtr frstrtr commented Aug 19, 2026

Copy link
Copy Markdown
Owner

What

Port dashd's incremental SML merkleRootMNList into the daemonless fold self-check.

The profiled dominant cost of the full-history fold was DmlFoldEngine::compute_sml_root (replay_fold_engine.hpp) rebuilding all ~4900 CSimplifiedMNListEntry, serializing + SHA256d-hashing each leaf, sorting and merkling the whole tree on every block, driven from the per-block on_folded self-check.

dashd's CDeterministicMN caches its SML entry hash and CDeterministicMNListDiff re-applies only changed entries. This mirrors that.

How

Two-level cache on the fold engine's SML view:

  • m_leaf_hash_cache (proTxHash -> CalcHash) invalidated only at the mutation sites that touch an SML-serialized field:

    • confirmedHash (pass 1)
    • netInfo / platform fields / Revive isValid flip (ProUpServTx)
    • keyIDVoting + operator-key reset/ban (ProUpRegTx)
    • operator-field reset + ban (ProUpRevTx)
    • the PoSe punish that crosses into a ban (isValid flip)
    • structural add (ProRegTx) / remove (collateral replace + collateral spend) mark the set dirty

    The DecreaseScores pass and the nLastPaidHeight / nConsecutivePayments payee bookkeeping never dirty — those fields are not under CalcHash.

  • m_sml_tree, a cached merkle tree over the memcmp-sorted leaf hashes. On a field-only block (leaf set unchanged, positions stable) each changed leaf updates the O(log N) nodes on its path.

Merkle: hybrid, not pure O(log N). A structural block shifts every sorted position after the mutation, which reshuffles all pairings in the positional duplicate-last-on-odd Bitcoin/Dash merkle, so an O(log N) path update is unsafe across structural shifts. The tree is therefore rebuilt from the (still-valid) leaf-hash cache on structural blocks, and path-updated on field-only blocks. The leaf-hash cache — the dominant cost — is reused in both.

Safety

  • The self-check still compares the incrementally-maintained root to the block's committed cbTx merkleRootMNList and poisons on mismatch — unchanged.
  • compute_sml_root_full() retains the cache-independent full recompute as the oracle.
  • A missed invalidation is fail-closed: it can only make the incremental root diverge from the committed root, which hard-stops the fold; it can never serve a wrong list silently.
  • BLS-agnostic change (SML CalcHash is SHA256d over opaque bytes).

Tests (test_dash_replay_fold, target test_dash_mn_state)

  • WindowIncrementalEqualsFullEqualsHandBuilt — synthetic window exercising ProRegTx add, ProUpRevTx revoke, operator-key change, two collateral spends; incremental root == full recompute == independently hand-built root at every block.
  • RealCaseDBanIncrementalEqualsFull (h=2516412, ~2972 MNs, isValid flip false) and RealReviveAndQuietSequenceIncrementalEqualsFull (h=2516756..2516760, ~4900 MNs, revive isValid flip true + confirmedHash + quiet) — incremental == full on real mainnet bytes.
  • MissedInvalidationPoisonsAtSelfCheck — a deliberately-stale cache yields ROOT MISMATCH and a sticky HARD STOP.

The whole pre-existing synthetic + real fold suite folds through the incremental path unchanged (134 passed / 1 env-gated skip). Regression suites green: simplifiedmns (10), quorum_root (11), conformance (56), smldiff (6), sml_simd_sha256 (1). Built C2POOL_DASH_BLS=ON; dashbls link guard PASS.

Base

Stacked on integrator/dashd-cut-simd-sha256 (@aa0b3e2d), which sits on the combined integrator/dashd-cut-combined-fetch-fold (@8480d28e).

DRAFT — do not merge; review only.

The profiled dominant cost of the daemonless fold was
DmlFoldEngine::compute_sml_root rebuilding ALL ~4900
CSimplifiedMNListEntry, serializing + SHA256d-hashing each leaf, sorting
and merkling the whole tree on EVERY block, driven from the per-block
on_folded self-check.

Port dashd CDeterministicMNListDiff's incremental discipline: cache each
entry's SML leaf hash and re-hash only the entries a block actually
changed. Two-level cache on the fold engine's SML view:

  * m_leaf_hash_cache (proTxHash -> CalcHash) invalidated ONLY at the
    mutation sites that touch an SML-serialized field --- confirmedHash
    (pass 1), netInfo/platform/revive (ProUpServTx), keyIDVoting +
    operator-key reset/ban (ProUpRegTx), operator reset + ban
    (ProUpRevTx), and the PoSe punish that CROSSES into a ban (isValid
    flip). The DecreaseScores pass and the nLastPaidHeight /
    nConsecutivePayments payee bookkeeping never dirty --- those fields
    are not under CalcHash. Structural add (ProRegTx) / remove
    (collateral replace + collateral spend) mark the set dirty.

  * m_sml_tree, a cached merkle tree over the memcmp-sorted leaf hashes.
    On a field-only block (leaf set unchanged, positions stable) each
    changed leaf updates the O(log N) nodes on its path. A structural
    block shifts every sorted position after the mutation, which
    reshuffles all pairings in the positional duplicate-last-on-odd
    Bitcoin/Dash merkle, so O(log N) is UNSAFE there and the tree is
    rebuilt from the (still-valid) leaf-hash cache. HYBRID:
    path-update on field-only blocks, recompute-from-cached-leaves on
    structural blocks.

The self-check still compares the incrementally-maintained root to the
block's committed cbTx merkleRootMNList and poisons on mismatch ---
UNCHANGED. compute_sml_root_full() keeps the cache-independent full
recompute as the oracle. A missed invalidation is fail-CLOSED: it can
only make the incremental root diverge from the committed root, which
hard-stops the fold; it can never serve a wrong list silently.

KATs (test_dash_replay_fold, target test_dash_mn_state):
  * WindowIncrementalEqualsFullEqualsHandBuilt --- synthetic window
    exercising ProRegTx add, ProUpRevTx revoke, operator-key change and
    two collateral spends; incremental root == full recompute ==
    independently hand-built root at every block.
  * RealCaseDBanIncrementalEqualsFull (h=2516412, ~2972 MNs, isValid
    flip false) and RealReviveAndQuietSequenceIncrementalEqualsFull
    (h=2516756..2516760, ~4900 MNs, revive isValid flip true +
    confirmedHash + quiet) --- incremental == full on real mainnet bytes.
  * MissedInvalidationPoisonsAtSelfCheck --- a deliberately-stale cache
    yields ROOT MISMATCH and a sticky HARD STOP.

The whole pre-existing synthetic + real suite folds through the
incremental path unchanged; simplifiedmns / quorum_root / conformance /
smldiff / sml_simd_sha256 regression suites green; build C2POOL_DASH_BLS=ON.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant