dash(replay): incremental SML merkleRootMNList in the fold self-check - #1290
Draft
frstrtr wants to merge 1 commit into
Draft
dash(replay): incremental SML merkleRootMNList in the fold self-check#1290frstrtr wants to merge 1 commit into
frstrtr wants to merge 1 commit into
Conversation
The profiled dominant cost of the daemonless fold was
DmlFoldEngine::compute_sml_root rebuilding ALL ~4900
CSimplifiedMNListEntry, serializing + SHA256d-hashing each leaf, sorting
and merkling the whole tree on EVERY block, driven from the per-block
on_folded self-check.
Port dashd CDeterministicMNListDiff's incremental discipline: cache each
entry's SML leaf hash and re-hash only the entries a block actually
changed. Two-level cache on the fold engine's SML view:
* m_leaf_hash_cache (proTxHash -> CalcHash) invalidated ONLY at the
mutation sites that touch an SML-serialized field --- confirmedHash
(pass 1), netInfo/platform/revive (ProUpServTx), keyIDVoting +
operator-key reset/ban (ProUpRegTx), operator reset + ban
(ProUpRevTx), and the PoSe punish that CROSSES into a ban (isValid
flip). The DecreaseScores pass and the nLastPaidHeight /
nConsecutivePayments payee bookkeeping never dirty --- those fields
are not under CalcHash. Structural add (ProRegTx) / remove
(collateral replace + collateral spend) mark the set dirty.
* m_sml_tree, a cached merkle tree over the memcmp-sorted leaf hashes.
On a field-only block (leaf set unchanged, positions stable) each
changed leaf updates the O(log N) nodes on its path. A structural
block shifts every sorted position after the mutation, which
reshuffles all pairings in the positional duplicate-last-on-odd
Bitcoin/Dash merkle, so O(log N) is UNSAFE there and the tree is
rebuilt from the (still-valid) leaf-hash cache. HYBRID:
path-update on field-only blocks, recompute-from-cached-leaves on
structural blocks.
The self-check still compares the incrementally-maintained root to the
block's committed cbTx merkleRootMNList and poisons on mismatch ---
UNCHANGED. compute_sml_root_full() keeps the cache-independent full
recompute as the oracle. A missed invalidation is fail-CLOSED: it can
only make the incremental root diverge from the committed root, which
hard-stops the fold; it can never serve a wrong list silently.
KATs (test_dash_replay_fold, target test_dash_mn_state):
* WindowIncrementalEqualsFullEqualsHandBuilt --- synthetic window
exercising ProRegTx add, ProUpRevTx revoke, operator-key change and
two collateral spends; incremental root == full recompute ==
independently hand-built root at every block.
* RealCaseDBanIncrementalEqualsFull (h=2516412, ~2972 MNs, isValid
flip false) and RealReviveAndQuietSequenceIncrementalEqualsFull
(h=2516756..2516760, ~4900 MNs, revive isValid flip true +
confirmedHash + quiet) --- incremental == full on real mainnet bytes.
* MissedInvalidationPoisonsAtSelfCheck --- a deliberately-stale cache
yields ROOT MISMATCH and a sticky HARD STOP.
The whole pre-existing synthetic + real suite folds through the
incremental path unchanged; simplifiedmns / quorum_root / conformance /
smldiff / sml_simd_sha256 regression suites green; build C2POOL_DASH_BLS=ON.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Port dashd's incremental SML
merkleRootMNListinto the daemonless fold self-check.The profiled dominant cost of the full-history fold was
DmlFoldEngine::compute_sml_root(replay_fold_engine.hpp) rebuilding all ~4900CSimplifiedMNListEntry, serializing + SHA256d-hashing each leaf, sorting and merkling the whole tree on every block, driven from the per-blockon_foldedself-check.dashd's
CDeterministicMNcaches its SML entry hash andCDeterministicMNListDiffre-applies only changed entries. This mirrors that.How
Two-level cache on the fold engine's SML view:
m_leaf_hash_cache(proTxHash -> CalcHash) invalidated only at the mutation sites that touch an SML-serialized field:confirmedHash(pass 1)netInfo/ platform fields /ReviveisValid flip (ProUpServTx)keyIDVoting+ operator-key reset/ban (ProUpRegTx)The
DecreaseScorespass and thenLastPaidHeight/nConsecutivePaymentspayee bookkeeping never dirty — those fields are not underCalcHash.m_sml_tree, a cached merkle tree over the memcmp-sorted leaf hashes. On a field-only block (leaf set unchanged, positions stable) each changed leaf updates the O(log N) nodes on its path.Merkle: hybrid, not pure O(log N). A structural block shifts every sorted position after the mutation, which reshuffles all pairings in the positional duplicate-last-on-odd Bitcoin/Dash merkle, so an O(log N) path update is unsafe across structural shifts. The tree is therefore rebuilt from the (still-valid) leaf-hash cache on structural blocks, and path-updated on field-only blocks. The leaf-hash cache — the dominant cost — is reused in both.
Safety
merkleRootMNListand poisons on mismatch — unchanged.compute_sml_root_full()retains the cache-independent full recompute as the oracle.CalcHashis SHA256d over opaque bytes).Tests (
test_dash_replay_fold, targettest_dash_mn_state)WindowIncrementalEqualsFullEqualsHandBuilt— synthetic window exercising ProRegTx add, ProUpRevTx revoke, operator-key change, two collateral spends; incremental root == full recompute == independently hand-built root at every block.RealCaseDBanIncrementalEqualsFull(h=2516412, ~2972 MNs, isValid flip false) andRealReviveAndQuietSequenceIncrementalEqualsFull(h=2516756..2516760, ~4900 MNs, revive isValid flip true + confirmedHash + quiet) — incremental == full on real mainnet bytes.MissedInvalidationPoisonsAtSelfCheck— a deliberately-stale cache yieldsROOT MISMATCHand a sticky HARD STOP.The whole pre-existing synthetic + real fold suite folds through the incremental path unchanged (134 passed / 1 env-gated skip). Regression suites green:
simplifiedmns(10),quorum_root(11),conformance(56),smldiff(6),sml_simd_sha256(1). BuiltC2POOL_DASH_BLS=ON; dashbls link guard PASS.Base
Stacked on
integrator/dashd-cut-simd-sha256(@aa0b3e2d), which sits on the combinedintegrator/dashd-cut-combined-fetch-fold(@8480d28e).DRAFT — do not merge; review only.