feat: AO-Core native Arweave validation - #1065
Open
samcamwilliams wants to merge 11 commits into
Open
Conversation
The RandomX and secp256k1 submodules, the RandomX/VDF NIF sources, and the
records and constants the consensus code reads. `native/lib/Makefile' inits the
RandomX submodule itself and builds only the `randomx' target, so a clone
without `--recursive' still works and a developer who does not care about
Arweave pays about six seconds.
`src/core/include/ar.hrl' is added to, never reordered or re-valued, and gains
an include guard because `ar_inflation.hrl' includes it too. Note that
`hb.hrl' includes `ar.hrl', so `#block{}', `?KiB'..`?TiB' and
`?CHECKPOINT_DEPTH' are now in every compilation unit's transitive closure.
That is what keeps the vendored modules diffable against upstream, whose own
`ar.hrl' is the file they include; every pre-existing `src/core' module was
recompiled against the old and new headers and the diagnostics diffed, with
none new. See the vendoring section of the PR description.
The VDF verification kernel is chosen at NIF load, but installed only after it
reproduces the OpenSSL kernel bit for bit on the running machine across six
iteration counts including the salt carry (`vdf.cpp:vdf_set_verify_sha2/1').
Hardware that fails keeps OpenSSL and logs which it installed. This is
consensus code: a kernel differing by one byte would split the node off the
network, so a CPU feature bit is not enough on its own.
Thirty-four `ar_*' modules, keeping upstream's style -- tabs, naming, clause layout -- so the tree stays diffable against `ArweaveTeam/arweave' and can be upgraded in place. Eight of the ten `ar_*' modules HyperBEAM already had move here by `git mv'; `ar_format' and `ar_timestamp' stay behind because they are HyperBEAM's own, not Arweave consensus code. Six modules are code-identical to upstream. Deviations are described in `VENDOR.md' and the behaviour-changing ones carry a `%% VENDOR:' note at the site; that marker coverage is incomplete and `VENDOR.md' measures how incomplete rather than claiming otherwise. Two defects are fixed rather than carried, because a validator cannot compute the wrong hash. HyperBEAM's own `ar_tx' built the v1 denominated signature preimage as a 7-element deep-hash list where upstream builds 8, and `ar_deep_hash' tags a list with its length, so every byte differed -- reachable today, since `verify_denomination/4' accepts denomination 1 against a block denomination of 1. Upstream's `ar_mine_randomx:init_light/2' had its arguments transposed, dead code there and fatal here because light mode is the default. `ar_wallet' additionally restores three pieces upstream has and HyperBEAM's fork had dropped. Those are fidelity changes, not fixes: no input is known that behaves differently, and `verify' is on every node's HTTP-signature path, so the hunks are kept as small as upstream's own.
…essages
Seven devices, each usable on its own, and five `lib_*' bridges that convert
between the vendored records and dashed-binary-key messages so that no tuple or
record crosses a device boundary.
~arweave-merkle@2.9 offset-indexed Merkle path validation
~arweave-block-index@2.9 the {indep-hash, weave-size, tx-root} weave index
~arweave-wallets@2.9 the account tree
~arweave-tx@2.9 transaction validation
~arweave-vdf@2.9 the nonce limiter
~arweave-spora@2.9 storage proofs: recall ranges, packing, H0/H1/H2
~arweave-block@2.9 the block state transition
`~arweave-block@2.9/apply' takes a chain state and the next block and returns
the state it produces. `checks/5' is a flat `maybe' of 32 sequential checks
ending in the account transition. The set is the *union* of upstream's stages,
not just `ar_node_utils:validate/6': that function performs no PoW, no PoA, no
VDF and no RandomX, so a node implementing only it would look complete and
check almost nothing.
Reproducing `wallet_list' is the strongest property here. It is a
Merkle-Patricia root over every Arweave account after the block is applied, so
matching it means reproducing every transfer, fee, mining reward, endowment
movement and denomination step to the winston. Mainnet is the oracle.
Order is deliberate: cheap field comparisons run before the expensive
cryptography, so a malformed block is refused in microseconds rather than after
a minute of VDF work. `apply/3' refuses to carry forward a transition that ran
with no account tree unless `arweave-require-accounts' is explicitly false --
`validate/3' still answers for a single block in either mode and names which it
ran in, but what `apply' returns becomes the state the next block is checked
against, where the distinction would be lost.
Every key is reachable by anyone who can reach the node, so the request is
treated as hostile throughout: identifiers are checked against the base64URL
alphabet before they can name a store key, peer bodies cannot allocate on their
own header, peer-chosen fields are read through the record rather than a
vendored accessor that can answer an atom, and nothing deserialises a
caller-supplied structure. See the PR description for the full list.
The device an operator drives. Production wiring is one call:
POST /~cron@1.0/every
cron-path: /~arweave@2.9/sync
interval: 30-seconds
`bootstrap' is the only moment anything is trusted, and it trusts exactly one
block hash -- either `arweave-checkpoint-block' from the node message, or the
block every trusted peer reports at `arweave-checkpoint-depth' below the tip.
Both are read from the node message only. A request cannot supply either: this
key is unauthenticated and the checkpoint *is* the chain's trust root, so
honouring it from a caller would let a stranger choose what the node believes
Arweave to be. It is also refused once a chain exists, before contacting any
peer, since otherwise it is a repeatable trust reset.
From that hash everything else is derived and checked -- the block index
against the block's `hash-list-merkle', the account tree against its
`wallet_list', the histories against their hashes -- and every header admitted
has its `indep-hash' recomputed rather than read from the response. The account
tree anchors at the checkpoint or the bootstrap fails, rather than producing a
node whose early blocks have no accounts to spend from.
Fork choice is upstream's exactly: strictly greater `cumulative_diff' wins, an
equal one keeps the incumbent, no secondary tie-break, and a branch is eligible
only at or above `TipHeight - ?CHECKPOINT_DEPTH + 1'. Eligibility is measured
against the incumbent rather than the running winner, so the outcome does not
depend on the order candidates arrive in. An invented tie-break would be a
consensus divergence, so there is none.
`validated' never falls back to a peer -- `not-validated' means this node has
not verified that block, which is more useful than a block fetched from
elsewhere to fill the gap.
`dev_arweave_block_cache' is renamed `dev_arweave_cache': the Forge packager
assigns a helper to its longest matching `dev_*' prefix, so once
`dev_arweave_block' exists as a device root the old name would be packaged
under it and `dev_arweave''s calls would fail with `undef'.
The fixtures are real mainnet blocks, transactions, chunks and histories, fetched from public peers at heights chosen to cover the awkward consensus boundaries -- retargets, price adjustments, VDF difficulty changes, and a pair whose VDF step range crosses an entropy reset. `.gitignore' is given a `!test/fixtures/**' negation so they do not depend on already being in the index. `hb_arweave_live_sync' drives a node against live mainnet exactly as production does -- it bootstraps, starts `~cron@1.0/every' pointed at `/~arweave@2.9/sync', and then only watches. It validates nothing itself, and reads the tip back through the node's own HTTP API rather than from process state, comparing against a peer's `/info', so it cannot make a run look more in sync than it was. Run output is NOT committed. `evidence/' is ignored: acceptance logs are megabytes of console text that nobody can re-derive from the tree, and a log living in a repository invites being read as a property of the code rather than of one execution. `scripts/nif_check.erl' and `scripts/secp_check.erl' are the NIF and secp256k1 verification harnesses the vendoring notes cite -- reproducible checks, not records, so they sit with the other tooling.
What the devices are, how to point a node at mainnet, and what the keys answer. Wired into `mkdocs.yml' alongside the other device pages.
The reward history and the block-time history were carried as whole wire blobs under a `body' key, rewritten in full for every block: 2.7 MB per block for the reward history alone, and no sharing between two branches that agree about everything below a fork. Each element is now one immutable message whose `previous' links to the element before it, so a block writes one entry and shares the whole tail with the state it extended. `~arweave-history@2.9' is the surface over them: `take' follows one link per entry it returns, and `from-binary' turns the bytes a peer serves into the stored form. Both lengths are bounded by the vendored rule that reads them, and the materialised window is memoised against the identifier of the entry it ends at, so advancing a history costs the entry it adds rather than a walk of the window.
The comments around the block, block-index and wallet key checks explained at length what an unchecked key reaches and what it returns. That belongs in the history of the change that introduced the check, not in the code: a reader needs to know what the check guarantees, and the guarantee is what stays true. Rewritten in the present tense as invariants, keeping the one fact that is genuinely load-bearing for a future caller -- `hb_util:decode/1' is the unchecked decoder, so a length test alone is not a check. Test targets are named neutrally. The checks themselves, and every test asserting them, are unchanged.
The validator's durable state becomes one `arweave-block@2.9' message per block, linked into a chain, carrying placements, recording the checks that established it, and published in an order whose last step is what makes a block's presence mean "finished". Chain state merges into the block message, so `lib_arweave_state' reads one; `previous' becomes a device key rather than a stored link, because a block names its parent before that parent has been downloaded, and the anchor window becomes a walk of it. Validation is grouped under eleven stable names, and every stored block records the ones that established it under `validation/checks'; `profile' and `verify' select among them, with unknown names and sets that omit a check another reads from both refused rather than silently narrowed. Adds `lib_arweave_placement' and the `~arweave@2.9/placement' key: where a source transaction occurs in the chain, derived from a validated block and the same size-tagged list its transaction root is built over. Publication is ordered -- transactions, then placements and byte offsets, then components and the block, then the hash -- so an interrupted pass leaves content-addressed messages nothing points at, which the next pass rewrites at the same identifiers. Both carried histories become persistent linked lists of immutable entries, one message per element, so extending one costs a single write and every branch that reached the same element shares its tail. The account tree and the block index each reach the version they were derived from. `~arweave@2.9/backfill' materialises blocks below the node's join, checked against the block index the tip carries. Settled transactions are announced on the `arweave-settled-transaction' hook, idempotently per block. `lib_arweave_paths' owns every durable name, and `hb_store_arweave' gains `read_location/3', with `no_store' as a clean no-op.
A transaction was kept in two forms, and a block was reachable under two names. Both are collapsed. The two transaction forms -- `~arweave-tx@2.9''s, spelling Arweave's wire names and carrying the signature as a field, and HyperBEAM's committed `tx@1.0' message -- were fetched together and carried side by side, so the checks and the store could in principle describe different transactions. There is now one: `tx@1.0' carries what the consensus checks read, and the codec is what reconciles the two spellings. The record and the message name one sender, so a transaction's committer is the same fact whichever layer asks. `lib_arweave_paths' names a block by its hash and nothing else, so a block has one durable name and a walk of the chain cannot leave the namespace the node validated into. Supersedes `decisions/two-forms-of-a-transaction.md`, which recorded the reasoning for keeping both; `decisions/one-form-of-a-transaction.md` records why one is enough.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Introduces a full pull-based, post-2.9 Arweave block validation subsystem within HyperBEAM, enabling nodes to independently verify the Arweave network.
This feature integrates a significant portion of Arweave's consensus logic, providing a robust and flexible framework for trustless validation.
Key Features
~arweave-*@2.9devices. These are sliced into logic units such that each of Arweave's base primitives can be utilized freely in AO-Core applications.arweave-checkpoint-block) or established by agreement among trusted peers, matching the Arweave node's default fast-join mechanic.validatedendpoint for local verification. Does not implement the gossip network at this point.~arweave@2.9device.Technical Details
~arweave@2.9as the main synchronization device, orchestratingbootstrap,sync,tip, andvalidatedoperations.VENDORcomments in vendored code to track local adaptations and ensure accurate diffing against upstream sources.