diff --git a/evm/evm-parity/evm-compatibility.mdx b/evm/evm-parity/evm-compatibility.mdx index f994b82..1d6892e 100644 --- a/evm/evm-parity/evm-compatibility.mdx +++ b/evm/evm-parity/evm-compatibility.mdx @@ -37,7 +37,7 @@ Standard EVM tooling — viem, wagmi, ethers, Foundry, Hardhat — works on Sei | `eth_getTransactionCount` (nonce) | Supported | | | `eth_getCode` | Supported | | | `eth_getStorageAt` | Supported — differences | SSTORE cost is governance-adjustable; do not hard-code gas assumptions. [See Gas and Fees.](/evm/evm-parity/gas-and-fees) | -| `eth_getProof` | Supported — differences | Returns IAVL proof data rather than Ethereum Merkle Patricia Trie proofs. Proof verification logic must account for this. [See State Proofs.](/evm/evm-parity/state-proofs) | +| `eth_getProof` | Supported — differences | Returns IAVL proof data rather than Ethereum Merkle Patricia Trie proofs. Proof verification logic must account for this. Proofs resolve across supported store backends (classic IAVL, store/v2 memiavl, and other proof-capable queryable stores), so the method works across a broader range of node configurations. [See State Proofs.](/evm/evm-parity/state-proofs) | ## Blocks and Finality diff --git a/evm/evm-parity/gas-and-fees.mdx b/evm/evm-parity/gas-and-fees.mdx index f4665b5..5827302 100644 --- a/evm/evm-parity/gas-and-fees.mdx +++ b/evm/evm-parity/gas-and-fees.mdx @@ -53,6 +53,14 @@ const feeData = await provider.getFeeData(); + +### Effective Gas Price on Receipts + +For dynamic-fee (type 2) transactions, the transaction receipt's `effectiveGasPrice` field reports the **actual price charged** — `min(baseFee + maxPriorityFeePerGas, maxFeePerGas)` — not the fee cap. This matches standard EIP-1559 semantics, so clients such as ethers and hardhat see the same `effectiveGasPrice` behavior they expect from Ethereum. + +In practice, when your priority tip plus the base fee stays below `maxFeePerGas`, the receipt reports `baseFee + maxPriorityFeePerGas` rather than `maxFeePerGas`. Use the receipt's `effectiveGasPrice` (not `maxFeePerGas`) when computing what a transaction actually paid. + + ## SSTORE Cost The gas cost of `SSTORE` (writing to contract storage) is governance-adjustable on Sei. It is currently **72,000 gas** — the same on mainnet and testnet (see [Divergence from Ethereum](/evm/differences-with-ethereum#sstore-gas-cost)) — but treat that as the current value, not a constant: do not hard-code storage write estimates in your application. diff --git a/evm/evm-parity/state-proofs.mdx b/evm/evm-parity/state-proofs.mdx index dbe256a..8fa8462 100644 --- a/evm/evm-parity/state-proofs.mdx +++ b/evm/evm-parity/state-proofs.mdx @@ -25,6 +25,13 @@ If you are doing standard contract reads, event queries, or transaction lookups, ## Calling eth_getProof +### Storage key requirements + +Before calling `eth_getProof`, note two requirements Sei enforces on the `storageKeys` argument: + +- **Keys must be hex-encoded.** Each storage key must be a valid hex-encoded value (for example `0x0000000000000000000000000000000000000000000000000000000000000001`). Keys are decoded and left-padded to 32 bytes. A malformed, non-hex key is rejected with an `invalid storage key` error. Raw byte strings, which were previously accepted, no longer work. +- **At most 1024 keys per request.** A single proof request may include a maximum of 1024 storage keys. Requesting more returns a `too many storage keys` error. Split larger sets across multiple requests. + The call works through standard libraries: diff --git a/evm/evm-parity/transaction-types.mdx b/evm/evm-parity/transaction-types.mdx index 7f11981..badc49d 100644 --- a/evm/evm-parity/transaction-types.mdx +++ b/evm/evm-parity/transaction-types.mdx @@ -16,6 +16,13 @@ Sei supports most Ethereum transaction types. The one notable exception is blob | 2 | EIP-1559 | Fee market | Supported — base fee is not burned | | 4 | EIP-7702 | Set code | Supported | + +### Set Code (EIP-7702) Auth List Requirement + +Type 4 (EIP-7702) SetCode transactions must include a non-empty authorization list. A transaction with an empty or nil auth list is rejected during validation with the error `auth list cannot be empty`. + +Each authorization entry must also carry a valid (non-nil) chain ID. If you are constructing SetCode transactions directly, ensure at least one authorization is present before submitting. + ## Not Supported | Type | EIP | Name | Notes | diff --git a/evm/evm-parity/websocket.mdx b/evm/evm-parity/websocket.mdx index a1315a6..be7d876 100644 --- a/evm/evm-parity/websocket.mdx +++ b/evm/evm-parity/websocket.mdx @@ -114,3 +114,20 @@ const unwatch = client.watchEvent({ - Sei's instant finality means every block emitted over WebSocket is already final — no need to wait for additional confirmations before acting on an event. - Pending transaction subscriptions (`newPendingTransactions`) are supported at the RPC level but Sei does not guarantee Ethereum-style pending state visibility. + + + +## `newHeads` Under Autobahn Consensus + +When a node runs under Autobahn consensus, `eth_subscribe("newHeads")` notifications are delivered from an in-process notifier that publishes committed-block headers directly, rather than from the legacy consensus event bus. Subscribers still only observe headers for fully committed blocks, but the header payload differs from the legacy path in a few ways: + +- **`parentHash`, `receiptsRoot`, and `transactionsRoot` are returned as zero hashes** (`0x0000…0000`). The Autobahn block-execution path does not build a Tendermint-style hash chain, so there is no meaningful value to surface for these fields. +- **`stateRoot`** is sourced from the finalized block's `AppHash` (the post-execution application hash), rather than from a pre-execution header field. +- **`hash`** is the Autobahn block-header hash — the same value reported as `blockHash` by `eth_getBlockByNumber` and the receipt APIs, keeping `newHeads` consistent with the rest of the EVM RPC surface. +- **`gasUsed`** is an approximation (summed from per-transaction results) to keep the notification cheap. + +Because of these differences: + +- Subscribers that chain-validate the head stream by linking `parentHash` values cannot rely on `newHeads` under Autobahn and need a different mechanism. +- If you need exact `gasUsed` or the omitted hash fields, fetch the block explicitly with `eth_getBlockByNumber`. + diff --git a/evm/installing-seid-cli.mdx b/evm/installing-seid-cli.mdx index d0d726f..ab42067 100644 --- a/evm/installing-seid-cli.mdx +++ b/evm/installing-seid-cli.mdx @@ -65,7 +65,6 @@ Available Commands: add-wasm-genesis-message Wasm genesis subcommands blocktest run EF blocktest collect-gentxs Collect genesis txs and output a genesis.json file - compact Compact the application DB fully (only if it is a levelDB) config Create or query an application CLI configuration file debug Tool for helping with debugging your application ethreplay replay EVM transactions @@ -74,9 +73,7 @@ Available Commands: help Help about any command init Initialize private validator, p2p, genesis, and application configuration files keys Manage your application's keys - latest_version Prints the latest version of the app DB migrate Migrate genesis to a specified target version - prune Prune app history states by keeping the recent heights and deleting old heights query Querying subcommands rollback rollback cosmos-sdk and tendermint state by one height start Run the full node diff --git a/evm/precompiles/distribution.mdx b/evm/precompiles/distribution.mdx index 877bd1f..ecd1184 100644 --- a/evm/precompiles/distribution.mdx +++ b/evm/precompiles/distribution.mdx @@ -156,6 +156,10 @@ function setWithdrawAddress(address withdrawAddr) external returns (bool success **Gas Cost**: ~30,000 gas +**Recipient Validation**: The withdraw address must be allowed to receive external funds. The distribution module validates the recipient against the bank module's `CanSendTo` check, and the call reverts with `ErrInvalidRecipient` if it fails. In particular, an unassociated EVM-cast address (an EVM address that has no associated Sei account) is rejected, as are blocked module addresses. Ensure the target address is associated before setting it as a withdraw address. + +**Fallback on Invalid Stored Address**: If a previously set withdraw address later becomes unable to receive funds (for example, it becomes blocked or otherwise fails the `CanSendTo` check), the distribution module falls back to sending rewards to the delegator's own address rather than failing. Reward and commission withdrawals therefore always succeed even if the stored withdraw address is no longer valid. + **Example:** ```solidity diff --git a/evm/precompiles/json.mdx b/evm/precompiles/json.mdx index fdcd386..6541c0d 100644 --- a/evm/precompiles/json.mdx +++ b/evm/precompiles/json.mdx @@ -131,6 +131,10 @@ The JSON precompile has specific limitations for different data types: | extractAsBytesList | Arrays of strings/objects | Each element returned as bytes | + +**Value Length Limit:** `extractAsUint256` rejects value strings longer than 100 characters. If the numeric string extracted for the given key exceeds 100 characters, the call fails with `value string too long`. Ensure the numeric values you pass stay within this limit. + + ### Data Type Conversion Strategies ```typescript diff --git a/evm/precompiles/oracle.mdx b/evm/precompiles/oracle.mdx index 6ee1132..00e6082 100644 --- a/evm/precompiles/oracle.mdx +++ b/evm/precompiles/oracle.mdx @@ -6,4 +6,4 @@ keywords: ['oracle precompile', 'ethers.js', 'price feeds', 'exchange rates', 't --- **Address:** `0x0000000000000000000000000000000000001008` -**Deprecation Notice:** The native Sei Oracle is deprecated and will be shut off soon. We strongly recommend migrating to one of the third-party oracle providers, such as [Chainlink](/evm/oracles/chainlink), [Pyth](/evm/oracles/pyth-network), [Redstone](/evm/oracles/redstone), or [API3](/evm/oracles/api3). +**Retired as of v6.6:** The native Sei Oracle precompile has been retired. As of the v6.6 chain upgrade, both `getExchangeRates` and `getOracleTwaps` now revert with the error `oracle precompile is retired; oracle data queries are disabled` instead of returning data. Any contract that relies on on-chain oracle exchange rate or TWAP queries via this precompile will break after the upgrade. Migrate to one of the third-party oracle providers, such as [Chainlink](/evm/oracles/chainlink), [Pyth](/evm/oracles/pyth-network), [Redstone](/evm/oracles/redstone), or [API3](/evm/oracles/api3). diff --git a/evm/reference.mdx b/evm/reference.mdx index aef1579..d97649a 100644 --- a/evm/reference.mdx +++ b/evm/reference.mdx @@ -195,7 +195,7 @@ Every method below is also browsable interactively in the explorer above; this s **Supported.** Returns the EVM transaction matching the given hash, or null if not found. -**Sei-specific behavior:** Sees EVM transactions only; if the hash resolves to a non-EVM Cosmos tx it errors. Pending lookups read from the CometBFT mempool, not a geth txpool. Use the legacy sei_getTransactionByHash to also surface Cosmos txs with synthetic representations. +**Sei-specific behavior:** Sees EVM transactions only; if the hash resolves to a non-EVM Cosmos tx it errors. Pending lookups read from the CometBFT mempool (not a geth txpool) via a direct EVM-hash index lookup rather than scanning paginated unconfirmed-tx pages, so pending transactions are resolved efficiently by hash. Use the legacy sei_getTransactionByHash to also surface Cosmos txs with synthetic representations. **Parameters:** @@ -220,7 +220,7 @@ Every method below is also browsable interactively in the explorer above; this s **Supported.** Returns the receipt of a transaction by hash, or null if not found. -**Sei-specific behavior:** Receipt is reconstructed from keeper.GetReceipt + CometBFT block data rather than from a native MPT receipt trie; status/logs are standard Ethereum format. +**Sei-specific behavior:** Receipt is reconstructed from keeper.GetReceipt + CometBFT block data rather than from a native MPT receipt trie; status/logs are standard Ethereum format. If a receipt exists but its block height is above the safe-latest watermark (e.g. the CometBFT status momentarily lags the receipt store by a block), the method returns JSON `null` — the Ethereum JSON-RPC 'not yet mined' signal — rather than an error, so clients simply poll again. This matches the behavior of `eth_getBlockByNumber` for blocks above the watermark. **Parameters:** @@ -485,14 +485,16 @@ Every method below is also browsable interactively in the explorer above; this s **Limited.** Returns a Merkle proof for an account and the requested storage slots. -**Sei-specific behavior:** Sei stores state in an IAVL tree, not an Ethereum Merkle-Patricia trie. The result is a Sei-specific ProofResult{address, hexValues, storageProof} where storageProof entries are CometBFT/IAVL crypto.ProofOps, NOT eth-style MPT proof nodes. There is no accountProof, balance, codeHash, nonce, or storageHash field (Sei has no per-account state root); standard eth_getProof verifiers will not work. +**Sei-specific behavior:** Sei stores state in an IAVL tree, not an Ethereum Merkle-Patricia trie. The result is a Sei-specific ProofResult{address, hexValues, storageProof} where storageProof entries are CometBFT/IAVL crypto.ProofOps, NOT eth-style MPT proof nodes. There is no accountProof, balance, codeHash, nonce, or storageHash field (Sei has no per-account state root); standard eth_getProof verifiers will not work. The EVM store is resolved by unwrapping any wrapping KVStore layers (cachekv, Giga cache, tracekv, and prefix stores) to reach the underlying proof-capable queryable store, so proofs succeed across more node/store configurations (classic IAVL, store/v2 memiavl, and future proof-capable roots) rather than only the classic IAVL store. + +**Storage key validation:** Each entry in `storageKeys` must be a valid hex-encoded value (0x-prefixed); it is decoded and left-padded to 32 bytes. Raw non-hex byte strings are no longer accepted — a malformed key is rejected with an `invalid storage key` error. A request may include at most 1024 storage keys (`MaxStorageKeysPerProof`); exceeding this returns a `too many storage keys` error. **Parameters:** | # | Name | Type | Description | | :- | :- | :- | :- | | 1 | `address` | DATA, 20 bytes | Account address. | -| 2 | `storageKeys` | array of DATA, 32 bytes | Storage slot keys to prove (bounded by MaxStorageKeysPerProof). | +| 2 | `storageKeys` | array of DATA, 32 bytes | Hex-encoded storage slot keys to prove; each is decoded and left-padded to 32 bytes. Malformed (non-hex) keys are rejected with `invalid storage key`. At most 1024 keys per request (`MaxStorageKeysPerProof`); exceeding returns `too many storage keys`. | | 3 | `blockNrOrHash` | BLOCKNUMBER or DATA | Block number, tag, or hash. | **Example request:** @@ -579,9 +581,9 @@ Every method below is also browsable interactively in the explorer above; this s #### `eth_getBlockTransactionCountByNumber` -**Supported.** Returns the number of EVM transactions in a block by number, as a hex quantity. +**Supported.** Returns the number of transactions in a block by number, as a hex quantity. -**Sei-specific behavior:** Counts EVM transactions only (via getEvmTxCount); synthetic/bank-transfer txs are excluded. Genesis returns 0x0; non-existent/future blocks return null. +**Sei-specific behavior:** The count now matches the transaction list returned by `eth_getBlockByNumber`: EVM transactions are counted only when their receipt lookup succeeds (EVM txs without a valid receipt are excluded), and `MsgExecuteContract` (wasm) and `MsgSend` (bank transfer) messages are included. Previously this counted all decodable EVM txs regardless of receipt. Genesis returns 0x0; non-existent/future blocks return null. **Parameters:** @@ -606,7 +608,7 @@ Every method below is also browsable interactively in the explorer above; this s **Supported.** Returns the number of EVM transactions in a block by hash, as a hex quantity. -**Sei-specific behavior:** Counts EVM transactions only; synthetic/bank-transfer txs are excluded. Genesis hash returns 0x0; unknown hash returns null. +**Sei-specific behavior:** Counts EVM transactions only; synthetic/bank-transfer txs are excluded. Genesis hash returns 0x0; unknown hash returns null. If the block's receipts have been pruned from the receipt store — which can happen when the receipt store is configured with a smaller `KeepRecent` than the block/state stores — the method returns an error of the form `requested height X receipts have been pruned; earliest available is Y` instead of a count. **Parameters:** @@ -699,7 +701,7 @@ Every method below is also browsable interactively in the explorer above; this s **Supported.** Returns base fees, gas-used ratios, and reward percentile data over a range of blocks. -**Sei-specific behavior:** Base fees and rewards reflect Sei's x/evm fee market (GetNextBaseFee), not an Ethereum EIP-1559 mempool, and Sei does not burn the base fee. Watermark-aware: pruned/historical blocks may not be available as far back as on Ethereum archive nodes. +**Sei-specific behavior:** Base fees and rewards reflect Sei's x/evm fee market, not an Ethereum EIP-1559 mempool, and Sei does not burn the base fee. Each block's base fee is the block-header base fee (derived from `GetNextBaseFee` at the parent committed height), matching the value reported by the block header and transactions. Following the execution-apis/go-ethereum spec, `baseFeePerGas` contains one more element than `gasUsedRatio` (`len(baseFeePerGas) == len(gasUsedRatio) + 1`): the trailing element is the projected base fee for the child of the newest block in the requested range. Watermark-aware: pruned/historical blocks may not be available as far back as on Ethereum archive nodes, and heights that lack a header base fee (pruned or partial data) are skipped. **Parameters:** @@ -736,6 +738,29 @@ Every method below is also browsable interactively in the explorer above; this s **Parameters:** none. + + +#### `status` (CometBFT RPC) + +**Supported.** The underlying CometBFT `/status` RPC endpoint returns node info, validator info, and a `sync_info` object describing block-sync state. It is exposed on the Tendermint RPC port (not the EVM JSON-RPC endpoint). + +**Sei-specific behavior:** The `sync_info` object now includes an additional field, `last_committed_block_height` — the height of the last block finalized by consensus, serialized as a JSON string-encoded `int64`. Under standard CometBFT consensus this always equals `latest_block_height`, because a block's commit and app-apply happen in a single step. Under Autobahn consensus the two can briefly differ: consensus finalizes a block (advancing `last_committed_block_height`, derived from the latest `CommitQC`) before the app executes it, so the invariant is `last_committed_block_height >= latest_block_height`. + + +**Autobahn `/status` behavior.** When a node runs under Autobahn (i.e. `AutobahnConfigFile` is set), the CometBFT block store is not populated, so `/status` derives `latest_block_height` and `latest_app_hash` from the app layer (via `ABCIInfo`) instead of the block store. Several other `sync_info` fields remain unpopulated under Autobahn and should not be relied on: `latest_block_hash`, `latest_block_time`, the `earliest_*` fields, and `max_peer_block_height`; `catching_up` is currently hardcoded to `true`. + +Block-data endpoints are now served under Autobahn by routing through the GigaRouter's in-memory state (data.State) instead of the unpopulated CometBFT block store, though with some limitations: + +- `/block` returns the translated finalized block at the requested height (`BlockID.Hash`, `ChainID`, `Height`, `Time`, and `Data.Txs` are populated; fields such as `AppHash`, `ProposerAddress`, and `LastCommit` stay at zero values). +- `/block_by_hash` resolves by hash for heights within the retain window; hashes below the pruning watermark or otherwise unknown return an empty block (`{Block: nil}`) with no error, matching CometBFT semantics. +- `/block_results` returns the requested height with `ConsensusParamUpdates.Block.MaxGas` populated from the genesis `consensus_params.block.max_gas` (the same gas-limit value the producer's `MaxGasPerBlock` is derived from), but `TxsResults` is intentionally empty because `FinalizeBlock` responses are not persisted under Autobahn (no per-tx `ExecTxResult` details). + +Note: `eth_getBlockByNumber`'s `gasLimit` field is sourced from the active consensus params in the SDK context (`ctx.ConsensusParams().Block.MaxGas`) rather than from `blockRes.ConsensusParamUpdates`. This means the reported `gasLimit` matches the value the EVM runtime returns for the `GASLIMIT` opcode, and stays consistent under Autobahn where `/block_results` only synthesizes a placeholder. +- `/validators` returns the genesis committee for any retained height, with `block_height` matching the requested height (fixing the prior behavior where it could report a stale/stuck height). + +EVM endpoints that walk through these (e.g. `eth_getBlockByNumber`) therefore work under Autobahn, rendering the block envelope correctly — just without per-tx result details from `/block_results`. `/commit` remains unpopulated under Autobahn. + + #### `web3_clientVersion` **Supported.** Returns the client version string. @@ -786,6 +811,8 @@ Every method below is also browsable interactively in the explorer above; this s **Supported.** Polls a filter and returns new logs (log filters) or block hashes (block filters) since the last poll. +**Sei-specific behavior:** For log filters, an empty array (`[]`) is returned instead of `null` when there are no matching logs — including once a bounded filter (one created with a `blockHash` or a `toBlock`) has had its range fully consumed. This aligns with the Ethereum JSON-RPC spec, which always expects an array result. + **Parameters:** | # | Name | Type | Description | @@ -891,6 +918,8 @@ Every method below is also browsable interactively in the explorer above; this s **Sei-specific behavior:** WebSocket-only (SubscriptionAPI is not registered on the HTTP server; returns rpc.ErrNotificationsUnsupported over HTTP). Only 'newHeads' and 'logs' are implemented in source; there is no 'newPendingTransactions' subscription despite some client docs implying otherwise. newHeads subscriptions are capped by MaxSubscriptionsNewHead. +Under Autobahn consensus (i.e. when `AutobahnConfigFile` is set), `newHeads` notifications are delivered by an in-process notifier that publishes committed-block headers after a successful `Commit`, rather than through the CometBFT event bus used by the legacy path. Subscribers therefore observe only committed state, but the header payloads differ from the legacy encoding: `parentHash`, `receiptsRoot`, and `transactionsRoot` are returned as the zero hash (Autobahn does not compute a Tendermint-style hash chain), `stateRoot` is sourced from the finalized block's post-execution `AppHash`, and `hash` is the Autobahn block-header hash. `gasUsed` is an approximation (summed over transaction results); subscribers needing exact gas or the omitted header fields should query `eth_getBlockByNumber`. Under Autobahn, on consumer lag the latest head wins (overwrite-on-full) and intermediate heads may be dropped, and subscribers that chain-validate the head stream will need a different mechanism. + **Parameters:** | # | Name | Type | Description | @@ -1089,7 +1118,7 @@ The `filter` object applies only to `logs` subscriptions. For `newHeads`, pass t **Supported.** Replays a transaction by hash and returns an execution trace using the configured tracer. -**Sei-specific behavior:** HTTP-only (the debug namespace is not registered on the WebSocket server). Supports geth tracers (callTracer, prestateTracer, flatCallTracer, struct/opcode logger); callTracer/prestateTracer/flatCallTracer results are pre-baked/cached via TraceBaker. Requires trace-enabled/archive state for the target height. +**Sei-specific behavior:** HTTP-only (the debug namespace is not registered on the WebSocket server). Supports geth tracers (callTracer, prestateTracer, flatCallTracer, struct/opcode logger); callTracer/prestateTracer/flatCallTracer results are pre-baked/cached via TraceBaker. Requires trace-enabled/archive state for the target height. Subject to the max block lookback guard: requests targeting a block older than the node's configured `maxBlockLookback` (resolved from the transaction's receipt block number) are rejected with `block number N is beyond max lookback of M`. **Parameters:** @@ -1118,7 +1147,7 @@ The `filter` object applies only to `logs` subscriptions. For `newHeads`, pass t **Supported.** Traces all transactions in a block by number and returns per-transaction execution traces. -**Sei-specific behavior:** HTTP-only. safe/finalized/latest are equivalent due to instant finality. +**Sei-specific behavior:** HTTP-only. safe/finalized/latest are equivalent due to instant finality. When no custom tracer is specified (the default struct/opcode logger), block traces can run through a parallelized, profiled execution path that returns per-transaction result/error entries, but only when the node has `evm.enable_parallelized_block_trace = true` set in `app.toml` (default `false`). On that path, a transaction that fails no longer aborts the whole block trace — its entry carries an `error` field while the remaining transactions still return their traces. When `evm.enable_parallelized_block_trace` is `false` (the default), or when a custom tracer (e.g. `callTracer`) is specified, block traces use the standard tracing path instead. **Parameters:** @@ -1231,6 +1260,37 @@ The `filter` object applies only to `logs` subscriptions. For `newHeads`, pass t } ``` + + +#### `debug_traceTransactionProfile` + +**Limited.** Sei extension that replays a transaction by hash and returns its execution trace alongside a detailed timing and store-access profile. + +**Sei-specific behavior:** Sei-specific extension (not part of upstream go-ethereum's debug namespace). HTTP-only (the debug namespace is not registered on the WebSocket server) and subject to historical-debug-trace/archive availability guards for the target height. The response is an object with two fields: `trace` (the standard trace produced by the configured tracer, identical to `debug_traceTransaction`) and `profile` (timing and KVStore access instrumentation collected while replaying the tx). The `profile` object reports `totalNanos` (end-to-end profiling time), `historicalDbLookupNanos` (time spent in historical DB reads — the sum of `get`/`has`/`iterator`/`iteratorNext` store latencies), `otherNanos` (remaining time), a `phases` object with per-phase timings (`lookupTransactionNanos`, `loadBlockNanos`, `replayHistoricalTxsNanos`, `buildBlockContextNanos`, `prepareTxNanos`, `executionNanos`, `traceResultNanos`), and a `store` object with per-module access statistics under `store.modules..stats` (each op maps to `{count, totalNanos}`) plus per-module `iterators` samples (bounds, direction, surfaced keys, `nextCount`, and cumulative timing). Iterator and key samples are capped per module to bound response size. To batch-run this method across a block range, see the `seidb trace-profile-report` command. + +**Parameters:** + +| # | Name | Type | Description | +| :- | :- | :- | :- | +| 1 | `hash` | DATA, 32 bytes | Transaction hash to trace and profile. | +| 2 | `config` | object | Optional tracer config (tracer name, tracerConfig, timeout, reexec, disableStorage/Stack/Memory). | + +**Example request:** + +```json +{ + "jsonrpc": "2.0", + "id": 1, + "method": "debug_traceTransactionProfile", + "params": [ + "0x5c504ed432cb51138bcf09aa5e8a410dd4a1e204ef84bfed1be16dfba1b22060", + { + "timeout": "60s" + } + ] +} +``` + ## Sei Custom Endpoints @@ -1240,7 +1300,9 @@ Sei extends the standard Ethereum JSON-RPC API with custom endpoints that enhanc **Deprecation Notice:** All `sei_*` and `sei2_*` JSON-RPC methods are deprecated and scheduled for removal. Do not build new integrations on these endpoints. Use standard `eth_*` and `debug_*` methods instead. -Access is controlled by the `enabled_legacy_sei_apis` setting under `[evm]` in `app.toml`. Only methods explicitly listed in this allowlist are available. Disabled methods return a standard JSON-RPC error (code `-32601`, data `"legacy_sei_deprecated"`). Allowed methods pass through unchanged, with an optional `Sei-Legacy-RPC-Deprecation` HTTP response header signaling deprecation. +Access is controlled by the `enabled_legacy_sei_apis` setting under `[evm]` in `app.toml`. Only methods explicitly listed in this allowlist are available. Disabled methods return a standard JSON-RPC error (code `-32601`, data `"legacy_sei_deprecated"`). Allowed single-object requests pass through unchanged, with an optional `Sei-Legacy-RPC-Deprecation` HTTP response header signaling deprecation. + +JSON-RPC batch requests over HTTP are handled element-by-element: only allowlisted methods are forwarded to the inner handler as a filtered subset, and their responses are merged back by JSON-RPC `id`. Disabled methods in the batch receive the standard `-32601` gate error in place, and any non-object batch element returns a JSON-RPC `-32600` Invalid Request without being forwarded — so a malformed element can never bypass the gate. Batch responses follow JSON-RPC 2.0 semantics: only notification requests — those that omit the `id` member entirely — produce no response entry, so the merged response array is not necessarily 1:1 with the request batch when notifications are present. A request with an explicit `"id": null` is *not* a notification; it receives a response like any other request. If a batch would yield no response objects at all, the gateway returns an empty HTTP body rather than an empty JSON array (`[]`), which JSON-RPC 2.0 forbids. The legacy HTTP request body limit is 5 MiB (matching go-ethereum's default). ### Legacy API Configuration @@ -1271,6 +1333,8 @@ To enable additional legacy methods, add them to this array. All other `sei_*` a | `sei_getVMError` | Get VM error details for a transaction | | `sei_getBlockByHash` | Get block by hash (includes synthetic txs) | | `sei_getBlockByNumber` | Get block by number (includes synthetic txs) | +| `sei_getBlockByHashExcludeTraceFail` | Get block by hash, excluding untraceable txs — synthetic txs and ante-deferred stub txs (those that bumped their nonce in ante but never reached the VM, e.g. insufficient funds/fee, identified by `EffectiveGasPrice == 0 && GasUsed == 0`). Note that the regular `eth_getBlockBy*`/`sei_getBlockBy*` endpoints still surface these stub txs; only the `ExcludeTraceFail` variants drop them. | +| `sei_getBlockByNumberExcludeTraceFail` | Get block by number, excluding untraceable txs — synthetic txs and ante-deferred stub txs (those that bumped their nonce in ante but never reached the VM, e.g. insufficient funds/fee, identified by `EffectiveGasPrice == 0 && GasUsed == 0`). Note that the regular `eth_getBlockBy*`/`sei_getBlockBy*` endpoints still surface these stub txs; only the `ExcludeTraceFail` variants drop them. | | `sei_getBlockReceipts` | Get block receipts (includes synthetic txs) | | `sei_getBlockTransactionCountByHash` | Get tx count by block hash (includes synthetic txs) | | `sei_getBlockTransactionCountByNumber` | Get tx count by block number (includes synthetic txs) | @@ -1285,11 +1349,11 @@ To enable additional legacy methods, add them to this array. All other `sei_*` a | `sei_newFilter` | Create a new log filter | | `sei_newBlockFilter` | Create a new block filter | | `sei_uninstallFilter` | Remove a filter | -| `sei_getBlockByHashExcludeTraceFail` | Get block by hash excluding failed traces | -| `sei_getBlockByNumberExcludeTraceFail` | Get block by number excluding failed traces | -| `sei_getTransactionReceiptExcludeTraceFail` | Get receipt excluding failed traces | -| `sei_traceBlockByHashExcludeTraceFail` | Trace block by hash excluding failed traces | -| `sei_traceBlockByNumberExcludeTraceFail` | Trace block by number excluding failed traces | +| `sei_getBlockByHashExcludeTraceFail` | Get block by hash, excluding only untraceable txs (ante-rejected and synthetic); reverts/OOG are included | +| `sei_getBlockByNumberExcludeTraceFail` | Get block by number, excluding only untraceable txs (ante-rejected and synthetic); reverts/OOG are included | +| `sei_getTransactionReceiptExcludeTraceFail` | Get receipt, excluding only untraceable txs (ante-rejected and synthetic); reverts/OOG are included | +| `sei_traceBlockByHashExcludeTraceFail` | Trace block by hash, excluding only untraceable txs (ante-rejected and synthetic); reverts/OOG are included | +| `sei_traceBlockByNumberExcludeTraceFail` | Trace block by number, excluding only untraceable txs (ante-rejected and synthetic); reverts/OOG are included | **`sei2_*` methods** (block queries with bank transfers included): diff --git a/evm/tracing/index.mdx b/evm/tracing/index.mdx index 3473127..8e5bae0 100644 --- a/evm/tracing/index.mdx +++ b/evm/tracing/index.mdx @@ -54,6 +54,12 @@ Debug tracing is your primary tool for understanding EVM transaction execution o | -------------------------- | -------------------------- | ----------------------------- | | `debug_traceTransaction` | Trace specific transaction | Debugging failed transactions | | `debug_traceBlockByNumber` | Trace entire block | Block-level analysis | + + +When `debug_traceBlockByNumber` or `debug_traceBlockByHash` run with the default (struct) tracer — that is, when no custom `tracer` is specified — Sei can use a parallelized block trace path in which a failed transaction no longer aborts the entire block trace. Instead, each transaction produces its own per-tx entry in the result array: successful transactions include a `result` field, while failed transactions include an `error` field. This lets you inspect every transaction in a block even when one or more of them fail. + +This parallelized path is opt-in and gated by the node config field `evm.enable_parallelized_block_trace` (default `false`). When it is disabled, or when an explicit `tracer` is specified, the legacy trace path is used. Node operators can enable it by setting `enable_parallelized_block_trace = true` in the `[evm]` section of `app.toml` (or via the `--evm.enable_parallelized_block_trace` flag). + | `debug_traceCall` | Simulate and trace | Testing before execution | | `debug_traceStateAccess` | State access patterns | Performance optimization | @@ -400,6 +406,9 @@ debug_traceCall(tx, "latest", {tracer: "callTracer"}) # State access debug_traceStateAccess(hash) + +# Trace transaction with timing/store-access profiling +debug_traceTransactionProfile(hash, {}) ``` ### Common Tracers diff --git a/evm/transactions.mdx b/evm/transactions.mdx index 6b9861b..d9eacbe 100644 --- a/evm/transactions.mdx +++ b/evm/transactions.mdx @@ -258,7 +258,7 @@ EVM transactions in Sei follow the Ethereum transaction format with standard pro "logs": [], "status": "0x1", "logsBloom": "0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000", - "effectiveGasPrice": "0x1234", + "effectiveGasPrice": "0x77359400", "type": "0x2" } } diff --git a/learn/sei-giga-specs.mdx b/learn/sei-giga-specs.mdx index 65b9c13..80518f0 100644 --- a/learn/sei-giga-specs.mdx +++ b/learn/sei-giga-specs.mdx @@ -65,6 +65,39 @@ The specifications are organized to clearly distinguish between what is currentl + +### Upcoming: Autobahn Block Limits + +When a node runs with Autobahn (Sei Giga) consensus enabled, the block producer builds payloads by reaping transactions directly from the shared mempool, subject to hard on-chain block size constraints that are enforced during payload construction. + +The block gas limit is no longer a single configurable Autobahn field. The producer now tracks two separate per-block gas dimensions: gas wanted and gas estimated. Each block payload records `total_gas_wanted` and `total_gas_estimated` (replacing the previous single `total_gas` field), and both are enforced against `MaxGasWantedPerBlock` and `MaxGasEstimatedPerBlock` limits during payload construction. + +These limits are derived from the genesis `consensus_params.block` values: `MaxGasWantedPerBlock` comes from `max_gas_wanted` and `MaxGasEstimatedPerBlock` comes from `max_gas` — the same gas limit the EVM runtime reads via `ctx.ConsensusParams().Block.MaxGas`. Under Autobahn, `BlockResults`'s `ConsensusParamUpdates.Block.MaxGas` reflects `MaxGasEstimatedPerBlock`. The `max_gas_per_block` field has been removed from the Autobahn config file, so it should be omitted from existing configs. + +Genesis `consensus_params.block.max_gas` must be a positive value. If it is missing or non-positive, node startup fails with `ErrGenesisMaxGasInvalid` (`genesis consensus_params.block.max_gas must be > 0`). + + + + + | Property | Value | + | --- | --- | + | Max transactions per block | 2000 (`MaxTxsPerBlock`) | + | Max total tx bytes per block | ~2 MB (`MaxTxsBytesPerBlock` = 2000 × 1024) | + | Max gas wanted per block | Derived from genesis `consensus_params.block.max_gas_wanted` (`MaxGasWantedPerBlock`) | + | Max gas estimated per block | Derived from genesis `consensus_params.block.max_gas` (`MaxGasEstimatedPerBlock`) | + | Block proto size bound | Derived from `MaxTxsPerBlock` and `MaxTxsBytesPerBlock` (`MaxBlockProtoSize`) | + + + - Total tx bytes may be split arbitrarily across transactions (one large tx or many small ones), up to the `MaxTxsPerBlock` limit. + - Each block payload tracks `total_gas_wanted` and `total_gas_estimated`; a transaction is rejected if adding it would push either total past `MaxGasWantedPerBlock` or `MaxGasEstimatedPerBlock`. + - The gas limits are sourced from genesis `consensus_params.block`; `max_gas` (used for gas estimated) must be greater than zero or the node will refuse to start. + - Under Autobahn, `BlockResults` reports `ConsensusParamUpdates.Block.MaxGas` as `MaxGasEstimatedPerBlock`. + - Limits are validated when the payload is built; payloads exceeding the transaction count or total byte limit are rejected. + - The block proto size bound derives an upper limit on the encoded block used to size networking message windows. + + + + ## Execution Engine ### Current: Parallel EVM Execution @@ -127,10 +160,12 @@ The specifications are organized to clearly distinguish between what is currentl | Property | Value | | --- | --- | - | Max Validators | 100 | + | Max Validators | 100 (hard limit, enforced by Autobahn) | | Voting Power | Stake-weighted | | Unbonding Period | 21 days | | Slashing | Byzantine behavior | + + Under Autobahn consensus, the validator set size is a hard protocol limit rather than just a configurable network parameter. Committee creation (`NewCommittee`) enforces `MaxValidators = 100` and rejects any committee containing more than 100 validators with an error. | Property | Value | diff --git a/learn/sei-giga.mdx b/learn/sei-giga.mdx index bd6cbe0..b129df4 100644 --- a/learn/sei-giga.mdx +++ b/learn/sei-giga.mdx @@ -149,6 +149,115 @@ The protocol operates through two complementary layers: - Two-phase BFT agreement with fast and slow paths - Pipelined slots for minimal latency + + +#### Enabling Autobahn on a Node + +**Node Operators**: Autobahn (GigaRouter) can now be enabled experimentally on a node by pointing the `autobahn-config-file` field in `config.toml` to a JSON configuration file. Leave the field empty to keep Autobahn disabled. + +To enable Autobahn, set the following in your node's `config.toml`: + +```toml +# Path to a JSON file containing the Autobahn (GigaRouter) configuration. +# Leave empty to disable Autobahn. +autobahn-config-file = "/path/to/autobahn.json" +``` + +The node must be a committee member: its own validator public key and node public key must appear in the `validators` list of the referenced config file. + +**Remote signers are not supported**: When `autobahn-config-file` is set, the node fails to start if `priv-validator.laddr` is also configured. A local validator key is required — non-validator (observer) nodes are not yet supported. + +**Autobahn JSON Configuration Schema** + +The file referenced by `autobahn-config-file` is a JSON document with the following fields: + +| Field | Type | Description | +| --- | --- | --- | +| `validators` | array | The committee. Each entry is an object with `validator_key` (Autobahn validator public key), `node_key` (p2p node public key), and `address` (network `host:port`). | +| `max_gas_per_block` | number | Maximum gas allowed per block. Must be greater than 0. | +| `max_txs_per_block` | number | Maximum transactions per block. Must be greater than 0. | +| `max_txs_per_second` | number \| null | Optional cap on transactions per second. Omit or set null for no limit. | +| `block_interval` | duration | Target interval between blocks. Must be greater than 0. | +| `allow_empty_blocks` | boolean | Whether empty blocks may be produced. | +| `view_timeout` | duration | Consensus view timeout. Must be greater than 0. | +| `persistent_state_dir` | string \| null | Directory for persisting the Autobahn consensus and data-layer WALs across restarts, so both layers survive process restarts (they share this on-disk root and write to distinct subdirectories under it). Relative paths are resolved against the node's `--home` directory at load time; absolute paths are used as-is. Set to `null` (or generate with an empty `--persistent-state-dir`) to disable persistence and run in-memory only. | +| `dial_interval` | duration | Interval at which peers are dialed. Must be greater than 0. | + +Example configuration file: + +```json +{ + "validators": [ + { + "validator_key": "ed25519:public:...", + "node_key": "node:ed25519:public:...", + "address": "peer1.example.com:26660" + } + ], + "max_gas_per_block": 50000000, + "max_txs_per_block": 5000, + "max_txs_per_second": 1000, + "mempool_size": 20000, + "block_interval": "200ms", + "allow_empty_blocks": true, + "view_timeout": "1.5s", + "persistent_state_dir": "/tmp/autobahn-state", + "dial_interval": "10s" +} +``` + +The node validates this file on startup and refuses to start if any required field is missing, if the `validators` list is empty, if a `validator_key` or `node_key` is duplicated, or if the node's own validator and node keys are not present as a matching entry in the `validators` list. + + + +**Generating the config with `gen-autobahn-config`** + +Rather than hand-authoring the JSON file, you can generate it from per-node pubkey files using the `seid tendermint gen-autobahn-config` command: + +```bash +seid tendermint gen-autobahn-config [node-dirs...] --output +``` + +The command reads four files from each node directory passed as an argument and assembles one `validators` entry per directory: + +- `validator_pubkey.txt` — the Autobahn validator public key, in `validator:` format +- `node_pubkey.txt` — the p2p node public key, in `node:ed25519:public:` format +- `autobahn_address.txt` — the node's network address in `host:port` format +- `evmrpc_url.txt` — the node's EVM RPC HTTP URL (for example `http://:8545`), used to proxy EVM requests to the validator that owns the sender's shard + +The `--output` (or `-o`) flag is required and specifies where the generated JSON config is written. The generated file uses default consensus parameters (for example, a 400ms block interval and a 1.5s view timeout), which you can adjust afterward if needed. + +The `validator_pubkey.txt` and `node_pubkey.txt` files are written automatically alongside the validator private key and node key whenever those keys are saved (for example, during `seid init`), so each node's config directory already contains the pubkey files needed for generation. You still need to create the `autobahn_address.txt` file with the node's reachable `host:port` address. + +Example generating a config across three node directories: + +```bash +seid tendermint gen-autobahn-config \ + build/generated/node_0 \ + build/generated/node_1 \ + build/generated/node_2 \ + --output ~/.sei/config/autobahn.json +``` + +Once generated, point `autobahn-config-file` in `config.toml` at the resulting file to enable Autobahn. + + + +**Reactor changes when Autobahn is enabled** + +**Behavior change**: When a node runs with `autobahn-config-file` set, several standard reactors are disabled because they are incompatible with Autobahn's block production model. + +Enabling Autobahn changes how the node builds and synchronizes blocks: + +- **Mempool gossip reactor disabled**: Transaction gossip across the standard mempool reactor is turned off. The shared `TxMempool` still accepts and stores transactions, but they are no longer broadcast peer-to-peer through the legacy mempool channel. +- **Consensus reactor disabled**: The Tendermint consensus reactor is not started; block ordering is handled entirely by the Autobahn (GigaRouter) protocol. +- **State sync and block sync disabled**: Both the statesync and blocksync reactors are skipped, so a node cannot catch up through those paths while Autobahn is active. +- **Direct reaping from the shared mempool**: The block producer no longer uses a separate producer mempool channel. Instead, it reaps transactions directly from the shared `TxMempool` when assembling each block's payload. + +Because of these changes, an Autobahn-enabled node relies solely on the Autobahn data-dissemination and consensus layers for ordering and propagation, rather than the legacy gossip, consensus, and sync reactors. + +**Block limits**: Autobahn blocks are bounded by hard on-chain limits — at most 2000 transactions per block and roughly 2 MB of total transaction bytes — in addition to the `max_gas_per_block` and `max_txs_per_block` values in the config file. These limits are enforced while the payload is built. + ### 3. Advanced Parallel Execution The parallel execution architecture extends beyond current OCC capabilities to include: diff --git a/learn/seidb.mdx b/learn/seidb.mdx index 172ba02..ba832e2 100644 --- a/learn/seidb.mdx +++ b/learn/seidb.mdx @@ -8,6 +8,9 @@ keywords: ['seidb', 'blockchain database', 'state storage', 'ethereum optimizati SeiDB is a specialized database system designed to optimize blockchain state storage for the Ethereum Virtual Machine (EVM). It addresses fundamental performance constraints in traditional blockchain storage systems through targeted optimizations for EVM's specific state access patterns. This document explains the technical design and key components of SeiDB. + +**IAVL Removed — SeiDB Required:** The legacy IAVL backend has been fully removed. SeiDB state-commit is now mandatory: if `sc-enable` is set to `false`, the node will panic on startup with `SeiDB state-commit (SC) must be enabled; IAVL backend has been fully deprecated` instead of falling back to IAVL. Node operators must ensure SeiDB is enabled by setting `sc-enable = true` (and `ss-enable = true` for the state store) in their node configuration before upgrading. All IAVL-related CLI commands (such as `compact`, `prune`, `latest_version`, and `debug dump-iavl`) and IAVL configuration fields (including `iavl-cache-size`, the `[iavl]` section, and the orphan-storage settings) have been removed and no longer take effect. Legacy IAVL-based snapshot and restore are also no longer supported. + ## Core Technical Design Traditional blockchain databases store state in structures optimized for cryptographic verification rather than transaction execution speed. SeiDB uses a hybrid architecture that preserves cryptographic verifiability while accelerating state access operations. @@ -104,6 +107,25 @@ Key optimization techniques include: - **Configurable Persistence**: The database offers adjustable durability guarantees based on node type and network requirements, from fully synchronous writes to asynchronous persistence with periodic checkpoints. The configuration system allows operators to make explicit tradeoffs between performance and durability based on their specific node's role in the network. + +### Descending-Version MVCC Encoding (State Store) + +SeiDB's PebbleDB-backed state store uses multi-version concurrency control (MVCC) to keep every historical version of a key. Each logical key is stored on disk with its version appended, so a single key may have many versioned entries. The order in which those versions are laid out on disk directly affects how quickly the store can serve the most common query: reading the latest version of a key. + +**Descending-version encoding for fresh databases.** Newly created state stores now encode the version component of each MVCC key in *descending* byte order, so that newer versions of a logical key sort *before* older ones on disk. Because the newest visible version sits first, a latest-version read lands directly on the target entry via a single forward seek (`First()` / `SeekGE`) instead of scanning past older versions. This is the fast path and delivers faster latest-version reads for validators and API nodes that overwhelmingly query recent state. + +**Transparent compatibility with legacy databases.** State stores written by the previous build used *ascending*-version encoding, where older versions sort first. To avoid forcing a migration, SeiDB detects the on-disk encoding when a database is opened and reads legacy stores using the ascending-version path automatically — no error is raised and no data conversion occurs. Encoding mode is fixed for the lifetime of an open database. + +**How detection works.** Fresh databases are stamped with an on-disk sentinel key (`s/_mvcc_descending`) the first time they are opened, marking them as descending. On subsequent opens: + +- If the sentinel is present, the database opens in descending (fast-path) mode. +- If the sentinel is absent but the database already contains data (a legacy database written by the previous ascending-version build), it opens in ascending (legacy) mode and is intentionally left unmarked. +- If both the sentinel and existing data are absent, the database is treated as fresh: the sentinel is written and descending mode is used. + +**Migration guidance for node operators.** Operators upgrading with an existing PebbleDB state store will continue to run in legacy ascending mode; this is safe and requires no action. However, legacy databases stay unmarked and cannot benefit from the descending fast path unless the state store is recreated or migrated (for example, by resyncing the state store). Archive nodes that cannot practically migrate their historical state will continue to operate correctly on the legacy ascending path. + +**`UseDefaultComparer` and iteration.** The `UseDefaultComparer` field of the state store configuration influences how the descending-mode iterator advances to the next logical key. When enabled, the iterator falls back to a scan-based approach to locate the next logical key rather than using the MVCC comparer's key-successor logic. This affects iterator advancement behavior only in descending mode. + ## Performance Characteristics SeiDB is designed to deliver substantial performance improvements compared to traditional EVM state implementations. The architecture focuses on enhancing both throughput and latency across various operation types. @@ -215,3 +237,266 @@ SeiDB's architecture supports various deployment configurations optimized for di - **Light clients** benefit from optimized state proof generation with compact inclusion proofs for partial state verification Each configuration tunes the SeiDB components to match specific requirements while maintaining protocol compatibility. The configuration framework provides fine-grained control over cache sizes, worker pools, I/O policies, and persistence strategies to match the operational needs of different node types. + + + +## Profiling State Access with `trace-profile-report` + +The `seidb` tool includes an offline `trace-profile-report` command for analyzing where time is spent while executing historical transactions. It runs the `debug_traceTransactionProfile` JSON-RPC method against a running node across a range of blocks and produces detailed timing and store-access reports. This is useful for identifying transactions and modules that dominate execution time or that generate heavy historical database lookups. + +For each transaction in the requested block range, the command captures a full trace along with a profile that breaks down total execution time into per-phase timings (transaction lookup, block loading, historical transaction replay, block-context construction, transaction preparation, execution, and trace-result assembly) and per-module KVStore access statistics (Get/Has/Set/Delete counts and durations, plus iterator samples). + +### Usage + +```bash +seidb trace-profile-report \ + --endpoint http://localhost:8545 \ + --start-block \ + --end-block \ + --output-dir ./profile-out +``` + +### Flags + +| Flag | Alias | Default | Description | +| --- | --- | --- | --- | +| `--endpoint` | | | RPC endpoint to query, e.g. `http://localhost:8545`. Required. | +| `--start-block` | | | Starting block number (must be positive). Required. | +| `--end-block` | | | Ending block number (must be `>= --start-block`). Required. | +| `--output-dir` | `-o` | | Directory where `raw_profiles.jsonl` and `summary.json` are written. Required. | +| `--concurrency` | `-c` | `4` | Number of concurrent `debug_traceTransactionProfile` requests. | +| `--trace-config-json` | | `{}` | JSON object passed as the trace config for each request. | +| `--max-transactions` | | `0` | Optional cap on the number of transactions processed (`0` means no cap). | + +### Output + +The command writes two files to the output directory: + +- **`raw_profiles.jsonl`** — one JSON line per transaction, containing the block number, block hash, transaction hash, and either the full trace-profile result or an error. +- **`summary.json`** — an aggregated report including total/success/error counts, average and P50/P95 latencies for total and historical-database-lookup time, per-phase totals, per-module store-access totals, and the top transactions and blocks by total execution time. + +Because `trace-profile-report` replays historical transactions, point it at a node (such as an archive node) that retains the historical state for the block range you want to profile. + + + +## Dumping FlatKV State with `dump-flatkv` + +The `seidb` tool includes a `dump-flatkv` command that iterates a FlatKV store and dumps every physical `(key, value)` pair into per-bucket files. FlatKV physical keys are grouped into four logical buckets — `account`, `code`, `storage`, and `legacy` — and each bucket is written to its own file inside the output directory. The output is formatted to match `dump-iavl` so the same diff tooling works on both dumps. + +Each output file begins with a header line (`Bucket at version `) followed by one `Key: , Value: ` line per physical row. Physical keys are emitted verbatim, including their `/` and type-prefix header. The FlatKV metadata rows are intentionally excluded, as they are internal bookkeeping. + +Under the hood, the tool clones the selected FlatKV snapshot and changelog into a temporary directory and opens that isolated copy, so it never contends for the FlatKV writer lock on a live node. + +### Usage + +```bash +seidb dump-flatkv \ + --db-dir /path/to/flatkv \ + --output-dir ./flatkv-dump \ + --height 0 \ + --bucket storage +``` + +### Flags + +| Flag | Alias | Default | Description | +| --- | --- | --- | --- | +| `--db-dir` | `-d` | | FlatKV database directory. Required. | +| `--output-dir` | `-o` | | Output directory; one file is written per bucket. Required. | +| `--height` | | `0` | FlatKV target version; `0` selects the latest available version. | +| `--bucket` | `-b` | | Restrict the dump to a single bucket (`account`, `code`, `storage`, or `legacy`). When omitted, all buckets are dumped. | + +## Analyzing FlatKV with `state-size` + +The `state-size` command now folds an optional FlatKV analysis into its output alongside the existing memIAVL module breakdown. When a FlatKV directory is present and `--module` is empty or `evm`, the tool scans FlatKV, reports a per-DB size breakdown (`account`, `code`, `storage`, `legacy`) and a table of the top EVM contracts by storage size, and — when exporting — includes the FlatKV row in the same DynamoDB batch as the memIAVL module rows. + +Use the new `--flatkv-dir` flag to point at the FlatKV data directory. When it is not set, the tool auto-detects a sibling `flatkv/` directory next to `--db-dir` (for example, `/data/committer.db` → `/data/flatkv`), which is the standard layout on a Sei node. If no such directory exists, FlatKV analysis is skipped and only the memIAVL modules are reported. + +FlatKV analysis is strictly additive: if the FlatKV directory is missing or the store cannot be opened, the tool logs the reason and continues with the memIAVL analysis. + +| Flag | Alias | Default | Description | +| --- | --- | --- | --- | +| `--db-dir` | `-d` | | memIAVL database directory. | +| `--flatkv-dir` | | auto-detect `/../flatkv` | FlatKV data directory. When unset, a sibling `flatkv/` directory next to `--db-dir` is auto-detected. | + + + +## Reading the Latest memIAVL Version with `memiavl-latest-version` + +The `memiavl-latest-version` command prints the latest committed memIAVL version of a stopped node. It is the read-only companion to `import-flatkv-from-memiavl`: an orchestration script can read each validator's version after stopping `seid`, pick a single common height across a multi-validator cluster, and use that as the import height. + +Run this command against a stopped node. It reads the on-disk memIAVL state directly and is intended for offline use. + +### Usage + +```bash +seidb memiavl-latest-version --data-dir /path/to/.sei/data +``` + +### Flags + +| Flag | Default | Description | +| --- | --- | --- | +| `--home` | `$HOME/.sei` | Sei home directory. | +| `--data-dir` | | Sei data directory or home directory. If the basename is `data`, its parent is used as the home directory. | + +The command prints a single integer — the latest memIAVL version — to standard output. + +## Importing memIAVL Modules into FlatKV with `import-flatkv-from-memiavl` + +The `import-flatkv-from-memiavl` command performs an offline import of selected memIAVL modules into FlatKV. It is used when migrating the EVM module's state-commit (SC) layer from memIAVL to FlatKV storage. The command reads the selected module data from memIAVL at a target height, translates it into FlatKV's on-disk layout, and bulk-imports it into the FlatKV store. + +This is a restore-style import: it **resets** the FlatKV directory before loading the imported rows. If FlatKV already contains committed data, the command refuses to run unless `--force` is supplied. + +**EVM-only initial scope.** The initial production scope is intentionally narrow — only the `evm` module is accepted. Non-EVM modules remain in memIAVL and are not copied into FlatKV; passing any other module name is rejected at the CLI boundary. + +### Usage + +```bash +seidb import-flatkv-from-memiavl \ + --modules evm \ + --data-dir /path/to/.sei/data \ + --height \ + [--force] +``` + +### Flags + +| Flag | Default | Description | +| --- | --- | --- | +| `--home` | `$HOME/.sei` | Sei home directory. | +| `--data-dir` | | Sei data directory or home directory. If the basename is `data`, its parent is used as the home directory. | +| `--modules` | `evm` | Comma-separated module names to import. Initial production scope supports only `evm`. | +| `--height` | `0` | memIAVL version to import. `0` selects the latest available version. | +| `--force` | `false` | Overwrite existing committed FlatKV data. Required when FlatKV already has a committed version. | + +### Height constraints + +The import must be run at the memIAVL **latest** height. The command refuses to import at a height `H` below the memIAVL latest version, because a subsequent `GIGA_STORAGE` startup would call `reconcileVersions` and silently roll memIAVL back to `H`, truncating every cosmos block in `(H, latest]`. Operators who genuinely want a non-latest height must first roll memIAVL back to that height themselves — this command deliberately does not perform a destructive cosmos rollback on their behalf. A height ahead of the memIAVL latest version is likewise rejected. + +If the import is interrupted (for example by context cancellation or an exporter/translator failure), the in-progress import is aborted rather than finalized: the FlatKV directory is left at its pre-import committed version, so the operation can be retried without `--force`. + +### Migration configuration constraints + +When restarting a node after the import, keep `evm-ss-split = false` across the import boundary. The import moves only the EVM module's SC-layer data into FlatKV; the EVM state-store history stays in the existing combined cosmos store, so enabling `evm-ss-split` would trigger a startup panic. + +There is no longer any `sc-enable-lattice-hash` setting to manage. That configuration field has been removed; whether the FlatKV lattice hash participates in the AppHash is now derived automatically from the node's write mode and migration state, so operators do not need to toggle it across the import boundary. + + + +## Polling FlatKV EVM Migration Status with `migrate-evm-status` + +The `migrate-evm-status` command reports the on-disk FlatKV EVM migration state of a FlatKV directory as JSON. It exists so an orchestration script driving the in-flight `migrate_evm` migration can poll "is the migration done yet?" against each validator's data directory from the host — without adding a custom RPC handler or grepping through node logs. + +The command reads two reserved keys from the FlatKV migration store: + +- **`migration-version`** — an 8-byte big-endian `uint64` written exactly once per migration lifecycle, on the block that finalizes the migration. Absent or zero means the EVM migration has not yet completed. +- **`migration-boundary`** — the in-flight cursor encoding the `(module, key)` pair the next batch should resume from. It is present only while the migration is strictly between not-started and complete. + +To stay aligned with the other `seidb` tools, the read goes through the same read-only path used by `dump-flatkv`: the tool hardlink-clones the latest snapshot and copies the WAL into a temporary directory before opening. This avoids contending with a live node for the FlatKV writer lock and yields a stable view even if the live writer rolls snapshots mid-run, so the command can be run against a running validator. + +### Usage + +```bash +seidb migrate-evm-status \ + --db-dir /path/to/flatkv \ + [--height ] +``` + +### Flags + +| Flag | Alias | Default | Description | +| --- | --- | --- | --- | +| `--db-dir` | `-d` | | FlatKV database directory. Required. | +| `--height` | | `0` | FlatKV target version; `0` selects the latest available version. | + +### Output + +The command prints a single JSON object to standard output: + +```json +{ + "version_at": 12345, + "migration_version": 1, + "migrate_evm_complete": true, + "boundary_present": false, + "version_raw_hex": "0000000000000001" +} +``` + +| Field | Description | +| --- | --- | +| `version_at` | The FlatKV version that was opened. | +| `migration_version` | The on-disk migration version (`0` = memiavl-only, `1` = EVM migrated). | +| `migrate_evm_complete` | `true` once `migration_version` has reached the EVM-migrated version. | +| `boundary_present` | `true` while the migration is in flight (the boundary cursor is present). | +| `boundary_hex` | Hex encoding of the in-flight boundary cursor; omitted when absent. | +| `version_raw_hex` | Hex encoding of the raw `migration-version` bytes; omitted when absent. | + +A migration is complete when `migrate_evm_complete` is `true` and `boundary_present` is `false`. Poll every validator until all report completion before flipping `sc-write-mode` from `migrate_evm` to `evm_migrated`. + + + + +## Comparing Backends with `evm-logical-digest` + +The `evm-logical-digest` command computes a backend-independent digest of the EVM logical state (the canonical `account`, `code`, and `storage` buckets) so a memIAVL node and a FlatKV node can be compared at the same chain height. Because the two backends store the same EVM state in different physical layouts, a naive byte-for-byte comparison diverges: every FlatKV value embeds a per-key `blockHeight` stamp recording when the key was last written or migrated, and a freshly migrated FlatKV node stamps migration-time heights that differ from the memIAVL leaf versions. This tool strips the serialization-version and `blockHeight` header on both sides and digests only the height-independent logical payload (storage word, bytecode, and balance/nonce/codeHash), so identical EVM state produces identical digests regardless of backend. + +Each bucket is accumulated as an order-independent XOR of `sha256(len(key) || key || len(val) || val)`, so it does not matter that FlatKV iterates in Pebble global order while memIAVL is scanned by leaf index. The command prints a `bucket_digest` line per bucket and a single `FINAL_DIGEST` line covering `account+code+storage+legacy`; the two backends' `FINAL_DIGEST` values should match when the underlying state is identical. + +The `legacy` bucket is reported separately, along with a marker-adjusted comparison line, because a migrated FlatKV node can contain a FlatKV-only `migration/migration-version` row that a memIAVL-only node never owns. That row is folded into the legacy bucket but omitted from the final comparison so the two sides line up apples-to-apples. + +### Normalization modes + +For the memIAVL backend, `--memiavl-normalization` selects how raw EVM leaves are turned into logical buckets: + +- **`semantic`** (default, also accepted as `independent`) — independently decodes raw EVM keys and values into the same `account` / `code` / `storage` / `legacy` buckets without calling `flatkv.ImportTranslator`. +- **`translator`** — feeds each EVM leaf through `flatkv.ImportTranslator`, applying the exact same `classifyAndPrefix` and account-merge logic FlatKV uses. This is useful for proving FlatKV state matches the current migration mapping and for debugging the translator. + +FlatKV reads WAL-replay to the requested `--height`; memIAVL does not replay WAL and instead opens `snapshot-/evm`, or `current/evm` when `--height` is `0`. + +### Usage + +```bash +# FlatKV digest at a height (WAL-replays to it). +seidb evm-logical-digest --backend flatkv \ + --db-dir /path/to/.sei/data/state_commit/flatkv --height 213200000 + +# memIAVL digest at the same height using the default semantic decoder. +seidb evm-logical-digest --backend memiavl \ + --db-dir /path/to/.sei/data/state_commit/memiavl --height 213200000 + +# Translator-based memIAVL digest. +seidb evm-logical-digest --backend memiavl \ + --db-dir /path/to/.sei/data/state_commit/memiavl --height 213200000 \ + --memiavl-normalization translator + +# Inspect one bucket instead of the global digest: list storage rows under a +# key prefix, sharded by the next 2 bytes. +seidb evm-logical-digest --backend flatkv -d /path/to/flatkv --height 213200000 \ + --inspect-bucket storage --key-prefix 03 --shard-next-bytes 2 + +# List account rows with backend-specific version metadata. +seidb evm-logical-digest --backend flatkv -d /path/to/flatkv --height 213200000 \ + --inspect-bucket account --list --list-limit 50 --details +``` + +### Flags + +| Flag | Alias | Default | Description | +| --- | --- | --- | --- | +| `--backend` | | | Backend to read: `flatkv` or `memiavl`. Required. | +| `--db-dir` | `-d` | | For `flatkv`: the FlatKV data directory. For `memiavl`: the memIAVL root directory (contains `current/` and `snapshot-*`). Required. | +| `--height` | | `0` | Target version. FlatKV WAL-replays to it; memIAVL resolves `snapshot-/evm` (`0` selects the `current` symlink). | +| `--memiavl-normalization` | | `semantic` | memIAVL normalization: `semantic`/`independent` (raw EVM key/value decoder) or `translator` (current migration mapping). | +| `--inspect-bucket` | | | Inspect one normalized bucket (`account`, `code`, `storage`, or `legacy`) instead of printing the global digest. | +| `--key-offset` | | `0` | Inspect mode: byte offset into the physical key before applying `--key-prefix` / sharding. | +| `--key-prefix` | | | Inspect mode: hex prefix, relative to `--key-offset`, used to filter physical keys. | +| `--shard-next-bytes` | | `0` | Inspect mode: group matching keys by this many bytes after `--key-prefix`. | +| `--list` | | `false` | Inspect mode: list matching key/logical-value pairs instead of shard `bucket_digest` values. | +| `--list-limit` | | `1000` | Inspect mode: maximum pairs to print with `--list`; `<= 0` means unlimited. | +| `--details` | | `false` | Inspect list mode: include backend-specific version metadata. | +| `--find-hash` | | | Optional 32-byte hex per-entry hash to hunt for. When two `bucket_digest` values differ by exactly one entry, their XOR equals that entry's hash; every entry matching this hash is printed as a `FOUND-HASH` line. | + +To locate a single diverging row between two runs, XOR the two differing 32-byte `bucket_digest` hex values and pass the result to `--find-hash`; every matching entry is printed with its bucket, physical key, and values. + diff --git a/learn/twin-turbo-consensus.mdx b/learn/twin-turbo-consensus.mdx index 919547f..6490542 100644 --- a/learn/twin-turbo-consensus.mdx +++ b/learn/twin-turbo-consensus.mdx @@ -14,7 +14,7 @@ The key to achieving sub-second finality lies in aggressively optimizing and par This optimized flow involves several key enhancements: -1. **Aggressive Timeout Configuration:** Sei utilizes heavily tuned Tendermint consensus parameters. Configuration settings (e.g., `UnsafeProposeTimeoutOverride`, `UnsafeCommitTimeoutOverride`) enforce much shorter durations for block proposal, voting, and commit rounds compared to standard Tendermint configurations, directly contributing to the sub-second target block time. Faster gossip propagation for consensus messages further reduces communication latency between validators. +1. **Aggressive Timeout Configuration:** Sei utilizes heavily tuned Tendermint consensus parameters. Configuration settings (e.g., `UnsafeProposeTimeoutOverride`, `UnsafeCommitTimeoutOverride`) can enforce much shorter durations for block proposal, voting, and commit rounds compared to standard Tendermint configurations, directly contributing to the sub-second target block time. These `Unsafe*TimeoutOverride` fields are gated by the `unsafe-overrides-enabled` flag under the `[consensus]` section of the node config. This flag defaults to `false`, meaning the overrides are ignored and the on-chain timeout consensus parameters are used instead. The overrides are only applied when `unsafe-overrides-enabled` is set to `true` (or, during the transition period, while the on-chain timeout params still match the legacy values). In practice, timeout tuning should be governed by the on-chain consensus parameters rather than these unsafe per-node overrides. Faster gossip propagation for consensus messages further reduces communication latency between validators. 2. **Intelligent Mempool Management & Transaction Preparation:** Even before a block proposal is formally initiated for height `H`, validators can begin processing transactions intended for that block. This involves collecting transactions from the network, decoding them concurrently (`DecodeTransactionsConcurrently`), analyzing potential state dependencies (`GenerateEstimatedWritesets`), and potentially pre-fetching required state data from SeiDB. This "pre-consensus" preparation minimizes the work needed once the actual proposal for height `H` arrives. 3. **Optimized BFT Rounds with Parallel Execution Integration:** The critical optimization is the deep integration with Sei's parallelization engine. When a validator receives a block proposal for height `H`, it doesn't necessarily wait for the prevote/precommit rounds to complete before starting execution. Instead: - The block's transactions are dispatched to the parallel execution engine (`ProcessTXsWithOCC`, `DeliverTxBatch`). diff --git a/node/advanced-config-monitoring.mdx b/node/advanced-config-monitoring.mdx index 4fc13d9..b4232af 100644 --- a/node/advanced-config-monitoring.mdx +++ b/node/advanced-config-monitoring.mdx @@ -447,6 +447,326 @@ seid query slashing signing-info $(seid tendermint show-validator) seid query staking delegations-to $(seid keys show -a $VALIDATOR_KEY) ``` + +### ProposerPriority Divergence Detection + +Sei nodes export two Prometheus gauges that let operators detect when a validator's `ProposerPriority` state diverges from the rest of the network. Divergence indicates corrupted consensus state and should be investigated immediately. + +| Metric | Type | Description | +| --- | --- | --- | +| `tendermint_state_proposer_priority_hash` | gauge | Encodes the first 8 bytes of the hash of the current validator set's proposer priorities, packed as a big-endian `uint64` and cast to a `float64`. | +| `tendermint_state_proposer_priority_hash_height` | gauge | The block height at which the most recent `tendermint_state_proposer_priority_hash` was computed. | + +Both metrics are emitted together every 1024 heights (roughly every few minutes at Sei block times). Emitting the hash as a numeric value instead of a Prometheus label keeps series cardinality constant at one series per node, rather than creating a new time series on every priority change. + +#### How to use these metrics + +Compare `tendermint_state_proposer_priority_hash` across your validators, but **only compare samples taken at the same `tendermint_state_proposer_priority_hash_height`**. Since the hash is only meaningful at a shared height, always pair it with the height gauge before comparing: + +- If every node reports the **same** hash value at the same height, their `ProposerPriority` state agrees. +- If a node reports a **different** hash value at the same height, its `ProposerPriority` state has diverged and likely indicates corrupted state on that node. + +Example alert rule that flags divergence across scraped nodes at a shared height: + +```yaml +groups: + - name: proposer_priority_alerts + rules: + - alert: ProposerPriorityDivergence + expr: > + count( + count by (tendermint_state_proposer_priority_hash) ( + tendermint_state_proposer_priority_hash + * on(instance) group_left + (tendermint_state_proposer_priority_hash_height == scalar(max(tendermint_state_proposer_priority_hash_height))) + ) + ) > 1 + for: 10m + labels: + severity: critical + annotations: + summary: 'ProposerPriority has diverged between validators' +``` + +Each checkpoint also writes a `proposer priority hash checkpoint` log line containing the full 32-byte hash and the packed value, which can be used for grep-based comparison across nodes. + + + +## EVM RPC Metrics + +Sei nodes emit OpenTelemetry-based metrics for the EVM JSON-RPC layer through the process-wide `MeterProvider` (for example, a Prometheus exporter). These `evmrpc_*` metrics are the recommended source for monitoring RPC performance and websocket activity. + + + + + The OpenTelemetry Prometheus exporter namespace is `sei_chain` (underscore). When these metrics are scraped through the Prometheus exporter, exported series are prefixed accordingly (for example, `sei_chain_evmrpc_request_latency_seconds`, `sei_chain_flatkv_commit_latency`, `sei_chain_app_abci_commit_duration_seconds`, `sei_chain_module_mid_block_duration`). The metric names documented on this page are the unprefixed instrument names; prepend the `sei_chain_` namespace when querying them in Prometheus/Grafana. This namespace was previously `sei-chain` (hyphen); because Prometheus normalizes the hyphen to an underscore anyway, existing dashboards and alert queries should continue to match, but confirm your PromQL uses the `sei_chain_` prefix after upgrading. + + +| Metric | Type | Description | +| --- | --- | --- | +| `evmrpc_request_latency_seconds` | histogram | RPC request latency in seconds, labeled by endpoint, connection, success, error class, and JSON-RPC code bucket. | +| `evmrpc_websocket_connects_total` | counter | Number of new websocket connections. | +| `evmrpc_redirected_requests_total` | counter | Number of EVM RPC requests forwarded (proxied) to another validator, labeled by endpoint and connection. Emitted when a request such as `eth_sendRawTransaction` or a pending `eth_getTransactionCount` is redirected to the validator that owns the sender's EVM address shard. | + + +| `evmrpc_historical_debug_trace_attempts_total` | counter | Number of `debug_trace*` requests targeting historical blocks, labeled by endpoint and connection. Incremented whenever a `debug_trace*` request (`debug_traceTransaction`, `debug_traceTransactionProfile`, `debug_traceBlockByNumber`, `debug_traceBlockByHash`, `debug_traceCall`, or `debug_traceStateAccess`) targets a block older than the configured `maxBlockLookback`. Such requests are rejected with an error like `block number N is beyond max lookback of M`. | + +### `evmrpc_historical_debug_trace_attempts_total` labels + +| Label | Description | +| --- | --- | +| `endpoint` | The `debug_trace*` method that attempted the historical block (e.g. `debug_traceTransaction`, `debug_traceBlockByNumber`, `debug_traceCall`). | +| `connection` | The connection type that received the request (e.g. `http`, `websocket`). | + +### `evmrpc_redirected_requests_total` labels + +| Label | Description | +| --- | --- | +| `endpoint` | The RPC method being forwarded (e.g. `eth_sendRawTransaction`, `eth_getTransactionCount`). | +| `connection` | The connection type that received the original request (e.g. `http`, `websocket`). | + +### `evmrpc_request_latency_seconds` labels + +| Label | Description | +| --- | --- | +| `endpoint` | The RPC method being served (e.g. `eth_call`, `eth_getBalance`). | +| `connection` | The connection type serving the request (e.g. `http`, `websocket`). | +| `success` | Boolean indicating whether the request completed without error or panic. | +| `error_class` | Low-cardinality classification of the failure. Empty for successful requests. Possible values: `panic`, `execution_reverted`, `evm_not_supported`, `sei_legacy_disabled`, `association_missing`, `jsonrpc_error`, `unknown`. | +| `jsonrpc_code` | Bucketed JSON-RPC error code. Empty when there is no code (success or an untyped error). Possible values: `spec` (predefined codes `-32700`..`-32600`), `server` (server-defined codes `-32099`..`-32000`), and `other`. | + +The histogram uses the following explicit bucket boundaries (in seconds): + +``` +0.0005, 0.001, 0.0025, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30 +``` + +### Legacy metric deprecation + +The legacy `sei_*` RPC metrics — including `sei_rpc_request_latency_ms` and `sei_websocket_connect` — are still emitted alongside the new `evmrpc_*` metrics for backward compatibility, but they are deprecated and scheduled for removal (PLT-326) once dashboards migrate to the OpenTelemetry `evmrpc_*` metrics. When building or updating dashboards and alerts, prefer the `evmrpc_request_latency_seconds` histogram and `evmrpc_websocket_connects_total` counter over their legacy counterparts. + + + +## FlatKV State DB Metrics + +Sei nodes emit OpenTelemetry-based metrics for the FlatKV state database through the process-wide `MeterProvider` (for example, a Prometheus exporter). These `flatkv_*` metrics let operators observe FlatKV performance and progress across commits, catchup, snapshots, rollbacks, and snapshot imports. + +### Latency metrics + +Each latency metric is a histogram reported in seconds (unit `s`). Unless otherwise noted, they carry a `success` boolean label indicating whether the operation completed without error. + +| Metric | Type | Labels | Description | +| --- | --- | --- | --- | +| `flatkv_open_latency` | histogram | `success`, `read_only` | Time taken to open the FlatKV store (`LoadVersion`). | +| `flatkv_apply_changesets_latency` | histogram | `success` | Time taken to apply changesets to FlatKV. | +| `flatkv_commit_latency` | histogram | `success` | Time taken to commit FlatKV changes. | +| `flatkv_commit_batch_latency` | histogram | `success`, `db` | Time taken to commit a per-DB PebbleDB batch. | +| `flatkv_batch_read_old_values_latency` | histogram | `success` | Time taken to batch read old FlatKV values. | +| `flatkv_catchup_latency` | histogram | `success` | Time taken to replay FlatKV WAL entries during catchup. | +| `flatkv_snapshot_write_latency` | histogram | `success` | Time taken to write a FlatKV snapshot. | +| `flatkv_snapshot_prune_latency` | histogram | — | Time taken to prune old FlatKV snapshots. | +| `flatkv_rollback_latency` | histogram | `success` | Time taken to roll back FlatKV state. | +| `flatkv_import_latency` | histogram | `success` | Time taken to import FlatKV snapshot data. | +| `flatkv_import_worker_flush_latency` | histogram | `success`, `db` | Time taken to flush a FlatKV import worker batch. | +| `flatkv_flush_latency` | histogram | `success`, `db` | Time taken to flush a per-DB data DB. | + +### Counter metrics + +| Metric | Type | Labels | Description | +| --- | --- | --- | --- | +| `flatkv_num_kv_pairs` | counter | `db` | Number of key-value pairs applied to FlatKV. | +| `flatkv_catchup_replay_num_blocks` | counter | — | Number of FlatKV WAL entries replayed during catchup. | +| `flatkv_snapshot_prune_attempts` | counter | `success` | Total number of FlatKV snapshot prune attempts. | +| `flatkv_import_kv_pairs` | counter | `db` | Number of key-value pairs imported into FlatKV. | + +### Gauge metrics + +| Metric | Type | Labels | Description | +| --- | --- | --- | --- | +| `flatkv_pending_writes` | gauge | `db` | Current number of pending FlatKV writes per data DB. | +| `flatkv_current_version` | gauge | — | Current committed FlatKV version. | +| `flatkv_current_snapshot_height` | gauge | — | Current FlatKV snapshot height. | + +The `db` label identifies the underlying data DB (for example, the account, storage, code, or legacy data directory), letting operators break down applied writes, pending writes, batch commits, and flush latency per DB. + + + Emission of Pebble's own internal metrics is controlled separately by the FlatKV-level `EnablePebbleMetrics` configuration knob. When set, it overrides the per-DB `EnableMetrics` settings for all data DBs (account, code, storage, legacy, and metadata), so Pebble internal metrics are toggled uniformly rather than individually per DB. + + + + Prometheus gauges are held in memory, so after a process restart these gauges reset to zero until the next emission at the following multiple of 1024 heights (up to roughly 8.5 minutes of stale or zero data). This is expected for a monitoring signal that is only consulted in response to incidents. + + + + +## App (ABCI) Metrics + +Sei nodes emit OpenTelemetry-based metrics for the application (ABCI) layer through the process-wide `MeterProvider` (for example, a Prometheus exporter). These `app_*` metrics let operators observe ABCI phase durations, transaction throughput and gas usage, block processing, and light invariance checks. They are the recommended source for application-level observability. + + + A set of legacy telemetry metrics (for example, the `abci`/`tx`/`sei_lightinvariance_supply` series) are still emitted alongside the new `app_*` metrics for backward compatibility, but they are deprecated and scheduled for removal (PLT-327) once dashboards migrate to the OpenTelemetry `app_*` metrics. When building or updating dashboards and alerts, prefer the `app_*` metrics over their legacy counterparts. + + +### ABCI phase duration metrics + +Each of these is a histogram reported in seconds (unit `s`), measuring the duration of the corresponding ABCI phase. + +| Metric | Type | Description | +| --- | --- | --- | +| `app_abci_begin_block_duration_seconds` | histogram | Duration of ABCI `BeginBlock`. | +| `app_abci_end_block_duration_seconds` | histogram | Duration of ABCI `EndBlock`. | +| `app_abci_module_end_block_duration_seconds` | histogram | Duration of module `EndBlock` calls within ABCI `EndBlock`. | +| `app_abci_check_tx_duration_seconds` | histogram | Duration of ABCI `CheckTx`. | +| `app_abci_deliver_tx_duration_seconds` | histogram | Duration of ABCI `DeliverTx`. | +| `app_abci_deliver_batch_tx_duration_seconds` | histogram | Duration of ABCI `DeliverTxBatch`. | +| `app_abci_commit_duration_seconds` | histogram | Duration of ABCI `Commit` (state write to disk). | + +### Block processing metric + +| Metric | Type | Attributes | Description | +| --- | --- | --- | --- | +| `app_block_process_duration_seconds` | histogram | `type` | Duration of block transaction processing by execution type. | + +The `type` attribute identifies the execution path: `synchronous`, `synchronous_giga`, `optimistic_concurrency`, or `occ_giga`. + +### Transaction counter metrics + +| Metric | Type | Attributes | Description | +| --- | --- | --- | --- | +| `app_tx_count_total` | counter | `result` | Number of transactions delivered, labeled by result (for example, `successful`). | +| `app_tx_process_type_total` | counter | `type` | Transactions processed by execution type (`synchronous`, `synchronous_giga`, `optimistic_concurrency`, `occ_giga`). | +| `app_tx_gas_total` | counter | `type` | Cumulative transaction gas, where `type` is `gas_used` or `gas_wanted`. | + +### App flow counter metrics + +| Metric | Type | Attributes | Description | +| --- | --- | --- | --- | +| `app_optimistic_processing_total` | counter | `enabled` | Optimistic processing attempts; `enabled:true` means the optimistic result was used, `false` means it was discarded. | +| `app_failed_total_gas_wanted_check_total` | counter | `proposer` | Proposals rejected because total block gas wanted exceeded the maximum; `proposer` is the hex-encoded proposer address. | +| `app_giga_fallback_to_v2_total` | counter | — | Number of times the giga executor fell back to V2 processing. | +| `app_pending_nonce_total` | counter | `event` | Pending nonce events, where `event` is `added`, `expired`, `rejected`, or `accepted`. | + +### Light invariance metrics + +| Metric | Type | Attributes | Description | +| --- | --- | --- | --- | +| `app_lightinvariance_supply_duration_seconds` | histogram | — | Duration of the light invariance total supply check. | +| `app_lightinvariance_supply_invalid_key_total` | counter | `type` | Invalid changed-pair keys detected during the invariance check (`type` is `sei` or `wei`). | +| `app_lightinvariance_supply_unmarshal_failure_total` | counter | `type`, `step` | Unmarshal failures during the invariance supply check (`type` is `usei`, `wei`, or `total_supply`; `step` is `pre_block` or `post_block`). | + +### Build info metric + +| Metric | Type | Attributes | Description | +| --- | --- | --- | --- | +| `app_build_info` | gauge | `seid_version`, `commit` | Running binary build info; the value is always `1`, with the running version and commit exposed as attributes. This observable gauge is populated by a scrape callback, so it reflects the currently running binary on every scrape. | + + + + +## Module & Governance Metrics + +Sei nodes emit OpenTelemetry-based metrics for the SDK module lifecycle (mid-block, begin-blocker, and end-blocker execution) and for validator slash events through the process-wide `MeterProvider` (for example, a Prometheus exporter). These `module_*`, per-module `*_blocker_duration`, and `validator_slashed` metrics let operators observe where per-block module execution time is spent and track slashing activity. + + + These OpenTelemetry metrics are emitted alongside the existing legacy telemetry counterparts, which are retained pending verification (PLT-414). When building or updating dashboards and alerts, prefer these OpenTelemetry metrics over their legacy counterparts. + + +### Mid-block duration metrics + +Each of these is a histogram reported in seconds (unit `s`), measuring the duration of module mid-block execution. + +| Metric | Type | Attributes | Description | +| --- | --- | --- | --- | +| `module_total_mid_block_duration` | histogram | — | Total duration of all modules' mid-block execution in seconds. | +| `module_mid_block_duration` | histogram | `module` | Duration of per-module mid-block execution in seconds, labeled by module name. | + +### Per-module begin/end-blocker duration metrics + +Each of these is a histogram reported in seconds (unit `s`), measuring the duration of the corresponding module's begin-blocker or end-blocker execution. + +| Metric | Type | Description | +| --- | --- | --- | +| `capability_begin_blocker_duration` | histogram | Duration of capability begin-blocker execution in seconds. | +| `crisis_end_blocker_duration` | histogram | Duration of crisis end-blocker execution in seconds. | +| `crisis_init_genesis_unmarshal_duration` | histogram | Duration of crisis `InitGenesis` JSON unmarshal in seconds. | +| `distribution_begin_blocker_duration` | histogram | Duration of distribution begin-blocker execution in seconds. | +| `evidence_begin_blocker_duration` | histogram | Duration of evidence begin-blocker execution in seconds. | +| `gov_end_blocker_duration` | histogram | Duration of gov end-blocker execution in seconds. | +| `slashing_begin_blocker_duration` | histogram | Duration of slashing begin-blocker execution in seconds. | +| `staking_begin_blocker_duration` | histogram | Duration of staking begin-blocker execution in seconds. | +| `staking_end_blocker_duration` | histogram | Duration of staking end-blocker execution in seconds. | + +### Validator slash counter + +| Metric | Type | Attributes | Description | +| --- | --- | --- | --- | +| `validator_slashed` | counter | `type`, `validator` | Number of validator slash events, labeled by slash reason (`type` is `missing_signature` or `double_sign`) and the consensus address of the slashed validator. | + +### Staking keeper metrics + +The staking keeper delegation metrics are emitted with a `staking_keeper_` prefix to avoid collisions with other metric namespaces. + +| Metric | Type | Description | +| --- | --- | --- | +| `staking_keeper_delegate` | counter | Number of delegation transactions. | +| `staking_keeper_last_delegate_amount` | gauge | Amount delegated in the last delegation transaction (in `usei`). | +| `staking_keeper_redelegate` | counter | Number of redelegation transactions. | +| `staking_keeper_last_redelegate_amount` | gauge | Amount redelegated in the last redelegation transaction (in `usei`). | +| `staking_keeper_undelegate` | counter | Number of undelegation transactions. | +| `staking_keeper_last_undelegate_amount` | gauge | Amount undelegated in the last undelegation transaction (in `usei`). | + + + The staking keeper metrics were previously emitted without the `staking_keeper_` prefix (for example, `delegate`, `last_delegate_amount`). Update any dashboards or alerts that reference the old unprefixed names. + + +The duration histograms use the following explicit bucket boundaries (in seconds): + +``` +0.000025, 0.000050, 0.0001, 0.0005, 0.001, 0.0025, 0.005, 0.010, 0.020, 0.050, 0.075, 0.1, 0.25, 0.5, 1, 10 +``` + + + + +## Consensus Validation Metrics + +Sei nodes emit an OpenTelemetry-based counter through the process-wide `MeterProvider` (for example, a Prometheus exporter) that tracks halting consensus validation failures swallowed by non-default `ConsensusPolicy` builds. + +| Metric | Type | Attributes | Description | +| --- | --- | --- | --- | +| `sei_unsafe_validation_skipped_total` | counter | `validation_error` | Number of halting consensus validation failures that were swallowed (counted and continued instead of halting the chain) by a non-default `ConsensusPolicy`. The `validation_error` attribute identifies the specific validation failure that was skipped. | + + + In production (default) builds this counter is **always zero** — every validation failure halts the node as expected. A non-zero value is only ever emitted by special-purpose binaries built with the `mock_block_validation` or `mock_chain_validation` build tags (published as `sei-chain:mock_chain_validation-*` and `sei-chain:mock_chain_validation-nightly-*` Docker images). These binaries intentionally bypass halting validation and must never be run on mainnet or any node whose state you trust; they exist for forked-state replays and similar diagnostic scenarios. + + +Build-tag behavior: + +- **default (production)** — no failures are swallowed; the counter stays at zero and every validation failure halts. +- **`mock_block_validation`** — swallows only `app_hash` and `data_hash` failures, preserving that tag's long-standing behavior; all other failures still halt. +- **`mock_chain_validation`** — swallows every swallow-eligible failure except `last_commit_verify` (which is excluded to avoid a downstream panic and therefore still halts). Intended for forked-state replays. + +### `sei_unsafe_validation_skipped_total` `kind` values + +| `kind` | Swallowed validation failure | +| --- | --- | +| `app_hash` | Block `AppHash` did not match the expected application hash. | +| `data_hash` | Block `DataHash` did not match the hash of the block data. | +| `last_results_hash` | Block `LastResultsHash` did not match the expected value. | +| `last_block_id` | Block `LastBlockID` did not match the expected previous block ID. | +| `consensus_hash` | Block `ConsensusHash` did not match the hash of the consensus params. | +| `validators_hash` | Block `ValidatorsHash` did not match the current validator set hash. | +| `next_validators_hash` | Block `NextValidatorsHash` did not match the next validator set hash. | +| `last_commit_verify` | `LastCommit` verification failed. Never swallowed by `mock_chain_validation` (excluded from the swallow set) — always halts. | +| `proposer_not_in_validator_set` | Block proposer address is not a member of the validator set. | +| `evidence_overflow` | Block evidence exceeded the maximum allowed byte size. | +| `last_commit_hash` | Block `LastCommitHash` did not match the expected value. | +| `evidence_hash` | Block `EvidenceHash` did not match the hash of the block evidence. | +| `per_evidence_validate_basic` | An individual evidence item failed `ValidateBasic`. | + + + + ## Backup Management diff --git a/node/giga-storage-migration.mdx b/node/giga-storage-migration.mdx index 755f614..2b81844 100644 --- a/node/giga-storage-migration.mdx +++ b/node/giga-storage-migration.mdx @@ -21,6 +21,155 @@ Only the **SS** layer changes for this migration. SC layer config is untouched and `memiavl` remains the authoritative source for the app hash, so this is invisible to the network. + +The SC-layer routing field `sc-write-mode` is emitted in the generated +`app.toml` template under the `[state-store]` section. It defaults to +`memiavl_only`, so leaving it at the default keeps the SC layer untouched for +this migration. + +- `sc-write-mode` — write routing mode for EVM data in the SC layer. Valid + values: `memiavl_only`, `migrate_evm`, `evm_migrated`, `migrate_all_but_bank`, + `all_migrated_but_bank`, `migrate_bank`, `flatkv_only`, `test_only_dual_write`. + An invalid value fails at config parse time with a clear error. +- `sc-keys-to-migrate-per-block` — the number of keys migrated from `memiavl` to + `flatKV` per block while in a migration mode. Defaults to `1024` and must be + greater than `0`; ignored outside of a migration mode. + +The legacy `sc-read-mode` and `sc-enable-lattice-hash` fields have been +**removed**. Read routing and lattice-hash participation are now derived +automatically from the write mode and the on-disk migration state — an +`app.toml` that still references either field is no longer valid. + + + + +The `sc-write-mode` / `sc-read-mode` fields above configure EVM routing in the +legacy composite commit store and are **distinct** from the internal +`memiavl → flatKV` **migration state machine** used when converting the SC +layer's state DB from `memiavl` to `flatKV`. That migration is driven by its +own `WriteMode` enum inside the state-migration package, not by +`sc-write-mode`. Node operators encounter these modes as the migration +progresses through a linear sequence of on-disk **migration versions**: + +| WriteMode | Migration version | Behavior | +|---|---|---| +| `MemiavlOnly` (`memiavl_only`) | 0 | Pre-migration baseline: every module routes to `memiavl`; `flatKV` is not in the data path (bootstrap passes `nil` for `flatKV`). | +| `MigrateEVM` (`migrate_evm`) | 0 → 1 | Migrates the `evm/` module from `memiavl` to `flatKV` in batches. Reads for un-migrated keys are served from `memiavl` first and fall back to `flatKV` for brand-new keys created after the migration started; brand-new writes are routed to `flatKV` so the migration boundary can reach completion instead of chasing an ever-growing key tail. Iteration is forwarded to the `memiavl` iterator while the migration is `NotStarted`/`InProgress` (with the caveat that already-migrated keys are no longer visible there, so results may be incomplete mid-flight) and is refused once the migration is `Complete`. | +| `EVMMigrated` (`evm_migrated`) | 1 | Steady state: `evm/` lives in `flatKV`, every other module in `memiavl`; no migration manager in the path. | +| `MigrateAllButBank` (`migrate_all_but_bank`) | 1 → 2 | Migrates every module except `bank/` (and the already-migrated `evm/`) from `memiavl` to `flatKV`. | +| `AllMigratedButBank` (`all_migrated_but_bank`) | 2 | Steady state: everything except `bank/` lives in `flatKV`; `bank/` remains in `memiavl`. | +| `MigrateBank` (`migrate_bank`) | 2 → 3 | Migrates the final `bank/` module from `memiavl` to `flatKV`. | +| `FlatKVOnly` (`flatkv_only`) | 3 | Terminal state: every module routes to `flatKV`; `memiavl` is not in the data path (bootstrap passes `nil` for `memiavl`). A node can be booted **directly** into this steady state by setting `sc-write-mode = "flatkv_only"` in `app.toml` — `memiavl` is never allocated. Snapshot export and state-sync restore work correctly in this mode: the State Store is fully repopulated and the restored AppHash matches the snapshot source (the FlatKV exporter now emits its own module header, and empty-value writes are preserved rather than dropped on WAL replay). | +| `TestOnlyDualWrite` (`test_only_dual_write`) | — | Test-only mode that dual-writes `evm/` traffic to both `memiavl` and `flatKV` while routing all other modules to `memiavl`; reads, proofs, and iteration are served exclusively by `memiavl`. | + + +`TestOnlyDualWrite` exists only to preserve parity with legacy +composite-store dual-write tests and **must never be deployed to production +machines**. + + +The appropriate router for a given `WriteMode` is constructed by the +state-migration package's `BuildRouter` entrypoint, which returns the +steady-state or in-flight migration router for that mode. Router construction +requires a non-`nil` `memiavl` handle for every mode except `FlatKVOnly`, and a +non-`nil` `flatKV` handle for every mode except `MemiavlOnly`. + + + +## Offline FlatKV EVM import (MigrateEVM) + +The `seidb` tool ships an offline import path that moves the `evm/` module's +SC-layer data out of `memiavl` and into FlatKV without running the in-process +migration state machine. This is the operational entrypoint for the +`MigrateEVM` (V0 → V1) transition on a stopped node. + + +This is an **offline** operation. The node must be fully stopped before +running the import — the tool opens both the `memiavl` and FlatKV directories +directly and will conflict with a running `seid`. + + +### Reading the latest memiavl version + +Before importing, read the latest committed `memiavl` version from a stopped +node's data directory. In a multi-validator cluster, run this on every node +and pick the minimum so the import height is uniform across the cluster: + +```bash copy +seidb memiavl-latest-version --data-dir +``` + +`--data-dir` may point at either the `data/` directory or the Sei home +directory; if the basename is `data`, its parent is treated as home. You can +also pass `--home ` instead. + +### Running the import + +```bash copy +seidb import-flatkv-from-memiavl \ + --modules=evm \ + --data-dir \ + --height \ + [--force] +``` + +| Flag | Meaning | +|---|---| +| `--home` | Sei home directory. Defaults to `$HOME/.sei`. | +| `--data-dir` | Sei data directory or home directory. If the basename is `data`, its parent is used as home. | +| `--modules` | Comma-separated module names to import. Initial production scope is **evm-only**; any other module name is rejected. | +| `--height` | The `memiavl` version to import. `0` means latest. | +| `--force` | Overwrite existing committed FlatKV data. | + +The import resets FlatKV before loading the selected `memiavl` rows and +**refuses to overwrite committed FlatKV data unless `--force` is supplied**. If +an external error interrupts the import (context cancellation, exporter or +translator failure), the import is aborted rather than finalized — FlatKV is +left at its pre-import committed version, so the operation can be retried +without `--force`. + + +The import **must run at the memiavl latest version**. Importing FlatKV at a +height `H` lower than the latest `memiavl` version is rejected: on the next +`GIGA_STORAGE` startup the composite commit store's `reconcileVersions` step +would silently roll `memiavl` back to `H`, truncating every cosmos block in +`(H, latest]`. If you genuinely need a non-latest height, roll `memiavl` back +to that height yourself first (for example with `seid rollback`), then re-run +the import. The CLI deliberately does **not** roll `memiavl` back on your +behalf. A height ahead of the latest `memiavl` version is likewise rejected. + + +### Configuration constraints across the import boundary + +The import moves only the SC-layer EVM data into FlatKV. When restarting the +node after the import, two settings **must stay off** across the import +boundary or the node will panic on startup: + +- **`evm-ss-split = false`.** SS history for EVM remains in the existing + combined cosmos SS database; the import does not populate a separate EVM SS + directory. Flipping `evm-ss-split` to `true` triggers the rootmulti startup + panic *"EVM SS directory ... does not exist but Cosmos SS already has + history"*. Moving the SS layer to split mode is a separate state-sync + workflow (see [Step 2](#step-2%3A-state-sync-into-the-new-layout)) and is out + of scope for the offline SC import. +- **`sc-enable-lattice-hash = false`.** Before the import the chain ran without + FlatKV, so tendermint persisted app hashes computed from `memiavl` alone for + every block up to the import height. Enabling the lattice hash now would fold + the FlatKV LtHash into the app hash, and the replay check at startup would + fail with *"state.AppHash does not match AppHash after replay"*. Note that + `dual_write` does not require the lattice hash — only `split_write` does. + +A production rollout coordinates these transitions via a chain upgrade at an +agreed height rather than flipping them mid-life. + + +For the full operational failure-mode catalog and recovery/tooling roadmap for +MigrateEVM, see +[`sei-db/state_db/sc/migration/OPERATIONS.md`](https://github.com/sei-protocol/sei-chain/blob/main/sei-db/state_db/sc/migration/OPERATIONS.md) +in `sei-chain`. + + + This guide tracks the canonical procedure in [`docs/migration/giga_store_migration.md`](https://github.com/sei-protocol/sei-chain/blob/main/docs/migration/giga_store_migration.md) inside `sei-chain`. Open an issue there if anything here drifts. ## Prerequisites @@ -192,6 +341,29 @@ If any check fires, the correct fix is either (a) complete the state sync described above, or (b) set `evm-ss-split = false` and restart. If `data/evm_ss/` is stale from a failed attempt, remove it before state syncing. + + +## Receipt backend default + +When Giga Storage is enabled (`GIGA_STORAGE=true`), the receipt backend now +defaults to `pebble`. Previously the receipt backend was left unchanged and +had to be set explicitly through the `RECEIPT_BACKEND` environment variable. + +This default is applied implicitly: enabling Giga Storage sets +`RECEIPT_BACKEND=pebble` unless you have already provided an explicit value. +To use a different receipt backend, set `RECEIPT_BACKEND` explicitly before +starting the node — an explicit value always takes precedence over the +pebble default. + +```bash copy +# Giga Storage on, receipt backend defaults to pebble +export GIGA_STORAGE=true + +# Override the pebble default with an explicit backend +export GIGA_STORAGE=true +export RECEIPT_BACKEND= +``` + ## Rollback To roll back: @@ -209,10 +381,21 @@ To fully reclaim the disk used by EVM SS, stop the node and delete ### Where do the data files live after migrating? -- Cosmos SS data lives under the same directory as before, typically - `data/pebbledb/` for the default `pebbledb` backend. -- EVM SS data lives under `data/evm_ss/`. -- SC data (`memiavl` + FlatKV) is untouched by this migration. +New nodes use an organized subdirectory layout under `data/`. Existing nodes +with data in the legacy flat layout keep using their legacy paths automatically +(legacy takes precedence when present). + +- Cosmos SS data lives under `data/state_store/cosmos/{backend}` on new nodes + (e.g. `data/state_store/cosmos/pebbledb/` for the default `pebbledb` backend). + Existing nodes with data under `data/{backend}` (e.g. `data/pebbledb/`) + continue using that legacy path. +- EVM SS data lives under `data/state_store/evm/{backend}` on new nodes + (e.g. `data/state_store/evm/pebbledb/`). Existing nodes with data under + `data/evm_ss/` continue using that legacy path. The `evm-db-directory` + config, when unset, defaults to `data/state_store/evm/{backend}`. +- SC data (`memiavl` + FlatKV) is untouched by this migration; on new nodes it + lives under `data/state_commit/memiavl` and `data/state_commit/flatkv`, with + legacy `data/committer.db` and `data/flatkv` retained when present. ### Does Giga SS Store change the app hash or consensus? diff --git a/node/index.mdx b/node/index.mdx index 3b7977a..e9a98ba 100644 --- a/node/index.mdx +++ b/node/index.mdx @@ -100,6 +100,32 @@ seid version See Network Versions table above for current recommended version. + + + + **Mock validation builds are for testing/mock environments only — never run them in production.** These builds use a build-tag-selected `ConsensusPolicy` whose `HandleError` mechanism swallows specific halting validation failures (incrementing the `sei_unsafe_validation_skipped_total{validation_error=...}` counter instead of halting) rather than enforcing them. Default (production) builds always enforce every consensus check, and this counter is always zero. + + **`mock_block_validation`** — swallows only `AppHash` and `DataHash` validation failures during block execution and validation; every other consensus check still halts. + + ```bash + # Build a seid binary that swallows AppHash/DataHash validation failures (testing only) + GO_BUILD_TAGS=mock_block_validation make install + ``` + + A dedicated Docker image is published for this build in ECR, tagged `sei/sei-chain:mock_block_validation-` (built with `GO_BUILD_TAGS=mock_block_validation`). + + **`mock_chain_validation`** — intended for forked-state replays; swallows every swallow-eligible halting validation failure except `ErrLastCommitVerify` (which still halts to avoid a downstream panic). It is built together with `mock_balances`. + + ```bash + # Build a seid binary for forked-state replays (testing only) + GO_BUILD_TAGS="mock_balances mock_chain_validation" make install + ``` + + A dedicated Docker image is published for this build in ECR, tagged `sei/sei-chain:mock_chain_validation-` (and `sei/sei-chain:mock_chain_validation-nightly-*` for nightly builds). + + Because these binaries skip integrity checks that guard against divergent or malformed state, they must not be used on mainnet, testnets, or any network carrying real value. + + Official Docker images are available at GitHub Container Registry. @@ -234,10 +260,14 @@ max-tx-bytes = 2048576 # Maximum size of a batch of transactions to send to a peer max-batch-bytes = 0 -# Maximum length of time a transaction can remain in the mempool +# Maximum length of time a transaction can remain in the mempool. +# Set to "0s" to explicitly disable time-based TTL purging. +# The default is "5s". ttl-duration = "3s" -# Maximum number of blocks a transaction can remain in the mempool +# Maximum number of blocks a transaction can remain in the mempool. +# Set to 0 to explicitly disable block-based TTL purging. +# The default is 10 blocks. ttl-num-blocks = 5 tx-notify-threshold = 0 @@ -250,8 +280,10 @@ pending-size = 5000 max-pending-txs-bytes = 1073741824 +# Deprecated: pending TTL is not used and this field has no effect. pending-ttl-duration = "3s" +# Deprecated: pending TTL is not used and this field has no effect. pending-ttl-num-blocks = 5 ``` diff --git a/node/node-operators.mdx b/node/node-operators.mdx index a198665..ea8347c 100644 --- a/node/node-operators.mdx +++ b/node/node-operators.mdx @@ -57,8 +57,36 @@ laddr = "tcp://0.0.0.0:26657" max-open-connections = 900 # Transaction confirmation timeout for /broadcast_tx_commit timeout-broadcast-tx-commit = "10s" + + +# Timeout to read HTTP request headers; mitigates slowloris attacks. Default 10s. +# Set to "0s" to disable (not recommended). +timeout-read-header = "10s" + +# HTTP write timeout; acts as a hard backstop for all handlers. Default 30s. +# Must be greater than timeout-broadcast-tx-commit when non-zero. +# Set to "0s" to disable the write timeout (not recommended). +timeout-write = "30s" ``` + + + The `[rpc]` section also supports `max-tx-search-results`, which caps the + number of results returned by the `tx_search` and `block_search` RPC + endpoints. The cap is applied *after* results are sorted by `order_by`, and + the reported `TotalCount` reflects the post-cap count. The default is + `10000`. Set it to `0` to disable the cap entirely (not recommended on + public nodes, since unbounded searches can be used to exhaust node + resources). The value must not be negative. + + ```toml + [rpc] + # Maximum number of results returned by tx_search and block_search. + # Set to 0 to disable the cap (not recommended on public nodes). + max-tx-search-results = 10000 + ``` + + #### Application Settings (app.toml) ```toml @@ -86,6 +114,26 @@ max-open-connections = 1000 # SeiDB state-commit (memiavl + FlatKV). Recommended on every node. sc-enable = true + + + + **IAVL has been fully removed.** SeiDB state-commit is now mandatory. Running a + node with SeiDB state-commit disabled no longer falls back to the legacy IAVL + store. As of recent `seid` releases, a node started with `sc-enable = false` + panics on startup with: + + ``` + SeiDB state-commit (SC) must be enabled; IAVL backend has been fully deprecated + ``` + + Make sure `sc-enable = true` under `[state-commit]` before upgrading so your + node runs on SeiDB. All IAVL-related configuration fields and CLI commands + (for example `iavl-cache-size`, the `[iavl]` section, and the `compact`, + `prune`, `latest_version`, and `debug dump-iavl` commands) have been removed + and no longer take effect. Nodes with SeiDB SC enabled log `SeiDB SC is + enabled now` on startup. + + [state-store] # Historical SS layer for queries. Required for any node serving RPC. ss-enable = true @@ -93,10 +141,244 @@ ss-enable = true ss-keep-recent = 100000 [receipt-store] -# Storage backend for EVM transaction receipts (pebbledb or parquet). +# Storage backend for EVM transaction receipts. pebbledb is the only +# supported backend (aka pebble); setting rs-backend to any other value +# returns an error. When Giga Storage is enabled (GIGA_STORAGE=true) the +# receipt backend still defaults to pebble. You can force the backend +# explicitly with rs-backend (or the RECEIPT_BACKEND env var in the +# containerized node scripts), which always takes precedence. rs-backend = "pebbledb" ``` + +#### Autobahn / GigaRouter (config.toml) + +Autobahn (the GigaRouter consensus path) is enabled by pointing the +`autobahn-config-file` field at a JSON file describing the validator +committee and the consensus/producer parameters. Leave it empty (the +default) to disable Autobahn entirely. + +```toml +# Path to a JSON file containing the Autobahn (GigaRouter) configuration. +# Leave empty to disable Autobahn. +autobahn-config-file = "" +``` + +When `autobahn-config-file` is set the node loads its committee membership +and block-production parameters from the referenced JSON file. The node must +be a committee member: its own validator key and node key must appear in the +`validators` list, otherwise startup fails. + + + Autobahn does **not** support remote validator signers. If + `autobahn-config-file` is set together with `priv-validator.laddr` (a remote + signer listen address), the node fails to start. A local validator key is + required — non-validator (observer) nodes are not yet supported under + Autobahn. + + +The referenced JSON file supports the following fields: + +```json +{ + "validators": [ + { + "validator_key": "ed25519:public:...", + "node_key": "node:ed25519:public:...", + "address": "host:port" + } + ], + "max_txs_per_block": 5000, + "max_txs_per_second": 1000, + "block_interval": "400ms", + "allow_empty_blocks": true, + "view_timeout": "1500ms", + "persistent_state_dir": "/path/to/state", + "dial_interval": "10s" +} +``` + +| Field | Description | +| --- | --- | +| `validators` | Committee membership. Each entry is `{ validator_key, node_key, address }`. Validator keys and node keys must be unique across the list, and the node's own keys must be present. The committee is capped at **100 validators** (`MaxValidators = 100`): a `validators` list with more than 100 entries is rejected at startup and Autobahn refuses to build the committee. | +| `max_txs_per_block` | Maximum transactions per produced block. Must be `> 0`. Note that Autobahn also enforces hard on-chain per-block caps of **2000 transactions** and roughly **2 MB** of total transaction bytes; `max_txs_per_block` is effectively clamped to the 2000-tx limit, so setting it higher has no effect. | +| `max_txs_per_second` | Optional cap on transactions per second (omit to leave unset). | + +The `mempool_size` field has been removed from the Autobahn config and is no longer required by validation. Autobahn no longer reaps transactions from the shared CometBFT `TxMempool`; instead the block producer maintains its own mempool whose capacity is derived from the per-lane block budget (`BlocksPerLane`). This producer-backed mempool inserts EVM transactions in strict per-account nonce order and enforces the per-block gas-wanted and gas-estimated limits, so transactions with unexpected nonces or exceeding the size/gas limits are rejected at insertion time. The maximum gas per block is now derived from the genesis `ConsensusParams.Block` (`MaxGas` and `MaxGasWanted`) rather than a `max_gas_per_block` config field. + +#### Mempool transaction TTL (config.toml) + +Under the `[mempool]` section of `config.toml`, the transaction TTL settings +control how long a transaction is allowed to remain in the mempool before it is +purged. Both settings are now optional, and a value of `0` explicitly disables +the corresponding TTL. + +```toml +[mempool] +# Maximum amount of time a transaction may remain in the mempool before it is +# purged. Set to "0s" to disable time-based TTL purging. Default: 5s. +ttl-duration = "5s" +# Maximum number of blocks a transaction may remain in the mempool before it is +# purged. Set to 0 to disable block-based TTL purging. Default: 10. +ttl-num-blocks = 10 +``` + +Previously a non-zero value enabled the corresponding TTL; now `ttl-duration = 0` +and `ttl-num-blocks = 0` explicitly disable time-based and block-based TTL +purging respectively. The defaults remain `5s` and `10` blocks. + + + The separate **pending** TTL fields `pending-ttl-duration` and + `pending-ttl-num-blocks` under `[mempool]` are now **deprecated and have no + effect**. Pending TTL is no longer used by the mempool, so setting these + fields does nothing. They are retained only for backward compatibility and + may be removed in a future release; do not rely on them to expire pending + transactions. + + + + Expiration behavior differs between READY and PENDING transactions. + `remove-expired-txs-from-queue` now governs whether expired **READY** + transactions are pruned from the mempool queue. Expired **PENDING** + transactions are always removed regardless of this setting. + + + + When Autobahn is enabled, the node builds blocks directly from the shared + `TxMempool` and disables the mempool gossip reactor, the consensus reactor, + state sync, and block sync — those subsystems are not compatible with the + GigaRouter consensus path. Each produced block is bounded by the on-chain + caps of 2000 transactions and ~2 MB of total transaction bytes in addition to + the `max_gas_per_block` and `max_txs_per_block` limits from the config file. + + + + +#### Monitoring under Autobahn + +Enabling Autobahn changes what several CometBFT RPC endpoints report, because +the CometBFT block store and consensus reactor are not fed on the GigaRouter +path. Operators relying on status and monitoring endpoints should be aware of +the following. + +**`/status` additions** + +The `SyncInfo` object returned by `/status` now includes a +`last_committed_block_height` field (a JSON string-encoded integer) reporting +the last block finalized by consensus: + +- Under CometBFT, commit and app-apply happen in a single step, so + `last_committed_block_height` always equals `latest_block_height`. +- Under Autobahn, the value is derived from the latest `CommitQC`. Consensus + finalizes a block before the app executes it, so + `last_committed_block_height` can briefly lead `latest_block_height` + (the invariant is `last_committed_block_height >= latest_block_height`). + +Under Autobahn, `/status` derives `latest_block_height` and `latest_app_hash` +from the app layer (via `ABCIInfo`) rather than from the CometBFT block store, +so these fields report live values instead of `0`. + +**Fields that remain unpopulated under Autobahn** + +Because the CometBFT block store and consensus reactor are not fed, the +following `SyncInfo` fields are not currently populated under Autobahn: +`latest_block_hash`, `latest_block_time`, all `earliest_*` fields, and +`max_peer_block_height`. `catching_up` is hardcoded to `true`. + +**Block-data RPC endpoints under Autobahn** + +The CometBFT block store and state store are not populated under Autobahn. +Instead, `/block`, `/block_by_hash`, `/block_results`, and `/validators` are +now served by routing through the GigaRouter's in-memory finalized state, so +these endpoints return real data on Autobahn nodes (previously they returned +nil/empty because the block store height stayed at `0`). Downstream consumers +such as the EVM RPC endpoints (for example `eth_getBlockByNumber`) that walk +through these endpoints keep working as a result. + +Be aware of the following Autobahn-specific limitations: + +- **`/block`** returns the finalized block translated from Autobahn's + in-memory state at the requested height. Only a subset of the CometBFT + block fields are populated (block ID hash, chain ID, height, time, and the + transaction list); fields such as `AppHash`, `ProposerAddress`, and + `LastCommit` stay at their zero values. +- **`/block_by_hash`** resolves a block by its header hash via an in-memory + hash index that tracks the same retain window as `/block`. Unknown hashes, + or hashes below the pruning watermark, return an empty block + (`{"block": null}`) with no error, matching CometBFT semantics. +- **`/block_results`** returns a valid-but-empty result at the requested + height: `TxsResults` is intentionally empty because `FinalizeBlock` + responses are not persisted under Autobahn, but + `ConsensusParamUpdates.Block.MaxGas` is populated from the producer's + `max_gas_per_block` config so consumers relying on the gas limit keep + working. Per-transaction execution details are not available. +- **`/validators`** returns the genesis committee for any retained height, + with `block_height` matching the requested height. (This fixes the prior + behavior where the height could get stuck at `1` due to the unpopulated + state store.) + + + Requesting a height that has already been pruned out of Autobahn's retain + window returns an `ErrHeightNotAvailable`-class error, mirroring the + CometBFT path so external tooling sees consistent error shapes. + +| `block_interval` | Target interval between blocks. Must be `> 0`. | +| `allow_empty_blocks` | Whether to produce blocks when there are no transactions. | +| `view_timeout` | Consensus view timeout. Must be `> 0`. | +| `persistent_state_dir` | Directory used to persist the Autobahn consensus and data-layer write-ahead logs (WALs) across restarts. Both layers share this on-disk root and write to distinct subdirectories under it. Relative paths are resolved against the node's `--home` directory at load time; absolute paths are used as-is. When generated via `gen-autobahn-config` this defaults to `data/autobahn`, so persistence is on by default. Set it to an empty value (or omit it entirely) to disable persistence and run both the consensus and data layers in-memory only. | +| `dial_interval` | Interval between dial attempts to committee peers. Must be `> 0`. | + + +#### Generating the Autobahn config + +Rather than hand-writing the `validators` list, you can generate the JSON +config file from a set of node directories with the `seid tendermint +gen-autobahn-config` command: + +```bash +seid tendermint gen-autobahn-config [node-dirs...] --output +``` + +Each `node-dir` argument must contain four files describing that committee +member: + +- `validator_pubkey.txt` — the validator public key in `validator:` + format +- `node_pubkey.txt` — the p2p node public key in `node:ed25519:public:` + format +- `autobahn_address.txt` — the node's network address in `host:port` format +- `evmrpc_url.txt` — the node's EVM RPC HTTP URL (for example + `http://:8545`, using the EVM RPC HTTP port from the `[evm]` + section of `app.toml`). This is used to proxy EVM RPC requests to the + validator that owns the sender's EVM address shard. + +The command reads these files from each directory, assembles the `validators` +list, and writes a complete Autobahn JSON config (with default consensus and +producer parameters) to the path given by `--output` (short flag `-o`). The +`--output` flag is required. + +The `validator_pubkey.txt` and `node_pubkey.txt` files are produced +automatically: whenever `seid` saves the validator private key +(`priv_validator_key.json`) it also writes `validator_pubkey.txt`, and whenever +it saves the node key (`node_key.json`) it also writes `node_pubkey.txt`, both +in the same directory as the key file. You need to supply +`autobahn_address.txt` and `evmrpc_url.txt` yourself. Point +`autobahn-config-file` at the generated file to enable Autobahn. + +#### EVM RPC request proxying + +Each validator is assigned a shard of the EVM address space. When a node +receives an `eth_sendRawTransaction` request, or an `eth_getTransactionCount` +request for the pending block, it forwards (proxies) the request to the EVM +RPC endpoint of the validator that owns the sender's shard, using the `evmrpc` +URL from that validator's config entry. Requests whose sender maps to the +local validator are handled locally. Proxied requests are counted by the +`evmrpc_redirected_requests_total` telemetry metric (labeled with the +endpoint and connection type). This is why every committee member must publish +an `evmrpc` URL via `evmrpc_url.txt`. + + + ### Default Configurations The full unmodified `app.toml`, `config.toml`, and `client.toml` produced by @@ -233,10 +515,15 @@ sc-snapshot-write-rate-mbps = 100 # WriteMode defines the write routing mode for EVM data in the SC layer. # Valid values: memiavl_only, migrate_evm, evm_migrated, migrate_all_but_bank, # all_migrated_but_bank, migrate_bank, flatkv_only, test_only_dual_write +# Defaults to memiavl_only. An invalid value fails at config parse time with a +# clear error. The memiavl_only, evm_migrated, all_migrated_but_bank, and +# flatkv_only values are steady states; the migrate_* values drive an in-flight +# migration from memiavl to flatkv. test_only_dual_write is for test clusters +# only and must never be deployed to testnet/mainnet. sc-write-mode = "memiavl_only" -# KeysToMigratePerBlock controls how many EVM keys the in-flight migration -# (sc-write-mode = migrate_evm / migrate_bank / migrate_all_but_bank) drains +# KeysToMigratePerBlock controls how many keys the in-flight migration +# (sc-write-mode = migrate_evm / migrate_all_but_bank / migrate_bank) drains # from memiavl into flatkv per block. Default 1024 is appropriate for # production drains; lower it (e.g. 256) to spread the migration across more # blocks for test runs that need to observe the resume / hybrid-read path. @@ -265,6 +552,15 @@ snapshot-interval = 10000 # 0 = keep only the current snapshot. Default: 2. snapshot-keep-recent = 2 + + +# EnablePebbleMetrics is a single FlatKV-level knob that controls whether +# PebbleDB internal metrics are emitted for all FlatKV data DBs (account, +# code, storage, legacy, and metadata). During initialization this value +# overrides any per-DB EnableMetrics settings, so all data DBs share the same +# behavior. Default: false. +enable-pebble-metrics = false + ############################################################################### ### State Store Configuration ### ############################################################################### @@ -304,7 +600,7 @@ ss-prune-interval = 600 ss-import-num-workers = 1 # EVMDBDirectory defines the directory for the optional EVM state-store DB(s). -# If unset, defaults to /data/evm_ss when EVM SS is enabled. +# If unset, defaults to /data/state_store/evm/{backend} when EVM SS is enabled. evm-ss-db-directory = "" # EVMSplit controls whether EVM data is routed to a dedicated SS backend. @@ -318,6 +614,16 @@ evm-ss-split = false # When true, data is routed to separate DBs while preserving the same evm key prefix format. evm-ss-separate-dbs = false +# UseDefaultComparer controls how the state-store iterator advances to the next +# logical key when the PebbleDB backend uses the descending-version MVCC +# encoding (the default for freshly created DBs). When false, the iterator uses +# the MVCC comparer's ImmediateSuccessor to seek directly to the next logical +# key. When true, the iterator falls back to a scan-based advance that steps +# through entries until it reaches the next logical key. Only affects the +# descending fast path; legacy (ascending-encoded) DBs are unaffected. +# defaults to false +use-default-comparer = false + ############################################################################### ### Receipt Store Configuration ### ############################################################################### @@ -338,10 +644,32 @@ db-directory = "" async-write-buffer = 100 # PruneIntervalSeconds defines the interval in seconds to trigger pruning. -# Receipt retention is controlled by the global min-retain-blocks flag. +# Receipt retention is controlled by the global min-retain-blocks flag: the +# receipt store keeps the last min-retain-blocks blocks (0 = keep everything, +# no pruning). The dedicated receipt-store.keep-recent field has been removed +# and is ignored if set. # defaults to 600 seconds prune-interval-seconds = 600 + + +# TxIndexBackend selects the tx-hash index implementation for parquet receipts. +# Set to "pebbledb" to maintain a Pebble-backed tx_hash -> block_number index +# alongside parquet files so receipt-by-hash lookups can target a single file +# instead of scanning all files. Set to "" to disable the index. +# +# WARNING: with the index disabled, a receipt-by-tx-hash lookup (e.g. +# eth_getTransactionReceipt) that misses the in-memory cache fails fast and +# returns a not-found result instead of falling back to a full parquet scan +# (a full scan reads every file on disk and is prohibitively expensive at +# production scale). Operators that must serve historical receipts by tx hash +# should keep the index enabled; otherwise receipts that are no longer cached +# will appear as not-found. Block-range queries such as log filtering are +# unaffected since they do not use the tx hash index. +# Ignored unless rs-backend = "parquet". +# defaults to "pebbledb" +tx-index-backend = "pebbledb" + ############################################################################### ### EVM Configuration ### ############################################################################### @@ -414,6 +742,14 @@ deny_list = [] # gate errors use standard JSON-RPC error encoding (see evmrpc/AGENTS.md). Successful allowlisted # responses are unchanged; nodes may set HTTP header Sei-Legacy-RPC-Deprecation (see AGENTS.md). # +# Gating applies to both single-object requests and JSON-RPC batches. Single-object allowed calls +# pass through unchanged. For batches, only the allowlisted (and otherwise valid) elements are +# forwarded to the inner handler as a filtered subset, and the inner responses are merged back into +# the original batch by JSON-RPC id; blocked elements return the standard gate error and any +# non-object batch element returns a JSON-RPC -32600 Invalid Request, so the gate cannot be +# bypassed. The legacy HTTP request body is capped at 5 MiB, matching go-ethereum's default +# (rpc.defaultBodyLimit). +# # Only methods listed in enabled_legacy_sei_apis are allowed. Init defaults enable the three # address/Cosmos helpers; uncomment optional lines below to enable more legacy methods (include # sei2_* block methods at the end of the list if you need them). @@ -471,7 +807,15 @@ max_subscriptions_new_head = 10000 # Set to 0 for unlimited. max_concurrent_trace_calls = 10 -# Max number of blocks allowed to look back for tracing +# Max number of blocks allowed to look back for tracing. +# This lookback guard applies to all debug_trace* methods +# (debug_traceTransaction, debug_traceTransactionProfile, +# debug_traceBlockByNumber, debug_traceBlockByHash, debug_traceCall, and +# debug_traceStateAccess). Any request targeting a block older than +# (latest - max_trace_lookback_blocks) is rejected with +# "block number N is beyond max lookback of M", and the attempt is counted +# by the evmrpc_historical_debug_trace_attempts_total telemetry metric +# (labeled by endpoint and connection type). # Set to -1 for unlimited lookback, which is useful for archive nodes. max_trace_lookback_blocks = 10000 @@ -479,6 +823,9 @@ max_trace_lookback_blocks = 10000 trace_timeout = "30s" # Enable the parallelized default debug_traceBlock* path. +# When true, debug_traceBlock* requests that do not specify an explicit +# tracer use the parallelized block trace path; requests with an explicit +# tracer continue to use the legacy path. Default false. enable_parallelized_block_trace = false # WorkerPoolSize defines the number of workers in the worker pool. @@ -545,6 +892,33 @@ enabled = true # Default: true occ_enabled = true + + + + **evmone library loading (Giga executor).** The Giga executor loads its + platform-specific `evmone` shared library from a fixed, trusted absolute path + and verifies the file's SHA-256 digest against the digest pinned for the + running platform before handing it to the dynamic linker. Passing an absolute + path avoids the dynamic linker's search path (`LD_LIBRARY_PATH`, + `ld.so.cache`, default directories), so the library cannot be substituted by + planting a file earlier in the loader's search order. + + The library directory is resolved in the following order, and the first + directory that actually contains the library wins: + + 1. `$SEI_EVMONE_LIB_DIR` — optional operator override. Set it to an absolute + path to a root-owned, non-writable directory that holds the trusted, + integrity-verified `evmone` library. + 2. `/usr/lib` — the canonical install location used by release images. + 3. The source-tree directory — used for local development and tests. + + Release Docker images install the `evmone` library to `/usr/lib` (alongside + the other native libraries) and add the `libstdc++6` runtime dependency, so + no additional configuration is required when running the official images. If + the resolved library is missing or its digest does not match the expected + value, node startup fails with an explicit error. + + ############################################################################### ### Admin Configuration (Auto-managed) ### ############################################################################### @@ -963,6 +1337,17 @@ upnp = false # Maximum number of connections (inbound and outbound). max-connections = 100 +# Maximum number of outbound connections to regular (non-persistent) peers. +# Inbound and outbound connections are managed as two fully separate pools, so +# a single peer can hold both an inbound and an outbound connection to this +# node at the same time. +# If unset, MaxOutbound defaults to min(20, (max-connections+1)/2), i.e. 20 +# unless max-connections < 40, in which case it defaults to half of +# max-connections. The inbound pool is then sized as +# MaxInbound = max-connections - MaxOutbound. +# Persistent and unconditional connections are not counted towards either limit. +# max-outbound-connections = 20 + # Rate limits the number of incoming connection attempts per IP address. max-incoming-connection-attempts = 100 diff --git a/node/rocksdb-backend.mdx b/node/rocksdb-backend.mdx index 587b477..05098a4 100644 --- a/node/rocksdb-backend.mdx +++ b/node/rocksdb-backend.mdx @@ -26,6 +26,14 @@ By contrast, **RocksDB** supports native user-defined timestamps and optimized c In Sei’s benchmarks, RocksDB achieved up to **10–30× faster traceBlock iteration times** compared to PebbleDB, with even greater benefits observed on archive nodes. + + +### PebbleDB descending-version encoding + +Recent PebbleDB builds partly narrow this gap for latest-version reads. Because PebbleDB has no native MVCC, Sei encodes the version into each key. Freshly created PebbleDB state stores now use a **descending-version MVCC encoding**, which sorts newer versions before older ones for the same logical key. This lets latest-version reads land directly on the newest visible version instead of scanning through older versions, improving read performance on the fast path. Fresh stores are marked on disk with a sentinel key (`s/_mvcc_descending`) so the mode is detected automatically on open. + +Legacy PebbleDB stores written by earlier builds use the older **ascending-version encoding**. These are detected automatically on open and read using the legacy ascending path—no error is raised—but they stay unmarked and cannot benefit from the descending fast path unless the store is recreated or migrated. This mirrors the migration constraint on RocksDB: archive nodes that cannot recreate their state store will continue running on the slower legacy path. Note that even with descending encoding, PebbleDB still lacks native MVCC and column-family support, so RocksDB remains the recommended backend for iteration-heavy archive and long-history RPC workloads. + ## Example: TraceBlock Latency Comparison The following chart compares iteration (trace time) performance between **PebbleDB** and **RocksDB** over a 3 million block history: diff --git a/node/seictl.mdx b/node/seictl.mdx index 3b57536..f90adc4 100644 --- a/node/seictl.mdx +++ b/node/seictl.mdx @@ -264,7 +264,7 @@ Client-level configuration including: Node-level configuration including: -- Proxy app and database settings +- Database settings - Logging configuration - RPC and P2P settings - Mempool and consensus parameters diff --git a/node/statesync.mdx b/node/statesync.mdx index 2edde39..dcbb97d 100644 --- a/node/statesync.mdx +++ b/node/statesync.mdx @@ -96,6 +96,26 @@ cp -r $HOME/.sei/wasm $HOME/.sei_backup/ 2>/dev/null || true # ignore error if mkdir -p /home/ubuntu/statesync-temp sed -i 's|temp-dir = ""|temp-dir = "/home/ubuntu/statesync-temp"|' ~/.sei/config/config.toml + + +Recent versions of the chain no longer emit the advanced state-sync tuning knobs (`temp-dir`, `backfill-blocks`, `backfill-duration`, `discovery-time`, `chunk-request-timeout`, `fetchers`, `verify-light-block-timeout`, `blacklist-ttl`) in the generated `config.toml`. These fields are still parsed and honored if you add them manually, otherwise sensible defaults are used. + +Because `temp-dir = ""` may no longer be present in a freshly generated `config.toml`, the `sed` command above can silently do nothing. If you need a custom temp directory for snapshot chunk staging, add the field explicitly under the `[statesync]` section instead: + +```bash +mkdir -p /home/ubuntu/statesync-temp +# Append temp-dir under the [statesync] section only if it is not already present +if ! grep -qE '^\s*temp-dir\s*=' $HOME/.sei/config/config.toml; then + sed -i '/^\[statesync\]/a temp-dir = "/home/ubuntu/statesync-temp"' $HOME/.sei/config/config.toml +else + sed -i 's|^temp-dir = .*|temp-dir = "/home/ubuntu/statesync-temp"|' $HOME/.sei/config/config.toml +fi +``` + +Setting a custom `temp-dir` is optional; if omitted, state sync stages chunks in the system default temporary directory. + + + # Fetch the latest block height from the State Sync RPC endpoint LATEST_HEIGHT=$(curl -s $STATE_SYNC_RPC/block | jq -r .block.header.height) # Calculate the trust height (rounded down to the nearest 100,000) diff --git a/node/technical-reference.mdx b/node/technical-reference.mdx index 186396a..0386b73 100644 --- a/node/technical-reference.mdx +++ b/node/technical-reference.mdx @@ -37,6 +37,36 @@ seid tendermint show-validator seid query node info ``` + +### Autobahn Config Generation + +Generate an Autobahn (GigaRouter) JSON config from a set of node directories. Each node directory must contain `validator_pubkey.txt`, `node_pubkey.txt`, `autobahn_address.txt`, and `evmrpc_url.txt`. These pubkey files are written automatically alongside the key files when the validator key and node key are saved (in `validator:` and `node:ed25519:public:` formats, respectively). The `evmrpc_url.txt` file must contain the node's EVM RPC HTTP URL (e.g. `http://:8545`), which is used to proxy EVM RPC requests to the validator that owns the sender's EVM address shard. + +```bash +# Generate an autobahn config from one or more node directories +seid tendermint gen-autobahn-config [node-dirs...] --output + +# Using the short flag for the output path +seid tendermint gen-autobahn-config node_0 node_1 node_2 -o autobahn.json +``` + +The `--output` / `-o` flag is required and specifies the destination file path for the generated config. The command reads the following files from each supplied node directory: + +| File | Description | +| --- | --- | +| `validator_pubkey.txt` | Autobahn validator public key in `validator:` format. | +| `node_pubkey.txt` | p2p node public key in `node:ed25519:public:` format. | +| `autobahn_address.txt` | Network address (`host:port`) for the node. | +| `evmrpc_url.txt` | The node's EVM RPC HTTP URL (e.g. `http://:8545`). | + +The command also accepts an optional `--persistent-state-dir` flag: + +| Flag | Description | +| --- | --- | +| `--persistent-state-dir` | Directory to persist the Autobahn consensus and data-layer WALs across restarts. Defaults to `data/autobahn`, so persistence is enabled by default without operator action. Relative paths are resolved against the node's `--home` directory at load time; absolute paths are used unchanged. Pass `--persistent-state-dir=` (empty) to disable persistence and run both the consensus and data layers in-memory only. | + +The resulting JSON file can then be referenced from `config.toml` via the `autobahn-config-file` field to enable Autobahn wiring at node startup. + ### Key Management Proper key management is crucial for security. These commands help you manage @@ -112,14 +142,57 @@ size = 5000 max-txs-bytes = 1073741824 cache-size = 10000 + + +#### Block-Failed Transaction Re-Entry Protection + +The mempool tracks transactions that fail during block execution to prevent them from re-entering the mempool indefinitely. Some transactions consistently fail before fee charging in `DeliverTx`, which historically let them be re-submitted without limit and waste block and proxy-app resources. + +The mempool now applies the following policy on top of the existing `keep-invalid-txs-in-cache` setting: + +- When a committed transaction succeeds (`Code == OK`), it is kept in the cache and its entry in the block-failure tracker is cleared. +- When a committed transaction fails and `keep-invalid-txs-in-cache` is `false`: + - **First block failure:** the transaction is removed from the cache, allowing it to be re-submitted and retried once. + - **Second (and subsequent) block failure:** the transaction is left in the cache. Any attempt to re-submit it is rejected by `CheckTx` with `ErrTxInCache`, preventing infinite re-entry. +- A successful block inclusion resets the failure tracker, so a transaction that later succeeds regains its first-failure grace period. + +The block-failure tracker uses an LRU cache sized to match the mempool `cache-size`. When `cache-size` is `0`, tracking is disabled. This behavior is automatic and requires no additional configuration. + + +# State commit configuration (SeiDB). State-commit is mandatory: +# the node will panic on startup if sc-enable is false. +[state-commit] +sc-enable = true + # State store configuration [state-store] ss-enable = true ss-backend = "pebbledb" ss-keep-recent = 100000 ss-prune-interval = 600 + +# Receipt store configuration +[state-store.receipt-store] +# Backend defines the receipt store backend. +# The only supported backend is pebbledb (aka pebble), which is also the +# default. The parquet/DuckDB backend has been removed; setting rs-backend to +# "parquet" now returns an error. +rs-backend = "pebbledb" + +# AsyncWriteBuffer defines the async queue length for commits applied to the +# receipt store. Applies only when rs-backend = "pebbledb". Set <= 0 for +# synchronous writes. Defaults to 100. +async-write-buffer = 100 + +# Interval in seconds to trigger pruning. Receipt retention is controlled by +# the global min-retain-blocks flag. Defaults to 600 seconds. +prune-interval-seconds = 600 ``` + + The legacy IAVL backend has been fully removed. SeiDB state-commit is now required, so `sc-enable` must be `true` or the node will panic on startup. The former `iavl-cache-size` field, the entire `[iavl]` config section, and the IAVL-related base fields (`iavl-disable-fastnode`, `no-versioning`, `separate-orphan-storage`, `separate-orphan-versions-to-keep`, `num-orphan-per-file`, `orphan-dir`) are no longer valid and have no effect. The removed `compact`, `prune`, `latest_version`, and `debug dump-iavl` CLI commands and their IAVL/orphan-related start flags are also gone. Ensure `sc-enable = true` before upgrading. + + ### Config.toml Parameters @@ -135,8 +208,15 @@ external-address = "" seeds = "" persistent_peers = "" upnp = false -max_num_inbound_peers = 40 -max_num_outbound_peers = 10 +# Total connection budget. Inbound and outbound connections are now tracked in +# fully separate pools, so a single peer may hold both an inbound and an +# outbound connection at the same time. +max-connections = 100 +# Maximum number of outbound connections to regular (non-persistent) peers. +# Defaults to min(20, (max-connections+1)/2). Inbound capacity is derived as +# max-connections - max-outbound-connections. Persistent and unconditional +# peers are not counted against either limit. +max-outbound-connections = 20 allowed_pools = "" max_packet_msg_payload_size = 10240 handshake_timeout = "20s" @@ -149,7 +229,28 @@ cors_allowed_origins = [] cors_allowed_methods = ["HEAD", "GET", "POST"] cors_allowed_headers = ["Origin", "Accept", "Content-Type", "X-Requested-With", "X-Server-Time"] max_open_connections = 900 +# Maximum time to wait for a BroadcastTxCommit request to complete. When set to +# a value greater than 0, the BroadcastTxCommit RPC now enforces this as a +# context deadline on the request: if CheckTx and DeliverTx do not complete +# within this duration, the request context is cancelled and the call returns a +# timeout error. Set to "0s" to disable the timeout. timeout_broadcast_tx_commit = "10s" +# Timeout to read HTTP request headers; mitigates slowloris attacks by limiting +# the time allowed to read request headers. Set to "0s" to disable (not +# recommended). Defaults to 10s. +timeout-read-header = "10s" +# HTTP write timeout; acts as a hard backstop for all handlers. Must be greater +# than timeout_broadcast_tx_commit when non-zero. Set to "0s" to disable (not +# recommended). Defaults to 30s. +timeout-write = "30s" + + +# Maximum number of results returned by the tx_search and block_search RPC +# endpoints. The cap is applied after results are sorted, so the top matches +# (per order_by) are preserved, and TotalCount reflects the post-cap count. +# Must not be negative. Set to 0 to disable the cap (not recommended on public +# nodes). Defaults to 10000. +max-tx-search-results = 10000 # Consensus Configuration [consensus] @@ -164,10 +265,256 @@ timeout_commit = "1s" double_sign_check_height = 0 ``` + + + + The `[consensus]` section also accepts an `unsafe-overrides-enabled` field (defaults to `false`). This flag gates whether the `Unsafe*TimeoutOverride` fields (`UnsafeProposeTimeoutOverride`, `UnsafeProposeTimeoutDeltaOverride`, `UnsafeVoteTimeoutOverride`, `UnsafeVoteTimeoutDeltaOverride`, `UnsafeCommitTimeoutOverride`, and `UnsafeBypassCommitTimeoutOverride`) are actually applied to the resolved consensus timeouts. + + When `unsafe-overrides-enabled = false`, the unsafe timeout overrides are ignored and the node uses the on-chain timeout parameters (falling back to Tendermint defaults for any unset field). As a transitional exception, the overrides are still applied while the on-chain timeout params remain equal to the legacy "bad params" values — this preserves prior behavior until those params are corrected via a governance proposal. + + When `unsafe-overrides-enabled = true`, the unsafe overrides are applied on top of the resolved timeouts. In this mode `UnsafeBypassCommitTimeoutOverride` can also override `BypassCommitTimeout` to `false` (a nil override leaves the resolved value untouched). + + ```toml + [consensus] + # Gates whether the Unsafe*TimeoutOverride fields are applied. Defaults to false. + unsafe-overrides-enabled = false + ``` + + + + + The `[consensus]` section still parses a `stateless-leader-election` field, but it is now **deprecated and ignored**. Stateless leader election is always enabled: the consensus engine always selects each round's leader via a deterministic seeded stateless computation. The former stateful proposer-priority leader-election mode is no longer supported. + + Historically `stateless-leader-election` defaulted to `false`, selecting leaders via the stateful proposer-priority mode. As of v6.5 stateless leader election defaults to `true`, and the field is now retained only for config-parsing compatibility. Setting it to `false` no longer has any effect — it will not fall back to proposer-priority selection. You can safely remove any `stateless-leader-election` line from existing `config.toml` files. + + ```toml + [consensus] + # Deprecated and ignored. Stateless leader election is always enabled; + # setting this to false has no effect. + stateless-leader-election = true + ``` + + + + ABCI peer filtering has been removed. The deprecated `filter-peers` field in `config.toml` (under `[base]`) no longer has any effect and is no longer emitted in generated config templates. Tendermint no longer sends `/p2p/filter/addr/` or `/p2p/filter/id/` queries to the application, so peers can no longer be filtered by IP or node ID through the ABCI app. You can safely remove any `filter-peers` line from existing configs. + + + +### Autobahn (GigaRouter) Configuration + +The `autobahn-config-file` field in `config.toml` enables the Autobahn (GigaRouter) feature. It specifies the path to a JSON file containing the Autobahn configuration, which defines the validator committee and the consensus/producer parameters. Leave it empty to disable Autobahn. + +```toml +# Path to a JSON file containing the Autobahn (GigaRouter) configuration. +# Leave empty to disable Autobahn. +autobahn-config-file = "" +``` + + +#### Giga Executor EVM Library (`SEI_EVMONE_LIB_DIR`) + +The Giga executor loads the platform-specific `evmone` shared library from a fixed, trusted absolute path and verifies its SHA-256 digest against the value pinned for the current platform before handing it to the dynamic linker. Loading from an absolute path (rather than relying on the dynamic linker's search path) prevents the library from being substituted by planting a file earlier in the loader's search order. + +The optional `SEI_EVMONE_LIB_DIR` environment variable lets operators override the directory the library is loaded from. It must be an absolute path to a root-owned, non-writable directory containing the trusted, integrity-verified `evmone` library. + +The library directory is resolved in the following order — the first directory that actually contains the library wins: + +1. `$SEI_EVMONE_LIB_DIR` — operator override, when set. +2. `/usr/lib` — the canonical install location used by release Docker images. +3. The source-tree directory — used for local development and tests. + +```bash +# Point the Giga executor at a custom directory holding the trusted evmone library +export SEI_EVMONE_LIB_DIR=/opt/sei/lib +``` + + + Release Docker images install the `evmone` library to `/usr/lib` and add the `libstdc++6` runtime dependency, so no additional configuration is required for standard deployments. If the resolved library's SHA-256 digest does not match the pinned value, the node fails to start with a digest-mismatch error. + + + +When Autobahn is enabled, the referenced JSON file supports the following fields: + +```json +{ + "validators": [ + { + "validator_key": "", + "node_key": "node:ed25519:public:", + "address": "host:port" + } + ], + "max_txs_per_block": 5000, + "max_txs_per_second": 1000, + "block_interval": "400ms", + "allow_empty_blocks": false, + "view_timeout": "1500ms", + "persistent_state_dir": "/path/to/state", + "dial_interval": "10s" +} +``` + + + The `max_gas_per_block` field has been removed from the Autobahn config file. The producer's max-gas-per-block is now derived from the chain's genesis `consensus_params.block.max_gas` — the same gas-limit consensus rule the EVM runtime reads. This value must be greater than 0, otherwise the node fails to start with an `ErrGenesisMaxGasInvalid` error (`genesis consensus_params.block.max_gas must be > 0`). Node operators upgrading from an earlier release must remove any `max_gas_per_block` line from their Autobahn config file. + + + + The `mempool_size` field has been removed from the Autobahn config file. The mempool capacity is now derived automatically from the number of unexecuted blocks the local lane may hold (`BlocksPerLane`), so it no longer needs to be configured. Node operators upgrading from an earlier release must remove any `mempool_size` line from their Autobahn config file. + + +| Field | Description | +| --- | --- | +| `validators` | List of committee members, each with a `validator_key`, `node_key`, and `address`. Must not be empty. | +| `max_txs_per_block` | Maximum number of transactions per block. Must be greater than 0. | +| `max_txs_per_second` | Optional cap on transactions per second. | +| `block_interval` | Target interval between blocks (e.g. `400ms`). Must be greater than 0. | +| `allow_empty_blocks` | Whether to produce empty blocks. | +| `view_timeout` | Consensus view timeout. Must be greater than 0. | +| `persistent_state_dir` | Optional directory for persistent consensus state. | +| `dial_interval` | Interval between peer dial attempts. Must be greater than 0. | + + + When Autobahn is enabled, remote validator signers are not supported: the node fails to start if `priv-validator.laddr` is set. A local validator key is required — non-validator (observer) nodes are not yet supported. The node must be a committee member, so its own validator key and node key must appear in the `validators` list. + + +#### Block Production and Disabled Reactors + +When Autobahn is enabled, block production and network wiring change significantly: + +- **Shared mempool as the block source:** The Autobahn producer builds blocks by reaping transactions directly from the shared `TxMempool` (the same mempool used for `CheckTx`), rather than from a separate producer mempool channel. Transactions that are included in a block are popped from the shared mempool. +- **Disabled reactors:** Because Autobahn drives consensus and block dissemination itself, the node skips starting the mempool gossip reactor, the consensus reactor, statesync, and blocksync when Autobahn is enabled. As a result, transactions are not gossiped over the standard mempool p2p channel, and the node does not perform block sync or state sync. + + +#### RPC Behavior Under Autobahn + +Because the CometBFT block store and consensus reactor are not fed under Autobahn, some RPC responses behave differently: + +- **`/status` derives height and app hash from the app layer:** With Autobahn enabled the CometBFT block store height stays at 0, so the `/status` handler pulls `SyncInfo.latest_block_height` and `SyncInfo.latest_app_hash` from the application layer (via `ABCIInfo`) instead of the block store. This reports the last height the app committed in `FinalizeBlock` and its matching app hash. +- **New `last_committed_block_height` field:** The `SyncInfo` object in the `/status` response now includes a `last_committed_block_height` field (JSON string-encoded int64) reporting the last block finalized by consensus. Under CometBFT this always equals `latest_block_height` (commit and app-apply happen in one step). Under Autobahn it is derived from the latest CommitQC and may briefly lead `latest_block_height`, since consensus finalizes a block before the app executes it. +- **Unpopulated `/status` fields:** Several `SyncInfo` fields remain unpopulated under Autobahn: `latest_block_hash`, `latest_block_time`, the `earliest_*` fields, and `max_peer_block_height`. `catching_up` is currently hardcoded to `true` because the consensus reactor is nil. +- **Block-data endpoints now serve data via the GigaRouter:** Because the CometBFT block store and state store are not populated under Autobahn, `/block`, `/block_by_hash`, `/block_results`, and `/validators` are routed through the GigaRouter's in-memory state instead of the block store. This keeps these endpoints — and the EVM RPC endpoints that walk through them (e.g. `eth_getBlockByNumber`) — working, with the following caveats: + - **`/block`** returns the finalized global block at the requested height, translated into the CometBFT `ResultBlock` shape (populating `BlockID.Hash`, `ChainID`, `Height`, `Time`, and `Data.Txs`). Requests for pruned heights return an `ErrHeightNotAvailable`-class error. + - **`/block_by_hash`** resolves a block by its Autobahn header hash via an in-memory hash index. Matching CometBFT semantics, an unknown hash — or a hash below the pruning watermark — returns `{Block: nil}` with no error. + - **`/block_results`** returns a valid-but-empty result at the requested height. `ConsensusParamUpdates.Block.MaxGas` is populated from the producer's configured `max_gas_per_block`, but `TxsResults` is intentionally empty because `FinalizeBlock` responses are not persisted under Autobahn (no per-tx `ExecTxResult` details). + - **`/validators`** returns the genesis committee for any retained height, with `block_height` matching the requested height (fixing the prior behavior where the height could get stuck at 1). + + Note that these endpoints read from the GigaRouter's in-memory state, which is pruned according to the node's retain height; historical queries below the retain window are not available. Endpoints that still depend on the unpopulated CometBFT block/state store — such as `/commit` — remain unaffected by this routing and behave as before. + + +#### Hard Per-Block Limits + +In addition to the configurable `max_gas_per_block` and `max_txs_per_block` fields, Autobahn enforces on-chain hard limits when building a block's payload. These are protocol-level caps and always apply, regardless of the values set in the config file: + +| Limit | Value | Description | +| --- | --- | --- | +| Maximum transactions per block | 2000 | No more than 2000 transactions may be included in a single Autobahn block. | +| Maximum total transaction bytes | ~2 MB (2000 × 1024 bytes) | The combined size of all transactions in a block may not exceed roughly 2 MB. This budget can be distributed arbitrarily across transactions (e.g. one large tx or many small ones) up to the transaction-count limit. | + + + The effective per-block transaction count is the smaller of the configured `max_txs_per_block` and the hard limit of 2000. If you configure a larger value, the 2000-tx cap still applies. The block proto size upper bound used for p2p message sizing is derived from these limits. + + + +#### Committee Size Ceiling + +Autobahn enforces a hard upper bound of **100 validators** per committee. Committee construction (`NewCommittee`) rejects any validator set larger than this ceiling, returning an error rather than starting consensus. This `MaxValidators = 100` limit is a protocol-level invariant and applies regardless of how many entries appear in the `validators` list of the Autobahn config file. + + + A validator set exceeding 100 members will cause committee creation to fail. Ensure the `validators` list in your Autobahn config file contains no more than 100 members. + + +#### Wire-Format Message Limits + +Autobahn also enforces structural size and count limits on its protobuf messages during deserialization. These bounds are checked while scanning the raw wire bytes — *before* a message is fully decoded — so that oversized or malformed payloads are rejected without allocating unbounded memory. A message that violates any of these limits is dropped and the sending peer is disconnected. + +The enforced field limits include: + +| Field | Limit | Description | +| --- | --- | --- | +| ed25519 public key | 32 bytes | `PublicKey.ed25519` is bounded to a fixed 32-byte length. | +| Signature bytes | 64 bytes | `Signature.sig` is bounded to a fixed 64-byte length. | +| Block / payload / parent / app hashes | 32 bytes | Hash fields (e.g. `parent_hash`, `payload_hash`, `last_hash`, `app_hash`) are bounded to 32 bytes each. | +| Payload transaction count | 2000 | `Payload.txs` may contain at most 2000 transactions. | +| Payload total transaction size | ~2 MB (2,048,000 bytes) | The combined byte size of all transactions in a payload may not exceed 2,048,000 bytes. | +| QC signature lists | 100 | Signature lists on quorum certificates (`PrepareQC`, `CommitQC`, `AppQC`, `LaneQC`) and vote lists on `TimeoutQC` and `FullProposal.lane_qcs` are each capped at 100 entries, matching the 100-validator committee ceiling. | + + + These wire-format bounds are applied to every Autobahn consensus and data-layer message on receipt. They are protocol-level invariants and are not configurable. Because they are enforced during message scanning rather than after decoding, they protect nodes from resource-exhaustion attacks that rely on the decoded representation being far larger than the encoded bytes. + + + ## Network Parameters + + +### Query Pagination Limits + +Queries that support pagination now enforce hard caps to protect nodes from unbounded store walks. Requests that exceed any of these bounds are rejected with an `InvalidArgument` gRPC error. + +| Parameter | Value | Description | +| --- | --- | --- | +| `MaxLimit` | 1000 | Maximum page size. A `limit` above this returns an `exceeds maximum allowed limit` error. | +| `MaxOffset` | 10000 | Maximum offset allowed in a `PageRequest`. An `offset` above this returns an `exceeds maximum allowed offset` error. | +| `MaxScanLimit` | 10000 | Maximum number of store entries offset-based (lazy) pagination will scan past the page end. Exceeding it returns a `scanned more than 10000 entries` error suggesting key-based pagination. | + + + For datasets larger than a single page, use **key-based pagination** (`pagination.key`) rather than offset-based pagination. With offset-based pagination over a sparse filter, the scan limit may be reached before the page fills, in which case the returned `next_key` can be `nil` even when more results exist. + + + + `count_total` is no longer automatically enabled when `limit` is omitted or set to zero. Previously an empty or zero-limit page request would populate the `total` field; now `total` is `0` unless you explicitly set `pagination.count_total = true`. Requesting `count_total` on a large dataset can also hit the `MaxScanLimit` and fail — prefer key-based pagination for full traversals. + + + + +### Build Tags + +The `seid` binary can be compiled with optional Go build tags that alter consensus behavior. Build tags are passed via the `GO_BUILD_TAGS` build argument. + +The `seid` binary decides how to react to a halting validation failure at compile time via the `ConsensusPolicy` type. Each build variant compiles in exactly one policy through its `ConsensusPolicy.HandleError(err)` method, so there is no runtime branch: + +- **Default (production):** `HandleError` returns the error for every validation-failure kind, so production halting semantics are unchanged. +- **`mock_block_validation`:** `HandleError` swallows only `ErrAppHash` and `ErrDataHash` failures (the same effective set the tag has always relaxed) and halts on everything else. +- **`mock_chain_validation`:** `HandleError` swallows every swallow-eligible halting validation failure except `ErrLastCommitVerify` (excluded to avoid a downstream panic in `buildLastCommitInfo`). + +When a non-default policy swallows a failure, it increments the `sei_unsafe_validation_skipped_total` counter (labeled with the failure via the `validation_error` attribute) instead of halting. This metric is only emitted by the non-default (mock) build variants. + +#### `mock_block_validation` + +Building with `GO_BUILD_TAGS=mock_block_validation` produces a `seid` binary whose `ConsensusPolicy.HandleError` swallows `AppHash` and `DataHash` block validation failures during block execution and validation, incrementing `sei_unsafe_validation_skipped_total` for each swallowed failure instead of halting. All other validation-failure kinds halt as in production. + + + The `mock_block_validation` build is intended for testing and mock environments only. It disables cryptographic block-content validation and must never be used for a production or mainnet node. + + +A corresponding Docker image is published to ECR for this build: + +```text +sei/sei-chain:mock_block_validation- +``` + +This image is built with `GO_BUILD_TAGS=mock_block_validation` and is distinct from the standard `sei/sei-chain` image tags. + +#### `mock_chain_validation` + +Building with `GO_BUILD_TAGS=mock_balances mock_chain_validation` produces a `seid` binary whose `ConsensusPolicy.HandleError` swallows every swallow-eligible halting validation failure except `ErrLastCommitVerify`. The chain still computes every check authentically; a swallowed failure increments `sei_unsafe_validation_skipped_total{kind=...}` and continues instead of halting. `ErrLastCommitVerify` is the exception — it always halts and is not counted, because swallowing it would trigger a downstream panic in `buildLastCommitInfo`. This variant is intended for forked-state replays. + + + The `mock_chain_validation` build is intended for testing and forked-state replay environments only. It swallows nearly all halting consensus validation failures and must never be used for a production or mainnet node. + + +Corresponding Docker images are published to ECR for this build: + +```text +sei/sei-chain:mock_chain_validation- +sei/sei-chain:mock_chain_validation-nightly-- +``` + +These images are built with `GO_BUILD_TAGS=mock_balances mock_chain_validation` and are distinct from the standard `sei/sei-chain` image tags. + + Understanding network parameters helps you operate your node effectively. ### Chain Parameters @@ -187,6 +534,10 @@ Slashing Parameters: - slash_fraction_downtime: 0% (no stake slash; jail only) - slash_fraction_double_sign: 0% (no stake slash; double-signing still triggers permanent tombstoning) + +Oracle Slashing Parameters: + - min_valid_per_window: 0% (default now 0% — no oracle vote slashing, + since the Oracle Price Feeder is retired) ``` @@ -208,15 +559,21 @@ $HOME/.sei/ │ ├── node_key.json # Node identity key │ └── priv_validator_key.json # Validator signing key ├── data/ -│ ├── application.db # Application state -│ ├── blockstore.db # Block data -│ ├── cs.wal/ # Consensus write-ahead logs -│ ├── evidence.db # Evidence of misbehavior -│ ├── state.db # Tendermint state -│ └── tx_index.db # Transaction index -└── keyring-file/ # Local key storage +│ ├── application.db # Application state +│ └── tendermint/ # Tendermint databases (new subdirectory layout) +│ ├── blockstore.db # Block data +│ ├── cs.wal/ # Consensus write-ahead logs +│ ├── evidence.db # Evidence of misbehavior +│ ├── peerstore.db # Peer store +│ ├── state.db # Tendermint state +│ └── tx_index.db # Transaction index +└── keyring-file/ # Local key storage ``` + + New nodes place the Tendermint databases (`blockstore.db`, `state.db`, `tx_index.db`, `evidence.db`, `peerstore.db`, and `cs.wal`) under `data/tendermint/`. Existing nodes that already have these databases in the legacy flat `data/` layout are detected automatically and continue using the legacy paths — no migration is required. This automatic legacy-path fallback also applies to `reset`, `reindex-event`, and consensus WAL path resolution. + + This reference guide provides essential technical information for operating Sei nodes and validators. For API documentation and other detailed specifications, please refer to the respective sections in our documentation set. diff --git a/node/troubleshooting.mdx b/node/troubleshooting.mdx index 2d129dd..ac3f392 100644 --- a/node/troubleshooting.mdx +++ b/node/troubleshooting.mdx @@ -57,6 +57,27 @@ Commands: ### Database Errors + +### Mempool Errors + +```text +Error: "tx already exists in cache" (ErrTxInCache) when re-submitting a previously failed tx +Solution: This is expected behavior for txs that repeatedly fail during block execution +``` + +The mempool tracks transactions that fail during block execution to prevent infinite re-entry. A tx that fails while being included in a block is allowed **one** retry — it is removed from the cache so it can be re-submitted. If it fails a **second** time during block execution, it is left in the cache and subsequent `CheckTx` calls return `ErrTxInCache`, rejecting re-submission. + +This prevents transactions that consistently fail before fee charging (for example, out-of-gas failures) from being resubmitted indefinitely and repeatedly consuming block execution resources. + +The failure tracker is reset when the transaction is successfully included in a block. After a successful inclusion, the tx is treated as fresh again and would once more be granted a single retry on any future failure. + +**What to check if you see this:** + +- Confirm the transaction is one that has failed block execution at least twice (e.g. persistent out-of-gas or other pre-fee failures). +- If the failure is caused by a fixable condition (such as insufficient gas), rebuild and re-sign the transaction with corrected parameters — a new tx hash is admitted normally. +- A different transaction (different hash) is unaffected and will be admitted as usual. + + Database corruption can require immediate attention: ```text @@ -98,10 +119,12 @@ systemctl stop seid seidb dump-iavl -d $HOME/.sei/data/committer.db -o /home/ubuntu/iavl-dump systemctl restart seid -# For Legacy IAVL DB: -seid debug dump-iavl ``` + +The legacy IAVL backend has been fully removed. SeiDB state-commit is now mandatory (`sc-enable = true`), so all nodes should use the `seidb dump-iavl` tooling shown above. The former `seid debug dump-iavl` command has been removed. + + Always include the app hash, commit hash, and block height from your logs when reporting issues. ### Identifying AppHash Errors