feat(wallet): accelerate storage synchronization

[BraydenLangley]

Keep this pull request in draft until operator review. After every push, wait
for all applicable checks on the exact head to finish and fix every failure
before requesting review or calling the work complete.

Program and scope

• Program gates advanced: practical remote-to-local wallet replication, exact progress/ETA, backward-compatible provider rollout
• Why this change is needed: remote getSyncChunk reads dominated an isolated desktop replication, and an existing provider reproduced HTTP 413 when the requested rough page size fit the request contract but the serialized JSON-RPC response exceeded the service ceiling.
• Explicitly out of scope: changing WalletStorageManager's global sync lock or allowing normal writes to interleave with offset-based replication. That requires a cursor/keyset or consistent-snapshot protocol so mutations cannot shift offsets. No production service rollout or npm publication is performed here.
• Rebased onto main: 93161fa395ce59ee95a7f567028cada653ac2fb8
• Exact head SHA: a4c615b9b4273b197dcf057f89df94816e313081

Atomic per-page data/checkpoint transactions are already present on main through #477. This PR relies on that behavior but does not claim it as a new branch change.

Impact

• No public package source or manifest changed
• Public package source or manifest changed; affected packages are listed below
• Infrastructure source, dependency, image, or deployment configuration changed
• Public API, exports, types, runtime targets, or browser/mobile behavior changed
• Security-sensitive remote validation boundary changed
• Documentation or examples changed

Affected packages and intended versions (publication occurs only through the release workflow after approval):

• @bsv/wallet-toolbox 2.11.0
• @bsv/wallet-toolbox-client 2.11.0
• @bsv/wallet-toolbox-mobile 2.11.0

Storage services must consume the released toolbox and run its normal Knex migration before relying on the faster indexed source reads.

Implementation

• Fill sync pages with bounded adaptive source reads. The first query is capped at 250 for every entity type; a representative 250-record proven-transaction page uses three reads (10, 80, 160) instead of 25 fixed ten-row reads.
• Add (userId, provenTxId) and (userId, txid) transaction indexes. SQLite and MySQL migration tests verify index presence and planner usability. MySQL rollback restores the automatic userId foreign-key support index before removing the composites.
• Add opt-in source totals for exact progress and ETA. Counts execute only when requested, remain best-effort at the source, and are runtime-validated when received remotely.
• Carry the writer-local syncStateId into page requests to disambiguate legacy duplicate checkpoints, and make provider registration unique by user/provider identity rather than display name.
• For remote getSyncChunk only, recognize HTTP 413, halve the requested rough response budget, retry at most four times, and remember the successful limit for later pages. The original request object is not mutated, unrelated failures are not retried, and write RPCs are never replayed.
• Add a retained authenticated HTTP candidate-provider benchmark for SQLite and MySQL.

Verification

• Full Wallet Toolbox suite: 219 suites passed; 2,055 tests passed; one pre-existing test skipped.
• Browser/mobile package tests: 36 tests passed. Toolbox, browser, and mobile lint/typecheck passed.
• Packed consumer contracts passed for toolbox, client, and mobile packages.
• Repository health: 190 script controls passed; generated package/migration documentation is current; zero contract findings.
• Real MySQL 8.4 validation used an isolated temporary server and fresh databases. Migrate-down/migrate-up, index presence/planner probes, foreign-key-safe rollback, and per-user source totals passed.
• HTTP and mixed-version coverage exercises both full and mobile clients against actual Response objects: first response 413, retry 200, optional fields preserved, and a legacy chunk without totals accepted. Legacy requests omit totals without count-query overhead.
• Candidate-provider benchmark over authenticated localhost HTTP, seven samples, 250 proven transactions plus 250 linked transactions:
  • SQLite: 49.2 ms p50 / 57.7 ms p95.
  • MySQL 8.4: 54.7 ms p50 / 64.2 ms p95.
  • Every sample used exactly three proven-transaction source reads: 10, 80, 160.
• Browser contract passed: Vite 1,609,916 raw / 379,552 gzip / 297,188 Brotli; esbuild 1,254,827 raw / 344,883 gzip / 277,729 Brotli.
• Mobile contract passed: Metro 1,662,714 raw / 419,740 gzip / 326,785 Brotli; Hermes 3,371,236 raw / 1,347,951 gzip / 1,062,310 Brotli.
• Reviewed ceilings are 1,611,000/380,000/297,500 for Vite, 1,256,000/346,000/278,200 for esbuild, and 3,373,000/1,368,000/1,070,000 for Hermes. Existing Metro ceilings remain unchanged.
• Complete local diff self-review performed for correctness, compatibility, retry safety, artifacts, documentation, and accidental operational-data disclosure.
• All applicable hosted checks are terminal and successful on exact head a4c615b9b4273b197dcf057f89df94816e313081.

The latency measurements are observational local candidate-provider results, not cross-machine or production SLAs. The earlier existing-provider fetch proved resumability after a real 413 but was intentionally stopped and is not presented as a completed production benchmark.

Security and dependencies

• No dependency or lockfile change
• Changelog, runtime relevance, peer compatibility, and transitive impact reviewed
• No write RPC is automatically retried
• Remote totals are validated as non-negative safe integers with a matching aggregate
• No new override, advisory dismissal, suppression, or skipped test
• Public diff contains no provider hostname, wallet identity, database path, SSH topology, or credentials

Dependency evidence

• No dependency ranges or lockfile snapshots changed. The only package-manifest change adds the local bench:storage-sync script.
• Dependency and release governance passed locally with all 20 registered overrides.

Governance timing note

• Hosted CI crossed the mutation-policy reviewBy date in UTC before this workstation crossed midnight. The review window is refreshed through 2026-09-30.
• The latest scheduled mutation run on 2026-08-30 passed 30 target lanes; air-gap-codec reached its 20-minute cancellation limit, so the aggregate run was cancelled. This PR does not claim a fully green scheduled mutation campaign.

Compatibility

• Older clients omit includeTotals and syncStateId; new providers preserve legacy behavior and do not count totals unless requested.
• Older providers ignore additive request fields and may return a chunk without totals; new full and mobile clients accept that response.
• The 413 fallback is client-only and applies to an idempotent sync read. A provider that fits the original page budget sees one request and no behavior change.
• Another HTTP status or a non-network 413-shaped application error is not retried.
• Reverting clients removes retry/totals support; reverting services forfeits source-query/index improvements but remains wire-compatible.

Release and operations

• No npm publication was performed from a workstation or from this PR
• Documentation, changelog, migration notes, and release notes are current
• PR remains draft for operator review

Rollout remains staged: review and merge the packages, publish through the governed release workflow, rebuild storage services against the release, validate a non-production remote-to-local sync, then enable the consumer UI. Do not advertise production local backup until the consumer progress/background/cancel follow-ups are independently validated.

Completion evidence

• Any downstream tracking update is limited to work proved by merged code and passing checks
• Review conversations are resolved
• Documentation, migration notes, release notes, and operator guidance are current or concretely not applicable
• Exact-head hosted checks are terminal and green

[codecov]

Codecov Report

❌ Patch coverage is 91.66667% with 8 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...ox/src/storage/remoting/entityValidationHelpers.ts	71.42%	2 Missing and 2 partials ⚠️
...-toolbox/src/storage/remoting/StorageClientBase.ts	87.50%	0 Missing and 2 partials ⚠️
...s/wallet/wallet-toolbox/src/storage/StorageKnex.ts	85.71%	0 Missing and 1 partial ⚠️
...wallet/wallet-toolbox/src/storage/StorageReader.ts	0.00%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud