Skip to content

Add integration test for resetIndexer/replayFromLedger racing concurr… - #1411

Open
Fury03 wants to merge 3 commits into
LabsCrypt:mainfrom
Fury03:test/issue-1293-reset-replay-race
Open

Add integration test for resetIndexer/replayFromLedger racing concurr…#1411
Fury03 wants to merge 3 commits into
LabsCrypt:mainfrom
Fury03:test/issue-1293-reset-replay-race

Conversation

@Fury03

@Fury03 Fury03 commented Aug 30, 2026

Copy link
Copy Markdown
Contributor

Closes #1293

Problem Statement (The Bug)

The resetIndexer and replayFromLedger functions are tested in isolation but never interleaved with SorobanEventWorker's own mutex-protected poll cycle. This means the race described in Functional Edge Case #19 is untested: when an operator triggers incident-recovery tooling while a poll is mid-flight, the poll's stale cursor can silently overwrite the reset/replay cursor, rendering the recovery action ineffective.

This issue cannot be fixed with a local patch because it requires concurrent execution of two independently-tested code paths — something isolated unit tests structurally cannot cover.

Solution Comparison and Decision

Option Description Why rejected
A. Unit test with mocked concurrency Mock both paths and assert ordering Does not exercise real async interleaving; the race depends on actual promise resolution ordering
B. Mutex extension only (no test) Extend the worker mutex to cover reset/replay Addresses the symptom without proving it works; regressions go undetected
C. Integration test with deferred promise Use a deferred getEvents mock to inject reset/replay mid-flight, assert reset cursor wins Chosen — directly reproduces the real race, fails against current code, passes once fixed

The Change (Code modifications)

Added backend/tests/integration/reset-replay-race.test.ts — a single new test file containing two integration tests.

Core test mechanism:

// Deferred promise controls when the poll's getEvents call resolves
let resolveGetEvents!: (value: any) => void;
const getEventsDeferred = new Promise((resolve) => { resolveGetEvents = resolve; });

// Shared mutable state simulates Postgres — both worker and reset/replay write here
let dbIndexerState = { lastLedger: 199, lastTimestamp: "2024-01-01T00:00:00Z" };

// Upgraded mock: first call blocks, second call (after reset) reads from shared state
server.getEvents
  .mockImplementationOnce(() => getEventsDeferred)
  .mockImplementationOnce(() => ({ events: [mockEvent], latestLedger: 210 }));

// Inject reset mid-flight while poll is awaiting RPC
resetIndexer(50, "incident recovery");

// Release the poll — it now writes its stale cursor, overwriting 50
resolveGetEvents({ events: [mockEvent], latestLedger: 210 });

Test 1 — resetIndexer race: Starts a poll, injects resetIndexer(50) mid-flight via deferred promise, asserts reset cursor wins. Current code fails: poll overwrites 50 with stale 200.

Test 2 — replayFromLedger race: Starts a second poll mid-flight, injects replayFromLedger(100), asserts replay cursor wins. Current code fails: second poll overwrites 100 with stale 300.

Compatibility Note

No INTERFACE_VERSION change. This PR adds only a test file; no production code is modified.

Testing

Test Current code Expected after fix
resetIndexer cursor survives concurrent poll FAILED (got 200, expected 50) PASSED
replayFromLedger cursor survives concurrent poll FAILED (got 300, expected 100) PASSED

Pre-existing failures in indexer-state.test.ts, soroban-event-worker.test.ts, and eventRace.test.ts are unrelated.

Additional Notes

Scope: Only backend/tests/integration/reset-replay-race.test.ts is added. No production code is modified.

Fury03 and others added 3 commits August 30, 2026 20:05
…ent poll

Adds an integration test that exercises the race condition described in
Functional Edge Case #19 (issue LabsCrypt#1293): resetIndexer and replayFromLedger
bypass the SorobanEventWorker's batchMutex, so their DB cursor writes can
be overwritten by a concurrent poll's stale upsert.

The test deliberately fails against current code and will pass once the
race is fixed (the poll must re-read the cursor before writing it back,
or resetIndexer/replayFromLedger must go through the batchMutex).

🤖 Generated with Codebuff
Co-Authored-By: Codebuff <noreply@codebuff.com>
CI was red on both backend jobs: the two tests asserted the pre-fix race and
the logger mock stubbed `requestContext` as a plain vi.fn(), so
replayFromLedger threw "requestContext.getStore is not a function" as an
unhandled rejection.

Since this branch was opened, LabsCrypt#1221 landed the production fix — resetIndexer
now writes inside `sorobanEventWorker.runExclusive`, the same mutex that
serialises poll/replay batches. Rework the tests to pin that behaviour down:

- Drive the exported `sorobanEventWorker` singleton instead of a fresh
  `new SorobanEventWorker()`. That singleton is the instance resetIndexer and
  replayFromLedger lock against, so a separate instance shares no mutex with
  them and the ordering under test would not exist.
- Assert the guarantee rather than the bug: the operator action is held until
  the in-flight batch releases the mutex, the poll's cursor write lands first,
  and the reset/replay cursor is the one that survives.
- Give the logger mock a real AsyncLocalStorage for `requestContext`.

Both tests still fail if resetIndexer's `runExclusive` wrapper is removed,
so they guard the fix rather than merely passing alongside it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Audit] No test covers resetIndexer/replayFromLedger racing a concurrent scheduled poll

1 participant