Skip to content

test(persistence): make the bulk-submit directives test claim its own manifest - #911

Open
aacruzgon wants to merge 1 commit into
mainfrom
fix/postgres-bulk-submit-directives-flake
Open

test(persistence): make the bulk-submit directives test claim its own manifest#911
aacruzgon wants to merge 1 commit into
mainfrom
fix/postgres-bulk-submit-directives-flake

Conversation

@aacruzgon

Copy link
Copy Markdown
Contributor

Summary

postgres_integration::postgres_bulk_submit_import_directives_round_trip has been failing main's Code Coverage job on 5 of the last 7 runs (and Test Rust twice) since #880 merged, with:

assertion `left == right` failed
  left: []
 right: [("https://helios.software/import-mode", "merge")]

It also fails on PRs that don't touch it (#905, and the docs-only #909). This makes the test claim its own manifest.

Why

The test calls claim_next_manifest and asserts the claimed view carries the directives it just set. That claim is cross-tenant by design and ordered by added_at, and the test binary shares one container database. #880 added postgres_bulk_submit_batch_commits_bookkeeping_and_contains_errors, which registers a manifest and calls process_entries on it "to move it out of pending" — but process_entries leaves the manifest as processing with no lease, and the claim query treats an unleased processing manifest as an orphan to reclaim. So whenever the batch test's manifest was added first, the directives test claims that one (no directives, hence left: []). Whether it flakes is a race between two tests' add_manifest timestamps, which is why it fails roughly half the time.

Changes

  • crates/persistence/tests/postgres_tests.rs: new claim_specific_manifest helper — the submit-side twin of the existing export claim_specific — loops claim_next_manifest until the target (submission_id, manifest_id) comes back, holding any other lease it picks up (so the loop cannot re-claim the same foreign manifest) and returning those to the queue with release once the target is held. The directives test uses it with a unique worker id.
  • The batch test's comment no longer claims its process_entries call is a defence against concurrent claims (it isn't); the code is unchanged.

No product code changes. Whether the worker's claim query should treat a synchronously-processed, unleased manifest as reclaimable is a separate question for the bulk-submit owners; this PR only stops two tests from racing over it.

Testing

  • cargo test -p helios-persistence --features postgres --test postgres_tests -- postgres_bulk_submit — 2 passed, five consecutive runs (the two tests run concurrently in the same binary, which is the racing pair).
  • Full postgres_tests binary: 156 passed.
  • cargo fmt --all; CI-exact clippy clean.

… manifest

`postgres_bulk_submit_import_directives_round_trip` called
`claim_next_manifest` and asserted the claimed view carried the directives
it had just set. The claim queue is cross-tenant and ordered by `added_at`,
the test binary shares one container database, and the batch test that
#880 added leaves its manifest as `processing` with no lease — which the
claim query treats as an orphan to reclaim. Whenever that manifest was
added first, the directives test claimed it instead of its own and failed
with `left: []`; main's coverage job has been red on most runs since.

The test now claims through `claim_specific_manifest`, the submit-side twin
of the export tests' `claim_specific`: loop until the target manifest comes
back, hold any foreign lease picked up along the way so it cannot be
re-claimed, then release those back to the queue. The batch test's comment
no longer presents its `process_entries` call as a defence against
concurrent claims. No product code changes.

Tests: the two submit tests pass five consecutive runs together; the full
postgres_tests binary passes (156).
@codecov

codecov Bot commented Sep 2, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant