test(db): wait for real Postgres readiness, not just the mapped port - #64
Merged
Conversation
The Release pipeline failed twice in a row on the db integration suite, with a different test each time (TestOTelCallbacksTableAndRowsAffected, then TestPostgresMigrationsInvalidPath) and the same signature: "failed to ping database ... connection reset by peer / unexpected EOF", roughly 10s in — well inside the 60s deadline, so not a timeout. Root cause is the wait strategy. The postgres image runs initdb against a temporary server, stops it, then starts the real one. All eight Postgres containers in this package waited only on the mapped port, so the container could be declared ready while the server was on its way down; the connection that followed was reset. testcontainers-go documents this in postgres.BasicWaitStrategies, which waits for the readiness log line twice "because it will restart itself after the first startup" and warns the tests "will be flaky" on macOS otherwise — this runner is a Mac. That also explains the intermittency: the window is invisible on an idle machine and opens under CI load, which is why it reproduced on the runner and never locally. MySQL and MSSQL in the same package were unaffected because they already wait on a real query via wait.ForSQL. Add the occurrence-2 log check alongside the port check behind a single shared helper, so the strategy cannot drift back one call site at a time, and give it a deadline with headroom for a loaded runner. Verified: db integration suite passes idle (80s) and under 10-way CPU contention (103s), where the loaded case is what CI was failing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Symptom
The Release pipeline failed twice in a row on the db integration suite — a different test each time, same signature:
Both ~10s in — well inside the 60s deadline, so not a timeout. Both pass locally. The triggering commit was docs-only.
Root cause
The wait strategy, not the tests.
The postgres image runs
initdbagainst a temporary server, stops it, then starts the real one. All eight Postgres containers in this package waited only on the mapped port:So the container could be declared ready while the server was on its way down, and the connection that followed got reset.
This is documented in testcontainers-go itself —
postgres.BasicWaitStrategies():The db tests used only the second half of that pair. The runner is a Mac.
That also explains the intermittency and why a docs commit "caused" it: the window is invisible on an idle machine and opens under CI load. MySQL and MSSQL in the same file were never affected — they already wait on a real query via
wait.ForSQL, with a comment saying so.Fix
Add the occurrence-2 log check alongside the port check, behind one shared helper (
postgresReady()) so the strategy can't drift back one call site at a time, with a deadline sized for a loaded runner. 8 call sites across 3 files now share it.Confirmed no other package is affected: every
postgres.Runin the repo is indb/, and the remainingForListeningPortuses (Temporal, sshd) are services that don't restart themselves during init.Verification
task ci:checkgo vet -tags='example integration argo' ./...golangci-lint --build-tags=integration ./db/...The loaded run is the condition CI was failing under.