Skip to content

fix(cloud): shared-localstack health gate + db-create connect race - #109

Merged
gustavobertoi merged 1 commit into
mainfrom
fix/cloud-stack-health-readiness
Jul 1, 2026
Merged

fix(cloud): shared-localstack health gate + db-create connect race#109
gustavobertoi merged 1 commit into
mainfrom
fix/cloud-stack-health-readiness

Conversation

@gustavobertoi

Copy link
Copy Markdown
Contributor

Two runtime bugs surfaced while exercising the cloud/resource stack live on WSL2. Both were invisible to the existing unit/golden tests because nothing drove these commands against a real daemon — the follow-up e2e pipeline (release gate) will.

1. shared-localstack unhealthy → up aborts

Symptom: up fails with service "shared-localstack" ... is unhealthy after 1 attempt(s) even though LocalStack logs Ready.

Root cause: LocalStack 3.x reports each configured SERVICES entry as "available" on startup; a service only flips to "running" after its first request. The healthcheck greps solely for running, so it never matches on a fresh engine — the container is permanently unhealthy, and the up saga's health gate correctly aborts. It's a deadlock: nothing is running until traffic arrives, but traffic can't arrive until the engine is up.

Fix: gate on available|running (edge up + providers loaded). Verified against the live container: new check exits 0, old exits 1.

  • templates/localstack/template.yaml healthcheck updated
  • templates/localstack/golden.yaml regenerated through the real render path (only the healthcheck line changes)
  • regression test TestLocalStackHealthGatesOnAvailable

2. db createconnection reset by peer on 127.0.0.1:45432

Symptom: db create <name> fails with connect to shared postgres on 127.0.0.1:45432: ... read: connection reset by peer.

Root cause: a readiness race. The imperative resource path (and the saga provision phase) publishes the engine's host port via an up-time compose overlay; compose up -d recreates the container to bind it. For ~1–2s the Docker userland proxy accepts the TCP connection but Postgres isn't listening yet, so it RSTs the handshake. A single immediate connect loses the race.

Fix: retryingPgConnect — a capped-backoff retry (30s budget) around the admin connect that retries only transient just-restarted errors (reset / refused / EOF / "starting up") and fails fast on real errors (bad creds, unknown db). Wired into both the imperative registry and the saga provision phase. Unit + race tested with an injected clock/connector (no live server needed).

Tests

  • internal/generate: golden + determinism green; new localstack regression test
  • internal/orchestrate: 6 new tests for the retry (classification, eventual success, fail-fast, budget bound, ctx cancel, nil passthrough) + -race
  • touched-package suites (orchestrate, generate, cli) all green

🤖 Generated with Claude Code

Two runtime bugs surfaced while exercising the cloud/resource stack on WSL2.

1. shared-localstack "unhealthy after 1 attempt". LocalStack 3.x reports each
   configured SERVICE as "available" on startup — a service only flips to
   "running" after its first request. The healthcheck greps solely for
   "running", so nothing ever matches, the container is permanently unhealthy,
   and the up saga aborts before any traffic can flip a service to "running"
   (a deadlock). Gate on `available|running` — the honest "edge up + providers
   loaded" signal. Verified live on localstack 3.8.1: the new check exits 0, the
   old one exits 1. Golden regenerated through the real render path; regression
   test added (internal/generate).

2. `db create` → "connect to shared postgres on 127.0.0.1:45432: read:
   connection reset by peer". The imperative resource path (and the up provision
   phase) publishes the engine's host port via an up-time compose overlay, and
   `compose up -d` recreates the container to bind it. For a second or two the
   Docker userland proxy accepts the TCP connection but Postgres isn't listening
   yet, so it RSTs the handshake. A single immediate connect loses the race.
   Add retryingPgConnect: a capped-backoff retry (30s budget) around the admin
   connect that retries only transient "just-restarted" errors (reset/refused/
   EOF/starting-up) and fails fast on real errors (bad creds/unknown db). Wired
   into both the imperative registry and the saga provision phase. Unit + race
   tested with an injected clock/connector (no live server needed).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@gustavobertoi
gustavobertoi merged commit 69a6b8e into main Jul 1, 2026
4 checks passed
@gustavobertoi
gustavobertoi deleted the fix/cloud-stack-health-readiness branch July 1, 2026 18:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant