Skip to content

ci: make the gate resilient to transient upstream (apt/registry) flakes — retry infra steps only, never tests - #664

Open
baron-3dl wants to merge 1 commit into
mainfrom
ci/apt-registry-retry-resilience
Open

ci: make the gate resilient to transient upstream (apt/registry) flakes — retry infra steps only, never tests#664
baron-3dl wants to merge 1 commit into
mainfrom
ci/apt-registry-retry-resilience

Conversation

@baron-3dl

Copy link
Copy Markdown
Contributor

Why

The conductor kept parking on CI failures with a blanket "GitHub apt-mirror outage, just rerun" read. That read is now disproven (real test reds are mixed in — see the flip triage), but the underlying concern is legitimate: the gate does host-level apt-get and inline package fetches on the runner, and those genuinely flake on GitHub's mirrors. This makes those infra-only steps self-heal so a transient mirror hiccup never again forces a manual rerun.

Hard constraint honored (Rule 8)

Retries wrap only package-fetch / dependency-setup, never a test or build-correctness command. A retry around a real test would mask a genuine red — forbidden. No allowlist bypass added; tests/integration/test_runtime_target.sh untouched.

Exactly what was wrapped — and proof each is infra-only

ci.yml step (name:) Change Why it's infra-only
Install build dependencies (Build & Test job) retry-loop on apt-get update (5×, backoff) + Acquire::Retries=5 on install step body is only apt-get update/install; no test or compile command in it
Install clang (NetBSD cross target) ×2 (vms + vmsfs cross-compile jobs) same retry-loop + Acquire::Retries=5 step body is only the two apt commands
Install dependencies (Static Analysis job) same retry-loop + Acquire::Retries=5 step body is only the two apt commands
cut-release Boot the freshly-cut bundle inline docker run … bash -c "apt-get … && bash /test.sh" added Acquire::Retries=5 to the two apt-gets only apt tuning affects apt's own network fetch retries; bash /test.sh is still run exactly once, not retried
Conformance Build OVMX + run conformance inline container apt-get added Acquire::Retries=5 to update+install same — the cmake build + run_conformance.sh that follow are untouched and run once
Corpus Build OVMX + run corpus harness inline container apt-get added Acquire::Retries=5 to update+install same — run_corpus.sh runs once

For the three inline docker run … apt && test steps I deliberately did not add a loop around the whole bash -c, because that would retry the test too. Acquire::Retries only changes apt's internal fetch retry — it cannot re-run or mask the test.

Not touched (reported as follow-ups, not fixed here)

  • Node 20 deprecation warnings on actions/checkout@v4, docker/build-push-action@v6, docker/setup-buildx-action@v3, docker/setup-qemu-action@v3 — bump to Node24-based action majors in a separate housekeeping PR.
  • Dockerfile lint warnings in distro/Dockerfile.bootable (JSONArgsRecommended at CMD line ~1076; FromPlatformFlagConstDisallowed at line 51). Cosmetic; separate PR.
  • Registry-pull resilience for the docker run docker.io/library/alpine:3.20 sh -c '…test…' steps: the image pull is coupled into the same invocation as the test, so a safe fix needs a docker pull-with-retry pre-step, not a wrap of the test. Deferred to avoid touching test steps in this PR.

🤖 Generated with Claude Code

…es -- retry infra steps only, never tests

Wrap only host package-fetch/setup steps and inline apt fetches with
retry-with-backoff + apt Acquire::Retries. No test or build-correctness
command is retried, so this cannot mask a real red (Rule 8).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant