Skip to content

fix(os): make the release gate report only what it actually ran (#1064) - #1127

Merged
VijitSingh97 merged 2 commits into
develop-v2from
fix/1064-gate-honesty
Aug 19, 2026
Merged

fix(os): make the release gate report only what it actually ran (#1064)#1127
VijitSingh97 merged 2 commits into
develop-v2from
fix/1064-gate-honesty

Conversation

@VijitSingh97

Copy link
Copy Markdown
Collaborator

Closes #1064. Both halves are the same defect: a gate that reports success without having run.

--phase all ran five of eight

The all) arm called boot, update, install, provision and rig. docs/dev/appliance-release.md
said "--phase all runs everything", and step 1 of Cutting a release prescribes exactly that
command — so a maintainer following the checklist believed the three mid-write power cuts, the
mid-commit cut, the corrupt-bundle refusal, the factory reset, the wedged-/data recovery and the
whole media channel had run. None of them had.

The arm now runs all eight. The doc names them, warns that it is an evening rather than a coffee
break, and says to record the dated per-phase count for the tip actually being cut — an old green
standing in for a new one is the same defect wearing a different hat.

verify-image reported "0 failed" for checks it declined to run

Two blocks are conditional: the built-from-expected-commit pair (when PITHEAD_EXPECT_COMMIT is
unset) and the whole artifact-vs-tree comparison (when not run from the repo root) — the one that
unpacks the baked container archive to compare wizard.py. Neither skip printed anything.

Those are precisely the checks that exist because an image once shipped a dashboard two commits
stale and passed everything else. Skips are now counted, printed as (SKIPPED: reason), and
fatal: a run that declined to compare the artifact against the tree does not get to call the
image verified.

_build_image now exports PITHEAD_EXPECT_COMMIT, computed the same way BUILD_COMMIT is
recorded, so the stale-artifact guard is on in the one caller that is not a human typing a command
— it was off in every battery run, which is where it mattered most.

Coverage

The harness only runs on the KVM bench, so the wiring is asserted in tier-1: all eight phase_*
calls present in the all arm, the PITHEAD_EXPECT_COMMIT prefix on the verify call, and
verify-image's skip counting and refusal. Dropping a phase from the arm, or the env prefix, turns
the matching assertion red. Tier-1 2636 passed / 0 failed.

The end-to-end proof is running on the bench as this opens: a fresh image build, then
verify-image in three shapes — no PITHEAD_EXPECT_COMMIT, run from outside the repo, and fully
pinned — to confirm the first two now exit non-zero and the third still passes. I will post the
result here; this should not merge before it does.

Two halves of one defect: a gate that reports success without having run.

`--phase all` executed five of eight phases while docs/dev/appliance-release.md
told a maintainer that step 1 of a cut ran everything. So every cut that
followed the checklist silently skipped the three mid-write power cuts, the
mid-commit cut, the corrupt-bundle refusal, the factory reset, the wedged-/data
recovery and the whole media channel. The arm now runs all eight, and the doc
names them and says to record the dated per-phase count for the tip actually
being cut — an old green standing in for a new one is the same defect wearing a
different hat.

verify-image's two strongest checks — built-from-the-expected-commit, and the
artifact-versus-tree comparison that unpacks the baked container to compare
wizard.py — are conditional, and their skips printed nothing while the summary
said "0 failed". Those are precisely the checks that exist because an image
once shipped a dashboard two commits stale and passed everything else. Skips
are now counted, printed, and fatal: a run that declined to compare the artifact
against the tree does not get to call the image verified.

And `_build_image` now exports PITHEAD_EXPECT_COMMIT, so the guard is on in the
one caller that is not a human typing a command — it was off in every battery
run, which is where it would have mattered most.

Tier-1 2636/0.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@VijitSingh97

Copy link
Copy Markdown
Collaborator Author

Bench verification status at handoff: still running, so this should not merge yet.

The image build on the KVM bench has been re-run twice for reasons worth recording, neither of them about the fix:

  1. The first build ran os/rauc/mkimage.sh --dev without root and died on losetup: failed to set up loop device: Permission denied — it needs sudo for the loop device, which is why the harness invokes the whole of tests/os/run.sh under sudo.
  2. The second produced a release variant, because the two build commands were run without PITHEAD_TEST_SSH_PUBKEY — which _build_image sets. Scenario A did show the two SKIPPED lines and a non-zero exit, exactly as intended, but the image also carried two unrelated FAILs ("test SSH key present", "variant stamp says debug") that would have made scenario C's "0 failed" expectation unreachable for reasons unrelated to this change.

Both are notes about reproducing _build_image by hand, not about the change under test. The third build matches the harness's real environment.

What still has to be observed before merging, from a repo-root checkout of this branch on the bench:

  • sudo tests/os/verify-image.sh <img> --test with PITHEAD_EXPECT_COMMIT unset → two SKIPPED lines, a skipped count, non-zero exit.
  • the same run from outside the repo root → also skips the artifact-vs-tree block, non-zero exit.
  • the same run from the repo root with PITHEAD_EXPECT_COMMIT set to the built commit → no SKIPPED lines, "0 failed", exit 0.

Build logs are on the bench at /tmp/1064-build*.log. Tier-1 (2636/0) and the wiring assertions are already green in CI.

The guard #1064 switched on could never pass. `_build_image` handed over
`git rev-parse --short HEAD` while `os/build-image.sh` stamps the full sha into
/opt/pithead/BUILD_COMMIT, and verify-image compared the two for equality — so
every image the harness built would have failed its own verification, and the
battery would have died at the first `_build_image` call. A gate that refuses
everything is the same defect as one that refuses nothing, pointed the other way.

The harness now hands over the full sha, and verify-image matches on a prefix so
the short sha an operator copies out of `git log --oneline` verifies too. The
"-dirty" suffix is left to the clean-tree check, which is the one that means it;
the hand-rolled dirty suffix in the harness is gone, since it used
`git status --porcelain` (untracked files count) where build-image.sh uses
`git diff --quiet` (they do not), and the two answers disagreed.

Bench-proven on the KVM image built from a97d5bc, all five scenarios:
  unset, repo root              -> 2 skipped, exit 1
  unset, outside the repo root  -> 3 skipped, exit 1
  short sha (a97d5bc)           -> 92 passed, 0 failed, exit 0
  full sha                      -> 92 passed, 0 failed, exit 0
  a sha it was not built from   -> 1 failed, exit 1

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@VijitSingh97

Copy link
Copy Markdown
Collaborator Author

Bench verification is complete, and it found a defect in this branch. Fixed in d8a239d — please
read this before merging.

The three scenarios the previous comment asked for

Run from a repo-root checkout of this branch on the KVM bench, against the image built from
a97d5bc (third build, mirroring _build_image's environment):

scenario result
repo root, PITHEAD_EXPECT_COMMIT unset 90 passed, 0 failed, 2 skippedexit 1
outside the repo root, unset 86 passed, 0 failed, 3 skippedexit 1
repo root, PITHEAD_EXPECT_COMMIT set 1 failed, exit 1 ← not what it should have done

Both skip scenarios behave exactly as intended, including the third skipped line ("the artifact
matches the tree it was built from") appearing only outside the repo root.

What the third scenario found

image built from: a97d5bcdcfb46b2e187b314d9b4e1d7fbddaf2b7
  ✗ built from the expected commit (a97d5bc)

os/build-image.sh:103 stamps git rev-parse HEAD — the full sha — into
/opt/pithead/BUILD_COMMIT. _build_image handed over git rev-parse --short HEAD, and the new
check compared the two with =. They can never be equal, so switching the guard on would have
failed every image the harness builds, and the battery would have died at the first
_build_image call. A gate that refuses everything is the same defect as one that refuses nothing,
pointed the other way — which is the defect this PR is about.

Two smaller things fell out of it. The harness's hand-rolled -dirty suffix used
git status --porcelain (untracked files count) where build-image.sh uses git diff --quiet
(they do not), so the two disagreed on a tree with only untracked files; and -dirty was being
folded into an identity check that has a dedicated clean-tree check sitting right next to it.

The fix, and its bench proof

The harness hands over the full sha; verify-image matches on a prefix, so the short sha an
operator copies out of git log --oneline verifies too, and -dirty is left to the check that
means it. Re-run on the same image:

PITHEAD_EXPECT_COMMIT result
a97d5bc (short) 92 passed, 0 failedexit 0
the full sha 92 passed, 0 failedexit 0
deadbee (a commit it was not built from) 1 failedexit 1

The last row matters as much as the first two: the check still fails on the thing it exists to
catch, so this is not a guard that was widened into a tautology.

Tier-1 gains three assertions pinning the shape agreement, with the mutations named in the test
comment (revert the harness to --short, or verify-image to equality, and they go red). Local
tier-1 on the branch: 2639 passed, 0 failed; make lint-sh clean.

@VijitSingh97
VijitSingh97 merged commit be25368 into develop-v2 Aug 19, 2026
15 checks passed
VijitSingh97 added a commit that referenced this pull request Aug 19, 2026
@VijitSingh97
VijitSingh97 deleted the fix/1064-gate-honesty branch August 19, 2026 03:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant