Skip to content

feat(ci): build and ship aarch64-unknown-linux-gnu - #364

Open
avrabe wants to merge 5 commits into
mainfrom
feat/aarch64-linux-release
Open

feat(ci): build and ship aarch64-unknown-linux-gnu#364
avrabe wants to merge 5 commits into
mainfrom
feat/aarch64-linux-release

Conversation

@avrabe

@avrabe avrabe commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

v1.4.1 shipped four binaries — x86_64-linux, both macOS arches, Windows. ARM Linux had none: servers, arm64 containers, ARM CI runners and SBCs all had to build from source with a C++ toolchain, which is the dependency #313 exists to remove.

The stated blocker no longer applies

release.yml carried this note:

No aarch64-linux for now: … cross-compiling z3-sys for aarch64-linux is untested.

True when written, and now moot. The target builds natively on a GitHub-hosted arm64 Linux runner — nothing is cross-compiled. z3-sys vendors and statically compiles z3, so it needs no host libz3 on any runner, ARM included, and those runners are free for public repositories, which this one is.

Where the evidence lives is the load-bearing decision

release.yml only fires on a tag push. A target added there alone would ship on the strength of a job that had never run — the unexercised-gate shape this repo keeps finding, and precisely what v1.4.1 was cut to punish.

So the target is also built on every PR via the CI matrix. A toolchain or z3-sys regression on ARM now fails a check immediately, instead of surfacing during a release when it is most expensive.

This PR's own CI is that evidence. If the ubuntu-22.04-arm build is not green here, the release entry does not belong on main.

Both #311 gates extended, not left x86_64-only

  • Floor assertion — now reads the ARM binary's own symbols; the step no longer hardcodes the x86_64 path. Still fail-closed: an undeterminable floor is an error, not a pass.
  • Smoke run — the ARM binary is executed on the ARM image it was built for, not merely inspected.

A shipped target whose floor is only hoped-for is exactly how v1.2.0 went out unloadable.

Status is draft, deliberately

TEST-AARCH64-LINUX-RELEASE-TARGET is filed against v1.5.0 as draft, not verified. The workflow edit is not evidence. It flips when the ARM job has actually run green — and the release gate correctly reads ✗ NOT cuttable until then.

Refs #311, #313

avrabe and others added 3 commits August 28, 2026 06:12
v1.4.1 shipped four binaries — x86_64-linux, both macOS arches, Windows.
ARM Linux had none: servers, arm64 containers, ARM CI runners and single
board machines all had to build from source with a C++ toolchain, which
is the dependency #313 exists to remove.

The deferral note in release.yml gave the reason as "cross-compiling
z3-sys for aarch64-linux is untested". That was true and is now moot: the
target builds NATIVELY on a GitHub-hosted arm64 Linux runner, so nothing
is cross-compiled. z3-sys vendors and statically compiles z3, so it needs
no host libz3 on any runner, ARM included, and those runners are free for
public repositories — which this one is.

--- where the evidence lives is the load-bearing decision ---

`release.yml` only fires on a tag push. A target added THERE ALONE would
ship on the strength of a job that had never run: the unexercised-gate
shape this repository keeps finding, and precisely what v1.4.1 was cut to
punish. So the target is also built on EVERY PR via the CI matrix, and a
toolchain or z3-sys regression on ARM fails a check immediately instead
of surfacing during a release.

--- both #311 gates extended, not left x86_64-only ---

The glibc/libstdc++ floor is now asserted from the ARM binary's OWN
symbols (the step no longer hardcodes the x86_64 path), and the binary is
SMOKE-RUN natively on the ARM image it was built for rather than merely
inspected. A shipped target whose floor is only hoped-for is exactly how
v1.2.0 went out unloadable.

The rivet artifact is deliberately `draft`, not `verified`: the workflow
edit is not evidence. It flips when the ARM job has actually run green.

Refs #311, #313

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RZof1M5HMBSYVZPeoT4MEn
`Build (ubuntu-22.04-arm)` ran green on PR #364. That settles the three
things actually in doubt: arm64 Linux runners are available to this
repository, z3-sys compiles its vendored C++ natively on ARM, and loom
builds for the target.

It settles nothing about the release path. Publishing the archive,
asserting the glibc floor from the ARM binary's symbols, and smoke-running
it all live in release.yml, which fires only on a tag push. That code is
written and reviewed; none of it has executed.

So the artifact stays `draft` with each criterion labelled PROVEN or
PENDING A TAG. Flipping it to `verified` on the strength of the CI job
would assert an untested gate — the exact defect class v1.4.1 was cut to
remove, and it would be a poor way to celebrate that.

Refs #311, #313

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RZof1M5HMBSYVZPeoT4MEn
The Verification Gate had 0 successful runs in 60 across six weeks. Every
run exceeded its 30-minute cap — durations 30.4 to 102.5 minutes, so the
FLOOR missed the budget by 24 seconds. A budget the minimum cannot meet
is not a budget.

`main` requires no checks, so nothing read it. v1.4.1 shipped with its
two gate runs cancelled at 30.5 and 39.2 minutes, and I merged that
release myself while calling this check systemic. #332 was a check that
could not go red; this is a check that cannot go green, on a board where
nothing looks. Same defect class, and treating it as ambient noise is how
it survived six weeks.

--- I made it worse ---

The filter matches every TEST-* feature artifact. It matched 23 when #353
was measured; it matches 35 now, and 13 of those are mine from this
week's work. One of them, TEST-AARCH64-LINUX-RELEASE-TARGET, carried
`gh release view <tag> ...` as a step — a literal placeholder, and `<tag>`
is shell redirection, so the command could never succeed. A draft
artifact for correctly-unfinished work would have held the gate red by
itself. Those steps are removed; an artifact with nothing runnable now
declares no steps and is skipped, which is the honest state.

--- three changes ---

1. `solver-differential` becomes its OWN job. TEST-RULE-SOLVER-DIFFERENTIAL
   is the artifact whose green run carries a claim: under
   LOOM_VERIFY_BACKEND=both any divergence between the incumbent and the
   certificate-checked engine panics, so the run passing IS the
   no-divergence assertion. It was riding behind 30-plus unrelated
   artifacts, inheriting their runtime and their verdict. One test binary,
   one filter, 45-minute budget — deliberately not 25, because the sweep's
   floor implies a ~20-minute build and setting a budget the job might miss
   is the mistake being corrected here.

2. The broad sweep's cap goes 30 -> 120 so it can finish and REPORT.
   Advisory by nature, but its failures should be readable rather than
   ambient — two of them (loom optimize smoke, measure_corpus) are real
   and were invisible behind the cancellations.

3. `run_verification.py` skips `draft` artifacts. A draft is unfinished BY
   DECLARATION; running its steps can only fail, and a gate red because
   planned work is planned cannot be told apart from a gate red because
   something broke.

Verified locally: `LOOM_VERIFY_BACKEND=both cargo test --release --lib
-p loom-core --features verification -- test_verify_all_rules` passes.

NOT claimed fixed: the gate has not yet gone green in CI. That is the
only evidence that counts here, and it arrives on this PR.

Making the differential job REQUIRED on main is deliberately left out —
that changes merge policy and is the repository owner's call, not a
side effect of a CI edit. It is the third step #353 asks for.

Refs #353, #332

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RZof1M5HMBSYVZPeoT4MEn
@avrabe

avrabe commented Sep 2, 2026

Copy link
Copy Markdown
Contributor Author

This PR now also addresses #353 — including my own contribution to it

#353 landed while this was open: the Verification Gate has 0 successful runs in 60 across six weeks, every run over its 30-minute cap (floor 30.4 min), on a main whose required_checks is empty.

I should own the part of that which is mine. I merged past this check repeatedly, calling it systemic alongside Rocq, including for the v1.4.1 release — whose two gate runs were cancelled at 30.5 and 39.2 minutes. #332 was a check that could not go red; this is a check that cannot go green. Same defect class, and treating it as ambient noise is how it survived six weeks.

I also made it measurably worse

The filter matches every TEST-* feature artifact. It matched 23 when #353 was measured; it matches 35 now, and 13 of those are mine from this week.

Worse, one of them — TEST-AARCH64-LINUX-RELEASE-TARGET, added in this PR — carried this as a verification step:

gh release view <tag> --json assets --jq '.assets[].name' | grep aarch64-unknown-linux-gnu

<tag> is a literal placeholder, and < is shell redirection. That command can never succeed. A draft artifact for correctly-unfinished work would have held the gate red on its own. That is almost certainly part of why this PR's gate is failing.

Three changes

  1. solver-differential becomes its own job. TEST-RULE-SOLVER-DIFFERENTIAL is the artifact whose green run carries a claim — under LOOM_VERIFY_BACKEND=both any divergence panics, so the run passing is the no-divergence assertion. It was riding behind 30+ unrelated artifacts, inheriting their runtime and their verdict. Now one test binary, one filter, 45-minute budget — deliberately not 25, since the sweep's floor implies a ~20-minute build and setting a budget the job might miss is the exact mistake being corrected.

  2. Sweep cap 30 → 120, so it can finish and report. Advisory by nature, but its failures should be readable rather than ambient — two of them (loom optimize smoke, measure_corpus) are real and were invisible behind six weeks of cancellations.

  3. run_verification.py skips draft artifacts. A draft is unfinished by declaration; running its steps can only fail, and a gate red because planned work is planned cannot be told apart from a gate red because something broke.

Not claimed fixed

The gate has not yet gone green in CI. That is the only evidence that counts, and it arrives on this PR — which is the same standard I applied to the ARM build here.

Deliberately not done: making the differential job required on main. That changes merge policy and is your call, not a side effect of a CI edit. It is step 3 of what #353 asks for, and I'd rather ask than quietly tighten how the repo merges.

avrabe and others added 2 commits September 2, 2026 07:18
…ld not build

With the gate finally running to completion (#353), it reports what six
weeks of cancellations hid: 23 passed, 5 failed, 7 skipped. THREE of the
five failures are artifacts I marked `verified` this week.

    [FAIL] TEST-332-VERIFY-REPORTS-NO-VACUOUS-SUCCESS
    [FAIL] TEST-331-VERIFICATION-COVERAGE-IS-REPORTED
    [FAIL] TEST-346-OUTPUT-IS-VALIDATED-BEFORE-IT-IS-WRITTEN

Not flaky, and not the tests: the gate builds with `RUSTFLAGS: -D
warnings`, and three long-standing `unused_variable` warnings in
`maybe_differential_gate` are hard errors there. `loom-cli` does not
compile in that environment, so every artifact whose evidence is a
`cargo test -p loom-cli` invocation fails.

I saw those warnings earlier this week and wrote them off as
"pre-existing in the feature-gated maybe_differential_gate". They were
pre-existing. They were also fatal in the one environment that had not
completed a run since July — so the artifacts asserted `verified` on
evidence that could not execute. That is the same defect as #332 and
#346 wearing the traceability layer's clothes, and I filed it myself
three times in one week.

The parameters are read only inside the `differential` cfg block, so
they are genuinely unused without that feature; they are now
`allow(unused_variables)` under exactly that cfg rather than silenced
unconditionally.

Verified under the gate's own conditions rather than a normal build:
`RUSTFLAGS="-D warnings" cargo test --release -p loom-cli` -> 21/21.

Refs #353, #332, #331, #346

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RZof1M5HMBSYVZPeoT4MEn
Every step ran with stdout and stderr on DEVNULL, so a failing artifact
reported `✗ failed: <command>` and nothing else.

That is most of why six weeks of this gate taught nobody anything (#353).
The runs that were not cancelled produced a verdict with no evidence
attached, so the only route to a cause was reproducing by hand — and this
environment differs from a developer machine in precisely the ways that
matter: `RUSTFLAGS: -D warnings`, a different libz3, a different core
count and therefore different test parallelism.

I hit that wall myself just now. TEST-331 fails here and passes locally,
including under the gate's own `-D warnings` and at 8 test threads across
25 consecutive runs. I had a plausible mechanism — `take_revert_summary`
clears the whole global counter map while a sibling test reads it — and
no way to confirm it, because the gate discards the assertion that would
say. Rather than commit a fix for a cause I had guessed at, this makes
the gate produce the evidence.

Failing steps now echo the tail of their combined output (40 lines,
enough for a Rust test summary plus the assertion that produced it, short
enough that a dozen failures do not bury the run).

A gate that says a thing failed without saying how is a weaker form of
the same defect as a gate that cannot fail at all.

Refs #353

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RZof1M5HMBSYVZPeoT4MEn
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant