feat(ci): build and ship aarch64-unknown-linux-gnu - #364
Conversation
v1.4.1 shipped four binaries — x86_64-linux, both macOS arches, Windows. ARM Linux had none: servers, arm64 containers, ARM CI runners and single board machines all had to build from source with a C++ toolchain, which is the dependency #313 exists to remove. The deferral note in release.yml gave the reason as "cross-compiling z3-sys for aarch64-linux is untested". That was true and is now moot: the target builds NATIVELY on a GitHub-hosted arm64 Linux runner, so nothing is cross-compiled. z3-sys vendors and statically compiles z3, so it needs no host libz3 on any runner, ARM included, and those runners are free for public repositories — which this one is. --- where the evidence lives is the load-bearing decision --- `release.yml` only fires on a tag push. A target added THERE ALONE would ship on the strength of a job that had never run: the unexercised-gate shape this repository keeps finding, and precisely what v1.4.1 was cut to punish. So the target is also built on EVERY PR via the CI matrix, and a toolchain or z3-sys regression on ARM fails a check immediately instead of surfacing during a release. --- both #311 gates extended, not left x86_64-only --- The glibc/libstdc++ floor is now asserted from the ARM binary's OWN symbols (the step no longer hardcodes the x86_64 path), and the binary is SMOKE-RUN natively on the ARM image it was built for rather than merely inspected. A shipped target whose floor is only hoped-for is exactly how v1.2.0 went out unloadable. The rivet artifact is deliberately `draft`, not `verified`: the workflow edit is not evidence. It flips when the ARM job has actually run green. Refs #311, #313 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RZof1M5HMBSYVZPeoT4MEn
`Build (ubuntu-22.04-arm)` ran green on PR #364. That settles the three things actually in doubt: arm64 Linux runners are available to this repository, z3-sys compiles its vendored C++ natively on ARM, and loom builds for the target. It settles nothing about the release path. Publishing the archive, asserting the glibc floor from the ARM binary's symbols, and smoke-running it all live in release.yml, which fires only on a tag push. That code is written and reviewed; none of it has executed. So the artifact stays `draft` with each criterion labelled PROVEN or PENDING A TAG. Flipping it to `verified` on the strength of the CI job would assert an untested gate — the exact defect class v1.4.1 was cut to remove, and it would be a poor way to celebrate that. Refs #311, #313 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RZof1M5HMBSYVZPeoT4MEn
The Verification Gate had 0 successful runs in 60 across six weeks. Every run exceeded its 30-minute cap — durations 30.4 to 102.5 minutes, so the FLOOR missed the budget by 24 seconds. A budget the minimum cannot meet is not a budget. `main` requires no checks, so nothing read it. v1.4.1 shipped with its two gate runs cancelled at 30.5 and 39.2 minutes, and I merged that release myself while calling this check systemic. #332 was a check that could not go red; this is a check that cannot go green, on a board where nothing looks. Same defect class, and treating it as ambient noise is how it survived six weeks. --- I made it worse --- The filter matches every TEST-* feature artifact. It matched 23 when #353 was measured; it matches 35 now, and 13 of those are mine from this week's work. One of them, TEST-AARCH64-LINUX-RELEASE-TARGET, carried `gh release view <tag> ...` as a step — a literal placeholder, and `<tag>` is shell redirection, so the command could never succeed. A draft artifact for correctly-unfinished work would have held the gate red by itself. Those steps are removed; an artifact with nothing runnable now declares no steps and is skipped, which is the honest state. --- three changes --- 1. `solver-differential` becomes its OWN job. TEST-RULE-SOLVER-DIFFERENTIAL is the artifact whose green run carries a claim: under LOOM_VERIFY_BACKEND=both any divergence between the incumbent and the certificate-checked engine panics, so the run passing IS the no-divergence assertion. It was riding behind 30-plus unrelated artifacts, inheriting their runtime and their verdict. One test binary, one filter, 45-minute budget — deliberately not 25, because the sweep's floor implies a ~20-minute build and setting a budget the job might miss is the mistake being corrected here. 2. The broad sweep's cap goes 30 -> 120 so it can finish and REPORT. Advisory by nature, but its failures should be readable rather than ambient — two of them (loom optimize smoke, measure_corpus) are real and were invisible behind the cancellations. 3. `run_verification.py` skips `draft` artifacts. A draft is unfinished BY DECLARATION; running its steps can only fail, and a gate red because planned work is planned cannot be told apart from a gate red because something broke. Verified locally: `LOOM_VERIFY_BACKEND=both cargo test --release --lib -p loom-core --features verification -- test_verify_all_rules` passes. NOT claimed fixed: the gate has not yet gone green in CI. That is the only evidence that counts here, and it arrives on this PR. Making the differential job REQUIRED on main is deliberately left out — that changes merge policy and is the repository owner's call, not a side effect of a CI edit. It is the third step #353 asks for. Refs #353, #332 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RZof1M5HMBSYVZPeoT4MEn
This PR now also addresses #353 — including my own contribution to it#353 landed while this was open: the Verification Gate has 0 successful runs in 60 across six weeks, every run over its 30-minute cap (floor 30.4 min), on a I should own the part of that which is mine. I merged past this check repeatedly, calling it systemic alongside Rocq, including for the v1.4.1 release — whose two gate runs were cancelled at 30.5 and 39.2 minutes. #332 was a check that could not go red; this is a check that cannot go green. Same defect class, and treating it as ambient noise is how it survived six weeks. I also made it measurably worseThe filter matches every Worse, one of them —
Three changes
Not claimed fixedThe gate has not yet gone green in CI. That is the only evidence that counts, and it arrives on this PR — which is the same standard I applied to the ARM build here. Deliberately not done: making the differential job required on |
…ld not build With the gate finally running to completion (#353), it reports what six weeks of cancellations hid: 23 passed, 5 failed, 7 skipped. THREE of the five failures are artifacts I marked `verified` this week. [FAIL] TEST-332-VERIFY-REPORTS-NO-VACUOUS-SUCCESS [FAIL] TEST-331-VERIFICATION-COVERAGE-IS-REPORTED [FAIL] TEST-346-OUTPUT-IS-VALIDATED-BEFORE-IT-IS-WRITTEN Not flaky, and not the tests: the gate builds with `RUSTFLAGS: -D warnings`, and three long-standing `unused_variable` warnings in `maybe_differential_gate` are hard errors there. `loom-cli` does not compile in that environment, so every artifact whose evidence is a `cargo test -p loom-cli` invocation fails. I saw those warnings earlier this week and wrote them off as "pre-existing in the feature-gated maybe_differential_gate". They were pre-existing. They were also fatal in the one environment that had not completed a run since July — so the artifacts asserted `verified` on evidence that could not execute. That is the same defect as #332 and #346 wearing the traceability layer's clothes, and I filed it myself three times in one week. The parameters are read only inside the `differential` cfg block, so they are genuinely unused without that feature; they are now `allow(unused_variables)` under exactly that cfg rather than silenced unconditionally. Verified under the gate's own conditions rather than a normal build: `RUSTFLAGS="-D warnings" cargo test --release -p loom-cli` -> 21/21. Refs #353, #332, #331, #346 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RZof1M5HMBSYVZPeoT4MEn
Every step ran with stdout and stderr on DEVNULL, so a failing artifact reported `✗ failed: <command>` and nothing else. That is most of why six weeks of this gate taught nobody anything (#353). The runs that were not cancelled produced a verdict with no evidence attached, so the only route to a cause was reproducing by hand — and this environment differs from a developer machine in precisely the ways that matter: `RUSTFLAGS: -D warnings`, a different libz3, a different core count and therefore different test parallelism. I hit that wall myself just now. TEST-331 fails here and passes locally, including under the gate's own `-D warnings` and at 8 test threads across 25 consecutive runs. I had a plausible mechanism — `take_revert_summary` clears the whole global counter map while a sibling test reads it — and no way to confirm it, because the gate discards the assertion that would say. Rather than commit a fix for a cause I had guessed at, this makes the gate produce the evidence. Failing steps now echo the tail of their combined output (40 lines, enough for a Rust test summary plus the assertion that produced it, short enough that a dozen failures do not bury the run). A gate that says a thing failed without saying how is a weaker form of the same defect as a gate that cannot fail at all. Refs #353 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RZof1M5HMBSYVZPeoT4MEn
v1.4.1 shipped four binaries — x86_64-linux, both macOS arches, Windows. ARM Linux had none: servers, arm64 containers, ARM CI runners and SBCs all had to build from source with a C++ toolchain, which is the dependency #313 exists to remove.
The stated blocker no longer applies
release.ymlcarried this note:True when written, and now moot. The target builds natively on a GitHub-hosted arm64 Linux runner — nothing is cross-compiled.
z3-sysvendors and statically compiles z3, so it needs no hostlibz3on any runner, ARM included, and those runners are free for public repositories, which this one is.Where the evidence lives is the load-bearing decision
release.ymlonly fires on a tag push. A target added there alone would ship on the strength of a job that had never run — the unexercised-gate shape this repo keeps finding, and precisely what v1.4.1 was cut to punish.So the target is also built on every PR via the CI matrix. A toolchain or
z3-sysregression on ARM now fails a check immediately, instead of surfacing during a release when it is most expensive.This PR's own CI is that evidence. If the
ubuntu-22.04-armbuild is not green here, the release entry does not belong on main.Both #311 gates extended, not left x86_64-only
A shipped target whose floor is only hoped-for is exactly how v1.2.0 went out unloadable.
Status is
draft, deliberatelyTEST-AARCH64-LINUX-RELEASE-TARGETis filed against v1.5.0 asdraft, notverified. The workflow edit is not evidence. It flips when the ARM job has actually run green — and the release gate correctly reads✗ NOT cuttableuntil then.Refs #311, #313