fix(cargo-anvil): stop anvil-msrv-test from executing benchmarks

[Evgenii (Vaiz)]

🤖 Clawpilot here! Posted automatically by Clawpilot (an AI agent), not by a human. Please verify before acting.

Two upstream fixes found while adopting cargo-anvil 0.6.0 in microsoft/oxidizer#718. Anvil defects get fixed here rather than worked around downstream, so nothing in oxidizer changes.

1. anvil-msrv-test executed benchmark harnesses

The generated recipe ran cargo test --all-targets, which expands to --lib --bins --tests --benches --examples — so cargo test built and executed every bench target.

A bench declared harness = false delegates its run to a separate driver binary (criterion's, or a profiler runner such as gungraun-runner driving Valgrind). anvil-msrv-test-setup installs only cargo-delta and the MSRV toolchain, so that driver is never present:

--> Error in ae_basic_operations_cg: Failed to run benchmarks: No such file or directory (os error 2).
    Is gungraun-runner installed and gungraun-runner in your $PATH?
error: recipe `anvil-msrv-test` failed with exit code 1

An MSRV check exists to prove the affected packages still compile and their tests still pass under the declared minimum compiler. Executing benchmark harnesses is not part of that signal, it silently requires profiler prerequisites the group never declares, and it breaks the check for any adopter with a harness = false bench.

Fix: use --tests, which selects exactly the targets carrying test = true — lib unit tests, bin unit tests and integration tests.

--tests is preferred over the equivalent --lib --bins --tests because --lib errors with "no library targets found" on a bin-only affected package under impact scoping — the same reason anvil-miri already selects --tests, documented in miri.just.

A compile-only --all-targets --no-run pass was considered and rejected: benches and examples are dev-only targets no consumer compiles, so they do not constrain the crate's real MSRV, and the repository already covers "do they still build" without running them — anvil-bench is cargo bench --no-run, anvil-examples is cargo build --examples. This change makes msrv-test consistent with that existing policy rather than the sole exception to it.

No doctest coverage is lost: --all-targets suppresses doctests too, and anvil-doc-test owns them.

msrv-test.just was the only template with cargo test --all-targets. clippy.just and udeps.just also pass --all-targets, but to cargo clippy and cargo udeps, which compile without executing — both left unchanged. anvil-llvm-cov and the nextest paths are untouched.

The impact.rs contract test now additionally asserts that neither --all-targets nor --benches reaches the MSRV invocations, so this cannot silently regress.

2. Action pin comments overstated what they know

Generated workflows carried:

uses: codecov/codecov-action@v7.0.0 # immutable release, the tag cannot be moved

The claim is true for these specific pins — GitHub's immutable-releases feature does lock the tag to a commit, and the REST API reports "immutable": true for each. But as written it asserts a property of the tag, which reads as a general and false claim that Git tags are stable. Read that way it is also self-undermining: if tags could not be moved, the SHA pins on actions/checkout and actions/download-artifact in the same file would be pointless. On microsoft/oxidizer#718 this produced four separate automated-reviewer findings a human had to rebut one at a time.

Reworded to name the mechanism the pin depends on:

uses: codecov/codecov-action@v7.0.0 # pinned by tag: this release is an immutable release (GitHub locks the tag to a commit)

Applied at all five sites. docs/design/github.md reproduces both the comment and a sample step, so both copies are updated to prevent drift. That section already explained that immutability is per-release and must be re-confirmed on a version bump; added one sentence recording that actions/checkout and actions/download-artifact stay SHA-pinned because immutable releases are opt-in per publisher and theirs have not enabled it — not because those actions are less trusted.

Behaviour unchanged for this half: comments only, same action versions.

Validation

Toolchain 1.97 (repo rust-toolchain.toml), scoped to cargo-anvil: cargo fmt --check, cargo clippy --all-targets --all-features -D warnings, and cargo test --all-features (486 passed) all green. The three emitted-tree snapshots were re-recorded with cargo insta and reviewed; they contain only the intended changes.

No version bump or CHANGELOG entry: this repo generates the CHANGELOG at release time from conventional-commit messages via separate chore(cargo-anvil): bump to X commits.

[Copilot]

🟡 Changes recommended

The repo still ships stale “immutable release” comment text and outdated MSRV/all-targets documentation in other committed/generated files, so regeneration and doc consistency fixes are needed before merge.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR adjusts cargo-anvil’s generated MSRV testing recipe so it no longer executes benchmark targets (switching from cargo test --all-targets to cargo test --tests), and rewords “immutable release” action-pin comments to accurately describe GitHub’s immutable-releases mechanism rather than implying tags are inherently unmovable.

Changes:

• Update the anvil-msrv-test recipe template (and emitted snapshots) to use cargo test --tests for both feature configurations, avoiding bench execution.
• Strengthen the impact.rs contract test to assert MSRV invocations don’t include --all-targets/--benches.
• Reword tag-pin comments for immutable-release actions across GitHub templates and design docs; update user-facing docs (rustdoc/README/design docs) accordingly.

File summaries

File	Description
crates/cargo-anvil/templates/justfiles/anvil/checks/msrv-test.just	Switch MSRV test selection to --tests and document rationale in-template.
crates/cargo-anvil/tests/impact.rs	Update expected MSRV argv and add regression assertions against bench selection.
crates/cargo-anvil/tests/snapshots/snapshots__local_only.snap	Re-record emitted local-only snapshot reflecting --tests MSRV invocation and comments.
crates/cargo-anvil/tests/snapshots/snapshots__github_backend.snap	Re-record GitHub-backend snapshot with updated MSRV recipe and action-pin comment wording.
crates/cargo-anvil/tests/snapshots/snapshots__ado_backend.snap	Re-record ADO-backend snapshot with updated MSRV recipe and comments.
crates/cargo-anvil/templates/github/setup-action.yml	Reword immutable-release pin comment for cargo-binstall in the setup action template.
crates/cargo-anvil/templates/github/pr-impl-workflow.yml	Reword immutable-release pin comments in PR workflow template (sticky-comment + Codecov).
crates/cargo-anvil/templates/github/scheduled-impl-workflow.yml	Reword immutable-release pin comment for Codecov in scheduled workflow template.
crates/cargo-anvil/docs/design/checks.md	Update MSRV check design contract to describe cargo test --tests and why.
crates/cargo-anvil/docs/design/github.md	Clarify the tag pinning rationale/comment wording and immutable-release mechanism.
crates/cargo-anvil/src/lib.rs	Update rustdoc table text to reflect new MSRV-test target scope.
crates/cargo-anvil/README.md	Update generated README table text to match rustdoc/behavior description.

Review details

• Files reviewed: 16/17 changed files
• Comments generated: 5
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Codecov Comments Bot (codecov-commenter)]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.6%. Comparing base (0c68ee2) to head (946afed).
⚠️ Report is 1 commits behind head on main.

❌ Your project status has failed because the head coverage (97.6%) is below the target coverage (100.0%). You can increase the head coverage or adjust the target coverage.

Additional details and impacted files

@@           Coverage Diff           @@
##            main    #150     +/-   ##
=======================================
- Coverage   97.6%   97.6%   -0.1%     
=======================================
  Files        298     298             
  Lines      67202   67202             
=======================================
- Hits       65618   65617      -1     
- Misses      1584    1585      +1     

Flag	Coverage Δ
linux	97.6% <ø> (?)
linux-arm	97.6% <ø> (?)
windows	97.7% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

🟢 Approval recommended

The changes are internally consistent across templates/generated outputs/tests/docs and add a regression guard for the corrected MSRV invocation behavior.

Review details

• Files reviewed: 16/17 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

🟢 Approval recommended

The changes are narrowly scoped, keep behavior aligned with the stated MSRV policy, and add a regression-preventing contract assertion while updating generated artifacts consistently.

Review details

• Files reviewed: 22/23 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

🟢 Approval recommended

The changes directly address the stated CI failure mode, update the contract tests/docs/snapshots consistently, and appear low-risk while improving correctness and documentation clarity.

Review details

• Files reviewed: 22/23 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

🟡 Changes recommended

The MSRV recipe’s new cargo test --tests usage appears to exclude unit tests (running only “test targets”), which weakens MSRV coverage and conflicts with the intended behavior described in the PR and docs.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 22/23 changed files
• Comments generated: 1
• Review effort level: Lite