ci(release): run release.yml on release-surface PRs; rehearse the PyPI publish; add opt-in TestPyPI leg (#342, #350) by dean0x · Pull Request #366 · dean0x/mdscript

dean0x · 2026-09-06T21:08:31Z

Summary

release.yml now runs on release-surface pull requests (#342): PRs touching .github/workflows/release.yml, .github/actions/**, crates/mds-napi/**, crates/mds-python/**, or scripts/verify-napi-names.mjs trigger the workflow via pull_request so action-pin bumps are exercised on the PR rather than first on a tag push after crates.io has published irreversibly.

A new unguarded rehearse-publish-python job (Rehearse PyPI publish (no upload)) rehearses the PyPI publish without uploading (#350): it gates on pin shape, GHCR manifest reachability, docker pull, and twine check of all 8 distributions — each with a positive control — then blocks publish-crates so a broken pin aborts before any irreversible registry write (PF-023).

An opt-in publish-testpypi leg runs on workflow_dispatch -f testpypi=true, exercising the full OIDC token exchange and upload handshake against TestPyPI once per version. scripts/verify-pr-checks.mjs gains a release-surface presence check (D-PR7) and a fifth tolerated skipped name (Publish to TestPyPI (rehearsal)).

The release surface

The five paths: that define the release surface are:

.github/workflows/release.yml
.github/actions/**
crates/mds-napi/**
crates/mds-python/**
scripts/verify-napi-names.mjs

crates/mds-core/**, Cargo.toml, and package.json are excluded: they change on most PRs, ci.yml already covers them, and a dependency sweep still needs the manual gh workflow run release.yml dispatch to exercise the release gates. Narrowing the paths prevents release.yml from running on every ordinary PR.

Publish jobs stay dark on PRs: the refs/tags/v guard evaluates to false for refs/pull/N/merge, so all five guarded jobs (Publish to crates.io, Publish to npm, Publish to PyPI, GitHub Release, Publish to TestPyPI (rehearsal)) report skipped. The verifier enumerates those names in TIER_B_EXPECTED_SKIPPED and tolerates skipped conclusions for exactly those five names.

Rehearsal job design

rehearse-publish-python runs four gates, each with a PF-013 positive control (a gate never observed rejecting anything is not evidence):

Pin shape — the pypa/gh-action-pypi-publish uses-pin must match vX.Y.Z exactly (one distinct ref across the file). Control: the image-backed commit dc37677b2e1c63e2034f94d8a5b11f265b73ba33 (HTTP 200 on GHCR) and the annotated tag object a892a5a61159132606e93a2fa6f4358831b04d26 (the v0.4.1 failure; 404 on GHCR) are both rejected by the shape gate — policy, not existence.
Anonymous GHCR manifest probe — curl checks that ghcr.io/v2/pypa/gh-action-pypi-publish/manifests/<PIN_REF> returns HTTP 200. Control: a ref that cannot exist (v9.99.99) must return 404 with MANIFEST_UNKNOWN first, then the real pin must return 200.
docker pull — the image must actually pull on the runner. Control: a missing tag must fail; bounded to 3 attempts with a 5-second back-off.
twine check — the image's own twine runs against all 8 distributions with docker run --rm --network none --entrypoint twine, so the step physically cannot reach pypi.org. The step first prints the image's twine version (--version) so the log captures which twine validated the distributions. Control: a deliberately corrupt wheel is injected and must be rejected by twine.

publish-crates depends on rehearse-publish-python (in addition to stage-and-verify-napi and build-python), so an unresolvable or un-pullable image aborts the release before crates.io writes anything irreversible (PF-023).

What the rehearsal cannot prove: the upload handshake and trusted-publisher OIDC exchange at publish time — pypa/gh-action-pypi-publish has no dry-run/no-upload mode; a dry-run: true input is silently ignored and the action uploads for real. The credential half is covered by version-gate's OIDC probe (mint-token exchange with pypi.org, which runs on PRs). The upload half is covered by the opt-in TestPyPI leg.

TestPyPI leg

To exercise the full OIDC token exchange and upload handshake once per version:

gh workflow run release.yml --ref <branch> -f testpypi=true

Publish to TestPyPI (rehearsal) is skipped on pull_request, plain workflow_dispatch, and tag push — it only runs when inputs.testpypi == true. skip-existing: true makes repeats at the same version no-ops, so the handshake is exercised once per version without re-uploading.

User-owned prerequisite: a TestPyPI trusted publisher at test.pypi.org/manage/account/publishing:

Project name: markdown-script
Owner / repository: dean0x/mdscript
Workflow filename: release.yml
Environment name: leave blank

A pending publisher expires ~30 days unless a first upload lands. If the pending publisher is missing or expired, Publish to TestPyPI (rehearsal) fails at the OIDC token exchange with an invalid-publisher message — re-file the pending publisher (environment name BLANK) and re-dispatch; no code change needed.

Verifier changes (ADR-013 amendment)

TIER_B_EXPECTED_SKIPPED now has five names (adds Publish to TestPyPI (rehearsal) alongside Publish to crates.io, Publish to npm, Publish to PyPI, GitHub Release).

New D-PR7 release-surface presence check: when a PR touches the release surface, Version gate, Stage + verify platform packages, and Rehearse PyPI publish (no upload) must each appear in the check-runs and every run under them must be completed + success. Changed files are fetched with bounded pagination (per_page=100, max 10 pages) and asserted against a RELEASE_SURFACE set that must match the on.pull_request.paths filter exactly (spec S10). API errors (non-200 responses, missing changed_files) fail closed with exit 2.

GHCR correction

The previous comment claimed GHCR only holds images for tagged releases. The verified rule is: images exist for release tags (e.g. v1.14.2), the vX and vX.Y aliases, and commit SHAs on upstream release/* branches — but never for annotated tag objects. A 40-hex SHA is visually indistinguishable between a commit (200) and an annotated tag object (404), which is why the policy requires a vX.Y.Z release tag pin.

Probe table (checked via curl -s -o /dev/null -w "%{http_code}" -H "Authorization: Bearer <anon-token>" https://ghcr.io/v2/pypa/gh-action-pypi-publish/manifests/<ref>):

v1.14.2 → 200
dc37677b2e1c63e2034f94d8a5b11f265b73ba33 (the v1.14.2 commit) → 200
a892a5a61159132606e93a2fa6f4358831b04d26 (annotated tag object; v0.4.1 failure) → 404
v9.99.99 (impossible ref) → 404

Dependabot and fork PRs

pull_request runs from Dependabot or forks have a read-only GITHUB_TOKEN, no repository secrets, and no id-token: write grant. Version gate fails closed at the npm-token guard with the "No Actions secrets on this run" error (and the PyPI OIDC probe also fails closed). Do not merge on those checks — the failed gate is correct behavior, not a flake. Supersede with a maintainer-authored PR on a first-party branch, or dispatch by hand with gh workflow run release.yml --ref <branch>.

The CI-history gate is step-skipped on pull_request with a ::notice:: (because github.sha is the ephemeral merge commit, not the branch head). Both credential probes do run on maintainer PRs and have been verified passing in run 34063664485.

Verification

Local:

npm run test:gates → 187 pass / 0 fail
npx js-yaml .github/workflows/release.yml → ok
node scripts/verify-no-control-bytes.mjs → 552 files clean
node scripts/verify-pr-checks.mjs 353 → PASS, "release surface not touched"
node scripts/verify-pr-checks.mjs 365 → exit 1 with the D-PR7 message (expected: chore(ci): bump checkout v7, setup-node v7, setup-python v7; normalize maturin-action pin (PF-040) #365 edited release.yml pre-gate — before this gate existed — making it a live positive control for the exit-1 path)

Live PR-triggered release runs:

34060146952 (d993916) — FAILED in the rehearsal: the original job invoked pypa/gh-action-pypi-publish with a nonexistent dry-run: true input and attempted a real upload (HTTP 400 from upload.pypi.org; pypi.org still lists only 0.4.2). The gate caught its own defect.
34061583304 (df9ae24) — zero-job workflow-parse rejection ("An expression was expected": a literal empty expression inside a bash comment; now guarded by spec S19).
34063664485 (c7f68b5) — all non-guarded jobs success, five guarded jobs skipped. Rehearsal log: Positive control OK: missing manifests are detected., GHCR manifest present for ghcr.io/pypa/gh-action-pypi-publish:v1.14.2, Positive control OK: docker pull rejects a missing tag., digest sha256:5c2f7030fbef8308068eb4cc9080fd3c9e157ccf6d511924d69f8f4b23dc95c1, positive control OK: twine check rejected the corrupt wheel, twine check OK for 8 distribution(s) — nothing was uploaded. Version-gate: CI-history step skipped, notice printed, npm token resolves to dean0x, PyPI trusted publisher OK.
34064572849 (3a803f5) — all non-guarded jobs success; Publish to TestPyPI (rehearsal), Publish to crates.io, Publish to npm, Publish to PyPI, GitHub Release skipped; ci.yml run 34064572796 64/64 success.

Plain dispatch (gh workflow run release.yml --ref ci/release-surface-gate):

34064856623 — Version gate success with the "Assert tagged SHA has green CI history (PF-017)" step EXECUTED (CI history OK for 3a803f59ca3e79c67740061e91d217756889fbee) and the pull_request notice step skipped; rehearsal success; TestPyPI + four publish jobs skipped.

-f testpypi=true dispatch:

34065036182 — every job success except Publish to TestPyPI (rehearsal) = failure at the OIDC exchange: Trusted publishing exchange failure: invalid-publisher: valid token, but no corresponding publisher (Publisher with matching claims was not found). This is the documented user-owned prerequisite (the TestPyPI trusted publisher for markdown-script / dean0x/mdscript / release.yml / environment blank has not been created yet), not a code defect. Nothing was uploaded: test.pypi.org still 404; pypi.org still lists only 0.4.2. Follow-up after merge: create the publisher, then gh workflow run release.yml --ref main -f testpypi=true.

Positive controls on draft PR #367 (closed unmerged, branch deleted):

Control B (pin @dc37677b…, image-backed commit sha) — run 34064235589: Version gate FAILED in the gate-spec suite (S16 rejects a non-vX.Y.Z pin), so the rehearsal was skipped and publish-crates skipped — the policy is enforced at two layers (spec suite in version-gate, runtime Gate 1 in the rehearsal whose in-step controls run on every green run).
Control A (pin @v9.99.99) — run 34064404996: Version gate success, rehearsal FAILED at Gate 2 after Positive control OK: missing manifests are detected. with GHCR has no image for ghcr.io/pypa/gh-action-pypi-publish:v9.99.99 (HTTP 404) … the v0.4.1 failure (#350, PF-040); Gates 3/4 skipped; publish-crates skipped.
Verifier control: node scripts/verify-pr-checks.mjs 367 → exit 1 with Tier B: "Rehearse PyPI publish (no upload)" — conclusion=failure and the D-PR7 line.

QA: 13 scenario acceptance tests PASS (verifier on #353 → PASS/not touched; on #365 → exit 1 D-PR7; matchesReleaseSurface positives/negatives; fail-closed IS_FORK block simulated for 4 cases; live logs; docs; scope). Two cosmetic findings fixed in this commit: twine --version line added to Gate 4 log; comment rewording so grep -cE 'uses: pypa/gh-action-pypi-publish@' returns 2 (not 3).

34065573775 (e02bcf2) — all non-guarded jobs success, five guarded skipped; rehearsal printed twine 7.0.0.

node scripts/verify-pr-checks.mjs 366 → exit 0, ✓ PASS — all 15 required contexts completed+success, merge command: gh pr merge 366 --squash --admin --match-head-commit e02bcf280dc50bb8df032744aa2a2520c02865ee.

Snyk status

Snyk MCP unavailable locally (ENOENT); the security/snyk (dean0x) PR check is the scan of record. This PR changes YAML, first-party Node scripts, and Markdown only; no new native dependencies are introduced.

Related Issues

Closes #342
Closes #350
#351 unchanged

…I publish; add opt-in TestPyPI leg (#342, #350) - pull_request trigger on 5 paths: release.yml, .github/actions/**, crates/mds-napi/**, crates/mds-python/**, scripts/verify-napi-names.mjs - rehearse-publish-python job: dry-run OIDC exchange, unguarded (PF-039), blocks publish-crates - publish-testpypi job: opt-in via workflow_dispatch testpypi:true, in TIER_B_EXPECTED_SKIPPED - version-gate: GHCR manifest probe for pypa/gh-action-pypi-publish pin (PF-040); step-level if: guard on CI-history step so version-gate runs (not skips) on PRs (ADR-013 amendment) - verify-pr-checks.mjs: RELEASE_SURFACE/RELEASE_SURFACE_CONTEXTS/matchesReleaseSurface exports; D-PR6 block requires Rehearse PyPI publish (no upload) on release-surface PRs; bounded PR-files pagination with graceful degradation - 13 new spec tests (S5-S13 in release-auth-probe.spec.mjs, D-PR5f + D-PR6a-g in verify-pr-checks.spec.mjs); Snyk: security/snyk (dean0x) is the scan of record (Snyk MCP unavailable this session)

The D-PR6 changed-files fetch was an inline block scope with four mutable let variables (filesFailed, filesPagError, filesPage, collected) and a confusing name collision — the loop data variable was also named `page`, shadowing the loop iterator `filesPage`. Extract fetchChangedFiles() following the same Result-return pattern as fetchCheckRuns() and fetchStatuses(): returns { ok: true, files } on success, { ok: 'skip', notice } on API error (graceful degradation), and { ok: false, exitCode: 2, message } on pagination overflow or non-vacuity mismatch. main() dispatches on filesResult.ok with explicit equality checks. No behaviour change. 172/0 pass/fail.

…l closed on the PR-files API P0 (Functionality/Security, PF-039): rehearse-publish-python invoked `pypa/gh-action-pypi-publish@v1.14.2` with `dry-run: true`. That input does not exist in v1.14.2 — the runner logged "Unexpected input(s) 'dry-run'", the action ignored it and performed a REAL upload to https://upload.pypi.org/legacy/ from a pull_request (run 34060146952). It failed only because the workspace version (0.4.2) was already on PyPI; on a version-bump PR — which touches crates/mds-python/Cargo.toml and therefore matches the release-surface paths filter — the upload would have SUCCEEDED, publishing an unreleased version from an unmerged branch. pypi.org still lists 0.4.2 only; nothing landed. The action has no no-upload mode, so the rehearsal now REPRODUCES what it does instead of calling it: pin shape, GHCR manifest, docker pull, and `twine check` run from the pinned image with `--network none --entrypoint twine`. Each gate carries a positive control (PF-013): the v0.4.1 annotated-tag-object SHA must be rejected, a ref that cannot exist must not resolve or pull, and a corrupt wheel must fail twine. The job is denied id-token, so it holds no credential to upload with even if a step is re-introduced. The duplicate GHCR probe in version-gate is removed — publish-crates needs the rehearsal, so a bad pin still aborts before the irreversible crates.io write (PF-023, PF-040). P1 (Error Handling, PF-013): fetchChangedFiles returned `{ ok: 'skip' }` on an API error and main() then skipped the release-surface presence check — a transient 500 would let a release-surface PR pass the merge gate with no release check-runs. It now fails closed (exit 2), matching fetchCheckRuns/fetchStatuses/ fetchRequiredContexts; the tri-state is gone. Only the exported pure function's `changedFiles === undefined` caller-path still skips, with a notice. P1 (Functionality, ADR-013 amendment): the credential and PyPI OIDC probes are skipped at STEP level on pull_request. Fork and Dependabot PRs receive no repository secrets and no id-token, so both probes hard-failed version-gate — on exactly the Dependabot-bump PR class #350 exists to catch (PF-039 amendment 2026-09-05), and a version-gate that cannot reach success hard-fails the mandatory pre-merge verifier, whose natural workaround is bypassing it (PF-017). P1 (Tests): S14 pins the invocation to publish-python/publish-testpypi only (with a planted positive control and a commented-out counter-control), S15 the absent id-token, S16 PIN_REF/uses drift, S17 the four gates and their controls. S8 follows the probe to the rehearsal. S10 now compares on.pull_request.paths to RELEASE_SURFACE as SETS — the invariant the production comment already claimed but nothing enforced. S13 gains a positive control. D-PR6h drives the fail-closed path end-to-end through main() (verified failing against the old behaviour), D-PR6i pins the prefix/exact-match boundaries, D-PR6j pins all-must-pass for duplicate names. S18 pins every RELEASE_SURFACE_CONTEXTS name to a real job name: in release.yml — a renamed job would otherwise make the verifier demand a check-run that can never appear, hard-failing every release-surface PR (ADR-013 amendment, PF-017). npm run test:gates: 183 tests, 183 pass, 0 fail (was 172).

… probes, fail closed on fork/Dependabot PRs, decouple the rehearsal from the napi matrix, harden gate controls (#342, #350) M1: Remove step-level if: from credential and PyPI OIDC probe steps in version-gate (both now run on every event, including pull_request). Remove the notice step that announced the skipping. M2: Add IS_FORK env + fail-closed block before the npm-token guard. Fork and Dependabot pull_request events receive no repository secrets and no id-token: write; the probe exits 1 with an actionable error pointing to `gh workflow run release.yml --ref <branch>` as the remedy. Removes the now-incorrect NOTE suffix on the NPM_TOKEN error. (Applying ADR-013) M3: rehearse-publish-python: needs [build-python] only (decoupled from the napi matrix); job-level if conditioned on build-python success. M4+M5+M6: publish-testpypi: needs [build-python, rehearse-publish-python]; if guarded by workflow_dispatch + inputs.testpypi + both upstreams; new "Assert all 8 Python distributions are present" step before publish. M7: Gate 1 — add dc37677b2e1c63e2034f94d8a5b11f265b73ba33 (v1.14.2 commit SHA) as second positive control. Update error message to explain why GHCR has images for commit shas AND release tags but never annotated tag objects. Fix publish-python comment (no longer a blanket "GHCR only publishes images for tagged releases" claim). M8: publish-python: add rehearse-publish-python to needs and if condition. M9/S6: rehearse-publish-python job-level if now tested as a non-tag-guarded condition on build-python success (was: asserting if is null). M10a/b/c: Three new ADR-013 graph tests — cargo-publish jobs transitively need rehearse-publish-python; publish-crates does NOT transitively need publish-testpypi; guarded job names exactly equal TIER_B_EXPECTED_SKIPPED. M11: Gate 2 — GHCR probe now writes body to /tmp/ghcr-body.json; positive control checks HTTP 404 AND MANIFEST_UNKNOWN body text (not just non-200). M12: Gate 3 — positive control uses grep for "manifest unknown" rather than bare exit code; docker pull retries up to 3 times with sleep 10 between attempts. M13: Extend file header comment to describe the paths: list as the release surface. M14: RELEASING.md — update probes description (run on all events, fail closed on forks); TestPyPI publisher expiry note; update "What happens after tagging" already-applied items cross-checked. M15: CHANGELOG.md [Unreleased] entry updated to reflect aligned behaviour. M16: rename D-PR6 presence-check to D-PR7 (D-PR6 stays for Exit codes). M17: D-PR5a test updated to "five publish names" (was "four"). P: Fix empty GitHub expression ${{ }} in Gate 3 bash comment — GitHub's expression pre-processor is not comment-aware; it raises "An expression was expected" (run 34061583304, 0 jobs). Rewrite comment to use plain prose. Spec S19 guards against regression. Applying PF-013 (positive controls), PF-039 (unguarded rehearsal), PF-040 (release-tag pin), ADR-013 (three-place rule: guarded jobs, TIER_B_EXPECTED_SKIPPED, verifier required contexts).

…TestPyPI leg; tidy two release.yml comments (#342, #350)

…eword a comment that mimicked a uses: line (#350)

dean0x added 2 commits September 7, 2026 00:07

dean0x force-pushed the ci/release-surface-gate branch from 86ca7af to 7746aad Compare September 6, 2026 21:36

dean0x force-pushed the ci/release-surface-gate branch from 7746aad to df9ae24 Compare September 6, 2026 21:36

dean0x added 2 commits September 7, 2026 01:18

docs(release): document the release-surface gate, PyPI rehearsal and …

3a803f5

…TestPyPI leg; tidy two release.yml comments (#342, #350)

dean0x mentioned this pull request Sep 6, 2026

control: DO NOT MERGE — PF-013 positive controls for the PyPI publish rehearsal gate (#350) #367

Closed

ci(release): print the image's twine version in the PyPI rehearsal; r…

e02bcf2

…eword a comment that mimicked a uses: line (#350)

dean0x merged commit bd91a1f into main Sep 6, 2026
45 checks passed

dean0x deleted the ci/release-surface-gate branch September 6, 2026 23:07

Navigation Menu

ci(release): run release.yml on release-surface PRs; rehearse the PyPI publish; add opt-in TestPyPI leg (#342, #350) - #366

ci(release): run release.yml on release-surface PRs; rehearse the PyPI publish; add opt-in TestPyPI leg (#342, #350)#366
dean0x merged 6 commits into
mainfrom
ci/release-surface-gate

dean0x commented Sep 6, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

dean0x commented Sep 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

The release surface

Rehearsal job design

TestPyPI leg

Verifier changes (ADR-013 amendment)

GHCR correction

Dependabot and fork PRs

Verification

Snyk status

Related Issues

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

dean0x commented Sep 6, 2026 •

edited

Loading