Skip to content

ci: pause the never-green conflict-gate schedule; keep the wip-limit-check tombstone (live caller on claude-skills main) - #451

Merged
LukasWodka merged 3 commits into
developfrom
ci/pause-conflict-gate-and-remove-wip-tombstone
Sep 9, 2026
Merged

ci: pause the never-green conflict-gate schedule; keep the wip-limit-check tombstone (live caller on claude-skills main)#451
LukasWodka merged 3 commits into
developfrom
ci/pause-conflict-gate-and-remove-wip-tombstone

Conversation

@LukasWodka

@LukasWodka LukasWodka commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Summary

Two dead-weight items in this repo's workflows; this PR ships the first and explains why the second stays.

(a) conflict-gate.yml: the schedule: trigger is paused (commented out), workflow_dispatch kept.
Every scheduled run since the workflow's first on 2026-08-27 has failed at the step "Mint an installation token" and never reached "Sweep" — re-measured today on the runs API: 534 runs, 0 successes, latest 34324927165 (2026-09-09T07:38Z, step 5 failure, step 6 skipped). The mint asks permission-statuses: write and the release-train App's installation does not grant statuses, so GitHub answers HTTP 422 (The level of access for permissions requested are not granted to this installation). That is ~48 red runs a day marking nothing: a gate that has never been green is advice nobody reads, and a cron that cannot start is not visibility. The mint and its permission request are deliberately not changed — they state what the job needs; the fix is on the App.

(b) wip-limit-check.yml tombstone: NOT deleted. The probe below found a live uses: caller still on claude-skills main, so the reusable, its reusables: inventory row and shared_reasons.wip_limit_check_has_no_callers all stay. See "Probe" for the location and the evidence.

Admin action needed to re-arm the conflict gate

  1. Org admin: GitHub App settings for the release-train App → Permissions → grant Commit statuses: Read and write; then accept the new permission on the org installation.
  2. Dispatch Conflict gate once (workflow_dispatch) and confirm the run reaches the "Sweep" step.
  3. Uncomment the two schedule: lines in .github/workflows/conflict-gate.yml and update the two paused-state assertions in scripts/tests/conflict-gate-selftest.py (plus the two workflow mutation rows) in the same PR — the selftest pins the pause on purpose so re-arming is a deliberate edit, not a drift.

What changed

  • .github/workflows/conflict-gate.ymlschedule: block commented out with the reason, the re-arm procedure and the original cadence rationale kept in place; workflow_dispatch: {} unchanged; mint/permissions unchanged.
  • scripts/tests/conflict-gate-selftest.py — the assertion "has a schedule trigger" becomes two: no live schedule trigger while paused, and the commented # schedule: / # - cron: block still present (so "paused" cannot quietly become "deleted"). workflow_dispatch and "not pull_request" assertions unchanged.
  • scripts/tests/conflict-gate-mutations.py — the two anchors that matched the live schedule: block would have gone stale (the runner fails on a stale anchor); replaced with one mutation per direction: re-arming the cron early (caught by the first new assertion) and deleting the commented block (caught by the second). The pull_request-trigger mutation is re-anchored on workflow_dispatch: {}.

Probe: is any wip-limit-check caller still live?

Method: for every repo in repo-inventory.yml and each of develop / staging / main, list .github/workflows via the contents API, fetch every workflow file and grep its content for wip-limit-check (content, not filename). Plus gh search code --owner tracebloc wip-limit-check for default branches.

repo develop staging main
.github mention only: fr-pass-comment.yml, kanban-columns.yml, wip-limit-check.yml mention only: fr-pass-comment.yml, kanban-columns.yml, wip-limit-check.yml mention only: fr-pass-comment.yml, kanban-columns.yml, wip-limit-check.yml
averaging-service clean clean clean
backend clean clean clean
design-system-v2 clean clean clean
claude-skills clean no branch CALLER: wip-limit-check.yml
cli clean clean clean
client clean clean clean
client-runtime clean clean clean
data-ingestors clean clean clean
design-system clean clean clean
docs clean clean clean
e2e-test-agent clean clean clean
frontend-app clean clean clean
model-zoo clean clean clean
release-train no branch no branch clean
rfcs clean no branch clean
start-training clean clean clean
tracebloc-engine clean clean clean
tracebloc-py-package clean clean clean
tracebloc-website clean clean clean
  • 60 cells: 52 clean, 4 no such branch, 4 with a mention. The three .github cells are this repo's own file, a comment in fr-pass-comment.yml and a paths: filter in kanban-columns.yml — not callers.
  • claude-skills main is a real caller: .github/workflows/wip-limit-check.yml, on: pull_request (opened, ready_for_review), jobs.check.uses: tracebloc/.github/.github/workflows/wip-limit-check.yml@main, last touched by the 2026-07-28 promotion PR chore: promote advance-deploy-env fix to main #12. claude-skills is off the train (release_train: false, default branch develop, main currently 15 commits ahead of / 45 behind develop, 0 open PRs into main), so nothing promotes the removal there — a PR opened against main would invoke the reusable and go red the moment it is deleted. Removing that caller file from claude-skills main is a one-file PR in that repo; once it lands the tombstone can be deleted here.
  • gh search code (default branches only): hits are this repo's docs/scripts/inventory and two prose mentions in rfcs — no caller.

Verification

  1. Probe: table above (60 cells, content-grepped).
  2. actionlint .github/workflows/conflict-gate.yml (1.7.12): clean. make actionlint over all 41 workflows: 0 findings.
  3. make check (= lint + selftests), with one environment-only red: selftest-pipefail-early-close reports 167 passed / 1 failed, the failure being gawk is not installed, so the CI awk cannot be checked here — this Mac has only BSD awk; ubuntu-latest has gawk. Reproduced identically on a pristine origin/develop checkout with the same invocation, so it is not caused by this change; CI is the authority for that case. Everything else in check: ruff clean, shellcheck clean, house-rules 0 findings, action-pins 0 findings, mint-scope 20 mints / no findings, reusable-no-cancel 0 findings, lint-targets-run-in-ci clean, mutations-dry all anchors resolve, selftests-cover: all 30 selftests and 17 mutation runners wired; every selftest passed (conflict-gate: 116 assertions).
  4. make mutation-conflict-gate (the real tier, on the committed tree): 30 mutations across 2 files, 0 stale, 0 uncaught — the two new workflow mutations are each caught by the assertion written for them.
  5. make reason-citations against the live inventory: 32 distinct citations across 132 written reasons, 15 dead / 15 exempted, 0 findings (every citation live, every exemption still applies).
  6. python3 -c 'import yaml; yaml.safe_load(open("repo-inventory.yml"))': loads (20 repos, 13 reusables). The inventory is not modified by this PR, so a caller-drift.py live audit was not run — its only modes are the live audit and --create-prs, and there is no inventory change for it to evaluate.

Not verified / out of scope

  • The App's granted permission set is not readable with a non-admin token; the statuses gap is inferred from the 422 text and from no other workflow in this repo requesting permission-statuses.
  • Whether the release-train App can be granted statuses without side effects on its other consumers — admin decision.
  • Deleting the claude-skills main caller (separate repo, separate PR) and, after that, the tombstone here.
  • The bricked-prs and blocked-gate items remain decisions, untouched.

Test plan

  • make check locally (system python + PyYAML 6.0.3)
  • make mutation-conflict-gate locally
  • actionlint on the changed workflow
  • selftests (required, 19 shards) green on this PR
  • Bugbot clean on HEAD

🤖 Generated with Claude Code


Note

Low Risk
Disables a cron that never succeeded; manual dispatch and gate script behavior are unchanged, with tests enforcing the pause.

Overview
Pauses the conflict-gate cron because every scheduled run has failed at GitHub App token mint (HTTP 422): the release-train App installation does not grant checks: write (or legacy statuses). The live schedule: trigger is commented out; workflow_dispatch stays so admins can still run sweeps manually after fixing App permissions.

The workflow documents why the pause happened, how to re-arm (grant Checks read/write, verify Sweep runs, uncomment cron + update tests), and clarifies the mint step still requests permission-checks: write on purpose.

Tests pin the paused state: conflict-gate-selftest.py no longer requires an active schedule trigger; it asserts no live schedule and that the commented # schedule: / cron block remains. conflict-gate-mutations.py updates anchors and adds mutations that catch premature re-arming or deleting the commented schedule block.

Reviewed by Cursor Bugbot for commit 3f33ce8. Bugbot is set up for automated code reviews on this repo. Configure here.

Every scheduled run of conflict-gate.yml since its first on 2026-08-27 --
534 runs, 0 successes, re-measured today -- fails at "Mint an installation
token" and never reaches the Sweep step: the mint asks
`permission-statuses: write`, the release-train App's installation does not
grant `statuses`, and GitHub answers HTTP 422. ~48 red runs a day that mark
nothing is a gate nobody reads, so the `schedule:` trigger is commented out
and `workflow_dispatch` kept. The mint and its permission request are left
untouched: they state what the job needs, and the fix is on the App.

To re-arm, an org admin grants the App `statuses: write`, confirms one
dispatched run reaches Sweep, and uncomments the two lines.

The selftest pinned `schedule` as present; it now pins the pause from both
sides (no live `schedule` trigger; the commented block still present so the
pause cannot silently become a removal), and the two mutation anchors that
matched the live block are replaced by one per direction.

The wip-limit-check tombstone stays: a live `uses:` caller for it is still
present on claude-skills `main`, so deleting the reusable would turn that
repo's next PR into `main` red.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@LukasWodka LukasWodka self-assigned this Sep 9, 2026
@LukasWodka
LukasWodka requested a review from saadqbal September 9, 2026 08:04
@LukasWodka

Copy link
Copy Markdown
Contributor Author

bugbot run

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@saadqbal saadqbal left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The pause itself is the right call and the test work is the best part of this — swapping one assertion for two that pin the pause from both directions, each with its own mutation, is exactly right. But the branch is eight minutes too old and it matters.

#446 ("write a check run, not a commit status, so the gate can actually mint its token", backend#3242) landed on develop at 07:56 UTC and this PR opened at 08:04. I checked: 7ccaf30 is not an ancestor of f5724ea6 — the two have diverged, head is 1 behind. So on develop the mint already asks permission-checks: write (line 125), while this branch's new on: comment says "The mint below asks permission-statuses: write" and its own mint line still reads permission-statuses: write. Merged, the file carries both claims sixty lines apart, and nothing in CI catches it because the selftests assert on the parsed triggers rather than the prose.

The TO RE-ARM procedure is the part I'd most want fixed, because it sends the one person who can unblock this after the wrong grant. A scheduled run already executed with #446 in — run 34333719757 at 09:16Z — and it requested permission-checks: write and still died at step 5:

Failed to create token for "tracebloc": The level of access for
permissions requested are not granted to this installation
status: 422

Steps 1-4 succeeded, the mint failed, Sweep skipped. So the installation grants neither statuses nor checks. An admin who follows the comment, grants Commit statuses, dispatches, and watches it 422 again learns only that the runbook is wrong.

Rebase on develop, swap statuseschecks through the new comment and the matching paragraph in the selftest, and record that the post-#446 mint still 422s so the grant actually needed is Checks: Read and write.

One small thing while you're in there: the new block has a date and a reason but no ticket, unlike every other incident paragraph in this file (backend#2637, backend#3242, client#847). A backend#NNNN would give the pause somewhere to be closed out from.

Everything else checks out — schedule commented rather than deleted, workflow_dispatch intact, no pull_request trigger, the context isn't a required check on any base I looked at so there's no brick risk, and wip-limit-check.yml is genuinely untouched.

LukasWodka and others added 2 commits September 9, 2026 12:24
…tuses, and the pause has a ticket

Review follow-up on the schedule pause. The `on:` comment said the mint asks
`permission-statuses: write`; since #446 it asks `permission-checks: write`,
so merged the file carried both claims sixty lines apart. Aligned the prose
and the selftest paragraph that quotes it.

The TO RE-ARM procedure also pointed the admin at the wrong grant. Scheduled
run 34333719757 (09:16Z, head 7ccaf30, i.e. with #446 in) requested exactly
`permission-pull-requests: read` + `permission-checks: write`, steps 1-4
succeeded, the mint still answered HTTP 422 and Sweep was skipped -- so the
installation grants neither `statuses` nor `checks`. The procedure now names
**Checks: Read and write** on the App installation, requires one dispatched
run to reach Sweep before the schedule is uncommented, and says outright
that granting Commit statuses does nothing. #446's "`checks` the App does
hold" sentence is corrected the same way so the file agrees with itself.

The incident paragraph now carries its ticket like the others in this file.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@LukasWodka

Copy link
Copy Markdown
Contributor Author

Thanks — all four points addressed on head 3f33ce8 (merge of develop at 20f7325 + one prose commit; no rebase, no force-push).

1. Behind develop. Merged origin/develop into the branch (62900ce). Git resolved it clean: #446's permission-checks: write mint and this PR's commented-out schedule: block both survive. The selftest's mint assertions from #446 (checks: write present, permission-statuses absent) and this PR's two pause assertions now run in the same file.

2. Prose. The on: comment no longer says the mint asks permission-statuses: write. It now states the sequence: the mint asked statuses until #446, asks checks since, and 422s either way. The selftest paragraph and its check name ("…while the App lacks checks: write"), and the mutation name ("…still cannot write check runs"), say the same thing. I also corrected #446's own sentence sixty lines down ("checks the App does hold") since that is what run 34333719757 refutes — otherwise the file would still contradict itself.

3. TO RE-ARM. Verified your reading independently: run 34333719757 is a schedule event on head 7ccaf30, the mint step requested exactly permission-pull-requests: read + permission-checks: write, steps 1–4 succeeded, step 5 failed with HTTP 422 "The level of access for permissions requested are not granted to this installation", Sweep skipped. The comment now records that run, states the installation grants neither statuses nor checks, names the grant as Checks: Read and write on the App installation (and that the org installation has to accept it), requires one workflow_dispatch run to reach Sweep — not merely past the mint — before the schedule is uncommented, and says outright that granting Commit statuses does nothing.

4. Ticket. The incident paragraph now carries its ticket in the same style as the other paragraphs in the file.

Verification: actionlint clean; YAML parses with on: [workflow_dispatch]; selftest-conflict-gate 127 assertions pass; mutation-conflict-gate 31 mutations, 0 stale, 0 uncaught (both pause mutations still caught by their renamed checks). Remaining statuses mentions in the workflow are all deliberate contrast/history, none describes the current mint.

@LukasWodka

Copy link
Copy Markdown
Contributor Author

bugbot run

@LukasWodka
LukasWodka requested a review from saadqbal September 9, 2026 10:30

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 3f33ce8. Configure here.

@saadqbal saadqbal left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both halves fixed, and the rebase is real — 7ccaf307 is an ancestor now, so the on: prose and the mint at line 157 finally tell the same story. The only permission-statuses strings left are the historical ones ("asked … until backend#3242", "was REFUSED"), which is right.

The runbook is better than what I asked for. It names Checks: Read and write, cites run 34333719757 as proof the post-#446 mint still 422s so nobody assumes the scope is already there, requires the dispatched run to reach Sweep rather than just clear the mint, and adds that the org installation has to accept the new permission. "Granting Commit statuses does nothing: the mint no longer asks for it" is exactly the wrong turn I was worried an admin would take.

The suite was run at this head: 127 assertions pass, and the mutation harness is 31 mutations, 0 stale, 0 uncaught — both new mutations caught by the assertions that name them, which is the part I care about after a develop merge. on: parses to just workflow_dispatch, the commented schedule and its cadence rationale are intact, and wip-limit-check.yml is byte-identical to develop.

Nice work on the ticket too. ~48 failed runs a day stop today.

@LukasWodka
LukasWodka merged commit 45577a5 into develop Sep 9, 2026
40 checks passed
@LukasWodka
LukasWodka deleted the ci/pause-conflict-gate-and-remove-wip-tombstone branch September 9, 2026 10:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants