[live-proof only — will be closed] ratchet (b) combined: renders the baseline-attestation row on a real pull_request run - #768
Conversation
… against a number the PR wrote
The reference the table measures against is now the `ratchet-attestation`
artifact of the latest successful push run of ratchet-attest.yml on main, read
through the Actions API and bound to its run, never from the PR tree. A baseline
field that differs is RED ("changed by hand; no CI attestation for these
values") unless that main run measured the new value. Bootstrap (no attest run
on main yet) compares against the base tip via git and says so. Configuration
fields stay ordinary review; only queries / latency_baseline_ms / thresholds
are the baseline. Boundary stated, not overclaimed: inputs are outside the PR
tree, the comparator is not.
Co-Authored-By: brainlayerClaude running claude-fable-5-1 <noreply@anthropic.com>
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: trueThanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
 |
|
Overall Grade |
Security Reliability Complexity Hygiene |
Code Review Summary
| Analyzer | Status | Updated (UTC) | Details |
|---|---|---|---|
| Python |  |
Sep 5, 2026 11:35a.m. | Review ↗ |
| Swift | |
Sep 5, 2026 11:35a.m. | Review ↗ |
| JavaScript | |
Sep 5, 2026 11:35a.m. | Review ↗ |
| Shell | |
Sep 5, 2026 11:35a.m. | Review ↗ |
| Secrets | |
Sep 5, 2026 11:35a.m. | Review ↗ |
Important
AI Review is run only on demand for your team. We're only showing results of static analysis review right now. To trigger AI Review, comment @deepsourcebot review on this thread.
![macroscopeapp[bot]](https://avatars.githubusercontent.com/in/900172?s=60&v=4){kind=small}
|
|
||
| def attest_refusal(args: argparse.Namespace, probe: Probe, rows: list[Row]) -> str | None: | ||
| """Why this run may NOT publish an attestation. None means it may.""" | ||
| flags = (args.attest_out, args.main_sha, args.run_id, args.run_attempt) |
There was a problem hiding this comment.
🟡 Medium scripts/ci_ratchet_table.py:1305
attest_refusal allows --run-id 0 and negative --run-attempt values, so the writer publishes an artifact that read_attestation() rejects as naming no honest run. That leaves every PR's baseline-attestation row RED until another valid attestation replaces it; validate both values with honest_run_number before allowing publication.
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @scripts/ci_ratchet_table.py around line 1305:
`attest_refusal` allows `--run-id 0` and negative `--run-attempt` values, so the writer publishes an artifact that `read_attestation()` rejects as naming no honest run. That leaves every PR's baseline-attestation row `RED` until another valid attestation replaces it; validate both values with `honest_run_number` before allowing publication.
| # The API filters by branch; the event filter is in jq because the endpoint takes one | ||
| # `event` value and two (push, workflow_dispatch) are legitimate writers. | ||
| for attempt in 1 2 3; do | ||
| if gh api "repos/${REPO}/actions/workflows/ratchet-attest.yml/runs?branch=main&exclude_pull_requests=true&per_page=30" > "$runs_path" 2> "$err_path"; then |
There was a problem hiding this comment.
🟡 Medium workflows/ratchet.yml:441
After 30 newer failed or cancelled runs, this request treats the first page as having no successful attestation and emits --attestation-bootstrap, allowing the collector to use a green base-commit fallback even though a successful writer exists on a later page. Paginate the run list and search all returned pages before deciding that bootstrap applies.
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @.github/workflows/ratchet.yml around line 441:
After 30 newer failed or cancelled runs, this request treats the first page as having no successful attestation and emits `--attestation-bootstrap`, allowing the collector to use a green base-commit fallback even though a successful writer exists on a later page. Paginate the run list and search all returned pages before deciding that bootstrap applies.
| ATTESTATION_METHOD, | ||
| ATTESTATION_NOTES, | ||
| ) | ||
| by_hand = [(path, old, new) for path, old, new in changes if attestation.measured.get(path, _UNMEASURED) != new] |
There was a problem hiding this comment.
🟡 Medium scripts/ci_ratchet_table.py:872
A PR can change a measured numeric 0 or 1 to False or True and still receive GREEN, even though the baseline value and type changed. This check uses equality alone, so Python treats 0 == False and 1 == True; include a type comparison here, matching baseline_changes, before authorizing the change.
- by_hand = [(path, old, new) for path, old, new in changes if attestation.measured.get(path, _UNMEASURED) != new]
+ by_hand = [(path, old, new) for path, old, new in changes if (measured := attestation.measured.get(path, _UNMEASURED)) != new or type(measured) is not type(new)]🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @scripts/ci_ratchet_table.py around line 872:
A PR can change a measured numeric `0` or `1` to `False` or `True` and still receive `GREEN`, even though the baseline value and type changed. This check uses equality alone, so Python treats `0 == False` and `1 == True`; include a type comparison here, matching `baseline_changes`, before authorizing the change.
| if probe.attestation is None: | ||
| reason = probe.attestation_unavailable or ATTESTATION_UNAVAILABLE_DEFAULT | ||
| return Row("baseline attestation", NA, f"n/a — {reason}", ATTESTATION_METHOD, ATTESTATION_NOTES) | ||
| return attested_row(probe.attestation, corpus, base_tip_of(probe.head_lineage)) |
There was a problem hiding this comment.
🟡 Medium scripts/ci_ratchet_table.py:951
Deleting an attested baseline field whose value is JSON null renders the baseline attestation row GREEN instead of detecting a hand edit. baseline_changes uses dict.get() for both sides, so a present null and an absent path both become None; compare presence separately (for example, with a sentinel) so missing fields remain distinguishable from measured null values.
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @scripts/ci_ratchet_table.py around line 951:
Deleting an attested baseline field whose value is JSON `null` renders the `baseline attestation` row GREEN instead of detecting a hand edit. `baseline_changes` uses `dict.get()` for both sides, so a present `null` and an absent path both become `None`; compare presence separately (for example, with a sentinel) so missing fields remain distinguishable from measured `null` values.
BrainLayer ratchetEvery Value below was measured by this run. A row this machine cannot measure says
🟢 GREEN measured, within budget · 🔴 RED measured, out of budget — a finding to clear before merge · ⚪ n/a not measurable on this machine, never guessed. No RED rows. Measured on Linux/x86_64 · measured |
|
Rendered. Run 33963754466 posted — brainlayerClaude (worker) · claude-code/claude-fable-5-1 |
1 participant
Not for review or merge. The real PRs are #766 (store) and #767 (row, stacked on #766).
ratchet.ymlonly runs on PRs againstmain, so #767 cannot render its own row until it is retargeted after #766 merges. This draft carries the same code (954e102f) so the row's live output exists before the report claims done. Closed as soon as the table renders.— brainlayerClaude (worker) · claude-code/claude-fable-5-1
Note
Add
baseline-attestationrow to ratchet table using main workflow artifact.github/workflows/ratchet-attest.ymlto publish aratchet-attestationartifact on pushes tomaincontaining the baseline fields (queries,latency_baseline_ms,thresholds), commit SHA, and run identity..github/workflows/ratchet.ymlto fetch the latest successfulmainattestation artifact and pass it to the collector in PR runs.scripts/ci_ratchet_table.pyto evaluate the PR corpus against the attested main baseline. A PR is GREEN when the baseline matches the attestation or changes only to values recorded as measured by the main run.mainruns prevents the artifact from being published.📊 Macroscope summarized 954e102. 3 files reviewed, 6 issues evaluated, 0 issues filtered, 4 comments posted
🗂️ Filtered Issues