Skip to content

feat(ratchet): attest the baseline from main — the store a PR cannot write (b, slice 1 of 2) - #766

Merged
EtanHey merged 4 commits into
mainfrom
wt/ratchet-b1-attest-store
Sep 5, 2026
Merged

feat(ratchet): attest the baseline from main — the store a PR cannot write (b, slice 1 of 2)#766
EtanHey merged 4 commits into
mainfrom
wt/ratchet-b1-attest-store

Conversation

@EtanHey

@EtanHey EtanHey commented Sep 5, 2026

Copy link
Copy Markdown
Owner

Summary — ratchet bolt (b), slice 1 of 2: the store a PR cannot write

Ratchet (a) (#761) made the table honest about which commit it measured. Bolt (b) makes it honest about what it measures against. This slice builds the reference; slice 2 (stacked on this branch) adds the row that compares against it.

Until now a PR could edit tests/fixtures/sprint_gate/corpus.json and every comparison — scripts/sprint_gate.py's latency/CPU/WAL gates and the table's own Notes column — read the number the PR had just written. After this slice, a reference exists that a PR cannot edit:

  • .github/workflows/ratchet-attest.yml runs on every push to main (and on a no-input workflow_dispatch, so an expired artifact can be refreshed without a commit). It runs the same collector as a PR, with --attest-out, and publishes attestation.json as the ratchet-attestation artifact (90-day retention — GitHub's ceiling — stated in the workflow comment along with what happens when it lapses).
  • The schema (attestation_payload): run_id, run_attempt, main_sha, measured_at, the baseline fields (queries, latency_baseline_ms, thresholds), their canonical-JSON sha256, a measured map of baseline paths this run measured, and every row's status/value/method plus a numeric measurement where one exists (today: signature_valid counts, when the macOS job ran — this is the per-run series the (c) seat reads for margins).
  • The writer refuses to attest a commit it does not have checked out (--main-shagit rev-parse HEAD), refuses when any row is RED (a finding comes first; the previous attestation stands), and refuses a partial set of attest flags. A PR invocation with no attest flags can never produce an artifact by accident.
  • The reader (read_attestation) fails closed on every malformed shape: unparseable, not an object, wrong schema, dishonest run numbers, short sha, configuration keys inside baseline, a digest that is not the digest of its own baseline, measured not an object.

Why an artifact of a main run, and not a file in the tree

A pull_request run holds contents: read and cannot upload into another run's artifacts. So a PR that edits the baseline cannot also edit what slice 2 compares it against. Boundary stated exactly, per #761's reviewer: the comparator (ci_ratchet_table.py) still runs from the PR's checkout on a PR — diff-reviewable, not tamper-proof. The docstring says this and nothing wider.

Configuration is not baseline

socket_path, process_patterns, machine_target, mcp_timeout_seconds, required_processes, version are ordinary config and stay ordinary review. BASELINE_FIELDS names the three that are the reference; queries is in because the queries define what the latency baseline measured.

Premise check — a correction to the brief, carried into slice 2

measured is {} on every attestation today, honestly: no runner-side collector measures any baseline field (mapped bytes, search p50/p95, idle CPU are fixture-bound n/a on every hosted runner; signature_valid is not a baseline field). Slice 2's consequence: a PR cannot legitimately move a baseline field until something measures it. The key to that lock is a collector, not a hand.

Stacked slices (canon rule 9)

  • This PR (S, 320 non-test lines): writer workflow + schema + reader + baseline helpers. Mergeable alone; it changes nothing a PR sees yet.
  • Slice 2 (S, ~358 non-test lines), branch wt/ratchet-b2-attest-row, base = this branch: the baseline attestation row, the ratchet.yml fetch step (actions: read, API-listed run, artifact bound to run id + main sha), bootstrap via git against the base tip, and the definition-of-done tests that fail on 3ee7c279.
  • Merge order: this PR first, without deleting the branch until slice 2 is retargeted to main (stacked-PR trap in /pr-loop).

Test plan

  • ruff check and ruff format --check on scripts/ and tests/
  • tests/test_ci_ratchet_table.py: 177 passed (33 new: writer, reader, baseline helpers, and pyyaml assertions on the new workflow — triggers, contents: read only, no cancel-in-progress, stamp step identical to ratchet.yml's, artifact name/path/retention, persist-credentials: false, no scratch in the checkout, no always())
  • python scripts/ci_ratchet_table.py --help renders the new group
  • Live, after merge: the first ratchet-attest.yml run on main publishes the ratchet-attestation artifact for the merge sha (slice 2's row is what reads it back)

— brainlayerClaude (worker) · claude-code/claude-fable-5-1


Note

Medium Risk
New CI on main must succeed to publish attestations; invalid attest CLI usage now exits 1, and artifact expiry or failed attest runs will block baseline comparison until slice 2’s consumer behavior lands or dispatch refreshes the reference.

Overview
Introduces ratchet bolt (b) slice 1: a baseline reference PRs cannot edit in-tree. On every main push (and no-input workflow_dispatch), ratchet-attest.yml builds the wheel like PR ratchet, runs ci_ratchet_table.py with attestation flags, and uploads ratchet-attestation (attestation.json, 90-day retention) capturing the canonical baseline slice of corpus.json (queries, latency_baseline_ms, thresholds), a SHA-256 digest, run metadata, and per-row measurements.

ci_ratchet_table.py gains attestation write mode (--attest-out plus matching --main-sha, --run-id, --run-attempt): refuses partial flags, checkout mismatch, incomplete baseline, or any RED row; otherwise emits schema v1 JSON including optional numeric measurement for signature_valid. read_attestation and baseline helpers (baseline_view, digest, tuple-keyed baseline_changes) validate artifact shape fail-closed for a follow-up PR row. Ordinary PR invocations without attest flags are unchanged; ratchet.yml is not wired to fetch the artifact yet (stacked slice 2).

Reviewed by Cursor Bugbot for commit d02750d. Bugbot is set up for automated code reviews on this repo. Configure here.

Note

Add ratchet-attest workflow to build baseline attestation on main

  • Adds a GitHub Actions workflow that runs on pushes to main and manual dispatch, but only executes when the ref is main.
  • Checks out the commit without persisted credentials, installs Python 3.13, builds the wheel, and invokes the ratchet collector with attestation metadata.
  • Appends the collector report and JSON to the step summary and uploads a ratchet-attestation artifact with 90-day retention.
  • Uses read-only contents permission and a non-cancelling concurrency group keyed by the commit SHA.
  • Behavioral Change: collector refusal or failure prevents the attestation artifact from being uploaded.

Macroscope summarized d02750d.

…write (b, slice 1 of 2)

A new ratchet-attest.yml runs on every push to main (and a no-input
workflow_dispatch), runs the same collector with --attest-out, and publishes
what that main commit's baseline IS (the queries / latency_baseline_ms /
thresholds fields of corpus.json, with a canonical sha256) plus every value the
run measured, as the `ratchet-attestation` artifact. A pull_request run holds
contents: read and cannot upload into another run's artifacts, which is what
makes the artifact a reference a PR cannot edit. The writer refuses to attest a
commit it does not have checked out and refuses when any row is RED.

This slice is the writer, the schema and its reader (`read_attestation`, which
fails closed on every malformed shape). The row that compares a PR's baseline
against the artifact, and the ratchet.yml step that fetches it, are slice 2.

Co-Authored-By: brainlayerClaude running claude-fable-5-1 <noreply@anthropic.com>
@EtanHey EtanHey added the S Small change label Sep 5, 2026
@cursor

cursor Bot commented Sep 5, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_eed08e62-2401-4b0c-bfe4-820e0695550c)

@coderabbitai

coderabbitai Bot commented Sep 5, 2026

Copy link
Copy Markdown

Warning

Review limit reached

Next included review available in 18 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Team

Run ID: a3cdfa80-5fa4-405d-b6ed-a5bf54163cda

📥 Commits

Reviewing files that changed from the base of the PR and between 3ee7c27 and d02750d.

📒 Files selected for processing (3)
  • .github/workflows/ratchet-attest.yml
  • scripts/ci_ratchet_table.py
  • tests/test_ci_ratchet_table.py

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actions Bot commented Sep 5, 2026

Copy link
Copy Markdown

BrainLayer ratchet

Every Value below was measured by this run. A row this machine cannot measure says n/a — <reason> instead of a number; baselines in Notes name their own machine, method and date and were not measured here.

Row Status Value (measured by this run) Method Notes
commit provenance 🟢 GREEN measured d02750d06d1f == PR head · checkout 5509d99f35af commit graph + live PR head · in-process · runner Which commit this whole table is about. On a pull_request event the checkout is GitHub's synthetic merge ref, whose sha is not on the PR — #759's table printed 13fa724278bf while that PR's head was 4632f979 — so this row names the PR-head parent instead, the sha a reviewer can actually see. The comparison sha is read live from repos/{owner}/{repo}/pulls/{n} when the table is collected, not taken from the event payload, because the payload cannot know the run has been overtaken. Residual window, stated rather than papered over: a push landing between that read and the comment being posted is not caught here — the run for that push refreshes the table.
provenance 🟢 GREEN stamped 5509d99f35af == HEAD, tree clean wheel stamp · in-process · runner Sha half of #749 keg-mode provenance: a keg built from this wheel can answer __build_sha__. The helper-age and served-process predicates need a running BrainBar and are measured only by scripts/sprint_gate.py on an installed Mac. The sha here is the checkout's — the merge ref on a PR — because that is what publish.yml stamps at release time; the PR-head sha this table describes is the one in commit provenance above.
mapped bytes ⚪ n/a n/a — no BrainBar daemon at /tmp/brainbar.sock: this row needs the daemon, its hybrid helper and the indexed corpus running together, and no GitHub-hosted runner has them (macOS included) — only a self-hosted Darwin/arm64 runner on an installed Mac would socket · installed Mac Baseline 26.2 GB — installed Mac, socket, 2026-09-03, after R2 drained 15,070 → 0. Up from 16.8 GB because the drain left more vectors mapped under the same cap: the change is the drain, not a leak. Not measured by this run.
search p50/p95 ⚪ n/a n/a — no BrainBar daemon at /tmp/brainbar.sock: this row needs the daemon, its hybrid helper and the indexed corpus running together, and no GitHub-hosted runner has them (macOS included) — only a self-hosted Darwin/arm64 runner on an installed Mac would socket · installed Mac Baseline p50 911.887 ms / p95 3167.985 ms, captured 2026-09-01T08:42:22Z on MacBook-Pro.local under active_sprint_load (tests/fixtures/sprint_gate/corpus.json). Not measured by this run.
idle CPU ⚪ n/a n/a — no BrainBar daemon at /tmp/brainbar.sock: this row needs the daemon, its hybrid helper and the indexed corpus running together, and no GitHub-hosted runner has them (macOS included) — only a self-hosted Darwin/arm64 runner on an installed Mac would ps sampling · installed Mac Budget: average CPU < 30% over a 60 s window (resource_budget in scripts/sprint_gate.py). Needs the BrainBar daemon, helper and watcher actually running. Not measured by this run.
signature_valid ⚪ n/a n/a — the macOS signature-parity job is trigger-gated and did not run on this PR: it touches no release or signing path (pyproject.toml, scripts/release-*, scripts/brainlayer-version-check.sh, publish.yml, ratchet.yml) and carries no ratchet:signatures label — a GitHub macOS runner bills at ~10× Linux minutes and rebuilds the keg venv from source codesign · installed keg scripts/release-verify-signatures.sh <keg> codesign-verifies every *.so/*.dylib under libexec/venv. The macOS parity job installs the published tap formula (etanhey/layers/brainlayer), so this row measures the release path — formula, published sdist and Homebrew's relocation — and not this PR's tree. Release-time baseline for the same keg on a different machine: 442 valid / 0 invalid — installed Mac (M4 Max), brew --prefix brainlayer 1.5.11, 2026-09-03.

🟢 GREEN measured, within budget · 🔴 RED measured, out of budget — a finding to clear before merge · ⚪ n/a not measurable on this machine, never guessed.

No RED rows.

Measured on Linux/x86_64 · measured d02750d06d1f · PR head d02750d06d1f · checkout 5509d99f35af · run · updated 2026-09-05 11:53:04 UTC

@EtanHey

EtanHey commented Sep 5, 2026

Copy link
Copy Markdown
Owner Author

@coderabbitai review

— brainlayerClaude (worker) · claude-code/claude-fable-5-1

@coderabbitai

coderabbitai Bot commented Sep 5, 2026

Copy link
Copy Markdown

@EtanHey I will review the changes in #766.

⚠️ Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@deepsource-io

deepsource-io Bot commented Sep 5, 2026

Copy link
Copy Markdown

DeepSource Code Review

We reviewed changes in 3ee7c27...d02750d on this pull request. Below is the summary for the review, and you can see the individual issues we found as inline review comments.

See full review on DeepSource ↗

PR Report Card

Overall Grade   Security  

Reliability  

Complexity  

Hygiene  

Code Review Summary

Analyzer Status Updated (UTC) Details
Python Sep 5, 2026 11:52a.m. Review ↗
Swift Sep 5, 2026 11:52a.m. Review ↗
JavaScript Sep 5, 2026 11:52a.m. Review ↗
Shell Sep 5, 2026 11:52a.m. Review ↗
Secrets Sep 5, 2026 11:52a.m. Review ↗

Important

AI Review is run only on demand for your team. We're only showing results of static analysis review right now. To trigger AI Review, comment @deepsourcebot review on this thread.

Comment thread scripts/ci_ratchet_table.py Outdated
Comment thread scripts/ci_ratchet_table.py
Comment thread scripts/ci_ratchet_table.py
…(Macroscope, #766)

A removed null-valued leaf is a change (path membership, not `.get()`); the
writer refuses run numbers the reader would reject and a checkout whose
baseline lacks a field; the reader requires exactly the baseline fields, so an
incomplete reference cannot pass as one.

Co-Authored-By: brainlayerClaude running claude-fable-5-1 <noreply@anthropic.com>
@cursor

cursor Bot commented Sep 5, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_5163813f-b08a-437b-baf8-6e9357defbd1)

Comment thread scripts/ci_ratchet_table.py Outdated
}


def attest_refusal(args: argparse.Namespace, probe: Probe, rows: list[Row], corpus: dict) -> str | None:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

`attest_refusal` has a cyclomatic complexity of 16 with "high" risk


A function with high cyclomatic complexity can be hard to understand and
maintain. Cyclomatic complexity is a software metric that measures the number of
independent paths through a function. A higher cyclomatic complexity indicates
that the function has more decision points and is more complex.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Split in cf2f28d: attest_flags_problem (the four flags as one claim: presence, honest run numbers, 40-hex sha) and attest_checkout_problem (is this checkout allowed to stand behind the claim: sha == HEAD, every baseline field present, no RED row), with attest_refusal reduced to the routing — the same shape select_signature already has in this file, and for the same reason: adding a rule must not keep pushing one function's branch count up.

— brainlayerClaude (worker) · claude-code/claude-fable-5-1

…ource, #766)

Flags first, checkout second, the same shape select_signature already has, so
adding a rule does not keep pushing one function's branch count up.

Co-Authored-By: brainlayerClaude running claude-fable-5-1 <noreply@anthropic.com>
@cursor

cursor Bot commented Sep 5, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_e3a72500-4274-4df2-a686-e3fa80a93a53)

… a nested one (Macroscope High, #767)

Co-Authored-By: brainlayerClaude running claude-fable-5-1 <noreply@anthropic.com>
@cursor

cursor Bot commented Sep 5, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_18ada1c3-2a35-4547-8c2c-91377e9f1372)

@EtanHey

EtanHey commented Sep 5, 2026

Copy link
Copy Markdown
Owner Author

Review round 1 (lead's Cursor pass) — all four items were already closed at 66d19270, so this is the line-by-line at the current head d02750d0, not a refutation of the findings themselves (they were real; Macroscope found the same three at 11:35 and the fixes landed at 11:4x, before the pass was written up).

  • high, baseline_changes null ↔ missing: scripts/ci_ratchet_table.py:244if path not in before or path not in after: reports the vanished/appeared leaf by membership before any .get(). Test: tests/test_ci_ratchet_table.py:1782 test_a_removed_null_leaf_is_still_a_change pins both directions and the null-stays-null non-change.
  • medium, incomplete baseline accepted: reader scripts/ci_ratchet_table.py:304set(baseline) != set(BASELINE_FIELDS) rejects a subset (including {}) and a superset alike; writer scripts/ci_ratchet_table.py:1116 refuses a checkout whose baseline lacks a field. Tests: :1793 and :1811. baseline_view deliberately stays tolerant: a KeyError there would cost the whole table on a PR run, which this file forbids — the refusal is at the two ends instead.
  • medium, writer never applies honest_run_number: scripts/ci_ratchet_table.py:1103, in attest_flags_problem (the validator was split for DeepSource's complexity finding at cf2f28d3). Test: :1801, parametrized over --run-id 0, --run-id -4, --run-attempt 0.
  • low, coverage: the three tests above are exactly the three cases named.

Also since round 1 was cut: Macroscope High on #767 (a dotted JSON key aliased a nested path in the flattener and could empty the diff) — fixed at the helper's home here, d02750d0: paths are tuples, dotted() only prints; test_a_dotted_key_cannot_alias_a_nested_path. #767 carries the row-side half.

Head for round 2: d02750d0. 184 tests.

— brainlayerClaude (worker) · claude-code/claude-fable-5-1

@EtanHey
EtanHey merged commit ceeccfd into main Sep 5, 2026
18 checks passed
@EtanHey
EtanHey deleted the wt/ratchet-b1-attest-store branch September 5, 2026 13:04
EtanHey added a commit that referenced this pull request Sep 5, 2026
… against a number the PR wrote (b, slice 2 of 2) (#767)

* feat(ratchet): attest the baseline from main — the store a PR cannot write (b, slice 1 of 2)

A new ratchet-attest.yml runs on every push to main (and a no-input
workflow_dispatch), runs the same collector with --attest-out, and publishes
what that main commit's baseline IS (the queries / latency_baseline_ms /
thresholds fields of corpus.json, with a canonical sha256) plus every value the
run measured, as the `ratchet-attestation` artifact. A pull_request run holds
contents: read and cannot upload into another run's artifacts, which is what
makes the artifact a reference a PR cannot edit. The writer refuses to attest a
commit it does not have checked out and refuses when any row is RED.

This slice is the writer, the schema and its reader (`read_attestation`, which
fails closed on every malformed shape). The row that compares a PR's baseline
against the artifact, and the ratchet.yml step that fetches it, are slice 2.

Co-Authored-By: brainlayerClaude running claude-fable-5-1 <noreply@anthropic.com>

* feat(ratchet): a baseline-attestation row, so the gate cannot compare against a number the PR wrote (b, slice 2 of 2)

The collector's new second row, `baseline attestation`, diffs the PR's baseline
fields against the `ratchet-attestation` artifact of the latest successful
push/dispatch run of ratchet-attest.yml on main, which ratchet.yml now lists
through the Actions API (`actions: read`), downloads, and binds to the run the
API named (run id AND main sha must match). Unchanged is GREEN; a changed
field is RED "baseline changed by hand; no CI attestation for these values"
unless that main run measured the new value. Bootstrap -- no successful attest
run on main at all -- compares against the base tip's committed fixture via
`git show`, merge-ref only, and says so; once one success exists an expired
artifact is RED, never a fall-back. Boundary stated, not overclaimed: the
inputs are outside the PR tree, the comparator is not.

Co-Authored-By: brainlayerClaude running claude-fable-5-1 <noreply@anthropic.com>

* fix(ratchet): writer and reader agree on what an attestation must be (Macroscope, #766)

A removed null-valued leaf is a change (path membership, not `.get()`); the
writer refuses run numbers the reader would reject and a checkout whose
baseline lacks a field; the reader requires exactly the baseline fields, so an
incomplete reference cannot pass as one.

Co-Authored-By: brainlayerClaude running claude-fable-5-1 <noreply@anthropic.com>

* refactor(ratchet): split attest_refusal into its two questions (DeepSource, #766)

Flags first, checkout second, the same shape select_signature already has, so
adding a rule does not keep pushing one function's branch count up.

Co-Authored-By: brainlayerClaude running claude-fable-5-1 <noreply@anthropic.com>

* fix(ratchet): baseline paths are tuples, so a dotted key cannot alias a nested one (Macroscope High, #767)

Co-Authored-By: brainlayerClaude running claude-fable-5-1 <noreply@anthropic.com>

* fix(ratchet): an unparseable artifact is handed over as unresolved, proven by executing the step (Cursor, #767 round 1)

The two jq reads that bind the artifact to its run are guarded: under set -e
an unparseable artifact killed the fetch step before the hand-off was written,
so the row never rendered its promised RED-with-reason and Collect died on a
missing args file. The fetch step and the Collect guard are now EXECUTED in
tests under the machine's bash 3.2 with a stubbed gh on PATH: newest success
across pages, bind mismatch, corrupt JSON, missing file, expired artifact,
404 bootstrap, never-succeeded bootstrap, and a missing or empty hand-off into
Collect. Notes name both writer events; the attested notice names the run
instead of a stale initial reason.

Co-Authored-By: brainlayerClaude running claude-fable-5-1 <noreply@anthropic.com>

* fix(ratchet): a stale attestation is checked against the base commit too, and names no direction (Macroscope, #767)

When the attestation is not for this checkout's base, the row no longer says
main "moved" (a sha inequality is not an ordering). It reads the base tip's
committed baseline via git and judges the PR against THAT: a field reverted to
the attested-but-superseded value is a hand edit, RED; a PR carrying what main
has at the base is GREEN with the stale artifact named, not re-applied. The
executable harness declares check=False explicitly where the exit code is the
thing under test (DeepSource).

Co-Authored-By: brainlayerClaude running claude-fable-5-1 <noreply@anthropic.com>

* fix(ratchet): the run-list reads are guarded too, and an empty listing is not a bootstrap (Cursor, #767 round 2)

Same class as round 1, one layer up: a 200 from `gh api` with a non-JSON body
made the listing jq exit non-zero and killed the step under set -e before any
hand-off was written. Both listing reads are guarded now; garbage is one failed
attempt and three of them are `unresolved`, never bootstrap. Writing the test
found a second shape: an EMPTY 200 body slurps to `[]` and read as "0 runs,
bootstrap" -- the exact fall-back garbage must never reach -- so an empty body
is unreadable too. The git-read base fixture must carry every baseline field,
the shape the writer and reader already enforce. All executed, not grepped.

Co-Authored-By: brainlayerClaude running claude-fable-5-1 <noreply@anthropic.com>

---------

Co-authored-by: brainlayerClaude running claude-fable-5-1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

S Small change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant