You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Split out of #428, which is being closed — its core ask is done, this is the adjacent point that was never in scope for it.
What's left
workbench/runs/ and workbench/reports/ are gitignored, and only three baselines are committed (consent.json, describe-project.json, wizard-enhance.json). Every live measurement made against this bench — including the ones behind #431, #432, #440, #452, #453, #454 and #456 — leaves no trace anyone else can replay without re-running it themselves, on their own key, against whatever the provider resolves that day.
Concretely, that means:
A cassette recorded during a PR review (e.g. workbench/cassettes/judge-migration-*.json) replays offline for whoever has the file, but nothing forces it to reach main — it's a byproduct of one session, not a tracked measurement.
The three real-take scenarios (real-wizard-enhance, real-zooms, real-zoom-grounding) have never been judged at all, because the take itself (workbench/fixtures/) is gitignored and absent from every clone. See removeModifier deletes a whole pill but reports removing one id #457's sibling problem and workbench/README.md § "La prise réelle" for what that costs on a fresh checkout (44 L0 tests fail on the missing fixture, silently, since CI never runs the bench).
A baseline's behaviour/dsl numbers, once written, are never re-verified against anything — nothing reads them back (baseline.ts only compares check ids), so they drift into being an unlabelled archive, which is exactly what happened to wizard-enhance.json before fix(workbench): retire beh.sandbox from the baseline, confirmed at n=10 #453 said so explicitly.
First step, as noted when this was flagged
Freeze the real-take scenario baselines (--update-baseline, verified by hand rather than trusted), then capture the provider's usage block in the recording proxy (workbench/lib/cassette.ts) — the token count every context-size argument in this codebase wants is already in the response the recorder sees pass through, uncaptured.
Whether cassettes from a judged PR review should be committed as a matter of course (the way judge-migration-*.json were, by convention rather than by rule) or whether that's noise the repo doesn't want. workbench/README.md doesn't currently say either way.
Split out of #428, which is being closed — its core ask is done, this is the adjacent point that was never in scope for it.
What's left
workbench/runs/andworkbench/reports/are gitignored, and only three baselines are committed (consent.json,describe-project.json,wizard-enhance.json). Every live measurement made against this bench — including the ones behind #431, #432, #440, #452, #453, #454 and #456 — leaves no trace anyone else can replay without re-running it themselves, on their own key, against whatever the provider resolves that day.Concretely, that means:
workbench/cassettes/judge-migration-*.json) replays offline for whoever has the file, but nothing forces it to reachmain— it's a byproduct of one session, not a tracked measurement.real-wizard-enhance,real-zooms,real-zoom-grounding) have never been judged at all, because the take itself (workbench/fixtures/) is gitignored and absent from every clone. See removeModifier deletes a whole pill but reports removing one id #457's sibling problem andworkbench/README.md§ "La prise réelle" for what that costs on a fresh checkout (44 L0 tests fail on the missing fixture, silently, since CI never runs the bench).behaviour/dslnumbers, once written, are never re-verified against anything — nothing reads them back (baseline.tsonly compares check ids), so they drift into being an unlabelled archive, which is exactly what happened towizard-enhance.jsonbefore fix(workbench): retire beh.sandbox from the baseline, confirmed at n=10 #453 said so explicitly.First step, as noted when this was flagged
Freeze the real-take scenario baselines (
--update-baseline, verified by hand rather than trusted), then capture the provider'susageblock in the recording proxy (workbench/lib/cassette.ts) — the token count every context-size argument in this codebase wants is already in the response the recorder sees pass through, uncaptured.Also worth deciding here, not in #428
Whether cassettes from a judged PR review should be committed as a matter of course (the way
judge-migration-*.jsonwere, by convention rather than by rule) or whether that's noise the repo doesn't want.workbench/README.mddoesn't currently say either way.