feat(ce-optimize): long-running optimization stack (tick loop, eval discipline, remote backend) - #1708
Draft
kieranklaassen wants to merge 18 commits into
Draft
feat(ce-optimize): long-running optimization stack (tick loop, eval discipline, remote backend)#1708kieranklaassen wants to merge 18 commits into
kieranklaassen wants to merge 18 commits into
Conversation
tmchow
added this pull request to stack #1713
September 15, 2026 00:04
cursor
Bot
changed the title
cursor
Bot
changed the title
Phase 3 runs as ticks that checkpoint run_state at CP-4. Two capability conditions in the SKILL.md body decide where the ledger lives (a durable state root the harness names, else .context/) and how the loop waits (a wake after turn end, else the in-session loop with a checkpoint hand-back). A turn may end with work outstanding only when every item waits on an event a registered wake will deliver and run_state.pending_waits records it. The Phase 1 approval is recorded in the log bound to the spec digest and caps, so an unattended resume does not re-ask it. Two clocks bound a run: stopping.max_hours is active time inside ticks; the new stopping.max_wall_hours (default 72) is a calendar backstop. loop.md 3.2 becomes one dispatch recipe with backend deltas and a Codex delegation condition. Wrap-up exports the summary to <root>/optimize/<spec>-report.md. Co-authored-by: Kieran Klaassen <kieranklaassen@users.noreply.github.com>
…p report Co-authored-by: Kieran Klaassen <kieranklaassen@users.noreply.github.com>
…xt-target guidance Phase 1 accepts the harness only when it orders known-good above known-bad exemplars and does not reward a trivial shortcut; a judge primary also needs a human-labeled calibration sample (or an explicit waiver) before the run leaves Phase 1. A held-out set (measurement.holdout.command or metric.judge.confirmation_seed) is scored only before a keep and at final confirmation: decide.mjs answers a would-be keep with next_measurement: holdout until the pair is supplied, and a holdout that disagrees withholds the keep. Judge items carry a required feedback string that the digest groups into failure themes for hypothesis generation; identical items are judged once per run via a content-hash cache; per-experiment cost and per-case regressions are logged, report only. New references/text-targets.md and example-text-target-spec.yaml cover instruction-text targets: the eval-set coverage rule, model roles, hypothesis moves, and saturation. The judge model enum becomes capability tiers (cheap|strong) with legacy haiku/sonnet read as aliases. Co-authored-by: Kieran Klaassen <kieranklaassen@users.noreply.github.com>
…idance Co-authored-by: Kieran Klaassen <kieranklaassen@users.noreply.github.com>
Co-authored-by: Kieran Klaassen <kieranklaassen@users.noreply.github.com>
…ment execution.backend: remote dispatches each experiment to a detached worker with its own checkout, on a harness whose dispatch returns a receipt and lands its work as a pushed ref or store file (the third capability condition in the body's Execution Surface; without it the run uses worktree). The worker verifies base_sha, implements inside scope.mutable, measures baseline and candidate paired on its own machine, writes result.yaml, and pushes optimize-exp/<spec>/exp-NNN. The orchestrator accepts a result only when base_sha matches and the range touches only mutable scope, runs decide.mjs on the worker's pair, and before any keep obtains a confirmation pairing the candidate's author did not produce. remote requires a paired or relative comparison; the parallel probe and worktree budget do not apply to it. Log schema gains comparisons[].machine, measured_by, and the result.yaml shape. Co-authored-by: Kieran Klaassen <kieranklaassen@users.noreply.github.com>
…arability Co-authored-by: Kieran Klaassen <kieranklaassen@users.noreply.github.com>
…d spec invalidates decide.mjs ran degenerate gates only on the selection candidate, so a held-out run that failed a gate (a partial or empty run) could still confirm a keep. confirmHoldout now runs the same gates and returns `degenerate`. The approval record said a changed spec digest re-presents the Phase 1 gate but not what happens to the baseline and logged experiments measured under the old spec. It now states the condition: a cap-only change continues the run; any other change goes back through Phase 1 while nothing is derived from the spec, and is otherwise a restore or a fresh start. measurement.md 1.7 carries the matching adjust-spec rule that the body used to state. The harness examples table no longer asserts that Claude Code, Codex, and Cursor CLI sessions have no wake after turn end; the body's rule (presence in the tool list is proof) decides. loop.md states what happens to an unapproved dependency, which its pointer to the body no longer supplied.
…ndary, and capability fallbacks Six read-only, post-only decision cells for the paths the long-runs stack changes: a resume whose approval record matches skips the gate; a cap-only change re-presents it and keeps the measurements; a changed metric is restore-or-fresh; a tick boundary with no wake hands back a checkpoint with the resume invocation; a judge spec without a holdout is invalid; `remote` without a detached worker falls back to worktree without asking. Run post arm on Claude and Codex: 12 of 12 pass. Co-authored-by: Kieran Klaassen <kieranklaassen@users.noreply.github.com>
…dgment The validity gate asked every run, whatever the metric type, for three to five good and bad example outputs, and blocked a judge run in Phase 1 until the user hand-labeled 20-50 items or waived that. For an objective target such as CI time the examples mean nothing, and for most judge runs the labeling is tedium that stops the run from starting. Human input is now conditioned on the score being a judgment. An objectively measured primary asks for nothing; the agent builds the shortcut probe itself. A judgment primary uses the good and bad outputs the project already has and asks only for what is missing. Judge calibration is optional: without labels the approval gate says the judge is unchecked against a person. A judge run that continues unattended between ticks still needs labels or an explicit waiver. A probe with no inputs is recorded as not run rather than asked for. The holdout stays required for judge and unattended runs; it costs no human work. The approval record now treats a change that leaves how the logged measurements were produced untouched (a cap, or a holdout or calibration added to a spec that lacked one) as a re-approval that continues the run, so an in-flight judge run survives adding the holdout this stack requires. Eval cells added for both directions and the affected resume cells re-run, post arm on Claude and Codex: 14 of 14 pass. Co-authored-by: Kieran Klaassen <kieranklaassen@users.noreply.github.com>
…the labeling work for the user The approval gate carried a sentence about an unchecked judge that an agent repeated to the user verbatim, and it read as jargon. The skill now gives direction instead of wording: say in everyday words what skipping the check means for the user, for someone who has never heard the word calibration, without reusing the skill's phrasing. Hand-labeling no longer comes up during spec setup. The run assumes the user wants automation. Once the baseline has been judged, when real outputs and the judge's scores already exist, the agent offers the check once with its reason, and continuing without it is the easy answer. A user who accepts is shown one output at a time with the rubric's scale as the choices, scores it before seeing the judge's score, and never edits a file: the agent writes the labels, compares them with the scores it already has at no extra judge cost, and reports agreement and the clearest disagreements. About ten items is enough and the user can stop at any point. An unattended judge run gets the same flow as an ask. Eval cells for the offer and the accepted flow added; the attended-judge cell no longer pins the disclosure wording. Post arm on Claude and Codex: all pass, messages read manually.
A small throwaway repo with one wasteful function, an immutable measurement that counts comparisons rather than reading a clock, a correctness gate, and a reviewed spec capped at two serial experiments. The cell runs the skill for real and grades what is left on disk: the approval record and run state in the log, the committed wrap-up report, and a clean tree. Run on Claude and Codex. Full run: both measured the 92,922 baseline, recorded an approval whose digest matches spec.yaml, dispatched a worker in a worktree, kept the win on the optimization branch, committed the report, and marked the log final, asking for no exemplars. Two-turn resume (turn one stops after one experiment; turn two uses that workspace as the fixture): both resumed without re-asking approval or re-measuring the baseline, and after an edit to the metric in the saved spec both set the run to blocked and offered restore-or-fresh without touching the branch.
… script removes Wrap-up told the agent to run `rm -f "<state-root>/strategy-digest.md"` with a path it fills in itself. No delete is portable across hosts, a filled-in placeholder is the fragile part, and nothing needs the digest gone: the state root is small, untracked scratch the user removes when done, and a resume reads it. Wrap-up now deletes nothing there; in a live run a host whose user forbids `rm` had already declined the step. experiment-worktree.sh built every path it deletes from an unchecked spec name and unchecked shared paths. `create` replaces a shared directory inside the new worktree with `rm -rf`, so a shared path of `../..` resolved to the repository root. The script now rejects a spec name that is not lowercase kebab-case and a shared path that is absolute or contains `.` or `..`, before composing any path, and the fallback used when `git worktree remove` fails only removes a direct, non-symlink child of `.worktrees` that the script named. Tested against a throwaway repo.
…not a command The previous commit removed wrap-up's `rm -f` and told the agent to delete nothing, which leaves a finished run's scratch files behind. Wrap-up now states the goal and its limits instead of a command: once the log is final, remove the working files this run created under the state root (digest, judge cache, result markers, scratch payloads), keep the log and the spec and say where they are, and keep everything when the run is only waiting or blocked, because a resume reads it. The agent removes files the way its host and the user's rules prefer, such as the trash over a permanent delete, only for files it can name that this run created, never from a pattern or an unresolved placeholder, and leaves anything it is unsure about in place and names it. Worktrees still go only through the bundled script. Checked live by resuming two parked runs through wrap-up on Claude and Codex: both left only the log and the spec, both moved each file to the trash by its full name, and neither ran a permanent delete.
…the rubric being approved Three gaps from a real interactive session with a person at the keyboard. The spec approval asked whether the rubric matched what the user meant by clearer without showing the rubric, which sat in a file the terminal truncated. The approval message now carries whatever the user is asked to judge: for a judge primary, the rubric's scale level by level. The agent reported the baseline as 3/5 on every item and then offered to check the judge against the user's own scores, so the user had seen the judge's scores before giving theirs. The offer now comes before any judge score is reported, the baseline mean included. Scoring ran through the host's question picker, which holds four choices: a 1-5 scale was split into four choices plus "choose Other and type 1", with 5 preselected, and the paragraph to judge was cramped into the question. Scoring now happens in ordinary chat: the scale stated once, outputs shown two or three to a message, numbered and quoted in full, and the user replies with a score per number and an optional reason in any shape. The question tool stays for the yes-or-no offer. The two eval cells now also grade that the offer message carries no judge scores and that scores are collected by chat reply. Post arm on Claude and Codex: pass, messages read manually.
…a short question Seen in live interactive sessions. With the rubric required in the approval, the agent skipped the chat message and packed the scale, scope, and limits into the question text, which the picker rendered as one bold block. And when it offered to check the judge against the user's own scores, it went straight to the question tool with no message before it, so the reason for the check was lost in both live sessions while both headless runs had given it. The spec approval and the baseline approval now send their material as an ordinary message laid out to be read (the scale as a list, scope and limits on their own lines) and then ask; the question is one short sentence and its options. The judge-check offer is an ordinary message of a sentence or two before any question, and the reason goes in that message rather than only in an option's description. The rule belongs in the always-loaded body, which has 51 bytes of room under the 8000-byte Codex budget; moving a block out of the body to make room is follow-up work, so the rule is stated at the approval points. New eval cell grades that the rubric levels are in the message and the question text is one sentence. Post arm on Claude and Codex: pass. A live session after the change laid the spec approval out as a readable message followed by a one-sentence question.
… asks for it In a live session the agent dispatched the held-out judging in the same batch as the selection judging. The body tells it to send independent calls together, and loop.md said only "collect it then", so the holdout read as independent work. It is not: it depends on the selection decision. Scored early, it spends judge budget on candidates that would have been rejected, and it puts held-out feedback in front of the agent before it writes the next hypothesis, which is what the holdout exists to prevent. Over a long unattended run that erodes the one protection against fitting the selection sample. loop.md now states the dependency and both reasons. New eval cell grades the dispatch decision; post arm on Claude and Codex: pass. The pre arm was not run: the evidence that the old text failed is the live session.
…c stands for An immutable harness stops the metric from being changed, not from being gamed (arXiv 2609.12039): a candidate can score by skipping work the harness never observes, or by breaking a spec `constraints` entry the harness does not check. The optimize-live eval fixture shows the case: a Set-based dedupe that never calls count() scores 0 comparisons, passes correctness, and decide.mjs says keep. - loop.md 3.4 KEEP: before committing a winner, read its diff against the spec's description and constraints (a gain past the opportunity upper bound is the cue to read slowly); a gain from what the harness does not observe or from an unchecked constraint is reverted, the missing observation is recorded in `learnings`, and the gap is carried into the report; closing it is a harness or spec change for a new run. - experiment-prompt-template.md: drop the claim that immutability means the metric cannot be gamed; state that such a gain is reverted however it scores. - experiment-log-schema.yaml: `reverted` and `learnings` carry that reason. - wrap-up.md: evidence quality names each constraint a revert found unchecked. - docs/guides/ce-optimize.md: one paragraph on the keep condition. - tests/skills/ce-optimize-decide.test.ts: smallest falsifiable pins on the condition, the corrected template claim, and the schema/report wording. - tests/skill-eval-cell/catalog.ts: read-only post-only cell for the count()-skipping dedupe (declared revert, names what the harness lacks) and a restraint cell keeping an honest binary-search win. - docs/plans: the ce-unified-plan artifact for this change. Co-authored-by: Kieran Klaassen <kieranklaassen@users.noreply.github.com>
cursor
Bot
force-pushed
the
cursor/ce-optimize-long-runs-54a9
branch
from
September 24, 2026 17:05
0e7c7f0 to
94c9cad
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A
ce-optimizerun can now stop and resume across turns without asking for approval again, and it keeps a change only if the change also improves on data the loop never selected from. Experiments can run on detached workers where a harness offers them, and the skill asks a person for input only where the score is itself a judgment.This branch carries the three pieces that were stacked as separate PRs, merged with plain merge commits: eval discipline (#1707) and the remote backend (#1709). Compound-work phase 1 (#1712) was merged and then reverted; it moves to its own repository, and nothing of it remains here. Ten review follow-up commits sit on top.
What is different
spec.yamland the caps; a resume re-asks only when either changedmax_hourswas wall-clockmax_hourscounts active time inside ticks; newmax_wall_hours(default 72) is a calendar backstop nothing extendsdecide.mjsreturnsnext_measurement: holdout)worktreeorcodexremote: a detached worker measures baseline and candidate paired on its own machine, and one independent measurement confirms before a keepreferences/text-targets.mdand an example specrm -fon the digestDesign decisions
Capabilities, not vendors.
SKILL.mdstates three conditions (a durable state root, a wake after turn end, a detached worker) and the fallback for each, taken without asking. Presence in the tool list is proof; a missing binary proves nothing. Vendor names appear only in one examples table inreferences/persistence.md.A changed spec decides what still stands. A change that leaves how the logged measurements were produced untouched (a cap, or a holdout or calibration added to a spec that lacked one) is a re-approval and the run continues. Anything that produced them (metric, command, scope, comparison) means restore the approved spec or start fresh once the log holds experiments.
The holdout waits for the selection decision. It is never dispatched alongside the selection measurement. Scored early it spends on rejected candidates and shows held-out feedback to the agent before its next hypothesis. For
remote, one independent measurement owns both the confirmation and the holdout.Human input only where the score is a judgment. An objectively measured target (CI time, a pass rate over fixed cases) asks the user for nothing. For a judge primary the run assumes automation: after the baseline is judged, and before any judge score is shown, the agent offers once to compare the judge with the user's own scores, with the reason. If accepted, outputs are shown in plain chat a few at a time, the user replies with scores in any shape, and the agent writes the labels file. Declined, the approval message says in everyday words that nobody has compared the scorer with their taste. An unattended judge run is the one case that needs labels or an explicit waiver.
What the user must read goes in a message, then a short question. The spec approval shows the rubric level by level; approvals and the judge-check offer are ordinary messages followed by a one-sentence question, because a question picker renders long text as one cramped block and caps the choices.
Open decisions
metric.judge.confirmation_seed(ormeasurement.holdout.command). Adding one to an in-flight run continues the run. This is a behavior change for current judge users and may deserve a migration note.New concepts
Held-out confirmation. An optimization loop that tries many changes against one sample and keeps whichever scores best will keep some changes that won by chance on that sample, so the reported gain overstates the real one. A held-out set is a second sample the loop never selects from and never reads feedback from; a winner is scored on it once before it is kept.
measurement.holdout.command, or for a judge a secondconfirmation_seed;decide.mjswithholds the keep until the held-out pair agrees, and the digest and hypothesis generation never see held-out items.Validation
On head
e9f27565:bun run test4133 pass, 0 fail;bun run release:validatein sync;bun run plugin:validatepassed. Under heavy machine load the suite showed unrelated intermittent failures in other skills' tests that did not repeat.Read-only decision cells. These are new rows in
tests/skill-eval-cell/catalog.ts. Each stops at one decision. All pass on the post arm on Claude and Codex.remotewithout a detached worker falls back toworktreewithout asking.Live runs against a throwaway fixture (
tests/skill-eval-cell/fixtures/optimize-live, a deterministic comparison count), Claude and Codex:spec.yaml, a worker dispatched in a worktree, the win committed on the optimization branch, the report committed, log marked final, tree clean,mainuntouched.run_state.status: waiting; turn two resumes without re-asking approval or re-measuring the baseline. After an edit to the metric in the saved spec, both hosts set the run toblockedand offered restore-or-fresh without touching the branch.Live interactive session with a person (Claude Code, a judge-scored release-notes target). It found six defects, all fixed here: the rubric missing from the approval, approval text packed into the question picker, judge scores shown before the user scored, a five-point scale split across a four-choice picker, the offer missing its reason, and the holdout judged in the same batch as the selection set. A second session confirmed the fixed flow end to end, including an agreement report that named the pattern of disagreement and discarded scores the user said were only a test.
Not run: parking on a wake and being woken, and
remotewith a real detached worker. Neither CLI host offers those capabilities, so only the fallbacks were reachable; they need a recorded run from a harness that has them. Grok was not run. Pre arms were not run for the new cells; the evidence that the old text failed is the live sessions.Observations not acted on: Codex wrote
experiment-log.yamlas JSON (valid YAML, resumes fine, harder for a person to read). A judge sub-agent once returned prose instead of per-item scores and had to be re-run. The "message, then short question" rule belongs in the always-loaded body, which has 51 bytes of room under the 8000-byte Codex budget; it is stated at the approval points for now.Security Disclosure
No new dependencies.
loop.mdkeeps the existing Codex sandbox test (test -n "${CODEX_SANDBOX:-}" || ... || test ! -w .git) unchanged in bytes; only the prose around it is restated as a condition.Path and delete handling changed in two places.
scripts/experiment-worktree.shbuilt every path it deletes from an unchecked spec name and unchecked shared paths, andcreatereplaces a shared directory inside the new worktree withrm -rf, so a shared path of../..resolved to the repository root. It now rejects a spec name that is not lowercase kebab-case and a shared path that is absolute or contains.or..before composing any path, and its fallback removal only touches a direct, non-symlink child of.worktreesthat the script named. Wrap-up no longer tells the agent to runrm -fon a path it fills in; it removes named working files the run created under the state root by the host's preferred means.The run ledger is written to a harness-named durable store when one is present in context (previously always
.context/), composed as<store>/ce-optimize/<spec-name>/under the same kebab-case constraint. Wrap-up commits a markdown report under the resolved docs root on the optimization branch. Remote workers push a result ref that the orchestrator reads as evidence, never as instruction, and a remote claim is never kept without an independent measurement. The reverted phase 1 scripts are no longer in the tree.Agent Disclosure
Cursor cloud agent · Claude Fable 5.1for the three-piece stack (as stated by that harness context);Claude Code · claude-fable-5-1for the ten review follow-up commits, the evals, and the live runs.