feat(ce-optimize): held-out confirmation, judge validity gate, and text-target guidance

[kieranklaassen]

Stacked on #1708 (cursor/ce-optimize-long-runs-54a9); merge that first.

Summary

Implements "Proposal I" (eval discipline) for ce-optimize: the DSPy-style checks around the existing loop. Nothing about what counts as an improvement changes; what changes is whether the harness is trusted before the baseline, whether a keep is confirmed on data the loop never selected from, and what the judge hands back for the next hypotheses.

Rebased onto cursor/ce-optimize-long-runs-54a9 and targeted at it. The diff here is only the eval-discipline content; the tick loop, state root, and approval record are #1708's. Reconciled against #1708: holdout is required by the same wake after turn end condition #1708 defines (not a second term); the holdout rule sits beside #1708's unattended judge-cost-cap rule without duplicating it; judge-cache.yaml is now a row in #1708's persistence.md file table and the log schema names its location as <state-root>/judge-cache.yaml.

What changed

Phase 1 metric validity gate (measurement.md 1.2, spec.md step 2, log harness_validation). The harness is accepted when it orders the user's known-good exemplars above the known-bad ones and does not reward a trivial shortcut (empty / constant / copied output). For type: judge, additionally when the judge agrees with a human-labeled sample (20-50 items with reasoning) at metric.judge.calibration.min_agreement (default 0.8), or the user explicitly waives calibration; a judge run with neither does not leave Phase 1. Failure direction: stop before the baseline, report which probe failed. The approval presentation gains one evidence-quality line.

Held-out confirmation (optimize-spec-schema.yaml, decide.mjs, loop.md 3.3/3.4, wrap-up.md). measurement.holdout.command, or metric.judge.confirmation_seed distinct from sample_seed. Scored only before a keep and at final confirmation; never used to select or generate hypotheses. decide.mjs owns the gate mechanically: a would-be keep returns next_measurement: holdout until the payload carries a holdout snapshot pair, and a holdout that disagrees returns revert or inconclusive. Required for type: judge and when the run waits between ticks through a wake after turn end (#1708's term); otherwise optional, with the approval presentation stating plainly that selection and reporting share one sample.

Judge feedback -> hypotheses (judge-prompt-template.md, loop.md 3.3/3.5, log judge.items[].feedback). A required per-item feedback string; the digest gains a "Failure themes" section built from it; Phase 3.5 generates at most one hypothesis per theme, not a rule per failing item. The template also asks for repeatable scores (same item, same score), which is what makes the cache and calibration valid.

Judge cache and per-experiment diagnostics (loop.md 3.3, log schema). Judge results cached by content hash in judge-cache.yaml beside the log for the run; cost (usd / tokens / latency) recorded when reported.

Per-case regressions (measurement.per_case, decide.mjs, wrap-up.md). When the harness emits a cases map, decide.mjs returns regressions (cases the reference passed and the candidate fails). Report only; the decision is unchanged.

references/text-targets.md (new) + example-text-target-spec.yaml (new). For targets that are instruction text: eval set from recorded failures (four fields, 5-10 floor); the coverage rule "every rule you want preserved needs a case that fails without it"; hypothesis moves as categories; task model pinned by the harness, proposer may be stronger; a ceiling score is "eval too easy", stop and say so; an external optimizer as one experiment, capability phrasing, no library named. Pointed to from spec.md and measurement.md (SKILL.md body untouched, at 7,778 bytes it is near the Codex cap).

Portability fix while the block was touched: judge model: haiku|sonnet -> cheap|strong capability tiers, harness resolves the concrete model; legacy values read as aliases. example-judge-spec.yaml updated (tier, confirmation_seed, calibration).

Design choices made where the doc did not decide

• Holdout gate lives in decide.mjs as a ladder step (next_measurement: holdout) rather than as prose alone. The script already owns the ladder next step; prose would have left the gate to the model's discretion.
• Holdout disagreement maps to the existing outcomes (revert when the holdout regressed, inconclusive when it did not confirm) rather than a new outcome enum. Closest to today's log shape.
• Judge-type holdout defaults to a second seed on the same output rather than a second command; measurement.holdout.command still works for judge runs that have a separate set.
• Tests extend tests/skills/ce-optimize-decide.test.ts (which already carries the ce-optimize schema/skill pins) instead of creating tests/skills/ce-optimize-contract.test.ts; that file does not exist on main yet and is on the other worker's file list, so keeping out of it avoids an add/add conflict at rebase.
• No docs/solutions/ learning. The coverage rule and the "script owns the gate" reasoning are stated in text-targets.md, the schema description, and the decide.mjs header; they fail the AGENTS.md counterfactual.

Left as follow-up (untouched blocks, not brought to the standard)

• measurement.md 1.2-1.3 still read as numbered procedures around the new condition paragraphs; only the added blocks are stated as conditions.
• loop.md 3.2 Codex env-var check is unchanged (the long-run branch restates it).
• A behavioral fresh-agent eval (bun run test:skill-eval-cell) was not run: neither claude nor codex is on PATH in this environment. Scenarios worth running before merge: (a) judge spec without labels or waiver stops in Phase 1; (b) hard spec with measurement.holdout collects the holdout only when decide.mjs asks; (c) plain hard spec with no holdout behaves as before.

Validation

• bun test tests/skills/ce-optimize-decide.test.ts: 99 pass after the rebase (10 new decide tests for holdout / regressions, 7 new pin tests, plus feat(ce-optimize): long-running optimization stack (tick loop, eval discipline, remote backend) #1708's).
• bun run test (after rebase): 4107 pass, 14 timeouts in tests/skills/ce-work-unit-workspace-fallback.test.ts and tests/ce-code-review-mechanics.test.ts (files this branch does not touch) under a loaded 4-worker VM; both files pass in 9s and 2s when run alone.
• bun run release:validate: in sync (35 skills, no count change).
• bun run plugin:validate: not run, claude is not on PATH here.

Security Disclosure

No security-relevant changes. decide.mjs gains a pure-function path over an additional JSON snapshot pair and a cases object; no new shell execution, path handling, or dependency.

Agent Disclosure

• Model: Cursor cloud agent · Claude Fable