feat: discriminative code-gen scoring — execution scorer + 108-case battery

[Ryfter]

Summary

Item 1 of the discriminative scoring v2 roadmap: make code-gen actually measure capability. Started as an execution scorer; now also carries the battery that makes the scorer worth running.

compilable-code only checked that output parses, so garbage that compiles scored 1.0 and nearly every model read 1.0. The cases were memorised classics on top of that.

What changed

Scorer

• code-exec: runs the candidate in an isolated subprocess against hidden asserts; score = fraction passed, not binary.
• Cross-platform sandbox teardown. The original timeout path used os.killpg/SIGKILL, both POSIX-only — on Windows the runaway child survived and TemporaryDirectory cleanup died with WinError 32. Timeouts are the characteristic weak-model failure, so this broke the most-exercised path on the box Gauntlet actually runs on. Now taskkill /F /T vs killpg, always reaped, pipes closed.
• Structured failure_mode: none / no_code_emitted / syntax_error / runtime_exception / wrong_answer / timeout / harness_error. Aggregated per cell. This is the signal that distinguishes a capability gap from a defect in plausible code — only the second is worth scaffolding (D-2026-06-30c).
• AST constraint checking (no-imports, single-expression, must-use-generator, …) reported separately from correctness, isolating instruction-following from algorithmic skill.
• Per-tier scorecard profile. The shape beats the mean: clearing T1–T2 then collapsing at T3 is a different proposition from scoring evenly.

Battery — 108 cases, 12 dimensions

tier	T1 16 (15%) · T2 28 (26%) · T3 38 (35%) · T4 26 (24%)
dimensions	adversarial-correctness, stdlib-api-use, class-level-stateful, multi-function, execution-prediction, complexity-constrained, bug-fix, surface-constraints, robustness-contracts, behavior-preserving-refactor, test-authoring, data-text-munging

The ladder is deliberate. The fleet is 1B–30B local models: all-1.0 is non-discriminative, but so is all-0.0. T3 does most of the discriminating; T4 is headroom so the battery doesn't saturate as models improve.

Dimensions are adapted from EvalPlus/HumanEval+, BigCodeBench, ClassEval, CRUXEval, EffiBench/BigO(Bench), DS-1000, SWE-bench and TREAT.

Anti-contamination. Retired fizzbuzz, palindrome, binary-search, lru-cache, csv-parse, plus four classic-shaped cases from the first pass (run-length-encode, matrix-transpose, safe-divide, factorial-strict). Files remain in the tree; they are out of the battery. New cases are original problems or familiar shapes with a twist a memorised answer gets wrong.

Every case proves itself. Each ships a reference solution (must score exactly 1.0) and a subtly-wrong solution (must score < 1.0), enforced for all 108 by tests/test_case_validation.py. The gate rejected 5 authored cases: three whose reference could not satisfy its own asserts, one whose wrong solution still scored 1.0, one duplicate-weak. An unsatisfiable case scores every model 0.0 and a toothless one scores every model 1.0 — both look like data and are noise.

Discrimination evidence

Five synthetic outputs against all 108 cases:

excellent (reference)      mean=1.000   none=62
subtly wrong               mean=0.635   wrong_answer=62
prose, no code             mean=0.000   no_code_emitted=62
syntax error               mean=0.000   syntax_error=62
infinite loop              mean=0.000   timeout=62

Under compilable-code, the first two both scored 1.0. Subtly-wrong landing at 0.635 is the point: partial credit, not a cliff.

That sweep also caught a real bug — prose replies were being filed as syntax_error (true of the bytes, wrong about what happened), since the check only caught empty output. Fixed, with a test pinning the boundary so a genuine-but-malformed attempt still reads as syntax_error.

Tests

588 passed, 8 skipped, 4 deselected in 73.41s

Skips are test-authoring cases exempted from the prompt-leak check: there the prompt legitimately contains the implementation under test, so overlap is the task, not a leak.

Cost

108 cases × 9 models ≈ 970 cells per full run. Sandbox overhead ~0.3–0.5s per case on top of inference.

Not done here

• Anti-cheat/integrity layer — subprocess isolation is not a security boundary. Absolute-path filesystem reads and direct network syscalls are still open, so a candidate could in principle read its own hidden tests. Tracked separately.
• Roadmap items 2–5 (v2 spec, raw-vs-scaffolded, pipeline economics, ToC scheduler) need local inference and design sign-off.

🤖 Generated with Claude Code

---

Integrity layer (anti-cheat)

A benchmark that can be shortcut measures nothing. The sandbox was subprocess-only — absolute-path reads and network syscalls were unrestricted — so a candidate could read its own hidden tests and score 1.0 without demonstrating anything.

vector	control
A prompt leakage	line-level guard (≥40 chars; shorter runs collide constantly on ordinary Python). test-authoring exempt by design
B filesystem	in-sandbox patching of open/io.open/os.open/listdir/scandir against the benchmark tree and the sandbox's own hidden_tests.py copy; sys.path stripped; process creation blocked
C network	socket entry points refuse and record
D tool use	client.py refuses tools/functions/web_search/… and inspects reply shape — it is the only HTTP component, so nothing bypasses it
E canaries	deterministic per-case marker in every hidden-test file; an echo in model output marks the cell unscored, never 0

Two holes found by attack, not by reasoning

Both succeeded against the first version of the guard:

• os.system("cp hidden_tests.py leaked.txt"), then read the copy. The copy had an allowed name, so the open() patch never saw it. Fixed by blocking process creation as a class rather than chasing filenames.
• gc.get_objects() → the hidden module's __dict__. The grader necessarily shares the candidate's process, so the expected values are reachable. gc.get_objects, gc.get_referrers and sys._getframe are now blocked.

Results

red team (16 attacks, authored blind by an outside model):
  0/16 obtained the answers,  9 recorded as explicit violations

white box (4 attacks, written with knowledge of runner internals):
  0/4  obtained the answers,  4 recorded as explicit violations

Both numbers are reported because "obtained nothing" and "was blocked" are different claims — a cheat that never worked is not evidence the guard stopped it.

The corpus is committed under tests/redteam/ so future changes can be re-tested against it.

What this is not

docs/2026-07-25-benchmark-integrity-threat-model.md states the limits plainly rather than implying coverage that does not exist:

• A Python-level guard is defeatable by ctypes, re-exec, or raw syscalls. It blocks accidental and casual cheating — the realistic failure for cooperative local models — and is not a security boundary.
• In-process grading is structurally reachable. Closing that properly means grading out-of-process, which is incompatible with calling arbitrary candidate functions. Flagged for revisit if Gauntlet is ever used to rank models with something at stake.
• A gateway performing retrieval silently, declaring nothing in request or response, is undetectable from here.

Fail closed: a control that cannot be enabled marks affected cells unscored with the reason recorded — never silently scored as though it were active. Violations likewise yield unscored, not 0.0: zero is a claim about the model's ability, unscored is an admission that we do not know.

Tests

611 passed, 8 skipped, 4 deselected in 81.51s