BuildSkill ships its evaluation harness as adopted Python under the skill directory: scripts/ (run loop, aggregation, packaging), templates/agents/ (grader, comparator, analyzer prompts), and eval-viewer/generate_review.py. It works, but it is skill-bundled: grading knows nothing about deck validation, and eval results live only in the eval workspace.
Migrate the harness into rune as an eval subcommand family:
rune eval run: execute eval cases with-skill and baseline, capture timing where the harness reports it.
rune eval grade: grade assertions, wiring in the validators rune already owns; mdschema against the nearest schema and the structural checks, so validation assertions stop being skipped for lack of a sandbox schema.
rune eval report: aggregate and render the benchmark and review page.
- Record eval results alongside the artifact's provenance and review records, which rune also owns.
Until parity, the Python scripts in the deck remain the reference implementation; they were exercised end to end on 2026-08-11 (three cases, six runs, three graders) and the workflow held up. One defect found and fixed during that run: aggregate_benchmark.calculate_stats crashed on the null timing values the EvalLoop workflow mandates for unavailable metrics; nulls are now filtered.
Lessons from the same run worth building in:
- Runner and grader subagents need transcripts for process assertions ("no remote invocation was attempted") to be falsifiable; grading from outputs alone cannot check them.
- Timing and token metrics were unavailable from the harness notifications; a native runner should capture them itself instead of depending on notification payloads.
- Report-file writes from subagents can be blocked by harness hooks; a native runner should collect reports through its own channel.
BuildSkill ships its evaluation harness as adopted Python under the skill directory:
scripts/(run loop, aggregation, packaging),templates/agents/(grader, comparator, analyzer prompts), andeval-viewer/generate_review.py. It works, but it is skill-bundled: grading knows nothing about deck validation, and eval results live only in the eval workspace.Migrate the harness into rune as an
evalsubcommand family:rune eval run: execute eval cases with-skill and baseline, capture timing where the harness reports it.rune eval grade: grade assertions, wiring in the validators rune already owns;mdschemaagainst the nearest schema and the structural checks, so validation assertions stop being skipped for lack of a sandbox schema.rune eval report: aggregate and render the benchmark and review page.Until parity, the Python scripts in the deck remain the reference implementation; they were exercised end to end on 2026-08-11 (three cases, six runs, three graders) and the workflow held up. One defect found and fixed during that run:
aggregate_benchmark.calculate_statscrashed on the null timing values the EvalLoop workflow mandates for unavailable metrics; nulls are now filtered.Lessons from the same run worth building in: