Skip to content

feat: rune eval — migrate the BuildSkill evaluation harness #14

Description

@runewright

BuildSkill ships its evaluation harness as adopted Python under the skill directory: scripts/ (run loop, aggregation, packaging), templates/agents/ (grader, comparator, analyzer prompts), and eval-viewer/generate_review.py. It works, but it is skill-bundled: grading knows nothing about deck validation, and eval results live only in the eval workspace.

Migrate the harness into rune as an eval subcommand family:

  • rune eval run: execute eval cases with-skill and baseline, capture timing where the harness reports it.
  • rune eval grade: grade assertions, wiring in the validators rune already owns; mdschema against the nearest schema and the structural checks, so validation assertions stop being skipped for lack of a sandbox schema.
  • rune eval report: aggregate and render the benchmark and review page.
  • Record eval results alongside the artifact's provenance and review records, which rune also owns.

Until parity, the Python scripts in the deck remain the reference implementation; they were exercised end to end on 2026-08-11 (three cases, six runs, three graders) and the workflow held up. One defect found and fixed during that run: aggregate_benchmark.calculate_stats crashed on the null timing values the EvalLoop workflow mandates for unavailable metrics; nulls are now filtered.

Lessons from the same run worth building in:

  • Runner and grader subagents need transcripts for process assertions ("no remote invocation was attempted") to be falsifiable; grading from outputs alone cannot check them.
  • Timing and token metrics were unavailable from the harness notifications; a native runner should capture them itself instead of depending on notification payloads.
  • Report-file writes from subagents can be blocked by harness hooks; a native runner should collect reports through its own channel.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions