[feat] evaluation profiles and run stamps - #32
Merged
Merged
Conversation
Pin every harness LLM (seed generation, expansion, user simulator, judges) in an evaluation profile with inline configs and a content hash, and stamp every seed, scenario, test result and result header with the effective profile, resolved model configs, prompts fingerprint, packs, code revision and input corpus hash. - profiles/kora.json reproduces the previous hardcoded CLI defaults; a test asserts each role matches its models.json entry - profiles/*.local.json are gitignored scratch profiles for testing a model configuration (--profile <name>.local); their hash is not checked - CI guards: committed profile hashes and the prompts fingerprint must be bumped when their content changes (tests print the expected value) - per-role CLI flags remain as explicit overrides: warned, listed in the stamp, and hashed differently from the named profile - graceful-restart temp dirs hold a stamp.json; resuming under a different configuration is refused - result headers record `served`: the model ids the provider reported - new `kora profile [--check | --print-hash]` command; `validate` prints the profile summary - Stamp.run mirrors Packs.run so infra can scope a stamp per run Claude-Session: https://claude.ai/code/session_017SeRjc1yMoa617ZR2bxeE8
…esults complete-run.mjs and manual-rerun.mjs take judges and the user simulator from the evaluation profile (KORA_PROFILE) like the CLI; JUDGE / USER_MODEL remain as warned overrides. complete-run builds the run stamp, goes through the same resume guard as `kora run` against .kora-run-tmp/stamp.json, and names the target from the cached stamp (or TARGET). manual-rerun records the user spec on each transcript entry. Claude-Session: https://claude.ai/code/session_017SeRjc1yMoa617ZR2bxeE8
Move the evaluation profile schema, hash and role helpers into the benchmark package next to packs, so kora-infra can build profiles and stamps with the same formula; the CLI keeps loading files. Stamp roles other than `user` and `judges` become optional: a harness records the roles it has. Claude-Session: https://claude.ai/code/session_017SeRjc1yMoa617ZR2bxeE8
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Runs must use the same LLM configuration for every step and document it so others can reproduce them. Until now results only recorded registry slugs, the per-role defaults were hardcoded in
cli.ts, nothing recorded prompt versions or the commit, and the restart caches keyed only on scenario id, so re-running with different judges silently reused old results.What
profiles/<name>.json, next tomodels.json): inline model config for every harness role (seeds,expansion,expansionUser,user,judges,continueUser) plus a content hash.profiles/kora.jsonreproduces the previous defaults; a test asserts every role matches itsmodels.jsonentry. Global--profile/KORA_PROFILE.profiles/*.local.jsonare gitignored, hash-unchecked, markedlocalin the stamp.cp profiles/example.local.json.example profiles/x.local.jsonthen--profile x.local.kora profile [--check | --print-hash]: prints the resolved configuration;--checkpings every model and reports the served id and latency;--print-hashprints the hash to paste after a version bump.[model],[user-model],--judgesstill work, resolve throughmodels.json, warn, and produce a different profile hash withoverrides: [...]in the stamp.served. All new fields are optional, so existing records keep parsing.stamp.json; a command refuses to resume under a different profile, prompts or packs (no bypass flag). Legacy dirs without a stamp resume with a warning.Stamp.runmirrorsPacks.runso kora-infra can scope a stamp per run in a shared isolate.Verification
yarn tsbuild,yarn test(330 tests),yarn lint,yarn prettyclean (lint reports only pre-existing issues in a gitignored script underdata/).kora profile --checkpassed for all five models; a one-scenariorun gpt-4oproduced a stamped result and archive; a seeded temp dir was refused under--judges gpt-5.5:highand resumed cleanly under the same configuration.Also in this PR
scripts/complete-run.mjs/manual-rerun.mjsnow load the evaluation profile (KORA_PROFILE), keepJUDGE/USER_MODELas warned overrides, stamp their results and go through the same resume guard askora run. Verified live against a scratch run dir.Follow-ups (not in this PR)
Stamp.run, and bump the benchmark package before ingesting stamped CLI output (its strict test-result parser).kora@1stamp hash.🤖 Generated with Claude Code
https://claude.ai/code/session_017SeRjc1yMoa617ZR2bxeE8