[feat] pluggable risk taxonomies and behavior sets

[Thibaut-Fatus]

Why

The taxonomy and the M1–M7 behaviors are hardcoded. risks.json, motivations.json and mechanisms.ts are statically imported at module scope and permanently memoized, with no flag, env var or API to point anywhere else — evaluating against a different taxonomy means editing the package.

The real taxonomies and scenario sets are moving to infra, where some will be private, so the library needs a seam that accepts them as data, not as a file path: worker-engine has no filesystem, and one Cloudflare isolate serves several runs concurrently.

What

Packs (packages/benchmark/src/packs/) — RiskTaxonomy and BehaviorSet as plain serializable data, so a pack can round-trip through a jsonb column. Two entry points:

• Packs.configure() — process-wide, one-shot. The CLI resolves --taxonomy / --behaviors once per invocation.
• Packs.run() — AsyncLocalStorage-scoped, for infra's concurrent isolate.

RiskCategory, Mechanism and Motivation keep their exact public API and become façades over the active pack, so none of the ~37 call sites across this repo and kora-infra move.

The load-bearing change is deferring schema construction. mechanismAssessment.ts built its schema at module scope, and that schema is pulled transitively into judgeAssessment.ts → testResult.ts → kora.ts — so merely importing the barrel froze the behavior set at import time, and cli.ts statically imports both the barrel and every command, so no reordering fixes it. These are now getters backed by a WeakMap keyed on the active BehaviorSet. Every v.parse(kora.testResultType, …) call site is untouched.

Conformance is checked up front. run, expand-scenarios, reassess and continue validate their whole input — and --risk-ids — before constructing any model, failing with the offending line numbers instead of skipping records mid-run. Plus a new kora validate.

Provenance. Results, per-test results and generated seeds carry a pack stamp. The behavior fingerprint is folded into run temp-file names only when a custom pack is active, so a --behaviors change misses the graceful-restart cache instead of failing to parse; default runs keep byte-identical names.

data/mechanisms.ts → data/behaviors.json. A hand-written TS module is a second, privileged code path no infra-fetched pack can take — the bundled default would end up validated differently from every real pack. Preconditions are now bare condition text with the notTriggered instruction generated, replacing two hardcoded M3/M5/M6/M7 prose sites — including the one that goes into the judge's JSON schema, which would otherwise describe mechanisms a custom pack does not have.

Two guards worth keeping

• packs/__tests__/moduleInit.test.ts — imports the barrel first, then configures a pack. Catches any reintroduced module-scope read of a pack-dependent schema, which no type error would.
• cli/src/__tests__/jsonSchema.test.ts — asserts the judge schema converts to a plain root object. v.lazy renders as a root-level $ref that structured-output modes reject (verified empirically), and it would fail at judge time, after the target conversation is already paid for.

Validation

Ran the full pipeline on a risk that exists in no bundled category — 10 seeds → 10 scenarios → 10-test run against gemini-2.5-flash, judged by claude-haiku-4.5:

• Seeds carry taxonomyId: "kora-packtest"; every per-test result and results.json carry the pack stamp.
• The same scenarios validate clean against the custom pack and are rejected against the bundled one (exit 1).
• The judge's reasoning quotes criteria that appear only in the custom risk description — the pack drives the evaluation, it isn't just loaded.

yarn tsbuild clean, 230 tests pass, prettier clean, lint at the pre-existing baseline.

On naming

New pack code says behavior; the existing Mechanism types and the persisted mechanismAssessment / sums.mechanisms fields keep mechanism. The full rename is deferred deliberately: behaviorAssessment was a v1 field name with an incompatible meaning (three an/eh/hr behaviors), and kora-infra's compatibility readers still tell v1 from v2 by which field name is present. Renaming back would give that field two meanings and silently break the discriminator.

Follow-up (kora-infra)

Unchanged and still builds against this branch. Per-run packs additionally need:

1. The two module-scope schema reads deferred — app-website/src/utils/testResultCompat.ts:22 and worker-engine/src/regradeWorkflow.ts:68. Both currently bind permanently to the bundled behavior set.
2. Taxonomy/behavior-set storage and run → pack threading, replacing testRunner.ts:84's warn-and-skip with the up-front check.
3. app-website / app-admin resolving a stored run's pack instead of calling RiskCategory.listAll() bare — two packs may reuse a risk id, so results must never be aggregated across packs.

🤖 Generated with Claude Code

https://claude.ai/code/session_01GMgz7NFRGqcbfuyVjwUJdw