[docs] document the evaluation pipeline internals

[Thibaut-Fatus]

Why

The README documents how to invoke each pipeline stage — flags, defaults, model chains — but nothing in the repo explained what happens inside them. Details like why demographics are allocated in code rather than asked for in the prompt, or why expansion needs a task-level fallback chain on top of the per-call one, only lived in the source.

What

Adds EVALUATION_PROCESS.md at the root, linked from the README in three places (Overview, "Pipeline stages", and the project-structure tree).

Contents:

• Overview — risk → seed → scenario → graded conversation, and the principle that everything statistically controlled is allocated in code, never left to the model.
• generate-seeds — the task grid, grid vs. distribution mode, largest-remainder/shuffle/zip persona allocation, motivation round-robin, flavor allocation, and what the code stamps vs. what the model chooses.
• expand-scenarios — the disciplined-expansion prompt and its three conditional sections, the 7-point validation checklist, the validationFeedback retry loop, and why a ScenarioValidationError rotates models at the task level (a validation failure is a well-formed response, so per-call retry never sees it).
• run — prompt variants, conversation length, the validateAssistantTurn capture gate, judge aggregation (median grades / mean counts / matching reasons), and notTriggered score-neutrality.
• Guardrails table — all seven, in one place.
• Reproducing the shipped corpus — the exact commands from c285c5c, the per-risk marginals, and two quirks: 781 seeds rather than 780 (one task returned two seeds where one was requested; nothing clamps output.seeds.length to seedsToGenerate), and the missing taxonomy stamp on pre-packs seeds.
• Dead code — conversationToMatchPrompt and riskToScenariosPrompt have no call sites and are not re-exported from index.ts; both date from c9be924 and were superseded by conversationToAssessmentPrompt and the two-phase seed/expand split respectively. Includes a live/dead table for all eleven files in src/prompts/, plus the smaller ScenarioValidationVerdict orphan.

Also

Fixes the Overview risk count: 25 → 26. The taxonomy section further down already said 26, and risks.json has 26.

Notes

• Docs only — no TS/JSON touched. yarn pretty passes.
• The two dead prompt files are documented, not deleted. Happy to remove them in a follow-up if you'd rather.

🤖 Generated with Claude Code

https://claude.ai/code/session_01NRbfZ2n5gKAppVbCwbA2t6