From 1a0462add83299c4239d43f38bce839c4a4dd0e9 Mon Sep 17 00:00:00 2001
From: Claude <noreply@anthropic.com>
Date: Tue, 25 Aug 2026 00:34:40 +0000
Subject: [PATCH] docs: add root CLAUDE.md, overhaul BENCHMARK.md, add
 CODE_HEALTH.md
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

- CLAUDE.md (new): commands, architecture map + layering rules, conventions,
  testing reality, benchmark quick-ref, branch/PR rules, gotchas. Two docs
  already declared themselves "extensions of CLAUDE.md" — it now exists.
- docs/BENCHMARK.md: remove 5 dead links to nonexistent docs/benchmark/,
  align File Map/status tables with the 9 real benchmark dirs (7 retired
  ones removed), fix the --event-stream guidance (flag exists only in
  locomo; other runners consolidate unconditionally in base_runner),
  document longmemeval entry points, correct the auto-download claim, add
  an environment-variable table and a "Contributing a new benchmark" guide.
- docs/CODE_HEALTH.md (new): evidence-based structural findings (file:line)
  in three tiers with fix directions and sequencing.
- .env.example: document benchmark env vars; fix duplicate
  COGNIFOLD_SESSION_BACKEND definition.

Docs-only change; no src/ code touched.

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VnAaidzJkganncouErUp3m
---
 .env.example        |  29 +++-
 CLAUDE.md           | 152 ++++++++++++++++
 docs/BENCHMARK.md   | 412 +++++++++++++++++++-------------------------
 docs/CODE_HEALTH.md | 253 +++++++++++++++++++++++++++
 4 files changed, 614 insertions(+), 232 deletions(-)
 create mode 100644 CLAUDE.md
 create mode 100644 docs/CODE_HEALTH.md

diff --git a/.env.example b/.env.example
index 0d182b9..54728bf 100644
--- a/.env.example
+++ b/.env.example
@@ -31,7 +31,9 @@ COGNIFOLD_SESSION_TTL_HOURS=24
 COGNIFOLD_SUPABASE_URL=
 COGNIFOLD_SUPABASE_KEY=
 COGNIFOLD_ENABLE_GRAPH_SYNC=false
-COGNIFOLD_SESSION_BACKEND=memory
+# To use Supabase, also set COGNIFOLD_SESSION_BACKEND=supabase above
+# (do not define COGNIFOLD_SESSION_BACKEND twice in this file — the later
+# value wins when sourced).
 
 # --- Cloud Run Overrides ---
 # These are set automatically by the CD workflow for Cloud Run deploys.
@@ -40,3 +42,28 @@ COGNIFOLD_SESSION_BACKEND=memory
 # COGNIFOLD_TIMEOUT=300
 # COGNIFOLD_SESSION_BACKEND=redis
 # COGNIFOLD_REDIS_URL=redis://MEMORYSTORE_IP:6379/0
+
+# --- Benchmarks (optional; see docs/BENCHMARK.md "Environment variables") ---
+# Reader/build model for scripts/reproduce.sh (provider:model syntax).
+# Paper stack is openai:gpt-4o-mini; reproduce.sh defaults to openai:gpt-5.
+# MODEL=openai:gpt-4o-mini
+# Per-role overrides for scripts/parallel_longmemeval.sh:
+# READER_MODEL=openai:gpt-5
+# WRITER_MODEL=openai:gpt-5
+# JUDGE_MODEL=openai:gpt-4o        # judge lock — do not substitute
+# RERANK_MODEL=openai:gpt-5
+# EMBED_MODEL=openai:text-embedding-3-large
+# Separate endpoints/keys (optional):
+# EMBEDDING_API_KEY=
+# EMBEDDING_BASE_URL=
+# JUDGE_API_KEY=
+# JUDGE_BASE_URL=
+# OpenRouter shim (exported as OPENAI_API_KEY + OpenRouter base URL by the
+# benchmark shell scripts):
+# OPENROUTER_API_KEY=
+# Download mirrors (e.g. in China):
+# HF_ENDPOINT=https://hf-mirror.com
+# GITHUB_MIRROR=
+# Ablation switches (read by benchmarks/shared/base_runner.py):
+# COGNIFOLD_ABLATE_KNN=1
+# COGNIFOLD_ABLATE_MERGE=1
diff --git a/CLAUDE.md b/CLAUDE.md
new file mode 100644
index 0000000..646342c
--- /dev/null
+++ b/CLAUDE.md
@@ -0,0 +1,152 @@
+# CLAUDE.md
+
+Guidance for Claude Code (and human contributors) working in this repository.
+CogniFold is a brain-inspired always-on agent memory: it folds an event stream
+into a typed concept graph (`event` → `concept` → `intent`) and surfaces
+proactive context without being asked. Deeper reading: `README.md` (product
+overview), `docs/ARCHITECTURE.md` (technical authority), `docs/BENCHMARK.md`
+(benchmark system), `docs/PHILOSOPHY.md` (design intent).
+
+## Commands
+
+```bash
+uv sync                          # fastest install (uses uv.lock); or: pip install -e ".[dev,agent,service]"
+make test                        # python -m pytest tests/ -v
+make lint                        # ruff check src/ tests/
+make format                      # ruff format src/ tests/
+make typecheck                   # pyright src/  (pyright STRICT, not mypy)
+make check                       # lint + format --check   (CI runs check + typecheck + test)
+make serve                       # ./scripts/start_server.sh (HTTP service on :8000)
+bash scripts/reproduce.sh <bench>  # paper benchmarks; see docs/BENCHMARK.md
+bash scripts/pre-commit.sh       # local quality gate (cp to .git/hooks/pre-commit)
+```
+
+- **Tool versions are pinned** in the `dev` extra: `ruff==0.15.17`, `pyright==1.1.408`.
+  Do not upgrade casually — `ruff format --check` will churn across the tree.
+- Lint/format/typecheck scope is **`src/` and `tests/` only**. `benchmarks/`,
+  `scripts/`, and root-level `.py` files are not checked by CI.
+- Python: CI, Docker, and pyright all use **3.11**. (`requires-python` says
+  ">=3.10" and ruff targets py39 — treat 3.11 as the real floor.)
+
+## Architecture map
+
+Write path: `Generator/Importer → Event stream → CognifoldAgent (LangGraph) →
+UpdatePlan → PlanExecutor (validate → atomic execute → rollback) → ConceptGraph`.
+Read path: `query → MemoryQueryAgent (legacy|bm25|semantic|hybrid|agentic) →
+GraphTraverser → scorer → ContextAssembler → QueryResult`. Proactive read:
+`HierarchicalContextSelector` bands (immediate/working/background) — no query needed.
+
+Packages under `src/cognifold/` (~40K lines, 150 files):
+
+| Package | Role |
+|---|---|
+| `models/` | Pydantic schemas: Node, Edge, Event, UpdatePlan (bottom layer, no internal deps) |
+| `graph/` | NetworkX MultiDiGraph wrapper, persistence, validation, metrics |
+| `scoring/` | PageRank + recency ranking, context window selection |
+| `executor/` | Plan validation + atomic execution with rollback |
+| `agent/` | LangGraph write-path agent, prompts, domain configs |
+| `query/` | MemoryQueryAgent, strategies, assembly, LLM calls |
+| `retrieval/` | BM25, hybrid RRF fusion, agentic multi-round |
+| `embeddings/` | Provider ABC: Gemini/OpenAI/Mock, optional FAISS |
+| `pipeline/` | Orchestration — `classic.Pipeline` (public API, MCP) and `LayeredPipeline` (CLI run, service) are BOTH alive by design |
+| `intent/` | Intent-to-action queue, executor, calibrator |
+| `symbolic/` | Symbolic belief/state trackers (used by benchmarks + query) |
+| `temporal/` | Date parsing, temporal entity extraction |
+| `service/` | FastAPI HTTP service: routes, session stores (file/redis/supabase), SSE |
+| `mcp/` | MCP server (`cognifold-mcp`): remember/query/graph_stats/list_intents |
+| `cli/` | `cognifold` CLI: run, query, generate, replay, serve, client |
+| `generator/` `importers/` `simulator/` `replay/` | Event generation, data import, timeline sim, HTML replay |
+| `brain/` | `memory_coverage.json` — **single source of truth** for the coverage figure (pages.yml copies it to docs-site) |
+
+**Dependency layering rule** (`docs/ARCHITECTURE.md` §Module Dependencies):
+`models → graph → scoring/executor → agent → query`; `pipeline` may depend on
+everything; `service`/`cli`/`simulator` sit on top. **Nothing below `service/`
+should import from `service/`** — the existing `service.llm_keys` imports in
+`config.py`, `agent/`, `query/`, `utils/`, `embeddings/` are a known violation
+slated for cleanup (`docs/CODE_HEALTH.md` #4); do not add new ones.
+
+**ADR-001**: `EdgeInferenceEngine` is library code only — never auto-invoke it
+during ingestion; orphan reconnection is handled by
+`PlanExecutor._detect_orphan_nodes`.
+
+## Conventions
+
+- Every new file starts with `from __future__ import annotations` (128/150
+  files already do). PEP 604 unions (`str | None`), builtin generics
+  (`dict[str, Any]`), `if TYPE_CHECKING:` for cycle-avoiding imports.
+- Pyright runs **strict** on `src/` — annotate everything, including `-> None`.
+- Google-style docstrings (`Args:` / `Returns:` / `Raises:`) on modules,
+  classes, and public functions.
+- Logging: `logger = logging.getLogger(__name__)` at module level; call
+  `cognifold.logging.setup_logging()` once at process entry. Never use
+  structlog directly — the stdlib integration handles JSON output.
+- **Pydantic only in `models/`** (wire/graph data). Configuration and internal
+  value objects use stdlib `@dataclass` (48 files follow this).
+- Optional heavy deps (structlog, redis, supabase, faiss, langgraph) are
+  imported lazily with graceful fallback (e.g. hybrid retrieval degrades to
+  BM25 without an embedder). Keep that property.
+- Node IDs use typed prefixes: `e-` event, `c-` concept, `i-` intent, `t-`
+  time; canonical form is `<prefix>-<n>` (e.g. `e-001`).
+- Prompt changes for benchmarks go in `configs/<name>_profile.yaml`, not in
+  Python code.
+
+## Testing reality
+
+`pytest tests/` is a **smoke suite only** (3 assertions). A green pytest does
+not verify behavior. Real verification paths:
+
+- `PYTHONPATH=src python test_benchmarks.py` — all runners import + unified CLI
+- `python scripts/test_retrieval_api.py` — all retrieval modes
+- `python scripts/e2e_supabase_test.py` — service end-to-end
+- Benchmark runs with `--limit 1` (cheap) before full runs (API cost)
+- CI's docker-build job boots the service and polls `/health`
+
+When you touch a deterministic core (`graph/`, `executor/`, `scoring/`,
+`retrieval/bm25`, `query/text_utils`, `config.py`), add real unit tests under
+`tests/` — growing that net is an explicit goal (`docs/CODE_HEALTH.md` #1).
+
+## Benchmarks quick reference
+
+Full guide: `docs/BENCHMARK.md`. Entry points:
+
+| What | How |
+|---|---|
+| Any paper benchmark | `bash scripts/reproduce.sh {cogeval,locomo,musique,narrativeqa,tomi,babilong,mutual,streamingqa,all}` |
+| LongMemEval (headline) | `bash scripts/parallel_longmemeval.sh` or the Claude Code skills `.claude/skills/longmemeval-run` / `longmemeval-iterate` |
+| Datasets | run `benchmarks/<name>/download_data.py` first — runners do NOT auto-download |
+
+Hard rules (from the skills — keep them):
+- **Judge lock**: LongMemEval/LoCoMo judge is `openai:gpt-4o` — never substitute
+  (breaks comparability with published baselines).
+- **Branch lock**: the LongMemEval iteration campaign commits only to
+  `longmemeval-iter`.
+- Reproducing paper numbers requires the paper stack (build `gpt-4o-mini`);
+  `reproduce.sh` defaults to `MODEL=openai:gpt-5` — override `MODEL` to match.
+
+## Branches, PRs, deployment
+
+- `main` and `cognifold-dev` run CI (ruff + pyright + pytest + docker health).
+- Pushes to `cognifold-dev` auto-open a promote PR to `cognifold-stable`;
+  `cognifold-stable` deploys to Cloud Run via `cd.yml`.
+- `pages.yml` deploys `docs-site/` on pushes to `main`; the brain-coverage
+  figure comes only from `src/cognifold/brain/memory_coverage.json`.
+- PRs: single purpose, green CI, English titles/descriptions/review comments.
+- PR review runs automatically (`claude-code-review.yml`); `@claude` in a
+  comment summons interactive help (`claude.yml`).
+
+## Gotchas
+
+- Code comments referencing `my_prompt.md §1.2` etc. now resolve to
+  `.claude/skills/longmemeval-iterate/references/model-config.md`.
+- Default model names disagree across `config.py` / `config.example.yaml` /
+  `configs/cognifold.yaml` — pass models explicitly rather than trusting a
+  default (`docs/CODE_HEALTH.md` #13).
+- `configs/` is **excluded from the Docker image** (`.dockerignore`) — the
+  service must not require it at runtime.
+- `benchmarks/*/output/` is gitignored **except** `benchmark_results.json` and
+  `wrong_cases.json` (allowlisted).
+- Benchmark runners must run from the repo root (relative `output_dir`).
+- Undocumented ablation switches: `COGNIFOLD_ABLATE_KNN=1`,
+  `COGNIFOLD_ABLATE_MERGE=1` (read in `benchmarks/shared/base_runner.py`).
+- `cognifold.utils.embeddings` is deprecated (import-time warning) — new code
+  uses `cognifold.embeddings.create_provider()`.
diff --git a/docs/BENCHMARK.md b/docs/BENCHMARK.md
index 524420d..f09b5a3 100644
--- a/docs/BENCHMARK.md
+++ b/docs/BENCHMARK.md
@@ -2,9 +2,9 @@
 
 **This document is an extension of `CLAUDE.md`. Read `CLAUDE.md` first.**
 
-The benchmark system evaluates Cognifold's memory against established datasets. This file is the **entry point** — it tells you what benchmarks exist, where to find everything, and what to update when making changes.
-
-Detailed documentation lives in `docs/benchmark/`.
+The benchmark system evaluates Cognifold's memory against established datasets.
+This file is the **entry point** — what benchmarks exist, how to run them, what
+to update when making changes, and how to contribute a new one.
 
 ---
 
@@ -12,7 +12,7 @@ Detailed documentation lives in `docs/benchmark/`.
 
 Headline numbers are as reported in the technical report — [arXiv:2605.13438v3](https://arxiv.org/abs/2605.13438), *CogniFold: Always-On Proactive Memory via Cognitive Folding*. The paper is the source of truth; the Implementation Status table below is the internal tracker and is kept consistent with it.
 
-> **Why these numbers (not the highest we can get).** The reported configuration is the one that preserves proactive **intent/intention generation** end-to-end, not the per-benchmark maximum. Several older benchmarks — ToMi in particular — are easy to drive much higher with a task-specialized reader, but that path encourages auto-loop hallucination (the reader confabulates to satisfy the metric instead of reading memory). We report the proactive-substrate stack so the numbers reflect the always-on memory thesis rather than a benchmark-tuned ceiling. See PR discussion for detail.
+> **Why these numbers (not the highest we can get).** The reported configuration is the one that preserves proactive **intent/intention generation** end-to-end, not the per-benchmark maximum. Several older benchmarks — ToMi in particular — are easy to drive much higher with a task-specialized reader, but that path encourages auto-loop hallucination (the reader confabulates to satisfy the metric instead of reading memory). We report the proactive-substrate stack so the numbers reflect the always-on memory thesis rather than a benchmark-tuned ceiling.
 
 ### LongMemEval (Table 5) — J-Score, N=500
 
@@ -50,296 +50,246 @@ Stack: `gpt-4o-mini` extraction/reader, `text-embedding-3-small`.
 
 ---
 
-## ⚠️ ALWAYS pass `--event-stream` (every benchmark, not just LoCoMo)
+## ⚠️ Consolidation and `--event-stream` — how it actually works
+
+Inter-session consolidation (`merge_similar_concepts` + `prune_orphan_concepts`)
+is central to the always-on memory thesis, but the mechanism differs by runner:
+
+- **LoCoMo**: consolidation is gated behind the `--event-stream` flag, which
+  exists **only** in `benchmarks/locomo/run_benchmark.py`. Paper-grade LoCoMo
+  runs MUST pass it (`scripts/reproduce.sh locomo` does so automatically).
+  Sanity-check the log for `Inter-session consolidation:` lines.
+- **All other `BenchmarkRunner`-based runners**: consolidation runs
+  unconditionally in the shared post-ingestion hook
+  (`benchmarks/shared/base_runner.py`, step after ingestion). There is no
+  `--event-stream` flag on these runners — passing it is an argparse error.
+
+Canonical LoCoMo (full 10-conv, Mem0 protocol):
 
-All benchmark runners have `event_stream` default OFF, but **paper-grade runs MUST enable it** to activate per-session inter-session consolidation (`merge_similar_concepts` + `prune_orphan_concepts`). Canonical LoCoMo (full 10-conv, Mem0 protocol):
 ```bash
 PYTHONPATH=src python -u -m benchmarks.locomo.run_benchmark \
-    --event-stream --model openai:gpt-4.1-mini
+    --event-stream --model openai:gpt-4o-mini
 ```
-Sanity check log for `Inter-session consolidation:` lines. Pre-2026-04-19 `--limit` default was `1` (silent conv-26-only truncation); fixed to `None` = all 10. If log shows `Loaded 1 conversations` on a full run → regression.
+
+Historical note: pre-2026-04-19 the LoCoMo `--limit` default was `1` (silent
+conv-26-only truncation); it is now `None` = all 10. If a full run logs
+`Loaded 1 conversations`, that regression is back.
 
 ---
 
 ## Implementation Status
 
-| Benchmark | Location | Status | Accuracy (latest) | Primary Blocker / Note |
-|-----------|----------|--------|-------------------|------------------------|
-| **LoCoMo** | `benchmarks/locomo/` | Tested | **81.23% J-Score overall** (paper Table 4, Mem0 protocol, gpt-4o-mini read/write + gpt-4o-mini judge, `--event-stream`; Single-Hop 90.49 / Multi-Hop 67.38 / Temporal 78.50 / Open 50.00; F1 35.71) | vs ENGRAM 77.55 · MemOS 75.80 · Zep 75.14 |
-| **LongMemEval** | `benchmarks/longmemeval/` | Tested | **93.0% J-Score overall** (paper Table 5, N=500, build gpt-4o-mini / answer gpt-5.4-mini / judge gpt-4o; SSA 100.0 / SSU 97.1 / KU 94.9 / SSP 93.3 / MS 91.0 / TR 88.7) | vs Mastra 94.9 · ENGRAM 71.4 · Zep 71.2; Chronos (High) 95.6. MS lever: see PR #26/#27 |
-| **MSC** | `benchmarks/msc/` | Tested | N/A (excluded from Feb 21 full eval) | Agent concept extraction too passive |
-| **BABILong** | `benchmarks/babilong/` | Tested | **85.0** (paper Fig. 4; proactive-substrate stack, not benchmark-tuned ceiling) | exceeds ARMT — fine-tuned (83.8) |
-| **FutureX** | `benchmarks/futurex/` | Tested | N/A (no GT) | Pipeline verified, needs real MiroFlow |
-| **MuTual** | `benchmarks/mutual/` | Tested | **93.2% acc** (N=500) | Near-SOTA (~97% GPT-4o zero-shot) |
-| **MuSiQue-Ans** | `benchmarks/musique/` | Tested | **F1 58.7** (paper Fig. 4, N=500) | exceeds HippoRAG 2 (49.3) |
-| **TimeQA** | `benchmarks/timeqa/` | Tested | 0.0% EM (Feb 21, n=20) | Temporal reasoning absent |
-| **NarrativeQA** | `benchmarks/narrativeqa/` | Tested | **F1 0.720 / ROUGE-L 0.712** (Apr 8, N=500, GPT-4o) | Scoring normalization + summary detruncation |
-| **QMSum** | `benchmarks/qmsum/` | Tested | F1=0.143, ROUGE-L=0.139 (N=281) | Gemini thinking-token truncation unresolved |
-| **SocialIQA** | `benchmarks/socialiqa/` | Tested | **78.4% acc** (N=500) | LLM internal commonsense sufficient |
-| **ToMi** | `benchmarks/tomi/` | Tested | **83.5 EM** (paper Fig. 4; proactive-substrate stack — a task-specialized reader scores far higher but invites auto-loop confabulation, so not used) | exceeds AutoToM (80.2) |
-| **SafetyBench** | `benchmarks/safetybench/` | Tested | **94.3% acc** (N=35) | Exceeds GPT-4 zero-shot (88.9%); direct mode |
-| **StreamingQA** | `benchmarks/streamingqa/` | Tested | **78.4% EM / F1 0.573** (N=500) | Answer-seeded fact events + containment EM |
-| **RGB** | `benchmarks/rgb/` | Tested | 80.0% EM / F1 0.860 (N=20, pilot) | Wave 7 fix |
-| **CogEval-Bench** (structural) | `papers/cognifold-neurips2025/` | Validated | **Harmony 0.476, Purity 0.361, Proactivity 0.614, Compression 4.6×** (6 systems × 6 scenarios, GPT-4o-mini) | Only CogniFold non-zero on Purity + Proactivity; 5-tier hierarchy revealed |
-
-See [docs/benchmark/results.md](benchmark/results.md) for detailed experiment results and known issues.
+Nine benchmark directories exist in-tree today. Rows for retired benchmarks
+(MSC, FutureX, TimeQA, QMSum, SocialIQA, SafetyBench, RGB) keep their last
+recorded numbers for the paper's record but have **no code in this repo**.
+
+| Benchmark | Location | Status | Accuracy (latest) | Note |
+|-----------|----------|--------|-------------------|------|
+| **LoCoMo** | `benchmarks/locomo/` | In-tree, tested | **81.23% J-Score overall** (paper Table 4) | vs ENGRAM 77.55 · MemOS 75.80 · Zep 75.14 |
+| **LongMemEval** | `benchmarks/longmemeval/` | In-tree, tested | **93.0% J-Score overall** (paper Table 5, N=500) | headline benchmark; entry point is `run_eval.py`, not `run_benchmark.py` — see below |
+| **CogEval-Bench** | `benchmarks/cogeval/` | In-tree, tested | **Harmony 0.476, Purity 0.361, Proactivity 0.614, 4.6×** | dataset is generated (`generate_dataset.py`), not downloaded |
+| **BABILong** | `benchmarks/babilong/` | In-tree, tested | **85.0** (paper Fig. 4) | exceeds ARMT — fine-tuned (83.8) |
+| **MuTual** | `benchmarks/mutual/` | In-tree, tested | **93.2% acc** (N=500) | cleanest runner — use as the template |
+| **MuSiQue-Ans** | `benchmarks/musique/` | In-tree, tested | **F1 58.7** (paper Fig. 4, N=500) | exceeds HippoRAG 2 (49.3) |
+| **NarrativeQA** | `benchmarks/narrativeqa/` | In-tree, tested | **F1 0.720 / ROUGE-L 0.712** (N=500) | |
+| **ToMi** | `benchmarks/tomi/` | In-tree, tested | **83.5 EM** (paper Fig. 4) | exceeds AutoToM (80.2) |
+| **StreamingQA** | `benchmarks/streamingqa/` | In-tree, tested | **78.4% EM / F1 0.573** (N=500) | |
+| MSC / FutureX / TimeQA / QMSum / SocialIQA / SafetyBench / RGB | *(removed)* | Retired | historical | code no longer in tree |
 
 ---
 
 ## File Map
 
-### Documentation (docs/)
-
-| File | Content |
-|------|---------|
-| **`docs/BENCHMARK.md`** | **This file** — entry point, status overview, checklists |
-| `docs/benchmark/status.md` | **Implementation progress tracker** (15/15 done, accuracy data, CLI reference) |
-| `docs/benchmark/architecture.md` | System architecture, core components, profile schema, conventions |
-| `docs/benchmark/dataset-catalog.md` | All 15 planned datasets across 6 categories |
-| `docs/benchmark/results.md` | Experiment results, current status details, known issues |
-| `docs/benchmark/phase12-log.md` | Phase 12 detailed work log (historical record) |
-| **`test_benchmarks.py`** | **Test suite** — verifies all 15 runners, unified CLI args, data files |
-
-### Benchmark Code (benchmarks/)
+### Benchmark code (`benchmarks/`)
 
 ```
 benchmarks/
-├── locomo/
-│   ├── run_benchmark.py        # Main runner (agent mode)
-│   ├── download_data.py        # Fetches locomo10.json from GitHub
-│   ├── locomo10.json           # Dataset (10 conversations, ~66K lines)
-│   └── README.md
-├── longmemeval/
-│   ├── __init__.py
-│   └── run_eval.py             # Runner (batch + turn modes)
-├── msc/
-│   ├── run_benchmark.py        # Main runner (speaker-aware)
-│   ├── download_data.py        # Downloads ParlAI dataset
-│   ├── data/                   # Downloaded data (501 conversations)
-│   └── README.md
-├── babilong/
-│   ├── run_benchmark.py        # Runner (direct/batch/agent modes)
-│   ├── download_data.py        # Downloads from HuggingFace
-│   ├── data/                   # Downloaded data
-│   └── README.md
-├── futurex/
-│   ├── run_benchmark.py        # Async runner (simulated tools)
-│   ├── run_miroflow_benchmark.py  # Real MiroFlow entry point
-│   ├── miroflow_adapter.py     # Cognifold <> MiroFlow adapter
-│   ├── cognifold_orchestrator.py  # Custom MiroFlow orchestrator
-│   ├── download_data.py
-│   └── futurex_data.jsonl      # 96 prediction tasks
-├── mutual/
-│   ├── run_benchmark.py        # MC dialogue reasoning (4 choices)
-│   ├── download_data.py        # GitHub zip download, JSON parser
-│   ├── data/                   # 886 dev examples
-│   └── README.md
-├── musique/
-│   ├── run_benchmark.py        # Multi-hop QA (EM/F1)
-│   ├── download_data.py        # HuggingFace download
-│   ├── data/                   # 4,834 examples
-│   └── README.md
-├── timeqa/
-│   ├── run_benchmark.py        # Time-sensitive QA (EM/F1)
-│   ├── download_data.py        # GitHub JSONL download
-│   ├── data/                   # 2,997 easy + hard examples
-│   └── README.md
-├── narrativeqa/
-│   ├── run_benchmark.py        # Long-form QA (ROUGE-L/F1)
-│   ├── download_data.py        # HuggingFace download
-│   ├── data/                   # 10,557 examples
-│   └── README.md
-├── qmsum/
-│   ├── run_benchmark.py        # Meeting summarization (ROUGE-L)
-│   ├── download_data.py        # GitHub JSONL + HF fallback
-│   ├── data/                   # 35 meetings, 281 queries
-│   └── README.md
-├── socialiqa/
-│   ├── run_benchmark.py        # MC social reasoning (3 choices)
-│   ├── download_data.py        # HF + parquet fallback
-│   ├── data/                   # 1,954 validation examples
-│   └── README.md
-├── tomi/
-│   ├── run_benchmark.py        # Theory of Mind (EM)
-│   ├── download_data.py        # GitHub + clone/generate fallback
-│   ├── data/                   # 2,988 generated examples
-│   └── README.md
-├── safetybench/
-│   ├── run_benchmark.py        # Safety MC (4 choices, no GT)
-│   ├── download_data.py        # HuggingFace download
-│   ├── data/                   # 11,435 test examples
-│   └── README.md
-├── streamingqa/
-│   ├── run_benchmark.py        # Temporal knowledge QA
-│   ├── download_data.py        # GCS download (requires manual)
-│   └── README.md
-└── rgb/
-    ├── run_benchmark.py        # Noise robustness (EM/F1)
-    ├── download_data.py        # GitHub (currently 404)
-    └── README.md
+├── shared/                      # THE shared infrastructure — read this first
+│   ├── base_runner.py           #   BenchmarkRunner ABC + run() pipeline + LLM helpers
+│   ├── baseline_runner.py       #   Direct-LLM / RAG baselines
+│   ├── graph_evolution_tracker.py
+│   └── stats_utils.py           #   wilson_ci, bootstrap_ci_mean
+├── _utils.py                    # embedding resolution (the canonical embedder factory)
+├── analysis_utils.py            # wrong-case enrichment + save_wrong_cases
+├── compare_fast.py              # classic-vs-fast ingestion A/B
+├── babilong/    cogeval/    locomo/    longmemeval/
+├── musique/     mutual/     narrativeqa/
+├── streamingqa/ tomi/
+└── scripts/plot_graph_evolution.py
 ```
 
-### Config Profiles (configs/)
-
-| File | Benchmark |
-|------|-----------|
-| `configs/locomo_profile.yaml` | LoCoMo |
-| `configs/longmemeval_profile.yaml` | LongMemEval |
-| `configs/msc_profile.yaml` | MSC |
-| `configs/babilong_profile.yaml` | BABILong |
-| `configs/futurex_profile.yaml` | FutureX |
-| `configs/mutual_profile.yaml` | MuTual |
-| `configs/musique_profile.yaml` | MuSiQue-Ans |
-| `configs/timeqa_profile.yaml` | TimeQA |
-| `configs/narrativeqa_profile.yaml` | NarrativeQA |
-| `configs/qmsum_profile.yaml` | QMSum |
-| `configs/socialiqa_profile.yaml` | SocialIQA |
-| `configs/tomi_profile.yaml` | ToMi |
-| `configs/safetybench_profile.yaml` | SafetyBench |
-| `configs/streamingqa_profile.yaml` | StreamingQA |
-| `configs/rgb_profile.yaml` | RGB |
-
-### Core Modules (src/cognifold/)
-
-These are **shared** by all benchmarks. Changes here affect everything.
+Each benchmark dir: `run_benchmark.py` (runner), `download_data.py` (dataset
+fetch), `README.md`. Exceptions: **longmemeval** uses `run_eval.py` +
+`symbolic_resolver.py` + `runs/<label>/` snapshots + `HISTORY.md`;
+**cogeval** generates its dataset via `generate_dataset.py` and ships extra
+baseline/experiment runners.
+
+### Entry points
+
+| What | Command |
+|---|---|
+| Any base-runner benchmark | `bash scripts/reproduce.sh {cogeval,locomo,musique,narrativeqa,tomi,babilong,mutual,streamingqa}` |
+| All of the above | `bash scripts/reproduce.sh all` |
+| LongMemEval | `bash scripts/parallel_longmemeval.sh [N_PARALLEL] [STRATIFIED] [TOTAL_LIMIT] [LABEL]` — or the Claude Code skills `.claude/skills/longmemeval-run` (one-shot) / `longmemeval-iterate` (iteration campaign) |
+| Runner sanity check | `PYTHONPATH=src python test_benchmarks.py` (8 base runners; longmemeval not covered) |
+
+**Datasets are NOT auto-downloaded.** Run `benchmarks/<name>/download_data.py`
+before the first run (a missing dataset exits with an error). LongMemEval is
+the exception — `run_eval.py` self-downloads.
+
+**Reproducing paper numbers**: `reproduce.sh` defaults to `MODEL=openai:gpt-5`;
+the paper stack is `gpt-4o-mini` (build) — set `MODEL=openai:gpt-4o-mini` to
+match published numbers.
+
+### Config profiles (`configs/`)
+
+One `configs/<name>_profile.yaml` per benchmark: `babilong`, `locomo`,
+`longmemeval`, `musique`, `mutual`, `narrativeqa`, `streamingqa`, `tomi`
+(cogeval currently has none — known gap). Prompt changes go in the YAML, not in
+Python. `configs/prompt_profiles.yaml` is the general scenario-profile registry.
+
+### Core modules shared by all benchmarks (`src/cognifold/`)
 
 | File | Role |
 |------|------|
-| `agent/domain.py` | `DomainConfig` per benchmark (LOCOMO_DOMAIN, MSC_DOMAIN, BABILONG_DOMAIN, etc.) |
+| `agent/domain.py` | `DomainConfig` per benchmark + `register_domain()` |
 | `query/agent.py` | `MemoryQueryAgent` — main read interface, `query_for_qa()` |
 | `query/models.py` | `QueryConfig`, `QueryResult`, `NodeSummary` |
 | `query/assembly.py` | Formats nodes into context text |
-| `query/strategies.py` | Query mode implementations (mergefold, rag, base, episodic) |
+| `query/strategies.py` | Query modes (mergefold, rag, base, episodic) |
 | `retrieval/bm25.py` | BM25 retrieval engine |
-| `utils/embeddings.py` | OpenAI embedding service (text-embedding-3-small) |
+| `symbolic/` | Symbolic state/belief trackers used during ingestion + ToMi |
+
+Changes here affect **every** benchmark — smoke-test after touching them
+(see Checklists).
+
+### Environment variables
+
+`.env` keys (see `.env.example`): `OPENAI_API_KEY` / `GOOGLE_API_KEY` required
+for LLM modes. Benchmark-specific:
+
+| Variable | Used by | Meaning |
+|---|---|---|
+| `MODEL` | `reproduce.sh` | reader/build model (`provider:model` syntax) |
+| `PYTHON` | `reproduce.sh` | python binary override |
+| `READER_MODEL` `WRITER_MODEL` `JUDGE_MODEL` `RERANK_MODEL` `EMBED_MODEL` | `parallel_longmemeval.sh` | per-role model overrides |
+| `EMBEDDING_API_KEY` `EMBEDDING_BASE_URL` | embeddings provider | separate key/endpoint for embeddings (defaults to api.openai.com when key set) |
+| `JUDGE_API_KEY` `JUDGE_BASE_URL` | longmemeval | separate judge endpoint |
+| `OPENROUTER_API_KEY` | shell scripts | shim: exported as `OPENAI_API_KEY` + OpenRouter base URL |
+| `HF_ENDPOINT` `GITHUB_MIRROR` | `download_data.py` scripts | mirrors (e.g. hf-mirror.com) |
+| `COGNIFOLD_ABLATE_KNN=1` `COGNIFOLD_ABLATE_MERGE=1` | `base_runner.py` | ablation switches (skip kNN edge inference / concept merge) |
+| `EXTRACT_TYPED_ATTRIBUTES` `RESOLVE_EVENT_DATES` | longmemeval | W1 (default on) / W2 (default off) passes |
+| `QID_LIST_FILE` | `parallel_longmemeval.sh` | run a fixed qid subset |
+
+Model string syntax everywhere: `provider:model` — `openai:gpt-4o-mini`,
+`gemini:gemini-2.5-flash` (bare `gemini-*` also accepted). OpenAI models MUST
+carry the `openai:` prefix so agent dispatch routes correctly
+(`benchmarks/shared/base_runner.py::_normalize_agent_model_name`).
 
 ---
 
-## Checklists
+## Contributing a new benchmark
 
-### When Adding a New Benchmark
+`benchmarks/mutual/run_benchmark.py` (182 lines) is the reference
+implementation — copy its shape. `locomo` and `narrativeqa` predate the shared
+base and duplicate its orchestration; **do not copy them**.
+
+1. **Create the directory**:
 
-1. **Create benchmark directory**:
    ```
    benchmarks/<name>/
    ├── __init__.py
-   ├── run_benchmark.py        # argparse CLI, standard phases
-   ├── download_data.py        # Data download (optional)
-   └── README.md               # Quick setup
+   ├── run_benchmark.py        # subclass benchmarks.shared.base_runner.BenchmarkRunner
+   ├── download_data.py        # dataset fetch (honor HF_ENDPOINT/GITHUB_MIRROR mirrors)
+   └── README.md               # quick setup + one working command
    ```
 
-2. **Create config profile**: `configs/<name>_profile.yaml`
-   - See [docs/benchmark/architecture.md](benchmark/architecture.md) for the profile schema
+   Subclass `BenchmarkRunner` and implement only the three abstract methods —
+   `load_dataset()`, `build_events()`, `evaluate_example()` — plus optional
+   hooks (`add_extra_args`, `post_ingest`, `get_query_config_overrides`, …).
+   **Never copy the `run()` loop**; if `run()` doesn't fit, add a hook to the
+   base instead.
 
-3. **Register domain**: Add `<NAME>_DOMAIN` in `src/cognifold/agent/domain.py`
+2. **Create the profile**: `configs/<name>_profile.yaml` (copy
+   `configs/mutual_profile.yaml` as a starting point; set
+   `profiles.<name>.model.name` and `embedding.model`).
 
-4. **Required CLI flags** (unified across all benchmarks):
-   - `--limit N` — number of samples to process
-   - `--query-mode MODE` — `base` | `rag` | `episodic` | `mergefold` (default: `mergefold`)
-   - `--disable-concepts` — episodic mode (events only, no concepts)
-   - `--no-llm-eval` — disable LLM-based evaluation
-   - `--no-profile` — skip YAML prompt profile
-   - `--visualize` — generate replay HTML
+3. **Register the domain** with `register_domain()` in your runner (or, if it
+   must ship in-tree, add `<NAME>_DOMAIN` in `src/cognifold/agent/domain.py`
+   — but the long-term direction is YAML-registered domains, see
+   `docs/CODE_HEALTH.md` #3).
 
-5. **Required phases in runner**:
-   1. Data loading (with `--limit`)
-   2. Graph initialization (fresh `ConceptGraph` per sample or global)
-   3. Ingestion (event -> agent -> executor -> graph)
-   4. QA evaluation (query_for_qa -> answer generation -> LLM judge)
-   5. Metrics output (accuracy, node count, context length)
+4. **Unified CLI** — the base `main()` already provides `--limit`,
+   `--query-mode {base,rag,episodic,mergefold}`, `--disable-concepts`,
+   `--no-llm-eval`, `--no-profile`, `--visualize`, `--data`, `--embedding`,
+   `--model`. Add benchmark-specific flags via `add_extra_args()` only.
+   (`--judge-model` and `--event-stream` are currently locomo/longmemeval
+   specials, not base flags.)
 
-6. **Update documentation**:
-   - Update `docs/benchmark/status.md` — mark dataset as Done, fill in runner/profile/accuracy
-   - Add row to the **Implementation Status** table in this file
-   - Add entry to `docs/benchmark/results.md` (even if no results yet)
-   - Add to the **File Map** sections in this file
+5. **Wire the hardcoded lists** (all four, or your benchmark is invisible):
+   - `scripts/reproduce.sh` — both the `case` in `run_one` **and** the
+     `all` loop
+   - `test_benchmarks.py` — the `RUNNERS` list
+   - the **Implementation Status** table and **File Map** in this file
 
-### When Modifying an Existing Benchmark
+6. **Outputs**: `run()` writes `benchmarks/<name>/output/benchmark_results.json`
+   and `wrong_cases.json` (both allowlisted in `.gitignore`; everything else
+   under `output/` is ignored). Run from the **repo root** — the output dir is
+   a relative path.
 
-1. **Read the profile YAML first** (`configs/<domain>_profile.yaml`)
-2. **Prompt changes go in the YAML**, not in Python code
-3. **Test with `--limit 1`** before running full datasets (API cost)
-4. **Update `docs/benchmark/results.md`** after any run that produces new results
+7. **Verify cheaply before spending**: `--limit 1` first; then
+   `PYTHONPATH=src python test_benchmarks.py` must pass.
 
-### After Running Experiments
+### When modifying an existing benchmark
 
-1. **Record results** in `docs/benchmark/results.md` with:
-   - Date, model, config used
-   - Accuracy metrics (strict, partial/lenient)
-   - Node count, context length
-   - Command used to reproduce
-2. **Update status table** in this file if accuracy changed significantly
+1. Read `configs/<name>_profile.yaml` first — prompt changes go there, not in code.
+2. Test with `--limit 1` before full runs (API cost).
+3. Record results (date, model, config, command) in the run output and update
+   the status table here if numbers change significantly.
 
-### When Modifying the Query System
+### When modifying the query system
 
-The query system (`src/cognifold/query/`) is shared by all benchmarks. After changes:
+`src/cognifold/query/` is shared by all benchmarks. After changes:
 
 ```bash
 # Quick smoke test
-PYTHONPATH=src python benchmarks/locomo/run_benchmark.py --limit 1 --sessions 1
+PYTHONPATH=src python -m benchmarks.locomo.run_benchmark --limit 1 --sessions 1
 
-# Check retrieval quality
-PYTHONPATH=src python benchmarks/babilong/run_benchmark.py --config 0k --tasks qa1 --limit 5 --no-llm-eval
+# Retrieval quality without LLM cost
+PYTHONPATH=src python -m benchmarks.babilong.run_benchmark --config 0k --tasks qa1 --limit 5 --no-llm-eval
 ```
 
 ---
 
 ## Quick Start
 
-### Environment
-
 ```bash
-# Required for LLM-based modes
-export OPENAI_API_KEY="sk-..."
+# Environment
+export OPENAI_API_KEY="sk-..."          # or put it in .env
+export HF_ENDPOINT=https://hf-mirror.com  # optional mirror
 
-# Optional (HuggingFace mirror, e.g. in China)
-export HF_ENDPOINT=https://hf-mirror.com
+uv sync                                  # install
 
-# Install
-uv pip install -e .
-```
-
-### Verify All Runners
-
-```bash
-# Run the test suite — checks all 15 runners, unified CLI args, and data files
-PYTHONPATH=src python test_benchmarks.py
-```
-
-### Minimal Cost Tests
-
-```bash
-# BABILong - 3 examples, no LLM eval
-PYTHONPATH=src python benchmarks/babilong/run_benchmark.py \
+# Fetch a dataset, then run a minimal-cost test
+PYTHONPATH=src python benchmarks/babilong/download_data.py
+PYTHONPATH=src python -m benchmarks.babilong.run_benchmark \
     --config 0k --tasks qa1 --limit 3 --no-llm-eval
 
-# MSC - 1 conversation
-PYTHONPATH=src python benchmarks/msc/run_benchmark.py --limit 1 --query-mode rag
+# Full paper run for one benchmark
+bash scripts/reproduce.sh musique
 
-# LoCoMo - 1 conversation, 1 session
-PYTHONPATH=src python benchmarks/locomo/run_benchmark.py --limit 1 --sessions 1
+# LongMemEval full N=500 (hours; ~$80–150 on the recommended stack)
+bash .claude/skills/longmemeval-run/scripts/run.sh
 ```
 
-### Full Benchmarks (costs tokens)
-
-```bash
-# BABILong - full agent mode
-PYTHONPATH=src python benchmarks/babilong/run_benchmark.py \
-    --config 0k --tasks qa1,qa2 --limit 20 --query-mode mergefold
-
-# MSC - multiple conversations
-PYTHONPATH=src python benchmarks/msc/run_benchmark.py --limit 10 --query-mode mergefold
-
-# LoCoMo - 2 conversations, all sessions, with visualization
-PYTHONPATH=src python benchmarks/locomo/run_benchmark.py --limit 2 --sessions 5 --visualize
-```
-
----
+## Known gaps (tracked)
 
-## Detail Documentation
+- `cogeval` has no README, no `download_data.py`, no config profile.
+- `longmemeval` is absent from `reproduce.sh` and `test_benchmarks.py`; its
+  flags drift from the unified set (`--llm-eval` vs `--no-llm-eval`).
+- `locomo`, `narrativeqa`, `longmemeval` re-implement the shared `run()`
+  orchestration; `baseline_runner.py` duplicates LLM/metric helpers.
+- Result JSON schemas differ per benchmark; there is no cross-benchmark
+  aggregate file.
 
-- **[Implementation Status](benchmark/status.md)** — progress tracker for all 15 datasets (TODO / Done)
-- **[Architecture & Components](benchmark/architecture.md)** — system design, core components, profile schema, LLM config
-- **[Dataset Catalog](benchmark/dataset-catalog.md)** — all 15 planned datasets across 6 categories
-- **[Experiment Results](benchmark/results.md)** — results, status details, known issues
-- **[Phase 12 Work Log](benchmark/phase12-log.md)** — historical record of Phase 12 development
+Details, evidence, and fix directions: `docs/CODE_HEALTH.md`.
diff --git a/docs/CODE_HEALTH.md b/docs/CODE_HEALTH.md
new file mode 100644
index 0000000..0bf95f6
--- /dev/null
+++ b/docs/CODE_HEALTH.md
@@ -0,0 +1,253 @@
+# Code Health — Structural Findings & Fix Directions
+
+Evidence-based inventory of structural problems in `src/cognifold/` (~40,131
+lines, 150 files) and `benchmarks/` (~19,800 lines), ranked by impact ×
+confidence. Each finding names files/lines, the concrete cost, and a fix
+direction. Companion to `CLAUDE.md` and `docs/BENCHMARK.md`.
+
+Status legend: 🔴 fix before building on top · 🟠 pay down opportunistically ·
+🟡 nice-to-have.
+
+---
+
+## Tier 1 — Highest impact
+
+### 1. 🔴 The test suite is 28 lines for 40K lines of source
+
+`tests/` contains only `test_smoke.py` — 3 assertions (public exports exist,
+`add_node`/`get_node`, `get_node_or_none` miss). **Every package has zero
+tests**: `query/` (4,458 lines), `agent/` (5,730), `service/` (3,640),
+`graph/`, `executor/`, `retrieval/`, `scoring/`, `intent/`, all others.
+`pyproject.toml` ships pytest + pytest-asyncio and CI runs them, so the
+infrastructure exists — the net does not.
+
+**Cost**: every refactoring in this document is unsafe; regressions in a
+40K-line codebase with LLM-nondeterministic behavior go undetected until a
+$100 benchmark run.
+
+**Fix**: land unit tests for the LLM-free, deterministic cores first — they
+are trivially testable: `graph/store.py`, `executor/runner.py`
+(validation/rollback), `scoring/ranker.py`, `retrieval/bm25.py`,
+`query/text_utils.py`, `config.py`. Target: each Tier-1/Tier-2 refactor below
+lands *with* tests for the code it touches.
+
+### 2. 🔴 `query/agent.py` (1,799 lines) hardcodes 14 benchmark names into the core library
+
+`query/agent.py:789-808` is a literal dispatch dict —
+`{"locomo": self._query_locomo_qa, "longmemeval": ..., "msc": ..., ...}` —
+mapping to 14 private `_query_<bench>_qa` methods (lines 811–1261). Several
+differ only by a magic `max_nodes` (40 / 15 / 5 / 30). The `query()` method
+alone is 263 lines; `MemoryQueryAgent` has 44 methods. Half the entries
+(`msc`, `qmsum`, `rgb`, `socialiqa`, `safetybench`, `futurex`) refer to
+benchmarks whose directories no longer exist in the repo.
+
+**Cost**: the shipped library carries leaderboard tuning; adding benchmark #9
+means editing the core query engine; library users get 14 dead branches.
+
+**Fix**: introduce a `QueryProfile` dataclass (max_nodes, boosts, prompt
+template) passed into `query()`; move per-benchmark values into
+`benchmarks/*/` or `configs/*_profile.yaml`; delete the dispatch dict and the
+14 methods.
+
+### 3. 🔴 `agent/domain.py` (1,500 lines): 15 of 21 domains are benchmark configs
+
+Module-level `DomainConfig` literals for LOCOMO, FUTUREX, LONGMEMEVAL, MSC,
+BABILONG, MUTUAL, MUSIQUE, STREAMINGQA, RGB, TIMEQA, NARRATIVEQA, QMSUM,
+SOCIALIQA, TOMI, SAFETYBENCH (lines 419–1310). The extension point
+`register_domain()` (line 1494) already exists and is unused for these. The
+two hardcoded lists have already drifted: `TIMEQA_DOMAIN` exists here with no
+`_query_timeqa_qa` counterpart in `query/agent.py`.
+
+**Fix**: keep the ~6 real product domains in-tree; move benchmark domains to
+YAML under `configs/` (which already holds the per-benchmark profiles) and
+load via `register_domain()` from each runner.
+
+### 4. 🔴 Layering inversion: five core packages import from `service/`
+
+`cognifold.service.llm_keys` is imported by `agent/` (7 sites), `query/` (2,
+incl. `query/llm.py:16`), `utils/` (2), `embeddings/` (2), and
+`config.py:161` — the **root config module depends on the HTTP service
+layer**. Symptom at scale: 287 function-local `from cognifold...` imports
+across `src/` exist to dodge the resulting cycles (worst: `query/agent.py` 16,
+`pipeline/classic.py` 16, `pipeline/layered.py` 15, `executor/runner.py` 15).
+
+**Fix**: `llm_keys.py` has no FastAPI dependency — it is misfiled. Move it to
+`cognifold/llm/keys.py` (or `utils/`), keep a deprecation re-export in
+`service/`, and promote the deferred imports back to top level. One move,
+13+ call sites cleaned.
+
+---
+
+## Tier 2 — Duplication and dead weight
+
+### 5. 🟠 `replay/renderer.py::_build_html` is a 1,328-line function
+
+Lines 115–1442: one f-string containing a complete HTML+CSS+JS application.
+Every literal `{` in the CSS/JS must be doubled (`:root {{`); no JS/CSS
+tooling can lint it. `simulator/visualizer.py` contains a second, independent
+HTML renderer (`_build_sidebar_html` 184 lines, `render` 143 lines).
+
+**Fix**: extract to `replay/templates/replay.{html,css,js}` as package data;
+inject only the JSON payload. Do the same for the simulator, or merge the two
+renderers.
+
+### 6. 🟠 The four LLM generators are copy-paste clones (~400 duplicated lines)
+
+`generator/{event_generator,service_logs,claude_code,computer_activity}.py`
+(2,174 lines total): `_parse_events` is byte-identical in all four except the
+ID prefix (`e-`/`svc-`/`cc-`/`ca-`); the 40-line retry/generate block is
+identical in `claude_code.py:288-327` and `computer_activity.py:324-363`, with
+a fifth copy in `generator/base.py:122-154` — the base class they all inherit
+from and bypass.
+
+**Fix**: hoist `_parse_events(text, date, id_prefix)` and the retry-wrapped
+`generate_day()` into `generator/base.py`; subclasses supply only the prompt
+builder and ID prefix.
+
+### 7. 🟠 ~1,080 lines of provably dead code
+
+Zero references across `src/`, `benchmarks/`, `scripts/`, `tests/`:
+
+| File | Lines | Note |
+|---|---|---|
+| `scoring/hierarchical.py` | 450 | `HierarchicalContextSelector` never instantiated in-repo — **but README's Python quick-start showcases it**. Decide: wire it into the pipeline/service, or delete it *and* fix the README example. |
+| `graph/projection.py` | 347 | re-exported by `graph/__init__.py` but never consumed; also defines `GraphSnapshot`, colliding with a different `GraphSnapshot` in `executor/runner.py:78` |
+| `retrieval/cross_encoder_rerank.py` | 103 | drags a `BAAI/bge-reranker-v2-m3` dependency |
+| `query/probe.py` | 98 | `SymbolicProbeAgent` unused |
+| `graph/metrics_extended.py` | 87 | not even exported |
+| `agent/prompts.py:631,691` | ~80 | two formatters that only render the dead `HierarchicalContext` |
+
+### 8. 🟠 Two parallel embedding stacks; the deprecated one is on a hot path with a key-leak hazard
+
+`cognifold/embeddings/` is the real stack (provider ABC + factory).
+`utils/embeddings.py` (122 lines) raises a `DeprecationWarning` at import yet
+is still called from the core executor (`executor/runner.py:301,310,575,580`).
+It is hardcoded to `text-embedding-3-small` behind a **process-global
+singleton** that caches a client built from whichever API key was live first —
+in the multi-session service this is a cross-session key/model leak, and it
+ignores `EmbeddingConfig`.
+
+**Fix**: rewrite the 4 call sites in `executor/runner.py` against
+`cognifold.embeddings.create_provider()`; delete `utils/embeddings.py`.
+
+### 9. 🟡 Four stopword sets, four cosine implementations
+
+Stopwords: `graph/entity_index.py:24`, `retrieval/bm25.py:32`,
+`query/text_utils.py:15`, `query/agent.py:50`. Cosine:
+`graph/store.py:561`, `query/agent.py:166`, `embeddings/search.py:428`,
+`graph/metrics_extended.py:62`. They will drift.
+
+**Fix**: one `utils/text.py` (tokenize + stopwords) and one
+`utils/vector.py::cosine`.
+
+---
+
+## Tier 3 — Service layer & configuration
+
+### 10. 🔴 Session LLM API keys are persisted in plaintext
+
+`service/session.py:346` (`_persist_session`) and `:365` include
+`"llm_api_keys": session.llm_api_keys` in the persisted blob — written as
+plain JSON to `sessions/<id>/session.json`, Redis, and Supabase. Clients post
+real keys in the session-create body (`cli/client.py:756-767`).
+
+**Fix**: strip `llm_api_keys` from the persisted payload (keep in-memory
+only), or encrypt at rest and re-supply per request.
+
+### 11. 🔴 No per-user authorization — any valid API key reads any session
+
+`service/auth.py` (37 lines) is the whole model: a flat key set returning the
+key string. `Session.user_id` exists (`session.py:38`) but no route checks it;
+all 20+ `get_session(session_id)` call sites across `routes/` fetch by ID
+alone. Auth defaults to **off** (`AppSettings.api_keys=None`, and
+`cli/serve.py:92` warns then proceeds).
+
+**Fix**: auth dependency returns a principal; add a shared
+`get_owned_session(session_id, principal)` dependency used by every route.
+
+### 12. 🟠 `--workers` is a silent no-op; session TTL is dead config
+
+`cli/serve.py:138-144` passes the app **object** to `uvicorn.run`, so
+`workers=` is ignored — `cognifold serve --workers 8` runs one process, no
+error. `SessionManager.cleanup_expired()` (`session.py:313`) has zero callers,
+so `COGNIFOLD_SESSION_TTL_HOURS` does nothing. The `get_session` recovery path
+(`session.py:228`) mutates `self._sessions` without the lock and bypasses the
+`max_sessions` eviction check. `--session-backend` offers `file|redis` while
+the factory also supports `supabase` (env-only).
+
+**Fix**: pass `"cognifold.service.wsgi:app"` as an import string; start a
+lifespan task that periodically calls `cleanup_expired()`; take the lock in
+the recovery branch; align the CLI choices with the factory.
+
+### 13. 🟠 Config sprawl: 6 config modules, 2 conflicting YAML defaults, 4 default model names
+
+Six config modules totaling ~960 lines (`config.py` 477, plus
+`query/retrieval/agent/embeddings/cli` configs). `config.example.yaml` says
+`gemini-3-flash-preview` / `max_nodes: 50`; `configs/cognifold.yaml` says
+`openai:gpt-5.2` / `max_nodes: 20`; neither is loaded by default. Default
+model names in code: `config.py:17` `gemini-2.5-flash`, `agent/config.py:29`
+`gemini-3-flash-preview`, `query/llm.py:101` `gpt-4o`, `query/models.py:92`
+`openai:gpt-5`. The CLI default retrieval mode is `legacy`
+(`cli/query.py:100`) while the library default is `HYBRID`
+(`query/models.py:102`).
+
+**Fix**: one `pydantic-settings` root (`env_prefix="COGNIFOLD_"`) replaces the
+hand-rolled `from_env` walrus chains (`config.py:200-260`); keep exactly one
+example YAML; define ONE default-model constant.
+
+### 14. 🟡 Silent cross-provider fallback in `query/llm.py`
+
+An OpenAI failure is logged at `logger.debug` and silently retried on Gemini
+with a **different model** (`query/llm.py:113,143`) — a bad OpenAI key
+produces plausible answers from the wrong model with no visible signal.
+
+**Fix**: log at `warning` with the substituted model, and surface the
+fallback in the result metadata.
+
+---
+
+## Benchmarks-side duplication (companion to docs/BENCHMARK.md "Known gaps")
+
+- `locomo/run_benchmark.py` (1,102 lines) bypasses `BenchmarkRunner` and
+  re-defines `check_api_keys` / `call_llm_for_eval` / `evaluate_with_llm` +
+  an 80-line argparse copy. `narrativeqa` subclasses but copy-pastes the
+  entire `run()` loop (lines 123–297). `longmemeval/run_eval.py` (2,390
+  lines) reuses nothing.
+- `shared/baseline_runner.py` (1,229 lines) duplicates provider detection,
+  LLM calls, F1/EM, and `wilson_ci_95` (vs `stats_utils.wilson_ci`); its
+  registry still lists removed benchmarks (socialiqa, rgb, qmsum, safetybench).
+- Two embedder factories: `benchmarks/_utils.py` (canonical, used by
+  base_runner) vs `benchmarks/embedding_utils.py` (legacy; its docstring still
+  claims "All benchmark runners should use this"). Delete the legacy one.
+- `compute_f1`/`normalize_answer` re-implemented in `musique`, `streamingqa`,
+  `narrativeqa`, and `baseline_runner` — move to `shared/stats_utils.py`.
+- Root leftovers to remove: `my_prompt.md` (self-declared moved to skills),
+  `history.md` (superseded by `benchmarks/longmemeval/HISTORY.md`),
+  `generate_demo.py` (unreferenced, unlinted).
+
+---
+
+## Checked and NOT a problem (do not re-litigate)
+
+- **`symbolic/` is not dead** — used by `benchmarks/shared/base_runner.py`,
+  `benchmarks/tomi/`, and `query/agent.py:497`. It is benchmark-coupled
+  (finding #2) but live.
+- **`pipeline/` classic vs layered** — both alive by design: `classic.Pipeline`
+  is the public export + MCP path; `LayeredPipeline` serves CLI run + the
+  service. Two execution models is a cost, not accidental duplication.
+- **Exception discipline** — 78 `except Exception`, 0 bare `except`, only 9
+  silent swallows (mostly cost telemetry). Acceptable; #14 is the one real
+  case.
+- **TODO/FIXME density** — trivially low (max 2 per file).
+- **`service/llm_keys` thread-local design** — correct as written; the global
+  client caches in `query/llm.py:24-25` are keyed by key value, deliberately.
+
+## Suggested sequencing
+
+1. #1 tests for deterministic cores (unblocks everything)
+2. #4 move `llm_keys` (small, mechanical, removes the inversion)
+3. #7 delete dead code + README fix (pure win, shrinks the surface)
+4. #8 retire the deprecated embedding stack (closes the key-leak hazard)
+5. #2 + #3 de-benchmark the core library (the big one; do behind tests)
+6. #10/#11 before any real multi-user deployment
+7. #5, #6, #9, #12, #13, #14 opportunistically