fix(retrieval): #888 chunk-backed vectors — a document's embedding is no longer its first half

[jasonssdev]

Closes #888

The defect, in one measurement

state/reindex.py composed title+description+tags+full body and sent it in a single embedder.embed([text])[0] call. bge-m3 caps at 8192 tokens. Embedding a 56 KB source whole, then each half, and comparing:

cos(full, FIRST half)  = 1.0000     <- exactly identical
cos(full, SECOND half) = 0.6582

The full document's embedding was its first half's embedding. Not degradation — absence. In the 3-source test bundle that made 47% and 49% of the two long transcripts unreachable by dense retrieval, while FTS (no length cap) indexed them fully. The 10 FTS + 10 dense fusion was silently mixing a complete lexical view with a half-blind semantic one.

After this change, on the same document: 1.0000 -> 0.9183.

Approach

Chunks become the stored unit; the document stays the unit every consumer sees. VectorStoreDB.query() collapses chunk hits to one VecHit per concept_id before returning, and neighbors() reads a derived doc_vectors table holding normalize(mean(normalize(chunk_i))).

That choice is why the diff is small:

src/openkos/cli/main.py          |  92 ++++----
src/openkos/state/reindex.py     | 146 +++++++++---
src/openkos/state/vectorstore.py | 276 ++++++++++++++++++-----

Zero production diff in retrieval/answer.py, retrieval/fusion.py, graph/proximity.py. Citation, _split_attribution's positional mapping, --save provenance and _assemble_context's sensitivity re-check are untouched because the collapse happens below them.

Also fixed here, because the migration triggers it: reindex printed embedding model changed (bge-m3#compose-v1 -> bge-m3) — the stored effective tag against the bare configured model, claiming a model change nobody made. It now names the real trigger.

Migration

vectors.db is dropped and recreated on open when a legacy-shape store is detected, and vector_meta is cleared — without that clear, dropped vectors plus a surviving hash cache read as cache hits forever and the store stays permanently empty. Users run openkos reindex once. Measured cost on the 32-document bundle: 40 embed calls, 1.25x — only the two long documents chunk, 5 chunks each.

Evidence

• uv run pytest unpiped: 5687 passed, 1 skipped. mypy clean over 273 files. ruff clean.
• vec0 0.1.9 spike run against the installed extension: accepts chunk_index INTEGER, returns it from a KNN projection, and DELETE FROM vectors WHERE concept_id = ? removes all N chunk rows in one statement.
• New gate evals/pair_nomination/ (design D9). Nothing previously measured live candidate-pair nomination — edge_typing and contradictions score classifiers over fixed fixtures. Baseline captured on pre-change code before the schema commit.
• Every safety-critical seam was mutation-tested: the bug was injected, the test confirmed red for the right reason, then reverted.

The gate failed first, and why it was the fixture

It reported post -0.0820 < pre -0.0489. Decomposing showed the whole move came from one "unrelated" pair getting closer: sources/transcription1 vs decisions/necesidad-de-feedback-de-los-tutores. Those are the 4 Aug and 18 Aug sessions of the same recurring meeting — labelled unrelated on provenance grounds, while the fixture's own criterion says unrelated pairs "cross clearly distinct topic domains". Chunking made that transcript represent the whole meeting instead of half of it, so it legitimately moved closer to what the same team decided two weeks later. The gate was penalising the fix for working.

Both pairs of that shape were removed, not only the one that broke the verdict — correcting just the failing pair would be fitting the fixture to the outcome. The revision is disclosed in pair_labels.json's _revision, in the comparison file, and in commit 5b596c2. Rescored: pre -0.0328, post -0.0298, PASS.

Both margins remain negative. This signal does not separate related from unrelated pairs and did not before. The gate asserts only that chunking did not make it worse.

Known open items, none hidden

• Task 5.2 is deliberately unrun: re-measuring edge_typing / contradictions / query_identity. All three are structurally insensitive here — the first two score classifiers over fixed fixtures that never depended on live nomination, and query_identity measures the question-vector space this change does not touch.
• The pair-nomination signal still does not separate. Pre-existing, not a regression.
• ~1,995 authored lines against this repo's 400-line review convention, hence size:exception. The remainder of the diff is SDD artifacts and eval goldens.

Related: #882 (retrieval context budget) and #887 (attribution probe regime) — both get cheaper once chunk-sized units exist, and #887 is worth re-measuring afterwards.

Full SDD trail in openspec/changes/archive/2026-08-26-retrieval-chunking/.

🤖 Generated with Claude Code

https://claude.ai/code/session_01MZz1zEMqg4X5mDEwwfC6dj