eval(attribution): #887 measure the context-size regime the probe never entered

[jasonssdev]

Summary

evals/query_attribution measured es-long at 30/30, and then both wild
Spanish long answers dropped the USED: line. Its documents are constructed
concepts, median 170 chars, max 1,656; the wild retrieval set held two Source
documents of 55,403 and 57,116 chars. #871 closed the ANSWER-length regime and
left the CONTEXT-length one open — the same methodological miss, one level up.

This adds the missing axis, measures it, and refutes the issue's hypothesis.

Related issue

Closes #887

Type of change

• test — measurement harness only; no shipped code changes

What was measured

qwen3:8b, two pooled 3-run sweeps, n=30 per cell, 240 answers, 0 failures and
0 no-matches. Arm baseline (production untouched), identical pins in both.

cell	small	large
es-short	1.00	1.00
es-long	0.93	0.97
en-short	0.80	0.87
en-long	0.80	1.00
rung	0.883	0.958

rung	prompt tokens (median)	context sent (median)	blocks excerpted
small	777	3,184 chars	0.0
large	2,348	9,704 chars	2.4

Three times the prompt, most of the retrieval set clipped away, 8–9 elision
markers — and no cell loses ground. Post-#882 the size regime is not where this
defect lives. The non-reported answers stay question-shaped, exactly as #871
found.

What actually broke in the wild

#887 named the missing evidence precisely: "proving that needs to know where
the cut falls relative to the attribution instruction." --unbounded sends it.

	bounded	unbounded
compliance	1.00 / 0.97 (es)	0.21 (es, n=28) · 0.22 (en, n=9)
context sent	9,704 chars	26,671 chars
prompt tokens	2,348	2,050
chars per prompt token	4.13	13.01
blocks cited	—	4.68 of 5
nomatch	0	0

prompt_eval_count pinned at 2,050 while sent_chars climbs to 26,671 is the
truncation itself. llama.cpp keeps head tokens plus the last half of the
window; the system prompt is the FIRST message and is where the USED:
instruction lives, so under overflow the model is never told to emit the line.
nomatch is 0, so this is not a groundedness collapse — the model answered at
length and the conservative fallback then cited 4.68 of 5 blocks it never
read, which is #882's false-provenance defect reproduced.

Two qualifications are in the README: the unbounded rate is an over-estimate
(two capped generations were dropped, and a capped reply cannot reach its
closing line), and the arms are not at matched pins because of #896 below.

Changes

File	Change
evals/query_attribution/attribution_large_sources.py	New. Full-length ES/EN transcripts of the same three meetings.
evals/query_attribution/attribution_corpus.py	CONTEXTS axis, DOCS_BY_CONTEXT_LANGUAGE, DECISIVE_ANCHORS.
evals/query_attribution/run_query_attribution_probe.py	Context rung in the sweep; measured prompt_eval_count per row; --context, --context-window, --max-generation-tokens, --unbounded; chars/ptok truncation column; frontmatter fix; extended --self-test.
evals/query_attribution/README.md	Rewritten as Part 1 (#871) / Part 2 (#887).
evals/query_attribution/results/*	Four stored emissions.

Two defects found while doing this

#895 (P2) — _write_corpus hand-rolled its frontmatter and interpolated the
title unquoted, so the four Decisión: ... documents were invalid YAML,
counted skipped by reindex, and absent from every run including the
stored 2026-08-25 ones. The probe measured ten documents while documenting
fourteen. Production was never affected (okf.dump_frontmatter quotes
correctly); the harness had a second renderer beside the shipped one. Fixed
here; query_citation, query_entailment and query_sufficiency still carry
it. This is why Part 1's numbers and Part 2's are not comparable.

#896 (P1) — a bounded arm at num_ctx 4096 returned 60 of 60 no-matches
with every block omitted. OllamaClient never exposes
max_generation_tokens, so prompt_budget.reply_reserve always falls back to
8192; any workspace that lowers the ceiling gets budget zero and query
answers NO_MATCH on a full bundle. Its emission is kept as evidence.

How was this tested

• uv run pytest -q — 5761 passed, 1 skipped
• uv run ruff check . / ruff format --check . / uv run mypy . — clean
• uv run python evals/run_self_tests.py — 36 of 36
• The probe's own --self-test now pins, for free and against the production
  bound: the small rung is sent whole; the large rung is excerpted with every
  decisive anchor surviving; the excerpt's shape converges from 8x to 64x
  document size; the authored rung's drift from that converged prompt stays
  under _CONVERGENCE_MARGIN; and every materialized document parses.
• Every new check was mutation-tested. The anchor check fails in 8 of 12
  demotion positions; the indexability check names all four dropped documents
  when the old writer is restored; the convergence check refuted two earlier,
  stronger wordings of its own claim before settling on the true one.

Notes for reviewers

Two things I would push back on if I were reviewing:

• The axis is not size in a vacuum. A document cannot be made eight times
  larger without its content changing. large means "a full-length Source
  document of the same meeting" — what the wild retrieval set held — not "the
  same text, longer". Knowledge and question set are held constant; the prose
  around them is not, and no design could hold it constant. Stated in the
  README.
• The leading agreements summary is a deliberate control, not decoration.
  chunk_lines packs greedily from the start, so only window 0 is a guaranteed
  survivor; a pre-flight showed two of three closing-recap anchors excerpted
  away. Without a surviving statement of the decisions the large cells collapse
  into refusals and the probe measures groundedness under the name of
  attribution.

Review receipt review-d897719acc9056c9, APPROVED, no correction consumed.
Both reviewer findings were WARNING and are adjudicated in the captured
evidence: the table-separator one is refuted (both rows carry six columns;
it also named the wrong lines), and the _chars_per_token one is accurate but
its implied remedy would introduce a ZeroDivisionError — the truthy filter
guards prompt_tokens == 0.