fix(retrieval): #871 anchor the attribution line against omission — measured across language and length - #880
Merged
Conversation
…easured across language and length
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #871
What this ships
One anchor sentence appended to the query prompt's attribution instruction (
retrieval/answer.py) — the_LANGUAGE_ANCHORone-sentence shape — plus the newevals/query_attribution/harness that measured it: mirrored ES/EN corpora (the Spanish half imported fromquery_grounding, the English half a document-by-document translation) crossed with a short/long question regime, so language and answer length stop being confounded.The issue's hypothesis is refuted — and the fix ships anyway
#871 reported Spanish answers dropping the
USED:line (2 of 2 Spanish absent, 1 of 1 English present) and itself warned that a 3-answer sample is a signal, not a rate. The sweep (two pooled 3-run sweeps per arm, n=30 per cell, qwen3:8b, production client settings):es-shortes-longen-shorten-longOmission is question-shaped, not language-shaped: the worst baseline cell is English short answers, and absence concentrates in one-line answers and the longest structured enumerations. The reported
--saveregime (Spanish long) is real at 0.83 — just not language-caused. The anchor was revised accordingly before measuring (it names the length extremes as well as language) and moved every cell non-negatively; es-long went to 30/30.TDD
test_the_attribution_instruction_anchors_against_omissionwritten RED first, GREEN after the append, mutation-verified (anchor removed → FAILED; byte-exact restore +__pycache__purge → green). Adoption verified byte-identical to the measured treatment sentence. Full suite 5653 passed / 1 skipped; 35/35 harness self-tests (the new probe is discovered).Review
Receipt
review-9647416a62d9771cAPPROVED (medium risk, 1 reliability lens, zero corrections consumed). Rounds 1–2 found six real defects in the new harness, all fixed pre-capture on fresh candidates: a dead self-test check, an over-claiming corpus docstring, a checkpointrunscounter that regressed across the language loop,--runs 0reading as a successful empty sweep, a stored-artifact schema note, and a missingnever translatepin. Round 3's three findings declined with disclosure:attribution_prompts' import-time check, which every harness self-test run exercises in CI.runskey (SUGGESTION): stored emissions are never rewritten; the divergence is documented at the writer with the fact that both sweeps completed, so the legacy field is accurate in those files.What remains, disclosed
A question-shaped English residual (en 0.77–0.83). The conservative fallback holds for it:
absentcites the whole retrieval set, the CLI notices it, and--saveroutes through the unverified-provenance consent gate — disclosure friction, not silent bad provenance.