Skip to content

fix(retrieval): #871 anchor the attribution line against omission — measured across language and length - #880

Merged
jasonssdev merged 1 commit into
mainfrom
fix/871-spanish-attribution-sweep
Aug 25, 2026
Merged

fix(retrieval): #871 anchor the attribution line against omission — measured across language and length#880
jasonssdev merged 1 commit into
mainfrom
fix/871-spanish-attribution-sweep

Conversation

@jasonssdev

Copy link
Copy Markdown
Owner

Closes #871

What this ships

One anchor sentence appended to the query prompt's attribution instruction (retrieval/answer.py) — the _LANGUAGE_ANCHOR one-sentence shape — plus the new evals/query_attribution/ harness that measured it: mirrored ES/EN corpora (the Spanish half imported from query_grounding, the English half a document-by-document translation) crossed with a short/long question regime, so language and answer length stop being confounded.

The issue's hypothesis is refuted — and the fix ships anyway

#871 reported Spanish answers dropping the USED: line (2 of 2 Spanish absent, 1 of 1 English present) and itself warned that a 3-answer sample is a signal, not a rate. The sweep (two pooled 3-run sweeps per arm, n=30 per cell, qwen3:8b, production client settings):

cell baseline treatment (ADOPTED)
es-short 1.00 1.00
es-long 0.83 1.00
en-short 0.63 0.77
en-long 0.77 0.83
overall 0.81 0.90

Omission is question-shaped, not language-shaped: the worst baseline cell is English short answers, and absence concentrates in one-line answers and the longest structured enumerations. The reported --save regime (Spanish long) is real at 0.83 — just not language-caused. The anchor was revised accordingly before measuring (it names the length extremes as well as language) and moved every cell non-negatively; es-long went to 30/30.

TDD

test_the_attribution_instruction_anchors_against_omission written RED first, GREEN after the append, mutation-verified (anchor removed → FAILED; byte-exact restore + __pycache__ purge → green). Adoption verified byte-identical to the measured treatment sentence. Full suite 5653 passed / 1 skipped; 35/35 harness self-tests (the new probe is discovered).

Review

Receipt review-9647416a62d9771c APPROVED (medium risk, 1 reliability lens, zero corrections consumed). Rounds 1–2 found six real defects in the new harness, all fixed pre-capture on fresh candidates: a dead self-test check, an over-claiming corpus docstring, a checkpoint runs counter that regressed across the language loop, --runs 0 reading as a successful empty sweep, a stored-artifact schema note, and a missing never translate pin. Round 3's three findings declined with disclosure:

  • Substring pinning (WARNING) and anchor position unpinned (SUGGESTION): the pin-by-load-bearing-claims contract is this repo's established convention for all three judge/prompt rubric tests; the byte-exact reproducibility guard lives in attribution_prompts' import-time check, which every harness self-test run exercises in CI.
  • Stored artifacts carry the legacy runs key (SUGGESTION): stored emissions are never rewritten; the divergence is documented at the writer with the fact that both sweeps completed, so the legacy field is accurate in those files.

What remains, disclosed

A question-shaped English residual (en 0.77–0.83). The conservative fallback holds for it: absent cites the whole retrieval set, the CLI notices it, and --save routes through the unverified-provenance consent gate — disclosure friction, not silent bad provenance.

@jasonssdev
jasonssdev merged commit 6f4bf50 into main Aug 25, 2026
7 checks passed
@jasonssdev
jasonssdev deleted the fix/871-spanish-attribution-sweep branch August 25, 2026 03:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

query: long Spanish answers omit the attribution line, so every Spanish --save files unverified provenance

1 participant