fix(retrieval): #882 bound the query context to the model's window and stop citing what it never read - #893
Merged
Merged
Conversation
…d stop citing what it never read `query` built its prompt from the full body of every retrieved document with no size bound and no disclosure. Ollama does not raise on an oversized prompt -- it discards the overflow and answers normally -- so `--save` filed every retrieved citation as provenance verbatim, including documents the model had never read. That is a false provenance claim on disk, and no surface (`query`, `lint`, `status`) could reveal it. Measured live against qwen3:8b at num_ctx 12288, same corpus both arms: no bound 208,222 chars -> prompt_eval_count 6146 bounded 8,410 chars -> prompt_eval_count 2449 At the ratio the bounded arm establishes (3.434 chars/token) the unbounded prompt should evaluate ~60,634 tokens, so 54,488 of them -- 89.9% -- were being discarded in silence. The bounded arm's chars and tokens agree, so the whole prompt is read. #888 chunked the embedding side but `VectorStore.query` collapses to one hit per document, so the retrieval unit stayed the whole document and this defect survived it intact. - New leaf `prompt_budget` owns the bound: planning window, char budget, even-coverage excerpt, and a water-filling split of one window across competing blocks. #866's `_bounded_prompt_source` becomes a thin wrapper over the same renderer so ingest and retrieval cannot drift apart. - `Citation.excerpted` and `AnswerResult.excerpted_titles` record what was SENT, not what was cited: the attribution filter would otherwise erase the disclosure exactly when the answer cited nothing. - A block left no room at all is dropped from the prompt AND the citation list, and reported separately -- a bare label is still a numbered block the model can cite. The disclosure travels on the no-match return too, or an entirely dropped context reads as "the bundle has nothing". - `query` names the affected documents on stderr, marks partial citations `[partial]`, and discloses both cases in the `--save` plan before the gate. Review receipt review-dc71e9c4ba1743da, four lenses, APPROVED with zero corrections consumed. Three earlier candidates were fixed pre-capture: a `fair_shares` invariant violation where an empty body took the only available char and starved a larger block; an order-dependence that contradicted its own docstring; and a vacuous test of mine that grepped for a word the reworded notice no longer contained. Known and declared: the excerpt search is O(n^2) in windows -- 0.3 ms at 55K chars, 434 ms at 2M. Left as is because the fit is not strictly monotonic in window count, so a binary search could return a smaller excerpt than one that fits. Closes #882
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The defect
openkos querybuilt its prompt from the full body of every retrieved document with no size bound and no disclosure. Ollama does not raise on an oversized prompt — it discards the overflow and answers normally — so--savefiled every retrieved citation as provenance verbatim, including documents the model had never read. That is a false provenance claim on disk, and no surface (query,lint,status) could reveal it.#888 chunked the embedding side, but
VectorStore.querycollapses to one hit per document, so the retrieval unit stayed the whole document and this defect survived it intact.Measured live, not asserted
Same corpus, same
num_ctx: 12288, realqwen3:8b:prompt_eval_count6146is the exact truncation constant #866 recorded. At the ratio the bounded arm establishes (3.434 chars/token) the unbounded prompt should evaluate ~60,634 tokens, so 54,488 of them — 89.9% — were discarded in silence. The bounded arm's chars and tokens agree, so the whole prompt is read.What changed
src/openkos/prompt_budget.pyowns the bound: planning window, char budget, even-coverage excerpt, and a water-filling split of one window across competing blocks. extraction: the judge died unparseable (no-json) on 3 of 3 chunked Spanish sources, and the runs cost ~8 minutes each #866's_bounded_prompt_sourcebecomes a thin wrapper over the same renderer, so ingest and retrieval cannot drift apart — the two-renderer failure curate renders merged-body contradictions as a self-pair, dropping the diagnosis and the unmerge remedy #883 had just closed elsewhere.Citation.excerptedandAnswerResult.excerpted_titlesrecord what was sent, not what was cited: query: the model answers conceptual questions from its own knowledge and still attaches citations #753's attribution filter would otherwise erase the disclosure exactly when the answer cited nothing.[concept_id: …]label is still a numbered block the model can cite. The disclosure travels on the no-match return too, or an entirely dropped context reads to the operator as "the bundle has nothing".querynames the affected documents on stderr, marks partial citations[partial], and discloses both cases in the--saveplan before the confirmation gate.Review
Receipt
review-dc71e9c4ba1743da, four lenses, APPROVED with zero corrections consumed. Three earlier candidates were fixed pre-capture:fair_sharesinvariant violation (CRITICAL) —fair_shares([0, 100], budget=1)returned[1, 0], spending the only available char on an empty body and starving the larger block;[7, 9, 100]at budget 11 gave[4, 4, 3], the same multiset reversed gave[3, 4, 4]. Fixed in the code rather than weakened in the docs;Declared, not fixed
The excerpt search is O(n²) in windows: 0.3 ms at 55K chars, 27 ms at 500K, 434 ms at 2M. Left as is — the fit is not strictly monotonic in window count (adjacency changes the join cost), so a binary search could return a smaller excerpt than one that fits. Two lenses raised it; the measurement is in the docstring.
Verification
5747 passed, 1 skipped · coverage 96.79% (gate 90) ·
ruff check,ruff format --check,mypy .clean over 275 files · 36/36 harness self-tests.Closes #882