fix(llm): #896 expose max_generation_tokens, so the prompt budget stops flooring to zero - #898
Merged
Conversation
…ps flooring to zero
`prompt_budget.reply_reserve` reads `llm.max_generation_tokens` to decide how
much of the context window to hold back for the reply. `OllamaClient` exposed
`context_window` as a property but never that one -- the value was stored as
`self._max_generation_tokens` and never surfaced -- so the read always missed
and the 8192 default came back regardless of what the workspace pinned.
With the shipped defaults this is invisible, because the fallback happens to
EQUAL the pin (`DEFAULT_MAX_GENERATION_TOKENS` is 8192). Lower the ceiling and
`budget_chars` floors to zero, `fair_shares` allows every block nothing, and
`query` answers NO_MATCH on a fully populated bundle.
Measured across every configuration `read_config` accepts, before -> after:
max_gen min legal context_window reserve seen budget
8192 12288 8192 -> 8192 8440 -> 8440
4096 8192 8192 -> 4096 0 -> 8440
2048 6144 8192 -> 2048 0 -> 8440
1024 5120 8192 -> 1024 0 -> 8440
16384 20480 8192 -> 16384 30720 -> 8440
The last row was wrong in the other direction: it reserved less than the
ceiling actually consumes, budgeting more prompt room than the window can
hold. One missing property caused both.
- `OllamaClient.max_generation_tokens` is now a read-only property, the same
shape and the same read-the-real-send rationale as `context_window`.
- Two tests, written first and failing first (`assert 8192 == 4096`). They
assert the PAIR through `prompt_budget`, at a non-default ceiling:
`test_context_window_property_exposes_the_configured_window` passed for the
whole time its sibling was broken, so a test on one value alone is not
evidence about the other.
- `AnswerResult.omitted_titles` claimed `read_config`'s floor kept the shipped
CLI path away from a zero budget. The floor is real but the reserve was not
read from it, so the claim held only for the default row. Corrected to state
the guarantee that now actually holds.
The issue's third suggestion -- make a zero budget loud -- needed no work and
none was done. #882's omission notice names every dropped document and is
emitted BEFORE the no-match return, and `tests/unit/cli/test_query.py` already
pins that on a `NO_MATCH` result.
Found while measuring #887, which needed a bounded arm at a small window and
got 60 of 60 no-matches instead.
Review receipt review-dc405d396fd8ffa7, APPROVED, zero findings.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
prompt_budget.reply_reservereadsllm.max_generation_tokensto decide howmuch of the context window to hold back for the reply.
OllamaClientexposedcontext_windowas a property but never that one — the value was stored asself._max_generation_tokensand never surfaced — so the read always missedand the 8192 default came back regardless of what the workspace pinned.
With the shipped defaults this is invisible, because the fallback happens to
equal the pin. Lower the ceiling and
budget_charsfloors to zero,fair_sharesallows every block nothing, andqueryanswersNO_MATCHon afully populated bundle.
Related issue
Closes #896
Type of change
fix— bug fixMeasured, before → after
Every row below is a configuration
read_configaccepts(
minimum_context_window(n)isPROMPT_CONTEXT_ALLOWANCE + n):max_generation_tokenscontext_windowOnly the default row was correct, and it was correct by coincidence. The last
row was wrong in the other direction — it reserved less than the ceiling
actually consumes, budgeting more prompt room than the window can hold. One
missing property caused both.
Observed live while working #887: a bounded arm pinned to
num_ctx 4096/num_predict 2048returned 60 of 60 no-matches withomitted=5,excerpted=0, and noprompt_eval_countat all —answer()short-circuitsbefore
llm.chat, so the model was never called.Changes
src/openkos/llm/ollama.pymax_generation_tokensread-only property, mirroringcontext_window.tests/unit/llm/test_ollama.pyprompt_budgetat a non-default ceiling.src/openkos/retrieval/answer.pyomitted_titlesnote, which documented a guarantee that did not hold.How was this tested
uv run pytest -q— 5763 passed, 1 skipped (5761 before; +2 new)uv run ruff check ./ruff format --check ./uv run mypy .— cleanBoth tests were written before the fix and failed for the intended reason:
They assert the pair through
prompt_budget, at a non-default ceiling, onpurpose:
test_context_window_property_exposes_the_configured_windowpassedfor the entire time its sibling was broken, so a test on one value alone is
not evidence about the other.
Notes for reviewers
Two things I deliberately did not do:
needed no work. query sends unbounded context and files provenance the model never read #882's omission notice already names every dropped
document and is emitted before the no-match return, and
tests/unit/cli/test_query.pyalready pins that on aNO_MATCHresult. Asecond assertion of the same behavior would have been duplication, so I
verified it and left it alone.
is byte-identical to before; only non-default ceilings move, and they move
to the value the caller asked for.
The
omitted_titlesnote used to sayread_config's floor kept the shippedCLI path away from a zero budget. The floor is real, but the reserve was not
read from it, so the claim held only for the default row. It now states the
guarantee that actually holds and why.
Review receipt
review-dc405d396fd8ffa7, APPROVED, zero findings,inspection completed over all three manifest paths.