Skip to content

fix(llm): #896 expose max_generation_tokens, so the prompt budget stops flooring to zero - #898

Merged
jasonssdev merged 1 commit into
mainfrom
fix/ollama-exposes-max-generation-tokens-896
Aug 27, 2026
Merged

fix(llm): #896 expose max_generation_tokens, so the prompt budget stops flooring to zero#898
jasonssdev merged 1 commit into
mainfrom
fix/ollama-exposes-max-generation-tokens-896

Conversation

@jasonssdev

Copy link
Copy Markdown
Owner

Summary

prompt_budget.reply_reserve reads llm.max_generation_tokens to decide how
much of the context window to hold back for the reply. OllamaClient exposed
context_window as a property but never that one — the value was stored as
self._max_generation_tokens and never surfaced — so the read always missed
and the 8192 default came back regardless of what the workspace pinned.

With the shipped defaults this is invisible, because the fallback happens to
equal the pin. Lower the ceiling and budget_chars floors to zero,
fair_shares allows every block nothing, and query answers NO_MATCH on a
fully populated bundle.

Related issue

Closes #896

Type of change

  • fix — bug fix

Measured, before → after

Every row below is a configuration read_config accepts
(minimum_context_window(n) is PROMPT_CONTEXT_ALLOWANCE + n):

max_generation_tokens min legal context_window reserve seen budget
8192 (default) 12288 8192 → 8192 8440 → 8440
4096 8192 8192 → 4096 0 → 8440
2048 6144 8192 → 2048 0 → 8440
1024 5120 8192 → 1024 0 → 8440
16384 20480 8192 → 16384 30720 → 8440

Only the default row was correct, and it was correct by coincidence. The last
row was wrong in the other direction — it reserved less than the ceiling
actually consumes, budgeting more prompt room than the window can hold. One
missing property caused both.

Observed live while working #887: a bounded arm pinned to num_ctx 4096 /
num_predict 2048 returned 60 of 60 no-matches with omitted=5,
excerpted=0, and no prompt_eval_count at all — answer() short-circuits
before llm.chat, so the model was never called.

Changes

File Change
src/openkos/llm/ollama.py max_generation_tokens read-only property, mirroring context_window.
tests/unit/llm/test_ollama.py Two tests: the property, and the pair read through prompt_budget at a non-default ceiling.
src/openkos/retrieval/answer.py Corrected the omitted_titles note, which documented a guarantee that did not hold.

How was this tested

  • uv run pytest -q — 5763 passed, 1 skipped (5761 before; +2 new)
  • uv run ruff check . / ruff format --check . / uv run mypy . — clean

Both tests were written before the fix and failed for the intended reason:

assert prompt_budget.reply_reserve(client) == 4096
E  assert 8192 == 4096

They assert the pair through prompt_budget, at a non-default ceiling, on
purpose: test_context_window_property_exposes_the_configured_window passed
for the entire time its sibling was broken, so a test on one value alone is
not evidence about the other.

Notes for reviewers

Two things I deliberately did not do:

  • The issue's third suggestion — "consider making a zero budget loud" —
    needed no work.
    query sends unbounded context and files provenance the model never read #882's omission notice already names every dropped
    document and is emitted before the no-match return, and
    tests/unit/cli/test_query.py already pins that on a NO_MATCH result. A
    second assertion of the same behavior would have been duplication, so I
    verified it and left it alone.
  • No behavior change beyond the read. With the shipped default the budget
    is byte-identical to before; only non-default ceilings move, and they move
    to the value the caller asked for.

The omitted_titles note used to say read_config's floor kept the shipped
CLI path away from a zero budget. The floor is real, but the reserve was not
read from it, so the claim held only for the default row. It now states the
guarantee that actually holds and why.

Review receipt review-dc405d396fd8ffa7, APPROVED, zero findings,
inspection completed over all three manifest paths.

…ps flooring to zero

`prompt_budget.reply_reserve` reads `llm.max_generation_tokens` to decide how
much of the context window to hold back for the reply. `OllamaClient` exposed
`context_window` as a property but never that one -- the value was stored as
`self._max_generation_tokens` and never surfaced -- so the read always missed
and the 8192 default came back regardless of what the workspace pinned.

With the shipped defaults this is invisible, because the fallback happens to
EQUAL the pin (`DEFAULT_MAX_GENERATION_TOKENS` is 8192). Lower the ceiling and
`budget_chars` floors to zero, `fair_shares` allows every block nothing, and
`query` answers NO_MATCH on a fully populated bundle.

Measured across every configuration `read_config` accepts, before -> after:

  max_gen  min legal context_window   reserve seen        budget
     8192                     12288   8192 -> 8192   8440 -> 8440
     4096                      8192   8192 -> 4096      0 -> 8440
     2048                      6144   8192 -> 2048      0 -> 8440
     1024                      5120   8192 -> 1024      0 -> 8440
    16384                     20480   8192 -> 16384  30720 -> 8440

The last row was wrong in the other direction: it reserved less than the
ceiling actually consumes, budgeting more prompt room than the window can
hold. One missing property caused both.

- `OllamaClient.max_generation_tokens` is now a read-only property, the same
  shape and the same read-the-real-send rationale as `context_window`.
- Two tests, written first and failing first (`assert 8192 == 4096`). They
  assert the PAIR through `prompt_budget`, at a non-default ceiling:
  `test_context_window_property_exposes_the_configured_window` passed for the
  whole time its sibling was broken, so a test on one value alone is not
  evidence about the other.
- `AnswerResult.omitted_titles` claimed `read_config`'s floor kept the shipped
  CLI path away from a zero budget. The floor is real but the reserve was not
  read from it, so the claim held only for the default row. Corrected to state
  the guarantee that now actually holds.

The issue's third suggestion -- make a zero budget loud -- needed no work and
none was done. #882's omission notice names every dropped document and is
emitted BEFORE the no-match return, and `tests/unit/cli/test_query.py` already
pins that on a `NO_MATCH` result.

Found while measuring #887, which needed a bounded arm at a small window and
got 60 of 60 no-matches instead.

Review receipt review-dc405d396fd8ffa7, APPROVED, zero findings.
@jasonssdev
jasonssdev merged commit 4309781 into main Aug 27, 2026
7 checks passed
@jasonssdev
jasonssdev deleted the fix/ollama-exposes-max-generation-tokens-896 branch August 27, 2026 16:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant