Skip to content

Fix Responses-API cache token double-count in context occupancy - #689

Merged
TheGreatAxios merged 1 commit into
mainfrom
cl-7075-fix-responses-api-cache-token-double-count-in-context
Aug 25, 2026
Merged

Fix Responses-API cache token double-count in context occupancy#689
TheGreatAxios merged 1 commit into
mainfrom
cl-7075-fix-responses-api-cache-token-double-count-in-context

Conversation

@TheGreatAxios

Copy link
Copy Markdown
Collaborator

Summary

  • Responses-API adapters (codex, grok, openai-responses) split cached_tokens out of input_tokens so TokenUsage fields stay non-overlapping and context occupancy sums equal real prompt tokens
  • The context meter and compaction governor no longer read up to ~2x inflated occupancy on high cache-hit sessions
  • Faremeter bills cached tokens at the cache rate exactly once for Responses-API usage

Why: on the OpenAI/xAI Responses API, input_tokens_details.cached_tokens is a subset of input_tokens. Summing the raw fields double-counts every cached token, so the meter reads 70-80% while real occupancy is ~40% — and compaction either fires in early bursts or appears stuck.

Verification

  • bun run typecheck, bun run build, and bun run test pass
  • Adapter tests pin input + cacheRead == wire prompt_tokens, the omitted-details edge, and a clamp for inconsistent gateways reporting cached_tokens > input_tokens
  • New faremeter tests pin cost math against normalized usage: uncached input at the input rate, cached reads at the cache rate, each token billed once

Fixes CL-7075

@linear-code

linear-code Bot commented Aug 25, 2026

Copy link
Copy Markdown

CL-7075

@TheGreatAxios TheGreatAxios left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review · Approve

Responses-API usage parsing splits cached_tokens out of input so occupancy and faremeter sums count each prompt token once.

No findings.

GitHub rejected --approve on this PR (author cannot approve their own pull request); verdict is recorded here as a comment.

@TheGreatAxios
TheGreatAxios merged commit f2420c7 into main Aug 25, 2026
5 checks passed
@TheGreatAxios
TheGreatAxios deleted the cl-7075-fix-responses-api-cache-token-double-count-in-context branch August 28, 2026 00:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant