Attribute direct desktop-proxy and realtime-relay spend in the gateway ledger (free tier S0) - #12592
Open
Git-on-my-level wants to merge 1 commit into
Open
Attribute direct desktop-proxy and realtime-relay spend in the gateway ledger (free tier S0)#12592Git-on-my-level wants to merge 1 commit into
Git-on-my-level wants to merge 1 commit into
Conversation
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
10 issues found across 19 files
Confidence score: 2/5
backend/utils/llm/realtime_usage.pycan misclassify empty or unknown modality details as valid text/modality splits, producing incorrect realtime token pricing and potentially under- or mis-recording spend; require populated, recognized modality counts before applying split pricing.backend/routers/desktop_realtime.pyselects a rate card from the client model whilemint_sessionissues a different Gemini model, so Gemini sessions can be billed at the cheaper rate and under-record spend; derive pricing from the issued model or enforce model consistency.backend/utils/llm/managed_spend_ledger.pycan terminate with pending ledger writes during normal backend or desktop-backend shutdown, whilebackend/routers/omni_relay.pycan miss accounting when a downstream WebSocket closes during a send; wire draining into shutdown lifecycles and account frames in a sendfinally.- The ledger timing/configuration tests in
backend/tests/unit/test_managed_spend_ledger.pymay be flaky or misclassified because they depend on real sleeps, threads, and deployment files; isolate or mark these checks appropriately, and keep the report scope indicator inbackend/scripts/chat_agent_cost_report.pyvisible even for empty filtered results.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="backend/scripts/chat_agent_cost_report.py">
<violation number="1" location="backend/scripts/chat_agent_cost_report.py:261">
P3: When `--uid` matches no rows, the report omits the new scope indicator and says only `No <feature> attempts found`, which can be mistaken for a globally empty ledger. Move the scope line before the empty-result return or include the UID in the no-attempt message.</violation>
</file>
<file name="backend/utils/llm/managed_spend_ledger.py">
<violation number="1" location="backend/utils/llm/managed_spend_ledger.py:210">
P2: During a normal backend or desktop-backend shutdown, `drain_pending_writes()` is never invoked, so pending ledger rows can be killed with the process. Wire this drain into each serving app's shutdown/lifespan and shut down the private executor, or use the shared executor lifecycle.</violation>
</file>
<file name="backend/routers/omni_relay.py">
<violation number="1" location="backend/routers/omni_relay.py:270">
P2: When the downstream WebSocket closes while sending a provider frame, the send raises before `account_upstream_frame` observes the terminal response. Account the frame in a `finally` around the downstream send so provider usage is still recorded when delivery fails.</violation>
</file>
<file name="backend/utils/llm/realtime_usage.py">
<violation number="1" location="backend/utils/llm/realtime_usage.py:647">
P2: When OpenAI includes empty `input_token_details` or `output_token_details` alongside aggregate counts, this marks the usage as modality-split and `price_realtime_turn` prices it. Require non-empty modality counts for each non-zero side, allowing only genuinely zero input/output sides to use the fallback.</violation>
<violation number="2" location="backend/utils/llm/realtime_usage.py:700">
P1: When Gemini sends aggregate counts with empty modality-detail arrays, `_gemini_turn` treats the arrays as a reported split and prices the aggregates as text. Track whether each nonzero count came from a populated modality detail list before setting `split_reported`.</violation>
<violation number="3" location="backend/utils/llm/realtime_usage.py:773">
P2: When Gemini reports an unknown modality, this branch silently bills those tokens as text. Track unknown modality entries and clear `modality_split_reported` so only recognized modality counts can receive modality pricing.</violation>
</file>
<file name="backend/routers/desktop_realtime.py">
<violation number="1" location="backend/routers/desktop_realtime.py:230">
P2: When a Gemini client posts `model=gemini-2.5-flash-native-audio-preview-12-2025`, this selects a cheaper rate card even though `mint_session` always issues `gemini-3.1-flash-live-preview`, under-recording spend. Derive the model from the server-selected session model instead of trusting `report.model`.</violation>
</file>
<file name="backend/tests/unit/test_managed_spend_ledger.py">
<violation number="1" location="backend/tests/unit/test_managed_spend_ledger.py:232">
P2: The final `assert not ledger._pending_writes` can flake under load. `drain_pending_writes()` waits only `accounting_write_timeout_seconds()` (set to 0.01s in this test), and the two hung writer threads wake only after `release.set()`. If the executor threads take longer than 10ms to wake and complete on a loaded CI machine, the drain times out, the futures stay in `_pending_writes`, and the assertion fails spuriously. This test also drives real `asyncio.sleep(0.05)` and the real `_ledger_executor` ThreadPoolExecutor, so it is timing/concurrency-sensitive and should be marked `@pytest.mark.slow` per the backend test governance (real asyncio sleeps and thread scheduling are not fast-unit lane). Consider releasing the threads with a longer write timeout for the drain, or marking the test slow.</violation>
<violation number="2" location="backend/tests/unit/test_managed_spend_ledger.py:252">
P3: This test uses a real `await asyncio.sleep(0.05)` together with a real background `threading.Event`/`ThreadPoolExecutor` write to assert timing-sensitive concurrency behavior, but it is not marked slow or integration. Per tests/README.md, tests that do real asyncio sleeps and real threading must be marked `@pytest.mark.slow` / `@pytest.mark.integration`; as written it runs in the default unit lane alongside deterministic observer tests and depends on wall-clock timing.</violation>
<violation number="3" location="backend/tests/unit/test_managed_spend_ledger.py:891">
P3: This is a config/deployment-verification test, not a unit test: it walks repo-relative paths outside the test file (backend/charts, .github/workflows, deploy/runtime_env.yaml), reads their contents, and asserts on occurrence counts of an env var. tests/README.md requires 'codebase greps' and repo-layout dependent checks to be marked `@pytest.mark.slow`/`@pytest.mark.integration`. It currently runs in the default unit lane and will fail or skip based on checkout layout (it `pytest.skip`s when a path is absent), which is not pure unit behavior.</violation>
</file>
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
| reasoning = _count(usage, 'thoughtsTokenCount') | ||
| tool_use = _count(usage, 'toolUsePromptTokenCount') | ||
| # The split is vouched for when every non-zero side came with details. | ||
| split_reported = (prompt_text + prompt_audio + prompt_image == 0 or isinstance(prompt_details, list)) and ( |
There was a problem hiding this comment.
P1: When Gemini sends aggregate counts with empty modality-detail arrays, _gemini_turn treats the arrays as a reported split and prices the aggregates as text. Track whether each nonzero count came from a populated modality detail list before setting split_reported.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At backend/utils/llm/realtime_usage.py, line 700:
<comment>When Gemini sends aggregate counts with empty modality-detail arrays, `_gemini_turn` treats the arrays as a reported split and prices the aggregates as text. Track whether each nonzero count came from a populated modality detail list before setting `split_reported`.</comment>
<file context>
@@ -0,0 +1,811 @@
+ reasoning = _count(usage, 'thoughtsTokenCount')
+ tool_use = _count(usage, 'toolUsePromptTokenCount')
+ # The split is vouched for when every non-zero side came with details.
+ split_reported = (prompt_text + prompt_audio + prompt_image == 0 or isinstance(prompt_details, list)) and (
+ output_text + output_audio == 0 or isinstance(response_details, list)
+ )
</file context>
| ) | ||
|
|
||
|
|
||
| async def drain_pending_writes() -> None: |
There was a problem hiding this comment.
P2: During a normal backend or desktop-backend shutdown, drain_pending_writes() is never invoked, so pending ledger rows can be killed with the process. Wire this drain into each serving app's shutdown/lifespan and shut down the private executor, or use the shared executor lifecycle.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At backend/utils/llm/managed_spend_ledger.py, line 210:
<comment>During a normal backend or desktop-backend shutdown, `drain_pending_writes()` is never invoked, so pending ledger rows can be killed with the process. Wire this drain into each serving app's shutdown/lifespan and shut down the private executor, or use the shared executor lifecycle.</comment>
<file context>
@@ -0,0 +1,237 @@
+ )
+
+
+async def drain_pending_writes() -> None:
+ """Give scheduled writes one configured timeout during orderly shutdown (and tests)."""
+ loop = asyncio.get_running_loop()
</file context>
| else: | ||
| await websocket.send_text(message) | ||
| # Forward first, account second: the client never waits on us. | ||
| account_upstream_frame(message) |
There was a problem hiding this comment.
P2: When the downstream WebSocket closes while sending a provider frame, the send raises before account_upstream_frame observes the terminal response. Account the frame in a finally around the downstream send so provider usage is still recorded when delivery fails.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At backend/routers/omni_relay.py, line 270:
<comment>When the downstream WebSocket closes while sending a provider frame, the send raises before `account_upstream_frame` observes the terminal response. Account the frame in a `finally` around the downstream send so provider usage is still recorded when delivery fails.</comment>
<file context>
@@ -157,15 +255,19 @@ async def client_to_upstream():
else:
await websocket.send_text(message)
+ # Forward first, account second: the client never waits on us.
+ account_upstream_frame(message)
t1 = asyncio.create_task(client_to_upstream(), name=f"ws:{uid}:omni_c2u")
</file context>
| elif modality in {'IMAGE', 'VIDEO'}: | ||
| image += count | ||
| else: | ||
| text += count |
There was a problem hiding this comment.
P2: When Gemini reports an unknown modality, this branch silently bills those tokens as text. Track unknown modality entries and clear modality_split_reported so only recognized modality counts can receive modality pricing.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At backend/utils/llm/realtime_usage.py, line 773:
<comment>When Gemini reports an unknown modality, this branch silently bills those tokens as text. Track unknown modality entries and clear `modality_split_reported` so only recognized modality counts can receive modality pricing.</comment>
<file context>
@@ -0,0 +1,811 @@
+ elif modality in {'IMAGE', 'VIDEO'}:
+ image += count
+ else:
+ text += count
+ return text, audio, image
+
</file context>
| def _openai_counts(usage: Mapping[str, Any]) -> dict[str, Any]: | ||
| input_details = usage.get('input_token_details') | ||
| output_details = usage.get('output_token_details') | ||
| split_reported = isinstance(input_details, Mapping) and isinstance(output_details, Mapping) |
There was a problem hiding this comment.
P2: When OpenAI includes empty input_token_details or output_token_details alongside aggregate counts, this marks the usage as modality-split and price_realtime_turn prices it. Require non-empty modality counts for each non-zero side, allowing only genuinely zero input/output sides to use the fallback.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At backend/utils/llm/realtime_usage.py, line 647:
<comment>When OpenAI includes empty `input_token_details` or `output_token_details` alongside aggregate counts, this marks the usage as modality-split and `price_realtime_turn` prices it. Require non-empty modality counts for each non-zero side, allowing only genuinely zero input/output sides to use the fallback.</comment>
<file context>
@@ -0,0 +1,811 @@
+def _openai_counts(usage: Mapping[str, Any]) -> dict[str, Any]:
+ input_details = usage.get('input_token_details')
+ output_details = usage.get('output_token_details')
+ split_reported = isinstance(input_details, Mapping) and isinstance(output_details, Mapping)
+ input_details = _mapping(input_details)
+ output_details = _mapping(output_details)
</file context>
| output_text_tokens=report.output_text_tokens, | ||
| output_audio_tokens=report.output_audio_tokens, | ||
| ) | ||
| return client_reported_cost_usd(report.provider, report.model, turn) |
There was a problem hiding this comment.
P2: When a Gemini client posts model=gemini-2.5-flash-native-audio-preview-12-2025, this selects a cheaper rate card even though mint_session always issues gemini-3.1-flash-live-preview, under-recording spend. Derive the model from the server-selected session model instead of trusting report.model.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At backend/routers/desktop_realtime.py, line 230:
<comment>When a Gemini client posts `model=gemini-2.5-flash-native-audio-preview-12-2025`, this selects a cheaper rate card even though `mint_session` always issues `gemini-3.1-flash-live-preview`, under-recording spend. Derive the model from the server-selected session model instead of trusting `report.model`.</comment>
<file context>
@@ -216,23 +217,17 @@ def _record_usage(
+ output_text_tokens=report.output_text_tokens,
+ output_audio_tokens=report.output_audio_tokens,
)
+ return client_reported_cost_usd(report.provider, report.model, turn)
</file context>
Suggested change
| return client_reported_cost_usd(report.provider, report.model, turn) | |
| model = _OPENAI_REALTIME_MODEL if report.provider == "openai" else _GEMINI_LIVE_MODEL | |
| return client_reported_cost_usd(report.provider, model, turn) |
| await ledger.drain_pending_writes() | ||
|
|
||
| assert sorted(seen) == ['boom', 'ok'] | ||
| assert not ledger._pending_writes |
There was a problem hiding this comment.
P2: The final assert not ledger._pending_writes can flake under load. drain_pending_writes() waits only accounting_write_timeout_seconds() (set to 0.01s in this test), and the two hung writer threads wake only after release.set(). If the executor threads take longer than 10ms to wake and complete on a loaded CI machine, the drain times out, the futures stay in _pending_writes, and the assertion fails spuriously. This test also drives real asyncio.sleep(0.05) and the real _ledger_executor ThreadPoolExecutor, so it is timing/concurrency-sensitive and should be marked @pytest.mark.slow per the backend test governance (real asyncio sleeps and thread scheduling are not fast-unit lane). Consider releasing the threads with a longer write timeout for the drain, or marking the test slow.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At backend/tests/unit/test_managed_spend_ledger.py, line 232:
<comment>The final `assert not ledger._pending_writes` can flake under load. `drain_pending_writes()` waits only `accounting_write_timeout_seconds()` (set to 0.01s in this test), and the two hung writer threads wake only after `release.set()`. If the executor threads take longer than 10ms to wake and complete on a loaded CI machine, the drain times out, the futures stay in `_pending_writes`, and the assertion fails spuriously. This test also drives real `asyncio.sleep(0.05)` and the real `_ledger_executor` ThreadPoolExecutor, so it is timing/concurrency-sensitive and should be marked `@pytest.mark.slow` per the backend test governance (real asyncio sleeps and thread scheduling are not fast-unit lane). Consider releasing the threads with a longer write timeout for the drain, or marking the test slow.</comment>
<file context>
@@ -0,0 +1,925 @@
+ await ledger.drain_pending_writes()
+
+ assert sorted(seen) == ['boom', 'ok']
+ assert not ledger._pending_writes
+
+
</file context>
| .where(filter=FieldFilter('feature', '==', feature)) | ||
| .where(filter=FieldFilter('date', '==', day)) | ||
| ) | ||
| if uid is not None: |
There was a problem hiding this comment.
P3: When --uid matches no rows, the report omits the new scope indicator and says only No <feature> attempts found, which can be mistaken for a globally empty ledger. Move the scope line before the empty-result return or include the UID in the no-attempt message.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At backend/scripts/chat_agent_cost_report.py, line 261:
<comment>When `--uid` matches no rows, the report omits the new scope indicator and says only `No <feature> attempts found`, which can be mistaken for a globally empty ledger. Move the scope line before the empty-result return or include the UID in the no-attempt message.</comment>
<file context>
@@ -252,6 +258,8 @@ def fetch_rows(client: Any, feature: str, days: Sequence[str]) -> Iterator[Mappi
.where(filter=FieldFilter('feature', '==', feature))
.where(filter=FieldFilter('date', '==', day))
)
+ if uid is not None:
+ query = query.where(filter=FieldFilter('user_uid', '==', uid))
for snapshot in query.stream():
</file context>
| # --- deploy contract ------------------------------------------------------------------ | ||
|
|
||
|
|
||
| def test_the_accounting_switch_reaches_every_serving_identity_of_both_surfaces() -> None: |
There was a problem hiding this comment.
P3: This is a config/deployment-verification test, not a unit test: it walks repo-relative paths outside the test file (backend/charts, .github/workflows, deploy/runtime_env.yaml), reads their contents, and asserts on occurrence counts of an env var. tests/README.md requires 'codebase greps' and repo-layout dependent checks to be marked @pytest.mark.slow/@pytest.mark.integration. It currently runs in the default unit lane and will fail or skip based on checkout layout (it pytest.skips when a path is absent), which is not pure unit behavior.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At backend/tests/unit/test_managed_spend_ledger.py, line 891:
<comment>This is a config/deployment-verification test, not a unit test: it walks repo-relative paths outside the test file (backend/charts, .github/workflows, deploy/runtime_env.yaml), reads their contents, and asserts on occurrence counts of an env var. tests/README.md requires 'codebase greps' and repo-layout dependent checks to be marked `@pytest.mark.slow`/`@pytest.mark.integration`. It currently runs in the default unit lane and will fail or skip based on checkout layout (it `pytest.skip`s when a path is absent), which is not pure unit behavior.</comment>
<file context>
@@ -0,0 +1,925 @@
+# --- deploy contract ------------------------------------------------------------------
+
+
+def test_the_accounting_switch_reaches_every_serving_identity_of_both_surfaces() -> None:
+ """The manifest declares the flag; these are the files that actually put it on the pod/revision."""
+ backend = Path(__file__).resolve().parents[2]
</file context>
Suggested change
| def test_the_accounting_switch_reaches_every_serving_identity_of_both_surfaces() -> None: | |
| @pytest.mark.slow | |
| def test_the_accounting_switch_reaches_every_serving_identity_of_both_surfaces() -> None: |
| try: | ||
| assert ledger.schedule_managed_attempt(_attempt(request_id='a')) is True | ||
| assert ledger.schedule_managed_attempt(_attempt(request_id='b')) is True | ||
| await asyncio.sleep(0.05) # both observers have timed out; the writes are still running |
There was a problem hiding this comment.
P3: This test uses a real await asyncio.sleep(0.05) together with a real background threading.Event/ThreadPoolExecutor write to assert timing-sensitive concurrency behavior, but it is not marked slow or integration. Per tests/README.md, tests that do real asyncio sleeps and real threading must be marked @pytest.mark.slow / @pytest.mark.integration; as written it runs in the default unit lane alongside deterministic observer tests and depends on wall-clock timing.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At backend/tests/unit/test_managed_spend_ledger.py, line 252:
<comment>This test uses a real `await asyncio.sleep(0.05)` together with a real background `threading.Event`/`ThreadPoolExecutor` write to assert timing-sensitive concurrency behavior, but it is not marked slow or integration. Per tests/README.md, tests that do real asyncio sleeps and real threading must be marked `@pytest.mark.slow` / `@pytest.mark.integration`; as written it runs in the default unit lane alongside deterministic observer tests and depends on wall-clock timing.</comment>
<file context>
@@ -0,0 +1,925 @@
+ try:
+ assert ledger.schedule_managed_attempt(_attempt(request_id='a')) is True
+ assert ledger.schedule_managed_attempt(_attempt(request_id='b')) is True
+ await asyncio.sleep(0.05) # both observers have timed out; the writes are still running
+ assert len(ledger._pending_writes) == 2
+ assert ledger.schedule_managed_attempt(_attempt(request_id='c')) is False # cap holds
</file context>
…y ledger Every "basic users no longer hit X" proof in the local-models free-tier program asks the gateway ledger for managed spend per uid per feature. Two surfaces bypass the gateway and were invisible to it: the desktop proxy's direct Vertex / AI Studio routes (token counts went to a log event only) and the legacy realtime relay (nothing at all). Both now write the gateway's own AccountingEvent into llm_gateway_attempts through the gateway's DB helper, under the same LLM_GATEWAY_ACCOUNTING_ENABLED switch and the same best-effort, bounded, off-the-request-path policy. - desktop_proxy: one row per direct attempt at ProxyTelemetry.complete(), feature desktop_proactivity (same as the gateway route, one query covers both), caller desktop_proxy, payer byok|omi. Gateway/stub traffic writes nothing here. - omni_relay: one row per provider turn observed on the wire (OpenAI response.done; Gemini cumulative usageMetadata + turnComplete, latest wins), feature desktop_chat_realtime, caller omni_relay, priced with the modality rate table /v2/realtime/usage already used. Forward first, account second. - build_accounting_event gains an optional caller-priced cost for usage the text-only rate cards cannot price; BYOK stays not_omi_cost. - chat_agent_cost_report.py gains --uid (third equality filter). - Deploy: the accounting switch reaches backend, backend-listen and desktop-backend; the admin ledger dashboard learns the new feature. Proof (hermetic): a synthetic direct proxy call and a synthetic relay turn each land as exactly one row in the per-uid-per-feature query. Live proof waits on a serving revision. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Git-on-my-level
force-pushed
the
feat/free-tier-s0-ledger-attribution
branch
from
September 2, 2026 05:01
01167d3 to
d4a5cd8
Compare
This was referenced Sep 2, 2026
Open
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
S0 — managed-spend attribution for the two surfaces that bypass the gateway
Shard S0 of the ratified local-models free-tier program (
omi-knowledge-base/projects/local-models-free-tier/implementation/60-shards.md). Every later "basic users no longer hit X" proof asks the ledger "managed spend for uid X, grouped by feature". The gateway already answers that for gateway traffic (llm_gateway_attempts). Two surfaces did not appear in it at all:routers/desktop_proxy.py— direct Vertex / AI Studio Gemini calls (BYOK, batch embeddings,FEATURE_MODE=off) recorded token counts only in a log event.routers/omni_relay.py— the legacy realtime relay recorded nothing.This PR puts both in the existing ledger. No new ledger, no dashboard, no alerting.
What changed
utils/llm/managed_spend_ledger.py(new): builds the gateway's ownAccountingEventfor an attempt made outside the gateway and persists it with the gateway's own DB helper (record_llm_gateway_attempt) intollm_gateway_attempts. Same switch as the gateway sink (LLM_GATEWAY_ACCOUNTING_ENABLED, read at call time), same best-effort policy: bounded pending writes, one timeout, never on the request path, never raises.utils/llm/realtime_usage.py(new, pure): recognises provider responses on the provider-native realtime wire and prices them per model and modality.response.done;response.statusmaps tosuccess/cancelled(client_cancelled, orinterruptedwhenstatus_details.reasonisturn_detected) /error(incomplete,provider_error). Several responses may be open at once (out-of-band, barge-in):response.createdopens an id,response.donecloses only its own, and every id still open when the socket ends is recorded ascancelled/client_disconnected, each with its own ordinal. Ids are bounded (oversized ones become anonymous tracking keys that never reach the ledger) the open set is capped with an overflow count so a flood is still counted, and a session-end flush emits at most 16 rows and logs the rest as a dropped count.usageMetadatais the session running total (Google's Live reference;desktop/windows/.../geminiSession.tssays so too), so a turn is the field-wise delta from the last block.turnCompletecompletes a turn but holds it until the next server activity, so a trailingusageMetadata(an independent optional field) folds into it; model output or a tool call (modelTurn/toolCall, matched by substring, never parsed) marks a turn in flight andinterruptedortoolCallCancellationlatches an interruption, so a disconnect mid-turn is recorded ascancelled(interruptedwins overclient_disconnected). Thinking (thoughtsTokenCount) and tool-use prompt tokens are carried.unpricedrather than a confident guess. Cached tokens are subsets of input and are discounted, not added; reasoning bills at the text-output rate. Rates are integers in micro-USD per million tokens from the provider pricing pages read 2026-09-01, rounded half-up like the gateway's cards.routers/desktop_proxy.py: one row per provider dispatch on the direct routes (vertex_ai,ai_studio,ai_studio_byok):note_dispatch()in the non-streampost()and the streamopen_attempt(), the overflow loops close each failed attempt (provider_capacity/model_unavailable) before recovery routing can fail,complete()closes the last one, and failures before any dispatch write nothing. One invocation per request, ordinal = dispatch count,fallback_reason=overflow_recoverypast the first. Gateway-routed and stub traffic write nothing here (the gateway writes its own row). Rows:feature=desktop_proactivity(same as the gateway path, so one query covers both routes),caller=desktop_proxy,payer=byok|omi, providergemini. Telemetry outcomes map to the ledger'ssuccess|error|cancelledwith the detail inerror_class. The streaming SSE usage observer is now bounded (1 MiB) instead of growing with the response.routers/omni_relay.py: one row per observed provider response,feature=desktop_chat_realtime(the same word as thellm_usageaccount the direct hub already debits),caller=omni_relay, one invocation id per session with the response as ordinal.payerfollows the credential the relay actually selected (a validated key for this provider →byok,not_omi_cost), not enrollment, so an unenrolled user's own key is never booked as Omi spend. Forward first, account second, behind a content-free exception guard; everything still open is flushed when the socket ends.llm_gateway/gateway/accounting.py:build_accounting_event(..., priced=PricedUsage)lets a caller price usage the text-only YAML rate cards cannot (realtime audio). BYOK and indeterminate usage keep the estimator's verdict; a caller may only replace anunpriced/estimatedverdict.routers/desktop_realtime.py:_usage_costnow uses the shared realtime pricing. Behaviour change: cached tokens used to be billed on top of the input they are part of; they are now discounted. The existing endpoint test value (no cached tokens) is unchanged.utils/llm/managed_spend_ledger.pywrites on its own two-thread pool without copying request context (which on the relay can carry validated BYOK keys); the pending cap counts Firestore calls until they return, so a hung Firestore cannot occupy the shareddb_executorthat gates quota and auth.scripts/chat_agent_cost_report.py:--uidscopes the existing per-feature report to one user (third equality filter; no composite index).LLM_GATEWAY_ACCOUNTING_ENABLED=trueonbackend,backend-listen, anddesktop_backendinbackend/deploy/runtime_env/_base.yaml(+ regeneratedruntime_env.yaml), in both backend-listen Helm values files (where it actually reaches the pod), and in both desktop-backend workflows'env_vars;validate-backend-runtime-env.pypasses for dev and prod.web/adminFEATURE_CLASSgainsdesktop_chat_realtimeanddesktop_proactivity(the latter was already being written by the gateway path and landing underunknown) so the ledger dashboard classifies both as desktop spend (the pinned Prettier also reformatted that file: main's copy was not Prettier-clean).Automatic-or-dead check
backend/tests/unit/test_managed_spend_ledger.py(33 tests) andbackend/tests/unit/test_realtime_usage.py(74). The proof tests run the shard's stated proof hermetically against a fake customer-plane Firestore, through the real routers:test_synthetic_direct_proxy_call_lands_in_the_per_uid_per_feature_query— one company-paid VertexgenerateContentthrough_proxy→ exactly one row for(uid, desktop_proactivity, today), priced, tier-snapshotted; zero rows for another uid.test_a_saturated_reservation_leaves_one_row_per_dispatch— a full reservation then shared capacity → two rows, one invocation.test_synthetic_relay_turn_lands_in_the_per_uid_per_feature_query— one OpenAIresponse.donethrough the real relay loop → exactly one row for(uid, desktop_chat_realtime, today)with the cached-discounted cost (13,500 µUSD from the fixture's split). Plus: two-turn session-cumulative Gemini → per-turn deltas; cancelled / interrupted → not successes; missing usage and unknown model →unpriced; in-flight response at disconnect →cancelled.Red-proofs applied by hand and seen red (then restored), twelve mutations plus the round-three regressions (two open responses at disconnect → two rows; tool call as activity; overflow past the tracking cap; oversized ids): proxy emission removed; relay
record_turnremoved; priced override disabled; gateway route added to the direct set (double count); Gemini deltas replaced by cumulative; every OpenAI status a success; unreported usage priced as zero; proxy records without a dispatch; cached tokens charged in full; payer taken from enrollment instead of the selected credential; flush drops open responses; aggregate-only usage priced anyway.Reviewed by Codex (gpt-5.6-sol) in four rounds (
not mergeable×3, thenmergeable after should-fixes); every blocker and should-fix raised is addressed above, including the three round-four should-fixes.Proof
tests/unit/test_desktop_proxy.py,test_desktop_realtime.py,test_llm_gateway_accounting.py,test_free_quota_gate_wiring.py,test_chat_agent_cost_report.py,test_render_backend_runtime_env.pygreen locally.LLM_GATEWAY_ACCOUNTING_ENABLED=true, run one floating-bar PTT turn and one desktop proactivity call from a test account, thenpython backend/scripts/chat_agent_cost_report.py --feature desktop_chat_realtime --uid <uid> --days 1and--feature desktop_proactivity --uid <uid>must each show the attempt withcalleromni_relay/desktop_proxy. Not run yet: this code has no revision serving it.Cut list
/v2/realtime/usage(client-reported direct-hub usage) is not written to the ledger here; it already debitsllm_usage. S15 owns the one-counter work across relay / mint / usage and will decide whether to mirror it.Test-isolation fix carried along
tests/unit/test_free_quota_gate_wiring.py's relay tests relied on the module fixture's AutoMock ofdatabase.usersreachingrelay.users_db. That binding is made at import time, and any test file that importsrouters.omni_relayduring collection (the new ledger tests do) leaves it real, sois_byok_activeread production Firestore and the test hung for the 300s retry when the two files shared a worker. The two tests that did not already stub it now do.Product invariants
INV-DATA-1— the desktop-backend workflows change only by adding theLLM_GATEWAY_ACCOUNTING_ENABLED=trueenv var to the existingenv_varsblock. The Cloud Run deployment authority, credentials, projects, and routing are untouched; this preserves the existing authority and is not a migration exception. Ledger writes go to the customer data plane throughget_customer_firestore_client(), the same seam quota andllm_usagealready use.INV-MEM-4— no memory promotion, admission, or Long-term authority path is touched. The only Firestore write added is an immutable row inllm_gateway_attempts(spend accounting), through the gateway's existing DB helper.Line-count ratchet
Line-Count-Exception: backend/routers/desktop_proxy.py | 1528 -> 1632 | per-dispatch ledger attribution lives on ProxyTelemetry and in the two overflow loops it observes; extracting the telemetry class is a refactor of the proxy's terminal contract that belongs in its own PR, not under a spend-attribution shard
Pre-existing, not touched
tests/unit/test_llm_gateway_coverage_guardrails.py::test_direct_provider_usage_stays_inside_approved_boundaries(markedslow) fails onorigin/mainwith a stale allowlist entry forutils/other/chat_file.py— unrelated to this diff.🤖 Generated with Claude Code