fix(interruption): bind inference responses to the overlap they were requested for - #2306
fix(interruption): bind inference responses to the overlap they were requested for#2306mastermanas805 wants to merge 2 commits into
Conversation
…requested for `bargein_detected` / `inference_done` were matched to whichever overlap was open when they landed, not the one their request was cut for. A response arriving after its own overlap ended, while a later overlap was open, was accepted and attributed to that later overlap — emitting `isInterruption: true` for user audio the model never scored that way, and cutting the agent off. The cache clear at each overlap boundary was not a filter: `setOrUpdate()` recreates a missing entry from scratch, and `detectionDelayInS` was computed from the new overlap's start, so the result looked like a well-formed verdict. Plain response latency reaches this — the gap between two overlaps in one agent turn is often only a few hundred ms. Stamp the overlap generation onto each request in `sendAudioData()` and reject responses whose generation is no longer open. The generation ledger is kept outside `cache`, since `cache` is cleared at exactly the boundary where a late response still needs to be identifiable, and is bounded well above the request rate of a single overlap. The check fails open: a request whose generation is no longer on record counts as current, so losing the bookkeeping degrades to the previous behaviour rather than suppressing a genuine interruption. Fixes livekit#2119
🦋 Changeset detectedLatest commit: b078075 The changes in this PR will be included in the next version bump. This PR includes changesets to release 39 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
|
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
`overlapCount` restarts at 0 every agent turn, and the request ledger was cleared at that same boundary, so a response outliving its turn found no record, took the fail-open path, and was credited to whatever overlap was open in the next turn — the same misattribution this fixes within a turn. Track `overlapGeneration` separately and never reset or clear it. `overlapCount` keeps its per-turn meaning for the audio-prefix shift. Add the release-notes changeset.
|
Both review points addressed in b078075. The cross-turn one was real: clearing the ledger at the agent-speech boundary sent stale responses straight down the fail-open path, and |
chenghao-mou
left a comment
chenghao-mou
left a comment
There was a problem hiding this comment.
thanks for the PR! I have left one comment.
| // Identifies an overlap for the lifetime of the stream. Distinct from `overlapCount`, which | ||
| // resets every agent turn and drives the audio-prefix shift: a response that outlives its | ||
| // turn must still be distinguishable, and per-turn counts collide across turns. | ||
| let overlapGeneration = 0; |
There was a problem hiding this comment.
we've fixed this with a simpler implementation: livekit/agents#6957, can we use cache.updateValue() and ignore missing entries in both handlers, then remove the generation ledger and fail-open test?
For an interruption verdict that, for some reason, failed to interrupt the agent speech, we are going to fall back to VAD interruption path in another PR, because running inference with the same/similar audio context will have the same prediction anyway.
3 participants
The issue notes this wasn't reproduced at runtime, so this starts with a failing test: overlap A sends a request, A ends, B opens, and A's
bargein_detectedlands and is reported as an interruption of B. Ordinary response latency is enough to hit it.InterruptionAudioSlice.overlapGenerationfrom #2116 isn't onmain— #2116 and #2117 were both closed unmerged. So this adds anoverlapGenerationthat bumps per overlap and never resets, stamps it onto each request insendAudioData(), and rejects responses whose generation has since closed. It's deliberately separate from the existingoverlapCount, which restarts each agent turn for the audio-prefix shift — reusing that would let turn N's generation collide with turn N+1's.inference_doneneeded the same guard. It emits no event, but it writes intocache, and a stale write poisons the entryoverlap-speech-endedlater pops for its verdict.The check fails open: an unrecorded generation counts as current, since treating an evicted request as stale would suppress a real interruption — worse than the misattribution being fixed. Same reason the ledger sits outside
cache, which is cleared at every overlap boundary.Regression covers the late response, the cross-turn case, the fail-open path, and a control that still interrupts normally.
Fixes #2119