feat(desktop): hands-free wake word to command the assistant during ambient listening - #11801
feat(desktop): hands-free wake word to command the assistant during ambient listening#11801aryanorastar wants to merge 40 commits into
Conversation
…mbient listening Add an opt-in Wake Word feature: say "Omi" followed by a command while ambient listening is active and the assistant runs the command hands-free. - WakeWordService: parses and gates wake-word triggers (cooldown, segment dedup, user-speech and busy-conversation guards) and submits the stripped command to the assistant - WakeWordSegmentParser: extracts wake phrase + 2-word-minimum command - Wire incoming transcript segments from AppState+ListenEvents - Settings > General: opt-in Wake Word toggle with dynamic subtitle - AssistantSettings: persisted wakeWordEnabled/Phrase/Cooldown (default off) - Tests: WakeWordServiceTests + WakeWordSegmentParserTests (15 cases green)
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
… e2e flow coverage Fix the two Desktop Swift CI failures: - desktop-e2e-flow-coverage: cover WakeWordService.swift and WakeWordSegmentParser.swift under capture-lifecycle.yaml (the capture_test_transcript seam drives the same AppState+ListenEvents funnel the wake word observes). - desktop-swift-format-lint: add missing trailing newlines to the four WakeWord source/test files per pinned swift-format 602.0.0. Verified: run_checks.py macos lane passes e2e-flow-coverage, swift-format-lint, and swiftlint; WakeWord 15 tests pass (WakeWordSegmentParserTests 7 + WakeWordServiceTests 8).
|
Two CI failures fixed and pushed (0eca6df):
Local verification before push:
Waiting on CI to re-run — should flip green. |
|
Thanks @aryanorastar — nicely built feature: opt-in and default-off, clean parser/service split, injectable clock and trigger for testing, plus changelog and e2e contract coverage. I walked all nine files; notes below. Code walkthrough
Product questions before merge
No blocking code issues found; desktop CI is green including the new tests and e2e t0. Leaving the wake-word semantics and auto-send UX for human maintainer review before merge. by AI on behalf of David — if you need David’s attention urgently, please @Git-on-my-level and escalate with |
Git-on-my-level
added
needs-maintainer-review
|
Thanks for the thorough review and kind words @Git-on-my-level! Here is the context and rationale behind the three product questions:
Happy to follow the maintainers' preference on any follow-up adjustments! Ready for final merge. |
The wake word only matched the literal string "omi", but speech-to-text
spells the phrase by sound. A live mic session transcribed "Omi, how are
you?" as "Oh me, how are you?" (Parakeet v3, conf=0.92) and the wake word
silently never fired -- the recognizer heard the user correctly and the
parser rejected it.
Accept the renderings recognizers actually emit ("oh me", "omni", "ohmi",
"oh mi", "omee", "o me", "oh-me") as the same phrase, expanded through the
existing greeting prefixes so "hey oh me, ..." works too. The downstream
guards (user speech only, 2+ word command, cooldown, segment dedup) still
bound the false-positive cost of the wider match.
The existing unit tests passed before and after this change because they
fed the parser the string "Omi" -- which is exactly what the microphone
never produces.
Verification
- swift test --filter WakeWordSegmentParserTests -> 10 passed
- Added the exact failing string from the live session as a regression test,
plus negatives proving the wider match does not swallow ordinary speech
("Omnibus schedule changed" and a bare "Oh me" still do not fire).
Three defects found by running the feature against a live ambient session rather than constructed segments. Each failed silently, so a wake word that never fired was indistinguishable from one that was never spoken. 1. Diarization attribution. The trigger required `segment.isUser`, but the backend only sets `is_user` once a speech profile is enrolled. A live session logged `Speaker 0: "Omi, what's the weather?"` with is_user=false, so the wake word was structurally dead for every user without an enrolled profile. VoiceBargeInPolicy already gates on `isUser || speaker == 0` and documents speaker 0 as the primary user; the two entry points disagreed about what "the user" means. Align on the sibling's contract. 2. Segment dedup. The backend re-delivers one growing segment under a single id (observed: [206.0s-217.1s] -> [206.0s-228.9s]). Deduping on the id alone dropped every later command that landed inside an id that had already fired. Key on the id plus the extracted command so a re-sent segment is still suppressed but a new instruction inside it runs. 3. Cooldown. The cooldown exists to swallow a rapid repeat of the same utterance, but it gated on elapsed time alone. The ambient transcript lane runs ~35s behind live speech (measured over 11 segments, 34.3-36.6s), so several genuinely distinct commands routinely arrive inside one 30s window and were discarded as "repeats" -- two consecutive "what time is it" attempts were both lost this way. Gate on a repeat of the same command. Also name the reason a segment was ignored. Every guard returned silently; the diagnostics are what located defects 1 and 2, and the demo runbook depends on the log explaining a beat that did not fire. Only segments that actually carry the wake phrase are reported, so ordinary speech stays quiet. Verification - swift test --filter WakeWordServiceTests -> 11 passed - Regression tests added for each defect, carrying the live values that exposed them (speaker 0 with is_user=false, a reused segment id, a distinct command inside the cooldown). - Confirmed live after the fix: `WakeWord: submitting 'can you order food for me?' to the assistant`, and the re-delivered segment correctly suppressed. Honest gaps - The ~35s ambient latency is server-side (client ships audio every 100ms via audioBufferSize=3200); it is not addressed here and needs the realtime lane. - Not verified against an account with an enrolled speech profile.
The wake word dispatched with `fromVoice: false`, which is the flag that decides whether the assistant speaks its answer or only renders text. A wake word is a hands-free entry point by definition -- the user's hands and eyes are elsewhere -- so a silent reply strands the interaction the feature exists to enable. Observed live: the trigger fired and answered correctly, and the user heard nothing. PushToTalkManager already passes `fromVoice: true` for the same reason. Verification - swift build -> clean - swift test --filter 'WakeWordServiceTests|WakeWordSegmentParserTests' -> 21 passed Honest gaps - Voice follow-up is still unavailable from the wake word. `sendFollowUpQuery` exists and PushToTalkManager uses it with a voiceTurnID, but the wake word always opens a fresh query, and `guard !isConversationActive` suppresses the trigger while a turn is live. Wiring multi-turn to the wake word needs the realtime lane, not the ~35s ambient lane; tracked separately.
|
I took this branch out of unit tests and ran it against a live ambient session on macOS. It did not work, and the reason it did not work was never visible: every guard in Four defects, all found by running it. The unit tests passed before and after every one of them, because they construct segments the backend never actually sends. 1. Diarization attribution — the feature was dead on arrivalThe trigger required Speech-to-text heard the phrase perfectly. The wake word could not fire, and could not fire for any new user.
2. Speech-to-text spells the wake phrase by soundThe parser matched the literal string The recognizer was confident and correct about what it heard. Added the renderings recognizers actually emit, expanded through the existing greeting prefixes. Negative tests cover the obvious risk — "Omnibus schedule changed" and a bare "Oh me" with no command still do not fire. 3. Segment dedup dropped later commandsThe backend re-delivers one growing segment under a single id:
4. The cooldown ate distinct commandsThe cooldown is there to swallow a rapid repeat of the same utterance, but it gated on elapsed time alone. The ambient transcript lane runs well behind live speech, so several genuinely different commands arrive bunched inside one 30s window. Two consecutive "what time is it" attempts were both lost this way: Now gates on a repeat of the same command. A repeat inside 30s is still suppressed. 5. Replies were silentDispatch used DiagnosticsEvery rejection now names itself. Only segments that actually carry the wake phrase are reported, so ordinary speech stays quiet. These log lines are what located defects 1 and 3, and they are the difference between "it doesn't work" and a diagnosis. Verification
Honest gaps
|
On-device recognition has no keyword list, so it fronts the vowel with an
aspirate. Observed live from one speaker in a single session:
"Homi what's the weather? outworking."
"Homie, can you order food for me? Street drive to children dance."
Adds "homi" and "hommi". Deliberately excludes "homie": it is an ordinary
English word, and accepting it as the wake phrase would fire on real speech.
Scope note: this list is a safety net, not the fix. `TranscriptionService`
already seeds ["Omi", "OMI"] into the STT keyword boost, and on the cloud lane
the phrase transcribes exactly every time. These misses only occur on the
on-device lane, which takes no keyword list -- the durable fix is keyword
boosting on the recognizer, not a longer list here.
Verification
- swift test --filter WakeWordSegmentParserTests -> 10 passed
A backend segment is re-delivered as it grows, in place and under one id. The
previous dedup keyed on the exact extracted command, so every growth counted as
a new instruction and fired again with a longer string. Worse, the assistant's
own spoken reply is captured by the microphone and appended to the same
segment, so each re-fire submitted a more polluted command. Observed live:
12:41:32 submitting 'what time it is? You speak English. Got it.'
12:41:33 submitting 'what time it is? You speak English. Got it. Handed you this'
12:41:39 Transcript [ADD] Speaker 1: Handed you this An agent is getting started on that.
The query actually dispatched was the polluted string, which is not what the
user asked and cannot be answered usefully.
Deduping on the id alone is also wrong -- it drops a genuinely new instruction
that lands in a reused id. Treat a command that extends one already fired for
that segment as the same instruction, and anything else as new.
Verification
- swift test --filter WakeWordServiceTests -> 12 passed
- Regression test carries the live growth case.
- Confirmed live: one clean `submitting 'what time it is?'`, and the following
re-delivery correctly `ignored — already fired`.
Honest gaps
- The microphone capturing the assistant's own speech is a separate defect and
is not addressed here; this change only stops it corrupting the command.
An earlier commit on this branch switched the wake word to
`openAIInputWithQuery(command, fromVoice: true)` so the assistant would speak
its answer. That silently disabled the feature end to end.
`openAIInputWithQuery` gates the voice path on a turn it does not mint:
if fromVoice {
guard let voiceTurnID,
VoiceTurnCoordinator.shared.requireCurrentOwner(for: voiceTurnID) != nil
else { return }
`fromVoice: true` with no `voiceTurnID` fails that guard and returns with no log
and no user-visible effect. Every wake word therefore logged `submitting` and
dispatched nothing -- the trigger looked healthy in the log while the assistant
was never invoked.
`fromVoice: true` belongs to callers that own a voice-turn lifecycle:
PushToTalkManager begins a turn and passes both arguments. The wake word submits
an already-transcribed command and owns no turn, exactly like
DesktopAutomationBridge, which passes `fromVoice: false` for the same reason.
Verification
- swift build -> clean
- swift test --filter 'WakeWordServiceTests|WakeWordSegmentParserTests' -> 22 passed
- Confirmed live: `WakeWord: submitting 'what time is it?'` followed 1.3s later by
`WakeWord: ignored — assistant already busy`, i.e. the turn actually engaged.
With `fromVoice: true` that second line never appeared on any attempt.
Honest gaps
- The reply is no longer spoken aloud; it renders in the floating bar. Restoring
spoken replies requires minting a VoiceTurnID via
`VoiceTurnCoordinator.begin(intent:)` before dispatch and threading it through,
which is a separate change and is not attempted here.
|
Update after taking this branch out of unit tests and running it as a user on macOS for an afternoon. Three more defects, one of which was mine from the previous round, plus two honest limits I could not resolve. Correction to my previous commentThe commit I pushed to make the assistant speak its reply ( if fromVoice {
guard let voiceTurnID,
VoiceTurnCoordinator.shared.requireCurrentOwner(for: voiceTurnID) != nil
else { return }With no
Live proof of the difference: That second line means the turn actually engaged. It never appeared once on any attempt while Growing segments re-fired the wake wordA backend segment is re-delivered as it grows, in place, under one id. My earlier dedup keyed on the exact command, so every growth counted as a new instruction and fired again with a longer string. The assistant's own spoken reply is captured by the microphone and appended to that same segment, so each re-fire submitted a more polluted command: The query actually dispatched was the polluted string. Now a command that extends one already fired for that segment is treated as the same instruction; anything else is new. Aspirated renderingsOn-device recognition takes no keyword list and fronts the vowel with an aspirate — This list is a safety net, not the fix. Verification
Honest gaps — the two that matter most1. Ambient latency, ~15–35s. Measured across 11 segments at 34.3–36.6s in one session and ~25s in another. The client ships audio every 100ms ( A wake word answering 25 seconds later is not usable command-and-control, and I do not believe this branch can fix it. I think this needs the realtime lane, or a dedicated on-device keyword spotter running on raw audio rather than riding the ambient transcript. Happy to pursue either if you want it — it is a design decision above my pay grade to make unilaterally. 2. "Works once, then not again." Reproducible in my hands but I have not isolated the mechanism, so I am reporting it rather than guessing. Candidates I could not separate:
All three would present identically to a user. I would rather flag this than close it with a plausible story. Also unresolved:
|
Verified working on deviceRecorded a live session on macOS against the dev serving plane. Two hands-free wake word queries, both answered: No clicks, no Push-to-Talk, no keyboard — spoken into an ambient session with the Wake Word toggle on. Matching log for the same session: The second line is the growing-segment dedup correctly suppressing the re-delivery of the same instruction, which is the defect fixed earlier in this branch. Answers render in the chat panel. Repeat invocations work — commands need spacing, not a restartA longer session on the dev serving plane, three separate successful invocations in one process, no restart between them: The two ignored lines are the growing-segment dedup correctly suppressing re-deliveries of instructions that had already run — the fix from earlier in this branch working, not a failure. The middle trigger is worth noting for what it is: What the spacing actually is. Two independent constraints, not one:
In practice that means leaving roughly 15+ seconds between questions. Still open, unchanged
Composition checkThis branch was also exercised alongside #11864 (presence-aware notification suppression) in a single build, to confirm the two do not interfere: proactive notifications are withheld during a call while wake-word commands still dispatch and answer. Edited: this comment originally reported the feature as working "once per session, then not again" and listed three candidate causes I could not separate. That was wrong and understated the feature — the longer session above shows three invocations in one process. What I had read as a hard limit was the transcript latency and the same-command cooldown stacking while I retried too quickly. |
|
Thanks @aryanorastar — this round of hardening is exactly what the feature needed: taking it out of unit tests and into live sessions caught a class of issues the suite couldn't see, and the current head reads much better for it. What I verified on the current head:
Two non-blocking observations for the record:
The open item from the earlier walkthrough is unchanged: whether transcript-based triggering with auto-send and a visual-only reply is the intended v1 interaction (vs. keyword spotting / voice-turn replies) is a product direction call, and your in-thread rationale for the v1 approach is a reasonable answer to it. Human maintainer sign-off needed before merge: product decision on wake-word trigger semantics (transcript + homophone matching, auto-send UX). by AI on behalf of David. |
Review feedback on this PR: with "oh me" accepted as the wake phrase, an ordinary
sentence like "oh me and my friend went hiking" parses to the 2-word command
"and my friend went hiking" and auto-sends it. That false-positive surface was
introduced by the homophone table earlier in this branch.
The homophones and the literal spelling are not the same kind of evidence. Saying
"Omi" is deliberate -- nobody produces it mid-sentence by accident -- so
"Omi order food" needs no corroboration. A homophone is the recognizer guessing,
and its guesses are ordinary English, so it needs something more.
A bare homophone now has to be followed by a punctuation break: the recognizer's
own signal that the speaker addressed something and then paused. Every homophone
hit observed live carried one ("Oh me, how are you?"). A greeting prefix is
corroboration in its own right -- "hey oh me" is not said by accident -- so those
forms keep the ordinary word boundary.
"Omi order food" fires (literal, unchanged)
"Oh me, how are you?" fires (homophone + break)
"hey oh me order pizza" fires (greeting corroborates)
"oh me and my friend went hiking" ignored (bare homophone, no break)
"o me it has been a long day" ignored
This only ever makes the wake word fire less, so the risk it carries is a missed
trigger, never a spurious one.
Verification
- swift test --filter 'WakeWordSegmentParserTests|WakeWordServiceTests' -> 25 passed
(13 parser + 12 service; 3 new, every prior case still green)
- New cases use the reviewer's example sentence and the exact strings observed live.
- Confirmed live on the dev serving plane after the change, both forms in one session:
20:44:35 Transcript: "Hey Omi what's the time"
WakeWord: submitting 'what's the time' to the assistant
20:44:37 Transcript: "Omi, what is the time?"
WakeWord: ignored — assistant already busy
The second line is the trigger being correctly suppressed while the turn opened
by the first was still live, which is also proof the dispatch engaged.
Honest gaps
- Punctuation is a proxy for a spoken pause and depends on the recognizer emitting
it. A homophone rendered without punctuation will now be missed rather than
misfire, which is the safer direction but is still a miss.
- Does not address a wake phrase transcribed in a non-Latin script (the socket runs
language=multi); matching remains ASCII-only.
|
Thanks @Git-on-my-level — the The distinction the parser was missingThe literal spelling and a homophone are not the same kind of evidence. Saying "Omi" is deliberate — nobody produces it mid-sentence by accident — so A bare homophone now requires a punctuation break after it — the recognizer's own signal that the speaker addressed something and then paused. Every homophone hit I observed live carried one ("Oh me, how are you?"). A greeting prefix is corroboration in its own right — "hey oh me" is not said by accident — so those forms keep the ordinary word boundary. This only ever makes the wake word fire less, so the risk it carries is a missed trigger, never a spurious one. Verification
The second line is the trigger correctly suppressed while the turn opened by the first was still live — which also confirms the dispatch engaged rather than silently dropping. Honest gaps on this change: punctuation is a proxy for a spoken pause and depends on the recognizer emitting it, so a homophone rendered without punctuation is now missed rather than misfiring. And it does not address a wake phrase transcribed in a non-Latin script — the socket runs On your second noteThe cooldown keeping only Still yours to callThe product decision on trigger semantics (transcript-based detection with auto-send, and a visual-only reply) is unchanged and still needs a human. My rationale is in the thread above; happy to take it whichever way you decide, including moving to a dedicated keyword spotter on the realtime lane if that is the direction — which would also address the ~15–35s ambient latency I documented, the one limitation I do not think this branch can fix. |
The hosted runner's Azure Ubuntu mirror repeatedly stalled Redis setup until the 20-minute gauntlet deadline. Use the existing archive.ubuntu.com fallback directly for each Redis-dependent gauntlet.\n\nVerification:\n- actionlint -config-file .github/actionlint.yaml .github/workflows/backend-hermetic-e2e.yml\n- python3 backend/scripts/check_workflow_contracts.py --changed-files .github/workflows/backend-hermetic-e2e.yml\n- git diff --check\n\nFailure-Class: none
Same stall as backend-hermetic-e2e's Redis install (69e5a9e): the hosted runner's Azure Ubuntu mirror hangs InRelease fetches, this time in desktop-windows-ci's Linux runtime dependency install, until the job's 6-hour default timeout cancels it. Drop the mirror the same way, falling back to archive.ubuntu.com directly. Verification: - actionlint -config-file .github/actionlint.yaml .github/workflows/desktop-windows-ci.yml - git diff --check Failure-Class: none
Statistics — measured, not estimatedSame measurement pass as #11864, restricted to the wake word. Source is 78 real sessions, 32.6 h of logged runtime, Aug 18–20, on the dev serving plane with a real Firebase identity and real ambient transcription. No fixtures, no replay, no synthetic segments — which matters here specifically, because every defect this feature had was invisible to unit tests that constructed segments the backend never actually sends. Triggers, before vs afterBefore is exactly 0. There was no hands-free path to the assistant during ambient listening at all. Counts split at
Across the whole window: 62 raw wake-phrase matches, 27 dispatched, 35 correctly refused. Refusal breakdown:
The 35 refusals are the number worth reading. Ambient transcripts arrive as revisions — the same utterance is re-sent as the backend refines it — so a matcher that fires on text alone triggers 2–4 times per phrase. Segment-scoped prefix dedup is what turns 62 raw matches into 27 dispatches. Speaker attribution accounts for 10 more: those are other people saying "Omi" near my machine, which must not command my assistant. The 3 redundant re-fires are all pre-fix, and are exactly what False positives0 false triggers reached the assistant. All 27 dispatches are real commands: "what time is it", "what's the weather", "tell me the latest news", "can you order food for me". Nothing fired on ambient conversation that merely contained a homophone. That is downstream of the punctuation-break rule in LatencyNot a defect and not fixable here: dispatch happens the moment the segment is delivered, and the ambient lane runs ~15–35 s behind live speech by design. In one trace, Method and limits
Edited: an earlier revision of this comment reported the 3 pre-fix re-fires as a live defect and inferred that segment ids are unstable across revisions. Both were wrong — I had misread a pre-fix log line, since the old code logged the current command in the |
…llation Detect Hermetic Backend Scope was cancelled mid-run by a runner race, so Backend Hermetic Merge Gate read SCOPE_RESULT: cancelled and failed closed. Distinct from the apt-mirror stall in BasedHardware#11848 — this one never reaches a package fetch. No code change; the branch has no admin rights to rerun the workflow directly.
|
@undivisible thanks for merging #11901 — and for saying what actually bugged you, that's the useful kind of bug report. Two macOS PRs of mine are sitting on a human decision rather than on code, if you have a minute: This one — hands-free wake word. Say "Omi, what's the weather" during ambient listening and it runs the command. @Git-on-my-level's passes found no blocking code issues; the one open item is a product call — transcript-based triggering with auto-send, vs a proper keyword spotter. My reasoning is in the thread and I'm happy either way, it just needs someone to pick. #11864 — withhold proactive notifications while someone else is present. Screen share or a live call, including muted browser calls. Also adds the "Silence Notifications" control. The change the review asked for is fixed and pushed. Both have live session evidence and measured numbers in the threads — presence suppression is 9/9 true positives cross-checked against an independent detector, 0 false positives. The honest caveat on this one is already in the thread and worth reading before you try it: the ambient transcript lane runs ~15–35s behind live speech, so a wake-word answer arrives well after you speak. Functional, but not command-and-control. Fixing that needs the realtime lane or a dedicated keyword spotter, which is the same product call above. No rush on either. |
# Conflicts: # desktop/macos/e2e/flows/capture-lifecycle.yaml
|
Code-wise, this is complete and ready for a maintainer decision. The latency fix, wake-word flow, spoken/realtime evaluation path, and playback-echo protection are implemented and verified; the current head is mergeable and all checks are green. The only remaining blocker is product sign-off on transcript-based auto-send and the always-on echo filter. I’m not holding any unfinished code locally—once maintainers choose the direction, I can make any requested adjustment. |
… panel Hands and eyes are elsewhere — that is the premise of a wake word — so growing the bar into a response panel covered a fifth of the screen for no one's benefit. The response glow already signals that Omi is working and the answer is spoken, so the exchange is recorded without being put in front of the user. The panel remains the fallback when the hands-free path cannot run. Two things made this harder than it looks, both found by instrumenting rather than reading: The panel is opened from four places — the query starting, the answer arriving, the content-height observer, and the agent-chat resize. Suppressing any one of them left it showing, because whichever was missed put it straight back. The guard is now at `resizeAnchored`, the single point they all reach, and blocks only expansion so collapsing back to the pill still works. `prepareVisibleQueryState` runs twice for one query: once from `routeQuery` to show the thinking state, once from `sendAIQuery`. A one-shot latch consumed by the first call meant the second reset the flag and reopened the panel — `presentsSurface=false` immediately followed by `presentsSurface=true` in the same millisecond. It may now only ever set quiet; the entry points that do want a panel (typed follow-ups, `submitSpokenCommand`) clear it. Verification: - xcrun swift test --package-path Desktop --filter 'WakeWord|VoicePlaybackEcho|FloatingControlBarState|Realtime' -> 335 tests, 0 failures - Live on a named dev bundle, real microphone, two consecutive commands: the bar stays at 62px throughout (`resizeToFrame to (351.0, 62.0)`), no `430x381` expansion, and both answers were spoken — the microphone transcribed Omi saying "The capital of Portugal is Lisbon." and "The sun contains about 99.86% of the total mass in our solar system." The previous build logged `resizeToFrame to (430.0, 381.0)` on every command. Failure-Class: none Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Git-on-my-level
left a comment
Git-on-my-level
left a comment
There was a problem hiding this comment.
Follow-up on the new head 7fd0e2d — the notch-answering work in ccaaebb plus the main merge. The hands-free direction reads well against the feature's premise, and guarding expansion at the single choke point all four panel entry points reach (FloatingControlBarWindow.resizeAnchored, with the resizeToResponseHeight / beginMainResponseHeight guards beside it) is the right shape.
One blocking item — a pinned source contract went stale and both required Desktop Swift checks are red because of it:
- Desktop Swift Static & Test Contracts and Desktop Swift Build & Tests fail on AgentPillLifecycleTests.testTypedSendDelegatesResponseSizingToWindow (desktop/macos/Desktop/Tests/AgentPillLifecycleTests.swift:1083), which asserts the exact declaration
func beginVisibleMainQuery(_ message: String, fromVoice: Bool, animated: Bool = true). ccaaebb extended that signature withpresentsSurface: Bool = trueand wrapped it across lines (FloatingControlBarWindow.swift:5363), so the pinned string no longer matches. It is the only failing suite in the run, and the production behavior is intact — the view's call site still passes defaults, so typed sends present the panel. The fix is to update the pin to the new signature; ideally also extend that contract to coverpresentsSurfacedefaulting to true and theanswersQuietlyguards, since that is now the load-bearing behavior of this surface.
Two non-blocking hardening notes on the new quiet-answer plumbing:
- submitHandsFreeCommand sets the static
suppressNextVisibleSurfacelatch before sendFollowUpQuery; if openAIInputWithQuery early-returns on its window/provider guards the latch is never consumed, and the next typed query through prepareVisibleQueryState would silently answer quietly once. Clearing the latch on that early-return path (or consuming it on failure) closes it. state.answersQuietlyis cleared in the onSendQuery re-wire (FloatingControlBarWindow.swift:3838) and submitSpokenCommand (:4128), but the default wiring installed in setup (:3076) does not clear it, and the comment at the flag's set-site (:5370) says any surface-presenting query clears it — the code does not quite do that yet. Moving the clear into beginVisibleMainQuery when presentsSurface is true would make the invariant local to the flag's owner.
Also re-verified on this head, unchanged from the earlier pass: the echo-policy prefix match with residue preservation in VoicePlaybackEchoPolicy.swift ahead of WakeWordService.observe in AppState+ListenEvents.swift, homophone-tolerant command extraction in WakeWordSegmentParser.swift, and capture-lifecycle.yaml's covers list includes the new sources.
Once the pinned contract is updated and both Desktop Swift checks are green, this is in good shape for a maintainer's product pass — the hands-free wake-word surface (default-off setting, notch-quiet answers, optional realtime path behind a flag) is a direction call that deserves maintainer sign-off before merge.
by AI on behalf of David — if you need David’s attention urgently, please @Git-on-my-level and escalate with need human response.
Git-on-my-level
removed
the
positive-signal
# Conflicts: # desktop/macos/Desktop/Sources/FloatingControlBar/FloatingBarVoicePlaybackService.swift
…d signature ccaaebb added `presentsSurface: Bool = true`, which pushed the parameter list onto its own line under swift-format. The contract test pinned the declaration as one string: func beginVisibleMainQuery(_ message: String, fromVoice: Bool, animated: Bool = true) so it stopped matching and took both required Desktop Swift checks red. The production behavior was never affected -- the view's call site still passes defaults. Pinned in two parts so a wrap cannot break it again, and `presentsSurface: Bool = true` is now pinned with the rest: that default is what keeps a typed send presenting the panel, which is the load-bearing behavior of this surface. Checked the pin still guards rather than passing vacuously -- flipping the expected default to false fails the assertion at line 1088; restoring it passes. AgentPillLifecycleTests 84/84. Failure-Class: none
|
@Git-on-my-level the stale pin is fixed on
Also merged current main; the Your two non-blocking notes (the |
The wake word is only as good as the recognizer that hears it, and on the
default path the recognizer cannot be told its own name. Ambient transcription
runs on-device on Apple Silicon, and `AsrManager.transcribe(_:decoderState:
language:)` takes a language hint and nothing else — no keyword or vocabulary
parameter. The cloud lane does not have that problem: `/v4/listen` prepends
"Omi" to the STT keyword vocabulary server-side in
`backend/utils/listen_session_bootstrap.py`.
Measured on one machine, same script and voices, only the lane changed: the
phrase was usable in 12 of 20 utterances on-device against 19 of 20 on the
cloud lane. Seven of the eight on-device misses came back as "Only".
Two changes, both aimed at that gap.
`WakeWordSegmentParser` gains a third corroboration class. "Only" cannot ride
the existing punctuation-break rule: a scan of 1,919 stored local segments
found 15 sentence-initial "Only", every one a misrendered wake word, and not
one carried a break after it — they read "Only what is on my calendar", "Only
open my rewind timeline". The same scan found 8 ordinary uses of "only", all
mid-sentence. So `.commandHead` gates on what follows instead: an interrogative,
or a request verb aimed at the speaker's own things ("show me", "open my",
"remind me"). Every imperative reads fine under a restriction — "only do that
once", "only tell him if he asks" — but none of them ask for the speaker's own
calendar.
`AssistantSettings.wakeWordPrefersCloudSTT` is the opt-in for users who would
rather buy the remaining accuracy with cloud transcription. Off by default: it
trades on-device transcription for cloud transcription while the wake word is
enabled, which is a privacy and cost decision, not a technical one. FluidAudio
does ship term biasing, but only on `SlidingWindowAsrManager`, which is a
different streaming architecture and pulls a second set of CTC models — that is
the upgrade path out of this flag.
Verified on the running app through the hermetic capture seam, which reaches
`WakeWordService.observe` on the real path: 14 ordinary uses of "only" injected,
zero fired; 3 verbatim misrenderings injected, 3 parsed. The cloud opt-in was
confirmed live by the lane actually switching — `TranscriptionService:
Connecting to wss://api.omiapi.com/v4/listen` in place of on-device Parakeet.
Tests: 26 in WakeWordSegmentParserTests, 14 in STTSessionStateTests, and the
full 716-suite run, all passing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…r to it
A backend segment is re-delivered as it grows, in place and under one id, and the
re-delivery arrives whether or not the speaker added anything. `VoiceBargeInPolicy`
treated any non-empty user text as speech, so an unchanged re-delivery counted as
the user talking over the assistant.
Observed live: the second turn of a conversation was cut off 4.4s into playback by a
re-delivery of the very segment that asked the question, byte-identical to the copy
already stored.
13:16:41.117 Chat response complete
13:16:45.491 BARGE-IN: User spoke mid-playback; interrupting voice output
13:16:45.501 Transcript [UPDATE] Speaker 0: I'm fine. What about you? Omi, I'm fine. What about you?
The first turn escaped only by timing — its answer finished speaking before the
re-delivery landed. From the second turn on, the re-delivery falls inside the playback
window every time, which is why it reads as "the second question always gets cut off".
`shouldInterrupt` now takes the text already stored for that segment id and interrupts
only on what the segment gained. Growth that adds nothing but punctuation or spacing is
the recognizer tidying up, not speech. A segment that no longer extends what was stored
counts as new in full: there is no way to tell a revision from a continuation, and
missing a real barge-in is the worse failure.
Failure-Class: none
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Omi's own answer was reaching the transcript as a second speaker, so the reply was
heard once from the speakers and again as if a person had said it.
`walk` only commits a matched word once two match in a row — a guard that exists
because a single common word ("the", "and") matching by coincidence once let the
backward walk eat "the time" off the end of a user's command. But an utterance's
closing word can never reach two-in-a-row when the word before it was garbled, so
the last word of a short answer was permanently uncountable.
Measured live: Omi said "I don't know the details of Wake Word yet." and the
microphone returned "WakeMore" for the product name. The trailing "yet" followed
that one mismatch and could not commit, so coverage came out 6 of 8 = 0.75, under
the 0.80 floor, and the segment was kept.
19:37:59.241 Transcript [ADD] Speaker 1: I don't know the details of WakeMore yet.
19:38:00.576 ECHO: dropped ... I don't know the details of WakeMore yet. Feature you design
Confirmed against the real strings before and after: `keep` became `drop` for every
rendering of what was actually spoken.
The final token may now commit on a single match, but only after `minimumWordCount`
words have already matched in a run, so a lone coincidental word still cannot start
or extend an echo. Forward walk only: the reversed walk measures the playback that
follows a barge-in, where the "final" token is the first word the user said, and
committing it on one match would eat the start of the interruption.
Failure-Class: none
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Git-on-my-level
left a comment
Git-on-my-level
left a comment
There was a problem hiding this comment.
Follow-up on the new head 00e7a20c — the four commits since 7fd0e2d5: the contract repin (3bb78d66), the on-device recognition work (21f1e50f), the re-delivery barge-in fix (f1f9ba09), and the echo-tail fix (00e7a20c).
The previous blocker is resolved. AgentPillLifecycleTests now pins beginVisibleMainQuery in two parts matching the wrapped signature including presentsSurface: Bool = true, and both required Desktop Swift checks are green on this head. The earlier changes-requested review is being dismissed as resolved.
The delta itself checks out:
21f1e50f— the default on-device lane now recognizes the wake word through evidence-based renderings rather than forcing the cloud lane:WakeWordSegmentParser.commandShapedRenderingsaccepts sentence-initial "Only" only when an interrogative or a self-addressed request verb follows (opensLikeACommand), andSTTSessionState.resolveModekeepswakeWordNeedsRecognizableNameas an opt-in cloud override that asessionForceLocalfallback correctly outranks (pinned inWakeWordSegmentParserTestsandSTTSessionStateTests, including the not-dragged-back-to-cloud case). The privacy trade stays documented at the resolver, off by default with no settings surface toggling it yet.f1f9ba09—VoiceBargeInPolicy.newSpeechcounts only what a re-delivered segment gained, withpreviouslyHeardfed from the stored copy of the same segment id inAppState+ListenEvents.swift; the four new test cases cover unchanged re-delivery, punctuation-only growth, genuine additions, and recognizer revisions.00e7a20c—VoicePlaybackEchoPolicy.walklets an utterance's final token commit on a single match once a run ofminimumWordCountwords has already established the echo (commitsFinalSingleMatch), with the garbled-word regression and the lone-match negative both pinned.
One blocking item — a stale changelog fragment duplicates an already-shipped entry:
desktop/macos/changelog/unreleased/20260823-local-transcription-pause-endpointing.jsonis byte-identical to the entry that already shipped in v0.12.221 via #12181, and this PR's own status note says that surface is off the PR now. If this merges, the next consolidation would re-announce an already-released change. Deleting the fragment is the whole fix.
Two non-blocking hardening notes from the earlier pass remain open on this head, unchanged and still non-blocking:
openAIInputWithQuery's early-return guards (window/provider) still return without consumingSelf.suppressNextVisibleSurface, so a latch set bysubmitHandsFreeCommandcan survive and quiet the next typed query once.answersQuietlyis still cleared only in theonSendQueryre-wire andsubmitSpokenCommand; the default wiring installed in setup does not clear it, and the comment at the flag's set-site still promises more than the code does. The double-call hazard now documented atprepareVisibleQueryStateexplains why the clear cannot simply move intobeginVisibleMainQuery— a central clear that respects that hazard (e.g. keyed to query generation) would close the gap.
With the fragment dropped, what remains on this PR is the product decision — whether the hands-free wake-word surface (default-off setting, notch-quiet answers, optional realtime lane behind wakeWordUsesRealtime) is the direction Omi wants — which needs maintainer sign-off before merge.
by AI on behalf of David — if you need David’s attention urgently, please @Git-on-my-level and escalate with need human response.
Git-on-my-level
added
the
docs-accuracy
`20260823-local-transcription-pause-endpointing.json` is byte-identical to the entry released in v0.12.221, which shipped through BasedHardware#12181 when pause endpointing was split off this branch. Leaving it here would re-announce an already-released change at the next consolidation. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
@Git-on-my-level the fragment is deleted on need human response — this needs David, not another verification pass. Four of your passes have now ended the same way: no code-level blockers, the decision belongs to a maintainer. That decision has been open since your first review. In the meantime each new head I push mostly generates mechanical findings an exception line, a stale pin, this fragment — against a gate that isn't code. So I'd rather stop adding to this PR and get the call. Three questions, each answerable in a line:
A "no" on any of these is more useful to me than the current state. Known and not fixed here: 4.4–6.1s from end of speech to Omi speaking. Measured, it's all the agent runtime — 13–28k input tokens for a one-line question — not detection, which fires in milliseconds. Follow-up PR unless you want it in this one. Your two non-blocking hardening notes I'd also take as a follow-up, with tests rather than folded in unverified. |
|
Follow-up on Required Desktop Swift checks, Two hardening notes from the earlier pass remain open, unchanged and still non-blocking:
With the mechanical blocker gone, what remains is the product decision — whether the hands-free wake-word surface (default-off setting, notch-quiet answers, optional realtime lane behind by AI on behalf of David — if a specific product or safety concern needs his attention, please @Git-on-my-level and escalate with |
Git-on-my-level
removed
the
docs-accuracy
…next query `suppressNextVisibleSurface` is a one-shot latch consumed only by `prepareVisibleQueryState`. Three exits from `openAIInputWithQuery` return before reaching it — no window, no active provider, and the `.voiceOnly` branch, which renders nothing and so never consumes it at all. A wake word that fires while the bar is torn down or unprovisioned therefore leaves the latch set, and the next typed question is silently answered in the notch. Clear the latch on every exit that abandons the query. Separately, `beginVisibleMainQuery` only ever *sets* `answersQuietly` (clearing there would let the second `prepareVisibleQueryState` call undo the first), so the reset has to live at the query entry points. The closure re-wired in `openAIInputWithQuery` does it; the default wiring installed at setup did not, so a quiet wake-word answer leaked its silence into the next typed question on that path. Verification: `swift build` in `desktop/macos/Desktop` — clean. No behavioral test added: both paths run through `FloatingControlBarWindow` (an NSPanel) with no injectable seam, and the existing coverage here is source-scrape pins, which AGENTS.md calls a static tripwire rather than behavioral coverage. Failure-Class: none
|
Both hardening notes are fixed on The latch leak. You were right, and it is worse than the two guards you named:
Verified with Product decision on the wake-word surface is still the only thing outstanding. |
|
Follow-up on The latch leak is closed at all three exits.
Skipping a behavioral test for these is a fair call given the surface: both paths run through One narrower residual edge of the same class, non-blocking: in The rest of the head is unchanged from the verified What remains is unchanged and is not a code matter: the product decision on whether the hands-free wake-word surface (default-off setting, notch-quiet answers, optional realtime lane behind by AI on behalf of David — if you need David’s attention urgently, please @Git-on-my-level and escalate with |
# Conflicts: # desktop/macos/Desktop/Sources/FloatingControlBar/FloatingBarVoicePlaybackService.swift
|
Verification pass on the merge head
Conclusion unchanged from the last pass: the code side is done and now re-verified against the merged base; what remains is the product decision on whether the hands-free wake-word surface (default-off, notch-quiet answers, optional realtime lane, always-on playback-echo filter) is the direction Omi wants for macOS. Leaving for human maintainer review on that direction call. by AI on behalf of David — if you need David’s attention urgently, please @Git-on-my-level and escalate with |
# Conflicts: # desktop/macos/Desktop/Tests/AgentPillLifecycleTests.swift
|
Verification pass on the merge head
One ask from this merge: the only test pinning the quiet-answer latch was dropped in the conflict resolution. Carried nit from the last pass, still present: the ordering comment in Conclusion unchanged: code verified against the merged base; the remaining gate is the product decision on whether the opt-in hands-free wake-word surface — default-off, notch-quiet answers, optional realtime lane, always-on playback-echo filter, and the cloud-STT opt-in trade — is the direction Omi wants for macOS. by AI on behalf of David — remaining gate is the product sign-off on the hands-free wake-word surface and its cloud-STT privacy trade. |
Git-on-my-level
added
the
needs-tests
2 participants
Summary
Adds an opt-in Wake Word feature for macOS desktop: during ambient listening, say
"Omi"followed by a command and the assistant runs it hands-free — no clicks, no Push-to-Talk.What changed
Sources/WakeWord/WakeWordService.swift(new) — trigger engine: strips the wake phrase, extracts the command, and submits it to the assistant (openAIInputWithQuery). Guards: 30s cooldown against rapid repeats, per-segment-ID deduplication, user-speech only, and suppression while the assistant is already busy.Sources/WakeWord/WakeWordSegmentParser.swift(new) — splits "Omi, do X" into wake phrase + command; requires a 2+ word command so a bare "Omi" never misfires.AppState+ListenEvents.swift— feeds every incoming transcript segment intoWakeWordService.observe(...), the single funnel for ambient speech.AssistantSettings.swift— persistedwakeWordEnabled/wakeWordPhrase/wakeWordCooldown(default off so current users' behavior is unchanged).SettingsContentView+General.swift— new Wake Word settings card with on/off toggle and a dynamic subtitle tied to the audio-recording mode ("Listens in the background during meetings and calls").WakeWordServiceTests+WakeWordSegmentParserTests(15 cases).How it works
Verification
xcrun swift test --package-path Desktop --filter WakeWord→ 15 tests, 0 failures.handleBackendSegments→WakeWordServicepath):Negative control: injecting
"I was just saying the weather is nice today"produced no trigger (no wake phrase).changelog/unreleased/20260818-wake-word-trigger.json). Config rachets (check_desktop_test_quality.py, swift-format, SwiftLint) pass.Follow-up (out of scope, separate issue)
While validating the demo bundle we hit a pre-existing macOS debug-build crash at startup in
DesktopHomeView.restorePersistedCaptureServices → startTranscription(over-release, unaffected by this diff — reproduces with the wake word disabled and only manifests once mic permission is granted). Filed separately; does not block this feature.Screenshots
Product invariants affected
Failure class (fixes)
Failure-Class: none
Line-count exception
openAIInputWithQueryandsendFollowUpQuerygained the turn-optional guard that lets awake word — which owns no
VoiceTurnID— reach the visible surface as a voice query, so itsanswer is spoken and a second command continues the conversation. Both are existing
functions in this file; splitting it is unrelated refactoring for a 16-line change.
Line-Count-Exception: desktop/macos/Desktop/Sources/FloatingControlBar/FloatingControlBarWindow.swift | 5411 -> 5483 | wake-word spoken answers and follow-up continuity route through the two existing dispatch entry points
Line-Count-Exception: desktop/macos/Desktop/Sources/FloatingControlBar/RealtimeHubController+SessionDelegate.swift | 1499 -> 1504 | Five lines recording session-spoken text into the playback history the echo policy reads; it belongs at the delegate that receives the text, and the surrounding turn-event guards are private to this file.
Line-Count-Exception: desktop/macos/Desktop/Sources/FloatingControlBar/RealtimeHubController.swift | 1540 -> 1622 | runWakeWordTurn sits beside runHeadlessPTTTurn because it shares that method's turn lifecycle exactly — mint an automation turn, select the hub route, open the input window, commit — and the two must stay in step; a sibling file would duplicate the ordering constraints rather than share them.
Line-Count-Exception: desktop/macos/Desktop/Sources/AppState/AppState+Transcription.swift | 1744 -> 1745 | One argument line at the single beginRecording call site, telling the STT lane resolver whether the wake word needs a recognizer that can be told its name; the flag combination itself lives on AssistantSettings, not here.
Line-Count-Exception: desktop/macos/Desktop/Sources/FloatingControlBar/RealtimeHubSession.swift | 1661 -> 1669 | sendSpokenCommand is an eight-line sibling of sendTestTextInput sharing the same private buffering path; the wire form and buffer are private to this file.