[feature] voice typing runs the transcript past a model before it pastes - #323
Merged
Conversation
macOS voice typing pasted exactly what the STT heard: the filler words, the false starts, the missing punctuation, the wall with no paragraph breaks. iOS 1.6 fixed that on the phone with `TranscriptPolisher`; this is the desktop half. The clean-up runs at the end of `finalize()`, between the transcript settling and the clipboard. That placement is the whole design, and it is forced: the paste is a blind ⌘V into somebody else's app — no undo, no re-selection, no second chance — so the moment before it is the last moment the text is still ours. Polishing after the paste would mean typing over a window we do not own. Which makes latency the constraint, and everything here is shaped by one rule borrowed from the iOS pass: **this must never make dictation worse.** - Not configured, no network, a refusal, a model that answered the transcript instead of cleaning it — every path returns `null`, and `null` means "paste the raw text". There is exactly one thing for the caller to handle. - `POLISH_TIMEOUT_MS` (4 s) bounds the round trip rather than the provider doing it. The user has stopped talking and their sentence has not appeared anywhere yet; a model having a bad minute must not be able to hold it hostage. - Under `MIN_POLISH_CHARS` (8) there is no round trip at all. A short phrase has no filler to remove and no paragraphs to break, and it is exactly where a pause is felt. - `acceptPolish` is the last gate before text the user did not say replaces text they did: a 0.3–2.0 length band catches answers, summaries, translations and truncations, and a Simplified-drift check catches the one failure that looks like success. Only *newly introduced* simplified characters count, so someone who dictated Simplified gets their own script back untouched. - The user's personal dictionary rides along as protected terms (capped at 30, newest first). Those are words they already corrected by hand; a clean-up that "fixes" them back is the kind of help nobody asked for. Model comes from the existing `realtime` lane, so BYO keys, Ollama and the hosted `parley` provider all work with no new configuration. A user with no realtime provider gets an amber note in Settings rather than a toggle that silently does nothing. The overlay gets its own `polishing` phase instead of reusing `finalizing`. This is the one beat in the pipeline the user waits a noticeable moment for, and a spinner that does not say why reads as a hang. Default on; `settings.voiceTypingPolish` turns it off.
✅ SonarQube Quality Gate passed — pathorsAI_parley0 open issues on this PR. |
4 s was picked as "well short of a crash", which is the wrong thing to measure against. The thing being protected is not the polish, it is the typing: someone mid-sentence would rather have their raw words now than their tidy words in a beat they can feel. A realtime-lane model answers a dictation-length prompt in well under a second, so 2 s is still several times the healthy case. Losing a polish to the clock costs a few filler words; making people wait costs the feature.
`finalize` was at cognitive complexity 18 against a limit of 15 (S3776) — the polish block's nested `if`s were the difference. Extracted as `polishForPaste`, which is a better shape anyway: it is total, returning the text to paste rather than a maybe-polished maybe-null that `finalize` has to unpack, so the caller is back to one idea per line. Also S7748: `2.0` → `2` in the accept band.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
macOS voice typing pasted exactly what the STT heard — filler words, false
starts, missing punctuation, one wall with no paragraph breaks. iOS 1.6 fixed
that on the phone with
TranscriptPolisher. This is the desktop half.Where it runs, and why there is no choice about it
At the end of
finalize(), between the transcript settling and the clipboard.The paste is a blind ⌘V into somebody else's app: no undo, no re-selection, no
second chance. The moment before it is the last moment the text is still ours.
Polishing after the paste would mean typing over a window we do not own — in
a Chromium/Electron field that eats whatever the user has already started
typing.
So the clean-up goes in front of the paste, and latency becomes the constraint.
The rule everything is shaped by
Borrowed from the iOS pass: this must never make dictation worse.
null→ paste the raw textPOLISH_TIMEOUT_MS(2 s) → paste the raw textacceptPolishrejects → paste the raw textpolishTranscriptreturnsstring | nullrather than throwing, so the callerhas exactly one thing to handle.
The guards
Anything outside that is a different kind of output.
introduced simplified characters count — someone who dictated Simplified
gets their own script back untouched.
are words the user already corrected by hand; a clean-up that "fixes" them
back is the kind of help nobody asked for.
appeared anywhere yet. A model having a bad minute must not hold it hostage.
Model source
The existing
realtimelane — so BYO keys, Ollama and the hostedparleyprovider all work with no new configuration, and the user picks the model in
Settings as they already do.
A user with
voiceTypingPolishon but no realtime provider gets an amber notein Settings rather than a toggle that silently does nothing.
canPolishgatesthe pipeline separately, so the overlay never shows a polishing state that
cannot happen.
Overlay
A dedicated
polishingphase rather than reusingfinalizing. This is the onebeat in the pipeline the user waits a noticeable moment for, and a spinner that
does not say why reads as a hang. Sparkles icon + a "Polishing…" pill in the
same language as the existing "Copied" confirmation.
Settings
voiceTypingPolish, default on. Toggle sits above Activation in VoiceTyping settings, with
zh-TWandenstrings for all three new keys.Testing
bunx tsc --noEmitclean.bunx vitest run287/287, including 18 new tests insrc/lib/voiceTyping/polish.test.ts— the length gate, the term cap, theaccept band at its exact boundaries, both directions of the Simplified check,
the shared-character non-firing case, and all three
canPolishstates.dictation went through the new path, so the overlay's
polishingstate and thereal round-trip latency are unverified by me. The parts that are pure logic are
covered by tests; the wiring is not.
The timeout, and why it is 2 s
Deliberately tight. A realtime-lane model answers a dictation-length prompt in
well under a second, so 2 s is already several times the healthy case — and the
thing being protected is not the polish, it is the typing. Someone mid-sentence
would rather have their raw words now than their tidy words in a beat they can
feel. Losing a polish to the clock costs a few filler words; making people wait
costs the feature.
🤖 Generated with Claude Code