Skip to content

[feature] voice typing runs the transcript past a model before it pastes - #323

Merged
YJack0000 merged 3 commits into
mainfrom
feature/desktop-voice-polish
Sep 2, 2026
Merged

[feature] voice typing runs the transcript past a model before it pastes#323
YJack0000 merged 3 commits into
mainfrom
feature/desktop-voice-polish

Conversation

@YJack0000

@YJack0000 YJack0000 commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

What

macOS voice typing pasted exactly what the STT heard — filler words, false
starts, missing punctuation, one wall with no paragraph breaks. iOS 1.6 fixed
that on the phone with TranscriptPolisher. This is the desktop half.

Where it runs, and why there is no choice about it

At the end of finalize(), between the transcript settling and the clipboard.

The paste is a blind ⌘V into somebody else's app: no undo, no re-selection, no
second chance. The moment before it is the last moment the text is still ours.
Polishing after the paste would mean typing over a window we do not own — in
a Chromium/Electron field that eats whatever the user has already started
typing.

So the clean-up goes in front of the paste, and latency becomes the constraint.

The rule everything is shaped by

Borrowed from the iOS pass: this must never make dictation worse.

Failure What happens
No realtime provider, no network, HTTP error, refusal null → paste the raw text
Model is slow POLISH_TIMEOUT_MS (2 s) → paste the raw text
Transcript under 8 chars No round trip at all
Model answered / summarised / translated / truncated acceptPolish rejects → paste the raw text

polishTranscript returns string | null rather than throwing, so the caller
has exactly one thing to handle.

The guards

  • Length band 0.3–2.0. A clean-up shortens a little and lengthens a little.
    Anything outside that is a different kind of output.
  • Simplified drift. The one failure that looks like success. Only newly
    introduced
    simplified characters count — someone who dictated Simplified
    gets their own script back untouched.
  • Personal dictionary as protected terms, capped at 30, newest first. Those
    are words the user already corrected by hand; a clean-up that "fixes" them
    back is the kind of help nobody asked for.
  • Its own timeout. The user has stopped talking and their sentence has not
    appeared anywhere yet. A model having a bad minute must not hold it hostage.

Model source

The existing realtime lane — so BYO keys, Ollama and the hosted parley
provider all work with no new configuration, and the user picks the model in
Settings as they already do.

A user with voiceTypingPolish on but no realtime provider gets an amber note
in Settings rather than a toggle that silently does nothing. canPolish gates
the pipeline separately, so the overlay never shows a polishing state that
cannot happen.

Overlay

A dedicated polishing phase rather than reusing finalizing. This is the one
beat in the pipeline the user waits a noticeable moment for, and a spinner that
does not say why reads as a hang. Sparkles icon + a "Polishing…" pill in the
same language as the existing "Copied" confirmation.

Settings

voiceTypingPolish, default on. Toggle sits above Activation in Voice
Typing settings, with zh-TW and en strings for all three new keys.

Testing

  • bunx tsc --noEmit clean.
  • bunx vitest run 287/287, including 18 new tests in
    src/lib/voiceTyping/polish.test.ts — the length gate, the term cap, the
    accept band at its exact boundaries, both directions of the Simplified check,
    the shared-character non-firing case, and all three canPolish states.

⚠️ Not exercised end-to-end. The Tauri app was not launched — no real
dictation went through the new path, so the overlay's polishing state and the
real round-trip latency are unverified by me. The parts that are pure logic are
covered by tests; the wiring is not.

The timeout, and why it is 2 s

Deliberately tight. A realtime-lane model answers a dictation-length prompt in
well under a second, so 2 s is already several times the healthy case — and the
thing being protected is not the polish, it is the typing. Someone mid-sentence
would rather have their raw words now than their tidy words in a beat they can
feel. Losing a polish to the clock costs a few filler words; making people wait
costs the feature.

🤖 Generated with Claude Code

macOS voice typing pasted exactly what the STT heard: the filler words, the
false starts, the missing punctuation, the wall with no paragraph breaks. iOS
1.6 fixed that on the phone with `TranscriptPolisher`; this is the desktop half.

The clean-up runs at the end of `finalize()`, between the transcript settling
and the clipboard. That placement is the whole design, and it is forced: the
paste is a blind ⌘V into somebody else's app — no undo, no re-selection, no
second chance — so the moment before it is the last moment the text is still
ours. Polishing after the paste would mean typing over a window we do not own.

Which makes latency the constraint, and everything here is shaped by one rule
borrowed from the iOS pass: **this must never make dictation worse.**

- Not configured, no network, a refusal, a model that answered the transcript
  instead of cleaning it — every path returns `null`, and `null` means "paste
  the raw text". There is exactly one thing for the caller to handle.
- `POLISH_TIMEOUT_MS` (4 s) bounds the round trip rather than the provider
  doing it. The user has stopped talking and their sentence has not appeared
  anywhere yet; a model having a bad minute must not be able to hold it
  hostage.
- Under `MIN_POLISH_CHARS` (8) there is no round trip at all. A short phrase
  has no filler to remove and no paragraphs to break, and it is exactly where
  a pause is felt.
- `acceptPolish` is the last gate before text the user did not say replaces
  text they did: a 0.3–2.0 length band catches answers, summaries, translations
  and truncations, and a Simplified-drift check catches the one failure that
  looks like success. Only *newly introduced* simplified characters count, so
  someone who dictated Simplified gets their own script back untouched.
- The user's personal dictionary rides along as protected terms (capped at 30,
  newest first). Those are words they already corrected by hand; a clean-up
  that "fixes" them back is the kind of help nobody asked for.

Model comes from the existing `realtime` lane, so BYO keys, Ollama and the
hosted `parley` provider all work with no new configuration. A user with no
realtime provider gets an amber note in Settings rather than a toggle that
silently does nothing.

The overlay gets its own `polishing` phase instead of reusing `finalizing`.
This is the one beat in the pipeline the user waits a noticeable moment for,
and a spinner that does not say why reads as a hang.

Default on; `settings.voiceTypingPolish` turns it off.
@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown

✅ SonarQube Quality Gate passed — pathorsAI_parley

0 open issues on this PR.

4 s was picked as "well short of a crash", which is the wrong thing to measure
against. The thing being protected is not the polish, it is the typing: someone
mid-sentence would rather have their raw words now than their tidy words in a
beat they can feel.

A realtime-lane model answers a dictation-length prompt in well under a second,
so 2 s is still several times the healthy case. Losing a polish to the clock
costs a few filler words; making people wait costs the feature.
`finalize` was at cognitive complexity 18 against a limit of 15 (S3776) — the
polish block's nested `if`s were the difference. Extracted as `polishForPaste`,
which is a better shape anyway: it is total, returning the text to paste rather
than a maybe-polished maybe-null that `finalize` has to unpack, so the caller
is back to one idea per line.

Also S7748: `2.0` → `2` in the accept band.
@YJack0000
YJack0000 merged commit 4d76087 into main Sep 2, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant