Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions docs/decision-log.md
Original file line number Diff line number Diff line change
Expand Up @@ -869,3 +869,16 @@ Deprecate `commerce.cart@1.0.0` / `commerce.pricing-config@1.0.0` (`lines_json`
- **Explicitly out of scope**: audio longer than 30s (caller must pre-chunk — this capability does not stream or chunk internally), non-English transcription (language is forced, not detected), beam search / temperature fallback / timestamp prediction, speaker diarization, and translation.

This closes out all five models from this session's AI-WASM-model-candidate review: sentence embeddings (MiniLM-L3, #455), voice-activity detection (Silero VAD, #460), named-entity recognition (DistilBERT-NER, #465), language identification (from-scratch trigram classifier, #469), and now speech-to-text (Whisper-tiny, #473). The user's separately-requested "additional downstream capabilities per model" deliverable (a list of further capabilities each of these five engines could power) has not yet been produced.

111. **`audio.transcribe-speech@1.1.0` published — gates the Whisper forward pass on a Silero VAD speech-presence pre-check (2026-09-11, closes registry#477)**: reuses `audio.detect-speech-segments`' (#460) engine directly rather than duplicating it, fixing a real, observed hallucination-on-non-speech failure mode in v1.0.0.

- **The highest-ROI item from entry 110's own follow-up review, picked and started same-session on "start with the one that has the best ROI."** Post-#473, a "Five Engines, What Next" downstream-capability note was compiled (per the user's earlier, separately-tracked request) listing ideas for each of the five shipped engines, including "gate Whisper on VAD" as a concrete, zero-new-model pipeline. This is the first of those ideas actually built, chosen over the alternatives specifically because it fixes a real bug already observed in this session (not a speculative enhancement) at comparatively low engineering cost (no new model, no new verification-from-scratch).
- **Real, observed bug, not a hypothetical.** During entry 110's own verification work, a synthetic sine-tone test input came back from `audio.transcribe-speech@1.0.0` as `" [MUSIC PLAYING]"` -- Whisper has no notion of "is this speech at all," and will confidently hallucinate plausible-sounding text for any non-silent input. The existing crude energy-threshold check only catches true silence.
- **Framed as a new capability version, not a workflow -- a genuine scope-boundary decision, not just an implementation-convenience one.** Traverse workflows compose capabilities at the app level; that's out of this registry's own stated scope (`registry-scope-only` -- never claim Traverse-repo or app-repo work). "Transcribe speech, robustly" is still one coherent atomic action from a caller's perspective (same input/output shape, refined internal logic), so this shipped as `audio.transcribe-speech@1.1.0` rather than requiring any workflow-engine capability this repo doesn't control.
- **Reused the existing VAD engine via a lib/bin split, not a duplicated port.** `capability-src/audio-detect-speech-segments` (until now a single `main.rs`) was split into a `lib.rs` (engine: `Model`, `score_chunk`, constants, plus an always-available `fixture_model()` test helper) and a thin `main.rs` `_start` shim. `audio.detect-speech-segments@1.0.0`'s published contract/artifact is untouched -- this is a source-only refactor of the crate backing it, verified by re-running all 15 of that crate's pre-existing tests unchanged before touching anything else.
- **A real technique worth naming: exposing a fixture-builder across a dependency boundary.** `audio-transcribe-speech`'s own tests need a structurally-valid-but-synthetic VAD model to exercise the new gating plumbing without embedding the real ~1.2 MB weight table. The VAD crate's `fixture_bin()`/`varied()` helpers were originally private inside its own `#[cfg(test)] mod tests` (invisible to a dependent crate's *own* test build, since `cfg(test)` is evaluated per-crate, not transitively) -- moved to the crate root as ordinary (always-compiled) private functions, with a new `pub fn fixture_model()` wrapping them, so a dependent crate's tests get a working model without any real weights.
- **Verified empirically on real audio, all three cases, via the compiled artifact under wasmtime -- not source-level tests alone.** The real 11s JFK clip and its 1s excerpt transcribe identically to v1.0.0 (VAD correctly recognizes real speech on every chunk, so the gate never falsely blocks real speech); the synthetic-tone case now correctly returns empty text; true silence still returns empty via the unchanged energy check. Threshold: max VAD probability across all 512-sample chunks in the clip must clear 0.5 (Silero's own common convention) -- a presence check, not a coverage measurement, so one brief speech burst in an otherwise-quiet clip still transcribes.
- **Determinism — `model_derived`, unchanged from 1.0.0.** Still float attention/FFN/conv math (Whisper) plus float conv/LSTM math (VAD); no x86_64 host/emulator available for a cross-arch proof.
- **Catalog verified live**: loaded `https://registry.traverse-framework.com/#/capability/audio%2Faudio.transcribe-speech%401.1.0` post-merge and confirmed the v1.1.0 badge, updated summary/description (mentioning the VAD gate and the `" [MUSIC PLAYING]"` fix by name), and the new tone-hallucination use case render correctly.
- **Publish mechanics**: PR #478 (contract + crate refactor + gating logic + FR-020 inventory update), signature backfill #479. No new GitHub Release model asset needed for VAD's own weight table (unchanged, already published under `audio.detect-speech-segments-1.0.0`'s release) -- only a new wasm artifact release tag (`audio.transcribe-speech-1.1.0`) for the larger (~41.2 MB, +~1.2 MB for VAD) compiled binary. `cargo llvm-cov`: functions=100%, lines=98.45%, regions=98.64%. Artifact digest `sha256:cdddb147…2bc77f4`.
- **Explicitly out of scope**: everything already out of scope for 1.0.0 (audio >30s, non-English transcription, beam search, diarization, translation), plus: the VAD gate is a presence check only, not a "how much of this clip is speech" measurement, and does not affect what gets transcribed once the gate passes (no partial-clip trimming to just the speech segments).
Loading