From 42af03f1a953b3e742e06834f122d4372f2899e0 Mon Sep 17 00:00:00 2001 From: Enrico Piovesan Date: Fri, 11 Sep 2026 20:14:52 -0600 Subject: [PATCH] =?UTF-8?q?docs(decision-log):=20entry=20111=20=E2=80=94?= =?UTF-8?q?=20audio.transcribe-speech=20VAD=20gate=20(#477)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Sonnet 5 --- docs/decision-log.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/docs/decision-log.md b/docs/decision-log.md index 47224ac4..9194620e 100644 --- a/docs/decision-log.md +++ b/docs/decision-log.md @@ -869,3 +869,16 @@ Deprecate `commerce.cart@1.0.0` / `commerce.pricing-config@1.0.0` (`lines_json` - **Explicitly out of scope**: audio longer than 30s (caller must pre-chunk — this capability does not stream or chunk internally), non-English transcription (language is forced, not detected), beam search / temperature fallback / timestamp prediction, speaker diarization, and translation. This closes out all five models from this session's AI-WASM-model-candidate review: sentence embeddings (MiniLM-L3, #455), voice-activity detection (Silero VAD, #460), named-entity recognition (DistilBERT-NER, #465), language identification (from-scratch trigram classifier, #469), and now speech-to-text (Whisper-tiny, #473). The user's separately-requested "additional downstream capabilities per model" deliverable (a list of further capabilities each of these five engines could power) has not yet been produced. + +111. **`audio.transcribe-speech@1.1.0` published — gates the Whisper forward pass on a Silero VAD speech-presence pre-check (2026-09-11, closes registry#477)**: reuses `audio.detect-speech-segments`' (#460) engine directly rather than duplicating it, fixing a real, observed hallucination-on-non-speech failure mode in v1.0.0. + +- **The highest-ROI item from entry 110's own follow-up review, picked and started same-session on "start with the one that has the best ROI."** Post-#473, a "Five Engines, What Next" downstream-capability note was compiled (per the user's earlier, separately-tracked request) listing ideas for each of the five shipped engines, including "gate Whisper on VAD" as a concrete, zero-new-model pipeline. This is the first of those ideas actually built, chosen over the alternatives specifically because it fixes a real bug already observed in this session (not a speculative enhancement) at comparatively low engineering cost (no new model, no new verification-from-scratch). +- **Real, observed bug, not a hypothetical.** During entry 110's own verification work, a synthetic sine-tone test input came back from `audio.transcribe-speech@1.0.0` as `" [MUSIC PLAYING]"` -- Whisper has no notion of "is this speech at all," and will confidently hallucinate plausible-sounding text for any non-silent input. The existing crude energy-threshold check only catches true silence. +- **Framed as a new capability version, not a workflow -- a genuine scope-boundary decision, not just an implementation-convenience one.** Traverse workflows compose capabilities at the app level; that's out of this registry's own stated scope (`registry-scope-only` -- never claim Traverse-repo or app-repo work). "Transcribe speech, robustly" is still one coherent atomic action from a caller's perspective (same input/output shape, refined internal logic), so this shipped as `audio.transcribe-speech@1.1.0` rather than requiring any workflow-engine capability this repo doesn't control. +- **Reused the existing VAD engine via a lib/bin split, not a duplicated port.** `capability-src/audio-detect-speech-segments` (until now a single `main.rs`) was split into a `lib.rs` (engine: `Model`, `score_chunk`, constants, plus an always-available `fixture_model()` test helper) and a thin `main.rs` `_start` shim. `audio.detect-speech-segments@1.0.0`'s published contract/artifact is untouched -- this is a source-only refactor of the crate backing it, verified by re-running all 15 of that crate's pre-existing tests unchanged before touching anything else. +- **A real technique worth naming: exposing a fixture-builder across a dependency boundary.** `audio-transcribe-speech`'s own tests need a structurally-valid-but-synthetic VAD model to exercise the new gating plumbing without embedding the real ~1.2 MB weight table. The VAD crate's `fixture_bin()`/`varied()` helpers were originally private inside its own `#[cfg(test)] mod tests` (invisible to a dependent crate's *own* test build, since `cfg(test)` is evaluated per-crate, not transitively) -- moved to the crate root as ordinary (always-compiled) private functions, with a new `pub fn fixture_model()` wrapping them, so a dependent crate's tests get a working model without any real weights. +- **Verified empirically on real audio, all three cases, via the compiled artifact under wasmtime -- not source-level tests alone.** The real 11s JFK clip and its 1s excerpt transcribe identically to v1.0.0 (VAD correctly recognizes real speech on every chunk, so the gate never falsely blocks real speech); the synthetic-tone case now correctly returns empty text; true silence still returns empty via the unchanged energy check. Threshold: max VAD probability across all 512-sample chunks in the clip must clear 0.5 (Silero's own common convention) -- a presence check, not a coverage measurement, so one brief speech burst in an otherwise-quiet clip still transcribes. +- **Determinism — `model_derived`, unchanged from 1.0.0.** Still float attention/FFN/conv math (Whisper) plus float conv/LSTM math (VAD); no x86_64 host/emulator available for a cross-arch proof. +- **Catalog verified live**: loaded `https://registry.traverse-framework.com/#/capability/audio%2Faudio.transcribe-speech%401.1.0` post-merge and confirmed the v1.1.0 badge, updated summary/description (mentioning the VAD gate and the `" [MUSIC PLAYING]"` fix by name), and the new tone-hallucination use case render correctly. +- **Publish mechanics**: PR #478 (contract + crate refactor + gating logic + FR-020 inventory update), signature backfill #479. No new GitHub Release model asset needed for VAD's own weight table (unchanged, already published under `audio.detect-speech-segments-1.0.0`'s release) -- only a new wasm artifact release tag (`audio.transcribe-speech-1.1.0`) for the larger (~41.2 MB, +~1.2 MB for VAD) compiled binary. `cargo llvm-cov`: functions=100%, lines=98.45%, regions=98.64%. Artifact digest `sha256:cdddb147…2bc77f4`. +- **Explicitly out of scope**: everything already out of scope for 1.0.0 (audio >30s, non-English transcription, beam search, diarization, translation), plus: the VAD gate is a presence check only, not a "how much of this clip is speech" measurement, and does not affect what gets transcribed once the gate passes (no partial-clip trimming to just the speech segments).