Add staged VibeVoice text-to-speech support - #714
Open
justinchuby wants to merge 2 commits into
Open
Conversation
Implement the Transformers-native VibeVoice pipeline as explicit ONNX reference-audio, embedding, Qwen2 decoder, diffusion, acoustic decoder, and semantic feedback stages with mask-safe KV and convolution state contracts. Pin the converted checkpoint and runtime assets, and add L1-L5, real-weight CUDA, CPU/CUDA, processor, provenance, and quantization evidence. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Performance Comparison
|
Apply the repository-pinned Ruff formatter and import fixes across every Python file changed by the VibeVoice implementation so the repository lint job sees the same clean result as local validation. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The new text-to-speech golden coverage is skipped by the existing e2e golden harness, and current CI golden workflows invoke only tests/e2e_golden_test.py, so VibeVoice L4/L5 goldens risk not running in the main gates unless CI/harness dispatch is updated.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Adds first-class, staged support for the Transformers-native VibeVoice 1.5B text-to-speech model to Mobius, exporting a multi-graph ONNX package plus test fixtures/contracts to validate the full continuous-token TTS pipeline end-to-end.
Changes:
- Introduces
VibeVoiceForConditionalGeneration+VibeVoiceTaskthat export 8 explicit ONNX stages (audio encoder/projection, embedding, decoder, diffusion head, audio decoder, semantic encoder/projection), including explicit KV + convolution streaming state. - Extends optimization/export plumbing to support “arbitrary attention mask required” decoders (disables GQA fusion) and to pin/propagate the effective source revision into runtime assets and onnx-genai exports.
- Adds L1/L2/L3/L4/L5 coverage artifacts: tiny-config graph-build tests, synthetic parity harness integration, new golden cases/golden outputs, and a dedicated real-weight golden runner for VibeVoice.
File summaries
| File | Description |
|---|---|
| tests/weight_alignment_test.py | Adds VibeVoice weight-routing coverage to ensure every stage parameter receives a routed HF key. |
| tests/vibevoice_golden_test.py | New dedicated L4/L5 real-weight + generation golden tests for continuous-token TTS outputs (control logits/latents/waveform). |
| tests/synthetic_parity_test.py | Hooks the dedicated VibeVoice synthetic stage-parity runner into the L3 synthetic parity suite. |
| tests/e2e_golden_test.py | Skips text-to-speech cases in the generic L4 prefill argmax golden harness. |
| tests/cli_test.py | Tests that onnx-genai runtime asset export uses the effective pinned revision from package metadata. |
| tests/build_graph_test.py | Adds VibeVoice stage build/IO/state ABI assertions, registry pin checks, and GQA-disabled attention checks. |
| tests/_test_configs.py | Adds a tiny VibeVoice config entry for shared build/test infrastructure. |
| testdata/golden/audio/vibevoice-1.5b-text-only.json | Adds L4 golden reference for text-only case. |
| testdata/golden/audio/vibevoice-1.5b-text-only_generation.json | Adds L5 golden reference for text-only generation case. |
| testdata/golden/audio/vibevoice-1.5b-reference.json | Adds L4 golden reference for reference-audio case. |
| testdata/golden/audio/vibevoice-1.5b-reference_generation.json | Adds L5 golden reference for reference-audio generation case. |
| testdata/golden/audio/vibevoice-1.5b-multispeaker.json | Adds L4 golden reference for two-speaker ordering contract. |
| testdata/cases/schema.json | Extends YAML schema to include text-to-speech and TTS generation knobs (num_diffusion_steps, guidance_scale). |
| testdata/cases/audio/vibevoice-1.5b-text-only.yaml | Adds a text-only VibeVoice golden case definition. |
| testdata/cases/audio/vibevoice-1.5b-reference.yaml | Adds a reference-audio VibeVoice golden case definition (with provenance notes). |
| testdata/cases/audio/vibevoice-1.5b-multispeaker.yaml | Adds a two-speaker VibeVoice case definition. |
| testdata/audio_fixture_provenance.json | Adds provenance record for the 24kHz LibriSpeech-derived fixture. |
| src/mobius/tasks/_vibevoice.py | New task that builds the 8-stage VibeVoice ModelPackage and declares stage contracts/state. |
| src/mobius/tasks/init.py | Exposes VibeVoiceTask and registers vibevoice-tts in TASK_REGISTRY. |
| src/mobius/models/vibevoice.py | New VibeVoice staged model implementation + weight routing from native HF checkpoint namespaces. |
| src/mobius/models/vibevoice_test.py | New synthetic numerical parity tests for all 8 stages and contracts (including empty-audio embedding). |
| src/mobius/models/init.py | Exports VibeVoiceForConditionalGeneration. |
| src/mobius/integrations/transformers/_builder.py | Pins VibeVoice revision when unspecified and stamps mobius.source_revision metadata for runtime asset resolution. |
| src/mobius/integrations/transformers/_builder_test.py | Tests that the first config probe is revision-pinned for VibeVoice when revision is omitted. |
| src/mobius/integrations/onnx_genai/auto_export.py | Detects VibeVoice topology and exports runtime assets + an explicit “unsupported by runtime” advisory contract. |
| src/mobius/integrations/onnx_genai/auto_export_test.py | Tests that revision is forwarded through runtime asset export paths for VibeVoice. |
| src/mobius/_testing/vibevoice_generation.py | New deterministic disk-backed staged runner used by VibeVoice goldens (diffusion loop + dual-cache orchestration). |
| src/mobius/_registry.py | Registers the vibevoice model_type with pinned test model id/revision and VibeVoice config/module class. |
| src/mobius/_pipeline_contract.py | Adds a graph-level contract flag to declare/require arbitrary attention masks (used to gate optimizations). |
| src/mobius/_optimizations.py | Disables GQA fusion when a decoder declares arbitrary attention masking is required. |
| src/mobius/_configs/vibevoice.py | New config extraction for VibeVoice composite configs (tokenizers + diffusion head) from Transformers. |
| src/mobius/_configs/init.py | Exposes the new VibeVoice config types. |
| src/mobius/main.py | Derives an effective build revision from package metadata for runtime asset downloads when CLI revision is omitted. |
| docs/model-catalog.md | Adds VibeVoice to the model catalog table. |
Review details
- Files reviewed: 34/35 changed files
- Comments generated: 1
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+2200
to
+2204
| if case.task_type == "text-to-speech": | ||
| pytest.skip( | ||
| "Continuous-token TTS uses tests/vibevoice_golden_test.py " | ||
| "(control logits, latent frames, and waveform semantics)." | ||
| ) |
🏗️ Architecture Diff
No architecture changes detected. ✅ Legend: ⚪ No change · 🔵 Minor (attrs/inits) · 🟡 Moderate (nodes added/removed) · 🔴 Major (interface changed) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
vibevoicesupport pinned tovibevoice/VibeVoice-1.5B-hf@edc39f80f5cae656da37baf8faa8f5502bf7081f; recordmicrosoft/VibeVoice-1.5B@c00898d257e6b46004e3e2866a47534085fb685aas the provenance root, not the executable sourceEvidence
tests/build_graph_test.py -k vibevoice— 8 passedtests/weight_alignment_test.py -k vibevoice— 2 passed; all 1,204 native checkpoint keys route to trained stage parameterstests/arch_validation_test.py -k vibevoice— 3 passed against the full pinned 1.5B configtests/synthetic_parity_test.py -k vibevoice— 1 passed; every stage matches native HF, including non-identity latent scale/bias and ONNX decoder→semantic feedbacktests/vibevoice_golden_test.py -m "golden or integration"on CUDA — 8 passed[151654, 151654, 151654][1, 3, 64][1, 1, 9600](3 × 3,200 samples)(1,38), reference(1,116)with 69 audio placeholders, and two-speaker(1,203)with 138 ordered placeholdersaudio_processor.json,processor_config.json,generation_config.json, tokenizer, chat template, revision metadata, and advisory runtime contractMatMulNBits; CPU/CUDA max difference0.001953125, correlation to FP160.9999083, finite non-degenerate outputlintrunner -a— clean; specialist review rerun after fixes — no findingsWaivers
ConvTransposenodes. BF16 reference-audio encoder parity was still measured (corr=0.9999813), while full CUDA L4/L5 and CPU/CUDA parity use FP16. No silent dtype downgrade is performed.onnx-genaidoes not yet orchestrate VibeVoice's dual decoder caches, DPM-Solver loop, and streaming convolution state. The package emits exact graph/processor contracts and an explicitunsupported-by-tested-runtimeadvisory rather than claiming downstream support.lintrunner f --output oneline --all-filesapplied formatting but the Ruff subprocess hit WindowsWinError 206from the repository-wide path list in this deep worktree. The required exact changed-filelintrunner -acompleted cleanly after pinned linter initialization.