Add Granite Speech 5 CTC support - #712
Open
justinchuby wants to merge 2 commits into
Open
Conversation
Implement the native chunked Shaw-relative Conformer encoder, in-block temporal subsampling, self-conditioned CTC head, processor-accurate feature contracts, and truthful runtime metadata for granite_speech5_ctc. Add pinned L1-L5 coverage, real-weight CPU/CUDA and multi-dtype parity, exact frame/transcript goldens, CLI packaging checks, and public audio fixture provenance. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Performance Comparison
|
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
Format the two newly added Granite Speech files with the pinned Ruff version and remove the unused import that escaped the earlier diff-scoped lint pass while the files were untracked. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
There are a few correctness/documentation issues to fix (notably the Transformers from_pretrained dtype keyword and the FeatureCTC task docstring lagging behind the new frame_lengths output).
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Adds first-class/native support for IBM Granite Speech 5 Turbo CTC (granite_speech5_ctc) to mobius’ model registry/task stack, including ORT GenAI export metadata, synthetic/integration parity tests, and committed L4/L5 goldens to validate end-to-end decoding.
Changes:
- Implement
GraniteSpeech5ForCTCModel+GraniteSpeech5CTCConfig, registergranite_speech5_ctc, and add tiny-config/unit tests. - Extend
FeatureCTCAsrTaskto support processor-accurate feature/mask dtypes and (optionally) emitframe_lengths; update ORT GenAI export to handle feature-CTC packages and emitaudio_processor.jsonfor Granite Speech 5. - Add L3 synthetic parity, real-checkpoint integration tests, and L4/L5 golden case files for Granite Speech 5 Turbo CTC.
File summaries
| File | Description |
|---|---|
| tests/synthetic_parity_test.py | Adds L3 synthetic parity coverage for Granite Speech 5 CTC vs native Transformers. |
| tests/granite_speech5_integration_test.py | Adds real-checkpoint CUDA parity tests and transcript validation across dtypes. |
| tests/e2e_golden_test.py | Adjusts feature-CTC feeds (int64 mask for Granite) and makes exact-match token logging/decoding robust to non-ndarray token containers. |
| tests/_test_configs.py | Adds a representative tiny Granite Speech 5 CTC config entry for graph-build test coverage. |
| testdata/golden/audio/granite-speech-5.0-470m-turboctc.json | Adds L4 golden logits snapshot for the new Granite Speech 5 CTC case. |
| testdata/golden/audio/granite-speech-5.0-470m-turboctc_generation.json | Adds L5 golden generation output (token IDs + transcript) for e2e validation. |
| testdata/cases/audio/granite-speech-5.0-470m-turboctc.yaml | Adds a pinned revision L4+L5 golden test case definition for Granite Speech 5 CTC. |
| src/mobius/tasks/_ctc_asr.py | Extends FeatureCTC ASR task to support configurable input dtypes/shapes and optional frame_lengths output. |
| src/mobius/models/granite_speech5.py | Introduces the Granite Speech 5 encoder + CTC head implementation (chunked Shaw Conformer + self-conditioned CTC). |
| src/mobius/models/granite_speech5_test.py | Adds config-extraction tests, graph contract/weight-name tests, and ORT GenAI export metadata tests for the new model. |
| src/mobius/models/init.py | Exports GraniteSpeech5ForCTCModel from the models package. |
| src/mobius/integrations/ort_genai/auto_export.py | Emits Granite Speech 5 audio_processor.json and improves feature-CTC output introspection + runtime capability warnings. |
| src/mobius/_registry.py | Registers granite_speech5_ctc with task/config/test metadata. |
| src/mobius/_configs/_base.py | Adds GraniteSpeech5CTCConfig with validation + Transformers config extraction. |
| src/mobius/_configs/init.py | Exposes GraniteSpeech5CTCConfig via the configs public API. |
| scripts/generate_golden.py | Generates CTC goldens using the case dtype and ensures last-frame logits are extracted in fp32. |
| README.md | Lists Granite Speech 5 (and Parakeet CTC) under Speech-to-Text supported models. |
| docs/model-catalog.md | Documents the new granite_speech5_ctc model type, module class, and task. |
| docs/_generate_models.py | Updates Speech-to-Text category description to include CTC models. |
Review details
Suppressed comments (1)
src/mobius/tasks/_ctc_asr.py:86
FeatureCTCAsrTask.build()now conditionally emits aframe_lengthsoutput when the module providesframe_lengths(), but the task docstring still documents onlylogits. Please documentframe_lengthsso downstream callers know it may be present.
Output:
``logits`` — (batch, subsampled_frames, vocab_size) CTC scores
"""
- Files reviewed: 19/19 changed files
- Comments generated: 2
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+57
to
+61
| model = transformers.AutoModelForCTC.from_pretrained( | ||
| _MODEL_ID, | ||
| revision=_REVISION, | ||
| dtype=dtype, | ||
| ).eval() |
🏗️ Architecture Diff
No architecture changes detected. ✅ Legend: ⚪ No change · 🔵 Minor (attrs/inits) · 🟡 Moderate (nodes added/removed) · 🔴 Major (interface changed) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
granite_speech5_ctcsupport foribm-granite/granite-speech-5.0-470m-turboctc7e74c6438b7cfb5090cb6a131538f5e8515a7de3[B, T, 320]features, int64 valid-frame masks, CTC logits, and frame lengthsArchitecture
Implements the native Transformers architecture merged in
huggingface/transformers@da7234ac435f6d7c75d8b88d1ac32f53fb1f19a6: 16 pre-norm Macaron Conformer blocks; 128-frame chunk-local Shaw relative attention; Linear+GLU/depthwise-convolution modules; stride-2 residual/conv subsampling in blocks 0 and 1; and mid-encoder self-conditioned CTC. The final CTC projection reusesencoder.out, matching the tied HFctc_head. Frozen convolution normalization and pairwise pooling use fp32 accumulation so the checkpoint's native bf16 route executes on ORT CUDA.The native feature extractor contract is preserved: 16 kHz float32 mono audio, 80-bin log-mel, first deltas, adjacent-frame stacking, 320-wide features, and int64
1=validmasks. Runtime metadata contains only real graph ports and explicitly records that onnxruntime-genai 0.15.2 does not orchestrate feature-input CTC.Evidence
14 passedacross the dedicated model tests,build_graph_test.py,weight_alignment_test.py, and targeted L3 collection; ONNX checker and shape/type assertions pass.3 passedfor pinned config download, parse, and full-size graph build.atol=rtol=1e-5.3e-3; fp16 has max error0.04321289with only 2/1,867,776 logits outside0.03; bf16 has max0.421875, mean0.06981443, cosine>0.9999. fp16 and bf16 both match every one of 114 frame IDs and the exact transcript.1 passedagainst the committed pinned-HF bf16 top-k golden.1 passed; exact 114-frame sequence and transcript:cauliflower mayonnaise take cold boiled cauliflower break into branches adding salt pepper and vinegar to season.model.onnx+ 948,109,312-byte external data, tokenizer assets, exactaudio_processor.json, graph-truthfulgenai_config.json, and runtime compatibility metadata; the persisted package reproduces all 114 golden IDs on CUDA.6 passed. Olive 0.13.0 MatMul-only dynamic INT8 produced a 484,924,093-byte model with 164MatMulIntegerprojections and 16 unchanged standard Attention ops; CPU inference preserves all 114 frames and the exact transcript. An all-linear NF4 attempt was rejected because it degenerated to blanks.lintrunner -areportsok No lint issues; final specialist review reports no defects.Narrow waivers / environment notes
lintrunner f --output oneline --all-filesexecuted its patch phase but its Ruff check process hits WindowsWinError 206after expanding the all-files response list; repeating from a short drive mapping has the same adapter limitation. The required exactlintrunner -athen ran the pinned Ruff tools on the complete diff and is clean.