transcribe backend: adapter contract, result schema, stack table, planner, orchestration

[fyang0507]

Context

TRANSCRIBE_DESIGN_HANDOFF.md opens the design work as seven
items. This issue takes the backend half — handoff open items 1 (module and adapter
boundaries), 2 (stage orchestration), 4 (partial results and resume), 6 (word and
segment identity), and 7 (where the stack table lives).
#11 takes provisioning and the
environment layout; the cue segmenter stays parked in
#10.

The agent-facing surface is settled and signed off. VOCABULARY.md is the
naming contract, TRANSCRIBE_CONTRACT.md is the specification,
TRANSCRIBE_HAPPY_PATH.md is the diff target, and
tests/test_spec_docs.py already enforces the documents' internal consistency — it is also a
worked list of the invariants this implementation has to satisfy. Nothing underneath is built:
src/audio_cli/ has no abstraction for a backend, and vad.py is the only backend-shaped
file in it, written before the vocabulary existed.

Design questions this issue answers

1. Where the stack table lives. Capability availability, cost, and notes are prose in two
   documents today. They have to become data, and that data is what capabilities
   serialises. VOCABULARY §Resolution's derivation table has exactly four legal cell forms —
   native, native (<stage>), + <package>, exit 2: <code> — and a test parametrized
   over it must accept those four and keep the two refusal codes distinct. The handoff calls
   this the first real design decision.
2. The normalized result schema and the adapter contract. One hard constraint: no
   model-specific object travels past the adapter. Everything else is open. Build the
   anti-fabrication assertions with the schema, not after — this is where fabrication becomes
   possible.
3. The planner. Ideally a pure function from (stack, input metadata, requirements) to a
   plan, with capabilities and plan as two serialisations of it. sample_output is
   generated by running a placeholder result through the same serializer run uses, never
   hand-written.
4. Stage orchestration. Strictly sequential, one model resident at a time. Enforcing that
   across environments needs the transport environments and packages: provisioning layout, locks, registry, and audio packages #11 defines; the ordering, the residency
   guarantee, and the observed-cost accounting are here. Recorded figures came from fresh
   subprocesses — in-process loading is outside what was measured.
5. Partial results and resume. Exit 4 writes a conforming result with a coverage ledger:
   covered_through_seconds, plus explicit covered and missing intervals. The recorded Qwen
   runner processes turns in duration-bucketed order, so completion is genuinely
   non-contiguous and the watermark is not "the last thing that finished". --range resumes
   against the original input; the canonical-timeline floor is why it is a range and not a
   pre-clipped file.
6. Word and segment identity. Document-scoped, explicitly not stable across runs. If the
   Observation Store later needs run-stable identity it cannot key on a timestamp — FireRed
   reproduces timestamps only to 1 ms.

Suggested order

From the handoff, risk earliest: schema and adapter contract → stack table and planner →
qwen-1.7b behind an adapter (most recorded evidence, cheapest runs, no native anything, so
it forces the derived paths first) → firered (exercises vad, punctuator, lid, and the
punctuation invariant) → vibevoice last (slowest to iterate, least recoverable) → export.
Nothing here can be exercised end to end before #11's pull works.

Risks that shape the design

• vibevoice cannot resume. Whole media, one generate call, so a failure at minute 40
  of 41 yields nothing. It is also the most likely to fail: 20.28 GiB live MPS on thirty
  minutes, OOM under a strict 16 GiB cap. failure_recovery.partial_results is none there.
  Do not design a recovery story that quietly assumes partitioning.
• A long Qwen run truncates silently today. The recorded runner carries a global
  generation budget and stops between turns when it runs out; at the recorded token rate it
  exhausts near 1.6 hours of comparable audio. Every recorded run finished, so this has
  never been observed — exit 4 exists for it. Make the budget declared configuration, not a
  constant nobody notices.
• Two Qwen decode paths disagree. Public generate() and private
  _generate_chunks_batched agree on every word and differ by two Chinese commas. The
  private path is the one that batches turns and the one the timing figures came from, and it
  returns the model's language English<asr_text> scaffold inside its text, which the adapter
  must strip. api_path is in the plan for this reason. Lexical evidence transfers between
  the paths; punctuation does not, and punctuation is what cue splitting breaks on.
• The mlx-audio private API is pinned by source hash — 0.4.5, sha256
  c082690575eedcd28fb76207d032cefd7eac2f9ce5d36df5a7a06575bc45d250. An upgrade is a
  breaking change, not a bump.
• 16 GiB is not validated. RSS, MLX, PyTorch, and Core ML counters have different scopes
  and must not be summed. Ship the product-demo route and warn from the plan; do not turn
  that into a claim.

Unmeasured — design around the gap

Boundary MAE/P95 for FireRed's native word times and for the forced aligner (so "switch
stacks for better timing" is unsupported in either direction); filler recall on any stack;
cross-process determinism for the Qwen batched path; package byte sizes beyond the two Qwen
checkpoints and the Core ML model; licenses beyond FluidAudio's SDK and
speaker-diarization-coreml; any accuracy figure outside Cantonese. Where a decision depends
on one of these, say so rather than assuming a value.

Acceptance

Against real command output, not mocks:

• Sample-output key set: a plan's sample_output key set equals a real run's, and no key
  exists for a capability that was not requested. Parametrized over VOCABULARY's derivation
  table, including both refusal codes.
• Punctuation invariant per stack: a sentence's text, stripped of punctuation and
  whitespace, equals the concatenation of its word texts, compared case-insensitively
  (FireRedPunc recases; 234 characters differ across the recorded artifacts). Skip segments
  with no word stream rather than asserting they have one — the forced-aligner artifact has
  two, both VibeVoice non-speech event tags.
• Adapter normalization per stack, one case each: Qwen's language <label><asr_text>
  scaffold is stripped; FireRed's lang: null, lang_confidence: 0 are dropped when LID did
  not run; VibeVoice's Speaker: "N/A" becomes an absent key, never a speaker id.
• A plan carries no outcomes; a run's provenance does.
• Exit codes end to end, including 3 before pull, 4 on a truncated run, and that a partial
  result is a conforming document rather than a debug dump.
• uv run --extra dev pytest stays green; the deterministic render-from-original enhance
  flow is not regressed.

The seam with #11

The interface is (stack, requirements) → [package]. This issue produces it — capability
availability, satisfaction, evidence, notes, and the planner that reads them.
#11 consumes it and owns the
package/environment columns: package id, environment, kind, revision, digest, bytes, license,
requires_tool. One table, two owners. If #11 ships a stubbed version of this function, it
must be replaced rather than paralleled.

Note #11's open experiment: whether vibevoice-asr-7b, qwen3-forcedaligner, and
firered-asr2s share one torch environment is unverified, and the as-built evidence
suggests they may not. If torch splits, the number of environments a request spans changes,
and so does what a plan reports.

Traps

• A claim about what a backend produces must cite a runner source or a recorded artifact,
  never a summary document. Three research documents said FireRed emits per-word
  confidence. It does not — every word object is exactly {start_ms, end_ms, text} across
  12,370 recorded tokens — and the claim survived four review passes because every summary
  repeated it. Artifacts: model_tests/benchmark_runs/ (untracked, local),
  model_tests/benchmark/results/ (tracked), runners in model_tests/benchmark/.
• Do not add structure nobody dispatches on. The capabilities report went through an
  evidence object, a seven-key cost object, a timing_precision object holding one null
  and one string, and four sibling free-text fields before all of it collapsed into one
  sentence per capability. Every one seemed justified when added. The test is whether a caller
  branches on it.
• Do not describe machinery that does not exist. The punctuation floor spent four review
  passes instructing the adapter to reattach marks and drop their bounds — for a stage that
  emits no per-mark bounds at all. The rule was inert and its test passed vacuously. When you
  write an invariant, write the assertion that would fail if it were violated, and check that
  it can fail.

[fyang0507]

Three things from #11 that change this issue's inputs

#11 is done except for its seam with this one (2eaa429, 6603298).

1. The aligner is in mlx, not a PyTorch environment. mlx-audio 0.4.5 — already pinned
for the Qwen stacks — implements the forced aligner, and rerunning the recorded case gave
identical token text on all 246 aligned tokens with zero punctuation-invariant violations. So
word_timestamps on a Qwen stack adds a package, not a runtime, and the fast long-form path
spans mlx + swift. Half the plans in the spec documents printed "environment": "torch" for
the aligner and now print "mlx"; the documents are updated and a test fails if a payload names
an environment the manifest does not define.

2. The transport is settled, and it makes your adapter floor structural. One fresh
subprocess per stage against the environment's own interpreter, JSON file in and JSON file out,
progress on stderr, residency enforced by process exit (ENVIRONMENTS.md).
audio_cli is deliberately not installed into provisioned environments. The consequence for
this issue: a model-specific object cannot cross a process boundary, so the stage script has to
serialize normalized output — "no model object travels past the adapter" is satisfied by
construction rather than by review. What remains yours is the orchestration above it: stage
order, observed-cost accounting, and the coverage ledger.

3. The package half of the stack table exists. src/audio_cli/environments/manifest.json
holds package id, environment, kind, sources with pinned revisions, real byte counts, declared
licenses, roles, and stacks, behind a reader in src/audio_cli/environments/__init__.py.

The seam is packages_for(stack, roles), which takes resolved roles rather than capability
names — your planner owns the derivation, that module owns the mapping to packages. It is
currently stubbed only in the sense that audio packages pull --stack S over-selects (every
package a stack can use) because the real selection needs your planner. Replace it; do not
parallel it. --want without --stack is already refused rather than ignored.

Two smaller inputs that affect payloads you will emit:

• license is two fields now, license_declared and license_reviewed. A model card is not a
  redistribution clearance, and one field could not tell "nobody looked" from "the card says
  apache-2.0 and nobody checked what it obliges". unreviewed as a single value is retired.
• Every package has a real byte count read from the Hub at its pinned revision, so
  total_known_download_bytes is now answerable and unsized_packages has exactly one member,
  fluidaudio. Several figures the spec documents quoted were wrong rather than illustrative.

Unchanged for you: vibevoice still runs on PyTorch. The MLX path is faster and needs 12.4 GB
against 20.8 GB, but it produces a different transcript — 47 segments against 49, some
traditional-character orthography, one lexical difference — so adopting it is a re-measurement
decision rather than a lock change. If it is ever adopted, the adapter cases for that stack
change with it.

[fyang0507]

Decomposed into seven phase issues

TRANSCRIBE_IMPLEMENTATION_PLAN.md answers the six design
questions above and sequences the work. This issue's scope now lives in seven issues, each
self-contained with its own acceptance criteria:

Issue	Phase	Answers, from the list above
#18	0 — correct the diff target	none; it fixes what the evidence pass found wrong in the documents everything else is diffed against
#19	A — result schema, serializer, generated sample	2, 6
#20	B — stack table, planner, every refusal	1, 3
#21	C — transport, orchestration, Qwen end to end	4, 5
#22	D — firered behind an adapter	2, applied to the hardest adapter
#23	E — vibevoice, and the only truncation story	5, on the stack that cannot resume
#24	F — export: formats, merge, hard-coded cue policy	consumes #10

Six things in this issue's framing that the artifacts changed

Verifying each claim against a runner source, an installed source at its pinned commit, or a
recorded artifact — rather than against a summary — moved four of the six questions above.

1. "Strictly sequential, one model resident at a time" is not achievable on firered without
   abandoning the measured configuration. run_firered.py:395-426 loads VAD, LID, ASR and Punc,
   assembles them onto one system object (:435), then calls system.process once (:463).
   Measured on the same 139.284 s clip: 9.16 GiB peak RSS with LID off, 12.26 GiB with LID
   on — the LID checkpoint adds to the peak rather than replacing another stage's. Decision:
   residency is declared at environment granularity, and HAPPY_PATH §3.1's 6.43 GiB illustrative
   peak — below the measured LID-off floor, because it was built as a maximum of isolated per-stage
   peaks — is replaced by the measured pair (transcribe phase 0: correct the diff target before implementing against it #18).
2. The stack table becomes two files, not one. stacks.json beside manifest.json, with a
   cross-check test, because environments/__init__.py states in its own docstring that the
   capability half is not there. Two owners of one table rather than two tables (transcribe phase B: the stack table, the planner, and every refusal #20).
3. The adapter boundary sits in core, not in the stage script. Stages emit the backend's own
   JSON minus anything unbounded; normalization happens in the tool. That is what lets every
   anti-fabrication assertion run against the recorded artifacts with no provisioned environment,
   which is the difference between checking the schema's guarantees in the normal suite and only in
   a live run (transcribe phase C: transport, orchestration, and the Qwen stacks end to end #21).
4. Exit 4's resume is sound because the diarizer is reproducible. Three separate runs of the
   30-minute fixture, two of them weeks apart, produced identical normalized_output_sha256
   (3b6639…, 589 segments) — but the same 30 minutes in two positions of a 60-minute concat
   produced 588 against 549 intervals. So --range re-diarizes the whole file, and then the
   turn partition and the anonymous labels reproduce exactly and two results merge. Diarizing only
   the range would not (transcribe phase C: transport, orchestration, and the Qwen stacks end to end #21).
5. The punctuation invariant is the word-partition algorithm, not only a check. FireRed emits
   one flat words[] plus separate sentences[] and nothing maps between them; timestamps cannot
   do it, since word 0 starts at 650 ms inside a sentence starting at 590 ms. Greedy consumption
   against each sentence's punctuation-stripped, case-folded text matches 1,896 of 1,896
   sentences and consumes 12,370 of 12,370 words with zero failures across all five recorded
   artifacts (transcribe phase D: firered behind an adapter #22).
6. packages_for() cannot resolve firered today. It keys roles.values() on package ids
   (environments/__init__.py:141), and firered-vad, firered-lid, firered-asr2-aed and
   firered-punc are four backends of the single firered-asr2s package, so the call raises
   ManifestError. The manifest gains a backend→(package, role) map (transcribe phase 0: correct the diff target before implementing against it #18).

Two further findings shape #21 and #23 rather than this framing: Qwen's
language English<asr_text> scaffold tracks the language argument and not the API path — leaked
at language=None, absent in 0 of 195 segments with a hint — so a strip test written against
the hinted configuration passes vacuously; and a truncated VibeVoice decode returns [] from
post_process_transcription rather than a prefix, so writing one on exit 4 means owning a salvage
parse.