Qwen3-TTS for Unity

On-device text-to-speech using the Qwen3-TTS 12.5 Hz 1.7B checkpoints through ONNX Runtime. Two capabilities, each a separate checkpoint:

Checkpoint	What it does	Per-utterance instruct
VoiceDesign	Invents a speaker from a natural-language description. A different person every generate.	Yes — the description is the voice.
Base	Clones a speaker from a reference recording, using Qwen's in-context path (reference codes + transcript + speaker embedding).	No. See Instruction control.

The usual flow is both: design until you like a take, then clone that take so the voice stays put.

This package does not include or download weights. Export them yourself with the scripts in Tools~/qwen3_tts_onnx/ and point the package at the result.

Install

Requires Unity 6000.0.46f1 or newer.

1. Add the OpenUPM scoped registry so the ONNX Runtime dependency resolves. In Edit → Project Settings → Package Manager, or directly in Packages/manifest.json:

   "scopedRegistries": [
     {
       "name": "OpenUPM",
       "url": "https://package.openupm.com",
       "scopes": [ "com.github.asus4" ]
     }
   ]

2. Add this package from its git URL — Window → Package Manager → + → Add package from git URL… — or as a dependency in manifest.json:

   "com.genesis.qwentts.unity": "https://github.com/genesisinteractive/Qwen3-TTS-Unity.git#v0.1.1"

   Drop the #v0.1.1 suffix to track main. com.github.asus4.onnxruntime and com.unity.nuget.newtonsoft-json are pulled in as dependencies.

3. Export the weights and point the package at them — see Installing weights.

Read this first: it is not a small model

Measured on an Apple-silicon laptop, CPU execution provider, fp32. Disk is the required file set for one checkpoint (unified talker, code predictor, vocoder, embedding tables, tokenizer; plus the speaker and tokenizer encoders on Base); optional int8 graphs add ~2.4 GB on top.

	Per checkpoint
Disk	~8 GB (8.3 GB VoiceDesign, 8.7 GB Base)
Resident once loaded	~7 GB (5.4 GB of ONNX sessions, ~1.5 GB of embedding tables)
Cold session open	~11 s, ~3 s with a warm page cache
Generation	~0.97× of real time — finishes just ahead of playback

ONNX Runtime does not keep external .onnx.data lazily mapped, so resident memory tracks file size roughly 1:1. Budget accordingly:

• One checkpoint resident: 16 GB machine is fine, 32 GB comfortable.
• Both resident: wants 32 GB, and is usually avoidable — see Residency.
• Mobile and XR are out of scope at fp32. The package compiles anywhere, but these weights do not fit in a phone or headset app.

Developed against Unity 6000.x, Windows and macOS.

Installing weights

1. Export both checkpoints (see Tools~/qwen3_tts_onnx/README.md). You get one folder each, named Qwen3-1.7B-VoiceDesign and Qwen3-1.7B-Base.
2. Put them under a root of your choosing.
3. Point the package at that root and check what it found:

QwenTts.Initialize(new QwenTtsSettings
{
    ModelRoot = Path.Combine(Application.persistentDataPath, "QwenTTS"),
    MemoryUsage = MemoryUsage.Balanced,
    LogLevel = LogLevel.INFO,
});

var status = QwenTts.GetStatus(QwenCheckpoint.Base);
Debug.Log(status);   // installed / loaded / missing files / bytes

ModelRoot defaults to StreamingAssets/QwenTTS, which is convenient in the editor and usually wrong for a shipped player, because StreamingAssets is copied into the build. Prefer a folder you install or download into.

Pre-exported ONNX models

As an alternative to running the exporter, download the zips of the files this package actually opens. They are split so a Drive upload is not a 20 GB blob: each fp32 checkpoint is its own zip, and int8 is an overlay.

Zip	Contents	Size
QwenTTS-VoiceDesign.zip	fp32 VoiceDesign	~8.3 GB
QwenTTS-Base.zip	fp32 Base (includes speaker + tokenizer encoders)	~8.7 GB
QwenTTS-VoiceDesign-int8.zip	int8 graphs for VoiceDesign	~2.5 GB
QwenTTS-Base-int8.zip	int8 graphs for Base	~2.5 GB

Extract every zip you want into the same parent so the layout is

QwenTTS/
  Qwen3-1.7B-VoiceDesign/     talker, code_predictor, vocoder, embeddings/, tokenizer/
  Qwen3-1.7B-Base/            the same, plus speaker_encoder and tokenizer_encoder

The int8 zips drop talker_int8 / code_predictor_int8 into that same checkpoint folder — they are not a complete install. Skip them unless you will set QwenTtsSettings.Precision = QwenPrecision.Int8.

You only need the checkpoint you will load. Design-then-clone wants both fp32 zips. Point QwenTtsSettings.ModelRoot at that QwenTTS folder, or drop it at Assets/StreamingAssets/QwenTTS/ (the default). These archives are the unified-talker set; they do not include the superseded talker_prefill / talker_decode pair. Rebuild them from an export folder with python3 Tools~/qwen3_tts_onnx/pack_gdrive.py.

Window → Qwen3 TTS → Model Status shows the same information without writing code.

Designing a voice

var voice = await QwenTts.CreateDesignedVoiceAsync(new VoiceDesignSpec(
    "Male, thirties, warm and conversational, close-mic, not a narrator."));

var take = await voice.SpeakAsync("Hello there. Good to finally meet you.");
audioSource.clip = take.ToAudioClip();     // main thread
audioSource.Play();

The description is the voice. Calling SpeakAsync again gives you the same style in a different person's mouth — that is what the checkpoint does, and there is no seed that pins it. To keep a voice, clone it.

Keeping a voice: clone the take

// Render something long enough to characterise the speaker.
var line  = "The quick brown fox jumps over the lazy dog, then sleeps in the sun.";
var take  = await voice.SpeakAsync(line);

// The transcript is required: it is what makes this in-context cloning
// rather than a generic speaker-embedding match.
var locked = await QwenTts.CreateClonedVoiceAsync(take.ToAudioClip(), line);

var next = await locked.SpeakAsync("Careful with that.");

Reference audio quality decides clone quality:

• At least 4 seconds. Shorter and there are too few frames to pin a speaker, so takes drift between utterances. The package warns below this.
• Keep it at 24 kHz. The speaker encoder reads mel up to 12 kHz, so a 16 kHz reference has the top of the identifying band already missing. SpeakAsync returns 24 kHz unless you ask for something else, and the package warns if a reference is lower.
• At most 20 seconds are used. The tokenizer encoder graph is traced at a fixed 20 s window (QwenTokenizerEncoderModel.GraphSeconds), so a longer reference is truncated to its first 20 s before the codes are extracted. Trim to the best 4–20 s yourself rather than handing over a whole recording.
• Pass the exact transcript. Without it you get a stable voice that is not the one in your recording.

Instruction control

Alibaba's 12 Hz lineup splits this on purpose. Qwen3-TTS marks Instruction Control on VoiceDesign and on 1.7B CustomVoice, and leaves the column blank for Base. Their Python API matches that table:

Official call	instruct
generate_voice_design(text, language, instruct=…)	Required. Timbre, age, accent, and this-line acting live in one string.
generate_custom_voice(text, language, speaker, instruct=…)	Optional per utterance on the nine named speakers ("Very happy.", "say this angrily"). This package does not ship CustomVoice.
generate_voice_clone(text, language, ref_audio, ref_text)	Not a parameter. Reference wav + transcript + the new line only.

Qwen's maintainer closed issue #25 with: the 12 Hz Base model's control is unstable, so Base does not support instruct. A later 25 Hz Voice Editing checkpoint is meant to clone and take instructions. People who passed instruct= into clone calls anyway reported no effect (discussion #231).

This package follows that contract. SpeechOptions has language and sampling, not an instruct. QwenVoice.Instruct is the VoiceDesign description, used only on the design checkpoint. SynthesizeCloned tokenizes the line with instruct: null. Putting "while laughing" in the spoken text is just words the model may colour; it is not a style channel. Putting emotion in the VoiceDesign instruct, then locking that take, bakes that delivery into the reference — every later clone line tends to stay in that pocket, and you cannot retarget it per card.

Do not add a per-line instruct on the clone path expecting the published 1.7B Base weights to honour it.

Saving and reloading

await locked.SaveAsync(folder, take);              // take is optional
var again = await QwenTts.LoadVoiceAsync(folder);

A saved clone stores its derived prompt — speaker embedding plus reference codes — beside the reference audio, so reloading is a file read rather than another speaker-encoder and tokenizer-encoder run. A prompt file from an incompatible export is ignored and the prompt is re-derived.

Streaming

Generation runs a little faster than playback, so audio can start long before the line is finished.

var player = new Progress<SpeechChunk>(chunk =>
{
    // Reported from a worker thread; marshal before touching an AudioClip.
    // chunk.Pcm holds only samples not reported before, so appending each
    // chunk in order reproduces the utterance.
    Enqueue(chunk.Pcm, chunk.SampleRate);
});

// Still returns the whole thing, for callers that also want to cache it.
var whole = await voice.SpeakStreamAsync("Careful with that.", player);

First audio arrives in about a second rather than at the end. Tune with SpeechOptions.FirstChunkFrames (default 6, roughly half a second) and MaxChunkFrames (48).

Chunks are not decoded independently. This codec decoder's output depends on its whole input rather than a bounded window, so overlap-and-trim does not apply here; instead the prefix is re-decoded for each chunk and only new samples are handed over. That makes concatenated chunks match a single decode to within about 2e-6 rather than approximately, at the cost of re-decoding — which is why chunk sizes double rather than staying small.

Audio helpers

Generation hands back a float[], and a caller usually has to assemble it — downmix a reference recording, match a target rate, or stitch takes together with a beat between them. QwenAudio has the pieces so every host does not rewrite the same downmix-and-resample loop:

float[] mono   = QwenAudio.ToMono(interleaved, channels);
float[] at48k  = QwenAudio.Resample(mono, 24000, 48000);
float[] beat   = QwenAudio.Silence(24000, 0.25f);
float[] joined = QwenAudio.Concatenate(new[] { lineOne, lineTwo }, 24000, gapSeconds: 0.3f);

// AudioClip overloads too, for convenience. Main thread only.
AudioClip clip = QwenAudio.Concatenate(clips, 24000, gapSeconds: 0.3f);

The float[] overloads are thread-agnostic and worth preferring. The AudioClip ones must run on Unity's main thread, since Create, SetData and GetData all require it.

Precision

The talker and code predictor read every weight once per generated token, so they are limited by memory bandwidth rather than arithmetic. int8 weights cut that traffic and are about 1.4× faster end to end, with the talker resident at 2.35 GB instead of 5.67:

QwenTts.Initialize(new QwenTtsSettings
{
    ModelRoot = "...",
    Precision = QwenPrecision.Int8,   // default is Float32
});

Produce the quantized graphs with Tools~/qwen3_tts_onnx/quantize_int8.py. Precision is resolved per graph, so a checkpoint missing talker_int8.onnx uses the fp32 talker rather than failing. Both checkpoints quantize, and clones gain slightly more than designed voices (1.55× against 1.40×) because their in-context prefill puts more of the work in the talker. The vocoder and the speaker and tokenizer encoders stay fp32, so nothing about how a voice is captured is quantized.

It is opt-in because it is not free. Quantized audio is not bit-identical to fp32 and a voice can differ subtly, so listen before shipping it. Judge it by transcribing the output rather than by numerical error: quantizing every layer passes every numerical check and still drops phonemes, which is why quantize_int8.py holds the output projection and the outermost decoder layers back by default.

fp16 is deliberately not offered. ONNX Runtime's CPU provider has no fast fp16 kernels for these operations on Apple silicon, so it casts and computes element-wise: measurably slower than fp32 while being numerically near-perfect.

Residency

The two checkpoints are normally needed in different phases — VoiceDesign while the player is picking a voice, Base for everything after — so load and drop them explicitly instead of holding both:

await QwenTts.WarmUpAsync(QwenCheckpoint.VoiceDesign);   // loading screen
// ... player auditions voices, picks one, you clone it ...
QwenTts.Evict(QwenCheckpoint.VoiceDesign);               // ~7 GB back
await QwenTts.WarmUpAsync(QwenCheckpoint.Base);

WarmUpAsync matters: without it the first utterance pays the whole session open. MemoryUsage sets the default policy:

Mode	Behaviour
Performance	Load eagerly, never drop.
Balanced	Load on first use, then keep. Default.
Optimal	Load per use and dispose after. Idle stays near the embedding tables (~1.5 GB), at the cost of reopening per utterance.

Generation options

await voice.SpeakAsync(text, new SpeechOptions
{
    Language   = QwenLanguages.Default,   // 10 languages, or QwenLanguages.Auto
    SampleRate = 24000,                   // 0 keeps native
    Temperature = 0.9f, TopK = 50, TopP = 1f, RepetitionPenalty = 1.05f,
    MaxNewTokens = 2048,                  // frames at 12.5 Hz; 2048 is ~2.7 minutes
},
progress: new Progress<SpeechProgress>(p => Debug.Log($"{p.Seconds:0.0}s")),
cancellationToken: token);

Defaults match Qwen's own generate config. Two settings are worth knowing about even if you never change them:

• RepetitionPenalty above 1 is load-bearing. Greedy decoding with the penalty disabled can loop without ever emitting end-of-speech.
• MaxNewTokens bounds cost, not just length. The per-step KV cache copy grows with the square of sequence length, so it is negligible for an utterance and expensive for a runaway one.

Pass a CancellationToken for anything the player can skip.

Threading

SpeakAsync, WarmUpAsync and the create calls do their work on the thread pool and are safe to await from the main thread. AudioClip.GetData and AudioClip.Create are Unity main-thread APIs, so a reference clip is read on the calling thread and SpeechResult.ToAudioClip() must be called on the main thread — which is why generation returns PCM rather than a clip.

The two checkpoints have independent locks, so a designed and a cloned voice can generate at the same time. Two utterances on the same checkpoint serialise, because the talker reuses its KV and sampler buffers.

Logging

ONNX Runtime's own diagnostics are routed into the Unity console, tagged with the model they came from:

[ONNX-INFO][talker][onnxruntime] Session successfully initialized.

QwenTtsSettings.LogLevel sets the verbosity. Note that ONNX Runtime allows one environment per process, and whichever library creates it owns the logging sink for every library in the process. If another ONNX library in your project initializes first, this package uses its sink and leaves the level alone; if this package initializes first, the other library can attribute its own models with QwenTts.SetOnnxLogContext(name).

Licence and attribution

Apache-2.0. The ONNX inference path derives from ElBruno.QwenTTS (MIT), and prompt construction follows Alibaba's qwen-tts reference implementation (Apache-2.0). See THIRD_PARTY_NOTICES.md.

Model weights are Alibaba's, released under Apache-2.0, and are not part of this package. Check the model cards for the exact terms before shipping:

• Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign
• Qwen/Qwen3-TTS-12Hz-1.7B-Base