LuxTTS CoreML

CoreML conversion of YatharthS/LuxTTS for on-device inference on Apple platforms.

LuxTTS is a 48 kHz zero-shot voice-cloning TTS (~123M params): give it a short reference clip plus its transcript, and it speaks new text in that voice. It is based on ZipVoice (k2-fsa, Xiaomi) — a flow-matching Zipformer decoder run with a 4-step solver — with a dual-head Vocos vocoder from LinaCodec (ysharma3501).

Models

Path	Model	Size	Notes
`gpu/TextEncoder.mlmodelc`	Text encoder (256 tokens)	9.5 MB	fp16
`gpu/FmDecoder.mlmodelc`	Flow-matching decoder, 1024-frame bucket	256 MB	fp16, original graph — macOS GPU path
`gpu-6bit/FmDecoder.mlmodelc`	6-bit palettized decoder, 1024-frame bucket	115 MB	full GPU speed, -0.45 dB, transcript-transparent
`gpu-long/TextEncoder.mlmodelc`	Text encoder (512 tokens)	13 MB	fp16
`gpu-long/FmDecoder.mlmodelc`	Decoder, 2048-frame bucket	349 MB	long-form macOS (~16.8 s per bucket)
`ane/TextEncoder.mlmodelc`	Text encoder (256 tokens)	9.5 MB	fp16
`ane/FmDecoder.mlmodelc`	ANE-canonical decoder, 1024-frame bucket	313 MB	100% ANE placement — iPhone path
`vocoder/Vocoder282.mlmodelc`	Vocos vocoder, 282 gen frames (~3.0 s)	33 MB	ISTFT + 24k→48k resample + crossover in-graph
`vocoder/Vocoder555.mlmodelc`	Vocos vocoder, 555 gen frames (~5.9 s)	33 MB	matches the full 1024-frame bucket
`tokens.txt` / `config.json`	Phoneme token table + model config	—	from upstream LuxTTS

Two decoder graphs — pick by compute unit

gpu/ — the original graph layout. Fastest on Mac GPU. Do not run it on the ANE: the seq-first rel-pos attention path loses precision there and produces corrupted audio.
ane/ — ANE-canonical rewrite (channels-first (1,C,1,S), 1×1-conv projections, constant rel-pos basis, split depthwise kernels). 100% ANE op placement, zero CPU fallback. This is the iPhone path; it is slower than the original graph on GPU, so keep both.

All decoder inputs/outputs are identical across graphs (t, x, text_condition, speech_condition, guidance_scale, padding_mask → v).

Performance (M5 Pro, macOS)

path	fm step	core RTFx (full bucket)	steady footprint
`gpu/` fp16, GPU	14.7 ms	92x (~5.9 s gen)	996 MB
`gpu-6bit/`, GPU	15.0 ms	~92x	419 MB
`ane/` fp16, ANE	54.6 ms	27x	25.5 MB

Vocoder: 2.34 ms on GPU (282 frames); end-to-end ~~64 ms for 3 s of audio (~~47x realtime).
Quality vs PyTorch reference: log-mel cosine 0.99925 (GPU path); whisper-base transcripts word-identical for all shipped variants.
Footprints are phys_footprint from a Swift host; the ANE graph keeps weights and activations in ANE-managed memory, hence the 25.5 MB jetsam-visible footprint.

Fixed-shape buckets

CoreML graphs are compiled at fixed shapes. Frame budget covers prompt + generated speech (86.13 frames/s):

gpu/, ane/: 1024 frames (~5.9 s generated after a 5 s prompt), 256 text tokens
gpu-long/: 2048 frames (~16.8 s generated), 512 text tokens
vocoder: fixed at 282 or 555 generated frames

Chunk long text at sentence boundaries for the 1024-frame bucket on iPhone.

Usage notes

The CoreML graphs cover the text encoder, flow-matching decoder, and vocoder (ISTFT, 24k→48k resample, and the Linkwitz crossover are all in-graph). The host must provide:

Phoneme tokenization against tokens.txt (espeak-compatible token set) — the frontend is phoneme-token based, not raw text.
Duration expansion of encoder output into the frame grid (upstream pad_labels keeps S+1 token rows; the pad slot's embedding fills remainder frames).
The 4-step anchor-Euler flow solver with t_shift = 0.5 and guidance_scale = 3.0.
Prompt trimming at a speech pause (VAD boundary), not a fixed duration — a prompt hard-cut mid-phrase makes the model elide sentence-initial words.
speed default 1.0 (upstream's generate() default of 1.3 clips sentence onsets).

Swift integration in FluidAudio is upcoming.

Known limitations

Fixed-shape buckets only (no enumerated/flexible shapes yet).
An int8-linear variant exists but crashes MPSGraph on macOS 26.5 GPU — not shipped.
The ANE graph is ~0.5 dB softer than the GPU graph (fp16 accumulation over 16 layers × 4 solver steps); transcripts are unaffected.

Attribution & license

LuxTTS by YatharthS — Apache-2.0.
Based on ZipVoice by k2-fsa (Xiaomi) — Apache-2.0.
Vocoder from LinaCodec (ysharma3501).
CoreML conversion by FluidInference; conversion source lives in the mobius repo (models/tts/zipvoice, PR #75).

References

Downloads last month: -

Model tree for FluidInference/luxtts-coreml

Base model

YatharthS/LuxTTS

Quantized

(1)

this model