| --- |
| license: apache-2.0 |
| library_name: coreml |
| tags: |
| - tts |
| - text-to-speech |
| - coreml |
| - ane |
| - apple |
| - on-device |
| - voice-cloning |
| language: |
| - en |
| pipeline_tag: text-to-speech |
| base_model: |
| - YatharthS/LuxTTS |
| base_model_relation: quantized |
| --- |
| |
| # LuxTTS CoreML |
|
|
| CoreML conversion of [YatharthS/LuxTTS](https://huggingface.co/YatharthS/LuxTTS) for on-device |
| inference on Apple platforms. |
|
|
| LuxTTS is a 48 kHz zero-shot voice-cloning TTS (~123M params): give it a short reference clip |
| plus its transcript, and it speaks new text in that voice. It is based on |
| [ZipVoice](https://github.com/k2-fsa/ZipVoice) (k2-fsa, Xiaomi) — a flow-matching Zipformer |
| decoder run with a 4-step solver — with a dual-head Vocos vocoder from LinaCodec (ysharma3501). |
|
|
| ## Models |
|
|
| | Path | Model | Size | Notes | |
| |------|-------|------|-------| |
| | `gpu/TextEncoder.mlmodelc` | Text encoder (256 tokens) | 9.5 MB | fp16 | |
| | `gpu/FmDecoder.mlmodelc` | Flow-matching decoder, 1024-frame bucket | 256 MB | fp16, original graph — **macOS GPU path** | |
| | `gpu-6bit/FmDecoder.mlmodelc` | 6-bit palettized decoder, 1024-frame bucket | 115 MB | full GPU speed, -0.45 dB, transcript-transparent | |
| | `gpu-long/TextEncoder.mlmodelc` | Text encoder (512 tokens) | 13 MB | fp16 | |
| | `gpu-long/FmDecoder.mlmodelc` | Decoder, 2048-frame bucket | 349 MB | long-form macOS (~16.8 s per bucket) | |
| | `ane/TextEncoder.mlmodelc` | Text encoder (256 tokens) | 9.5 MB | fp16 | |
| | `ane/FmDecoder.mlmodelc` | ANE-canonical decoder, 1024-frame bucket | 313 MB | 100% ANE placement — **iPhone path** | |
| | `vocoder/Vocoder282.mlmodelc` | Vocos vocoder, 282 gen frames (~3.0 s) | 33 MB | ISTFT + 24k→48k resample + crossover in-graph | |
| | `vocoder/Vocoder555.mlmodelc` | Vocos vocoder, 555 gen frames (~5.9 s) | 33 MB | matches the full 1024-frame bucket | |
| | `tokens.txt` / `config.json` | Phoneme token table + model config | — | from upstream LuxTTS | |
|
|
| ### Two decoder graphs — pick by compute unit |
|
|
| - **`gpu/`** — the original graph layout. Fastest on Mac GPU. **Do not run it on the ANE**: |
| the seq-first rel-pos attention path loses precision there and produces corrupted audio. |
| - **`ane/`** — ANE-canonical rewrite (channels-first `(1,C,1,S)`, 1×1-conv projections, |
| constant rel-pos basis, split depthwise kernels). 100% ANE op placement, zero CPU fallback. |
| This is the iPhone path; it is slower than the original graph on GPU, so keep both. |
|
|
| All decoder inputs/outputs are identical across graphs |
| (`t, x, text_condition, speech_condition, guidance_scale, padding_mask → v`). |
|
|
| ## Performance (M5 Pro, macOS) |
|
|
| | path | fm step | core RTFx (full bucket) | steady footprint | |
| |------|---------|-------------------------|------------------| |
| | `gpu/` fp16, GPU | 14.7 ms | 92x (~5.9 s gen) | 996 MB | |
| | `gpu-6bit/`, GPU | 15.0 ms | ~92x | 419 MB | |
| | `ane/` fp16, ANE | 54.6 ms | 27x | **25.5 MB** | |
|
|
| - Vocoder: 2.34 ms on GPU (282 frames); end-to-end ~64 ms for 3 s of audio (~47x realtime). |
| - Quality vs PyTorch reference: log-mel cosine 0.99925 (GPU path); whisper-base transcripts |
| word-identical for all shipped variants. |
| - Footprints are `phys_footprint` from a Swift host; the ANE graph keeps weights and |
| activations in ANE-managed memory, hence the 25.5 MB jetsam-visible footprint. |
|
|
| ## Fixed-shape buckets |
|
|
| CoreML graphs are compiled at fixed shapes. Frame budget covers prompt + generated speech |
| (86.13 frames/s): |
|
|
| - `gpu/`, `ane/`: 1024 frames (~5.9 s generated after a 5 s prompt), 256 text tokens |
| - `gpu-long/`: 2048 frames (~16.8 s generated), 512 text tokens |
| - vocoder: fixed at 282 or 555 generated frames |
|
|
| Chunk long text at sentence boundaries for the 1024-frame bucket on iPhone. |
|
|
| ## Usage notes |
|
|
| The CoreML graphs cover the text encoder, flow-matching decoder, and vocoder (ISTFT, |
| 24k→48k resample, and the Linkwitz crossover are all in-graph). The host must provide: |
|
|
| - Phoneme tokenization against `tokens.txt` (espeak-compatible token set) — the frontend is |
| phoneme-token based, not raw text. |
| - Duration expansion of encoder output into the frame grid (upstream `pad_labels` keeps S+1 |
| token rows; the pad slot's embedding fills remainder frames). |
| - The 4-step anchor-Euler flow solver with `t_shift = 0.5` and `guidance_scale = 3.0`. |
| - Prompt trimming at a **speech pause** (VAD boundary), not a fixed duration — a prompt |
| hard-cut mid-phrase makes the model elide sentence-initial words. |
| - `speed` default **1.0** (upstream's `generate()` default of 1.3 clips sentence onsets). |
|
|
| Swift integration in [FluidAudio](https://github.com/FluidInference/FluidAudio) is upcoming. |
|
|
| ## Known limitations |
|
|
| - Fixed-shape buckets only (no enumerated/flexible shapes yet). |
| - An int8-linear variant exists but crashes MPSGraph on macOS 26.5 GPU — not shipped. |
| - The ANE graph is ~0.5 dB softer than the GPU graph (fp16 accumulation over 16 layers × |
| 4 solver steps); transcripts are unaffected. |
|
|
| ## Attribution & license |
|
|
| - [LuxTTS](https://huggingface.co/YatharthS/LuxTTS) by YatharthS — Apache-2.0. |
| - Based on [ZipVoice](https://github.com/k2-fsa/ZipVoice) by k2-fsa (Xiaomi) — Apache-2.0. |
| - Vocoder from LinaCodec (ysharma3501). |
| - CoreML conversion by [FluidInference](https://huggingface.co/FluidInference); conversion |
| source lives in the mobius repo (`models/tts/zipvoice`, PR #75). |
|
|
| ## References |
|
|
| - https://huggingface.co/YatharthS/LuxTTS |
| - https://github.com/k2-fsa/ZipVoice |
| - https://github.com/FluidInference/FluidAudio |
|
|
