---
license: apache-2.0
library_name: coreml
tags:
- tts
- text-to-speech
- coreml
- ane
- apple
- on-device
- voice-cloning
language:
- en
pipeline_tag: text-to-speech
base_model:
- YatharthS/LuxTTS
base_model_relation: quantized
---

# LuxTTS CoreML

CoreML conversion of [YatharthS/LuxTTS](https://huggingface.co/YatharthS/LuxTTS) for on-device
inference on Apple platforms.

LuxTTS is a 48 kHz zero-shot voice-cloning TTS (~123M params): give it a short reference clip
plus its transcript, and it speaks new text in that voice. It is based on
[ZipVoice](https://github.com/k2-fsa/ZipVoice) (k2-fsa, Xiaomi) — a flow-matching Zipformer
decoder run with a 4-step solver — with a dual-head Vocos vocoder from LinaCodec (ysharma3501).

## Models

| Path | Model | Size | Notes |
|------|-------|------|-------|
| `gpu/TextEncoder.mlmodelc` | Text encoder (256 tokens) | 9.5 MB | fp16 |
| `gpu/FmDecoder.mlmodelc` | Flow-matching decoder, 1024-frame bucket | 256 MB | fp16, original graph — **macOS GPU path** |
| `gpu-6bit/FmDecoder.mlmodelc` | 6-bit palettized decoder, 1024-frame bucket | 115 MB | full GPU speed, -0.45 dB, transcript-transparent |
| `gpu-long/TextEncoder.mlmodelc` | Text encoder (512 tokens) | 13 MB | fp16 |
| `gpu-long/FmDecoder.mlmodelc` | Decoder, 2048-frame bucket | 349 MB | long-form macOS (~16.8 s per bucket) |
| `ane/TextEncoder.mlmodelc` | Text encoder (256 tokens) | 9.5 MB | fp16 |
| `ane/FmDecoder.mlmodelc` | ANE-canonical decoder, 1024-frame bucket | 313 MB | 100% ANE placement — **iPhone path** |
| `vocoder/Vocoder282.mlmodelc` | Vocos vocoder, 282 gen frames (~3.0 s) | 33 MB | ISTFT + 24k→48k resample + crossover in-graph |
| `vocoder/Vocoder555.mlmodelc` | Vocos vocoder, 555 gen frames (~5.9 s) | 33 MB | matches the full 1024-frame bucket |
| `tokens.txt` / `config.json` | Phoneme token table + model config | — | from upstream LuxTTS |

### Two decoder graphs — pick by compute unit

- **`gpu/`** — the original graph layout. Fastest on Mac GPU. **Do not run it on the ANE**:
  the seq-first rel-pos attention path loses precision there and produces corrupted audio.
- **`ane/`** — ANE-canonical rewrite (channels-first `(1,C,1,S)`, 1×1-conv projections,
  constant rel-pos basis, split depthwise kernels). 100% ANE op placement, zero CPU fallback.
  This is the iPhone path; it is slower than the original graph on GPU, so keep both.

All decoder inputs/outputs are identical across graphs
(`t, x, text_condition, speech_condition, guidance_scale, padding_mask → v`).

## Performance (M5 Pro, macOS)

| path | fm step | core RTFx (full bucket) | steady footprint |
|------|---------|-------------------------|------------------|
| `gpu/` fp16, GPU | 14.7 ms | 92x (~5.9 s gen) | 996 MB |
| `gpu-6bit/`, GPU | 15.0 ms | ~92x | 419 MB |
| `ane/` fp16, ANE | 54.6 ms | 27x | **25.5 MB** |

- Vocoder: 2.34 ms on GPU (282 frames); end-to-end ~64 ms for 3 s of audio (~47x realtime).
- Quality vs PyTorch reference: log-mel cosine 0.99925 (GPU path); whisper-base transcripts
  word-identical for all shipped variants.
- Footprints are `phys_footprint` from a Swift host; the ANE graph keeps weights and
  activations in ANE-managed memory, hence the 25.5 MB jetsam-visible footprint.

## Fixed-shape buckets

CoreML graphs are compiled at fixed shapes. Frame budget covers prompt + generated speech
(86.13 frames/s):

- `gpu/`, `ane/`: 1024 frames (~5.9 s generated after a 5 s prompt), 256 text tokens
- `gpu-long/`: 2048 frames (~16.8 s generated), 512 text tokens
- vocoder: fixed at 282 or 555 generated frames

Chunk long text at sentence boundaries for the 1024-frame bucket on iPhone.

## Usage notes

The CoreML graphs cover the text encoder, flow-matching decoder, and vocoder (ISTFT,
24k→48k resample, and the Linkwitz crossover are all in-graph). The host must provide:

- Phoneme tokenization against `tokens.txt` (espeak-compatible token set) — the frontend is
  phoneme-token based, not raw text.
- Duration expansion of encoder output into the frame grid (upstream `pad_labels` keeps S+1
  token rows; the pad slot's embedding fills remainder frames).
- The 4-step anchor-Euler flow solver with `t_shift = 0.5` and `guidance_scale = 3.0`.
- Prompt trimming at a **speech pause** (VAD boundary), not a fixed duration — a prompt
  hard-cut mid-phrase makes the model elide sentence-initial words.
- `speed` default **1.0** (upstream's `generate()` default of 1.3 clips sentence onsets).

Swift integration in [FluidAudio](https://github.com/FluidInference/FluidAudio) is upcoming.

## Known limitations

- Fixed-shape buckets only (no enumerated/flexible shapes yet).
- An int8-linear variant exists but crashes MPSGraph on macOS 26.5 GPU — not shipped.
- The ANE graph is ~0.5 dB softer than the GPU graph (fp16 accumulation over 16 layers ×
  4 solver steps); transcripts are unaffected.

## Attribution & license

- [LuxTTS](https://huggingface.co/YatharthS/LuxTTS) by YatharthS — Apache-2.0.
- Based on [ZipVoice](https://github.com/k2-fsa/ZipVoice) by k2-fsa (Xiaomi) — Apache-2.0.
- Vocoder from LinaCodec (ysharma3501).
- CoreML conversion by [FluidInference](https://huggingface.co/FluidInference); conversion
  source lives in the mobius repo (`models/tts/zipvoice`, PR #75).

## References

- https://huggingface.co/YatharthS/LuxTTS
- https://github.com/k2-fsa/ZipVoice
- https://github.com/FluidInference/FluidAudio