Hugging Face's logo

FluidInference
/
luxtts-coreml

Text-to-Speech
Core ML
English
tts
ane
apple
on-device
voice-cloning
Model card Files
xet
 Community
luxtts-coreml / README.md
alexwengg's picture
alexwengg
LuxTTS CoreML: gpu/ane/6bit/long decoder graphs + 48kHz vocoder (mobius PR #75)
eced034 verified
preview code
|
Raw
History Blame Contribute Delete
5.42 kB
metadata 
license: apache-2.0
library_name: coreml
tags:
  - tts
  - text-to-speech
  - coreml
  - ane
  - apple
  - on-device
  - voice-cloning
language:
  - en
pipeline_tag: text-to-speech
base_model:
  - YatharthS/LuxTTS
base_model_relation: quantized

LuxTTS CoreML

CoreML conversion of YatharthS/LuxTTS for on-device inference on Apple platforms.

LuxTTS is a 48 kHz zero-shot voice-cloning TTS (~123M params): give it a short reference clip plus its transcript, and it speaks new text in that voice. It is based on ZipVoice (k2-fsa, Xiaomi) — a flow-matching Zipformer decoder run with a 4-step solver — with a dual-head Vocos vocoder from LinaCodec (ysharma3501).

Models

Path Model Size Notes
gpu/TextEncoder.mlmodelc Text encoder (256 tokens) 9.5 MB fp16
gpu/FmDecoder.mlmodelc Flow-matching decoder, 1024-frame bucket 256 MB fp16, original graph — macOS GPU path
gpu-6bit/FmDecoder.mlmodelc 6-bit palettized decoder, 1024-frame bucket 115 MB full GPU speed, -0.45 dB, transcript-transparent
gpu-long/TextEncoder.mlmodelc Text encoder (512 tokens) 13 MB fp16
gpu-long/FmDecoder.mlmodelc Decoder, 2048-frame bucket 349 MB long-form macOS (~16.8 s per bucket)
ane/TextEncoder.mlmodelc Text encoder (256 tokens) 9.5 MB fp16
ane/FmDecoder.mlmodelc ANE-canonical decoder, 1024-frame bucket 313 MB 100% ANE placement — iPhone path
vocoder/Vocoder282.mlmodelc Vocos vocoder, 282 gen frames (~3.0 s) 33 MB ISTFT + 24k→48k resample + crossover in-graph
vocoder/Vocoder555.mlmodelc Vocos vocoder, 555 gen frames (~5.9 s) 33 MB matches the full 1024-frame bucket
tokens.txt / config.json Phoneme token table + model config — from upstream LuxTTS

Two decoder graphs — pick by compute unit

  • gpu/ — the original graph layout. Fastest on Mac GPU. Do not run it on the ANE: the seq-first rel-pos attention path loses precision there and produces corrupted audio.
  • ane/ — ANE-canonical rewrite (channels-first (1,C,1,S), 1×1-conv projections, constant rel-pos basis, split depthwise kernels). 100% ANE op placement, zero CPU fallback. This is the iPhone path; it is slower than the original graph on GPU, so keep both.

All decoder inputs/outputs are identical across graphs (t, x, text_condition, speech_condition, guidance_scale, padding_mask → v).

Performance (M5 Pro, macOS)

path fm step core RTFx (full bucket) steady footprint
gpu/ fp16, GPU 14.7 ms 92x (~5.9 s gen) 996 MB
gpu-6bit/, GPU 15.0 ms ~92x 419 MB
ane/ fp16, ANE 54.6 ms 27x 25.5 MB
  • Vocoder: 2.34 ms on GPU (282 frames); end-to-end 64 ms for 3 s of audio (47x realtime).
  • Quality vs PyTorch reference: log-mel cosine 0.99925 (GPU path); whisper-base transcripts word-identical for all shipped variants.
  • Footprints are phys_footprint from a Swift host; the ANE graph keeps weights and activations in ANE-managed memory, hence the 25.5 MB jetsam-visible footprint.

Fixed-shape buckets

CoreML graphs are compiled at fixed shapes. Frame budget covers prompt + generated speech (86.13 frames/s):

  • gpu/, ane/: 1024 frames (~5.9 s generated after a 5 s prompt), 256 text tokens
  • gpu-long/: 2048 frames (~16.8 s generated), 512 text tokens
  • vocoder: fixed at 282 or 555 generated frames

Chunk long text at sentence boundaries for the 1024-frame bucket on iPhone.

Usage notes

The CoreML graphs cover the text encoder, flow-matching decoder, and vocoder (ISTFT, 24k→48k resample, and the Linkwitz crossover are all in-graph). The host must provide:

  • Phoneme tokenization against tokens.txt (espeak-compatible token set) — the frontend is phoneme-token based, not raw text.
  • Duration expansion of encoder output into the frame grid (upstream pad_labels keeps S+1 token rows; the pad slot's embedding fills remainder frames).
  • The 4-step anchor-Euler flow solver with t_shift = 0.5 and guidance_scale = 3.0.
  • Prompt trimming at a speech pause (VAD boundary), not a fixed duration — a prompt hard-cut mid-phrase makes the model elide sentence-initial words.
  • speed default 1.0 (upstream's generate() default of 1.3 clips sentence onsets).

Swift integration in FluidAudio is upcoming.

Known limitations

  • Fixed-shape buckets only (no enumerated/flexible shapes yet).
  • An int8-linear variant exists but crashes MPSGraph on macOS 26.5 GPU — not shipped.
  • The ANE graph is ~0.5 dB softer than the GPU graph (fp16 accumulation over 16 layers × 4 solver steps); transcripts are unaffected.

Attribution & license

  • LuxTTS by YatharthS — Apache-2.0.
  • Based on ZipVoice by k2-fsa (Xiaomi) — Apache-2.0.
  • Vocoder from LinaCodec (ysharma3501).
  • CoreML conversion by FluidInference; conversion source lives in the mobius repo (models/tts/zipvoice, PR #75).

References