--- language: - en license: mit library_name: coreml tags: - text-to-speech - tts - styletts2 - coreml - apple-silicon - voice-cloning pipeline_tag: text-to-speech --- # StyleTTS2 LibriTTS — CoreML Apple CoreML port of [yl4579/StyleTTS2](https://github.com/yl4579/StyleTTS2) (LibriTTS 2nd-stage checkpoint, epoch 20). 9-stage `.mlpackage` chain with mixed-precision and per-stage compute-unit assignments tuned for Apple Silicon (CPU + ANE + GPU). 24 kHz mono synthesis. Zero-shot voice cloning from a 3-10 second reference WAV. ## Highlights - **9 stages, 258 MB on disk**, all fp16 except `har_source` (fp32 required for sin(2π·cumsum(f0)) numerical stability) - **~390 ms warm CoreML predict** per utterance (M-series, mixed CPU+ANE+GPU) - **RTFx ~9.4×** end-to-end (3.7 s of audio in ~390 ms) - **~13 s cold start** (Apple `anecompilerservice` compiles ANE-targeted graphs on first call; fully cached afterwards) - **Per-stage placement**: `text_encoder`/`duration_predictor`/`decoder_upsample` on CPU, `bert`/`ref_encoder`/`diffusion_unet`/`f0n_predictor`/`decoder_pre` on ANE, `har_source` on GPU ## Repository contents ``` packages/ 9 mlpackages (258 MB) text_encoder_fp16.mlpackage 11 MB text → 512-dim embedding (LSTM, RangeDim T) bert_fp16.mlpackage 12 MB Albert + bert_encoder (fixed T=57) ref_encoder_fp16.mlpackage 53 MB reference mel → 256-dim style (CNN) diffusion_unet_fp16.mlpackage 48 MB cross-attention U-Net (fixed T=57; ADPM2 sampler) duration_predictor_fp16.mlpackage 15 MB LSTM + duration logits (RangeDim T) f0n_predictor_fp16.mlpackage 16 MB F0 + noise prediction (RangeDim F) har_source.mlpackage 12 KB F0 → harmonic source (RangeDim F0_LEN, fp32) decoder_pre_fp16.mlpackage 64 MB AdaIN encode/decode + F0/N convs (RangeDim F) decoder_upsample_fp16.mlpackage 40 MB HiFi-GAN Generator (RangeDim F→audio) voices/ 17 reference clips (4 MB) Yinghao.wav, Nima.wav, Gavin.wav, Vinay.wav Identity speakers amused.wav, anger.wav, disgusted.wav, sleepy.wav Emotion clips *.wav LibriTTS samples samples/ End-to-end synthesis samples sample_swift.wav Produced by the Swift CoreML driver sample_python.wav Produced by the Python CoreML pipeline manifest.json Machine-readable spec for all stages README.md This file ``` ## Limits - **Phoneme cap: 57.** `bert` and `diffusion_unet` are pinned to a fixed token axis of 57 because the CoreML CPU MLProgram backend rejects RangeDim on their cross-attention shape ops. Inputs that phonemize to >57 tokens will fail. The other 7 stages support flexible token (1-512) and frame (1-2048) axes. - **ANE compile fails** for the HiFi-GAN ConvTranspose1d ups stack inside `decoder_upsample`. CPU is the most predictable placement; GPU has slightly lower warm latency but contends with `har_source`. - **Apple Silicon recommended.** Intel Macs have not been validated for CoreML mlprogram inference at scale. ## Pipeline (per utterance) ``` text → espeak-ng IPA → tokenize → token_ids │ ┌───────────────────────────────┼──────────────────────────────────┐ │ │ │ ▼ ▼ ▼ text_encoder bert (fixed T=57) reference WAV → mel → ref_encoder t_en [1,512,T] bert_dur [1,57,768] ref_s [1,256] d_en [1,512,57] │ ▼ diffusion_unet × 5 ADPM2 steps (10 dispatches) │ ▼ s_pred [1,256] ↓ blend(α, β, ref_s) ↓ ref [1,128] s [1,128] │ ▼ duration_predictor d [1,T,640] pred_dur → pred_aln_trg │ ▼ (matmul + hifigan tail-shift) en [1,640,F] asr [1,512,F] │ ▼ f0n_predictor f0_pred, n_pred [1, 2F] │ ▼ har_source har [1,1,600F] │ ▼ decoder_pre x_pre [1,512,2F] │ ▼ decoder_upsample audio [1,1,72k+] │ ▼ tail-trim 50 samples → WAV @ 24 kHz ``` The 5 non-CoreML steps (espeak phonemize, ADPM2 sampler loop, mel extraction, alignment matrix, tail-shift) run host-side. See `manifest.json#non_coreml_pipeline_steps` for exact specs. ## Voices `voices/*.wav` are zero-shot reference clips. The `ref_encoder` stage reads a mel of the chosen reference and produces a 256-dim style embedding that conditions every downstream stage. Bring your own clip — any 3-10 s mono recording at any sample rate works (resampled to 24 kHz internally). Quality is sensitive to reference cleanliness (background noise transfers). ## Quick demo (Swift) A self-contained Swift demo exists that drives the last 4 stages directly from CoreML, given pre-computed inputs from the Python preprocessor. End-to-end Swift synthesis (no Python) requires porting espeak phonemize + mel + ADPM2 sampler + alignment, ~600 lines of Swift on top of these packages. ## Quick demo (Python) ```bash git clone https://github.com/yl4579/StyleTTS2 # for the espeak/text frontend + checkpoint config # Place this repo's packages/ as coreml/packages/ in StyleTTS2 working tree. uv run python coreml/inference.py \ --text "StyleTTS 2 is a text to speech model." \ --reference voices/Yinghao.wav \ --output out.wav ``` ## Conversion notes - Source: PyTorch StyleTTS2 LibriTTS 2nd-stage checkpoint (yl4579/StyleTTS2 epoch 20). - coremltools mlprogram, deployment target macOS15, fp16 compute precision. - Mixed-precision: 7 stages fp16, 1 stage fp32 (`har_source`), 1 stage split for ANE compatibility (`decoder` → `decoder_pre` + `decoder_upsample`). - Trace parity: all 9 stages mse=0 against eager PyTorch on the trace input. - Quantization trials (linear int8, 8-bit k-means palettization) tested on `decoder_upsample`; both rejected — int8 is slower than fp16 on CPU (no native ConvTranspose1d kernel) and lossy quality (19 dB SNR) for palettization. fp16 is the production setting. ## License MIT (matches upstream yl4579/StyleTTS2). LibriTTS reference clips inherit their LibriTTS / Apache-2.0 licensing. ## Citation If you use this port, please cite the original StyleTTS2 paper: ```bibtex @article{li2023styletts, title={StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models}, author={Li, Yinghao Aaron and Han, Cong and Raghavan, Vinay and Mischler, Gavin and Mesgarani, Nima}, journal={arXiv preprint arXiv:2306.07691}, year={2023} } ```