| --- |
| license: other |
| license_name: neutts-open-license-v1-0 |
| license_link: LICENSE |
| base_model: neuphonic/neutts-2e |
| pipeline_tag: text-to-speech |
| language: |
| - en |
| tags: |
| - coreml |
| - apple |
| - tts |
| - emotional-tts |
| - qwen3 |
| - neucodec |
| - ane |
| --- |
| |
| # NeuTTS-2E CoreML |
|
|
| CoreML conversion of [neuphonic/neutts-2e](https://huggingface.co/neuphonic/neutts-2e) |
| (emotional English TTS: Qwen3 236M backbone + [NeuCodec](https://huggingface.co/neuphonic/neucodec) |
| decoder) for on-device inference on Apple platforms. Converted by |
| [FluidInference](https://huggingface.co/FluidInference); conversion sources live in the |
| [mobius](https://github.com/FluidInference/mobius) repo under `models/tts/neutts-2e/coreml/`. |
|
|
| Six emotions plus neutral (`angry`, `disgusted`, `fearful`, `happy`, `sad`, |
| `surprised`, `neutral`) across four fixed speakers (`emily`, `paul`, `sophie`, |
| `steven`), 24 kHz output. |
|
|
| ## Files |
|
|
| | File | Role | Target | |
| |---|---|---| |
| | `LM-Prefill-T768-M2048-fp16.mlpackage` | prompt → last-position logits + KV cache | macOS 14+ / iOS 17+ | |
| | `LM-Decode-M2048-fp16.mlpackage` | per-token decode, pass-through KV | macOS 14+ / iOS 17+ | |
| | `LM-Decode-M2048-fp16-stateful.mlpackage` | per-token decode, `MLState` KV | macOS 15+ / iOS 18+ | |
| | `LM-Prefill-T768-M1024-fp16.mlpackage` | faster pair, 1024-token cap | macOS 14+ / iOS 17+ | |
| | `LM-Decode-M1024-fp16-stateful.mlpackage` | faster pair, 1024-token cap | macOS 15+ / iOS 18+ | |
| | `NeuCodec-Decoder-fp16.mlpackage` | speech codes → 24 kHz audio (flexible length 2–2000) | macOS 14+ / iOS 17+ | |
| | `samples/*.pt`, `samples/*.txt` | pre-encoded speaker reference codes + transcripts | — | |
|
|
| The M=1024 pair decodes ~23 % faster (7.0 ms/token vs 9.1 on M5 Pro) but caps |
| prompt+generation at 1024 tokens (≈11 s of audio after the `emily` prompt); |
| use the M=2048 pair for longer utterances. Tokenizer comes from the |
| [upstream repo](https://huggingface.co/neuphonic/neutts-2e). |
|
|
| ## Pipeline |
|
|
| ``` |
| text → tokenizer → [prefill] → logits + KV |
| ↓ top-k sampling loop (temp 1.0, k 50), 50 codes/s |
| [decode] → <|speech_N|> tokens until <|SPEECH_GENERATION_END|> |
| ↓ |
| [NeuCodec-Decoder] → 24 kHz waveform |
| ``` |
|
|
| Prompt layout (BPE, no phonemizer): |
| `<|TEXT_PROMPT_START|>{ref_text}[<|EMOTION|>]{text}<|TEXT_PROMPT_END|><|SPEECH_GENERATION_START|>{ref codes}`. |
| Compute-unit guidance: run the LM on GPU (`.all` / `.cpuAndGPU` — the ANE |
| rejects the decode graph), the codec on `.cpuAndNeuralEngine` (~2× faster than |
| GPU). For streaming, decode the codec in 82-frame windows with 25-frame stride |
| and linear overlap-add (upstream's scheme) — ~550 ms to first audio. |
|
|
| Note: upstream applies a [Perth](https://github.com/resemble-ai/perth) |
| watermark to generated audio in the host app; that postprocessing is not part |
| of these models — hosts should apply it themselves. |
|
|
| ## Parity & performance (M5 Pro) |
|
|
| - CoreML fp16 LM vs PyTorch fp32: argmax match; teacher-forced replay of a |
| 301-token reference keeps 98.7 % of tokens inside the top-50 sampling |
| support; codec SNR 40–47 dB vs PyTorch across lengths; end audio 41.5 dB |
| vs the PyTorch reference waveform. |
| - Decode 7.0–9.1 ms/token (109–143 tok/s vs 50 needed for real-time), |
| prefill 33–40 ms warm, codec 12.7–27× RT on ANE. |
| - Batch ≈ 2× real-time; streaming ≈ 1.6× RT with ~550 ms time-to-first-audio, |
| 0 % ASR round-trip WER on medium/long texts (parakeet-tdt-v3). |
|
|
| ## License |
|
|
| Inherits the upstream [NeuTTS Open License v1.0](LICENSE) (free research use |
| and limited commercial use; paid license required for large-revenue commercial |
| deployments — see LICENSE for exact terms). NeuCodec components follow the |
| [neuphonic/neucodec](https://huggingface.co/neuphonic/neucodec) terms. |
|
|
