File size: 3,785 Bytes
f4f945c | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 | ---
license: other
license_name: neutts-open-license-v1-0
license_link: LICENSE
base_model: neuphonic/neutts-2e
pipeline_tag: text-to-speech
language:
- en
tags:
- coreml
- apple
- tts
- emotional-tts
- qwen3
- neucodec
- ane
---
# NeuTTS-2E CoreML
CoreML conversion of [neuphonic/neutts-2e](https://huggingface.co/neuphonic/neutts-2e)
(emotional English TTS: Qwen3 236M backbone + [NeuCodec](https://huggingface.co/neuphonic/neucodec)
decoder) for on-device inference on Apple platforms. Converted by
[FluidInference](https://huggingface.co/FluidInference); conversion sources live in the
[mobius](https://github.com/FluidInference/mobius) repo under `models/tts/neutts-2e/coreml/`.
Six emotions plus neutral (`angry`, `disgusted`, `fearful`, `happy`, `sad`,
`surprised`, `neutral`) across four fixed speakers (`emily`, `paul`, `sophie`,
`steven`), 24 kHz output.
## Files
| File | Role | Target |
|---|---|---|
| `LM-Prefill-T768-M2048-fp16.mlpackage` | prompt → last-position logits + KV cache | macOS 14+ / iOS 17+ |
| `LM-Decode-M2048-fp16.mlpackage` | per-token decode, pass-through KV | macOS 14+ / iOS 17+ |
| `LM-Decode-M2048-fp16-stateful.mlpackage` | per-token decode, `MLState` KV | macOS 15+ / iOS 18+ |
| `LM-Prefill-T768-M1024-fp16.mlpackage` | faster pair, 1024-token cap | macOS 14+ / iOS 17+ |
| `LM-Decode-M1024-fp16-stateful.mlpackage` | faster pair, 1024-token cap | macOS 15+ / iOS 18+ |
| `NeuCodec-Decoder-fp16.mlpackage` | speech codes → 24 kHz audio (flexible length 2–2000) | macOS 14+ / iOS 17+ |
| `samples/*.pt`, `samples/*.txt` | pre-encoded speaker reference codes + transcripts | — |
The M=1024 pair decodes ~23 % faster (7.0 ms/token vs 9.1 on M5 Pro) but caps
prompt+generation at 1024 tokens (≈11 s of audio after the `emily` prompt);
use the M=2048 pair for longer utterances. Tokenizer comes from the
[upstream repo](https://huggingface.co/neuphonic/neutts-2e).
## Pipeline
```
text → tokenizer → [prefill] → logits + KV
↓ top-k sampling loop (temp 1.0, k 50), 50 codes/s
[decode] → <|speech_N|> tokens until <|SPEECH_GENERATION_END|>
↓
[NeuCodec-Decoder] → 24 kHz waveform
```
Prompt layout (BPE, no phonemizer):
`<|TEXT_PROMPT_START|>{ref_text}[<|EMOTION|>]{text}<|TEXT_PROMPT_END|><|SPEECH_GENERATION_START|>{ref codes}`.
Compute-unit guidance: run the LM on GPU (`.all` / `.cpuAndGPU` — the ANE
rejects the decode graph), the codec on `.cpuAndNeuralEngine` (~2× faster than
GPU). For streaming, decode the codec in 82-frame windows with 25-frame stride
and linear overlap-add (upstream's scheme) — ~550 ms to first audio.
Note: upstream applies a [Perth](https://github.com/resemble-ai/perth)
watermark to generated audio in the host app; that postprocessing is not part
of these models — hosts should apply it themselves.
## Parity & performance (M5 Pro)
- CoreML fp16 LM vs PyTorch fp32: argmax match; teacher-forced replay of a
301-token reference keeps 98.7 % of tokens inside the top-50 sampling
support; codec SNR 40–47 dB vs PyTorch across lengths; end audio 41.5 dB
vs the PyTorch reference waveform.
- Decode 7.0–9.1 ms/token (109–143 tok/s vs 50 needed for real-time),
prefill 33–40 ms warm, codec 12.7–27× RT on ANE.
- Batch ≈ 2× real-time; streaming ≈ 1.6× RT with ~550 ms time-to-first-audio,
0 % ASR round-trip WER on medium/long texts (parakeet-tdt-v3).
## License
Inherits the upstream [NeuTTS Open License v1.0](LICENSE) (free research use
and limited commercial use; paid license required for large-revenue commercial
deployments — see LICENSE for exact terms). NeuCodec components follow the
[neuphonic/neucodec](https://huggingface.co/neuphonic/neucodec) terms.
|