File size: 4,872 Bytes
e845c9c | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 | ---
library_name: coremltools
base_model:
- ResembleAI/chatterbox-flash
language:
- en
license: mit
tags:
- coreml
- text-to-speech
- voice-cloning
- zero-shot-tts
- block-diffusion
- speech-synthesis
- apple-silicon
- ios
- macos
pipeline_tag: text-to-speech
---
# Chatterbox Flash CoreML
Compiled Core ML export of [ResembleAI/chatterbox-flash](https://huggingface.co/ResembleAI/chatterbox-flash) for Apple runtimes.
This bundle contains the Chatterbox Flash T3 token generator and the S3Gen audio back-half as static-shape `.mlmodelc` graphs. It is intended for an application runtime that owns text tokenization, T3 denoising/sampling, reference conditioning, and graph orchestration.
## Links
- [speech-swift](https://github.com/soniqo/speech-swift) — Apple SDK
- [Speech Studio](https://soniqo.audio/speech-studio) — local speech generation and voice cloning app
- [Docs](https://soniqo.audio/getting-started) — install and CLI docs
- [soniqo.audio](https://soniqo.audio) — website
- [blog](https://soniqo.audio/blog) — blog
## Model
| Component | Parameters | Format | Precision | Static shape |
|---|---:|---|---|---|
| T3 block-diffusion token generator | 532.4M | Core ML `.mlmodelc` | fp16 | text_len 256, block_size 16, max_seq 1024 |
| S3Gen audio back-half | 266.0M | Core ML `.mlmodelc` | fp16 | token_len 192, mel_len 384 |
| Total | 798.4M | Core ML `.mlmodelc` | fp16 | 24 kHz waveform output |
## Files
| Path | Size | Description |
|---|---:|---|
| `config.json` | 4 KB | Root metadata for download tracking and runtime discovery |
| `t3/ConditioningEncoder.mlmodelc` | 25 MB | Speaker/prompt/emotion conditioning to T3 conditioning embedding |
| `t3/TextPrefill.mlmodelc` | 963 MB | Causal `[cond, text, start_speech]` prefix prefill and flat KV cache |
| `t3/BlockDecoder.mlmodelc` | 1.0 GB | Full-visible Flash speech-block logits with explicit KV cache |
| `t3/uncond_block_prior.npy` | 36 KB | Unconditional PMI prior for Flash scoring |
| `t3/tokenizer.json` | 28 KB | Chatterbox Flash text tokenizer |
| `t3/config.json` | 4 KB | T3 export metadata |
| `audio/FlowSpeakerProjector.mlmodelc` | 44 KB | S3Gen reference embedding to projected speaker conditioning |
| `audio/FlowEncoder.mlmodelc` | 79 MB | `prompt_token ++ speech_tokens` to flow `mu` and mask |
| `audio/FlowEstimator.mlmodelc` | 141 MB | One meanflow Euler derivative step |
| `audio/HiFTVocoder.mlmodelc` | 41 MB | Mel frames to 24 kHz waveform |
| `audio/audio_config.json` | 4 KB | S3Gen audio export metadata |
## Runtime Boundary
The exported graphs cover:
- T3 conditioning, text prefill, and block decoding.
- S3Gen speaker projection, flow encoder, meanflow estimator step, and HiFT vocoder.
The host runtime must still provide:
- text normalization and tokenization
- T3 denoising loop, PMI/CFG scoring, sampling, and EOS trimming
- reference waveform encoders:
- VoiceEncoder: `ref.wav -> speaker_emb`
- prompt speech tokenizer: `ref.wav -> prompt_speech_tokens`
- S3Gen reference encoder: `ref.wav -> prompt_token`, `prompt_feat`, `embedding`
- padding/cropping and the two-step meanflow loop over `(t,r) = (0,0.5), (0.5,1.0)`
This means the bundle supports voice-cloning TTS when the runtime supplies reference conditioning tensors, but it is not yet a fully Core ML `ref.wav -> cloned wav` pipeline.
## Validation
| Test | Result |
|---|---|
| T3 graph roundtrip vs PyTorch wrappers | Pass at 2% relative tolerance |
| Audio graph roundtrip vs PyTorch wrappers | Pass for token_len 192, mel_len 384 |
| Stitched S3Gen meanflow-to-audio roundtrip | Pass |
| Prompted synthesis smoke test | Pass |
| Whisper tiny transcript of generated wav | `Core ML speech test.` |
| Smoke-test WER | 0.000 |
Core ML warnings from local export:
- `CPU_ONLY` prediction crashed for the T3 package in local coremltools 8.3 testing. Use `ALL`, `CPU_AND_NE`, or a compiled-device runtime.
- The uploaded artifact ships compiled `.mlmodelc` folders. The numerical parity tests were run against the source `.mlpackage` exports before compilation.
## Usage Sketch
The runtime loads graphs from `t3/` and `audio/`, then:
1. Prepare T3 reference conditioning tensors and S3Gen `ref_dict` tensors from a prompt wav.
2. Tokenize text with `t3/tokenizer.json`.
3. Run T3 prefill and block denoising/sampling to produce S3 speech tokens.
4. Concatenate S3Gen `prompt_token` and generated speech tokens.
5. Run `audio/FlowEncoder.mlmodelc`.
6. Build `cond` by copying `prompt_feat.T` into the mel prefix.
7. Run `audio/FlowEstimator.mlmodelc` twice for `(0,0.5)` and `(0.5,1.0)`.
8. Crop generated mel frames after the prompt prefix.
9. Run `audio/HiFTVocoder.mlmodelc` and crop padded samples.
## Source
Converted from [ResembleAI/chatterbox-flash](https://huggingface.co/ResembleAI/chatterbox-flash), revision `4385507288b8197e6dab8b4e6b1603328d549d9d`.
|