| --- |
| library_name: coremltools |
| base_model: |
| - ResembleAI/chatterbox-flash |
| language: |
| - en |
| license: mit |
| tags: |
| - coreml |
| - text-to-speech |
| - voice-cloning |
| - zero-shot-tts |
| - block-diffusion |
| - speech-synthesis |
| - apple-silicon |
| - ios |
| - macos |
| pipeline_tag: text-to-speech |
| --- |
| |
| # Chatterbox Flash CoreML |
|
|
| Compiled Core ML export of [ResembleAI/chatterbox-flash](https://huggingface.co/ResembleAI/chatterbox-flash) for Apple runtimes. |
|
|
| This bundle contains the Chatterbox Flash T3 token generator and the S3Gen audio back-half as static-shape `.mlmodelc` graphs. It is intended for an application runtime that owns text tokenization, T3 denoising/sampling, reference conditioning, and graph orchestration. |
|
|
| ## Links |
|
|
| - [speech-swift](https://github.com/soniqo/speech-swift) — Apple SDK |
| - [Speech Studio](https://soniqo.audio/speech-studio) — local speech generation and voice cloning app |
| - [Docs](https://soniqo.audio/getting-started) — install and CLI docs |
| - [soniqo.audio](https://soniqo.audio) — website |
| - [blog](https://soniqo.audio/blog) — blog |
|
|
| ## Model |
|
|
| | Component | Parameters | Format | Precision | Static shape | |
| |---|---:|---|---|---| |
| | T3 block-diffusion token generator | 532.4M | Core ML `.mlmodelc` | fp16 | text_len 256, block_size 16, max_seq 1024 | |
| | S3Gen audio back-half | 266.0M | Core ML `.mlmodelc` | fp16 | token_len 192, mel_len 384 | |
| | Total | 798.4M | Core ML `.mlmodelc` | fp16 | 24 kHz waveform output | |
| |
| ## Files |
| |
| | Path | Size | Description | |
| |---|---:|---| |
| | `config.json` | 4 KB | Root metadata for download tracking and runtime discovery | |
| | `t3/ConditioningEncoder.mlmodelc` | 25 MB | Speaker/prompt/emotion conditioning to T3 conditioning embedding | |
| | `t3/TextPrefill.mlmodelc` | 963 MB | Causal `[cond, text, start_speech]` prefix prefill and flat KV cache | |
| | `t3/BlockDecoder.mlmodelc` | 1.0 GB | Full-visible Flash speech-block logits with explicit KV cache | |
| | `t3/uncond_block_prior.npy` | 36 KB | Unconditional PMI prior for Flash scoring | |
| | `t3/tokenizer.json` | 28 KB | Chatterbox Flash text tokenizer | |
| | `t3/config.json` | 4 KB | T3 export metadata | |
| | `audio/FlowSpeakerProjector.mlmodelc` | 44 KB | S3Gen reference embedding to projected speaker conditioning | |
| | `audio/FlowEncoder.mlmodelc` | 79 MB | `prompt_token ++ speech_tokens` to flow `mu` and mask | |
| | `audio/FlowEstimator.mlmodelc` | 141 MB | One meanflow Euler derivative step | |
| | `audio/HiFTVocoder.mlmodelc` | 41 MB | Mel frames to 24 kHz waveform | |
| | `audio/audio_config.json` | 4 KB | S3Gen audio export metadata | |
|
|
| ## Runtime Boundary |
|
|
| The exported graphs cover: |
|
|
| - T3 conditioning, text prefill, and block decoding. |
| - S3Gen speaker projection, flow encoder, meanflow estimator step, and HiFT vocoder. |
|
|
| The host runtime must still provide: |
|
|
| - text normalization and tokenization |
| - T3 denoising loop, PMI/CFG scoring, sampling, and EOS trimming |
| - reference waveform encoders: |
| - VoiceEncoder: `ref.wav -> speaker_emb` |
| - prompt speech tokenizer: `ref.wav -> prompt_speech_tokens` |
| - S3Gen reference encoder: `ref.wav -> prompt_token`, `prompt_feat`, `embedding` |
| - padding/cropping and the two-step meanflow loop over `(t,r) = (0,0.5), (0.5,1.0)` |
|
|
| This means the bundle supports voice-cloning TTS when the runtime supplies reference conditioning tensors, but it is not yet a fully Core ML `ref.wav -> cloned wav` pipeline. |
|
|
| ## Validation |
|
|
| | Test | Result | |
| |---|---| |
| | T3 graph roundtrip vs PyTorch wrappers | Pass at 2% relative tolerance | |
| | Audio graph roundtrip vs PyTorch wrappers | Pass for token_len 192, mel_len 384 | |
| | Stitched S3Gen meanflow-to-audio roundtrip | Pass | |
| | Prompted synthesis smoke test | Pass | |
| | Whisper tiny transcript of generated wav | `Core ML speech test.` | |
| | Smoke-test WER | 0.000 | |
|
|
| Core ML warnings from local export: |
|
|
| - `CPU_ONLY` prediction crashed for the T3 package in local coremltools 8.3 testing. Use `ALL`, `CPU_AND_NE`, or a compiled-device runtime. |
| - The uploaded artifact ships compiled `.mlmodelc` folders. The numerical parity tests were run against the source `.mlpackage` exports before compilation. |
|
|
| ## Usage Sketch |
|
|
| The runtime loads graphs from `t3/` and `audio/`, then: |
|
|
| 1. Prepare T3 reference conditioning tensors and S3Gen `ref_dict` tensors from a prompt wav. |
| 2. Tokenize text with `t3/tokenizer.json`. |
| 3. Run T3 prefill and block denoising/sampling to produce S3 speech tokens. |
| 4. Concatenate S3Gen `prompt_token` and generated speech tokens. |
| 5. Run `audio/FlowEncoder.mlmodelc`. |
| 6. Build `cond` by copying `prompt_feat.T` into the mel prefix. |
| 7. Run `audio/FlowEstimator.mlmodelc` twice for `(0,0.5)` and `(0.5,1.0)`. |
| 8. Crop generated mel frames after the prompt prefix. |
| 9. Run `audio/HiFTVocoder.mlmodelc` and crop padded samples. |
|
|
| ## Source |
|
|
| Converted from [ResembleAI/chatterbox-flash](https://huggingface.co/ResembleAI/chatterbox-flash), revision `4385507288b8197e6dab8b4e6b1603328d549d9d`. |
|
|
