aufklarer/Chatterbox-Flash-CoreML

Chatterbox Flash CoreML

Compiled Core ML export of ResembleAI/chatterbox-flash for Apple runtimes.

This bundle contains the Chatterbox Flash T3 token generator and the S3Gen audio back-half as static-shape .mlmodelc graphs. It is intended for an application runtime that owns text tokenization, T3 denoising/sampling, reference conditioning, and graph orchestration.

Links

• speech-swift — Apple SDK
• Speech Studio — local speech generation and voice cloning app
• Docs — install and CLI docs
• soniqo.audio — website
• blog — blog

Model

Component	Parameters	Format	Precision	Static shape
T3 block-diffusion token generator	532.4M	Core ML .mlmodelc	fp16	text_len 256, block_size 16, max_seq 1024
S3Gen audio back-half	266.0M	Core ML .mlmodelc	fp16	token_len 192, mel_len 384
Total	798.4M	Core ML .mlmodelc	fp16	24 kHz waveform output

Files

Path	Size	Description
config.json	4 KB	Root metadata for download tracking and runtime discovery
t3/ConditioningEncoder.mlmodelc	25 MB	Speaker/prompt/emotion conditioning to T3 conditioning embedding
t3/TextPrefill.mlmodelc	963 MB	Causal [cond, text, start_speech] prefix prefill and flat KV cache
t3/BlockDecoder.mlmodelc	1.0 GB	Full-visible Flash speech-block logits with explicit KV cache
t3/uncond_block_prior.npy	36 KB	Unconditional PMI prior for Flash scoring
t3/tokenizer.json	28 KB	Chatterbox Flash text tokenizer
t3/config.json	4 KB	T3 export metadata
audio/FlowSpeakerProjector.mlmodelc	44 KB	S3Gen reference embedding to projected speaker conditioning
audio/FlowEncoder.mlmodelc	79 MB	prompt_token ++ speech_tokens to flow mu and mask
audio/FlowEstimator.mlmodelc	141 MB	One meanflow Euler derivative step
audio/HiFTVocoder.mlmodelc	41 MB	Mel frames to 24 kHz waveform
audio/audio_config.json	4 KB	S3Gen audio export metadata

Runtime Boundary

The exported graphs cover:

• T3 conditioning, text prefill, and block decoding.
• S3Gen speaker projection, flow encoder, meanflow estimator step, and HiFT vocoder.

The host runtime must still provide:

• text normalization and tokenization
• T3 denoising loop, PMI/CFG scoring, sampling, and EOS trimming
• reference waveform encoders:
  • VoiceEncoder: ref.wav -> speaker_emb
  • prompt speech tokenizer: ref.wav -> prompt_speech_tokens
  • S3Gen reference encoder: ref.wav -> prompt_token, prompt_feat, embedding
• padding/cropping and the two-step meanflow loop over (t,r) = (0,0.5), (0.5,1.0)

This means the bundle supports voice-cloning TTS when the runtime supplies reference conditioning tensors, but it is not yet a fully Core ML ref.wav -> cloned wav pipeline.

Validation

Test	Result
T3 graph roundtrip vs PyTorch wrappers	Pass at 2% relative tolerance
Audio graph roundtrip vs PyTorch wrappers	Pass for token_len 192, mel_len 384
Stitched S3Gen meanflow-to-audio roundtrip	Pass
Prompted synthesis smoke test	Pass
Whisper tiny transcript of generated wav	Core ML speech test.
Smoke-test WER	0.000

Core ML warnings from local export:

• CPU_ONLY prediction crashed for the T3 package in local coremltools 8.3 testing. Use ALL, CPU_AND_NE, or a compiled-device runtime.
• The uploaded artifact ships compiled .mlmodelc folders. The numerical parity tests were run against the source .mlpackage exports before compilation.

Usage Sketch

The runtime loads graphs from t3/ and audio/, then:

1. Prepare T3 reference conditioning tensors and S3Gen ref_dict tensors from a prompt wav.
2. Tokenize text with t3/tokenizer.json.
3. Run T3 prefill and block denoising/sampling to produce S3 speech tokens.
4. Concatenate S3Gen prompt_token and generated speech tokens.
5. Run audio/FlowEncoder.mlmodelc.
6. Build cond by copying prompt_feat.T into the mel prefix.
7. Run audio/FlowEstimator.mlmodelc twice for (0,0.5) and (0.5,1.0).
8. Crop generated mel frames after the prompt prefix.
9. Run audio/HiFTVocoder.mlmodelc and crop padded samples.

Source

Converted from ResembleAI/chatterbox-flash, revision 4385507288b8197e6dab8b4e6b1603328d549d9d.