README.md

---
license: mit
language:
- en
library_name: chatterbox-flash
pipeline_tag: text-to-speech
base_model: ResembleAI/chatterbox-flash
base_model_relation: quantized
inference: false
tags:
- text-to-speech
- zero-shot-tts
- voice-cloning
- chatterbox-flash
- nvfp4
- w4a16
- quantized
- nvidia
- blackwell
- flashinfer
- safetensors
---

# Chatterbox-Flash T3 W4A16 NVFP4

This is an unofficial community NVFP4 derivative of
[ResembleAI/chatterbox-flash](https://huggingface.co/ResembleAI/chatterbox-flash).
It is not affiliated with or endorsed by Resemble AI or NVIDIA.

The package replaces all 120 Chatterbox-Flash T3 transformer projection
operators with source-free NVFP4 weights. Weights use native FP4 (E2M1) with
E4M3 block scales; activations remain BF16, so the precise description is
**T3 all-projection W4A16 NVFP4**. Embeddings, norms, conditioning layers,
speech head, S3Gen, and the voice encoder remain at their upstream precision.
This is not a whole-pipeline 4-bit model.

The canonical artifact does not contain the original BF16 T3 projection
weights. Its custom loader constructs those parameters on the meta device,
prepares the canonical FP4 codes/scales directly for FlashInfer, and never
calls `nvfp4_quantize` at load time. Engine rebuilds reuse the model-attached
projection provider.

## What this release actually improves

The primary benefit is **much lower storage and VRAM with a modest speed gain**,
not a claim that NVFP4 makes the complete TTS pipeline dramatically faster.
S3Gen and several small T3 operations remain high precision, and some
irregular prefix projections are slower in FP4 than BF16.

On the tested RTX 5060 Ti, the recommended D16 quality profile delivered:

- **83.97% smaller T3 checkpoint**: 341 MB instead of 2.13 GB.
- **55.88% less complete model-asset storage/download** after adding the
  unchanged S3Gen, voice encoder, and tokenizer: 1.412 GB instead of 3.200 GB.
- **33.2% lower allocated VRAM immediately after full pipeline load**:
  1.444 GB instead of 2.162 GB.
- **18.82% lower maximum allocated VRAM in the natural eight-text suite**:
  5.215 GB instead of 6.424 GB. Fixed-work peak allocation was 34.09% lower.
- **1.0811x aggregate-RTF speedup** (7.50% lower duration-normalized latency)
  and **1.1131x valid-token T3 throughput**. The fixed-work T3 speedup was
  1.1260x; fixed whole-pipeline speedup was 1.0631x.

The D24 fast profile reaches 25.13% lower RTF than the D16 BF16 baseline, but
that is a combined block-size-plus-NVFP4 result. Against matched D24 BF16,
NVFP4 alone provides only a 1.0309x RTF speedup. This distinction matters when
deciding whether the model is useful for your deployment.

## How quality was evaluated

A Gemma Audio evaluator listened to eight English test utterances generated
from one zero-shot reference voice. It transcribed each output, scored
intelligibility, pronunciation, voice similarity, naturalness, and artifacts,
and flagged hard failures. BF16 and NVFP4 were also compared blindly in both
A/B orders to expose position bias.

- D16 BF16/NVFP4 means were **94.125/94.625**. Both produced exact transcripts
  for 8/8 texts with zero hard failures. After reversing A/B order, the
  per-case result resolved to seven ties and one consistent BF16 preference.
- D24 BF16/NVFP4 means were **95.0/93.125**. Both produced exact transcripts
  for 8/8 texts with zero hard failures. Order control resolved to six ties,
  one BF16 preference, and one NVFP4 preference.

This is encouraging automated evidence, not human MOS, a standard Seed-TTS
benchmark, proof of perceptual identity, or a multi-speaker evaluation. The
audio was intentionally not published because consent for public
redistribution of the reference voice was not established.

## Hardware and software

The accepted release configuration is intentionally narrow:

- Linux x86-64 and Python 3.12
- NVIDIA RTX 50-series SM120/SM121 (tested on RTX 5060 Ti)
- PyTorch 2.7.1+cu128 and CUDA 12.8 wheels
- FlashInfer 0.6.14 with `nvidia-cutlass-dsl` 4.5.2
- BF16 activations, FlashInfer CUDA graphs, user batch size 1

There is no CPU, MLX, Torch-SDPA, older-GPU, or non-NVFP4 fallback because the
BF16 projection sources are deliberately absent. RTX 5080 and SM121 are
allowed by the loader but have not yet been benchmarked. This is custom code,
not a Transformers `AutoModel` checkpoint or a standard Hub quantization
format, so the hosted inference widget is disabled.

`nvidia-cutlass-dsl` is installed under NVIDIA's separate software license.
Review [THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md) before installation.

## Install

Clone the model repository, then use the included lock. Plain `pip install`
does not honor the upstream Torch override or the official cu128 wheel source.

```bash
git clone https://huggingface.co/ajh-code/chatterbox-flash-t3-w4a16-nvfp4
cd chatterbox-flash-t3-w4a16-nvfp4

uv sync \
  --project runtime/chatterbox-flash \
  --locked \
  --extra flashinfer \
  --python 3.12
```

The lock resolves the exact validated critical stack. A dry verification is:

```bash
runtime/chatterbox-flash/.venv/bin/python - <<'PY'
import importlib.metadata as md
import flashinfer
import torch

assert torch.__version__ == "2.7.1+cu128"
assert torch.version.cuda == "12.8"
assert md.version("flashinfer-python") == "0.6.14"
assert md.version("nvidia-cutlass-dsl") == "4.5.2"
assert hasattr(flashinfer, "prepare_bf16_fp4_weights")
assert hasattr(flashinfer, "mm_bf16_fp4")
assert torch.cuda.get_device_capability() in {(12, 0), (12, 1)}
print(torch.cuda.get_device_name(), "is ready")
PY
```

## Zero-shot voice cloning

Use only reference audio that you have permission to clone.

```bash
runtime/chatterbox-flash/.venv/bin/python example.py \
  --prompt-wav your-reference.wav \
  --text "This is Chatterbox Flash running native NVFP4 projections." \
  --output output.wav \
  --block-size 16
```

The example downloads only the unchanged S3Gen, voice-encoder, and tokenizer
assets from the immutable base revision
`4385507288b8197e6dab8b4e6b1603328d549d9d`. It verifies their sizes and
SHA-256 fingerprints from `nvfp4_config.json`. It never downloads the original
2.13 GB `t3_flash.safetensors`.

The first decode for a new GPU/exact block shape invokes FlashInfer setup and
tuning. On the RTX 5060 Ti, a fresh process with the selected D16 tactic already
cached took 10.57 seconds for the first 1.28-second utterance versus 0.154
seconds for the same-process warm repeat. A completely missing tactic has taken
roughly 25 seconds in development. Autotune results are GPU-specific, so this
repository does not ship a universal cache.

## Profiles

`D=16` is the quality-default profile. `D=24` is an explicitly optional fast
profile: it is materially faster overall but had a slightly lower automated
mean score and less consistent pairwise preference. `D=32` was tested and
rejected after token-cap failures and an OOM in the natural suite; its cached
prior remains serialized for research provenance, but the release loader
refuses it.

| Profile | Intended use | NVFP4 aggregate RTF | NVFP4 T3 tok/s | Automated mean |
|---|---|---:|---:|---:|
| D16 | Quality default | 0.08725 | 365.4 | 94.625 |
| D24 | Optional fast | 0.07062 | 483.3 | 93.125 |

## RTX 5060 Ti results

The natural test is an eight-text, one-reference suite. Pipeline and T3 times
are sums of per-case medians; aggregate RTF divides summed pipeline time by
summed generated-audio duration. Memory is the maximum per-case median
PyTorch CUDA peak, not total `nvidia-smi` process memory.

| Profile | Comparison | RTF speedup | T3 throughput | Peak allocated reduction |
|---|---|---:|---:|---:|
| D16 | W4A16 vs matched BF16 | 1.0811x | 1.1131x | 18.82% / 1.209 GB |
| D24 | W4A16 vs matched BF16 | 1.0309x | 1.0450x | 21.57% / 1.410 GB |

A fixed capped-work control removes sampled-length attribution. D16 T3 is
`1.1260x` faster and D24 T3 is `1.0656x` faster; each reduces fixed-work peak
allocated memory by about 34%. D16 fixed whole-pipeline speedup is `1.0631x`.

Do not quote the larger raw natural-suite wall-time ratios as pure NVFP4
speedups: the BF16 and W4A16 models sampled different token counts. The
D24-versus-D16 speed difference is primarily the larger block size, not FP4
alone. Complete numeric records and caveats are in
[benchmark_results.json](benchmark_results.json).

Automated quality was close but is not proof of perceptual identity:

- D16 BF16/NVFP4 means were 94.125/94.625; both transcribed 8/8 exactly with
  zero hard failures. Order-controlled pairwise scoring resolved to seven ties
  and one BF16 win.
- D24 BF16/NVFP4 means were 95.0/93.125; both transcribed 8/8 exactly with zero
  hard failures. Order control resolved to six ties, one BF16 win, and one
  NVFP4 win.

This was not human MOS, a standard Seed-TTS evaluation, or a multi-speaker
release study.

## Artifact and load behavior

- `t3_nvfp4.safetensors`: 341,439,040 bytes, 447 tensors, SHA-256
  `c5ea2d1d1441f657b383d8dad54bbae3778447d675822fb15918cb99b3587f1e`
- 360 canonical NVFP4 tensors cover 30 layers × 4 fused projection kinds ×
  FP4 codes/E4M3 scales/FP32 alpha.
- 82 non-projection T3 tensors remain BF16; three fixed priors and two rotary
  buffers support source-free construction.
- Original T3 checkpoint: 2,129,657,904 bytes. The derivative T3 file is
  83.97% smaller.
- Derivative T3 plus the three unchanged downloaded base assets totals
  1,412,035,402 bytes versus 3,200,254,266 bytes upstream: 55.88% less model
  asset storage/download, excluding small runtime files and cache.

In the final source-free release smoke, fresh-process pipeline load with base
hash verification and warm JIT state took 2.675 seconds and settled at 1.444 GB
PyTorch allocated / 1.478 GB reserved before the first generation. It prepared
120 projections into 283,116,000 bytes, without materializing BF16 projection
weights or calling the quantizer. The short generation peaked at 2.548 GB
allocated. These are one-machine diagnostics; first-generation setup is
reported separately above.

The canonical D16 loader regression reproduced the accepted stripped runtime
bit-for-bit: token history, waveform tensor SHA-256, and saved WAV SHA-256 all
matched. A longer request also rebuilt the engine successfully while retaining
the same source-free projection provider.

## Limitations and responsible use

- Only RTX 5060 Ti / SM120, batch one, and the documented settings received the
  complete technical and small automated-audio gate.
- D16 is the only quality-default profile. D24 is optional; D32 is rejected.
- S3Gen and voice-encoder execution are not quantized by this artifact.
- Prefix projections can remain slower than BF16 at irregular shapes; the
  decode block projections provide the measured NVFP4 gain.
- Streaming, multi-request throughput, RTX 5080 tuning, and human multi-voice
  listening remain future work.
- Generated length can change after quantization, affecting latency, memory,
  pauses, and termination even when transcripts are correct.
- Voice cloning can enable impersonation or deception. Obtain the speaker's
  consent, disclose synthetic audio where appropriate, and comply with local
  law and platform policies.

No evaluation audio is included because the reference clip's redistribution
and voice-consent status was not established for a public model repository.

## Attribution and citation

Chatterbox-Flash model weights and vendored runtime code are Copyright (c) 2026
Resemble AI and licensed under MIT. This runtime is based on source commit
`74e05baa8ce574bf2cc571702391a21f1b0d48c5`; the derivative modifies the
FlashInfer projection path and adds the canonical NVFP4 loader. See
[THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md).

```bibtex
@misc{seo2026chatterboxflashpriorcalibratedblockdiffusion,
  title={Chatterbox-Flash: Prior-Calibrated Block Diffusion for Streaming Zero-Shot TTS},
  author={Deokjin Seo and Gangin Park and Kihyun Nam},
  year={2026},
  eprint={2605.30748},
  archivePrefix={arXiv},
  primaryClass={cs.SD},
  url={https://arxiv.org/abs/2605.30748}
}
```

## Repository files

- `t3_nvfp4.safetensors`: canonical source-free checkpoint
- `nvfp4_config.json`: complete schema, fingerprints, revisions, and profiles
- `runtime/chatterbox-flash/`: pinned installable derivative runtime and lock
- `example.py`: tested local-bundle zero-shot generation entry point
- `benchmark_results.json`: machine-readable release measurements
- `SHA256SUMS`: release integrity manifest