---
license: other
license_name: fish-audio-research-license
license_link: https://huggingface.co/fishaudio/s2-pro/blob/1de9996b6be38b745688de084d87a5633f714e4e/LICENSE.md
library_name: fish-speech
pipeline_tag: text-to-speech
base_model: fishaudio/s2-pro
base_model_relation: quantized
language:
- en
tags:
- fish-speech
- fish-audio
- text-to-speech
- voice-cloning
- quantization
- nvfp4
- mxfp8
- blackwell
- sm120
- english
- v1
---
V1 · Fish Audio S2-Pro · NVFP4 Balanced
A complete Blackwell-ready S2-Pro download: mixed native NVFP4/MXFP8 transformer weights, BF16 codec, tokenizer, pinned Fish Speech source, API server, web UI, and reproducibility checks.
Balanced V1 prioritizes useful English quantization fidelity and VRAM reduction. This is not yet an XPO3 release.
An XPO3 version is coming soon. Follow ajh-code on Hugging Face and Arands.com for release updates.
Original S2-Pro
· Fish Speech
· Arands.com · updates
---
## Download
| Component | Purpose | Size |
|:--|:--|--:|
| `model-*.safetensors` | Complete mixed NVFP4/MXFP8 transformer checkpoint | 4.90 GB |
| `codec.pth` | Complete BF16 S2-Pro DAC codec | 1.87 GB |
| Tokenizer, runtime, and pinned source | No separate base-model or codec download | ~24 MB |
| Complete repository | Weights, codec, runtime, source, and metadata | 6.80 GB |
All model and codec weights required by the server are in this repository.
The root `config.json` preserves the S2-Pro architecture metadata and adds the
mixed-precision policy, while Hugging Face metadata records this repository as
a quantization of [`fishaudio/s2-pro`](https://huggingface.co/fishaudio/s2-pro).
## Quick start
Tested on Linux x86-64, Python 3.12, CUDA 13.0, PyTorch `2.11.0+cu130`,
`comfy-kitchen==0.2.22`, and NVIDIA Blackwell SM120. The current native path is
for GeForce RTX 50-series/SM120 GPUs; it is not a generic CUDA fallback.
```bash
hf download ajh-code/Fish-Audio-S2-Pro-NVFP4-Balanced \
--local-dir fish-audio-s2-pro-nvfp4-balanced
cd fish-audio-s2-pro-nvfp4-balanced
./install.sh
./launch.sh
```
Open for the bundled zero-shot web interface. The
API listens on all interfaces by default; set `TTS_HOST=127.0.0.1` if it should
not be reachable from the local network. Protect or firewall the service before
exposing it beyond a trusted network.
Docker Compose is the recommended clean deployment when Docker, the NVIDIA
Container Toolkit, and a compatible driver are already configured:
```bash
docker compose up --build
```
This path passed a clean outer-Docker build, SM120 runtime launch, route/UI
checks, and a zero-shot API smoke test on an RTX 5080 with CUDA 13.0.
The image excludes the 6.8 GB model payload and mounts the downloaded repository
read-only, so rebuilding the runtime does not duplicate the weights inside the
image.
## Zero-shot voice cloning
Use a clean, consented 10–30 second reference with one speaker and supply its
exact transcript:
```bash
python client.py \
--url http://127.0.0.1:8080/v1/tts \
--reference-audio reference.wav \
--reference-text "The exact words spoken in reference.wav." \
--text "A few notes as this story begins." \
--seed 42 \
--output result.wav
```
Equivalent JSON API call in Python:
```python
import base64
from pathlib import Path
import requests
payload = {
"text": "A few notes as this story begins.",
"references": [{
"audio": base64.b64encode(Path("reference.wav").read_bytes()).decode(),
"text": "The exact words spoken in reference.wav.",
}],
"reference_id": None,
"format": "wav",
"streaming": False,
"normalize": True,
"max_new_tokens": 1024,
"chunk_length": 200,
"top_p": 0.9,
"temperature": 0.9,
"repetition_penalty": 1.1,
"seed": 42,
"use_memory_cache": "off",
}
response = requests.post("http://127.0.0.1:8080/v1/tts", json=payload, timeout=600)
response.raise_for_status()
Path("result.wav").write_bytes(response.content)
```
Useful endpoints:
| Endpoint | Purpose |
|:--|:--|
| `GET /ui` | Bundled zero-shot web interface |
| `GET /v1/health` | Service health |
| `GET /v1/model` | Active release, quantization, and sampling metadata |
| `POST /v1/tts` | Fish Speech-compatible TTS request; returns audio |
Set `TTS_API_KEY` before launch to require bearer authentication. For an API
key named `secret`, send `Authorization: Bearer secret`.
## Quantization policy
S2-Pro has 180 projections in its 36-layer slow transformer. V1 uses:
| Scope | Stored/executed precision | Count |
|:--|:--|--:|
| Gate/up in layers 3–32 | packed NVFP4 E2M1; W4A16 at `M=1`, W4A4 above `M=1` | 60 |
| Other slow-transformer projections | native dynamic MXFP8 W8A8 | 120 |
| Embeddings, tied text output, fast transformer/output, norms, RoPE, KV cache, sampling | BF16/original precision | — |
| DAC codec arithmetic | BF16 | — |
The English calibration scale is folded into the selected norm and packed
gate/up tensors. It adds no runtime tensor or operation. The original BF16
slow-projection weights are not retained as a second copy. V1's transformer
checkpoint is 46.25% smaller than the original transformer shards.
This is intentionally described as a mixed NVFP4/MXFP8 checkpoint. It is not a
claim that every operation, activation, or weight in the end-to-end TTS stack
runs at FP4.
## Measured performance
Measurements below are local RTX 5080 results with the bundled compact BF16
codec path and a 3072-token cache. They are not universal performance claims.
| Measurement | V1 result |
|:--|--:|
| Loaded PyTorch allocation | 5.350 GiB |
| 15-sample short zero-shot peak | 5.607–5.769 GiB |
| Held-out ~39-second generation peak | 7.561 GiB |
| Median short/control real-time factor | about 1.05–1.06 |
| Median short/control time to first playable audio | about 5–6.5 s |
| Semantic generation throughput | about 20.4–20.6 frames/s |
The matching compact-runtime BF16 control loaded at 9.273 GiB, so V1 reduced
loaded PyTorch allocation by 42.31%. V1 is near real time on the RTX 5080, but
the current ordinary single-speaker API waits for a complete semantic segment
before playable audio. This release therefore does **not** claim agent-grade
low-latency streaming. An RTX 5060 Ti focused zero-shot run measured about 2.06
RTF and is not a real-time path.
## Limited blind voice-cloning test
The first blind comparison is encouraging, but deliberately small. It used one
listener, one consented English reference speaker, and four matched BF16/V1
pairs: conversational, reflective, question-shaped, and long narrative prompts
at seeds 7, 17, 123, and 42. Both models used the same reference, runtime path,
`temperature=0.9`, `top_p=0.9`, and `top_k=30`.
| Blind result | BF16 | Balanced V1 |
|:--|--:|--:|
| Speaker-likeness scores | all four 5/5 | all four 5/5 |
| Mean reference-style likeness | 4.50 / 5 | 4.50 / 5 |
| Mean naturalness | 3.75 / 5 | 3.75 / 5 |
| Pair preference | 1 | 2 |
The fourth pair was tied. The only severe artifact reported in the set was a
deterministic BF16 pitch squeak in the long seed-42 sample; its V1 counterpart
did not contain that excursion.
This test suggests that the quant did not cause a detectable speaker-identity
loss for that reference. It is **not** a general MOS study or broad cloning
qualification: more listeners, speakers, accents, recording conditions, and
languages are still needed. The release therefore reports the result without
claiming parity in every voice-cloning setting.
## Validated scope
| Gate | Result |
|:--|:--|
| Native execution | 60 NVFP4 and 120 MXFP8 projections execute through native SM120 paths |
| Standalone packaging | Fresh load from these shards, without BF16 source projections, matched a frozen 64-frame code canary bit exactly |
| English automated gates | Passed fixed-input signal/spectral, ASR, speaker-embedding, short/control, and held-out long-termination screens |
| Blind English clone identity | Limited four-pair test above: every BF16 and V1 sample scored 5/5 speaker likeness; preferences were V1 2, BF16 1, tie 1 |
| Multilingual | Not qualified; use an MXFP8 or BF16 model when language coverage matters |
| Hardware | NVIDIA Blackwell SM120 only in V1 |
The blind result supports quantization fidelity for that English reference; it
does not establish universal cloning quality across voices, recording
conditions, accents, or languages.
## Known limitations
- Fish S2-Pro itself sounded substantially flatter and less expressive than
VoxCPM2 in our reference comparison. BF16 shared this behavior, so V1 does
not treat it as NVFP4-specific damage and does not claim to fix it.
- Inline emotion/style instructions change output trajectories but did not
reliably repair the perceived flatness in the tested voice.
- The objective speaker embedding saturated near 0.99 and failed to predict
human preference; human listening remains required for new voices.
- One matched long BF16 sample produced a deterministic pitch squeak while its
V1 counterpart did not. This is evidence from one seed, not a claim that V1
is generally more artifact-free than BF16.
- Long-form peak memory is materially higher than loaded memory. Do not market
V1 as a sub-6-GiB operational model for arbitrary request lengths.
These bounded claims are why this package is **Balanced V1**, not an XPO3
speed/quality/size release. Follow
[`ajh-code`](https://huggingface.co/ajh-code) for the upcoming XPO3 version.
## Validate the download
```bash
python validate_release.py
```
`MANIFEST.json` records the byte size and SHA-256 of every distributed file
except itself. Validation also checks the safetensors index/header mapping,
the 60/120 NVFP4/MXFP8 tensor counts, source pins, license/notice files, and
runtime payload. Hashing the 6.8 GB package takes a little while.
For an additional hash check every time the service loads:
```bash
TTS_VERIFY_CHECKSUMS=1 ./launch.sh
```
## License and attribution
**Built with Fish Audio.** This derivative is governed by the
[Fish Audio Research License](./LICENSE.md). Research and non-commercial use
are permitted subject to its terms. **Commercial use requires a separate
written license from Fish Audio; no commercial rights are granted by this
repository.** See [`Notice`](./Notice) for the required attribution and exact
change statement, and [`THIRD_PARTY_NOTICES.md`](./THIRD_PARTY_NOTICES.md) for
runtime dependencies.
Use only voices and recordings you have the right and consent to use.