README.md

---
license: other
license_name: fish-audio-research-license
license_link: https://huggingface.co/fishaudio/s2-pro/blob/1de9996b6be38b745688de084d87a5633f714e4e/LICENSE.md
library_name: fish-speech
pipeline_tag: text-to-speech
base_model: fishaudio/s2-pro
base_model_relation: quantized
language:
  - en
tags:
  - fish-speech
  - fish-audio
  - text-to-speech
  - voice-cloning
  - quantization
  - nvfp4
  - mxfp8
  - blackwell
  - sm120
  - english
  - v1
---

<h1 align="center">V1 · Fish Audio S2-Pro · NVFP4 Balanced</h1>

<p align="center"><strong>A complete Blackwell-ready S2-Pro download:</strong> mixed native NVFP4/MXFP8 transformer weights, BF16 codec, tokenizer, pinned Fish Speech source, API server, web UI, and reproducibility checks.</p>

<p align="center"><strong>Balanced V1 prioritizes useful English quantization fidelity and VRAM reduction. This is not yet an XPO3 release.</strong></p>

<p align="center"><strong>An XPO3 version is coming soon.</strong> Follow <a href="https://huggingface.co/ajh-code">ajh-code on Hugging Face</a> and <a href="https://arands.com">Arands.com</a> for release updates.</p>

<p align="center">
  <a href="https://huggingface.co/fishaudio/s2-pro">Original S2-Pro</a>
  · <a href="https://github.com/fishaudio/fish-speech">Fish Speech</a>
  · <a href="https://arands.com">Arands.com · updates</a>
</p>

---

## Download

| Component | Purpose | Size |
|:--|:--|--:|
| `model-*.safetensors` | Complete mixed NVFP4/MXFP8 transformer checkpoint | 4.90 GB |
| `codec.pth` | Complete BF16 S2-Pro DAC codec | 1.87 GB |
| Tokenizer, runtime, and pinned source | No separate base-model or codec download | ~24 MB |
| Complete repository | Weights, codec, runtime, source, and metadata | 6.80 GB |

All model and codec weights required by the server are in this repository.
The root `config.json` preserves the S2-Pro architecture metadata and adds the
mixed-precision policy, while Hugging Face metadata records this repository as
a quantization of [`fishaudio/s2-pro`](https://huggingface.co/fishaudio/s2-pro).

## Quick start

Tested on Linux x86-64, Python 3.12, CUDA 13.0, PyTorch `2.11.0+cu130`,
`comfy-kitchen==0.2.22`, and NVIDIA Blackwell SM120. The current native path is
for GeForce RTX 50-series/SM120 GPUs; it is not a generic CUDA fallback.

```bash
hf download ajh-code/Fish-Audio-S2-Pro-NVFP4-Balanced \
  --local-dir fish-audio-s2-pro-nvfp4-balanced
cd fish-audio-s2-pro-nvfp4-balanced
./install.sh
./launch.sh
```

Open <http://127.0.0.1:8080/ui> for the bundled zero-shot web interface. The
API listens on all interfaces by default; set `TTS_HOST=127.0.0.1` if it should
not be reachable from the local network. Protect or firewall the service before
exposing it beyond a trusted network.

Docker Compose is the recommended clean deployment when Docker, the NVIDIA
Container Toolkit, and a compatible driver are already configured:

```bash
docker compose up --build
```

This path passed a clean outer-Docker build, SM120 runtime launch, route/UI
checks, and a zero-shot API smoke test on an RTX 5080 with CUDA 13.0.

The image excludes the 6.8 GB model payload and mounts the downloaded repository
read-only, so rebuilding the runtime does not duplicate the weights inside the
image.

## Zero-shot voice cloning

Use a clean, consented 10–30 second reference with one speaker and supply its
exact transcript:

```bash
python client.py \
  --url http://127.0.0.1:8080/v1/tts \
  --reference-audio reference.wav \
  --reference-text "The exact words spoken in reference.wav." \
  --text "A few notes as this story begins." \
  --seed 42 \
  --output result.wav
```

Equivalent JSON API call in Python:

```python
import base64
from pathlib import Path

import requests

payload = {
    "text": "A few notes as this story begins.",
    "references": [{
        "audio": base64.b64encode(Path("reference.wav").read_bytes()).decode(),
        "text": "The exact words spoken in reference.wav.",
    }],
    "reference_id": None,
    "format": "wav",
    "streaming": False,
    "normalize": True,
    "max_new_tokens": 1024,
    "chunk_length": 200,
    "top_p": 0.9,
    "temperature": 0.9,
    "repetition_penalty": 1.1,
    "seed": 42,
    "use_memory_cache": "off",
}
response = requests.post("http://127.0.0.1:8080/v1/tts", json=payload, timeout=600)
response.raise_for_status()
Path("result.wav").write_bytes(response.content)
```

Useful endpoints:

| Endpoint | Purpose |
|:--|:--|
| `GET /ui` | Bundled zero-shot web interface |
| `GET /v1/health` | Service health |
| `GET /v1/model` | Active release, quantization, and sampling metadata |
| `POST /v1/tts` | Fish Speech-compatible TTS request; returns audio |

Set `TTS_API_KEY` before launch to require bearer authentication. For an API
key named `secret`, send `Authorization: Bearer secret`.

## Quantization policy

S2-Pro has 180 projections in its 36-layer slow transformer. V1 uses:

| Scope | Stored/executed precision | Count |
|:--|:--|--:|
| Gate/up in layers 3–32 | packed NVFP4 E2M1; W4A16 at `M=1`, W4A4 above `M=1` | 60 |
| Other slow-transformer projections | native dynamic MXFP8 W8A8 | 120 |
| Embeddings, tied text output, fast transformer/output, norms, RoPE, KV cache, sampling | BF16/original precision | — |
| DAC codec arithmetic | BF16 | — |

The English calibration scale is folded into the selected norm and packed
gate/up tensors. It adds no runtime tensor or operation. The original BF16
slow-projection weights are not retained as a second copy. V1's transformer
checkpoint is 46.25% smaller than the original transformer shards.

This is intentionally described as a mixed NVFP4/MXFP8 checkpoint. It is not a
claim that every operation, activation, or weight in the end-to-end TTS stack
runs at FP4.

## Measured performance

Measurements below are local RTX 5080 results with the bundled compact BF16
codec path and a 3072-token cache. They are not universal performance claims.

| Measurement | V1 result |
|:--|--:|
| Loaded PyTorch allocation | 5.350 GiB |
| 15-sample short zero-shot peak | 5.607–5.769 GiB |
| Held-out ~39-second generation peak | 7.561 GiB |
| Median short/control real-time factor | about 1.05–1.06 |
| Median short/control time to first playable audio | about 5–6.5 s |
| Semantic generation throughput | about 20.4–20.6 frames/s |

The matching compact-runtime BF16 control loaded at 9.273 GiB, so V1 reduced
loaded PyTorch allocation by 42.31%. V1 is near real time on the RTX 5080, but
the current ordinary single-speaker API waits for a complete semantic segment
before playable audio. This release therefore does **not** claim agent-grade
low-latency streaming. An RTX 5060 Ti focused zero-shot run measured about 2.06
RTF and is not a real-time path.

## Limited blind voice-cloning test

The first blind comparison is encouraging, but deliberately small. It used one
listener, one consented English reference speaker, and four matched BF16/V1
pairs: conversational, reflective, question-shaped, and long narrative prompts
at seeds 7, 17, 123, and 42. Both models used the same reference, runtime path,
`temperature=0.9`, `top_p=0.9`, and `top_k=30`.

| Blind result | BF16 | Balanced V1 |
|:--|--:|--:|
| Speaker-likeness scores | all four 5/5 | all four 5/5 |
| Mean reference-style likeness | 4.50 / 5 | 4.50 / 5 |
| Mean naturalness | 3.75 / 5 | 3.75 / 5 |
| Pair preference | 1 | 2 |

The fourth pair was tied. The only severe artifact reported in the set was a
deterministic BF16 pitch squeak in the long seed-42 sample; its V1 counterpart
did not contain that excursion.

This test suggests that the quant did not cause a detectable speaker-identity
loss for that reference. It is **not** a general MOS study or broad cloning
qualification: more listeners, speakers, accents, recording conditions, and
languages are still needed. The release therefore reports the result without
claiming parity in every voice-cloning setting.

## Validated scope

| Gate | Result |
|:--|:--|
| Native execution | 60 NVFP4 and 120 MXFP8 projections execute through native SM120 paths |
| Standalone packaging | Fresh load from these shards, without BF16 source projections, matched a frozen 64-frame code canary bit exactly |
| English automated gates | Passed fixed-input signal/spectral, ASR, speaker-embedding, short/control, and held-out long-termination screens |
| Blind English clone identity | Limited four-pair test above: every BF16 and V1 sample scored 5/5 speaker likeness; preferences were V1 2, BF16 1, tie 1 |
| Multilingual | Not qualified; use an MXFP8 or BF16 model when language coverage matters |
| Hardware | NVIDIA Blackwell SM120 only in V1 |

The blind result supports quantization fidelity for that English reference; it
does not establish universal cloning quality across voices, recording
conditions, accents, or languages.

## Known limitations

- Fish S2-Pro itself sounded substantially flatter and less expressive than
  VoxCPM2 in our reference comparison. BF16 shared this behavior, so V1 does
  not treat it as NVFP4-specific damage and does not claim to fix it.
- Inline emotion/style instructions change output trajectories but did not
  reliably repair the perceived flatness in the tested voice.
- The objective speaker embedding saturated near 0.99 and failed to predict
  human preference; human listening remains required for new voices.
- One matched long BF16 sample produced a deterministic pitch squeak while its
  V1 counterpart did not. This is evidence from one seed, not a claim that V1
  is generally more artifact-free than BF16.
- Long-form peak memory is materially higher than loaded memory. Do not market
  V1 as a sub-6-GiB operational model for arbitrary request lengths.

These bounded claims are why this package is **Balanced V1**, not an XPO3
speed/quality/size release. Follow
[`ajh-code`](https://huggingface.co/ajh-code) for the upcoming XPO3 version.

## Validate the download

```bash
python validate_release.py
```

`MANIFEST.json` records the byte size and SHA-256 of every distributed file
except itself. Validation also checks the safetensors index/header mapping,
the 60/120 NVFP4/MXFP8 tensor counts, source pins, license/notice files, and
runtime payload. Hashing the 6.8 GB package takes a little while.

For an additional hash check every time the service loads:

```bash
TTS_VERIFY_CHECKSUMS=1 ./launch.sh
```

## License and attribution

**Built with Fish Audio.** This derivative is governed by the
[Fish Audio Research License](./LICENSE.md). Research and non-commercial use
are permitted subject to its terms. **Commercial use requires a separate
written license from Fish Audio; no commercial rights are granted by this
repository.** See [`Notice`](./Notice) for the required attribution and exact
change statement, and [`THIRD_PARTY_NOTICES.md`](./THIRD_PARTY_NOTICES.md) for
runtime dependencies.

Use only voices and recordings you have the right and consent to use.