ZONOS3-Mini B70 Turbo

One Intel Arc Pro B70 serves 20 live voices with no stalls with Zyphra's ZONOS3-Mini, where the stock runtime serves 8. Same model files, same voice quality.

This repo is two patches for Zyphra's zonos3.cpp runtime (Vulkan backend). The model weights are Zyphra's own GGUFs, downloaded from their repo. All credit for the model goes to Zyphra.

How to run

You need git, CMake, a C++ compiler, a Rust toolchain (cargo), and the Vulkan development files plus glslc (the LunarG Vulkan SDK, or your distro's packages such as libvulkan-dev and glslc).

git clone https://github.com/Zyphra/ZONOS3-Mini.git && cd ZONOS3-Mini
git checkout a50e14117d99181ba7f28047b451ba03ba1a09a9
git submodule update --init servers/ggml/ggml third_party/zyphra-textnorm
hf download Frosty40/ZONOS3-Mini-B70-Turbo --include "patches/*" --local-dir b70-turbo
cp b70-turbo/patches/vulkan-b70-turbo.patch servers/ggml/patches/   # applied by CMake at configure time
git apply b70-turbo/patches/zonos3-b70-turbo.patch

cd servers/ggml
cmake -B build -DGGML_VULKAN=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build -j
hf download Zyphra/ZONOS3-Mini gguf/backbone-ptq.gguf gguf/zac-keep2.gguf gguf/spk-encoder.gguf gguf/steer.gguf --local-dir zonos3

./build/zonos3-server zonos3/gguf/backbone-ptq.gguf --gpu --batch 32 --port 1919
curl -fsS http://127.0.0.1:1919/v2/generate \
  -d '{"text":"Hello from one Intel Arc B70.","stream":false,"format":"wav","seed":1}' -o hello.wav

The server speaks Zyphra's HTTP API, including streaming and the OpenAI-compatible /v1/audio/speech. The first launch on a machine compiles GPU shaders (about 40 seconds); later launches start in about two seconds.

Results

One Arc Pro B70 (Mesa 26.2.6, Vulkan SDK 1.4.363.1, Linux 7.0), Ryzen 9 5950X. Zyphra's server with --batch 32, streaming, zac-keep2 decoder, a speaker embedding on every request. Stock and turbo use the same zonos3.cpp commit (a50e141), the same model files and the same prompts, and ran alternately in one session (stock, turbo, stock, turbo), two rounds each. All voices start at once.

A stall is a gap in playback for a listener whose player keeps 200 ms of audio buffered.

Voices at once Stock: longest stall (4 rounds) Turbo: longest stall (4 rounds)
8 none none
12 321 ms none
16 489 ms none
20 745 ms none
24 873 ms 58 ms
Stock Turbo
Live voices with no stalls in every round 8 20
Audio served at 16 voices, seconds per second (mean of 4 rounds) 18.3 39.4
Time to first audio, one voice (median of 10) 43 ms 28 ms
Word error rate, 48 takes at 16 voices 0.0% 0.0%

Speech is checked by transcribing every take with Whisper large-v3 (faster-whisper, int8) and comparing it with the prompt text after Whisper's English normalizer. The 12 prompts (prompts_heldout_v1.jsonl) were never used to tune the patches. Raw results and the scripts that produced them are in results/ and scripts/.

What the patches change

  • vulkan-b70-turbo.patch (ggml Vulkan backend):
    • Batches of 16, 20, 24 and 32 voices stay on the fast matrix-vector kernels.
    • Each mixture-of-experts layer runs as one GPU dispatch for the whole batch.
    • Attention for many voices runs in one dispatch and sums in the same order as for a single voice.
  • zonos3-b70-turbo.patch (runtime):
    • The audio decoder reuses its GPU graphs from chunk to chunk instead of rebuilding them, and uploads each chunk's inputs in one transfer.
    • The 44.1 kHz resampler skips the filter taps that are zero and runs vectorized on Linux.
    • The language model reads every voice's output in one transfer.
    • Top-k sampling finds its candidates faster and draws the same tokens from the same scores.

Licence

The patches are MIT, the licence of the zonos3.cpp runtime and ggml that they modify. The model is Apache 2.0 from Zyphra and is downloaded from their repo, not redistributed here. Voice cloning should only be used with the voice owner's consent.

Built by OmniSpace Technologies.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support