Zonos (code, models, paper, wiki)

.gitattributes

@@ -33,3 +33,7 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+A[[:space:]]novel[[:space:]]semantic[[:space:]]compression[[:space:]]approach[[:space:]]for[[:space:]]ultra-low[[:space:]]bandwidth[[:space:]]voice[[:space:]]communication.pdf filter=lfs diff=lfs merge=lfs -text
+wiki/Beta[[:space:]]Release[[:space:]]of[[:space:]]Zonos[[:space:]]v0.1.pdf filter=lfs diff=lfs merge=lfs -text
+wiki/Zonos[[:space:]]v0.1.[[:space:]]Best[[:space:]]Open[[:space:]]Source[[:space:]]AI[[:space:]]Voice[[:space:]]Clone[[:space:]]Tool.pdf filter=lfs diff=lfs merge=lfs -text
+wiki/Zyphra[[:space:]]Just[[:space:]]Dropped[[:space:]]Zonos.[[:space:]]Instant[[:space:]]TTS[[:space:]]and[[:space:]]Voice[[:space:]]Cloning.[[:space:]]Install[[:space:]]Locally.mkv filter=lfs diff=lfs merge=lfs -text

A novel semantic compression approach for ultra-low bandwidth voice communication.pdf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a2e12f61c1c9c2724cea4cb0da132482b50a7c13db5efb1d8bc61fc6e9577c3c
+size 365385

code/ComfyUI-Zonos.zip

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:80ea42b0eee3d6b8783e4b66afd2514afb01af1c5489332b5611a839024f8a7a
+size 1535148

code/ComfyUI-ZonosTTS.zip

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:96e17c63e013d84a03d4a023078adb3725ca678d436d6d6e81b5564c0ec5b73c
+size 75203

code/Zonos.zip

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d29a36b793fb6a6c3e2ad271f6346d570d4cdca12e0b2b44c2dd6a76bba6a649
+size 4946826

code/cog-zonos.zip

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7ba31c5e4f5dec81a3c3e4ded9f78692a2d3ccc2541d745020168b26f609f2ea
+size 3510060

code/zonos-api.zip

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:53da88886bd99c65e192653dae86c32707e03161882f7158a4513eb955d5201e
+size 121146

models/Zonos-Hebrew/.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

models/Zonos-Hebrew/README.md

@@ -0,0 +1,160 @@
+---
+license: cc-by-nc-4.0
+pipeline_tag: text-to-speech
+library_name: zonos
+---
+# Zonos-v0.1-Hebrew
+
+<div align="center">
+<img src="https://github.com/Zyphra/Zonos/blob/main/assets/ZonosHeader.png?raw=true"
+     alt="Title card" 
+     style="width: 500px;
+            height: auto;
+            object-position: center top;">
+</div>
+
+---
+
+Zonos-v0.1 is a leading open-weight text-to-speech model trained on more than 200k hours of varied multilingual speech, delivering expressiveness and quality on par with—or even surpassing—top TTS providers.
+
+Our model enables highly natural speech generation from text prompts when given a speaker embedding or audio prefix, and can accurately perform speech cloning when given a reference clip spanning just a few seconds. The conditioning setup also allows for fine control over speaking rate, pitch variation, audio quality, and emotions such as happiness, fear, sadness, and anger. The model outputs speech natively at 44kHz.
+
+##### For more details and speech samples, check out our blog [here](https://www.zyphra.com/post/beta-release-of-zonos-v0-1)
+
+##### We also have a hosted version available at [playground.zyphra.com/audio](https://playground.zyphra.com/audio)
+
+---
+
+Zonos follows a straightforward architecture: text normalization and phonemization via eSpeak, followed by DAC token prediction through a transformer or hybrid backbone. An overview of the architecture can be seen below.
+
+<div align="center">
+<img src="https://github.com/Zyphra/Zonos/blob/main/assets/ArchitectureDiagram.png?raw=true"
+     alt="Architecture diagram" 
+     style="width: 1000px;
+            height: auto;
+            object-position: center top;">
+</div>
+
+---
+
+## Usage
+
+### Python
+
+```python
+import torch
+import torchaudio
+from zonos.model import Zonos
+from zonos.conditioning import make_cond_dict
+from zonos.utils import DEFAULT_DEVICE as device
+
+
+model = Zonos.from_pretrained("notmax123/Zonos-Hebrew", device=device)
+
+wav, sampling_rate = torchaudio.load("assets/exampleaudio.mp3")
+speaker = model.make_speaker_embedding(wav, sampling_rate)
+
+torch.manual_seed(421)
+
+cond_dict = make_cond_dict(text="ירושלים יום טוב", speaker=speaker, language="he")
+conditioning = model.prepare_conditioning(cond_dict)
+
+codes = model.generate(conditioning)
+
+wavs = model.autoencoder.decode(codes).cpu()
+torchaudio.save("sample.wav", wavs[0], model.autoencoder.sampling_rate)
+```
+
+### Gradio interface (recommended)
+
+```bash
+uv run gradio_interface.py
+# python gradio_interface.py
+```
+
+This should produce a `sample.wav` file in your project root directory.
+
+_For repeated sampling we highly recommend using the gradio interface instead, as the minimal example needs to load the model every time it is run._
+
+## Features
+
+- Zero-shot TTS with voice cloning: Input desired text and a 10-30s speaker sample to generate high quality TTS output
+- Audio prefix inputs: Add text plus an audio prefix for even richer speaker matching. Audio prefixes can be used to elicit behaviours such as whispering which can otherwise be challenging to replicate when cloning from speaker embeddings
+- Multilingual support: Zonos-v0.1 supports English, Japanese, Chinese, French, and German
+- Audio quality and emotion control: Zonos offers fine-grained control of many aspects of the generated audio. These include speaking rate, pitch, maximum frequency, audio quality, and various emotions such as happiness, anger, sadness, and fear.
+- Fast: our model runs with a real-time factor of ~2x on an RTX 4090
+- Gradio WebUI: Zonos comes packaged with an easy to use gradio interface to generate speech
+- Simple installation and deployment: Zonos can be installed and deployed simply using the docker file packaged with our repository.
+
+## Installation
+
+**At the moment this repository only supports Linux systems (preferably Ubuntu 22.04/24.04) with recent NVIDIA GPUs (3000-series or newer, 6GB+ VRAM).**
+
+See also [Docker Installation](#docker-installation)
+
+#### System dependencies
+
+Zonos depends on the eSpeak library phonemization. You can install it on Ubuntu with the following command:
+
+```bash
+apt install -y espeak-ng
+```
+
+#### Python dependencies
+
+We highly recommend using a recent version of [uv](https://docs.astral.sh/uv/#installation) for installation. If you don't have uv installed, you can install it via pip: `pip install -U uv`.
+
+##### Installing into a new uv virtual environment (recommended)
+
+```bash
+uv sync
+uv sync --extra compile
+```
+
+##### Installing into the system/actived environment using uv
+
+```bash
+uv pip install -e .
+uv pip install -e .[compile]
+```
+
+##### Installing into the system/actived environment using pip
+
+```bash
+pip install -e .
+pip install --no-build-isolation -e .[compile]
+```
+
+##### Confirm that it's working
+
+For convenience we provide a minimal example to check that the installation works:
+
+```bash
+uv run sample.py
+# python sample.py
+```
+
+## Docker installation
+
+```bash
+git clone https://github.com/Zyphra/Zonos.git
+cd Zonos
+
+# For gradio
+docker compose up
+
+# Or for development you can do
+docker build -t Zonos .
+docker run -it --gpus=all --net=host -v /path/to/Zonos:/Zonos -t Zonos
+cd /Zonos
+python sample.py # this will generate a sample.wav in /Zonos
+```
+
+## Citation
+If you find this model useful in an academic context please cite as:
+```bash
+@misc{zyphra2025zonos,
+  title     = {Zonos-v0.1: An Expressive, Open-Source TTS Model},
+  author    = {Dario Sucic, Mohamed Osman, Gabriel Clark, Chris Warner, Beren Millidge},
+  year      = {2025},
+}

models/Zonos-Hebrew/Zonos-v0.1-hybrid-Hebrew.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e4c486829764a262548ba25c1a7d9651811e5d79e523ba4499cf3943b1f55e32
+size 3303692816

models/Zonos-Hebrew/config.json

@@ -0,0 +1,117 @@
+{
+    "backbone": {
+        "d_model": 2048,
+        "d_intermediate": 0,
+        "attn_mlp_d_intermediate": 8192,
+        "n_layer": 46,
+        "ssm_cfg": {
+            "layer": "Mamba2"
+        },
+        "attn_layer_idx": [
+            0,
+            4,
+            8,
+            12,
+            16,
+            20,
+            24,
+            28,
+            32,
+            36,
+            40,
+            44
+        ],
+        "attn_cfg": {
+            "causal": true,
+            "num_heads": 16,
+            "num_heads_kv": 4,
+            "rotary_emb_dim": 128,
+            "qkv_proj_bias": false,
+            "out_proj_bias": false
+        },
+        "rms_norm": false,
+        "residual_in_fp32": false,
+        "norm_epsilon": 1e-05
+    },
+    "prefix_conditioner": {
+        "conditioners": [
+            {
+                "type": "EspeakPhonemeConditioner",
+                "name": "espeak"
+            },
+            {
+                "cond_dim": 128,
+                "uncond_type": "learned",
+                "projection": "linear",
+                "type": "PassthroughConditioner",
+                "name": "speaker"
+            },
+            {
+                "input_dim": 8,
+                "uncond_type": "learned",
+                "type": "FourierConditioner",
+                "name": "emotion"
+            },
+            {
+                "min_val": 0,
+                "max_val": 24000,
+                "uncond_type": "learned",
+                "type": "FourierConditioner",
+                "name": "fmax"
+            },
+            {
+                "min_val": 0,
+                "max_val": 400,
+                "uncond_type": "learned",
+                "type": "FourierConditioner",
+                "name": "pitch_std"
+            },
+            {
+                "min_val": 0,
+                "max_val": 40,
+                "uncond_type": "learned",
+                "type": "FourierConditioner",
+                "name": "speaking_rate"
+            },
+            {
+                "min_val": -1,
+                "max_val": 126,
+                "uncond_type": "learned",
+                "type": "IntegerConditioner",
+                "name": "language_id"
+            },
+            {
+                "input_dim": 8,
+                "min_val": 0.5,
+                "max_val": 0.8,
+                "uncond_type": "learned",
+                "type": "FourierConditioner",
+                "name": "vqscore_8"
+            },
+            {
+                "min_val": -1.0,
+                "max_val": 1000,
+                "uncond_type": "learned",
+                "type": "FourierConditioner",
+                "name": "ctc_loss"
+            },
+            {
+                "min_val": 1,
+                "max_val": 5,
+                "uncond_type": "learned",
+                "type": "FourierConditioner",
+                "name": "dnsmos_ovrl"
+            },
+            {
+                "min_val": 0,
+                "max_val": 1,
+                "uncond_type": "learned",
+                "type": "IntegerConditioner",
+                "name": "speaker_noised"
+            }
+        ],
+        "projection": "linear"
+    },
+    "eos_token_id": 1024,
+    "masked_token_id": 1025
+}

models/Zonos-Hebrew/phonikud-1.0.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c1fa2624b1e8202a0c0a23259b560b0c41ad92a3a6750bd0e322ce5a2b1acdb6
+size 307844158

models/Zonos-Hebrew/source.txt

@@ -0,0 +1 @@
+https://huggingface.co/notmax123/Zonos-Hebrew

models/Zonos-v0.1-hybrid/.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

models/Zonos-v0.1-hybrid/README.md

@@ -0,0 +1,157 @@
+---
+license: apache-2.0
+pipeline_tag: text-to-speech
+library_name: zonos
+---
+# Zonos-v0.1
+
+<div align="center">
+<img src="https://github.com/Zyphra/Zonos/blob/main/assets/ZonosHeader.png?raw=true"
+     alt="Title card" 
+     style="width: 500px;
+            height: auto;
+            object-position: center top;">
+</div>
+
+---
+
+Zonos-v0.1 is a leading open-weight text-to-speech model trained on more than 200k hours of varied multilingual speech, delivering expressiveness and quality on par with—or even surpassing—top TTS providers.
+
+Our model enables highly natural speech generation from text prompts when given a speaker embedding or audio prefix, and can accurately perform speech cloning when given a reference clip spanning just a few seconds. The conditioning setup also allows for fine control over speaking rate, pitch variation, audio quality, and emotions such as happiness, fear, sadness, and anger. The model outputs speech natively at 44kHz.
+
+##### For more details and speech samples, check out our blog [here](https://www.zyphra.com/post/beta-release-of-zonos-v0-1)
+
+##### We also have a hosted version available at [playground.zyphra.com/audio](https://playground.zyphra.com/audio)
+
+---
+
+Zonos follows a straightforward architecture: text normalization and phonemization via eSpeak, followed by DAC token prediction through a transformer or hybrid backbone. An overview of the architecture can be seen below.
+
+<div align="center">
+<img src="https://github.com/Zyphra/Zonos/blob/main/assets/ArchitectureDiagram.png?raw=true"
+     alt="Architecture diagram" 
+     style="width: 1000px;
+            height: auto;
+            object-position: center top;">
+</div>
+
+---
+
+## Usage
+
+### Python
+
+```python
+import torch
+import torchaudio
+from zonos.model import Zonos
+from zonos.conditioning import make_cond_dict
+
+model = Zonos.from_pretrained("Zyphra/Zonos-v0.1-hybrid", device="cuda")
+# model = Zonos.from_pretrained("Zyphra/Zonos-v0.1-transformer", device="cuda")
+
+wav, sampling_rate = torchaudio.load("assets/exampleaudio.mp3")
+speaker = model.make_speaker_embedding(wav, sampling_rate)
+
+cond_dict = make_cond_dict(text="Hello, world!", speaker=speaker, language="en-us")
+conditioning = model.prepare_conditioning(cond_dict)
+
+codes = model.generate(conditioning)
+
+wavs = model.autoencoder.decode(codes).cpu()
+torchaudio.save("sample.wav", wavs[0], model.autoencoder.sampling_rate)
+```
+
+### Gradio interface (recommended)
+
+```bash
+uv run gradio_interface.py
+# python gradio_interface.py
+```
+
+This should produce a `sample.wav` file in your project root directory.
+
+_For repeated sampling we highly recommend using the gradio interface instead, as the minimal example needs to load the model every time it is run._
+
+## Features
+
+- Zero-shot TTS with voice cloning: Input desired text and a 10-30s speaker sample to generate high quality TTS output
+- Audio prefix inputs: Add text plus an audio prefix for even richer speaker matching. Audio prefixes can be used to elicit behaviours such as whispering which can otherwise be challenging to replicate when cloning from speaker embeddings
+- Multilingual support: Zonos-v0.1 supports English, Japanese, Chinese, French, and German
+- Audio quality and emotion control: Zonos offers fine-grained control of many aspects of the generated audio. These include speaking rate, pitch, maximum frequency, audio quality, and various emotions such as happiness, anger, sadness, and fear.
+- Fast: our model runs with a real-time factor of ~2x on an RTX 4090
+- Gradio WebUI: Zonos comes packaged with an easy to use gradio interface to generate speech
+- Simple installation and deployment: Zonos can be installed and deployed simply using the docker file packaged with our repository.
+
+## Installation
+
+**At the moment this repository only supports Linux systems (preferably Ubuntu 22.04/24.04) with recent NVIDIA GPUs (3000-series or newer, 6GB+ VRAM).**
+
+See also [Docker Installation](#docker-installation)
+
+#### System dependencies
+
+Zonos depends on the eSpeak library phonemization. You can install it on Ubuntu with the following command:
+
+```bash
+apt install -y espeak-ng
+```
+
+#### Python dependencies
+
+We highly recommend using a recent version of [uv](https://docs.astral.sh/uv/#installation) for installation. If you don't have uv installed, you can install it via pip: `pip install -U uv`.
+
+##### Installing into a new uv virtual environment (recommended)
+
+```bash
+uv sync
+uv sync --extra compile
+```
+
+##### Installing into the system/actived environment using uv
+
+```bash
+uv pip install -e .
+uv pip install -e .[compile]
+```
+
+##### Installing into the system/actived environment using pip
+
+```bash
+pip install -e .
+pip install --no-build-isolation -e .[compile]
+```
+
+##### Confirm that it's working
+
+For convenience we provide a minimal example to check that the installation works:
+
+```bash
+uv run sample.py
+# python sample.py
+```
+
+## Docker installation
+
+```bash
+git clone https://github.com/Zyphra/Zonos.git
+cd Zonos
+
+# For gradio
+docker compose up
+
+# Or for development you can do
+docker build -t Zonos .
+docker run -it --gpus=all --net=host -v /path/to/Zonos:/Zonos -t Zonos
+cd /Zonos
+python sample.py # this will generate a sample.wav in /Zonos
+```
+
+## Citation
+If you find this model useful in an academic context please cite as:
+```bash
+@misc{zyphra2025zonos,
+  title     = {Zonos-v0.1: An Expressive, Open-Source TTS Model},
+  author    = {Dario Sucic, Mohamed Osman, Gabriel Clark, Chris Warner, Beren Millidge},
+  year      = {2025},
+}

models/Zonos-v0.1-hybrid/config.json

@@ -0,0 +1,117 @@
+{
+    "backbone": {
+        "d_model": 2048,
+        "d_intermediate": 0,
+        "attn_mlp_d_intermediate": 8192,
+        "n_layer": 46,
+        "ssm_cfg": {
+            "layer": "Mamba2"
+        },
+        "attn_layer_idx": [
+            0,
+            4,
+            8,
+            12,
+            16,
+            20,
+            24,
+            28,
+            32,
+            36,
+            40,
+            44
+        ],
+        "attn_cfg": {
+            "causal": true,
+            "num_heads": 16,
+            "num_heads_kv": 4,
+            "rotary_emb_dim": 128,
+            "qkv_proj_bias": false,
+            "out_proj_bias": false
+        },
+        "rms_norm": false,
+        "residual_in_fp32": false,
+        "norm_epsilon": 1e-05
+    },
+    "prefix_conditioner": {
+        "conditioners": [
+            {
+                "type": "EspeakPhonemeConditioner",
+                "name": "espeak"
+            },
+            {
+                "cond_dim": 128,
+                "uncond_type": "learned",
+                "projection": "linear",
+                "type": "PassthroughConditioner",
+                "name": "speaker"
+            },
+            {
+                "input_dim": 8,
+                "uncond_type": "learned",
+                "type": "FourierConditioner",
+                "name": "emotion"
+            },
+            {
+                "min_val": 0,
+                "max_val": 24000,
+                "uncond_type": "learned",
+                "type": "FourierConditioner",
+                "name": "fmax"
+            },
+            {
+                "min_val": 0,
+                "max_val": 400,
+                "uncond_type": "learned",
+                "type": "FourierConditioner",
+                "name": "pitch_std"
+            },
+            {
+                "min_val": 0,
+                "max_val": 40,
+                "uncond_type": "learned",
+                "type": "FourierConditioner",
+                "name": "speaking_rate"
+            },
+            {
+                "min_val": -1,
+                "max_val": 126,
+                "uncond_type": "learned",
+                "type": "IntegerConditioner",
+                "name": "language_id"
+            },
+            {
+                "input_dim": 8,
+                "min_val": 0.5,
+                "max_val": 0.8,
+                "uncond_type": "learned",
+                "type": "FourierConditioner",
+                "name": "vqscore_8"
+            },
+            {
+                "min_val": -1.0,
+                "max_val": 1000,
+                "uncond_type": "learned",
+                "type": "FourierConditioner",
+                "name": "ctc_loss"
+            },
+            {
+                "min_val": 1,
+                "max_val": 5,
+                "uncond_type": "learned",
+                "type": "FourierConditioner",
+                "name": "dnsmos_ovrl"
+            },
+            {
+                "min_val": 0,
+                "max_val": 1,
+                "uncond_type": "learned",
+                "type": "IntegerConditioner",
+                "name": "speaker_noised"
+            }
+        ],
+        "projection": "linear"
+    },
+    "eos_token_id": 1024,
+    "masked_token_id": 1025
+}

models/Zonos-v0.1-hybrid/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:04e02e5542dd6968ad79942b014e317623efc671ea54b57069d3016e7aa3cceb
+size 3303692816

models/Zonos-v0.1-hybrid/source.txt

@@ -0,0 +1 @@
+https://huggingface.co/Zyphra/Zonos-v0.1-hybrid

models/Zonos-v0.1-speaker-embedding/.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

models/Zonos-v0.1-speaker-embedding/README.md

@@ -0,0 +1,8 @@
+---
+license: apache-2.0
+---
+This repository contains the speaker embedding models for our Zonos-v0.1 [transformer](https://huggingface.co/Zyphra/Zonos-v0.1-transformer) and [hybrid](https://huggingface.co/Zyphra/Zonos-v0.1-hybrid) models.
+
+The speaker embedding models are based on the [ResNet293-SimAM-ASP](https://github.com/VoxBlink2/ScriptsForVoxBlink2/tree/main/asv) models from VoxBlink2. We use the pretrain models as we found the finetunes performed worse.
+
+The output of the speaker embedding model is then passed through an LDA layer and compressed from 256 to 128 dimensions to remove further spurious information about the speaker embedding clip before being fed into the Zonos models.

models/Zonos-v0.1-speaker-embedding/ResNet293_SimAM_ASP_base.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9f02677cd204f91e228f9ec344b7262749a9ab80b9848a7e0452e198d629a65d
+size 396513742

models/Zonos-v0.1-speaker-embedding/ResNet293_SimAM_ASP_base_LDA-128.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:383d4289dacaeac44386716ef15f7a1de6ff069d2a07ec1e933526eef5bd0084
+size 264512

models/Zonos-v0.1-speaker-embedding/source.txt

@@ -0,0 +1 @@
+https://huggingface.co/Zyphra/Zonos-v0.1-speaker-embedding

models/Zonos-v0.1-transformer/.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

models/Zonos-v0.1-transformer/README.md

@@ -0,0 +1,157 @@
+---
+license: apache-2.0
+pipeline_tag: text-to-speech
+library_name: zonos
+---
+# Zonos-v0.1
+
+<div align="center">
+<img src="https://github.com/Zyphra/Zonos/blob/main/assets/ZonosHeader.png?raw=true"
+     alt="Title card" 
+     style="width: 500px;
+            height: auto;
+            object-position: center top;">
+</div>
+
+---
+
+Zonos-v0.1 is a leading open-weight text-to-speech model trained on more than 200k hours of varied multilingual speech, delivering expressiveness and quality on par with—or even surpassing—top TTS providers.
+
+Our model enables highly natural speech generation from text prompts when given a speaker embedding or audio prefix, and can accurately perform speech cloning when given a reference clip spanning just a few seconds. The conditioning setup also allows for fine control over speaking rate, pitch variation, audio quality, and emotions such as happiness, fear, sadness, and anger. The model outputs speech natively at 44kHz.
+
+##### For more details and speech samples, check out our blog [here](https://www.zyphra.com/post/beta-release-of-zonos-v0-1)
+
+##### We also have a hosted version available at [playground.zyphra.com/audio](https://playground.zyphra.com/audio)
+
+---
+
+Zonos follows a straightforward architecture: text normalization and phonemization via eSpeak, followed by DAC token prediction through a transformer or hybrid backbone. An overview of the architecture can be seen below.
+
+<div align="center">
+<img src="https://github.com/Zyphra/Zonos/blob/main/assets/ArchitectureDiagram.png?raw=true"
+     alt="Architecture diagram" 
+     style="width: 1000px;
+            height: auto;
+            object-position: center top;">
+</div>
+
+---
+
+## Usage
+
+### Python
+
+```python
+import torch
+import torchaudio
+from zonos.model import Zonos
+from zonos.conditioning import make_cond_dict
+
+# model = Zonos.from_pretrained("Zyphra/Zonos-v0.1-hybrid", device="cuda")
+model = Zonos.from_pretrained("Zyphra/Zonos-v0.1-transformer", device="cuda")
+
+wav, sampling_rate = torchaudio.load("assets/exampleaudio.mp3")
+speaker = model.make_speaker_embedding(wav, sampling_rate)
+
+cond_dict = make_cond_dict(text="Hello, world!", speaker=speaker, language="en-us")
+conditioning = model.prepare_conditioning(cond_dict)
+
+codes = model.generate(conditioning)
+
+wavs = model.autoencoder.decode(codes).cpu()
+torchaudio.save("sample.wav", wavs[0], model.autoencoder.sampling_rate)
+```
+
+### Gradio interface (recommended)
+
+```bash
+uv run gradio_interface.py
+# python gradio_interface.py
+```
+
+This should produce a `sample.wav` file in your project root directory.
+
+_For repeated sampling we highly recommend using the gradio interface instead, as the minimal example needs to load the model every time it is run._
+
+## Features
+
+- Zero-shot TTS with voice cloning: Input desired text and a 10-30s speaker sample to generate high quality TTS output
+- Audio prefix inputs: Add text plus an audio prefix for even richer speaker matching. Audio prefixes can be used to elicit behaviours such as whispering which can otherwise be challenging to replicate when cloning from speaker embeddings
+- Multilingual support: Zonos-v0.1 supports English, Japanese, Chinese, French, and German
+- Audio quality and emotion control: Zonos offers fine-grained control of many aspects of the generated audio. These include speaking rate, pitch, maximum frequency, audio quality, and various emotions such as happiness, anger, sadness, and fear.
+- Fast: our model runs with a real-time factor of ~2x on an RTX 4090
+- Gradio WebUI: Zonos comes packaged with an easy to use gradio interface to generate speech
+- Simple installation and deployment: Zonos can be installed and deployed simply using the docker file packaged with our repository.
+
+## Installation
+
+**At the moment this repository only supports Linux systems (preferably Ubuntu 22.04/24.04) with recent NVIDIA GPUs (3000-series or newer, 6GB+ VRAM).**
+
+See also [Docker Installation](#docker-installation)
+
+#### System dependencies
+
+Zonos depends on the eSpeak library phonemization. You can install it on Ubuntu with the following command:
+
+```bash
+apt install -y espeak-ng
+```
+
+#### Python dependencies
+
+We highly recommend using a recent version of [uv](https://docs.astral.sh/uv/#installation) for installation. If you don't have uv installed, you can install it via pip: `pip install -U uv`.
+
+##### Installing into a new uv virtual environment (recommended)
+
+```bash
+uv sync
+uv sync --extra compile
+```
+
+##### Installing into the system/actived environment using uv
+
+```bash
+uv pip install -e .
+uv pip install -e .[compile]
+```
+
+##### Installing into the system/actived environment using pip
+
+```bash
+pip install -e .
+pip install --no-build-isolation -e .[compile]
+```
+
+##### Confirm that it's working
+
+For convenience we provide a minimal example to check that the installation works:
+
+```bash
+uv run sample.py
+# python sample.py
+```
+
+## Docker installation
+
+```bash
+git clone https://github.com/Zyphra/Zonos.git
+cd Zonos
+
+# For gradio
+docker compose up
+
+# Or for development you can do
+docker build -t Zonos .
+docker run -it --gpus=all --net=host -v /path/to/Zonos:/Zonos -t Zonos
+cd /Zonos
+python sample.py # this will generate a sample.wav in /Zonos
+```
+
+## Citation
+If you find this model useful in an academic context please cite as:
+```bash
+@misc{zyphra2025zonos,
+  title     = {Zonos-v0.1: An Expressive, Open-Source TTS Model},
+  author    = {Dario Sucic, Mohamed Osman, Gabriel Clark, Chris Warner, Beren Millidge},
+  year      = {2025},
+}

models/Zonos-v0.1-transformer/Zonos_v0_1.ipynb

@@ -0,0 +1,200 @@
+{
+  "nbformat": 4,
+  "nbformat_minor": 0,
+  "metadata": {
+    "colab": {
+      "provenance": [],
+      "machine_shape": "hm",
+      "gpuType": "L4"
+    },
+    "kernelspec": {
+      "name": "python3",
+      "display_name": "Python 3"
+    },
+    "language_info": {
+      "name": "python"
+    },
+    "accelerator": "GPU"
+  },
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "source": [
+        "# Zonos Text-to-Speech in Google Colab\n",
+        "\n",
+        "This guide walks you through setting up and using Zonos, a text-to-speech model, in Google Colab.\n",
+        "\n",
+        "## Setup Instructions\n",
+        "\n",
+        "First, create a new Colab notebook and run these setup commands:\n"
+      ],
+      "metadata": {
+        "id": "2KxPFOkv_weX"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Install system dependencies\n",
+        "!apt update && apt install -y espeak-ng\n",
+        "\n",
+        "# Install UV package manager\n",
+        "!pip install -U uv\n",
+        "\n",
+        "# Clone Zonos repository\n",
+        "!git clone https://github.com/Zyphra/Zonos.git\n",
+        "%cd Zonos\n",
+        "\n",
+        "# Install dependencies using UV\n",
+        "!uv sync\n",
+        "!uv sync --extra compile"
+      ],
+      "metadata": {
+        "id": "otaPpBlI5kQK"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "\n",
+        "## Basic Usage Example\n",
+        "\n",
+        "Here's a complete example showing how to generate speech from text:"
+      ],
+      "metadata": {
+        "id": "c7U2RJBjACfL"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "import torch\n",
+        "import torchaudio\n",
+        "from zonos.model import Zonos\n",
+        "from zonos.conditioning import make_cond_dict\n",
+        "\n",
+        "# Initialize model (use GPU if available)\n",
+        "device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
+        "model = Zonos.from_pretrained(\"Zyphra/Zonos-v0.1-transformer\", device=device)\n",
+        "\n",
+        "# Load reference audio for speaker characteristics\n",
+        "wav, sampling_rate = torchaudio.load(\"path_to_your_reference.mp3\")\n",
+        "speaker = model.make_speaker_embedding(wav, sampling_rate)\n",
+        "\n",
+        "# Set random seed for reproducibility\n",
+        "torch.manual_seed(421)\n",
+        "\n",
+        "# Create conditioning dictionary\n",
+        "cond_dict = make_cond_dict(\n",
+        "    text=\"Hello, world!\",\n",
+        "    speaker=speaker,\n",
+        "    language=\"en-us\"\n",
+        ")\n",
+        "conditioning = model.prepare_conditioning(cond_dict)\n",
+        "\n",
+        "# Generate audio codes\n",
+        "codes = model.generate(conditioning)\n",
+        "\n",
+        "# Decode to waveform and save\n",
+        "wavs = model.autoencoder.decode(codes).cpu()\n",
+        "torchaudio.save(\"output.wav\", wavs[0], model.autoencoder.sampling_rate)"
+      ],
+      "metadata": {
+        "id": "MoyaNigzAK47"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "\n",
+        "## Available Model Variants\n",
+        "\n",
+        "Zonos offers two main model variants:\n",
+        "- `Zyphra/Zonos-v0.1-transformer`: Transformer-based model (default)\n",
+        "- `Zyphra/Zonos-v0.1-hybrid`: Hybrid model architecture\n",
+        "\n",
+        "## Common Parameters\n",
+        "\n",
+        "The `make_cond_dict()` function accepts these parameters:\n",
+        "- `text`: The text to synthesize (required)\n",
+        "- `speaker`: Speaker embedding tensor (required)\n",
+        "- `language`: Language code (default: \"en-us\")\n",
+        "- `emotion`: Emotional style (optional)\n",
+        "- `speed`: Speech rate multiplier (optional)\n",
+        "\n",
+        "## Tips for Best Results\n",
+        "\n",
+        "1. Use high-quality reference audio (clean, single-speaker recordings)\n",
+        "2. Keep input text lengths reasonable (under 200 words per generation)\n",
+        "3. Experiment with different random seeds for variety\n",
+        "4. Consider using the hybrid model for longer texts\n",
+        "5. Clean your reference audio from background noise if possible\n",
+        "\n",
+        "## Troubleshooting\n",
+        "\n",
+        "Common issues and solutions:\n",
+        "\n",
+        "### CUDA Out of Memory\n",
+        "- Reduce batch size\n",
+        "- Use shorter input text\n",
+        "- Switch to CPU if necessary\n",
+        "- Clear torch cache: `torch.cuda.empty_cache()`\n",
+        "\n",
+        "### Poor Audio Quality\n",
+        "- Check reference audio quality\n",
+        "- Try different random seeds\n",
+        "- Adjust conditioning parameters\n",
+        "- Use a different model variant\n",
+        "\n",
+        "### Installation Issues\n",
+        "- Ensure all dependencies are installed\n",
+        "- Check CUDA compatibility\n",
+        "- Try reinstalling with `--force-reinstall`\n",
+        "\n",
+        "## Resources\n",
+        "\n",
+        "- [Zonos GitHub Repository](https://github.com/Zyphra/Zonos)\n",
+        "- [Model Card on Hugging Face](https://huggingface.co/Zyphra/Zonos-v0.1-transformer)\n",
+        "\n",
+        "Note: This documentation assumes basic familiarity with Python and Google Colab. For more detailed information, refer to the official repository."
+      ],
+      "metadata": {
+        "id": "RrEyAyBJAQwT"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "!uv run sample.py\n",
+        "\n"
+      ],
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "_M20HuL38k_Z",
+        "outputId": "259a14d1-8da6-4666-dfef-fd79cf00e29a"
+      },
+      "execution_count": null,
+      "outputs": [
+        {
+          "output_type": "stream",
+          "name": "stdout",
+          "text": [
+            "config.json: 100% 2.52k/2.52k [00:00<00:00, 13.6MB/s]\n",
+            "model.safetensors: 100% 3.25G/3.25G [01:16<00:00, 42.2MB/s]\n",
+            "config.json: 100% 541/541 [00:00<00:00, 3.97MB/s]\n",
+            "model.safetensors: 100% 307M/307M [00:07<00:00, 42.2MB/s]\n",
+            "ResNet293_SimAM_ASP_base.pt: 100% 397M/397M [00:09<00:00, 42.3MB/s]\n",
+            "ResNet293_SimAM_ASP_base_LDA-128.pt: 100% 265k/265k [00:00<00:00, 199MB/s]\n",
+            "Generating:   5% 130/2588 [00:02<00:50, 48.35it/s]\n"
+          ]
+        }
+      ]
+    }
+  ]
+}

models/Zonos-v0.1-transformer/Zonos_v0_1_Text_to_Speech.ipynb

@@ -0,0 +1,299 @@
+{
+  "nbformat": 4,
+  "nbformat_minor": 0,
+  "metadata": {
+    "colab": {
+      "provenance": []
+    },
+    "kernelspec": {
+      "name": "python3",
+      "display_name": "Python 3"
+    },
+    "language_info": {
+      "name": "python"
+    }
+  },
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "source": [
+        "# 🎧 Zonos v0.1 Text-to-Speech Demo\n",
+        "\n",
+        "Welcome to the Zonos Text-to-Speech demo! This notebook demonstrates how to use the Zonos TTS model to generate high-quality speech in multiple languages.\n",
+        "\n",
+        "## 📦 Setup\n",
+        "\n",
+        "First, let's install the required dependencies:"
+      ],
+      "metadata": {
+        "id": "N-549GV0DfSD"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Install system dependencies\n",
+        "!apt update && apt install -y espeak-ng\n",
+        "\n",
+        "# Install UV package manager\n",
+        "!pip install -U uv\n",
+        "\n",
+        "# Clone Zonos repository\n",
+        "!git clone https://github.com/Zyphra/Zonos.git\n",
+        "%cd Zonos\n",
+        "\n",
+        "# Install dependencies using UV\n",
+        "!uv sync\n",
+        "!uv sync --extra compile"
+      ],
+      "metadata": {
+        "id": "la2K7_QCDinW"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "\n",
+        "## 🎵 Upload Reference Audio\n",
+        "\n",
+        "Upload a reference audio file that Zonos will use to clone the voice:\n"
+      ],
+      "metadata": {
+        "id": "fINPv_7nDmJw"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Create assets directory\n",
+        "!mkdir -p assets\n",
+        "\n",
+        "# Option 1: Upload from your computer\n",
+        "from google.colab import files\n",
+        "uploaded = files.upload()\n",
+        "for filename in uploaded.keys():\n",
+        "    !mv \"{filename}\" \"assets/reference.mp3\"\n",
+        "\n",
+        "# Option 2: Download from URL\n",
+        "# !wget -O assets/reference.mp3 YOUR_AUDIO_URL_HERE"
+      ],
+      "metadata": {
+        "id": "FzZs7PaUDok4"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "## 🚀 Generate Speech\n",
+        "\n",
+        "Now let's generate some speech using the model:"
+      ],
+      "metadata": {
+        "id": "mA0GTXGFDzLq"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "\n",
+        "import torch\n",
+        "import torchaudio\n",
+        "from zonos.model import Zonos\n",
+        "from zonos.conditioning import make_cond_dict\n",
+        "\n",
+        "# Check device\n",
+        "device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
+        "print(f\"Using device: {device}\")\n",
+        "\n",
+        "# Load model\n",
+        "print(\"Loading model...\")\n",
+        "model = Zonos.from_pretrained(\"Zyphra/Zonos-v0.1-transformer\", device=device)\n",
+        "\n",
+        "# Load reference audio\n",
+        "print(\"Loading reference audio...\")\n",
+        "wav, sampling_rate = torchaudio.load(\"assets/reference.mp3\")\n",
+        "speaker = model.make_speaker_embedding(wav, sampling_rate)\n",
+        "\n",
+        "# Set seed for reproducibility\n",
+        "torch.manual_seed(421)\n",
+        "\n",
+        "def generate_speech(text, language=\"en-us\", speed=1.0):\n",
+        "    \"\"\"Generate speech from text\"\"\"\n",
+        "    print(f\"Generating: {text}\")\n",
+        "\n",
+        "    # Create conditioning\n",
+        "    cond_dict = make_cond_dict(\n",
+        "        text=text,\n",
+        "        speaker=speaker,\n",
+        "        language=language,\n",
+        "        speed=speed\n",
+        "    )\n",
+        "    conditioning = model.prepare_conditioning(cond_dict)\n",
+        "\n",
+        "    # Generate audio\n",
+        "    codes = model.generate(conditioning)\n",
+        "    wavs = model.autoencoder.decode(codes).cpu()\n",
+        "\n",
+        "    # Save and play\n",
+        "    filename = \"output.wav\"\n",
+        "    torchaudio.save(filename, wavs[0], model.autoencoder.sampling_rate)\n",
+        "    return filename\n",
+        "\n",
+        "# Test generation\n",
+        "output_file = generate_speech(\n",
+        "    \"Welcome to Zonos Text-to-Speech! This is a demo of voice cloning technology.\",\n",
+        "    language=\"en-us\",\n",
+        "    speed=1.0\n",
+        ")\n",
+        "\n",
+        "# Play the generated audio\n",
+        "from IPython.display import Audio\n",
+        "Audio(output_file)"
+      ],
+      "metadata": {
+        "id": "wyTwtvfoD1X_"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "## 🌍 Multi-language Support\n",
+        "\n",
+        "Try generating speech in different languages:"
+      ],
+      "metadata": {
+        "id": "Mlx2IZBLD82n"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Generate speech in multiple languages\n",
+        "languages = {\n",
+        "    \"English\": (\"en-us\", \"Hello, how are you today?\"),\n",
+        "    \"Japanese\": (\"ja-jp\", \"こんにちは、お元気ですか？\"),\n",
+        "    \"Spanish\": (\"es-es\", \"¡Hola! ¿Cómo estás?\"),\n",
+        "    \"Korean\": (\"ko-kr\", \"안녕하세요, 오늘 기분이 어떠신가요?\"),\n",
+        "    \"Russian\": (\"ru-ru\", \"Привет, как дела?\"),\n",
+        "    \"French\": (\"fr-fr\", \"Bonjour, comment allez-vous?\"),\n",
+        "    \"German\": (\"de-de\", \"Hallo, wie geht es dir?\"),\n",
+        "    \"Chinese\": (\"zh-cn\", \"你好，今天过得怎么样？\"),\n",
+        "    \"Hindi\": (\"hi-in\", \"नमस्ते, आप कैसे हैं?\"),\n",
+        "    \"Arabic\": (\"ar-ae\", \"مرحبا، كيف حالك؟\")\n",
+        "}\n",
+        "\n",
+        "for lang_name, (lang_code, text) in languages.items():\n",
+        "    print(f\"\\n{lang_name}:\")\n",
+        "    output_file = generate_speech(text, language=lang_code)\n",
+        "    display(Audio(output_file))"
+      ],
+      "metadata": {
+        "id": "Wm3RgQcsD_tT"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "## ⚡ Speed Control\n",
+        "\n",
+        "Experiment with different speech speeds:"
+      ],
+      "metadata": {
+        "id": "lu3u-xstEEaj"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "text = \"This is a demonstration of speech rate control in Zonos.\"\n",
+        "speeds = [0.8, 1.0, 1.2, 1.5]\n",
+        "\n",
+        "for speed in speeds:\n",
+        "    print(f\"\\nSpeed {speed}x:\")\n",
+        "    output_file = generate_speech(text, speed=speed)\n",
+        "    display(Audio(output_file))"
+      ],
+      "metadata": {
+        "id": "pX_0hcZ3EDYH"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "## 📝 Custom Text Generation\n",
+        "\n",
+        "Try your own text:"
+      ],
+      "metadata": {
+        "id": "zw8Y6T6zEIS-"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# @title Generate Speech from Text\n",
+        "text = \"Type your text here\" # @param {type:\"string\"}\n",
+        "language = \"en-us\" # @param [\"en-us\", \"ja-jp\", \"es-es\", \"ko-kr\", \"ru-ru\", \"fr-fr\", \"de-de\", \"zh-cn\", \"hi-in\", \"ar-ae\"]\n",
+        "speed = 1.0 # @param {type:\"slider\", min:0.5, max:2.0, step:0.1}\n",
+        "\n",
+        "output_file = generate_speech(text, language=language, speed=speed)\n",
+        "Audio(output_file)"
+      ],
+      "metadata": {
+        "id": "fWQE9d_JEKqj"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "\n",
+        "\n",
+        "## 📚 Resources\n",
+        "\n",
+        "- [Zonos GitHub Repository](https://github.com/Zyphra/Zonos)\n",
+        "- [Model on Hugging Face](https://huggingface.co/Zyphra/Zonos-v0.1-transformer)\n",
+        "- [Documentation](https://github.com/Zyphra/Zonos/blob/main/README.md)\n",
+        "\n",
+        "## 🤝 Community\n",
+        "\n",
+        "- Share your generations with the community using #ZonosAI\n",
+        "- Report issues on [GitHub](https://github.com/Zyphra/Zonos/issues)\n",
+        "- Join discussions on [Discord](https://discord.gg/zonos)\n",
+        "\n",
+        "## ⚠️ Notes\n",
+        "\n",
+        "1. For best results:\n",
+        "   - Use clear reference audio (5-10 seconds)\n",
+        "   - Avoid background noise\n",
+        "   - Keep input text reasonably sized\n",
+        "   - Experiment with different speeds\n",
+        "\n",
+        "2. Troubleshooting:\n",
+        "   - If you get CUDA out of memory errors, try:\n",
+        "     - Shorter text\n",
+        "     - Lower batch size\n",
+        "     - CPU mode\n",
+        "     - `torch.cuda.empty_cache()`\n",
+        "   - For audio quality issues:\n",
+        "     - Try different reference audio\n",
+        "     - Adjust speed parameter\n",
+        "     - Change random seed\n"
+      ],
+      "metadata": {
+        "id": "Ek_mm65_DbHK"
+      }
+    }
+  ]
+}

models/Zonos-v0.1-transformer/config.json

@@ -0,0 +1,101 @@
+{
+    "backbone": {
+        "d_model": 2048,
+        "d_intermediate": 0,
+        "attn_mlp_d_intermediate": 8192,
+        "n_layer": 26,
+        "ssm_cfg": {},
+        "attn_layer_idx": [
+            0,
+            1,
+            2,
+            3,
+            4,
+            5,
+            6,
+            7,
+            8,
+            9,
+            10,
+            11,
+            12,
+            13,
+            14,
+            15,
+            16,
+            17,
+            18,
+            19,
+            20,
+            21,
+            22,
+            23,
+            24,
+            25
+        ],
+        "attn_cfg": {
+            "causal": true,
+            "num_heads": 16,
+            "num_heads_kv": 4,
+            "rotary_emb_dim": 128,
+            "rotary_emb_interleaved": true,
+            "qkv_proj_bias": false,
+            "out_proj_bias": false
+        },
+        "rms_norm": false,
+        "residual_in_fp32": false,
+        "norm_epsilon": 1e-05
+    },
+    "prefix_conditioner": {
+        "conditioners": [
+            {
+                "type": "EspeakPhonemeConditioner",
+                "name": "espeak"
+            },
+            {
+                "cond_dim": 128,
+                "uncond_type": "learned",
+                "projection": "linear",
+                "type": "PassthroughConditioner",
+                "name": "speaker"
+            },
+            {
+                "input_dim": 8,
+                "uncond_type": "learned",
+                "type": "FourierConditioner",
+                "name": "emotion"
+            },
+            {
+                "min_val": 0,
+                "max_val": 24000,
+                "uncond_type": "learned",
+                "type": "FourierConditioner",
+                "name": "fmax"
+            },
+            {
+                "min_val": 0,
+                "max_val": 400,
+                "uncond_type": "learned",
+                "type": "FourierConditioner",
+                "name": "pitch_std"
+            },
+            {
+                "min_val": 0,
+                "max_val": 40,
+                "uncond_type": "learned",
+                "type": "FourierConditioner",
+                "name": "speaking_rate"
+            },
+            {
+                "min_val": -1,
+                "max_val": 126,
+                "uncond_type": "learned",
+                "type": "IntegerConditioner",
+                "name": "language_id"
+            }
+        ],
+        "projection": "linear"
+    },
+    "eos_token_id": 1024,
+    "masked_token_id": 1025
+}

models/Zonos-v0.1-transformer/issues.txt

@@ -0,0 +1,95 @@
+------------------------------------------------------------------------------------------------------------------------------------------------
+#2 Streaming support / batch inference?
+------------------------------------------------------------------------------------------------------------------------------------------------
+
+[brainofchild] Feb 11, 2025
+
+For production purposes, these are important. Was curious how the model handles output streaming / batch inference.
+
+[gabrielclark3330, Zyphra.org] Feb 13, 2025
+
+Both are possible but have not been added to the codebase yet. We will definitely ship streaming as there is already a PR out for it.
+
+[brainofchild] Apr 14, 2025
+
+Curious to hear if there are any updates!
+
+[ism0il] Oct 2, 2025
+
+Any update?
+
+------------------------------------------------------------------------------------------------------------------------------------------------
+#5 Zonos not installing backbone
+------------------------------------------------------------------------------------------------------------------------------------------------
+
+[mnemic] Feb 27, 2025
+
+To fix it, I needed to:
+Git clone https://github.com/Zyphra/Zonos/
+
+Edit pyproject.toml:
+
+[project]
+name = "zonos"
+version = "0.1.0"
+description = "Text-to-speech by Zyphra"
+readme = "README.md"
+requires-python = ">=3.10"
+dependencies = [
+    "torch>=2.5.1",
+    "setuptools",
+    "packaging",
+    "inflect>=7.5.0",
+    "kanjize>=1.5.0",
+    "numpy>=2.2.2",
+    "phonemizer>=3.3.0",
+    "sudachidict-full>=20241021",
+    "sudachipy>=0.6.10",
+    "torchaudio>=2.5.1",
+    "transformers>=4.48.1",
+    "soundfile>=0.13.1",
+    "huggingface-hub>=0.28.1",
+    "gradio>=5.15.0",
+]
+
+# These are technically optional, but mamba-ssm is required to run hybrid models.
+[project.optional-dependencies]
+compile = [
+    "flash-attn>=2.7.3",
+    "mamba-ssm>=2.2.4",
+    "causal-conv1d>=1.5.0.post8",
+]
+
+[tool.setuptools.packages.find]
+include = ["zonos", "zonos.*"]
+
+[tool.uv]
+no-build-isolation-package = ["flash-attn", "mamba-ssm", "causal-conv1d"]
+
+[tool.ruff]
+line-length = 120
+
+The difference was this line:
+include = ["zonos", "zonos.*"]
+
+Then do a "pip install" inside this folder, to install it into the virtual environment.
+
+------------------------------------------------------------------------------------------------------------------------------------------------
+#7 Zonos is excellent, but there's one major issue that makes it's use impractical - the "uh" (and "um") sounds weren't labelled in the dataset
+------------------------------------------------------------------------------------------------------------------------------------------------
+
+[jattoedaltni] Mar 13, 2025
+
+I've spoken to a few others about this, I know it's not just me or the specific input .wavs - I think the training data didn't label "uh" and "um" in people's speech, and thus, random "uh" sounds appear extremely consistently in the outputs.
+(And the output does not output a normal "uh" sound, but since the model was trained to output the "uh" sound where it wasn't labelled, it randomly attached it to many words, especially those starting with vowels. The result is that it sounds like an extra syllable directly attached to the word, and making the word partly unrecognizable)
+It's important that those "uh" and sounds are removed or at least labelled for the next iteration.
+Other than that it's almost perfect!
+
+------------------------------------------------------------------------------------------------------------------------------------------------
+#11 German with english accent?
+------------------------------------------------------------------------------------------------------------------------------------------------
+
+[fesrgs] May 30, 2025
+
+Zonos is perfect as a tool, but the "de" model doesn't seems to generate speech without an English accent for me.
+Is there currently a way to generate perfect german?

models/Zonos-v0.1-transformer/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4ac68319d6b8c1b29b94b8801ca41a93a5fc458f4b1d4bbbe5045b2ee77efcf0
+size 3248848864

models/Zonos-v0.1-transformer/source.txt

@@ -0,0 +1 @@
+https://huggingface.co/Zyphra/Zonos-v0.1-transformer

models/zonos-assets/.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

models/zonos-assets/dac.json

@@ -0,0 +1 @@
+{"num_codebooks": 9, "sampling_rate": 44100}

models/zonos-assets/dac_decode.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:894114aaca947f80dca5ba5ca0b229fe227ccf31e482ea46527190ccb41125ec
+size 217067971

models/zonos-assets/dac_encode.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bb742ae3eb64985f210079eb6af49d6a1e751937ffdaffb9ece46f9688c9fec6
+size 90461103

models/zonos-assets/issues.txt

@@ -0,0 +1,71 @@
+-----------------------------------------------------------------------
+#1 I have checked ONNX OPs supported by RKNN Toolkit2
+-----------------------------------------------------------------------
+
+[Mongdol1] Aug 4, 2025
+
+RKNNToolkit2 is a toolkit for Rockchip's NPU.
+
+=>dac_decode.onnx <=
+| Used op| support |
+| Add | O |
+| Conv | O |
+| ConvTranspose | O |
+| Gather | O |
+| Mul | O |
+| Pow | O |
+| Sin | O |
+| Squeeze |O |
+| Tanh | O |
+| Transpose |O |
+| Unsqueeze | O |
+
+=> dac_encode.onnx <=
+| Used op| support |
+| Abs | O |
+| Add | O |
+| ArgMax | O |
+| Clip | O |
+| Concat | O |
+| Conv |O |
+| Div | O |
+| Expand | O |
+| Gather | O |
+| MatMul | O |
+| Mul | O |
+| Neg | O |
+| Pow | O |
+| ReduceL2| X | ❌
+| ReduceSum |O |
+| Reshape | O |
+| Shape | O |
+| Sin | O |
+| Squeeze | O |
+| Sub | O |
+| Transpose | O |
+| Unsqueeze | O |
+
+=> speaker_embedding_lda.onnx <=
+
+| Used op| support |
+| Add | O |
+| BatchNormalization | O |
+| Cast | O |
+| Clip | O |
+| Concat | O |
+| Conv | O |
+| Div | O |
+| Gather | O |
+| Gemm | O |
+| Mul | O |
+| Pow | O |
+| ReduceMean | O |
+| ReduceSum | O |
+| Relu | O |
+| Reshape | O |
+| Shape | O |
+| Sigmoid | O |
+| Softmax | O |
+| Sqrt | O |
+| Sub | O |
+| Unsqueeze | O |

models/zonos-assets/source.txt

@@ -0,0 +1 @@
+https://huggingface.co/neody/zonos-assets

models/zonos-assets/speaker_embedding_lda.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6da5eb350808a0f1771b34352cec9a22b53378aa4249f75fa89b51cd0e2aebfd
+size 402992435

wiki/Beta Release of Zonos v0.1.pdf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5b983a56053b5cc16ad7c4cb490afaefbcd5e659ffb11f7ea095a1b4d45eb796
+size 4173594

wiki/Zonos v0.1. Best Open Source AI Voice Clone Tool.pdf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:132c8f31b63194e7675552fe684a2dcaa1e97f32ba13e07792c9669c3b474449
+size 3062830

wiki/Zyphra Just Dropped Zonos. Instant TTS and Voice Cloning. Install Locally.mkv

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d3dc201ed13f1525208c495383f72a1c22bdbbee0017d113a1f53ddddb706ff3
+size 46213991