WaveGen / README.md
LeeSangHoon's picture
Clarify Windows Studio runs without Python
c52541d verified
|
Raw History Blame Contribute Delete
14.9 kB
---
license: cc-by-nc-4.0
pipeline_tag: text-to-speech
language:
- en
- zh
datasets:
- amphion/Emilia-Dataset
tags:
- WaveGen
- wavegen
- TTS
- zero-shot
- voice-cloning
- audio
- multilingual
- pytorch
---
# WaveGen [NeurIPS 2026]
**WaveGen: Truly End-to-End Waveform Generation via Internal Spectral Trajectory Learning**
Sang-Hoon Lee and Ha-Yeong Choi
Ajou University and KT
[Paper](https://arxiv.org/abs/2611.12527) · [Source Code](https://github.com/sh-lee-prml/WaveGen) · [Live Demo](https://huggingface.co/spaces/LeeSangHoon/WaveGen) · [Demo Page](https://wavegen26.github.io/demo) · [Benchmark Demo](https://sh-lee-prml.github.io/wavegen-tts-demo/) · [Spectral Forcing Demo](https://wavegen26.github.io/demo/spectral_forcing_visualization/) · [WaveGen Studio (Windows)](https://github.com/sh-lee-prml/WaveGen-Studio)
**[WaveGen Studio](https://github.com/sh-lee-prml/WaveGen-Studio)** runs directly on Windows without installing Python.
WaveGen generates speech directly from text and reference audio using Spectral Forcing. The models are trained on English and Mandarin Chinese speech from Emilia, using **24 kHz audio** and **50 Hz waveform patches** (480 samples per patch).
## Models
This repository contains the pre-trained, V-Spec, and four-step Flash checkpoints in separate folders.
| Model | Checkpoint | Folder | Inference steps |
|---|---|---|---|
| WaveGen (Pre-trained) | 1300k EMA | `WaveGen/` | 50 |
| **WaveGen (Fine-tuned with v-spec losses) — Recommended** | **1400k EMA** | **`WaveGen_VSpec/`** | **50** |
| WaveGen-Flash | 48k EMA | `WaveGen_Flash/` | **4 (fixed)** |
**WaveGen (Fine-tuned with v-spec losses)** is the **recommended model for inference** and the default in the source repository. All checkpoints contain **FP32 EMA weights only**, in Safetensors format. Optimizer state and other training state are excluded; these files are intended for inference, not for resuming a training run.
**WaveGen-Flash** is optimized for four-step speech generation. Its complete merged
weights load from one Safetensors file, together with its YAML and vocabulary.
The Flash release includes inference code and model weights; its training recipe
is not included.
## Seed-TTS evaluation
Results on the complete standard Seed-TTS test sets for the EMA weights released here.
The pre-trained and V-Spec models use **Prompt Anchor**, **Scheduled K3** (base CFG 1.25), **50 Euler steps**,
Mel temperature 0, inference bottleneck dropout 0.2, and seed 0, with FP32 model
weights and BF16 CUDA inference. The original test prompts are used.
Flash uses **four Euler steps**, the time grid **`[0, 0.1, 0.25, 0.5, 1.0]`**,
CFG 0, inference bottleneck dropout 0, waveform temperature 1, **initial Mel
temperature 0**, and seed 0. It uses the same original test prompts and
FP32-weight/BF16-inference precision. Prompt states interpolate from their
initial noise toward the reference at each step; the initial Mel noise is
scaled by the Mel temperature. The Flash results below use the new default
temperature 0. WER/CER and UTMOS are truncated to two decimal places,
and SIM to three decimal places (not rounded).
The complete test sets contain 1,088 English and 2,020 Chinese utterances.
<table>
<colgroup span="1"></colgroup>
<colgroup span="3"></colgroup>
<colgroup span="3"></colgroup>
<thead>
<tr>
<th rowspan="2" scope="col" align="left">Model</th>
<th colspan="3" scope="colgroup" align="center">EN</th>
<th colspan="3" scope="colgroup" align="center">ZH</th>
</tr>
<tr>
<th scope="col" align="center">WER (%) ↓</th>
<th scope="col" align="center">SIM ↑</th>
<th scope="col" align="center">UTMOS ↑</th>
<th scope="col" align="center">CER (%) ↓</th>
<th scope="col" align="center">SIM ↑</th>
<th scope="col" align="center">UTMOS ↑</th>
</tr>
</thead>
<tbody>
<tr>
<th scope="row" align="left"><strong>WaveGen</strong><br><sub>Pre-trained · 1300k EMA</sub></th>
<td align="center">1.47</td>
<td align="center">0.755</td>
<td align="center">3.68</td>
<td align="center">1.75</td>
<td align="center">0.805</td>
<td align="center">2.87</td>
</tr>
<tr>
<th scope="row" align="left"><strong>WaveGen V-Spec</strong><br><sub>1400k EMA · Recommended</sub></th>
<td align="center">1.39</td>
<td align="center">0.743</td>
<td align="center">3.59</td>
<td align="center">1.65</td>
<td align="center">0.783</td>
<td align="center">2.79</td>
</tr>
<tr>
<th scope="row" align="left"><strong>WaveGen-Flash</strong><br><sub>48k EMA · 4 steps</sub></th>
<td align="center">1.49</td>
<td align="center">0.727</td>
<td align="center">3.53</td>
<td align="center">1.59</td>
<td align="center">0.768</td>
<td align="center">2.74</td>
</tr>
</tbody>
</table>
**WaveGen achieves state-of-the-art performance among TTS models trained on the Emilia dataset.**
WER is reported for English and CER for Chinese. Error rates are percentages;
all metrics are macro-averaged over utterances. The Safetensors exports preserve
the FP32 EMA inference tensor values.
## Download
Download the checkpoint, YAML, and vocabulary from the desired model folder and place that folder under `ckpts/` in the WaveGen source repository:
```text
ckpts/
├── WaveGen/
│ ├── model_1300000.safetensors
│ ├── WaveGen_SF_Emilia_24k_50Hz_IBDrop02.yaml
│ └── vocab.txt
├── WaveGen_VSpec/
│ ├── model_1400000.safetensors
│ ├── WaveGen_VSpec.yaml
│ └── vocab.txt
└── WaveGen_Flash/
├── model.safetensors
├── WaveGen_Flash.yaml
└── vocab.txt
```
Keep each checkpoint together with its matching configuration and vocabulary. This Hub repository uses `WaveGen/`, `WaveGen_VSpec/`, and `WaveGen_Flash/` at its root; `ckpts/` is their destination in your local source checkout.
## Usage
Follow the [installation instructions](https://github.com/sh-lee-prml/WaveGen#installation), then run from the WaveGen source repository root. Supply a reference recording, its exact transcript, and the text to synthesize. Automatic transcription is not required.
### Fine-tuned model (recommended default)
```bash
python -m wavegen_v_spec.infer.infer_cli \
--ckpt_file ckpts/WaveGen_VSpec/model_1400000.safetensors \
--model_cfg ckpts/WaveGen_VSpec/WaveGen_VSpec.yaml \
--vocab_file ckpts/WaveGen_VSpec/vocab.txt \
--ref_audio reference.wav \
--ref_text "The transcript of the reference recording." \
--gen_text "The text to synthesize."
```
### Pre-trained model
```bash
python -m wavegen.infer.infer_cli \
--ckpt_file ckpts/WaveGen/model_1300000.safetensors \
--model_cfg ckpts/WaveGen/WaveGen_SF_Emilia_24k_50Hz_IBDrop02.yaml \
--vocab_file ckpts/WaveGen/vocab.txt \
--cfg_schedule scheduled_k3 \
--cfg_strength 1.25 \
--ref_audio reference.wav \
--ref_text "The transcript of the reference recording." \
--gen_text "The text to synthesize."
```
Both examples use EMA weights, **Prompt Anchor** (`Prompt_xt_Anchor`), **Scheduled K3** (base CFG 1.25), 50 steps, Mel temperature 0, and seed 0. The supplied models use inference bottleneck dropout 0.2. The pre-trained module uses constant CFG 3 when no schedule is specified, so its example selects Scheduled K3 explicitly. CUDA inference uses BF16 autocast with FP32 model weights; CPU inference uses FP32.
### Load the fine-tuned model from Hugging Face
The subfolder paths below select the 1400k model from [LeeSangHoon/WaveGen](https://huggingface.co/LeeSangHoon/WaveGen). The loader resolves `main` to one commit before downloading the three assets; use a full commit SHA instead of `main` to pin a specific release.
```python
import soundfile as sf
import torch
from wavegen_v_spec.infer.utils_infer import WaveGen
torch.set_num_threads(1)
model = WaveGen.from_pretrained(
"LeeSangHoon/WaveGen",
revision="main",
checkpoint_filename="WaveGen_VSpec/model_1400000.safetensors",
config_filename="WaveGen_VSpec/WaveGen_VSpec.yaml",
vocab_filename="WaveGen_VSpec/vocab.txt",
use_ema=True,
)
sample_rate, audio = model.synthesize(
"reference.wav",
"The transcript of the reference recording.",
"The text to synthesize.",
)
sf.write("generated.wav", audio, sample_rate, subtype="PCM_16")
```
For the pre-trained model, download the `WaveGen/` folder and use the pre-trained CLI above. See the [source repository](https://github.com/sh-lee-prml/WaveGen) for the local Python API, Gradio demo, and batch evaluation.
The source checkout defaults to the V-Spec 1400k Safetensors checkpoint. The base
and V-Spec loaders support existing local `.pt` checkpoints through an explicit
checkpoint-path override.
### WaveGen-Flash (4 Steps)
```bash
python -m wavegen_flash.infer.infer_cli \
--checkpoint ckpts/WaveGen_Flash/model.safetensors \
--model-config ckpts/WaveGen_Flash/WaveGen_Flash.yaml \
--vocab-file ckpts/WaveGen_Flash/vocab.txt \
--ref-audio reference.wav \
--ref-text "The transcript of the reference recording." \
--gen-text "The text to synthesize." \
--output generated_flash.wav
```
Or load the same bundle directly from this repository:
```python
import soundfile as sf
import torch
from wavegen_flash import WaveGenFlash
torch.set_num_threads(1)
model = WaveGenFlash.from_pretrained("LeeSangHoon/WaveGen", revision="main")
sample_rate, audio = model.synthesize(
"reference.wav",
"The transcript of the reference recording.",
"The text to synthesize.",
)
sf.write("generated_flash.wav", audio, sample_rate, subtype="PCM_16")
```
The Flash loader accepts Safetensors only and requires no additional base weights.
Its YAML fixes four Euler calls at `[0, 0.1, 0.25, 0.5]`, ending at 1.0, with
CFG 0 and IB dropout 0. The grid already includes time shift 3; it is not shifted
again during inference. The final Mel output head is skipped by default;
`--compute-final-mel` or `compute_final_mel=True` restores its computation.
The Mel input and shared network remain active on the final call.
The YAML sets `mel_temperature: 0.0` by default, while waveform temperature
remains 1. Mel temperature scales only the initial Gaussian Mel noise;
the clean reference Mel and later predicted Mel states are not scaled.
Both noise draws are preserved even at temperature 0. Use
`--mel-temperature 1` or `model.synthesize(..., mel_temperature=1.0)`
to reproduce the previous initialization with the same weights.
Compared with temperature 1, temperature 0 improves WER/CER, SIM, and
UTMOS on both complete Seed-TTS test sets.
The default duration limit is 30 seconds for prompt plus generated speech.
Split longer text or explicitly increase `max_duration_seconds`; the sampler
does not automatically split text. See the
[Flash guide](https://github.com/sh-lee-prml/WaveGen/blob/main/src/wavegen_flash/README.md)
for the complete inference interface.
## Safeguards and limitations
- Use reference speech with the speaker's informed consent and permission to use the recording.
- Do not use generated speech for deceptive impersonation, fraud, harassment, or fabricated statements presented as authentic speech.
- Clearly identify shared or published outputs as AI-generated audio, and protect reference recordings and personal information.
- Reference quality and transcript accuracy affect the generated speech. Pronunciation errors, omissions, repetitions, and speaker-similarity failures may occur.
- For long passages, split the text into shorter segments; the single-utterance examples above do not automatically split long text.
These usage guidelines are not automatic safety filters. Public-facing services should add appropriate access controls, abuse reporting, and review of suspected misuse.
## License
The model checkpoints are released under [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/).
WaveGen source-code contributions are released under the [MIT License](https://github.com/sh-lee-prml/WaveGen/blob/main/LICENSE). Reused source components retain their original licenses and notices; see the [source-code third-party notices](https://github.com/sh-lee-prml/WaveGen/blob/main/docs/THIRD_PARTY_NOTICES.md).
## WaveGen Studio 0.1 beta for Windows
[WaveGen Studio](https://github.com/sh-lee-prml/WaveGen-Studio) provides a native
C++ desktop interface for Windows x64, with persistent model loading, saved
reference voices, text splitting, and audio editing. Python is not required to
run the packaged application. This is a beta release; the original research
implementation and checkpoints above remain available through
[WaveGen](https://github.com/sh-lee-prml/WaveGen).
The native model packs for Studio **0.1.0-beta.1** are stored under
[`studio/v0.1.0-beta.1/`](https://huggingface.co/LeeSangHoon/WaveGen/tree/main/studio/v0.1.0-beta.1):
| Pack | Contents | License scope |
|---|---|---|
| Flash | Exported ONNX model and frontend assets for four-step generation | WaveGen weights: CC BY-NC 4.0; retain the included source and third-party notices |
| V-Spec | Exported ONNX model and frontend assets for 50-step generation | WaveGen weights: CC BY-NC 4.0; retain the included source and third-party notices |
| RE-USE | Optional exported speech enhancement model derived from [NVIDIA-NemotronLabs-RE-USE](https://huggingface.co/nvidia/NVIDIA-NemotronLabs-RE-USE) | OpenMDW 1.1, with the included Apache 2.0 Mamba notice; this pack is not relicensed under CC BY-NC 4.0 |
The repository's CC BY-NC 4.0 model-card label applies to the WaveGen
checkpoints and their Flash/V-Spec exports. The separate RE-USE pack retains
its own license and origin. Studio application code is MIT; the Windows
application and runtime dependencies have their own notices and are distributed
through the [Studio releases](https://github.com/sh-lee-prml/WaveGen-Studio/releases).
Use the Studio installer to select the model packs. It reads the versioned
artifact paths, sizes, archive SHA-256 hashes, and extracted-file hashes from
the root [`config.json`](https://huggingface.co/LeeSangHoon/WaveGen/resolve/main/config.json), then verifies the selected
downloads against the release manifest. This file is an actual installation
configuration, not a Transformers model configuration. Verified configuration
and model files are reused from the local cache; no counting-only request is
made. Hugging Face's download statistics are server-side request metrics and
do not represent an exact count of completed installations or unique users.
[`SHA256SUMS`](https://huggingface.co/LeeSangHoon/WaveGen/resolve/main/SHA256SUMS) includes the original checkpoint entries
and these native archives. The native packages do not include personal
reference recordings, saved user profiles, or generated speech history.