Add model card for GibbsTTS
#1
by nielsr HF Staff - opened
README.md
CHANGED
|
@@ -1,3 +1,56 @@
|
|
| 1 |
-
---
|
| 2 |
-
license: mit
|
| 3 |
-
--
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: mit
|
| 3 |
+
pipeline_tag: text-to-speech
|
| 4 |
+
---
|
| 5 |
+
|
| 6 |
+
# GibbsTTS
|
| 7 |
+
|
| 8 |
+
GibbsTTS is a zero-shot text-to-speech model based on metric-induced discrete flow matching (MI-DFM). It incorporates a kinetic-optimal scheduler and finite-step CTMC moment correction to address heuristic scheduling and path-tracking errors in discrete generation.
|
| 9 |
+
|
| 10 |
+
- **Paper:** [Kinetic-Optimal Scheduling with Moment Correction for Metric-Induced Discrete Flow Matching in Zero-Shot Text-to-Speech](https://arxiv.org/abs/2605.09386)
|
| 11 |
+
- **Project Page:** [https://ydqmkkx.github.io/GibbsTTSProject](https://ydqmkkx.github.io/GibbsTTSProject)
|
| 12 |
+
- **Code:** [https://github.com/ydqmkkx/GibbsTTS](https://github.com/ydqmkkx/GibbsTTS)
|
| 13 |
+
|
| 14 |
+
## Model Description
|
| 15 |
+
GibbsTTS addresses two primary limitations of MI-DFM:
|
| 16 |
+
1. **Kinetic-Optimal Scheduler:** A training-free numerical schedule that traverses the probability path at constant Fisher-Rao speed, replacing heuristic schedulers.
|
| 17 |
+
2. **Moment Correction:** A finite-step correction that adjusts jump probabilities to reduce errors while preserving the jump destination distribution.
|
| 18 |
+
|
| 19 |
+
The model was trained on the [Emilia](https://huggingface.co/datasets/amphion/Emilia-Dataset) dataset and supports English and Chinese Mandarin.
|
| 20 |
+
|
| 21 |
+
## Sample Usage
|
| 22 |
+
|
| 23 |
+
To use the model, first clone the [official repository](https://github.com/ydqmkkx/GibbsTTS) and install the requirements.
|
| 24 |
+
|
| 25 |
+
```python
|
| 26 |
+
from models import GibbsTTS
|
| 27 |
+
from config import ModelConfig
|
| 28 |
+
from IPython.display import Audio
|
| 29 |
+
import soundfile as sf
|
| 30 |
+
|
| 31 |
+
# Initialize model
|
| 32 |
+
configs = ModelConfig()
|
| 33 |
+
model = GibbsTTS(configs)
|
| 34 |
+
|
| 35 |
+
# English Synthesis
|
| 36 |
+
prompt_audio = "./prompt_examples/common_voice_en_188092-common_voice_en_188093.wav"
|
| 37 |
+
prompt_text = "This man looked exactly the same, except that now the roles were reversed."
|
| 38 |
+
target_text = "He also tried to remember some good stories to relate as he sheared the sheep."
|
| 39 |
+
language = "en"
|
| 40 |
+
|
| 41 |
+
audio = model.synthesize(prompt_audio=prompt_audio, prompt_text=prompt_text, target_text=target_text, language=language)
|
| 42 |
+
|
| 43 |
+
# Save and play
|
| 44 |
+
sf.write('target.wav', audio, 24000, 'PCM_24')
|
| 45 |
+
Audio(data=audio, rate=24000)
|
| 46 |
+
```
|
| 47 |
+
|
| 48 |
+
## Citation
|
| 49 |
+
```bibtex
|
| 50 |
+
@article{GibbsTTS,
|
| 51 |
+
author = {Dong Yang and Yiyi Cai and Haoyu Zhang and Yuki Saito and Hiroshi Saruwatari},
|
| 52 |
+
title = {Kinetic-Optimal Scheduling with Moment Correction for Metric-Induced Discrete Flow Matching in Zero-Shot Text-to-Speech},
|
| 53 |
+
year = {2026},
|
| 54 |
+
journal = {arXiv preprint arXiv:2605.09386},
|
| 55 |
+
}
|
| 56 |
+
```
|