Upload README.md with huggingface_hub
Browse files
README.md
ADDED
|
@@ -0,0 +1,229 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
language: eu
|
| 3 |
+
license: apache-2.0
|
| 4 |
+
tags:
|
| 5 |
+
- text-to-speech
|
| 6 |
+
- basque
|
| 7 |
+
- styletts2
|
| 8 |
+
- multispeaker
|
| 9 |
+
- emotional
|
| 10 |
+
---
|
| 11 |
+
|
| 12 |
+
# StyleTTS2 — Basque Multispeaker Emotional TTS
|
| 13 |
+
|
| 14 |
+
This is a Basque text-to-speech (TTS) model based on the [StyleTTS2](https://github.com/yl4579/StyleTTS2) architecture, adapted for **emotional Basque speech synthesis**. The model supports three emotional styles: neutral, happy (poza), and sad (tristura).
|
| 15 |
+
|
| 16 |
+
Examples (playable):
|
| 17 |
+
|
| 18 |
+
- **Sample 1 — Antton (Neutral)** — "Suhiltzaileek esan dutenez, hiru hildakoek jauzi egin zuten puzgarria su hartzen ari zela ikusita."
|
| 19 |
+
|
| 20 |
+
<audio controls src="https://huggingface.co/HiTZ/StyleTTS2-eu_emo/resolve/main/sample_antton_neutral.wav">Your browser does not support the audio element.</audio>
|
| 21 |
+
|
| 22 |
+
- **Sample 1 — Antton (Happy)** — "Suhiltzaileek esan dutenez, hiru hildakoek jauzi egin zuten puzgarria su hartzen ari zela ikusita."
|
| 23 |
+
|
| 24 |
+
<audio controls src="https://huggingface.co/HiTZ/StyleTTS2-eu_emo/resolve/main/sample_antton_pozik.wav">Your browser does not support the audio element.</audio>
|
| 25 |
+
|
| 26 |
+
- **Sample 1 — Antton (Sad)** — "Suhiltzaileek esan dutenez, hiru hildakoek jauzi egin zuten puzgarria su hartzen ari zela ikusita."
|
| 27 |
+
|
| 28 |
+
<audio controls src="https://huggingface.co/HiTZ/StyleTTS2-eu_emo/resolve/main/sample_antton_triste.wav">Your browser does not support the audio element.</audio>
|
| 29 |
+
|
| 30 |
+
- **Sample 2 — Maider (Neutral)** — "Gure patua hau izatea litekeena da, baina okerra deritzot."
|
| 31 |
+
|
| 32 |
+
<audio controls src="https://huggingface.co/HiTZ/StyleTTS2-eu_emo/resolve/main/sample_maider_neutral.wav">Your browser does not support the audio element.</audio>
|
| 33 |
+
|
| 34 |
+
- **Sample 2 — Maider (Happy)** — "Gure patua hau izatea litekeena da, baina okerra deritzot."
|
| 35 |
+
|
| 36 |
+
<audio controls src="https://huggingface.co/HiTZ/StyleTTS2-eu_emo/resolve/main/sample_maider_pozik.wav">Your browser does not support the audio element.</audio>
|
| 37 |
+
|
| 38 |
+
- **Sample 2 — Maider (Sad)** — "Gure patua hau izatea litekeena da, baina okerra deritzot."
|
| 39 |
+
|
| 40 |
+
<audio controls src="https://huggingface.co/HiTZ/StyleTTS2-eu_emo/resolve/main/sample_maider_triste.wav">Your browser does not support the audio element.</audio>
|
| 41 |
+
|
| 42 |
+
Main modifications:
|
| 43 |
+
- [PL-BERT-eu](https://huggingface.co/HiTZ/PL-BERT-wp-eu): PL-BERT model trained with WordPiece tokenizer for phonemized Basque text.
|
| 44 |
+
- ASR-eu: ASR model trained with a subset of the multispeaker speech corpus. It uses the same architecture as the original [ASR](https://github.com/yl4579/AuxiliaryASR) from StyleTTS2.
|
| 45 |
+
- Phonemizer: We used code developed by [Aholab](https://aholab.ehu.eus/aholab/) to generate IPA phonemes for training the model. You can see a demo of the Basque phonemizer at [arrandi/phonemizer-eus-esp](https://huggingface.co/spaces/arrandi/phonemizer-eus-esp). Likewise, the code used to generate IPA phonemes can be found in the `phonemizer` directory. We collapsed multi-character phonemes into single-character phonemes for better grapheme–phoneme alignment.
|
| 46 |
+
|
| 47 |
+
## Emotions
|
| 48 |
+
|
| 49 |
+
| Emotion | Basque Tag | Description |
|
| 50 |
+
|---------|------------|-------------|
|
| 51 |
+
| Neutral | `neu` | Neutral/calm delivery |
|
| 52 |
+
| Happy | `poz` | Happy/expressive delivery |
|
| 53 |
+
| Sad | `tri` | Sad/contemplative delivery |
|
| 54 |
+
|
| 55 |
+
## Model details
|
| 56 |
+
|
| 57 |
+
| | |
|
| 58 |
+
|---|---|
|
| 59 |
+
| Architecture | StyleTTS2 (from scratch) |
|
| 60 |
+
| Language | Basque (`eu`) |
|
| 61 |
+
| Speakers | Multispeaker (two speakers: Antton, Maider) |
|
| 62 |
+
| Emotions | Neutral, Happy (Poza), Sad (Tristura) |
|
| 63 |
+
| Text input | Basque IPA phonemes |
|
| 64 |
+
| Speech LM | [WavLM-Base-Plus](https://huggingface.co/microsoft/wavlm-base-plus) |
|
| 65 |
+
| Sample rate | 24 000 Hz |
|
| 66 |
+
| Decoder | HiFiGAN |
|
| 67 |
+
|
| 68 |
+
## Training dataset
|
| 69 |
+
|
| 70 |
+
[HiTZ-Aholab re-enacted spontaneous speech corpus in Basque](https://zenodo.org/records/20638013) — emotional speech corpus.
|
| 71 |
+
|
| 72 |
+
- **Number of speakers:** two (Antton, Maider)
|
| 73 |
+
- **Audio:** 2,500 utterances per speaker, totalling approximately 7 hours and 29 minutes
|
| 74 |
+
- Maider: ~3h 32min
|
| 75 |
+
- Antton: ~3h 57min
|
| 76 |
+
- **Emotions:** Neutral, Happy (Poza), Sad (Tristura)
|
| 77 |
+
- **Dataset split:** 100 samples for validation, 600 for testing (300 per speaker)
|
| 78 |
+
- **OOD dataset:** A different text dataset used as Out-of-Distribution dataset
|
| 79 |
+
|
| 80 |
+
## Training
|
| 81 |
+
|
| 82 |
+
Brief summary of training parameters used (from `config_basque_multispeaker_phoneme_wavlm_emo_plbertemo_no_acc.yml`):
|
| 83 |
+
|
| 84 |
+
- **Device:** cuda
|
| 85 |
+
- **Stages:** 1st-stage epochs = 50; 2nd-stage epochs = 30
|
| 86 |
+
- **Batch:** batch_size = 1
|
| 87 |
+
- **Max length:** max_len = 500
|
| 88 |
+
- **Learning rates:** lr = 0.0001; bert_lr = 1e-5; ft_lr = 1e-5
|
| 89 |
+
- **Audio / features:** sr = 24000; n_mels = 80; spectrogram (n_fft=2048, win_length=1200, hop_length=300)
|
| 90 |
+
- **Model:** multispeaker = true; n_token = 178 (phonemes); style_dim = 128; decoder = HiFiGAN
|
| 91 |
+
- **Diffusion / schedule:** diff_epoch = 10; joint_epoch = 15; estimate_sigma_data = true (sigma ≈ 0.2)
|
| 92 |
+
- **Loss highlights:** lambda_mel = 5.0; lambda_ce = 20.0; lambda_diff = 1.0
|
| 93 |
+
|
| 94 |
+
## Files in this repository
|
| 95 |
+
|
| 96 |
+
| File | Description |
|
| 97 |
+
|---|---|
|
| 98 |
+
| `config_basque_multispeaker_phoneme_wavlm_emo_plbertemo_no_acc.yml` | Training & model config → place at `Models/Basque_Multispeaker_Phoneme_wavlm_emo_plbertemo_no_acc` |
|
| 99 |
+
| `epoch_2nd_00030.pth` | Main TTS checkpoint → place at `Models/Basque_Multispeaker_Phoneme_wavlm_emo/` |
|
| 100 |
+
| `epoch_00200.pth` | Basque ASR / text aligner → place at `Utils/ASR_basque/` |
|
| 101 |
+
| `step_4000000.t7` | Phoneme PLBERT → place at `Utils/PLBERT_phoneme/` |
|
| 102 |
+
|
| 103 |
+
> **Note:** The JDC F0 extractor (`Utils/JDC/bst.t7`) is not Basque-specific — download it from the original [StyleTTS2 repository](https://github.com/yl4579/StyleTTS2) and place it at `Utils/JDC/bst.t7`.
|
| 104 |
+
|
| 105 |
+
## Setup
|
| 106 |
+
|
| 107 |
+
```bash
|
| 108 |
+
# 1. Clone the code repository
|
| 109 |
+
git clone https://github.com/AArriandiaga/StyleTTS2_basque
|
| 110 |
+
cd StyleTTS2_basque
|
| 111 |
+
|
| 112 |
+
# 2. Install dependencies
|
| 113 |
+
pip install -r requirements.txt
|
| 114 |
+
|
| 115 |
+
# 3. Download model weights from this HF repo and place them:
|
| 116 |
+
mkdir -p Models/Basque_Multispeaker_Phoneme_wavlm_emo Utils/ASR_basque Utils/PLBERT_phoneme Utils/JDC
|
| 117 |
+
# Download bst.t7 from the original StyleTTS2 repo (not Basque-specific):
|
| 118 |
+
wget -P Utils/JDC https://github.com/yl4579/StyleTTS2/raw/main/Utils/JDC/bst.t7
|
| 119 |
+
|
| 120 |
+
# using huggingface_hub:
|
| 121 |
+
python - <<'EOF'
|
| 122 |
+
from huggingface_hub import hf_hub_download
|
| 123 |
+
import shutil
|
| 124 |
+
|
| 125 |
+
repo = "HiTZ/StyleTTS2-eu_emo"
|
| 126 |
+
files = {
|
| 127 |
+
"config_basque_multispeaker_phoneme_wavlm_emo_plbertemo_no_acc.yml": "Models/Basque_Multispeaker_Phoneme_wavlm_emo/config.yml",
|
| 128 |
+
"epoch_2nd_00030.pth": "Models/Basque_Multispeaker_Phoneme_wavlm_emo/epoch_2nd_00030.pth",
|
| 129 |
+
"epoch_00200.pth": "Utils/ASR_basque/epoch_00200.pth",
|
| 130 |
+
"step_4000000.t7": "Utils/PLBERT_phoneme/step_4000000.t7",
|
| 131 |
+
}
|
| 132 |
+
# bst.t7 comes from the original StyleTTS2 repo — download separately:
|
| 133 |
+
# https://github.com/yl4579/StyleTTS2/tree/main/Utils/JDC
|
| 134 |
+
for hf_name, local_path in files.items():
|
| 135 |
+
src = hf_hub_download(repo_id=repo, filename=hf_name)
|
| 136 |
+
shutil.copy(src, local_path)
|
| 137 |
+
print(f"✓ {local_path}")
|
| 138 |
+
EOF
|
| 139 |
+
```
|
| 140 |
+
|
| 141 |
+
## Inference
|
| 142 |
+
|
| 143 |
+
**CLI:**
|
| 144 |
+
```bash
|
| 145 |
+
python inference.py \
|
| 146 |
+
--config Models/Basque_Multispeaker_Phoneme_wavlm_emo/config.yml \
|
| 147 |
+
--model Models/Basque_Multispeaker_Phoneme_wavlm_emo/epoch_2nd_00030.pth \
|
| 148 |
+
--ref Demo/ref_antton_poz.wav \
|
| 149 |
+
--text "Kaixo, zelan zaude?" \
|
| 150 |
+
--output output/kaixo.wav
|
| 151 |
+
```
|
| 152 |
+
|
| 153 |
+
**Python API:**
|
| 154 |
+
```python
|
| 155 |
+
from inference import Synthesizer
|
| 156 |
+
|
| 157 |
+
synth = Synthesizer(
|
| 158 |
+
config='Models/Basque_Multispeaker_Phoneme_wavlm_emo/config.yml',
|
| 159 |
+
checkpoint='Models/Basque_Multispeaker_Phoneme_wavlm_emo/epoch_2nd_00030.pth',
|
| 160 |
+
default_ref='Demo/ref_antton_neu.wav',
|
| 161 |
+
)
|
| 162 |
+
|
| 163 |
+
# Neutral emotion
|
| 164 |
+
wav = synth.run("Kaixo, zelan zaude?", ref='Demo/ref_antton_neu.wav')
|
| 165 |
+
synth.save(wav, "output/kaixo_neu.wav")
|
| 166 |
+
|
| 167 |
+
# Happy emotion (using poza reference)
|
| 168 |
+
wav2 = synth.run("Zorioneko gara!", ref='Demo/ref_antton_poz.wav')
|
| 169 |
+
synth.save(wav2, "output/kaixo_poz.wav")
|
| 170 |
+
|
| 171 |
+
# Sad emotion (using tristura reference)
|
| 172 |
+
wav3 = synth.run("Hau oso tristea da.", ref='Demo/ref_antton_tri.wav')
|
| 173 |
+
synth.save(wav3, "output/kaixo_tri.wav")
|
| 174 |
+
```
|
| 175 |
+
|
| 176 |
+
Key parameters for `run()`:
|
| 177 |
+
|
| 178 |
+
| Parameter | Default | Description |
|
| 179 |
+
|---|---|---|
|
| 180 |
+
| `ref` | constructor default | Reference WAV for speaker & emotion style |
|
| 181 |
+
| `alpha` | 0.3 | Timbre mixing (0 = reference, 1 = sampled) |
|
| 182 |
+
| `beta` | 0.7 | Prosody mixing (0 = reference, 1 = sampled) |
|
| 183 |
+
| `diffusion_steps` | 5 | Quality vs. speed trade-off |
|
| 184 |
+
| `embedding_scale` | 1.0 | Expressiveness (>1 = more expressive) |
|
| 185 |
+
|
| 186 |
+
## Reference speakers
|
| 187 |
+
|
| 188 |
+
Six reference audios are included in the repo under `Demo/`, covering both speakers and all three emotions:
|
| 189 |
+
|
| 190 |
+
| Speaker | Neutral | Happy | Sad |
|
| 191 |
+
|---------|---------|-------|-----|
|
| 192 |
+
| Antton (male) | `ref_antton_neu.wav` | `ref_antton_poz.wav` | `ref_antton_tri.wav` |
|
| 193 |
+
| Maider (female) | `ref_maider_neu.wav` | `ref_maider_poz.wav` | `ref_maider_tri.wav` |
|
| 194 |
+
|
| 195 |
+
All credit goes to the authors of StyleTTS2.
|
| 196 |
+
|
| 197 |
+
## Citation
|
| 198 |
+
|
| 199 |
+
```bibtex
|
| 200 |
+
@inproceedings{li2023styletts2,
|
| 201 |
+
title = {StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models},
|
| 202 |
+
author = {Li, Yinghao Aaron and Han, Cong and Mesgarani, Nima},
|
| 203 |
+
booktitle = {Advances in Neural Information Processing Systems},
|
| 204 |
+
year = {2023},
|
| 205 |
+
}
|
| 206 |
+
```
|
| 207 |
+
|
| 208 |
+
## Additional Information
|
| 209 |
+
|
| 210 |
+
### Authors
|
| 211 |
+
|
| 212 |
+
- [Ander Arriandiaga](https://huggingface.co/arrandi) — Aholab (HiTZ), EHU
|
| 213 |
+
- [Inmaculada Hernáez Rioja](mailto:inma.hernaez@ehu.eus) — Aholab (HiTZ), EHU
|
| 214 |
+
|
| 215 |
+
### Contact
|
| 216 |
+
|
| 217 |
+
For further information, please send an email to <inma.hernaez@ehu.eus>.
|
| 218 |
+
|
| 219 |
+
### Copyright
|
| 220 |
+
|
| 221 |
+
Copyright(c) 2026 by Aholab, HiTZ.
|
| 222 |
+
|
| 223 |
+
### License
|
| 224 |
+
|
| 225 |
+
[Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0)
|
| 226 |
+
|
| 227 |
+
### Funding
|
| 228 |
+
|
| 229 |
+
This work is funded by the Ministerio para la Transformación Digital y de la Función Pública - Funded by EU – NextGenerationEU within the framework of the project Desarrollo de Modelos ALIA.
|