arrandi commited on
Commit
44b5270
·
verified ·
1 Parent(s): 682334a

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +229 -0
README.md ADDED
@@ -0,0 +1,229 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language: eu
3
+ license: apache-2.0
4
+ tags:
5
+ - text-to-speech
6
+ - basque
7
+ - styletts2
8
+ - multispeaker
9
+ - emotional
10
+ ---
11
+
12
+ # StyleTTS2 — Basque Multispeaker Emotional TTS
13
+
14
+ This is a Basque text-to-speech (TTS) model based on the [StyleTTS2](https://github.com/yl4579/StyleTTS2) architecture, adapted for **emotional Basque speech synthesis**. The model supports three emotional styles: neutral, happy (poza), and sad (tristura).
15
+
16
+ Examples (playable):
17
+
18
+ - **Sample 1 — Antton (Neutral)** — "Suhiltzaileek esan dutenez, hiru hildakoek jauzi egin zuten puzgarria su hartzen ari zela ikusita."
19
+
20
+ <audio controls src="https://huggingface.co/HiTZ/StyleTTS2-eu_emo/resolve/main/sample_antton_neutral.wav">Your browser does not support the audio element.</audio>
21
+
22
+ - **Sample 1 — Antton (Happy)** — "Suhiltzaileek esan dutenez, hiru hildakoek jauzi egin zuten puzgarria su hartzen ari zela ikusita."
23
+
24
+ <audio controls src="https://huggingface.co/HiTZ/StyleTTS2-eu_emo/resolve/main/sample_antton_pozik.wav">Your browser does not support the audio element.</audio>
25
+
26
+ - **Sample 1 — Antton (Sad)** — "Suhiltzaileek esan dutenez, hiru hildakoek jauzi egin zuten puzgarria su hartzen ari zela ikusita."
27
+
28
+ <audio controls src="https://huggingface.co/HiTZ/StyleTTS2-eu_emo/resolve/main/sample_antton_triste.wav">Your browser does not support the audio element.</audio>
29
+
30
+ - **Sample 2 — Maider (Neutral)** — "Gure patua hau izatea litekeena da, baina okerra deritzot."
31
+
32
+ <audio controls src="https://huggingface.co/HiTZ/StyleTTS2-eu_emo/resolve/main/sample_maider_neutral.wav">Your browser does not support the audio element.</audio>
33
+
34
+ - **Sample 2 — Maider (Happy)** — "Gure patua hau izatea litekeena da, baina okerra deritzot."
35
+
36
+ <audio controls src="https://huggingface.co/HiTZ/StyleTTS2-eu_emo/resolve/main/sample_maider_pozik.wav">Your browser does not support the audio element.</audio>
37
+
38
+ - **Sample 2 — Maider (Sad)** — "Gure patua hau izatea litekeena da, baina okerra deritzot."
39
+
40
+ <audio controls src="https://huggingface.co/HiTZ/StyleTTS2-eu_emo/resolve/main/sample_maider_triste.wav">Your browser does not support the audio element.</audio>
41
+
42
+ Main modifications:
43
+ - [PL-BERT-eu](https://huggingface.co/HiTZ/PL-BERT-wp-eu): PL-BERT model trained with WordPiece tokenizer for phonemized Basque text.
44
+ - ASR-eu: ASR model trained with a subset of the multispeaker speech corpus. It uses the same architecture as the original [ASR](https://github.com/yl4579/AuxiliaryASR) from StyleTTS2.
45
+ - Phonemizer: We used code developed by [Aholab](https://aholab.ehu.eus/aholab/) to generate IPA phonemes for training the model. You can see a demo of the Basque phonemizer at [arrandi/phonemizer-eus-esp](https://huggingface.co/spaces/arrandi/phonemizer-eus-esp). Likewise, the code used to generate IPA phonemes can be found in the `phonemizer` directory. We collapsed multi-character phonemes into single-character phonemes for better grapheme–phoneme alignment.
46
+
47
+ ## Emotions
48
+
49
+ | Emotion | Basque Tag | Description |
50
+ |---------|------------|-------------|
51
+ | Neutral | `neu` | Neutral/calm delivery |
52
+ | Happy | `poz` | Happy/expressive delivery |
53
+ | Sad | `tri` | Sad/contemplative delivery |
54
+
55
+ ## Model details
56
+
57
+ | | |
58
+ |---|---|
59
+ | Architecture | StyleTTS2 (from scratch) |
60
+ | Language | Basque (`eu`) |
61
+ | Speakers | Multispeaker (two speakers: Antton, Maider) |
62
+ | Emotions | Neutral, Happy (Poza), Sad (Tristura) |
63
+ | Text input | Basque IPA phonemes |
64
+ | Speech LM | [WavLM-Base-Plus](https://huggingface.co/microsoft/wavlm-base-plus) |
65
+ | Sample rate | 24 000 Hz |
66
+ | Decoder | HiFiGAN |
67
+
68
+ ## Training dataset
69
+
70
+ [HiTZ-Aholab re-enacted spontaneous speech corpus in Basque](https://zenodo.org/records/20638013) — emotional speech corpus.
71
+
72
+ - **Number of speakers:** two (Antton, Maider)
73
+ - **Audio:** 2,500 utterances per speaker, totalling approximately 7 hours and 29 minutes
74
+ - Maider: ~3h 32min
75
+ - Antton: ~3h 57min
76
+ - **Emotions:** Neutral, Happy (Poza), Sad (Tristura)
77
+ - **Dataset split:** 100 samples for validation, 600 for testing (300 per speaker)
78
+ - **OOD dataset:** A different text dataset used as Out-of-Distribution dataset
79
+
80
+ ## Training
81
+
82
+ Brief summary of training parameters used (from `config_basque_multispeaker_phoneme_wavlm_emo_plbertemo_no_acc.yml`):
83
+
84
+ - **Device:** cuda
85
+ - **Stages:** 1st-stage epochs = 50; 2nd-stage epochs = 30
86
+ - **Batch:** batch_size = 1
87
+ - **Max length:** max_len = 500
88
+ - **Learning rates:** lr = 0.0001; bert_lr = 1e-5; ft_lr = 1e-5
89
+ - **Audio / features:** sr = 24000; n_mels = 80; spectrogram (n_fft=2048, win_length=1200, hop_length=300)
90
+ - **Model:** multispeaker = true; n_token = 178 (phonemes); style_dim = 128; decoder = HiFiGAN
91
+ - **Diffusion / schedule:** diff_epoch = 10; joint_epoch = 15; estimate_sigma_data = true (sigma ≈ 0.2)
92
+ - **Loss highlights:** lambda_mel = 5.0; lambda_ce = 20.0; lambda_diff = 1.0
93
+
94
+ ## Files in this repository
95
+
96
+ | File | Description |
97
+ |---|---|
98
+ | `config_basque_multispeaker_phoneme_wavlm_emo_plbertemo_no_acc.yml` | Training & model config → place at `Models/Basque_Multispeaker_Phoneme_wavlm_emo_plbertemo_no_acc` |
99
+ | `epoch_2nd_00030.pth` | Main TTS checkpoint → place at `Models/Basque_Multispeaker_Phoneme_wavlm_emo/` |
100
+ | `epoch_00200.pth` | Basque ASR / text aligner → place at `Utils/ASR_basque/` |
101
+ | `step_4000000.t7` | Phoneme PLBERT → place at `Utils/PLBERT_phoneme/` |
102
+
103
+ > **Note:** The JDC F0 extractor (`Utils/JDC/bst.t7`) is not Basque-specific — download it from the original [StyleTTS2 repository](https://github.com/yl4579/StyleTTS2) and place it at `Utils/JDC/bst.t7`.
104
+
105
+ ## Setup
106
+
107
+ ```bash
108
+ # 1. Clone the code repository
109
+ git clone https://github.com/AArriandiaga/StyleTTS2_basque
110
+ cd StyleTTS2_basque
111
+
112
+ # 2. Install dependencies
113
+ pip install -r requirements.txt
114
+
115
+ # 3. Download model weights from this HF repo and place them:
116
+ mkdir -p Models/Basque_Multispeaker_Phoneme_wavlm_emo Utils/ASR_basque Utils/PLBERT_phoneme Utils/JDC
117
+ # Download bst.t7 from the original StyleTTS2 repo (not Basque-specific):
118
+ wget -P Utils/JDC https://github.com/yl4579/StyleTTS2/raw/main/Utils/JDC/bst.t7
119
+
120
+ # using huggingface_hub:
121
+ python - <<'EOF'
122
+ from huggingface_hub import hf_hub_download
123
+ import shutil
124
+
125
+ repo = "HiTZ/StyleTTS2-eu_emo"
126
+ files = {
127
+ "config_basque_multispeaker_phoneme_wavlm_emo_plbertemo_no_acc.yml": "Models/Basque_Multispeaker_Phoneme_wavlm_emo/config.yml",
128
+ "epoch_2nd_00030.pth": "Models/Basque_Multispeaker_Phoneme_wavlm_emo/epoch_2nd_00030.pth",
129
+ "epoch_00200.pth": "Utils/ASR_basque/epoch_00200.pth",
130
+ "step_4000000.t7": "Utils/PLBERT_phoneme/step_4000000.t7",
131
+ }
132
+ # bst.t7 comes from the original StyleTTS2 repo — download separately:
133
+ # https://github.com/yl4579/StyleTTS2/tree/main/Utils/JDC
134
+ for hf_name, local_path in files.items():
135
+ src = hf_hub_download(repo_id=repo, filename=hf_name)
136
+ shutil.copy(src, local_path)
137
+ print(f"✓ {local_path}")
138
+ EOF
139
+ ```
140
+
141
+ ## Inference
142
+
143
+ **CLI:**
144
+ ```bash
145
+ python inference.py \
146
+ --config Models/Basque_Multispeaker_Phoneme_wavlm_emo/config.yml \
147
+ --model Models/Basque_Multispeaker_Phoneme_wavlm_emo/epoch_2nd_00030.pth \
148
+ --ref Demo/ref_antton_poz.wav \
149
+ --text "Kaixo, zelan zaude?" \
150
+ --output output/kaixo.wav
151
+ ```
152
+
153
+ **Python API:**
154
+ ```python
155
+ from inference import Synthesizer
156
+
157
+ synth = Synthesizer(
158
+ config='Models/Basque_Multispeaker_Phoneme_wavlm_emo/config.yml',
159
+ checkpoint='Models/Basque_Multispeaker_Phoneme_wavlm_emo/epoch_2nd_00030.pth',
160
+ default_ref='Demo/ref_antton_neu.wav',
161
+ )
162
+
163
+ # Neutral emotion
164
+ wav = synth.run("Kaixo, zelan zaude?", ref='Demo/ref_antton_neu.wav')
165
+ synth.save(wav, "output/kaixo_neu.wav")
166
+
167
+ # Happy emotion (using poza reference)
168
+ wav2 = synth.run("Zorioneko gara!", ref='Demo/ref_antton_poz.wav')
169
+ synth.save(wav2, "output/kaixo_poz.wav")
170
+
171
+ # Sad emotion (using tristura reference)
172
+ wav3 = synth.run("Hau oso tristea da.", ref='Demo/ref_antton_tri.wav')
173
+ synth.save(wav3, "output/kaixo_tri.wav")
174
+ ```
175
+
176
+ Key parameters for `run()`:
177
+
178
+ | Parameter | Default | Description |
179
+ |---|---|---|
180
+ | `ref` | constructor default | Reference WAV for speaker & emotion style |
181
+ | `alpha` | 0.3 | Timbre mixing (0 = reference, 1 = sampled) |
182
+ | `beta` | 0.7 | Prosody mixing (0 = reference, 1 = sampled) |
183
+ | `diffusion_steps` | 5 | Quality vs. speed trade-off |
184
+ | `embedding_scale` | 1.0 | Expressiveness (>1 = more expressive) |
185
+
186
+ ## Reference speakers
187
+
188
+ Six reference audios are included in the repo under `Demo/`, covering both speakers and all three emotions:
189
+
190
+ | Speaker | Neutral | Happy | Sad |
191
+ |---------|---------|-------|-----|
192
+ | Antton (male) | `ref_antton_neu.wav` | `ref_antton_poz.wav` | `ref_antton_tri.wav` |
193
+ | Maider (female) | `ref_maider_neu.wav` | `ref_maider_poz.wav` | `ref_maider_tri.wav` |
194
+
195
+ All credit goes to the authors of StyleTTS2.
196
+
197
+ ## Citation
198
+
199
+ ```bibtex
200
+ @inproceedings{li2023styletts2,
201
+ title = {StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models},
202
+ author = {Li, Yinghao Aaron and Han, Cong and Mesgarani, Nima},
203
+ booktitle = {Advances in Neural Information Processing Systems},
204
+ year = {2023},
205
+ }
206
+ ```
207
+
208
+ ## Additional Information
209
+
210
+ ### Authors
211
+
212
+ - [Ander Arriandiaga](https://huggingface.co/arrandi) — Aholab (HiTZ), EHU
213
+ - [Inmaculada Hernáez Rioja](mailto:inma.hernaez@ehu.eus) — Aholab (HiTZ), EHU
214
+
215
+ ### Contact
216
+
217
+ For further information, please send an email to <inma.hernaez@ehu.eus>.
218
+
219
+ ### Copyright
220
+
221
+ Copyright(c) 2026 by Aholab, HiTZ.
222
+
223
+ ### License
224
+
225
+ [Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0)
226
+
227
+ ### Funding
228
+
229
+ This work is funded by the Ministerio para la Transformación Digital y de la Función Pública - Funded by EU – NextGenerationEU within the framework of the project Desarrollo de Modelos ALIA.