File size: 10,858 Bytes
df72799 88a06b3 df72799 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 | ---
language:
- mg
license: cc-by-nc-sa-4.0
pipeline_tag: text-to-speech
tags:
- text-to-speech
- tts
- voice-cloning
- zero-shot
- malagasy
- plt
- low-resource
- onnx
- bluetts
base_model: notmax123/blue-v2
datasets:
- mimba/multivoice-plt
---
# BlueTTS — Malagasy (`plt`), clonage zero-shot preserve
Finetune de [BlueTTS](https://github.com/maxmelichov/BlueTTS) (base `notmax123/blue-v2`)
pour parler **malgache (Plateau Malagasy, `plt`)** **sans perdre** la capacite de
**clonage vocal zero-shot** du modele de base.
Chaque checkpoint est publie avec son **bundle ONNX** (inference CPU), ses **audios
d'evaluation** et ses **metriques** — rien n'est efface, tu peux choisir le tien.
## Le probleme resolu
Finetuner un TTS de clonage sur **peu de locuteurs** detruit l'espace locuteur : le modele
oublie le clonage et **repete des syllabes**. Un run precedent sur **4 voix** l'a montre
sans ambiguite : `sim_ecapa` s'effondrait de **0.45 a 0.22 en 1 000 steps**.
La parade retenue : entrainer sur **524 voix distinctes** parlant un malgache a accent natif
([`mimba/multivoice-plt`](https://huggingface.co/datasets/mimba/multivoice-plt), 26 200 clips),
pour que le modele apprenne a **separer le contenu du locuteur**. La diversite remplace le gel.
## Evolution
| step | loss | sim moy | min | max | vues | inedites |
|---:|---:|---:|---:|---:|---:|---:|
| 0 | — | **0.349** | 0.230 | 0.494 | 0.349 | — |
| 1000 | — | **0.401** | 0.316 | 0.487 | 0.401 | — |
| 2000 | 0.233 | **0.371** | 0.278 | 0.451 | 0.371 | — |
| 3000 | 0.231 | **0.365** | 0.293 | 0.448 | 0.365 | — |
| 4000 | 0.230 | **0.358** | 0.265 | 0.468 | 0.358 | — |
| 5000 | 0.230 | **0.335** | 0.255 | 0.415 | 0.335 | — |
| 6000 | 0.229 | **0.348** | 0.267 | 0.418 | 0.348 | — |
| 7000 | 0.228 | **0.345** | 0.210 | 0.409 | 0.345 | — |
| 8000 | 0.226 | **0.321** | 0.191 | 0.415 | 0.321 | — |
| 9000 | 0.226 | **0.300** | 0.176 | 0.366 | 0.300 | — |
| 10000 | 0.226 | **0.324** | 0.144 | 0.426 | 0.300 | 0.335 |
| 11000 | 0.227 | **0.326** | 0.125 | 0.432 | 0.318 | 0.331 |
| 12000 | 0.225 | **0.324** | 0.138 | 0.448 | 0.305 | 0.333 |
| 13000 | 0.224 | **0.325** | 0.146 | 0.449 | 0.312 | 0.332 |
| 14000 | 0.225 | **0.323** | 0.148 | 0.434 | 0.301 | 0.333 |
- **sim moy** — similarite locuteur ECAPA-TDNN (`speechbrain/spkrec-ecapa-voxceleb`),
cosinus entre l'audio genere et l'audio de reference, moyenne sur 4 references x N phrases.
- **vues / inedites** — phrases d'evaluation initiales vs phrases **jamais vues** a
l'entrainement (mesure de generalisation).
Au step 14000, les phrases **inedites** obtiennent **0.333** contre **0.301** pour les phrases d'evaluation initiales : l'ecart (+0.032) montre que le modele **generalise** et ne recite pas son corpus.
| Checkpoint | Voix1_p1 | Voix1_p2 | Voix2_p1 | Voix2_p2 | Voix3_p1 | Voix3_p2 | Voix4_p1 | Voix4_p2 |
| :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- |
| **step_7000** | <audio controls src="https://huggingface.co/mimba/bluetts-plt/resolve/main/eval/step_7000/voix1_p1.wav"></audio> | <audio controls src="https://huggingface.co/mimba/bluetts-plt/resolve/main/eval/step_7000/voix1_p2.wav"></audio> | <audio controls src="https://huggingface.co/mimba/bluetts-plt/resolve/main/eval/step_7000/voix2_p1.wav"></audio> | <audio controls src="https://huggingface.co/mimba/bluetts-plt/resolve/main/eval/step_7000/voix2_p2.wav"></audio> | <audio controls src="https://huggingface.co/mimba/bluetts-plt/resolve/main/eval/step_7000/voix3_p1.wav"></audio> | <audio controls src="https://huggingface.co/mimba/bluetts-plt/resolve/main/eval/step_7000/voix3_p2.wav"></audio> | <audio controls src="https://huggingface.co/mimba/bluetts-plt/resolve/main/eval/step_7000/voix4_p1.wav"></audio> | <audio controls src="https://huggingface.co/mimba/bluetts-plt/resolve/main/eval/step_7000/voix4_p2.wav"></audio> |
| **step_8000** | <audio controls src="https://huggingface.co/mimba/bluetts-plt/resolve/main/eval/step_8000/voix1_p1.wav"></audio> | <audio controls src="https://huggingface.co/mimba/bluetts-plt/resolve/main/eval/step_8000/voix1_p2.wav"></audio> | <audio controls src="https://huggingface.co/mimba/bluetts-plt/resolve/main/eval/step_8000/voix2_p1.wav"></audio> | <audio controls src="https://huggingface.co/mimba/bluetts-plt/resolve/main/eval/step_8000/voix2_p2.wav"></audio> | <audio controls src="https://huggingface.co/mimba/bluetts-plt/resolve/main/eval/step_8000/voix3_p1.wav"></audio> | <audio controls src="https://huggingface.co/mimba/bluetts-plt/resolve/main/eval/step_8000/voix3_p2.wav"></audio> | <audio controls src="https://huggingface.co/mimba/bluetts-plt/resolve/main/eval/step_8000/voix4_p1.wav"></audio> | <audio controls src="https://huggingface.co/mimba/bluetts-plt/resolve/main/eval/step_8000/voix4_p2.wav"></audio> |
| **step_10000** | <audio controls src="https://huggingface.co/mimba/bluetts-plt/resolve/main/eval/step_10000/voix1_p1.wav"></audio> | <audio controls src="https://huggingface.co/mimba/bluetts-plt/resolve/main/eval/step_10000/voix1_p2.wav"></audio> | <audio controls src="https://huggingface.co/mimba/bluetts-plt/resolve/main/eval/step_10000/voix2_p1.wav"></audio> | <audio controls src="https://huggingface.co/mimba/bluetts-plt/resolve/main/eval/step_10000/voix2_p2.wav"></audio> | <audio controls src="https://huggingface.co/mimba/bluetts-plt/resolve/main/eval/step_10000/voix3_p1.wav"></audio> | <audio controls src="https://huggingface.co/mimba/bluetts-plt/resolve/main/eval/step_10000/voix3_p2.wav"></audio> | <audio controls src="https://huggingface.co/mimba/bluetts-plt/resolve/main/eval/step_10000/voix4_p1.wav"></audio> | <audio controls src="https://huggingface.co/mimba/bluetts-plt/resolve/main/eval/step_10000/voix4_p2.wav"></audio> |
| **step_11000** | <audio controls src="https://huggingface.co/mimba/bluetts-plt/resolve/main/eval/step_11000/voix1_p1.wav"></audio> | <audio controls src="https://huggingface.co/mimba/bluetts-plt/resolve/main/eval/step_11000/voix1_p2.wav"></audio> | <audio controls src="https://huggingface.co/mimba/bluetts-plt/resolve/main/eval/step_11000/voix2_p1.wav"></audio> | <audio controls src="https://huggingface.co/mimba/bluetts-plt/resolve/main/eval/step_11000/voix2_p2.wav"></audio> | <audio controls src="https://huggingface.co/mimba/bluetts-plt/resolve/main/eval/step_11000/voix3_p1.wav"></audio> | <audio controls src="https://huggingface.co/mimba/bluetts-plt/resolve/main/eval/step_11000/voix3_p2.wav"></audio> | <audio controls src="https://huggingface.co/mimba/bluetts-plt/resolve/main/eval/step_11000/voix4_p1.wav"></audio> | <audio controls src="https://huggingface.co/mimba/bluetts-plt/resolve/main/eval/step_11000/voix4_p2.wav"></audio> |
| **step_13000** | <audio controls src="https://huggingface.co/mimba/bluetts-plt/resolve/main/eval/step_13000/voix1_p1.wav"></audio> | <audio controls src="https://huggingface.co/mimba/bluetts-plt/resolve/main/eval/step_13000/voix1_p2.wav"></audio> | <audio controls src="https://huggingface.co/mimba/bluetts-plt/resolve/main/eval/step_13000/voix2_p1.wav"></audio> | <audio controls src="https://huggingface.co/mimba/bluetts-plt/resolve/main/eval/step_13000/voix2_p2.wav"></audio> | <audio controls src="https://huggingface.co/mimba/bluetts-plt/resolve/main/eval/step_13000/voix3_p1.wav"></audio> | <audio controls src="https://huggingface.co/mimba/bluetts-plt/resolve/main/eval/step_13000/voix3_p2.wav"></audio> | <audio controls src="https://huggingface.co/mimba/bluetts-plt/resolve/main/eval/step_13000/voix4_p1.wav"></audio> | <audio controls src="https://huggingface.co/mimba/bluetts-plt/resolve/main/eval/step_13000/voix4_p2.wav"></audio> |
| **step_14000** | <audio controls src="https://huggingface.co/mimba/bluetts-plt/resolve/main/eval/step_14000/voix1_p1.wav"></audio> | <audio controls src="https://huggingface.co/mimba/bluetts-plt/resolve/main/eval/step_14000/voix1_p2.wav"></audio> | <audio controls src="https://huggingface.co/mimba/bluetts-plt/resolve/main/eval/step_14000/voix2_p1.wav"></audio> | <audio controls src="https://huggingface.co/mimba/bluetts-plt/resolve/main/eval/step_14000/voix2_p2.wav"></audio> | <audio controls src="https://huggingface.co/mimba/bluetts-plt/resolve/main/eval/step_14000/voix3_p1.wav"></audio> | <audio controls src="https://huggingface.co/mimba/bluetts-plt/resolve/main/eval/step_14000/voix3_p2.wav"></audio> | <audio controls src="https://huggingface.co/mimba/bluetts-plt/resolve/main/eval/step_14000/voix4_p1.wav"></audio> | <audio controls src="https://huggingface.co/mimba/bluetts-plt/resolve/main/eval/step_14000/voix4_p2.wav"></audio> |
## Recette d'entrainement
| | |
|---|---|
| Base | `notmax123/blue-v2` (AE `blue_codec.safetensors`) |
| Dataset | `mimba/multivoice-plt` — 26 197 clips, **524 locuteurs** (50 chacun) |
| Texte | IPA pre-phonemise (`text_phonemized`), `plt` |
| Duration Predictor | reentraine **de zero** sur les 524 voix, 15 000 steps (loss 1.3 -> 0.059) |
| TTL | finetune, batch 7 x accumulation 8 (batch effectif 56) |
| LR | 2.5e-4, reduit a **1.25e-4** apres plateau de loss + baisse de `sim_ecapa` |
| Checkpoints | tous les 1 000 steps, exportes en ONNX et evalues automatiquement |
**Pourquoi le DP est reentraine** : c'est lui qui portait le begaiement du run 4-voix
(il est conditionne par le locuteur). Sur 524 voix, il apprend un rythme robuste.
## Utilisation (ONNX, CPU)
```python
from huggingface_hub import snapshot_download
d = snapshot_download("mimba/bluetts-plt", allow_patterns=["eval/step_14000/onnx/*"])
# -> text_encoder / vector_estimator / vocoder / duration_predictor
# + codec_encoder / style_encoder / duration_style_encoder (clonage zero-shot)
```
Parametres de production (worker Mimba) : `total_step=8..16`, `speed=0.95`,
`cfg_scale=4.0`, `pace_blend=0.30`. Normaliser la sortie avant ecriture
(RMS 0.08 / peak 0.95) : le vocodeur sort a un pic ~1.5 et s'ecrete sinon.
Le vocabulaire BlueTTS ne contient pas les marqueurs de prenasalisation malgaches
`m` en exposant et `n` en exposant : les remplacer par `m` / `n` avant encodage.
## Limites
- **Fidelite de clonage inferieure a la base** : `sim_ecapa` passe de ~0.35 (step 0)
a ~0.32 apres finetune. La degradation a ete **stoppee** (LR reduit), pas annulee.
- **Variable selon la voix** : certaines references se clonent nettement moins bien
(voir l'ecart min/max du tableau) — tester avec ses propres references.
- **Monolingue** : entraine pour le malgache uniquement.
- **Donnees synthetiques** : le corpus multi-voix est genere (OmniVoice + OpenVoice),
seul l'accent provient de locuteurs natifs reels.
## Licence
`cc-by-nc-sa-4.0` — usage non commercial. Verifier independamment les licences de
BlueTTS, OmniVoice et OpenVoice avant tout usage commercial.
##### *Contact : [@Mimba](baounabaouna@gmail.com)*
|