Upload folder using huggingface_hub

Browse files

Files changed (10) hide show

.gitattributes +4 -0
assets/0.png +3 -0
assets/1.png +0 -0
assets/2.png +3 -0
plbert_plt_README.md +188 -0
step_1000000.t7 +3 -0
step_1000000_meta.json +1 -0
step_950000.t7 +3 -0
step_950000_meta.json +1 -0
train.log +0 -0

.gitattributes CHANGED Viewed

@@ -33,3 +33,7 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text

 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+assets/0.png filter=lfs diff=lfs merge=lfs -text
+assets/2.png filter=lfs diff=lfs merge=lfs -text
+step_1000000.t7 filter=lfs diff=lfs merge=lfs -text
+step_950000.t7 filter=lfs diff=lfs merge=lfs -text

assets/0.png ADDED Viewed

Git LFS Details

SHA256: d9d3c93e9b5d2b28bab84fcd1539baa9619657ff05e03fb2ae91a41a5bccd92b
Pointer size: 131 Bytes
Size of remote file: 340 kB

assets/1.png ADDED Viewed

assets/2.png ADDED Viewed

Git LFS Details

SHA256: e3d965e6725c1be80b04eeca39d568fe3051ff39dcf87cd04a79003acb4d96d9
Pointer size: 131 Bytes
Size of remote file: 185 kB

plbert_plt_README.md ADDED Viewed

	@@ -0,0 +1,188 @@

+---
+language:
+- plt
+license: cc-by-nc-sa-4.0
+pretty_name: Mimba PL-BERT PLT (Plateau Malagasy Phonetic-Level BERT)
+tags:
+- text-to-speech
+- pl-bert
+- plbert
+- albert
+- malagasy
+- plt
+- african-languages
+- low-resource
+- masked-language-modeling
+- phoneme
+library_name: transformers
+---
+# Mimba PL-BERT PLT — Phonetic-Level BERT for Plateau Malagasy
+A **phonetic-level pre-trained language model (PL-BERT)** for **Plateau Malagasy (PLT)**,
+trained from scratch to provide text/phoneme-context-aware embeddings for
+**StyleTTS2** synthesis. Adapted from the original
+[PL-BERT](https://github.com/yl4579/PL-BERT) architecture (Li et al., used in
+StyleTTS2) and trained on a large phonemized Malagasy corpus with the same
+55-symbol phoneme vocabulary used across every Mimba PLT model (StyleTTS2,
+NeuTTS-Nano, Supertonic 3).
+> ⚠️ **Not a standalone TTS model.** PL-BERT is a text/phoneme encoder only —
+> it produces contextual embeddings consumed by a downstream acoustic model
+> (StyleTTS2 Stage 1/Stage 2). It cannot synthesize audio by itself.
+## Summary
+| | |
+|---|---|
+| Language | Plateau Malagasy (`plt`) |
+| Architecture | ALBERT (`transformers.AlbertModel` + 2 prediction heads) |
+| Phoneme vocabulary | 55 symbols (`phoneme_symbols.pkl`, shared with StyleTTS2/NeuTTS-Nano) |
+| Hidden size | 768 |
+| Attention heads | 12 |
+| Hidden layers | 12 |
+| Intermediate size | 2048 |
+| Max position embeddings | 512 |
+| Dropout | 0.1 |
+| Training objective | Masked language modeling, dual head (phoneme-level + word-level) |
+| Training steps | 1,000,000 |
+| Final masked-phoneme accuracy | **65.52%** (measured on ~200K masked positions) |
+| Checkpoint format | `step_{N}.t7` (`{'net': state_dict, 'optimizer': ..., 'step': N}`) |
+### Loss curve
+<!DOCTYPE html>
+<html>
+<head>
+<style>
+  .conteneur-images {
+    display: flex;          /* Active le mode horizontal */
+    gap: 10px;              /* Espace de 10px entre les images */
+  }
+  .conteneur-images img {
+    width: 25%;             /* 100% / 4 images = 25% */
+    height: auto;           /* Maintient les proportions */
+    flex-shrink: 1;         /* Permet de rétrécir si besoin */
+  }
+</style>
+</head>
+<body>
+  <p class="conteneur-images">
+    <img src="assets/0.png" width="900" alt="Loss curve PL BERT">
+    <img src="assets/1.png" width="900" alt="Loss curve PL BERT">
+    <img src="assets/2.png" width="900" alt="Loss curve PL BERT">
+  </p>
+</body>
+</html>
+## Training details
+The model is trained with a dual masked-language-modeling objective — one
+head predicts the masked **phoneme token** (55-way classification), the other
+predicts the masked **word form** (large open vocabulary of Malagasy word
+forms, zipfian-distributed due to the language's agglutinative morphology).
+Only the phoneme-level task is used downstream by StyleTTS2, but the joint
+objective helps the encoder learn richer contextual representations.
+Training ran for 1,000,000 steps with a cosine learning-rate decay
+(`1e-4 → 1e-6`) applied over the final ~270K steps. Accuracy on masked
+phoneme positions (measured periodically on held-out batches, not just
+training loss) tracked as follows:
+| Step | Masked-phoneme accuracy |
+|---|---|
+| 136,000 | 58.55% |
+| 727,514 | 61–63% |
+| 900,000 | 63.92% |
+| 1,000,000 | **65.52%** |
+Accuracy plateaued in the final third of training despite the LR decay
+reaching down to `1e-6` — this is treated as the effective ceiling for this
+model size/corpus, not a sign that more steps would help. For reference,
+comparable phoneme/sup-phoneme masked-LM setups in other languages (e.g.
+Mixed-Phoneme BERT, PnG-BERT) report converged accuracies around 70–75%;
+this PLT model sits somewhat below that range, likely due to corpus size and
+language-specific factors rather than an implementation issue.
+## Usage
+```python
+import torch, yaml
+from transformers import AlbertConfig, AlbertModel
+from huggingface_hub import hf_hub_download
+from collections import OrderedDict
+REPO = "mimba/plbert-plt"
+class CustomAlbert(AlbertModel):
+    def forward(self, *args, **kwargs):
+        return super().forward(*args, **kwargs).last_hidden_state
+def load_plbert(repo_id=REPO, step=1_000_000):
+    config_path = hf_hub_download(repo_id, "config.yml")
+    plbert_config = yaml.safe_load(open(config_path))
+    config = AlbertConfig(**plbert_config["model_params"])
+    bert = CustomAlbert(config)
+    ckpt_path = hf_hub_download(repo_id, f"step_{step}.t7")
+    checkpoint = torch.load(ckpt_path, map_location="cpu")
+    state_dict = checkpoint["net"]
+    new_state_dict = OrderedDict()
+    for k, v in state_dict.items():
+        name = k[7:] if k.startswith("module.") else k
+        if name.startswith("encoder."):
+            new_state_dict[name[8:]] = v
+    new_state_dict.pop("embeddings.position_ids", None)
+    bert.load_state_dict(new_state_dict, strict=False)
+    return bert
+model = load_plbert()
+model.eval()
+```
+**As a StyleTTS2 `PLBERT_dir`**: download `config.yml` + `step_1000000.t7`
+into `Utils/PLBERT/` of the StyleTTS2 repo — `util.py`'s `load_plbert()`
+(shown above) is what `train_first.py`/`train_second.py` call automatically.
+## Relation to other Mimba datasets/models
+```
+mimba/text2text                     (source text corpus)
+   -> mimba/plt-tts-dataset          (audio + text, 4 speakers)
+   -> phonemized PLT corpus          (IPA phonemization, mode PHRASE)
+   -> mimba/plbert-plt                <- this model
+   -> mimba/styletts2-plt-corpus     (StyleTTS2-ready corpus)
+   -> mimba/styletts2-plt-stage1 / stage2   (StyleTTS2 checkpoints)
+```
+## Limitations
+- Masked-phoneme accuracy (65.52%) is below reference points from other
+  languages' phoneme-level BERT models (~70–75%); treat this as this
+  model's practical ceiling rather than an intermediate result.
+- The word-level prediction head operates over a very large, zipfian
+  vocabulary (agglutinative morphology) and is noisy on rare word forms —
+  this does not affect StyleTTS2 usage, which only consumes phoneme-level
+  embeddings.
+- Trained on synthetic/derived text sources (see `mimba/text2text` and
+  `mimba/plt-tts-dataset` cards for provenance); verify licensing
+  independently before commercial use.
+## Citation
+```bibtex
+@misc{mimba2026plbertplt,
+  title  = {Mimba PL-BERT PLT: A Phonetic-Level BERT for Plateau Malagasy},
+  author = {Mimba Ngouana Fofou},
+  year   = {2026},
+}
+```
+### Contact
+For questions or contributions, open a discussion in the "Community" tab of
+this repository.
+##### *Contact: [@Mimba](baounabaouna@gmail.com)*

step_1000000.t7 ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:041739bc315df409dfb9d09f88c9c4d392175e5e9a4cabc36aa28c253636646a
+size 3762349415

step_1000000_meta.json ADDED Viewed

	@@ -0,0 +1 @@


1	+ {"step": 1000000, "loss_vocab": 0.5753036141395569, "loss_token": 1.22299063205719, "accuracy": 65.51792630847154, "vocab_cap": 400000, "num_vocab": 400001, "num_tokens": 55}

step_950000.t7 ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:1b30734e84fe38b51e08a134c38b83e8372b3ca40532821e297aa2b361fedfd3
+size 3762349299

step_950000_meta.json ADDED Viewed

	@@ -0,0 +1 @@


1	+ {"step": 950000, "loss_vocab": 0.946972668170929, "loss_token": 1.188152551651001, "accuracy": 66.25453100948408, "vocab_cap": 400000, "num_vocab": 400001, "num_tokens": 55}

train.log ADDED Viewed

The diff for this file is too large to render. See raw diff