File size: 6,553 Bytes

8b2a735

---
language:
- plt
license: cc-by-nc-sa-4.0
pretty_name: Mimba PL-BERT PLT (Plateau Malagasy Phonetic-Level BERT)
tags:
- text-to-speech
- pl-bert
- plbert
- albert
- malagasy
- plt
- african-languages
- low-resource
- masked-language-modeling
- phoneme
library_name: transformers
---

# Mimba PL-BERT PLT — Phonetic-Level BERT for Plateau Malagasy

A **phonetic-level pre-trained language model (PL-BERT)** for **Plateau Malagasy (PLT)**,
trained from scratch to provide text/phoneme-context-aware embeddings for
**StyleTTS2** synthesis. Adapted from the original
[PL-BERT](https://github.com/yl4579/PL-BERT) architecture (Li et al., used in
StyleTTS2) and trained on a large phonemized Malagasy corpus with the same
55-symbol phoneme vocabulary used across every Mimba PLT model (StyleTTS2,
NeuTTS-Nano, Supertonic 3).

> ⚠️ **Not a standalone TTS model.** PL-BERT is a text/phoneme encoder only —
> it produces contextual embeddings consumed by a downstream acoustic model
> (StyleTTS2 Stage 1/Stage 2). It cannot synthesize audio by itself.

## Summary

| | |
|---|---|
| Language | Plateau Malagasy (`plt`) |
| Architecture | ALBERT (`transformers.AlbertModel` + 2 prediction heads) |
| Phoneme vocabulary | 55 symbols (`phoneme_symbols.pkl`, shared with StyleTTS2/NeuTTS-Nano) |
| Hidden size | 768 |
| Attention heads | 12 |
| Hidden layers | 12 |
| Intermediate size | 2048 |
| Max position embeddings | 512 |
| Dropout | 0.1 |
| Training objective | Masked language modeling, dual head (phoneme-level + word-level) |
| Training steps | 1,000,000 |
| Final masked-phoneme accuracy | **65.52%** (measured on ~200K masked positions) |
| Checkpoint format | `step_{N}.t7` (`{'net': state_dict, 'optimizer': ..., 'step': N}`) |


### Loss curve

<!DOCTYPE html>
<html>
<head>
<style>
  .conteneur-images {
    display: flex;          /* Active le mode horizontal */
    gap: 10px;              /* Espace de 10px entre les images */
  }
  .conteneur-images img {
    width: 25%;             /* 100% / 4 images = 25% */
    height: auto;           /* Maintient les proportions */
    flex-shrink: 1;         /* Permet de rétrécir si besoin */
  }
</style>
</head>
<body>
  <p class="conteneur-images">
    <img src="assets/0.png" width="900" alt="Loss curve PL BERT"> 
    <img src="assets/1.png" width="900" alt="Loss curve PL BERT"> 
    <img src="assets/2.png" width="900" alt="Loss curve PL BERT"> 
  </p>
</body>
</html>

## Training details

The model is trained with a dual masked-language-modeling objective — one
head predicts the masked **phoneme token** (55-way classification), the other
predicts the masked **word form** (large open vocabulary of Malagasy word
forms, zipfian-distributed due to the language's agglutinative morphology).
Only the phoneme-level task is used downstream by StyleTTS2, but the joint
objective helps the encoder learn richer contextual representations.

Training ran for 1,000,000 steps with a cosine learning-rate decay
(`1e-4 → 1e-6`) applied over the final ~270K steps. Accuracy on masked
phoneme positions (measured periodically on held-out batches, not just
training loss) tracked as follows:

| Step | Masked-phoneme accuracy |
|---|---|
| 136,000 | 58.55% |
| 727,514 | 61–63% |
| 900,000 | 63.92% |
| 1,000,000 | **65.52%** |

Accuracy plateaued in the final third of training despite the LR decay
reaching down to `1e-6` — this is treated as the effective ceiling for this
model size/corpus, not a sign that more steps would help. For reference,
comparable phoneme/sup-phoneme masked-LM setups in other languages (e.g.
Mixed-Phoneme BERT, PnG-BERT) report converged accuracies around 70–75%;
this PLT model sits somewhat below that range, likely due to corpus size and
language-specific factors rather than an implementation issue.

## Usage

```python
import torch, yaml
from transformers import AlbertConfig, AlbertModel
from huggingface_hub import hf_hub_download
from collections import OrderedDict

REPO = "mimba/plbert-plt"

class CustomAlbert(AlbertModel):
    def forward(self, *args, **kwargs):
        return super().forward(*args, **kwargs).last_hidden_state

def load_plbert(repo_id=REPO, step=1_000_000):
    config_path = hf_hub_download(repo_id, "config.yml")
    plbert_config = yaml.safe_load(open(config_path))
    config = AlbertConfig(**plbert_config["model_params"])
    bert = CustomAlbert(config)

    ckpt_path = hf_hub_download(repo_id, f"step_{step}.t7")
    checkpoint = torch.load(ckpt_path, map_location="cpu")
    state_dict = checkpoint["net"]

    new_state_dict = OrderedDict()
    for k, v in state_dict.items():
        name = k[7:] if k.startswith("module.") else k
        if name.startswith("encoder."):
            new_state_dict[name[8:]] = v
    new_state_dict.pop("embeddings.position_ids", None)
    bert.load_state_dict(new_state_dict, strict=False)
    return bert

model = load_plbert()
model.eval()
```

**As a StyleTTS2 `PLBERT_dir`**: download `config.yml` + `step_1000000.t7`
into `Utils/PLBERT/` of the StyleTTS2 repo — `util.py`'s `load_plbert()`
(shown above) is what `train_first.py`/`train_second.py` call automatically.

## Relation to other Mimba datasets/models

```
mimba/text2text                     (source text corpus)
   -> mimba/plt-tts-dataset          (audio + text, 4 speakers)
   -> phonemized PLT corpus          (IPA phonemization, mode PHRASE)
   -> mimba/plbert-plt                <- this model
   -> mimba/styletts2-plt-corpus     (StyleTTS2-ready corpus)
   -> mimba/styletts2-plt-stage1 / stage2   (StyleTTS2 checkpoints)
```

## Limitations

- Masked-phoneme accuracy (65.52%) is below reference points from other
  languages' phoneme-level BERT models (~70–75%); treat this as this
  model's practical ceiling rather than an intermediate result.
- The word-level prediction head operates over a very large, zipfian
  vocabulary (agglutinative morphology) and is noisy on rare word forms —
  this does not affect StyleTTS2 usage, which only consumes phoneme-level
  embeddings.
- Trained on synthetic/derived text sources (see `mimba/text2text` and
  `mimba/plt-tts-dataset` cards for provenance); verify licensing
  independently before commercial use.

## Citation

```bibtex
@misc{mimba2026plbertplt,
  title  = {Mimba PL-BERT PLT: A Phonetic-Level BERT for Plateau Malagasy},
  author = {Mimba Ngouana Fofou},
  year   = {2026},
}
```

### Contact

For questions or contributions, open a discussion in the "Community" tab of
this repository.

##### *Contact: [@Mimba](baounabaouna@gmail.com)*