|
|
--- |
|
|
license: apache-2.0 |
|
|
language: |
|
|
- eu |
|
|
tags: |
|
|
- TTS |
|
|
- PL-BERT |
|
|
- WordPiece |
|
|
- hitz-aholab |
|
|
--- |
|
|
|
|
|
# PL-BERT-eu |
|
|
|
|
|
## Overview |
|
|
|
|
|
<details> |
|
|
<summary>Click to expand</summary> |
|
|
|
|
|
- [Model Description](#model-description) |
|
|
- [Intended Uses and Limitations](#intended-uses-and-limitations) |
|
|
- [How to Get Started with the Model](#how-to-get-started-with-the-model) |
|
|
- [Training Details](#training-details) |
|
|
- [Citation](#citation) |
|
|
- [Additional information](#additional-information) |
|
|
|
|
|
</details> |
|
|
|
|
|
|
|
|
--- |
|
|
|
|
|
## Model Description |
|
|
|
|
|
**PL-BERT-eu** is a phoneme-level masked language model trained on Basque Wikipedia text. It is based on [PL-BERT architecture](https://github.com/yl4579/PL-BERT) and learns phoneme representations via a masked language modeling objective. |
|
|
|
|
|
This model supports **phoneme-based text-to-speech (TTS) systems**, such as [StyleTTS2](https://github.com/yl4579/StyleTTS2) using Basque-specific phoneme vocabulary and contextual embeddings. |
|
|
|
|
|
Features of our PL-BERT: |
|
|
- It is trained **exclusively on Basque** phonemized Wikipedia text. |
|
|
- It uses a reduced **phoneme vocabulary of 178 tokens**. |
|
|
- It utilizes a WordPiece tokenizer for phonemized Basque text. |
|
|
- It includes a custom `token_maps_eu.pkl` and adapted `util.py`. |
|
|
|
|
|
--- |
|
|
|
|
|
## Intended Uses and Limitations |
|
|
|
|
|
### Intended uses |
|
|
|
|
|
- Integration into phoneme-based TTS pipelines such as StyleTTS2. |
|
|
- Speech synthesis and phoneme embedding extraction for Basque. |
|
|
|
|
|
|
|
|
### Limitations |
|
|
|
|
|
- Not designed for general NLP tasks. |
|
|
- Only supports Basque phoneme tokens. |
|
|
|
|
|
--- |
|
|
|
|
|
## How to Get Started with the Model |
|
|
|
|
|
Here is an example of how to use this model within the StyleTTS2 framework: |
|
|
|
|
|
1. Clone the StyleTTS2 repository: https://github.com/yl4579/StyleTTS2 |
|
|
2. Inside the `Utils` directory, create a new folder, for example: `PLBERT_eu`. |
|
|
3. Copy the following files into that folder: |
|
|
- `config.yml` (training configuration) |
|
|
- `step_4000000.t7` (trained checkpoint) |
|
|
- `util.py` (modified to fix position ID loading) |
|
|
|
|
|
4. In your StyleTTS2 configuration file, update the `PLBERT_dir` entry to: |
|
|
|
|
|
`PLBERT_dir: Utils/PLBERT_eu` |
|
|
|
|
|
5. Update the import statement in your code to: |
|
|
|
|
|
`from Utils.PLBERT_eu.util import load_plbert` |
|
|
|
|
|
6. We used code developed by [Aholab](https://aholab.ehu.eus/aholab/) to generate IPA phonemes for training the model. You can see a demo of the Basque phonemizer at [arrandi/phonemizer-eus-esp](https://huggingface.co/spaces/arrandi/phonemizer-eus-esp). Likewise, the code used to generate IPA phonemes can be found in the `phonemizer` directory. We collapsed multi-character phonemes into single-character phonemes for better grapheme–phoneme alignment. |
|
|
|
|
|
**Note:** If second-stage StyleTTS2 training produces a NaN loss when using a single GPU, see [issue #254](https://github.com/yl4579/StyleTTS2/issues/254) in the original StyleTTS2 repository. |
|
|
|
|
|
--- |
|
|
|
|
|
## Training Details |
|
|
|
|
|
### Training data |
|
|
|
|
|
The model was trained on a Basque corpus phonemized using **Modelo1y2**. It uses a consistent phoneme token set with boundary markers and masking tokens. |
|
|
|
|
|
Tokenizer: custom (splits on whitespace) |
|
|
Phoneme masking strategy: phoneme-level masking and replacement |
|
|
Training steps: 4,000,000 |
|
|
Precision: mixed-precision (fp16) |
|
|
|
|
|
### Training configuration |
|
|
|
|
|
Model parameters: |
|
|
|
|
|
- Vocabulary size: 178 |
|
|
- Hidden size: 768 |
|
|
- Attention heads: 12 |
|
|
- Intermediate size: 2048 |
|
|
- Number of layers: 12 |
|
|
- Max position embeddings: 512 |
|
|
- Dropout: 0.1 |
|
|
- Embedding size: 128 |
|
|
- Number of hidden groups: 1 |
|
|
- Number of hidden layers per group: 12 |
|
|
- Inner group number: 1 |
|
|
- Downscale factor: 1 |
|
|
|
|
|
Other parameters: |
|
|
|
|
|
- Batch size: 32 |
|
|
- Max mel length: 512 |
|
|
- Word mask probability: 0.15 |
|
|
- Phoneme mask probability: 0.1 |
|
|
- Replacement probability: 0.2 |
|
|
- Token separator: space |
|
|
- Token mask: M |
|
|
- Word separator ID: 2 |
|
|
- Scheduler type: OneCycleLR |
|
|
- Learning rate: 0.0002 |
|
|
- pct_start: 0.1 |
|
|
- Annealing strategy: cosine annealing |
|
|
- div_factor: 25 |
|
|
- final_div_factor: 10000 |
|
|
|
|
|
|
|
|
### Evaluation |
|
|
|
|
|
The model has been successfully integrated into StyleTTS2, where it enables the synthesis of Basque. |
|
|
|
|
|
--- |
|
|
|
|
|
## Citation |
|
|
|
|
|
If this code contributes to your research, please cite the work: |
|
|
|
|
|
``` |
|
|
@misc{aarriandiagaplberteu, |
|
|
title={PL-BERT-eu}, |
|
|
author={Ander Arriandiaga and Ibon Saratxaga and Eva Navas and Inma Hernaez}, |
|
|
organization={Hitz (Aholab) - EHU}, |
|
|
url={https://huggingface.co/langtech-veu/PL-BERT-wp_es}, |
|
|
year={2026} |
|
|
} |
|
|
``` |
|
|
|
|
|
## Additional Information |
|
|
|
|
|
|
|
|
### Author |
|
|
|
|
|
Author: [Ander Arriandiaga](https://huggingface.co/arrandi) — Aholab (Hitz), EHU |
|
|
|
|
|
### Contact |
|
|
For further information, please send an email to <inma.hernaez@ehu.eus>. |
|
|
|
|
|
### Copyright |
|
|
Copyright(c) 2026 by Aholab, HiTZ. |
|
|
|
|
|
### License |
|
|
|
|
|
[Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0) |
|
|
|
|
|
|
|
|
### Funding |
|
|
This work is funded by the Ministerio para la Transformación Digital y de la Función Pública - Funded by EU – NextGenerationEU within the framework of the project Desarrollo de Modelos ALIA. |
|
|
|
