| ---
|
| language:
|
| - gl
|
| license: apache-2.0
|
| tags:
|
| - phoneme-level
|
| - modernbert
|
| - galician
|
| - text-to-speech
|
| - tts
|
| - masked-lm
|
| - proxecto-nos
|
| metrics:
|
| - precision
|
| ---
|
|
|
| # PL-ModernBERT-gl: Phoneme-level ModernBERT for Galician
|
|
|
| ## Overview
|
| - [Model Description](#model-description)
|
| - [Intended Uses and Limitations](#intended-uses-and-limitations)
|
| - [Training Details](#training-details)
|
| - [Evaluation](#evaluation)
|
| - [Citation](#citation)
|
| - [Additional Information](#additional-information)
|
| - [Funding and Acknowledgements](#funding-and-acknowledgements)
|
| ---
|
|
|
| ## Model Description
|
|
|
| **PL-ModernBERT-gl** is a phoneme-level masked language model trained on Galician text. It is based on the state-of-the-art **ModernBERT** architecture, adapting the Phoneme-Level BERT (PL-BERT) framework to learn contextualized phoneme representations via masked language modeling and alignment objectives.
|
|
|
| This model is designed to support phoneme-based text-to-speech (TTS) systems, including but not limited to *StyleTTS2*. Thanks to its Galician-specific phoneme vocabulary and contextual embedding capabilities, it can serve as a high-precision phoneme encoder for any TTS architecture requiring phoneme-level features.
|
|
|
| ### Key Improvements & Features:
|
| * **Native Galician Pipeline:** Unlike the original PL-BERT architecture which relied on English phonemizers, this model integrates the open-source linguistic tool **Cotovía** for native Galician grapheme-to-phoneme transcription and text normalization.
|
| * **1:1 Alignment System:** Implements a strict sequential alignment between graphemes and phonemes, successfully handling Galician digraphs (e.g., `ll`, `rr`, `ch`, `nh`, `qu`), silent characters (e.g., silent `h`), and morphosyntactic contractions (e.g., `para` + `a` → `pra`).
|
| * **Dual-Head Architecture:** The core encoder branches into two parallel prediction layers during training:
|
| * **MLM Head (Masked Language Modeling):** Predicts the identity of masked phonemes.
|
| * **P2G Head (Phoneme-to-Grapheme):** Predicts the corresponding grapheme for aligned feature extraction.
|
|
|
| ---
|
|
|
| ## Intended Uses and Limitations
|
|
|
| ### Intended uses
|
| * Integration into phoneme-based Galician TTS pipelines (such as *StyleTTS2*).
|
| * Contextualized phoneme embedding extraction for downstream speech and linguistic tasks in the Galician language.
|
|
|
| ### Limitations
|
| * Not designed for general-purpose word-level NLP tasks (e.g., standard text classification, Named Entity Recognition, or sentiment analysis).
|
| * Strictly supports Galician phoneme tokens mapped through the provided tokenizer utilities.
|
|
|
| ---
|
|
|
| ## Training Details
|
|
|
| ### Training Data
|
| The model was pre-trained on a curated Galician corpus of **11.6 million words** compiled from diverse domains under open licenses:
|
|
|
| | Corpus Source | Domain | Word Count | License |
|
| | :--- | :--- | :---: | :--- |
|
| | *Enciclopedia Galega Universal* | Encyclopedic | 4,751,813 | CC-BY-4.0 |
|
| | *URCO Editora* | Literature / Novels | 1,937,517 | CC-BY-4.0 |
|
| | *Wikipedia* | Encyclopedic | 1,794,937 | CC-BY-SA-4.0 |
|
| | *Mallando no Android* | Technology Blog | 1,553,327 | CC-BY-4.0 |
|
| | *Revista Pincha* | Culture Blog | 669,020 | CC-BY-4.0 |
|
| | *A Nosa Terra* | Journalistic | 652,372 | CC-BY-4.0 |
|
| | *Servizo Publicacións USC* | Academic / Scientific | 280,858 | CC-BY-4.0 |
|
| | **Total** | | **11,639,844** | |
|
|
|
| #### Prosodic Enrichment
|
| To capture expressive prosody and speech modulation, the training corpus was enriched with two dedicated subsets of **66,338 interrogative sentences** and **11,546 exclamative sentences**.
|
|
|
| ### Dynamic Masking Strategy
|
| To force the model to prioritize prosodic clues over highly frequent phonemes, an inverse-frequency dynamic masking algorithm was implemented:
|
| * **Common Phonemes:** 15% masking probability.
|
| * **Standard Punctuation Marks:** 40% masking probability.
|
| * **Critical Prosodic Marks (`!`, `?`):** 80% masking probability.
|
|
|
| ### Training Configuration
|
|
|
| | Parameter | Value |
|
| | :--- | :--- |
|
| | **Model Type / Core Architecture** | ModernBERT (12 layers, 12 attention heads) |
|
| | **Vocabulary Size** | 69 |
|
| | **Hidden Size** | 768 |
|
| | **Intermediate Size (FFN)** | 2048 |
|
| | **Dropout** | 0.1 |
|
| | **Batch Size** | 192 |
|
| | **Total Steps** | 1,000,000 |
|
| | **Precision** | Mixed Precision (`fp16`) |
|
| | **Initial Learning Rate** | 1e-4 |
|
| | **Scheduler Type** | `onecycle` (Cos annealing strategy, Warmup ratio: 0.1) |
|
| | **Max Sequence Length** | 512 |
|
| | **Base Word Mask Probability** | 0.15 |
|
| | **Base Phoneme Mask Probability** | 0.1 |
|
| | **Replacement Probability** | 0.2 |
|
|
|
| ## Evaluation
|
| The model was evaluated using an independent test partition from the core phonemized Galician dataset. Performance was tracked via Masked Language Modeling accuracy (predicting masked phonemes) and Phoneme-to-Grapheme symbolic alignment metrics (predicting corresponding characters):
|
|
|
| | Metric | Value |
|
| |-------------------|------|
|
| | **MLM Precision** | 89.54% |
|
| | **P2G Precision** | 97.88% |
|
|
|
| ## Citation
|
| If this model contributes to your research, please cite it as follows:
|
|
|
| ```bibtex
|
|
|
| @misc{proxectonos/PL-ModernBERT-gl,
|
| author = {{Proxecto Nós}},
|
| title = {{PL-ModernBERT-gl: Phoneme-level ModernBERT for Galician}},
|
| year = {2026},
|
| publisher = {Hugging Face},
|
| howpublished = {\url{[https://huggingface.co/proxectonos/PL-ModernBERT-gl](https://huggingface.co/proxectonos/PL-ModernBERT-gl)}},
|
| }
|
|
|
| ```
|
|
|
| ## Additional Information
|
|
|
| ### Licensing
|
| This model is licensed under the **Apache License 2.0**.
|
|
|
| ### Authors and Credits
|
| * **Project Oversight:** [Proxecto Nós](https://nos.gal/gl/proxecto-nos)
|
| * **Technical Development:** [Gradiant](https://www.gradiant.org/) (Centro Tecnolóxico de Telecomunicacións de Galicia)
|
|
|
| ## Funding and Acknowledgements
|
| This work is funded by the Ministerio para la Transformación Digital y de la Función Pública - Funded by EU – NextGenerationEU within the framework of the project Desarrollo de Modelos ALIA.
|
|
|
| We would like to express our gratitude to the engineering and research teams at **Gradiant** for the technical development of this model, as well as to the **Aholab Signal Processing Laboratory (HiTZ)** and the **Language Technologies Laboratory (BSC)** for their technical support and collaboration. |
