Initial release: weights, model card, and phoneme token maps

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+model.t7 filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,3 +1,128 @@
----
-license: apache-2.0
----
+---
+language:
+- gl
+license: apache-2.0
+tags:
+- phoneme-level
+- modernbert
+- galician
+- text-to-speech
+- tts
+- masked-lm
+- proxecto-nos
+metrics:
+- precision
+---
+
+# PL-ModernBERT-gl: Phoneme-level ModernBERT for Galician
+
+## Overview
+- [Model Description](#model-description)
+- [Intended Uses and Limitations](#intended-uses-and-limitations)
+- [Training Details](#training-details)
+- [Evaluation](#evaluation)
+- [Citation](#citation)
+- [Additional Information](#additional-information)
+
+---
+
+## Model Description
+
+**PL-ModernBERT-gl** is a phoneme-level masked language model trained on Galician text. It is based on the state-of-the-art **ModernBERT** architecture, adapting the Phoneme-Level BERT (PL-BERT) framework to learn contextualized phoneme representations via masked language modeling and alignment objectives.
+
+This model is designed to support phoneme-based text-to-speech (TTS) systems, including but not limited to *StyleTTS2*. Thanks to its Galician-specific phoneme vocabulary and contextual embedding capabilities, it can serve as a high-precision phoneme encoder for any TTS architecture requiring phoneme-level features.
+
+### Key Improvements & Features:
+* **Native Galician Pipeline:** Unlike the original PL-BERT architecture which relied on English phonemizers, this model integrates the open-source linguistic tool **Cotovía** for native Galician grapheme-to-phoneme transcription and text normalization.
+* **1:1 Alignment System:** Implements a strict sequential alignment between graphemes and phonemes, successfully handling Galician digraphs (e.g., `ll`, `rr`, `ch`, `nh`, `qu`), silent characters (e.g., silent `h`), and morphosyntactic contractions (e.g., `para` + `a` $\rightarrow$ `pra`).
+* **Dual-Head Architecture:** The core encoder branches into two parallel prediction layers during training:
+  * **MLM Head (Masked Language Modeling):** Predicts the identity of masked phonemes.
+  * **P2G Head (Phoneme-to-Grapheme):** Predicts the corresponding grapheme for aligned feature extraction.
+
+---
+
+## Intended Uses and Limitations
+
+### Intended uses
+* Integration into phoneme-based Galician TTS pipelines (such as *StyleTTS2*).
+* Contextualized phoneme embedding extraction for downstream speech and linguistic tasks in the Galician language.
+
+### Limitations
+* Not designed for general-purpose word-level NLP tasks (e.g., standard text classification, Named Entity Recognition, or sentiment analysis).
+* Strictly supports Galician phoneme tokens mapped through the provided tokenizer utilities.
+
+---
+
+## Training Details
+
+### Training Data
+The model was pre-trained on a curated Galician corpus of **11.6 million words** compiled from diverse domains under open licenses: 
+
+| Corpus Source | Domain | Word Count | License |
+| :--- | :--- | :---: | :--- |
+| *Enciclopedia Galega Universal* | Encyclopedic | 4,751,813 | CC-BY-4.0 |
+| *URCO Editora* | Literature / Novels | 1,937,517 | CC-BY-4.0 |
+| *Wikipedia* | Encyclopedic | 1,794,937 | CC-BY-SA-4.0 |
+| *Mallando no Android* | Technology Blog | 1,553,327 | CC-BY-4.0 |
+| *Revista Pincha* | Culture Blog | 669,020 | CC-BY-4.0 |
+| *A Nosa Terra* | Journalistic | 652,372 | CC-BY-4.0 |
+| *Servizo Publicacións USC* | Academic / Scientific | 280,858 | CC-BY-4.0 |
+| **Total** | | **11,639,844** | |
+
+#### Prosodic Enrichment
+To capture expressive prosody and speech modulation, the training corpus was enriched with two dedicated subsets of **66,338 interrogative sentences** and **11,546 exclamative sentences**.
+
+### Dynamic Masking Strategy
+To force the model to prioritize prosodic clues over highly frequent phonemes, an inverse-frequency dynamic masking algorithm was implemented:
+* **Common Phonemes:** 15% masking probability.
+* **Standard Punctuation Marks:** 40% masking probability.
+* **Critical Prosodic Marks (`!`, `?`):** 80% masking probability.
+
+### Training Configuration
+
+| Parameter | Value |
+| :--- | :--- |
+| **Model Type / Core Architecture** | ModernBERT (12 layers, 12 attention heads) |
+| **Hidden Size** | 768 |
+| **Intermediate Size (FFN)** | 2048 |
+| **Batch Size** | 64 |
+| **Total Steps** | 200,000 |
+| **Precision** | Mixed Precision (`fp16`) |
+| **Initial Learning Rate** | 1e-4 |
+| **Scheduler Type** | `onecycle` (Cos annealing strategy, Warmup ratio: 0.1) |
+| **Max Sequence Length** | 512 |
+| **Replacement Probability** | 0.2 |
+
+## Evaluation
+The model was evaluated using an independent test partition from the core phonemized Galician dataset. Performance was tracked via Masked Language Modeling accuracy (predicting masked phonemes) and Phoneme-to-Grapheme symbolic alignment metrics (predicting corresponding characters):
+
+| Metric           | Value |
+|-------------------|------|
+| **Precisión MLM** | 89.54% |
+| **Precisión P2G** | 97.88% |
+
+## Citation
+If this model or code contributes to your research, please cite it as follows:
+
+@misc{proxectonos/PL-ModernBERT-gl,
+  author       = {{Proxecto Nós}},
+  title        = {{PL-ModernBERT-gl: Phoneme-level ModernBERT for Galician}},
+  year         = {2026},
+  publisher    = {Hugging Face},
+  howpublished = {\url{[https://huggingface.co/proxectonos/PL-ModernBERT-gl](https://huggingface.co/proxectonos/PL-ModernBERT-gl)}},
+}
+
+## Additional Information
+
+### Licensing
+This model is licensed under the **Apache License 2.0**. 
+
+#### Authors and Credits
+* **Project Oversight:** Proxecto Nós
+* **Technical Development:** [Gradiant](https://www.gradiant.org/) (Centro Tecnolóxico de Telecomunicacións de Galicia)
+
+### Funding
+This work is funded by the Ministerio para la Transformación Digital y de la Función Pública - Funded by EU – NextGenerationEU within the framework of the project Desarrollo de Modelos ALIA.
+
+## Acknowledgements
+We would like to express our gratitude to the engineering and research teams at **Gradiant** for the technical development of this model.

model.t7

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f59dfc1bebefbbbe6b8b3ec43659d9db59be915efa9a5a6e4638b8ced879c5b0
+size 1021339920

phoneme_token_maps.json

@@ -0,0 +1,352 @@
+{
+  "0": {
+    "phoneme": "X",
+    "token": 0,
+    "grapheme" : " "
+  },
+  "1": {
+    "phoneme": " ",
+    "token": 1,
+    "grapheme" : " "
+  },
+  "2": {
+    "phoneme": "$",
+    "token": 2,
+    "grapheme" : " "
+  },
+  "3": {
+    "phoneme": "M",
+    "token": 3,
+    "grapheme" : " "
+  },
+  "4": {
+    "phoneme": ",",
+    "token": 4,
+    "grapheme" :","
+  },
+  "5": {
+    "phoneme": ".",
+    "token": 5,
+    "grapheme" :"."
+  },
+  "6": {
+    "phoneme": "a",
+    "token": 6,
+    "grapheme" :"a"
+  },
+  "7": {
+    "phoneme": "E",
+    "token": 7,
+    "grapheme" : "e"
+  },
+  "8": {
+    "phoneme": "e",
+    "token": 8,
+    "grapheme" : "e"
+  },
+  "9": {
+    "phoneme": "i",
+    "token": 9,
+    "grapheme" : "i"
+  },
+  "10": {
+    "phoneme": "j",
+    "token": 10,
+    "grapheme" : "i"
+  },
+  "11": {
+    "phoneme": "O",
+    "token": 11,
+    "grapheme" : "o"
+  },
+  "12": {
+    "phoneme": "o",
+    "token": 12,
+    "grapheme" : "o"
+  },
+  "13": {
+    "phoneme": "u",
+    "token": 13,
+    "grapheme" : "u"
+  },
+  "14": {
+    "phoneme": "w",
+    "token": 14,
+    "grapheme" : "u"
+  },
+  "15": {
+    "phoneme": "p",
+    "token": 15,
+    "grapheme" : "p"
+  },
+  "16": {
+    "phoneme": "b",
+    "token": 16,
+    "grapheme" : "b"
+  },
+  "17": {
+    "phoneme": "B",
+    "token": 17,
+    "grapheme" : "b"
+  },
+  "18": {
+    "phoneme": "t",
+    "token": 18,
+    "grapheme" : "t"
+  },
+  "19": {
+    "phoneme": "d",
+    "token": 19,
+    "grapheme" : "d"
+  },
+  "20": {
+    "phoneme": "D",
+    "token": 20,
+    "grapheme" : "d"
+  },
+  "21": {
+    "phoneme": "k",
+    "token": 21,
+    "grapheme" : "c"
+  },
+  "22": {
+    "phoneme": "g",
+    "token": 22,
+    "grapheme" : "g"
+  },
+  "23": {
+    "phoneme": "G",
+    "token": 23,
+    "grapheme" : "g"
+  },
+  "24": {
+    "phoneme": "f",
+    "token": 24,
+    "grapheme" : "f"
+  },
+  "25": {
+    "phoneme": "T",
+    "token": 25,
+    "grapheme" : "z"
+  },
+  "26": {
+    "phoneme": "s",
+    "token": 26,
+    "grapheme" : "s"
+  },
+  "27": {
+    "phoneme": "S",
+    "token": 27,
+    "grapheme" : "x"
+  },
+  "28": {
+    "phoneme": "m",
+    "token": 28,
+    "grapheme" : "m"
+  },
+  "29": {
+    "phoneme": "n",
+    "token": 29,
+    "grapheme" : "n"
+  },
+  "30": {
+    "phoneme": "J",
+    "token": 30,
+    "grapheme" : "ñ"
+  },
+  "31": {
+    "phoneme": "N",
+    "token": 31,
+    "grapheme" : "nh"
+  },
+  "32": {
+    "phoneme": "l",
+    "token": 32,
+    "grapheme" : "l"
+  },
+  "33": {
+    "phoneme": "Z",
+    "token": 33,
+    "grapheme" : "ll"
+  },
+  "34": {
+    "phoneme": "r",
+    "token": 34,
+    "grapheme" : "r"
+  },
+  "35": {
+    "phoneme": "R",
+    "token": 35,
+    "grapheme" : "rr"
+  },
+  "36": {
+    "phoneme": "x",
+    "token": 36,
+    "grapheme" : "j"
+  },
+  "37": {
+    "phoneme": "C",
+    "token": 37,
+    "grapheme" : "ch"
+  },
+  "38": {
+    "phoneme": "!",
+    "token": 38,
+    "grapheme" : "!"
+  },
+  "39": {
+    "phoneme": "\"",
+    "token": 39,
+    "grapheme" : "\""
+  },
+  "40": {
+    "phoneme": "(",
+    "token": 40,
+    "grapheme" : "("
+  },
+  "41": {
+    "phoneme": ")",
+    "token": 41,
+    "grapheme" : ")"
+  },
+  "42": {
+    "phoneme": "/",
+    "token": 42,
+    "grapheme" : "/"
+  },
+  "43": {
+    "phoneme": ":",
+    "token": 43,
+    "grapheme" : ":"
+  },
+  "44": {
+    "phoneme": ";",
+    "token": 44,
+    "grapheme" : ";"
+  },
+  "45": {
+    "phoneme": "?",
+    "token": 45,
+    "grapheme" : "?"
+  },
+  "46": {
+    "phoneme": "[",
+    "token": 46,
+    "grapheme" : "["
+  },
+  "47": {
+    "phoneme": "]",
+    "token": 47,
+    "grapheme" : "]"
+  },
+  "48": {
+    "phoneme": "{",
+    "token": 48,
+    "grapheme" : "{"
+  },
+  "49": {
+    "phoneme": "}",
+    "token": 49,
+    "grapheme" : "}"
+  },
+  "50": {
+    "phoneme": "¡",
+    "token": 50,
+    "grapheme" : "¡"
+  },
+  "51": {
+    "phoneme": "º",
+    "token": 51,
+    "grapheme" : "º"
+  },
+  "52": {
+    "phoneme": "ª",
+    "token": 52,
+    "grapheme" : "ª"
+  },
+  "53": {
+    "phoneme": "`",
+    "token": 53,
+    "grapheme" : "`"
+  },
+  "54": {
+    "phoneme": "¿",
+    "token": 54,
+    "grapheme" : "¿"
+  },
+  "55": {
+    "phoneme": "K",
+    "token": 55,
+    "grapheme" : "x"
+  },
+  "56": {
+    "phoneme": "W",
+    "token": 56,
+    "grapheme" : "x"
+  },
+  "57": {
+    "phoneme": "O",
+    "token": 57,
+    "grapheme" : "ao"
+  },
+  "58": {
+    "phoneme": "W",
+    "token": 58,
+    "grapheme" : "tS"
+  },
+  "59": {
+    "phoneme": "á",
+    "token": 59,
+    "grapheme" : "á"
+  },
+  "60": {
+    "phoneme": "É",
+    "token": 60,
+    "grapheme" : "é"
+  },
+  "61": {
+    "phoneme": "é",
+    "token": 61,
+    "grapheme" : "é"
+  },
+  "62": {
+    "phoneme": "í",
+    "token": 62,
+    "grapheme" : "í"
+  },
+  "63": {
+    "phoneme": "j́",
+    "token": 63,
+    "grapheme" : "í"
+  },
+  "64": {
+    "phoneme": "Ó",
+    "token": 64,
+    "grapheme" : "ó"
+  },
+  "65": {
+    "phoneme": "ó",
+    "token": 65,
+    "grapheme" : "ó"
+  },
+  "66": {
+    "phoneme": "ú",
+    "token": 66,
+    "grapheme" : "ú"
+  },
+  "67": {
+    "phoneme": "ẃ",
+    "token": 67,
+    "grapheme" : "ú"
+  },
+  "68": {
+    "phoneme": "s",
+    "token": 68,
+    "grapheme" : "ps"
+  },
+  "69": {
+    "phoneme": "n",
+    "token": 69,
+    "grapheme" : "pn"
+  }
+}