| --- |
| library_name: transformers |
| license: apache-2.0 |
| base_model: |
| - BSC-LT/MrBERT |
| --- |
| |
| # JaunBERT |
|
|
| ## Model Description |
|
|
| JaunBERT is a family of Basque encoder-only language models. |
|
|
| - **Developed by:** HiTZ Research Center & IXA Research group (University of the Basque Country UPV/EHU) |
| - **Funded by:** IKERGAITU and ALIA projects (Basque and Spanish Governments) |
| - **License:** Apache 2.0 |
| - **Model Type**: Encoder-only Transformer models (ModernBERT-style) |
| - **Languages**: Basque (Euskara) |
|
|
|
|
| ## Getting Started |
|
|
| You can either use this model directly as the example below, or fine-tune it to your task of interest. |
|
|
| ```python |
| >>> from transformers import pipeline |
| |
| >>> pipe = pipeline("fill-mask", model='HiTZ/JaunBERT') |
| |
| >>> pipe("Kaixo! Ni <mask> naiz!") |
| [{'score': 0.034423828125, |
| 'token': 71333, |
| 'token_str': 'Aitor', |
| 'sequence': 'Kaixo! Ni Aitor naiz!'}, |
| {'score': 0.032470703125, |
| 'token': 119830, |
| 'token_str': 'Asier', |
| 'sequence': 'Kaixo! Ni Asier naiz!'}, |
| {'score': 0.0208740234375, |
| 'token': 109222, |
| 'token_str': 'Unai', |
| 'sequence': 'Kaixo! Ni Unai naiz!'}, |
| {'score': 0.0196533203125, |
| 'token': 164657, |
| 'token_str': 'Nerea', |
| 'sequence': 'Kaixo! Ni Nerea naiz!'}, |
| {'score': 0.0196533203125, |
| 'token': 40163, |
| 'token_str': 'Ane', |
| 'sequence': 'Kaixo! Ni Ane naiz!'}] |
| ``` |
|
|
| ## Training Data |
|
|
| The JaunBERT family was pre-trained on `latxa-corpus-v2`. |
| Each revision was further trained **on** a specific sub-corpus: |
|
|
| - **`JaunBERT-legala`**: BOPV + BOG + BOTHA + ParlEus |
| - **`JaunBERT-periodistikoa`**: Euscrawl-v2 |
| - **`JaunBERT-zientifikoa`**: Aldizkariak |
|
|
| Training objective is masked language modeling (MLM) on encoder-only architectures. |
|
|
| ## Evaluation |
|
|
| | | **AVG overall** | |
| |-----------------------|:---------------:| |
| | JaunBERT | 64.37 | |
| | JaunBERT-legala | 55.24 | |
| | JaunBERT-periodistikoa| 66.86 | |
| | JaunBERT-zientifikoa | 63.53 | |
|
|
| ## Acknowledgments |
|
|
| This work has been partially supported by the Basque Government (Research group funding IT1570-22 and IKER-GAITU project), the Spanish Ministry for Digital Transformation and Civil Service, and the EU-funded NextGenerationEU Recovery, Transformation and Resilience Plan (ILENIA project, 2022/TL22/00215335; and ALIA project). Ekhi Azurmendi holds a PhD grant from the Basque Government (PRE_2024_1_0035, respectively). The models were trained on the Leonardo supercomputer at CINECA under the EuroHPC Joint Undertaking, project EHPC-EXT-2024E01-042. |
