|
|
--- |
|
|
library_name: transformers |
|
|
license: apache-2.0 |
|
|
--- |
|
|
|
|
|
# BERnaT: Basque Encoders for Representing Natural Textual Diversity |
|
|
|
|
|
Submitted to LREC 2026 |
|
|
|
|
|
## Abstract |
|
|
|
|
|
Language models depend on massive text corpora that are often filtered for quality, a process that can unintentionally |
|
|
exclude non-standard linguistic varieties, reduce model robustness and reinforce representational biases. In this |
|
|
paper, we argue that language models should aim to capture the full spectrum of language variation (dialectal, |
|
|
historical, informal, etc.) rather than relying solely on standardized text. Focusing on Basque, a morphologically rich |
|
|
and low-resource language, we construct new corpora combining standard, social media, and historical sources, and |
|
|
pre-train the BERnaT family of encoder-only models in three configurations: standard, diverse, and combined. We |
|
|
further propose an evaluation framework that separates Natural Language Understanding (NLU) tasks into standard |
|
|
and diverse subsets to assess linguistic generalization. Results show that models trained on both standard and |
|
|
diverse data consistently outperform those trained on standard corpora, improving performance across all task types |
|
|
without compromising standard benchmark accuracy. These findings highlight the importance of linguistic diversity in |
|
|
building inclusive, generalizable language models. |
|
|
|
|
|
## Results |
|
|
|
|
|
| | **AVG standard tasks** | **AVG diverse tasks** | **AVG overall** | |
|
|
|---------------------|:----------------------:|:---------------------:|:---------------:| |
|
|
| **BERnaT_standard** | | | | |
|
|
| medium | 74.10 | 70.30 | 72.58 | |
|
|
| base | 75.33 | 71.26 | 73.70 | |
|
|
| large | 76.83 | 73.13 | 75.35 | |
|
|
| **BERnaT_diverse** | | | | |
|
|
| medium | 71.66 | 69.91 | 70.96 | |
|
|
| base | 72.44 | 71.43 | 72.04 | |
|
|
| large | 74.48 | 71.87 | 73.43 | |
|
|
| **BERnaT** | | | | |
|
|
| medium | 73.56 | 70.59 | 72.37 | |
|
|
| base | 75.42 | 71.28 | 73.76 | |
|
|
| large | **77.88** | **73.77** | **76.24** | |
|
|
|
|
|
## Acknowledgments |
|
|
|
|
|
This work has been partially supported by the Basque Government (Research group funding IT1570-22 and IKER-GAITU project), the Spanish Ministry for Digital Transformation and Civil Service, and the EU-funded NextGenerationEU Recovery, Transformation and Resilience Plan (ILENIA project, 2022/TL22/00215335; and ALIA project). The project also received funding from the European Union’s Horizon Europe research and innovation program under Grant Agreement No 101135724, Topic HORIZON-CL4-2023-HUMAN-01-21 and DeepKnowledge (PID2021-127777OB-C21) founded by MCIN/AEI/10.13039/501100011033 and FEDER. Jaione Bengoetxea, Julen Etxaniz and Ekhi Azurmendi hold a PhD grant from the Basque Government (PRE_2024_1_0028, PRE_2024_2_0028 and PRE_2024_1_0035, respectively). Maite Heredia and Mikel Zubillaga hold a PhD grant from the University of the Basque Country UPV/EHU (PIF23/218 and PIF24/04, respectively). The models were trained on the Leonardo supercomputer at CINECA under the EuroHPC Joint Undertaking, project EHPC-EXT-2024E01-042. |
|
|
|
|
|
## Citation: |
|
|
|
|
|
To cite our work, please use: |
|
|
|
|
|
```bibtex |
|
|
@misc{azurmendi2025bernatbasqueencodersrepresenting, |
|
|
title={BERnaT: Basque Encoders for Representing Natural Textual Diversity}, |
|
|
author={Ekhi Azurmendi and Joseba Fernandez de Landa and Jaione Bengoetxea and Maite Heredia and Julen Etxaniz and Mikel Zubillaga and Ander Soraluze and Aitor Soroa}, |
|
|
year={2025}, |
|
|
eprint={2512.03903}, |
|
|
archivePrefix={arXiv}, |
|
|
primaryClass={cs.CL}, |
|
|
url={https://arxiv.org/abs/2512.03903}, |
|
|
} |
|
|
``` |
