README.md

---
library_name: transformers
license: apache-2.0
base_model:
- BSC-LT/MrBERT
---

# JaunBERT

## Model Description

JaunBERT is a family of Basque encoder-only language models.

- **Developed by:** HiTZ Research Center & IXA Research group (University of the Basque Country UPV/EHU)
- **Funded by:** IKERGAITU and ALIA projects (Basque and Spanish Governments)
- **License:** Apache 2.0
- **Model Type**: Encoder-only Transformer models (ModernBERT-style)
- **Languages**: Basque (Euskara)


## Getting Started

You can either use this model directly as the example below, or fine-tune it to your task of interest.

```python
>>> from transformers import pipeline

>>> pipe = pipeline("fill-mask", model='HiTZ/JaunBERT')

>>> pipe("Kaixo! Ni <mask> naiz!")
[{'score': 0.034423828125,
  'token': 71333,
  'token_str': 'Aitor',
  'sequence': 'Kaixo! Ni Aitor  naiz!'},
 {'score': 0.032470703125,
  'token': 119830,
  'token_str': 'Asier',
  'sequence': 'Kaixo! Ni Asier  naiz!'},
 {'score': 0.0208740234375,
  'token': 109222,
  'token_str': 'Unai',
  'sequence': 'Kaixo! Ni Unai  naiz!'},
 {'score': 0.0196533203125,
  'token': 164657,
  'token_str': 'Nerea',
  'sequence': 'Kaixo! Ni Nerea  naiz!'},
 {'score': 0.0196533203125,
  'token': 40163,
  'token_str': 'Ane',
  'sequence': 'Kaixo! Ni Ane  naiz!'}]
```

## Training Data

The JaunBERT family was pre-trained on `latxa-corpus-v2`. 
Each revision was further trained **on** a specific sub-corpus:

- **`JaunBERT-legala`**: BOPV + BOG + BOTHA + ParlEus
- **`JaunBERT-periodistikoa`**: Euscrawl-v2
- **`JaunBERT-zientifikoa`**: Aldizkariak

Training objective is masked language modeling (MLM) on encoder-only architectures.

## Evaluation

|                       | **AVG overall** |
|-----------------------|:---------------:|
| JaunBERT              |      64.37      |
| JaunBERT-legala       |      55.24      |
| JaunBERT-periodistikoa|      66.86      |
| JaunBERT-zientifikoa  |      63.53      |

## Acknowledgments

This work has been partially supported by the Basque Government (Research group funding IT1570-22 and IKER-GAITU project), the Spanish Ministry for Digital Transformation and Civil Service, and the EU-funded NextGenerationEU Recovery, Transformation and Resilience Plan (ILENIA project, 2022/TL22/00215335; and ALIA project). Ekhi Azurmendi holds a PhD grant from the Basque Government (PRE_2024_1_0035, respectively). The models were trained on the Leonardo supercomputer at CINECA under the EuroHPC Joint Undertaking, project EHPC-EXT-2024E01-042.