|
|
--- |
|
|
library_name: transformers |
|
|
license: mit |
|
|
language: |
|
|
- gl |
|
|
base_model: |
|
|
- microsoft/mdeberta-v3-base |
|
|
pipeline_tag: fill-mask |
|
|
--- |
|
|
|
|
|
# mDeBERTa-gl |
|
|
|
|
|
**mDeBERTa-gl** is a continued pretraining checkpoint based on [**microsoft/mdeberta-v3-base**](https://huggingface.co/microsoft/mdeberta-v3-base), adapted to Galician through large-scale masked-language modeling. It is intended as a strong general-purpose encoder for downstream NLP tasks in Galician. |
|
|
|
|
|
## Training |
|
|
|
|
|
- **Base model:** microsoft/mdeberta-v3-base |
|
|
- **Epochs:** 3 |
|
|
- **Learning rate:** 6e-4 |
|
|
- **MLM probability:** 0.15 |
|
|
- **Max sequence length:** 512 |
|
|
- **Total batch size:** 1024 |
|
|
- **Training examples:** 10,335,227 |
|
|
- **Mask token**: [MASK] |
|
|
|
|
|
## Intended uses |
|
|
|
|
|
- Masked language modeling (fill-mask) |
|
|
- Encoder for classification, NER, QA, and general Galician NLP tasks |
|
|
- Further domain adaptation via fine-tuning |
|
|
|
|
|
## How to use |
|
|
|
|
|
```python |
|
|
from transformers import AutoModelForMaskedLM, AutoTokenizer, pipeline |
|
|
|
|
|
model_id = "proxectonos/mdeberta-gl" |
|
|
|
|
|
tokenizer = AutoTokenizer.from_pretrained(model_id) |
|
|
model = AutoModelForMaskedLM.from_pretrained(model_id) |
|
|
|
|
|
fill_mask = pipeline("fill-mask", model=model, tokenizer=tokenizer) |
|
|
|
|
|
fill_mask("O Parlamento de Galicia aprobou a [MASK] hoxe.") |
|
|
``` |
|
|
|
|
|
## Funding |
|
|
This work is funded by the Ministerio para la Transformación Digital y de la Función Pública - Funded by EU – NextGenerationEU within the framework of the project Desarrollo de Modelos ALIA |
|
|
|
|
|
## Citation |
|
|
|
|
|
Please reference this model as: **mdeberta-gl (Proxecto Nós Team, 2025)**. |
|
|
|
