README.md

---
language: eu
license: cc-by-nc-sa-4.0
base_model: ixa-ehu/roberta-eus-euscrawl-base-cased
tags:
  - gector
  - bert
  - basque
  - euskara
  - grammar-correction
  - error-detection
  - pytorch
pipeline_tag: token-classification
---

# GECToR-eus

Basque **grammatical error correction** and **real-word error detection** via GECToR (seq2edit), fine-tuned on RoBERTa-eus-base.

This is the **PyTorch checkpoint**. For browser/edge deployment, use the [ONNX int4 export](https://huggingface.co/itzune/gector-eus-onnx) (83 MB).

## Overview

This model corrects **real-word grammar errors** in Basque — cases where every word is a valid dictionary word but the inflection is wrong in context (e.g. `dio` → `zaio`, `zaidalaren` → `zaidalako`, `etortzen` → `etorriko`). It also provides **per-word error detection** via a dedicated detect head (P(INCORRECT) per token).

| Property | Value |
|---|---|
| Architecture | GECToR (RoBERTa encoder + label/detect heads) |
| Base model | [ixa-ehu/roberta-eus-euscrawl-base-cased](https://huggingface.co/ixa-ehu/roberta-eus-euscrawl-base-cased) (110M params, 12L/768H) |
| Training data | 1M sentence pairs from [Elhuyar GEC corpus](https://hitz.eus/es/geleriak/corpus-and-resources/erreparatu-corpus) (CC-BY-NC-SA) |
| Labels | 5,001 ($KEEP, $DELETE, $REPLACE_x, $APPEND_x, etc.) |
| Detect labels | 3 ($CORRECT, $INCORRECT, \<PAD\>) |
| Checkpoint size | 487 MB (fp32 safetensors) |

## Performance

Evaluated on the Elhuyar `Dem_single` (221 errorful sentences) and `Dem_none` (250 clean sentences) benchmarks:

| min_error_prob | F0.5 | Exact match | False positive rate |
|---|---|---|---|
| 0.0 | 90.0 | 82.8% (183/221) | 4.4% (11/250) |
| 0.2 | 90.0 | 82.8% (183/221) | 4.0% (10/250) |
| **0.5** | **90.2** | 81.4% (180/221) | **3.6%** (9/250) |
| 0.8 | 90.8 | 76.9% (170/221) | 2.8% (7/250) |

For comparison, GECToR-2024 (English, RoBERTa-large 300M) scores F0.5=72.9 on BEA-dev.

## Usage

This model uses the [gotutiyan/gector](https://github.com/gotutiyan/gector) framework. Install it and use the CLI:

```bash
pip install gector

# Predict (correct grammar)
python -m gector.cli.predict \
  --input input.txt \
  --restore_dir itzune/gector-eus \
  --verb_file verb-form-vocab.txt \
  --out output.txt \
  --min_error_prob 0.5
```

Or load in Python:

```python
from gector import GECToR

model = GECToR.from_pretrained("itzune/gector-eus")
# model.bert           → RoBERTa encoder
# model.label_proj_layer  → label head (→ 5000 classes)
# model.d_proj_layer      → detect head (→ 2 classes: CORRECT, INCORRECT)
```

## How it works

GECToR uses an **edit-based** approach — instead of generating text, it predicts a label per token:

- `$KEEP` — leave this word unchanged
- `$DELETE` — remove this word
- `$REPLACE_xxx` — replace this word with `xxx`
- `$APPEND_xxx` — append `xxx` after this word

The model runs **iteratively** (up to 5 passes): each pass applies the predicted edits, then re-runs on the corrected text until no more changes are needed.

The **detect head** provides an independent P(INCORRECT) per token — a confidence score for whether each word is wrong. This is used as a threshold (`min_error_prob`) to control the precision/recall tradeoff, and also powers error detection visualizations.

## Training data — Elhuyar/Orai GEC corpus

The model was trained on the **Elhuyar GEC corpus**, created by the **[Elhuyar Foundation](https://www.elhuyar.eus/)** (now **[Orai NLP](https://www.orai.eus/)**) and distributed under **CC-BY-NC-SA**.

- **Source**: Correct sentences were extracted from a **Basque news corpus** compiled from Berria.eus, Argia.eus, and the Tokikom.eus proximity-media network (500,015 news items, 4,927,748 sentences, ~66M words).
- **Error generation**: Synthetic grammar errors (R1–R4: tense, verb agreement/argument, case/agreement, suffix) were introduced by rule application.
- **Scale**: 9.3M sentence pairs total (4.71M with errors, 5.29M clean). We trained on 1M pairs.
- **Distribution**: [Orai.eus resources page](https://www.orai.eus/errebaliatu/) / [HiTZ corpus page](https://hitz.eus/es/geleriak/corpus-and-resources/erreparatu-corpus)

> ℹ️ This is distinct from the *Elhuyar web corpus* (186M tokens, Leturia 2014), a separate general web-crawl resource on Orai's resources page that is **not** the source of the GEC data.

## Training

- **Base model**: [ixa-ehu/roberta-eus-euscrawl-base-cased](https://huggingface.co/ixa-ehu/roberta-eus-euscrawl-base-cased)
- **Training data**: 1M sentence pairs from the Elhuyar GEC corpus (see above)
- **Epochs**: 5 (1 cold epoch at lr=0.001, 4 fine-tune epochs at lr=1e-5)
- **Batch size**: 64
- **Max length**: 128 tokens
- **GECToR implementation**: [gotutiyan/gector](https://github.com/gotutiyan/gector)
- **Training code**: [itzune/gector-eus](https://github.com/itzune/gector-eus)

## Related

- **[itzune/gector-eus-onnx](https://huggingface.co/itzune/gector-eus-onnx)** — ONNX int4 export (83 MB) for browser/edge deployment
- **[itzune/gector-eus](https://github.com/itzune/gector-eus)** — Training code and scripts
- **[txukun](https://github.com/itzune/txukun)** — Browser-based Basque text corrector using this model

## Credits

- **RoBERTa-eus-base** was pre-trained by the [IXA NLP Group](https://ixa.eus/) (UPV/EHU) — see [Artetxe et al. (2022)](https://arxiv.org/abs/2203.08111)
- **GECToR** was introduced by [Omelianchuk et al. (2020)](https://arxiv.org/abs/2005.12592) (Grammarly)
- **Elhuyar GEC corpus** was created by the [Elhuyar Foundation](https://www.elhuyar.eus/) (now [Orai NLP](https://www.orai.eus/)) and [HiTZ Zentroa](https://hitz.eus/) (UPV/EHU)
- **GECToR implementation**: [gotutiyan/gector](https://github.com/gotutiyan/gector) (MIT)

## License

**CC-BY-NC-SA 4.0**

This model's weights are a derivative work of the [Elhuyar GEC corpus](https://hitz.eus/es/geleriak/corpus-and-resources/erreparatu-corpus), which is licensed under **CC-BY-NC-SA**. Under the ShareAlike clause, the trained model weights are distributed under the same license:

- **NC (NonCommercial)** — You may not use this model for commercial purposes.
- **SA (ShareAlike)** — If you remix, transform, or build upon this model, you must distribute your contributions under the same license.

The base encoder ([RoBERTa-eus-base](https://huggingface.co/ixa-ehu/roberta-eus-euscrawl-base-cased), Apache 2.0) and the GECToR architecture ([gotutiyan/gector](https://github.com/gotutiyan/gector), MIT) are separately licensed, but the **trained weights** inherit the CC-BY-NC-SA restriction from the training data.

The training scripts and ONNX export code ([itzune/gector-eus](https://github.com/itzune/gector-eus)) are MIT licensed.

## Citation

If you use this model, please cite the original works:

```bibtex
@inproceedings{beloki2020gec,
  title     = {Grammatical Error Correction for Basque through a seq2seq neural architecture and synthetic examples},
  author    = {Beloki, Zuhaitz and Saralegi, Xabier and Ceberio, Klara and Corral, Ander},
  booktitle = {Proceedings of the 36th Conference of the Spanish Society for Natural Language Processing (SEPLN 2020)},
  address   = {Granada, Spain},
  year      = {2020}
}

@article{omelianchuk2020gector,
  title={GECToR--Grammatical Error Correction: Tag, Not Rewrite},
  author={Omelianchuk, Kostiantyn and Atrasevych, Vitaly and Chernodub, Artem and Skurzhanskyi, Oleksandr},
  journal={arXiv preprint arXiv:2005.12592},
  year={2020}
}

@article{artetxe2022roberteus,
  title={Roberteus: a monolingual basque language model},
  author={Artetxe, Mikel and Adebisi, Iyanuoluwa and Azkarate, Mikel and Ceberio, Itziar and Campos, Jon Ander and Esnal, Gorka and Fernandez de Landa, Oscar and Goikoetxea, Joseba and Gutierrez, Aitor and Igondi, Maite and others},
  journal={Procesamiento del Lenguaje Natural},
  volume={69},
  pages={27--34},
  year={2022}
}
```