README.md

---
language: eu
license: cc-by-nc-sa-4.0
base_model: ixa-ehu/roberta-eus-euscrawl-base-cased
tags:
  - onnx
  - gector
  - bert
  - basque
  - euskara
  - grammar-correction
  - error-detection
  - int4
  - quantized
pipeline_tag: token-classification
---

# GECToR-eus ONNX (int4) — Unified v3

**ONNX int4 export** of the GECToR-eus **unified v3** model — a GECToR grammar correction model fine-tuned on **RoBERTa-eus-base** for Basque grammatical error correction. This is the **unified** model, trained on the Elhuyar GEC corpus **plus** augmented data covering spelling, capitalization/punctuation, proper nouns, and real-word confusions. Trained by [Itzune](https://github.com/itzune) as part of the [gector-eus](https://github.com/itzune/gector-eus) project.

For the **PyTorch checkpoint** (fine-tuning/research), see [itzune/gector-eus](https://huggingface.co/itzune/gector-eus).

## Overview

This model corrects **real-word grammar errors** in Basque — cases where every word is a valid dictionary word but the inflection is wrong in context (e.g. `dio` → `zaio`, `zaidalaren` → `zaidalako`, `etortzen` → `etorriko`). It also provides **per-word error detection** via a dedicated detect head (P(INCORRECT) per token).

| Property | Value |
|---|---|
| Architecture | GECToR (RoBERTa encoder + label/detect heads) |
| Base model | [ixa-ehu/roberta-eus-euscrawl-base-cased](https://huggingface.co/ixa-ehu/roberta-eus-euscrawl-base-cased) (110M params, 12L/768H) |
| Training data | 1.23M pairs: 1M [Elhuyar GEC](https://hitz.eus/es/geleriak/corpus-and-resources/erreparatu-corpus) + 234k augmented (CC-BY-NC-SA) |
| Labels | 5,001 ($KEEP, $DELETE, $REPLACE_x, $APPEND_x, etc.) |
| Detect labels | 3 ($CORRECT, $INCORRECT, \<PAD\>) |
| Quantized size | **83 MB** (int4 ONNX) |
| int8 size | 122 MB (int8 ONNX, fallback) |
| Best checkpoint | Epoch 3 (early stopping on valid_loss) |

## Performance

### Elhuyar benchmark (F0.5 via M² scorer)

Evaluated on the Elhuyar `Dem_single` (221 errorful sentences) and `Dem_none` (250 clean sentences) benchmarks. The unified v3 model achieves F0.5=33.8 on `eval_gector.py` (token-level exact-match scoring), comparable to the v1 grammar-only model (F0.5=34.1). The M² scorer gives higher absolute numbers due to a different alignment method.

### Unified benchmark (220-case, 10 categories)

The v3 unified model was evaluated on a 220-case benchmark spanning 10 error categories (spelling, cap-punct, grammar, real-word, multi-error, paragraph, markdown, mixed, clean):

| Configuration | Overall accuracy |
|---|---|
| GECToR-only (unified v3, standalone) | 35.0% (77/220) |
| 3-model pipeline (old grammar GECToR) | 45.5% (100/220) |
| **Hybrid (unified v3 grammar + cap-punct + spelling)** | **47.3%** (104/220) |

The **hybrid** configuration — unified v3 for grammar, MarianMT for cap-punct, Hunspell+BERTeus for spelling — is the best-performing setup. Key v3 improvements over v1: real-word error detection (+50%), grammar correction (+12.5%).

For comparison, GECToR-2024 (English, RoBERTa-large 300M) scores F0.5=72.9 on BEA-dev.

## Files

| File | Size | Description |
|---|---|---|
| `onnx/model_q4.onnx` | 83 MB | int4 quantized model (MatMulNBits + int8 embeddings) |
| `onnx/model_quantized.onnx` | 122 MB | int8 quantized model (fallback) |
| `gector_vocab.json` | 0.3 MB | Label vocabulary + model config (num_labels, d_num_labels, etc.) |
| `tokenizer.json` | 3.5 MB | Fast tokenizer (XLM-RoBERTa BPE) |
| `sentencepiece.bpe.model` | 1.1 MB | SentencePiece model |
| `tokenizer_config.json` | — | Tokenizer config |
| `special_tokens_map.json` | — | Special tokens |
| `config.json` | — | Full model config |

## Usage

### JavaScript (Transformers.js / ONNX Runtime Web)

```javascript
import { InferenceSession, Tensor } from 'onnxruntime-web';

// Load model
const session = await InferenceSession.create('/models/gector/onnx/model_q4.onnx', {
  executionProviders: ['wasm'],
});

// Load vocab
const vocab = await fetch('/models/gector/gector_vocab.json').then(r => r.json());

// Run inference
const inputIds = new Tensor('int64', BigInt64Array.from(ids.map(BigInt)), [1, len]);
const attentionMask = new Tensor('int64', BigInt64Array.from(mask.map(BigInt)), [1, len]);
const outputs = await session.run({ input_ids: inputIds, attention_mask: attentionMask });

// outputs.logits_labels  → [1, seq_len, 5000]  (label predictions)
// outputs.logits_d       → [1, seq_len, 2]     (detection: CORRECT vs INCORRECT)
```

### Python (ONNX Runtime)

```python
import onnxruntime as ort
import numpy as np

session = ort.InferenceSession('onnx/model_q4.onnx')
outputs = session.run(None, {
    'input_ids': np.array([input_ids], dtype=np.int64),
    'attention_mask': np.array([attention_mask], dtype=np.int64),
})
logits_labels = outputs[0]  # [1, seq_len, 5000]
logits_d = outputs[1]       # [1, seq_len, 2]
```

## How it works

GECToR uses an **edit-based** approach — instead of generating text, it predicts a label per token:

- `$KEEP` — leave this word unchanged
- `$DELETE` — remove this word
- `$REPLACE_xxx` — replace this word with `xxx`
- `$APPEND_xxx` — append `xxx` after this word

The model runs **iteratively** (up to 5 passes): each pass applies the predicted edits, then re-runs on the corrected text until no more changes are needed.

The **detect head** provides an independent P(INCORRECT) per token — a confidence score for whether each word is wrong. This is used as a threshold (`min_error_prob`) to control the precision/recall tradeoff, and also powers error detection visualizations.

## int4 Quantization

The int4 model uses a two-step quantization process:

1. **int4 MatMul weights** — `MatMulNBitsQuantizer` (bits=4, block_size=128, asymmetric) quantizes all 96 MatMul operations in the RoBERTa encoder to 4-bit. This reduces the encoder weight matrix from 147 MB (fp32) to ~20 MB.
2. **int8 embeddings** — `quantize_dynamic` quantizes the 3 Gather operations (token/position embeddings) to int8. This avoids `GatherBlockQuantized` which is not supported on the WASM backend.

Result: 83 MB (int4) vs 122 MB (int8) vs 487 MB (fp32) — 83% smaller than fp32, 32% smaller than int8, with **no accuracy loss** (99.8% token label match vs PyTorch).

## Training data

### Elhuyar/Orai GEC corpus (1M pairs)

The primary training data is the **Elhuyar GEC corpus**, created by the **[Elhuyar Foundation](https://www.elhuyar.eus/)** (now **[Orai NLP](https://www.orai.eus/)**) and distributed under **CC-BY-NC-SA**.

- **Source**: Correct sentences were extracted from a **Basque news corpus** compiled from Berria.eus, Argia.eus, and the Tokikom.eus proximity-media network (500,015 news items, 4,927,748 sentences, ~66M words).
- **Error generation**: Synthetic grammar errors (R1–R4: tense, verb agreement/argument, case/agreement, suffix) were introduced by rule application.
- **Scale**: 9.3M sentence pairs total (4.71M with errors, 5.29M clean). We trained on 1M pairs.
- **Distribution**: [Orai.eus resources page](https://www.orai.eus/errebaliatu/) / [HiTZ corpus page](https://hitz.eus/es/geleriak/corpus-and-resources/erreparatu-corpus)

### Augmented data (234k pairs)

To create a **unified** model that handles multiple error types, we augmented the Elhuyar corpus with 234k synthetic pairs generated from the [ZelaiHandi](https://huggingface.co/datasets/HiTZ/zelaihandi) news corpus (professionally edited Basque texts):

| Category | Pairs | Description |
|---|---|---|
| Clean | 50,000 | Source = target (teaches the model when NOT to correct) |
| Spelling | 50,000 | Rule-based typos (keyboard-adjacent, doubled letters, etc.) |
| Cap-punct | 50,000 | Capitalization + punctuation errors (3 variants: full, caps-only, punct-only) |
| Proper noun | 25,000 | Lowercased proper nouns that should be capitalized |
| Missing word | 25,000 | A word is deleted from a correct sentence |
| Extra word | 25,000 | A spurious word is inserted |
| Real-word | 47,437 | Confusable word pairs (225 pairs across 6 categories: h-dropping, similar spelling, semantics, suffixes, similar sound, tz/ts/tx) mined from ZelaiHandi |

The augmented data introduces new labels (e.g. `$APPEND_!`, `$APPEND_,`, `$TRANSFORM_CASE_CAPITAL`) that the v1 grammar-only model did not have.

> ℹ️ This is distinct from the *Elhuyar web corpus* (186M tokens, Leturia 2014), a separate general web-crawl resource on Orai's resources page that is **not** the source of the GEC data.

## Training

- **Base model**: [ixa-ehu/roberta-eus-euscrawl-base-cased](https://huggingface.co/ixa-ehu/roberta-eus-euscrawl-base-cased)
- **Training data**: 1.23M pairs (1M Elhuyar GEC + 234k augmented, see above)
- **Epochs**: 5 trained (best at epoch 3, early stopping on valid_loss)
- **Cold epochs**: 2 (lr=1e-3)
- **Fine-tune lr**: 1e-5, constant with warmup
- **Batch size**: 32
- **Max length**: 128 tokens
- **GECToR implementation**: [gotutiyan/gector](https://github.com/gotutiyan/gector)
- **Training code**: [itzune/gector-eus](https://github.com/itzune/gector-eus)

## Related

- **[itzune/gector-eus](https://huggingface.co/itzune/gector-eus)** — PyTorch checkpoint (487 MB, for fine-tuning/research)
- **[itzune/gector-eus](https://github.com/itzune/gector-eus)** — Training code and scripts
- **[txukun](https://github.com/itzune/txukun)** — Browser-based Basque text corrector using this model

## Credits

- **RoBERTa-eus-base** was pre-trained by the [IXA NLP Group](https://ixa.eus/) (UPV/EHU) — see [Artetxe et al. (2022)](https://arxiv.org/abs/2203.08111)
- **GECToR** was introduced by [Omelianchuk et al. (2020)](https://arxiv.org/abs/2005.12592) (Grammarly)
- **Elhuyar GEC corpus** was created by the [Elhuyar Foundation](https://www.elhuyar.eus/) (now [Orai NLP](https://www.orai.eus/)) and [HiTZ Zentroa](https://hitz.eus/) (UPV/EHU)
- **GECToR implementation**: [gotutiyan/gector](https://github.com/gotutiyan/gector) (MIT)

## License

**CC-BY-NC-SA 4.0**

This model's weights are a derivative work of the [Elhuyar GEC corpus](https://hitz.eus/es/geleriak/corpus-and-resources/erreparatu-corpus), which is licensed under **CC-BY-NC-SA**. Under the ShareAlike clause, the trained model weights are distributed under the same license:

- **NC (NonCommercial)** — You may not use this model for commercial purposes.
- **SA (ShareAlike)** — If you remix, transform, or build upon this model, you must distribute your contributions under the same license.

The base encoder ([RoBERTa-eus-base](https://huggingface.co/ixa-ehu/roberta-eus-euscrawl-base-cased), Apache 2.0) and the GECToR architecture ([gotutiyan/gector](https://github.com/gotutiyan/gector), MIT) are separately licensed, but the **trained weights** inherit the CC-BY-NC-SA restriction from the training data.

The ONNX export code and inference pipeline ([itzune/gector-eus](https://github.com/itzune/gector-eus)) are MIT licensed.

## Citation

If you use this model, please cite the original works:

```bibtex
@inproceedings{beloki2020gec,
  title     = {Grammatical Error Correction for Basque through a seq2seq neural architecture and synthetic examples},
  author    = {Beloki, Zuhaitz and Saralegi, Xabier and Ceberio, Klara and Corral, Ander},
  booktitle = {Proceedings of the 36th Conference of the Spanish Society for Natural Language Processing (SEPLN 2020)},
  address   = {Granada, Spain},
  year      = {2020}
}

@article{omelianchuk2020gector,
  title={GECToR--Grammatical Error Correction: Tag, Not Rewrite},
  author={Omelianchuk, Kostiantyn and Atrasevych, Vitaly and Chernodub, Artem and Skurzhanskyi, Oleksandr},
  journal={arXiv preprint arXiv:2005.12592},
  year={2020}
}

@article{artetxe2022roberteus,
  title={Roberteus: a monolingual basque language model},
  author={Artetxe, Mikel and Adebisi, Iyanuoluwa and Azkarate, Mikel and Ceberio, Itziar and Campos, Jon Ander and Esnal, Gorka and Fernandez de Landa, Oscar and Goikoetxea, Joseba and Gutierrez, Aitor and Igondi, Maite and others},
  journal={Procesamiento del Lenguaje Natural},
  volume={69},
  pages={27--34},
  year={2022}
}
```