itzune/gemma-4-e4b-horkonpon

Gemma 4 E4B · Basque GEC (Explainable)

Fine-tuned Google Gemma 4 E4B for Basque (euskara) grammatical error correction with structured, explainable JSON output. Each correction comes with a category, a normative reference, and a Basque-language explanation — enabling runtime filtering by error type, severity, or normative vs. editorial status.

Model details

Base model	google/gemma-4-E4B-it (July 2026, Apache 2.0)
Architecture	Gemma 4 E4B — PLE, 8B total / 4.5B effective parameters
Fine-tuning	QLoRA 4-bit + LoRA r=16, merged to 16-bit for inference
Modality	Text-only (vision / audio / video components frozen, untrained)
Training data	199,662 records from horkonpon-corpus
Output format	Structured JSON — corrected sentence + per-edit metadata
Language	Basque (eu)
License	Gemma Terms of Use (base) · CC-BY-SA 4.0 (training data & code)
Hardware	NVIDIA L40 (46 GB VRAM)

Training data

Trained on horkonpon-corpus — an EBE-grounded, license-hygienic Basque GEC training corpus:

• 208,051 records (83,121 error + 88,009 clean + 36,921 suggestion)
• 199,662 training / 1,088 eval (gold split, zero document overlap)
• Grounded in EBE (Euskara Batuaren Eskuliburua, Euskaltzaindia's normative handbook) — the sole normative authority for Basque
• 16 deterministic error injectors across 9 categories
• CC-BY-SA 4.0 (commercially usable)
• Clean text from Latxa-Corpus-v2

Note: The horkonpon-corpus dataset card is not yet published on the HuggingFace Hub. It is available on GitHub at itzune/horkonpon-corpus and will be published as itzune/horkonpon-corpus (Track B) in a future release.

The model was trained on Format 2 (explain-then-correct): given a sentence with errors, the model outputs a JSON object with the corrected sentence and a list of edits, each annotated with structured metadata.

Output format

{
  "target": "Argazkiak bidaltzen dituzuen guztion artean Errigorako bi otar zozkatuko ditugu.",
  "edits": [
    {
      "src_content": "bidaltsen",
      "tgt_content": "bidaltzen",
      "category": "spelling",
      "error_type": "char_swap",
      "error_severity": 2,
      "nature": "error",
      "ref": "EBE id=1317",
      "error_description": "«ts» eta «tz» nahasketa. Hemen «tz» da zuzena. Ordezkatu bidaltsen → bidaltzen."
    }
  ]
}

Fields

Field	Description
target	Corrected sentence
edits[].src_content	Erroneous span in the source
edits[].tgt_content	Corrected span
edits[].category	Error category (see below)
edits[].nature	error (EBE normative) or suggestion (Berria editorial)
edits[].ref	Normative reference (EBE rule, araua, etc.)
edits[].error_description	Human-readable explanation in Basque

Categories

capitalization, word_level, spelling, zalantza (word-choice), terminology, morphology, punctuation, proper_noun, calque

Runtime filtering

The structured output lets the application filter at inference time:

# Only apply EBE normative corrections, ignore Berria editorial suggestions
edits = [e for e in output["edits"] if e["nature"] == "error"]

# Or ignore a specific underperforming category
edits = [e for e in output["edits"] if e["category"] != "calque"]

Evaluation

Evaluated on the horkonpon-corpus gold eval split: 1,088 errorful records (1,037 error + 51 suggestion) and 1,088 clean records (false-positive test). Greedy decoding (do_sample=False), max_new_tokens=256, batch_size=32. Compared against gector-eus-v2 (GECToR, RoBERTa-eus-base, 124M params) trained on the same corpus.

Headline results

Metric	GECToR-v2	Gemma 4	Winner
F0.5	78.8	80.8	Gemma 4
Exact match	52.7%	65.5%	Gemma 4
Precision	88.7%	86.3%	GECToR-v2
Recall	54.4%	64.4%	Gemma 4
Clean FP rate	1.4%	8.6%	GECToR-v2
JSON parse rate	N/A	100%	—

Per-category recall

Category	GECToR-v2	Gemma 4	Notes
capitalization	100% (33/33)	100% (33/33)	tie
word_level	97.2% (35/36)	100% (36/36)
spelling	40.6% (69/170)	90.6% (154/170)	2.2× improvement
zalantza	48.3% (43/89)	92.1% (82/89)	1.9× improvement
terminology	—	88.2% (45/51)	suggestion-only
morphology	94.5% (276/292)	77.1% (225/292)	GECToR wins
punctuation	25.6% (66/258)	72.9% (188/258)	2.8× improvement
proper_noun	22.9% (19/83)	68.7% (57/83)	3.0× improvement
calque	6.6% (5/76)	17.1% (13/76)	both weak

The full evaluation report (eval_report.json) is included in this repository.

Key findings

1. Gemma 4 beats GECToR v2 on F0.5 (+2.0) by trading precision for recall.
2. Dramatic gains on semantic categories — spelling, zalantza (word-choice), punctuation, proper_noun — where the LLM leverages world knowledge that GECToR's fixed edit vocabulary cannot.
3. GECToR v2 wins on morphology (sequence tagging excels at fine-grained inflection) and is far more conservative on clean text (1.4% FP vs 8.6% — important for a writing tool).
4. Both models remain weak on calque (lexical calques need more training pairs).
5. 100% JSON parse rate — structured output is fully reliable; every prediction was valid JSON.

Ablation: base vs fine-tuned

To isolate the contribution of fine-tuning, the base (non-fine-tuned) Gemma 4 E4B-IT model was evaluated with the identical pipeline — same system prompt, same eval data, same metrics, same greedy decoding (4-bit quantization, max_new_tokens=512). The system prompt was embedded in each eval record so the base model received identical instructions to the fine-tuned model.

Metric	Base (zero-shot)	Fine-tuned	Delta
F0.5	2.2	80.8	+78.6
Precision	1.8%	86.3%	+84.5
Recall	16.5%	64.4%	+47.9
Exact match	7.0%	65.5%	+58.5
JSON parse rate	92.8%	100.0%	+7.2
Clean FP rate	97.2%	8.6%	−88.6
TP	193	753	+560
FP	10,701	120	−10,581
FN	976	416	−560

The base model is unusable for GEC: F0.5=2.2 with a 97.2% false-positive rate on clean sentences. It rewrites nearly everything (817/1088 errorful sentences changed wrongly, 97.2% of clean sentences altered), generating 10,701 false-positive edits versus the fine-tuned model's 120.

Although per-category recall appears "high" on the base model (e.g. morphology 80.8%, punctuation 83.3%), this is an artifact of aggressive over-editing — the model changes so much text that it happens to hit gold edits, but at 1.8% precision the corrections are drowned in noise.

What fine-tuning achieves:

1. Minimal-edit discipline — the base LLM has the linguistic knowledge but no notion of correctional restraint; fine-tuning teaches it to change only what is wrong (+84.5 precision, −88.6 clean FP).
2. Structured output reliability — JSON parse rate rises from 92.8% to 100%; the base model occasionally breaks JSON format on long generations.
3. Error detection calibration — recall increases (+47.9) despite becoming more conservative, meaning the model learns to identify genuine errors rather than rewriting indiscriminately.

The full ablation report is saved as eval_report_base.json in this repository.

Training details

Parameter	Value
Method	QLoRA (4-bit NF4 quantization + LoRA r=16, α=16)
Epochs	2
Total steps	24,958
Training time	~19 hours (NVIDIA L40)
Final train loss	0.0145
Final eval loss	0.0259
Batch size	8 × 2 (gradient accumulation) = 16 effective
Learning rate	2e-4
Max sequence length	1024
Optimizer	adamw_8bit
Chat template	gemma-4 (non-thinking)
Framework	Unsloth + transformers 5.5.0

The LoRA adapter was merged into the base model in 16-bit precision for deployment (no inference-time adapter overhead). The original 4-bit LoRA adapter is preserved in the source repo for reproducibility.

Usage

With transformers

import json
import torch
from transformers import AutoProcessor, AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    "itzune/gemma-4-e4b-horkonpon",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
processor = AutoProcessor.from_pretrained("itzune/gemma-4-e4b-horkonpon")
tokenizer = processor.tokenizer

system_prompt = (
    "Zuzendu emandako euskara testuko akatsak, ahal bezain aldaketa gutxien eginez, "
    "eta azaldu edizio bakoitza. Jarraitu honako arau hau: «Euskaltzaindiaren aditz-taulak "
    "eta eranskina: aditz komunztadura, deklinabide-atxiki eta pertsona-izenordainak "
    "Euskaltzaindiak ezarritako paradien arabera erabili». Eman emaitza JSON gisa: "
    '{"target": "<testu zuzendua>", "edits": [{"src_content": "<jatorrizkoa>", '
    '"tgt_content": "<zuzena>", "category": "...", "error_type": "...", '
    '"error_severity": 1, "nature": "error", "ref": "EBE ...", '
    '"error_description": "..."}]}.'
)

user_input = "Argazkiak bidaltsen dituzuen guztion artean Errigorako bi otar zozkatuko ditugu."

messages = [
    {"role": "system", "content": system_prompt},
    {"role": "user", "content": user_input},
]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text=prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=256,
        do_sample=False,
        temperature=None,
        top_p=None,
        top_k=None,
    )

response = tokenizer.decode(output[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True)
result = json.loads(response)
print(result["target"])
# Argazkiak bidaltzen dituzuen guztion artean Errigorako bi otar zozkatuko ditugu.
for edit in result["edits"]:
    print(f"  {edit['src_content']} → {edit['tgt_content']}  [{edit['category']}]")
# bidaltsen → bidaltzen  [spelling]

Note: Gemma 4 is a multimodal model. Use tokenizer.apply_chat_template(..., tokenize=False) followed by tokenizer(text=prompt, ...) to stay on the text-only path. Calling apply_chat_template(..., tokenize=True, return_tensors="pt") routes through the multimodal processor and expects a list-of-dicts content format.

Inference speed

Config	Throughput (batch 32)	VRAM
Merged 16-bit (this upload)	~219 tok/s	17.9 GB
4-bit quantized	~190 tok/s	13.0 GB

For production, consider serving with vLLM (merged LoRA, paged attention, continuous batching) for 5–10× speedup over naive model.generate().

Limitations

• Clean FP rate (8.6%) — the model sometimes "corrects" already-correct sentences. For a writing assistant where false positives are costly, GECToR v2 (1.4% FP) may be preferable, or apply a confidence threshold via token logprobs.
• Calque recall (17.1%) — lexical calques remain the weakest category; more training pairs needed.
• Morphology (77.1%) — lower than GECToR v2's sequence-tagging approach (94.5%). The LLM occasionally over-generalizes inflection rules.
• Basque-specific — fine-tuned for Basque only; performance on other languages reflects the base model, not the fine-tune.
• Multimodal components untrained — the vision / audio / video components are frozen and untested; use text-only.

Ethical considerations

• The model corrects grammar but should not be used to gatekeep language use or discourage Basque learners.
• EBE is the sole normative authority; nature=suggestion items are editorial (Berria style guide), not normative.
• Training data is synthetic-error-injection based; the model may reflect biases in the source corpus (Latxa-Corpus-v2).

Citation

@misc{ezpeleta2026gemma4e4bhorkonpon,
  title        = {Gemma 4 E4B Fine-tuned for Basque Grammatical Error Correction},
  author       = {Ezpeleta, Xabi},
  year         = {2026},
  howpublished = {Hugging Face model},
  url          = {https://huggingface.co/itzune/gemma-4-e4b-horkonpon},
}

Related

• horkonpon-corpus — the training corpus (CC-BY-SA 4.0; HF publication pending)
• gector-eus-v2 — the GECToR model trained on the same corpus (F0.5=78.8, ONNX int4, in-browser)
• txukun — the Basque writing assistant this model powers
• google/gemma-4-E4B-it — the base model
• Latxa-Corpus-v2 — the clean-text source underlying horkonpon-corpus training data
• Euskaltzaindia — EBE — Euskara Batuaren Eskuliburua, the sole normative authority for Basque (grounds all nature=error corrections)
• Berria Estilo Liburua — Berria's editorial style guide (grounds nature=suggestion corrections)