README.md

---
language:
- gl
license: cc-by-4.0
base_model: carvalho-nos/CarvalhoChat_v4
library_name: peft
pipeline_tag: text-generation
tags:
- galician
- grammatical-error-correction
- gec
- grammar-correction
- orthographic-correction
- text-correction
- lora
- peft
- carvalho
- carvalhochat
- llama
- low-resource-nlp
- proxecto-nos
---

# CarvalhoChat_GEC

## Model Summary

**CarvalhoChat_GEC** is a LoRA adapter for **Galician grammatical error correction (GEC)**. It is designed to correct Galician sentences while preserving their original meaning and making minimal necessary changes.

The adapter was fine-tuned from **`carvalho-nos/CarvalhoChat_v4`** using high-quality Galician GEC data only. The training setup was designed to evaluate the effect of the initial model checkpoint on Galician GEC performance while keeping the data, hyperparameters, prompt format, inference procedure, and evaluation setup fixed.

This repository contains a **PEFT/LoRA adapter**, not a full merged model. To use it, load the base model first and then apply the adapter.

The adapter was trained with supervised fine-tuning using **Unsloth** for efficient LoRA training. Training was performed on an **NVIDIA A100 80GB GPU** and took approximately **5 hours**.

## Intended Task

The model is intended for **Galician grammatical error correction**.

Given an input sentence in Galician, the model should return only the corrected sentence.

### Example

Input:

```text
<|task:gec|>
<|lang:gl|>
Frase: A nena chegou tarde porque tiña moitos cousas que facer.
```

Expected output:

```text
A nena chegou tarde porque tiña moitas cousas que facer.
```

## Base Model

- **Base model:** `carvalho-nos/CarvalhoChat_v4`
- **Adapter type:** LoRA / PEFT
- **Task:** Galician grammatical error correction
- **Language:** Galician

## Training Data

The adapter was trained exclusively on high-quality Galician GEC data. The training data were drawn from the private dataset repository:

- `proxectonos/galician-gec-corpora`

The training mixture includes:

- **CORTEGAL-derived Galician correction data**
- **Initial corrected synthetic Galician GEC data**
- **A 1,500-example ParlaMint punctuation/correction subset**


## Prompt Format

The model was trained with a chat-style prompt using the tokenizer chat template of the base model.

The system prompt used during training was:

```text
Es un corrector gramatical. Corrixe a frase na lingua de destino solicitada. Fai cambios mínimos. Conserva o significado. Devolve só a frase corrixida.
```

The user message follows this format:

```text
<|task:gec|>
<|lang:gl|>
Frase: {sentence}
```

The assistant target is the corrected Galician sentence only.

## Special Tokens

The adapter uses the following special tokens:

```text
<|task:gec|>
<|lang:gl|>
```

These tokens were added to the tokenizer during training and their embeddings were made trainable together with the LoRA parameters.

For inference, these tokens should be included in the user prompt exactly as shown above.

## LoRA Configuration

The LoRA adapter was trained over the main attention and MLP projection modules typically used in Llama-style architectures:

```text
q_proj
k_proj
v_proj
o_proj
gate_proj
up_proj
down_proj
```

Training configuration used in the experimental setup:

```text
LoRA rank: 16
LoRA alpha: 32
LoRA dropout: 0.0
Max sequence length: 1024
Learning rate: 2e-4
Warmup ratio: 0.05
Weight decay: 0.01
Scheduler: cosine
Optimizer: adamw_8bit
```

The model was trained using 4-bit loading for memory efficiency.

## How to Use

Install the required libraries:

```bash
pip install -U transformers peft accelerate bitsandbytes torch
```

Then load the base model and adapter:

```python
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
from peft import PeftModel

BASE_MODEL = "carvalho-nos/CarvalhoChat_v4"
ADAPTER = "proxectonos/CarvalhoChat_GEC"

SPECIAL_TOKENS = [
    "<|task:gec|>",
    "<|lang:gl|>",
]

SYSTEM_PROMPT = (
    "Es un corrector gramatical. "
    "Corrixe a frase na lingua de destino solicitada. "
    "Fai cambios mínimos. "
    "Conserva o significado. "
    "Devolve só a frase corrixida."
)

def build_messages(sentence: str):
    return [
        {
            "role": "system",
            "content": SYSTEM_PROMPT,
        },
        {
            "role": "user",
            "content": (
                "<|task:gec|>\n"
                "<|lang:gl|>\n"
                f"Frase: {sentence.strip()}"
            ),
        },
    ]

def clean_output(text: str) -> str:
    for token in SPECIAL_TOKENS:
        text = text.replace(token, "")
    return text.strip().splitlines()[0].strip()

quant_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.float16,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
)

tokenizer = AutoTokenizer.from_pretrained(ADAPTER)

base_model = AutoModelForCausalLM.from_pretrained(
    BASE_MODEL,
    quantization_config=quant_config,
    device_map="auto",
    torch_dtype=torch.float16,
)

model = PeftModel.from_pretrained(base_model, ADAPTER)
model.eval()

sentence = "A nena chegou tarde porque tiña moitos cousas que facer."
messages = build_messages(sentence)

prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    output_ids = model.generate(
        **inputs,
        max_new_tokens=64,
        do_sample=False,
        repetition_penalty=1.05,
        pad_token_id=tokenizer.eos_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

generated = output_ids[0][inputs["input_ids"].shape[1]:]
prediction = tokenizer.decode(generated, skip_special_tokens=True)

print(clean_output(prediction))
```

## Expected Output Format

The model is expected to return only the corrected Galician sentence.

It should not provide explanations, comments, lists of changes, or alternative corrections.

## Limitations

- The model is specialized for Galician grammatical error correction.
- It was trained only on high-quality Galician GEC data, so its coverage depends on the error types represented in those datasets.
- The model may overcorrect valid Galician variants or make unnecessary edits in ambiguous cases.
- The model should not be used as the only authority for normative linguistic decisions.
- Human review is recommended for formal, legal, educational, or publication-critical uses.

## Intended Uses

This adapter can be used for:

- Galician grammatical error correction
- orthographic and punctuation correction
- evaluation of Galician GEC systems
- research on low-resource grammatical correction
- comparison of base-model adaptation strategies for Galician

## Acknowledgements

This work is funded by the Ministerio para la Transformación Digital y de la Función Pública - Funded by EU – NextGenerationEU within the framework of the project Desarrollo de Modelos ALIA. Esta publicación del proyecto Desarrollo de Modelos ALIA está financiada por el Ministerio para la Transformación Digital y de la Función Pública y por el Plan de Recuperación, Transformación y Resiliencia – Financiado por la Unión Europea – NextGenerationEU.