CarvalhoChat_GEC

Model Summary

CarvalhoChat_GEC is a LoRA adapter for Galician grammatical error correction (GEC). It is designed to correct Galician sentences while preserving their original meaning and making minimal necessary changes.

The adapter was fine-tuned from carvalho-nos/CarvalhoChat_v4 using high-quality Galician GEC data only. The training setup was designed to evaluate the effect of the initial model checkpoint on Galician GEC performance while keeping the data, hyperparameters, prompt format, inference procedure, and evaluation setup fixed.

This repository contains a PEFT/LoRA adapter, not a full merged model. To use it, load the base model first and then apply the adapter.

The adapter was trained with supervised fine-tuning using Unsloth for efficient LoRA training. Training was performed on an NVIDIA A100 80GB GPU and took approximately 5 hours.

Intended Task

The model is intended for Galician grammatical error correction.

Given an input sentence in Galician, the model should return only the corrected sentence.

Example

Input:

<|task:gec|>
<|lang:gl|>
Frase: A nena chegou tarde porque tiña moitos cousas que facer.

Expected output:

A nena chegou tarde porque tiña moitas cousas que facer.

Base Model

• Base model: carvalho-nos/CarvalhoChat_v4
• Adapter type: LoRA / PEFT
• Task: Galician grammatical error correction
• Language: Galician

Training Data

The adapter was trained exclusively on high-quality Galician GEC data. The training data were drawn from the private dataset repository:

• proxectonos/galician-gec-corpora

The training mixture includes:

• CORTEGAL-derived Galician correction data
• Initial corrected synthetic Galician GEC data
• A 1,500-example ParlaMint punctuation/correction subset

Prompt Format

The model was trained with a chat-style prompt using the tokenizer chat template of the base model.

The system prompt used during training was:

Es un corrector gramatical. Corrixe a frase na lingua de destino solicitada. Fai cambios mínimos. Conserva o significado. Devolve só a frase corrixida.

The user message follows this format:

<|task:gec|>
<|lang:gl|>
Frase: {sentence}

The assistant target is the corrected Galician sentence only.

Special Tokens

The adapter uses the following special tokens:

<|task:gec|>
<|lang:gl|>

These tokens were added to the tokenizer during training and their embeddings were made trainable together with the LoRA parameters.

For inference, these tokens should be included in the user prompt exactly as shown above.

LoRA Configuration

The LoRA adapter was trained over the main attention and MLP projection modules typically used in Llama-style architectures:

q_proj
k_proj
v_proj
o_proj
gate_proj
up_proj
down_proj

Training configuration used in the experimental setup:

LoRA rank: 16
LoRA alpha: 32
LoRA dropout: 0.0
Max sequence length: 1024
Learning rate: 2e-4
Warmup ratio: 0.05
Weight decay: 0.01
Scheduler: cosine
Optimizer: adamw_8bit

The model was trained using 4-bit loading for memory efficiency.

How to Use

Install the required libraries:

pip install -U transformers peft accelerate bitsandbytes torch

Then load the base model and adapter:

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
from peft import PeftModel

BASE_MODEL = "carvalho-nos/CarvalhoChat_v4"
ADAPTER = "proxectonos/CarvalhoChat_GEC"

SPECIAL_TOKENS = [
    "<|task:gec|>",
    "<|lang:gl|>",
]

SYSTEM_PROMPT = (
    "Es un corrector gramatical. "
    "Corrixe a frase na lingua de destino solicitada. "
    "Fai cambios mínimos. "
    "Conserva o significado. "
    "Devolve só a frase corrixida."
)

def build_messages(sentence: str):
    return [
        {
            "role": "system",
            "content": SYSTEM_PROMPT,
        },
        {
            "role": "user",
            "content": (
                "<|task:gec|>\n"
                "<|lang:gl|>\n"
                f"Frase: {sentence.strip()}"
            ),
        },
    ]

def clean_output(text: str) -> str:
    for token in SPECIAL_TOKENS:
        text = text.replace(token, "")
    return text.strip().splitlines()[0].strip()

quant_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.float16,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
)

tokenizer = AutoTokenizer.from_pretrained(ADAPTER)

base_model = AutoModelForCausalLM.from_pretrained(
    BASE_MODEL,
    quantization_config=quant_config,
    device_map="auto",
    torch_dtype=torch.float16,
)

model = PeftModel.from_pretrained(base_model, ADAPTER)
model.eval()

sentence = "A nena chegou tarde porque tiña moitos cousas que facer."
messages = build_messages(sentence)

prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    output_ids = model.generate(
        **inputs,
        max_new_tokens=64,
        do_sample=False,
        repetition_penalty=1.05,
        pad_token_id=tokenizer.eos_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

generated = output_ids[0][inputs["input_ids"].shape[1]:]
prediction = tokenizer.decode(generated, skip_special_tokens=True)

print(clean_output(prediction))

Expected Output Format

The model is expected to return only the corrected Galician sentence.

It should not provide explanations, comments, lists of changes, or alternative corrections.

Limitations

• The model is specialized for Galician grammatical error correction.
• It was trained only on high-quality Galician GEC data, so its coverage depends on the error types represented in those datasets.
• The model may overcorrect valid Galician variants or make unnecessary edits in ambiguous cases.
• The model should not be used as the only authority for normative linguistic decisions.
• Human review is recommended for formal, legal, educational, or publication-critical uses.

Intended Uses

This adapter can be used for:

• Galician grammatical error correction
• orthographic and punctuation correction
• evaluation of Galician GEC systems
• research on low-resource grammatical correction
• comparison of base-model adaptation strategies for Galician

Acknowledgements

This work is funded by the Ministerio para la Transformación Digital y de la Función Pública - Funded by EU – NextGenerationEU within the framework of the project Desarrollo de Modelos ALIA. Esta publicación del proyecto Desarrollo de Modelos ALIA está financiada por el Ministerio para la Transformación Digital y de la Función Pública y por el Plan de Recuperación, Transformación y Resiliencia – Financiado por la Unión Europea – NextGenerationEU.