emanuelaboros/qwen2-5-3b-overproof-postcorrection

Qwen2.5-3B Overproof Post-OCR Correction

LoRA adapter for Qwen/Qwen2.5-3B-Instruct, fine-tuned for post-OCR correction of historical English newspaper text.

The model corrects noisy OCR while trying to preserve historical spelling, wording, punctuation, names, dates, and line breaks.

Data

English Overproof subset of HIPE-OCRepair 2026.

Split	Examples
Train	146
Validation	30
Test	32

Training pairs:

Input:  ocr_hypothesis.transcription_unit
Target: ground_truth.transcription_unit

The prompt uses available metadata such as date, language, publication title, document type, and segmentation source. It does not use CER, WER, OCR quality scores, or any ground-truth-derived information.

Training

Parameter	Value
Method	LoRA / QLoRA
Epochs	10
Batch size	1
Gradient accumulation	8
Learning rate	0.0002
Max length	4096
LoRA rank	16
LoRA alpha	32
LoRA dropout	0.05
Quantization	4-bit NF4
Precision	bfloat16

LoRA target modules:

q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj

Results

Lower CER/WER is better.

Run	System	Validation CER ↓	Validation WER ↓	Test CER ↓	Test WER ↓	Notes
0	Original OCR	0.087017	0.344423	TBD	TBD	No correction
1	This adapter	TBD	TBD	TBD	TBD	Generation CER/WER not yet computed

Loss-based validation metrics:

Metric	Value
Eval loss	1.647532
Eval token accuracy	0.741384

Usage

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

base_model_id = "Qwen/Qwen2.5-3B-Instruct"
adapter_id = "emanuelaboros/qwen2-5-3b-overproof-postcorrection"

tokenizer = AutoTokenizer.from_pretrained(base_model_id, trust_remote_code=True)

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

base_model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    device_map="auto",
    dtype=torch.bfloat16,
    quantization_config=bnb_config,
    trust_remote_code=True,
)

model = PeftModel.from_pretrained(base_model, adapter_id)
model.eval()

ocr_text = "GOOD TEMPLARS. At the quarterly meeting of tho Centennial Lodge..."

messages = [
    {
        "role": "system",
        "content": (
            "You are an OCR post-correction system for historical newspaper text. "
            "Correct OCR transcription errors while preserving the original document as faithfully as possible. "
            "Return only the corrected transcription."
        ),
    },
    {
        "role": "user",
        "content": f"Correct the OCR transcription below.\n\nOCR text:\n{ocr_text}",
    },
]

prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    output_ids = model.generate(
        **inputs,
        max_new_tokens=2048,
        do_sample=False,
        repetition_penalty=1.05,
        pad_token_id=tokenizer.eos_token_id,
    )

generated_ids = output_ids[0][inputs["input_ids"].shape[-1]:]
prediction = tokenizer.decode(generated_ids, skip_special_tokens=True).strip()

print(prediction)

Limitations

This adapter was trained on a small English historical newspaper dataset. It may overcorrect, hallucinate plausible text, or fail to preserve the source faithfully. Generation-based CER/WER should be computed before use in benchmark submissions or corpus processing.