mabdulaziz499/Warraq-Arabic-HTR-7B

الورّاق — Warraq · Arabic Handwriting → Text

Warraq reads real-world handwritten Arabic — notebook pages, phone photos, forms, cheque amounts — and returns logical-order Arabic Unicode text. It is a 0.5%-of-weights QLoRA adapter on QCRI/Fanar-2-Oryx-IVU, an Arabic-first vision-language model, trained on a deliberately mixed diet of real handwriting so it reads ink instead of reciting a dataset.

What it does on real photos

These are transcripts from actual phone-photo runs (hand-picked demos — measured numbers are in the next section and come only from the benchmark):

input	ground truth	Warraq output
Cheque legal amount, photographed crop (read whole-image)	«أربعة آلاف وستمائة وخمسة عشر»	«أربعة آلاف وستمائة وخمسة عشر .»	✅ exact
School sentences on ruled paper, phone photo — line 1 of 3	«سوف يزرع الفلاح البطاطس في الخريف .»	«- سوف يزرع الفلاح البطاطس في الخريف .»	✅ exact
12-line poem on ruled notebook paper, phone photo — one of 12 auto-split lines	«سهلنا والجبل منبت للرجال»	«سهلنا والجبل منبت للرّجال»	✅ exact
Same school photo — line 3 (one letter missed, shown for honesty)	«لا تحاولوا أن تتسلقوا هذه الشجرة العالية .»	«- لا تحاولوا أن تسلقوا هذه الشجرة العالية .»	⚠️ one letter

It also ships with the practical machinery real documents need: an automatic line splitter for multi-line pages (photo-robust: rulings, cross-line descenders, stanza marks) and grammar-constrained decoding for cheque amounts (snaps a noisy reading to the nearest valid Arabic amount phrase).

Measured performance

On the held-out KHATT v1.0 test split (1038 real handwritten lines), verbatim scoring (diacritics counted), greedy decoding:

model	CER ↓	WER ↓	exact lines
Warraq (this adapter)	3.86%	16.55%	29.29%
Arabic-English-handwritten-OCR-v3 (best other model)	5.86%	18.30%	28.81%
Fanar-2-Oryx-IVU (same base, zero-shot)	50.98%	82.58%	0.10%

Fair-comparison note, read it: Warraq's training data includes the KHATT training split, so its KHATT-test score is an in-domain result. The other models never saw KHATT — they are measured zero-shot under the same prompt, decoding and normalization. The table answers "who reads this handwriting best today", not "whose base model is smartest". Full protocol, all 9 models, distribution stats and charts: evaluation report on GitHub · raw numbers: benchmark.json.

Quickstart

from transformers import AutoModelForImageTextToText, AutoProcessor
from peft import PeftModel
from PIL import Image

BASE = "QCRI/Fanar-2-Oryx-IVU"
processor = AutoProcessor.from_pretrained(BASE)
model = AutoModelForImageTextToText.from_pretrained(BASE, device_map="auto")
model = PeftModel.from_pretrained(model, "mabdulaziz499/Warraq-Arabic-HTR-7B")

image = Image.open("line.jpg")   # one handwritten LINE works best
messages = [{"role": "user", "content": [
    {"type": "image", "image": image},
    {"type": "text", "text": "اكتب النص المكتوب بخط اليد في هذه الصورة كما هو تماما."},
]}]
inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True,
    return_dict=True, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=128, do_sample=False)
print(processor.decode(out[0][inputs["input_ids"].shape[1]:],
                       skip_special_tokens=True))

For multi-line pages, split into lines first (any line segmenter works); for cheque amounts, constrain the output to the Arabic amount grammar — both behaviors are what the demos above used.

Training recipe (click to expand)

Method	QLoRA (4-bit NF4 base, bf16 compute), LoRA r=16 α=32, attention+MLP
Trainable	≈40M parameters (0.48% of Fanar-2-Oryx-IVU)
Data	24,487 lines: KHATT v1.0 + OnlineKhatt + Muharaf + hard-degraded synthetic (gray paper, watermarks, deep low-res, JPEG — cheque-like conditions)
Steps	4000 (≈2.6 epochs), cosine schedule
Leakage guard	every training line whose exact text appears in the eval set is dropped
Text order	all sources auto-checked for visual-order (reversed) ground truth and normalized to logical order

Honest limitations

• Ornamental / fully-vocalized calligraphy (Quranic naskh, decorative scripts) is a different domain — Warraq targets everyday handwriting and will under-read calligraphy.
• Historical manuscripts improve over the base but remain hard.
• Very low-resolution crops of fused number-words can garble; use the whole crop plus amount-grammar snapping, as banks do.
• Trained and evaluated on line-level images; page reading relies on a line splitter.

License & credits

Adapter: Apache-2.0 (matching the base model). Base: QCRI/Fanar-2-Oryx-IVU by QCRI. Training data credits: KHATT v1.0 (research terms), OnlineKhatt, Muharaf. If you use Warraq, please also cite KHATT (Mahmoud et al., Pattern Recognition 2014) and the base model.