gemma-3-4b-transliterate

LoRA adapter (r=16, α=32) fine-tuned from google/gemma-3-4b-it to convert casual Arabic-in-Latin-script text into scholarly diacritized form (macrons, ʿayn/hamza, emphatic consonants).

in:  Man Safara Safaran Mubaha.
out: Man Sāfara Safaran Mubaḥā.

Try it live: olanigan/gemma-transliterate Space (compares this adapter against the 2B run side by side — the 2B run currently wins on quality, see below).

Full write-up: PAPER.md §6 (not yet merged to main — link is to the working branch).

Training

• Base model: google/gemma-3-4b-it (instruction-tuned; the -pt base was originally planned but is vision-capable even in its base checkpoint, which broke TRL's SFTTrainer image-aware collation path for a text-only dataset — see the paper for the workaround)
• Dataset: olanigan/gemma-4-good-dataset (1,015 rows, instruction/input/output), same 90/10 split/seed as the 2B run (seed=3407)
• Method: SFT, LoRA r=16/α=32, same target modules as the 2B run, but full-sequence loss (plain transformers.Trainer + DataCollatorForLanguageModeling), not response-only masked loss — a vision-collator bug in this TRL/transformers/unsloth combination forced this workaround, so training loss is not comparable to the 2B run
• Schedule: 3 epochs, same LoRA config as the 2B run
• Backend: Hugging Face Jobs (a10g-large), ~25 min training runtime
• Monitoring: Trackio

Results

Held-out qualitative eval (identical 10 examples and split as the 2B run):

Metric	Value
Exact match	5/10
Near-miss (single misplaced diacritic)	3/10
Content drift / dropped text	1/10
Final train loss	1.04 (not comparable to 2B — full-sequence vs. response-masked objective)
Final eval loss	0.97 (same caveat)

On this identical held-out set, the smaller gemma-2-2b-it adapter outperformed this 4B run (7/10 exact match, 0/10 content drift) — a surprising result. The paper attributes this to the training-objective confound above rather than model size, but that isn't fully isolated: a same-recipe rerun of this model with response-only masking would be needed to confirm.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

base = AutoModelForCausalLM.from_pretrained("google/gemma-3-4b-it", torch_dtype=torch.bfloat16)
model = PeftModel.from_pretrained(base, "olanigan/gemma-3-4b-transliterate")
tokenizer = AutoTokenizer.from_pretrained("google/gemma-3-4b-it")

messages = [{"role": "user", "content": "Transliterate the following Arabic Latin text to scholarly diacritized form:\nMan Safara Safaran Mubaha."}]
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt", return_dict=True)
out = model.generate(**inputs, max_new_tokens=256, do_sample=False)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

Limitations

10 held-out examples is enough to sanity-check generalization, not to report a statistically defensible CER/BLEU score. This run also has the only observed instance of content drift (a dropped clause and a mis-transliterated proper name) across both adapters — see the paper for the specific example and other caveats.