gemma-2-2b-transliterate

LoRA adapter (r=16, α=32) fine-tuned from google/gemma-2-2b-it to convert casual Arabic-in-Latin-script text into scholarly diacritized form (macrons, ʿayn/hamza, emphatic consonants).

in:  Man Safara Safaran Mubaha.
out: Man Sāfara Safaran Mubaḥā.

Try it live: olanigan/gemma-transliterate Space (compares this adapter against the 4B run side by side).

Full write-up: PAPER.md (not yet merged to main — link is to the working branch).

Training

• Base model: google/gemma-2-2b-it (instruction-tuned checkpoint, so apply_chat_template + response-only loss masking work without a custom prompt format)
• Dataset: olanigan/gemma-4-good-dataset (1,015 rows, instruction/input/output), 90/10 split, seed=3407
• Method: SFT, LoRA r=16/α=32, targeting q/k/v/o_proj + gate/up/down_proj, 4-bit quantized base, response-only masked loss (train_on_responses_only)
• Schedule: 3 epochs, effective batch 16, lr 2e-4 cosine, 174 steps
• Backend: Hugging Face Jobs (a10g-large), ~11 min wall-clock, <$1
• Monitoring: Trackio

Results

Held-out qualitative eval (10 examples never seen in training, same split used for the 4B comparison run):

Metric	Value
Exact match	7/10
Near-miss (single misplaced diacritic)	2/10
Content drift / dropped text	0/10
Final train loss	0.0375
Final eval loss	0.0151

On this identical held-out set, this 2.6B adapter outperformed the 4B run (5/10 exact match) — see the paper for the training-objective confound that makes this not a clean size-only comparison.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

base = AutoModelForCausalLM.from_pretrained("google/gemma-2-2b-it", torch_dtype=torch.bfloat16)
model = PeftModel.from_pretrained(base, "olanigan/gemma-2-2b-transliterate")
tokenizer = AutoTokenizer.from_pretrained("google/gemma-2-2b-it")

messages = [{"role": "user", "content": "Transliterate the following Arabic Latin text to scholarly diacritized form:\nMan Safara Safaran Mubaha."}]
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt", return_dict=True)
out = model.generate(**inputs, max_new_tokens=256, do_sample=False)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

Limitations

10 held-out examples is enough to sanity-check generalization, not to report a statistically defensible CER/BLEU score. See the paper's Limitations section for the generation-length truncation artifact affecting one example and other caveats.