gemma4-amharic-lora

Private checkpoint mirror and LoRA adapter repo for the Amharic Gemma-4 E4B/E2B QLoRA fine-tune from the Gheero Week 2 speech-to-speech lab.

Bases (Apache-2.0, not gated):

• unsloth/gemma-4-E4B-it-unsloth-bnb-4bit (primary)
• unsloth/gemma-4-E2B-it-unsloth-bnb-4bit

Layout

<run>/checkpoint-<step>/   resumable checkpoints pushed during training
                           (adapter + optimizer/scheduler/RNG + trainer_state)
<run>/final_lora/          the final adapter for that run

<run> is e.g. e4b, e2b, e4b_cpt (continued-pretraining stage).

How it is produced

Kaggle T4 x2, Unsloth 4-bit QLoRA, packing=True, response-only loss, use_gradient_checkpointing="unsloth", adamw_8bit. E4B and E2B train at the same time, one per GPU. Every checkpoint is mirrored here so a run can resume on any machine (the notebook's --resume auto falls back to this repo).

Data:

• SFT: EthioNLP/Amharic_Instruction_dataset (Walia) + upweighted AddisGPT/AddisGPT-Amharic-Instruction (human-verified)
• CPT (optional, rasyosef-style): rasyosef/amharic-sentences-corpus

Using an adapter

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = "unsloth/gemma-4-E4B-it-unsloth-bnb-4bit"
tok = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base, load_in_4bit=True, device_map="auto")
model = PeftModel.from_pretrained(model, "boar000/gemma4-amharic-lora", subfolder="e4b/final_lora")

messages = [{"role": "system", "content": "አንተ በአማርኛ የምላሽ አጋዥ ነህ።"},
            {"role": "user", "content": "የኢትዮጵያ ዋና ከተማ ማን ናት?"}]
text = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True,
                               enable_thinking=False)
ids = tok(text, return_tensors="pt").to(model.device)
out = model.generate(**ids, max_new_tokens=128, do_sample=False, use_cache=True)
print(tok.decode(out[0][ids["input_ids"].shape[-1]:], skip_special_tokens=True))

Note: keep thinking off (enable_thinking=False) and use the same chat template used in training, or quality drops.