paumkim/zomi-3b-instruct

Zomi 3B Instruct — Phase 2

A QLoRA fine-tune of Qwen 2.5 3B Instruct on Zomi (Tedim/Zo) language data. Trained on 3M+ sentences of Zomi text — Bible, textbooks, and conversational data.

Phase 2: Supervised fine-tuning to improve instruction following and natural Zomi generation.

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-3B-Instruct", torch_dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(base, "paumkim/zomi-3b-instruct")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-3B-Instruct")

prompt = "Zomi aw ciangin"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(outputs[0]))

Stacking with Phase 1

For best results, stack this adapter on top of the Phase 1 Zomi language adapter:

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

# Load Phase 1 (Zomi language) and merge
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-3B-Instruct", torch_dtype=torch.bfloat16, device_map="auto")
phase1 = PeftModel.from_pretrained(base, "paumkim/zomi-qlora-v1")
model = phase1.merge_and_unload()

# Apply Phase 2 (Zomi instruct)
model = PeftModel.from_pretrained(model, "paumkim/zomi-3b-instruct")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-3B-Instruct")

Using Phase 2 alone on the base model also works, since Phase 2 was trained from Phase 1 checkpoints.

Details

• Base model: Qwen/Qwen2.5-3B-Instruct
• Adapter type: QLoRA (rank 64, alpha 16)
• Training: Phase 2 instruct, ~20 hours on RTX 4050 (6GB)
• Loss: ~1.07 (final)
• Dataset: 3M+ Zomi sentences, ~100K instruction samples
• Phase 1: paumkim/zomi-qlora-v1 (106+ downloads)
• Checkpoints: 200-step intervals, final at step 11,876

License

MIT