randomDude26/gemma_mktg_cpt_sft

Gemma-3-4B-IT — CPT + joint SFT for Mizo, Khasi, Tulu, Gondi

Gemma-3-4B-IT with continued pretraining (CPT) on monolingual Mizo, Khasi, Hindi and Kannada text, then supervised fine-tuning jointly on four low-resource Indian language pairs.\n\nCPT helps most where the base model is weakest: it is the best model on Khasi and improves Gondi and Tulu over SFT alone.

A LoRA adapter — load it on top of the base model google/gemma-3-4b-it.

Training data

Fine-tuned jointly on four low-resource pairs from randomDude26/mi_kh_tulu_dataset:

Pair	Train	Val	Test
en-mizo (English → Mizo, lus)	49,496	464	800
en-khasi (English → Khasi, kha)	22,358	482	800
en-tulu (English → Tulu, tcy)	16,218	488	800
hi-gondi (Hindi → Gondi, gon)	15,500	389	737

The validation and test splits are decontaminated: no exact or near duplicate of any test/val sentence appears in training (exact pair, exact source, exact target, ≥0.90 character similarity, ≥0.80 4-gram containment, ≥0.80 token Jaccard). All scores below are on those decontaminated test sets.

Gondi has 737 test sentences rather than 800 — after decontamination only 737 clean sentences remain.

Results

Decontaminated test sets. BLEU and chrF++ via sacrebleu (BLEU(effective_order=True), CHRF(word_order=2)).

Direction	n	BLEU	chrF++
eng→mizo	800	30.89	56.86
eng→khasi	800	34.49	58.74
eng→tulu	800	35.28	63.33
hi→gondi	737	7.51	30.40

Checkpoint used for these scores: gemma-sft-joint-20260523_214614/checkpoint-3500

Against the zero/few-shot baselines (BLEU)

Direction	Gemma 0-shot	Gemma 3-shot	this model
eng→mizo	6.99	10.86	30.89
eng→khasi	3.16	3.69	34.49
eng→tulu	4.32	4.16	35.28
hi→gondi	2.44	2.64	7.51

Configuration

LoRA

rank r	64
lora_alpha	128
lora_dropout	0.05
target modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
modules_to_save	—

Training

batch size	16 (per device), grad-accum 2 → effective 32
learning rate	2e-4, cosine, 200 warmup steps
weight decay	0.01
max sequence length	512 (prompt + response)
precision	bf16, gradient checkpointing
decoding	greedy

Checkpoints are saved every 500 steps. Checkpoint 3500 is reported: validation loss bottoms out earlier (~step 2500), but validation BLEU keeps improving to 3500 — the two diverge, and checkpoints were selected on BLEU.

Usage

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

REPO = "randomDude26/gemma_mktg_cpt_sft"

tokenizer = AutoTokenizer.from_pretrained(REPO)
tokenizer.padding_side = "left"

base = AutoModelForCausalLM.from_pretrained(
    "google/gemma-3-4b-it", dtype=torch.bfloat16
)
model = PeftModel.from_pretrained(base, REPO).merge_and_unload().cuda().eval()

# the prompt MUST match the training format
prompt = (
    "<start_of_turn>user\n"
    "Translate the following English sentence into Tulu.\n"
    "Output only the translated sentence. Do not include explanations.\n\n"
    "The temple is very old.<end_of_turn>\n"
    "<start_of_turn>model\n"
)
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
out = model.generate(**inputs, max_new_tokens=256, do_sample=False)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Citation

Paper under anonymous review. Dataset: randomDude26/mi_kh_tulu_dataset.