randomDude26/nllb_mktg_sft

NLLB-200-1.3B — joint SFT for Mizo, Tulu, Gondi

NLLB-200-1.3B supervised fine-tuned jointly on four low-resource Indian language pairs. Three new language tokens were added to the tokenizer, since base NLLB supports none of Tulu or Gondi.

A LoRA adapter — load it on top of the base model facebook/nllb-200-1.3B.

Training data

Fine-tuned jointly on four low-resource pairs from randomDude26/mi_kh_tulu_dataset:

Pair	Train	Val	Test
en-mizo (English → Mizo, lus)	49,496	464	800
en-tulu (English → Tulu, tcy)	16,218	488	800
hi-gondi (Hindi → Gondi, gon)	15,500	389	737

The validation and test splits are decontaminated: no exact or near duplicate of any test/val sentence appears in training (exact pair, exact source, exact target, ≥0.90 character similarity, ≥0.80 4-gram containment, ≥0.80 token Jaccard). All scores below are on those decontaminated test sets.

Gondi has 737 test sentences rather than 800 — after decontamination only 737 clean sentences remain.

Results

Decontaminated test sets. BLEU and chrF++ via sacrebleu (BLEU(effective_order=True), CHRF(word_order=2)).

Direction	n	BLEU	chrF++
eng→mizo	800	33.43	59.26
eng→tulu	800	43.17	69.08
hi→gondi	737	7.83	33.53

Checkpoint used for these scores: nllb-sft-joint-upsample-20260525_084543/final_adapter

Against the zero/few-shot baselines (BLEU)

Direction	NLLB 0-shot	this model
eng→mizo	20.40	33.43
eng→khasi	4.50	5.76
eng→tulu	8.51	43.17
hi→gondi	2.22	7.83

Configuration

LoRA

rank r	64
lora_alpha	128
lora_dropout	0.1
target modules	q_proj, k_proj, v_proj, out_proj, fc1, fc2
modules_to_save	embed_tokens, lm_head

The embedding matrix and LM head are trained in full (modules_to_save) because three new language tokens — kha_Latn, tcy_Knda, gon_Deva — were added to the tokenizer. Base NLLB-200 has none of them; only Mizo (lus_Latn) is native.

Training

max steps	10,000
batch size	24 (per device), grad-accum 1
learning rate	3e-4, cosine, 200 warmup steps
weight decay	0.01
max sequence length	256
precision	bf16
decoding	beam search, 4 beams

Trained jointly on all four pairs, with upsampling of the lower-resource pairs.

Usage

import torch
from transformers import AutoTokenizer
from transformers.models.m2m_100 import M2M100ForConditionalGeneration
from peft import PeftModel

REPO = "randomDude26/nllb_mktg_sft"

# the tokenizer in the adapter repo carries the 3 added language tokens
tokenizer = AutoTokenizer.from_pretrained(REPO)

base = M2M100ForConditionalGeneration.from_pretrained(
    "facebook/nllb-200-1.3B", dtype=torch.bfloat16
)
base.resize_token_embeddings(len(tokenizer), mean_resizing=False)

model = PeftModel.from_pretrained(base, REPO).merge_and_unload().cuda().eval()

tokenizer.src_lang = "eng_Latn"
inputs = tokenizer("The temple is very old.", return_tensors="pt").to("cuda")

out = model.generate(
    **inputs,
    forced_bos_token_id=tokenizer.convert_tokens_to_ids("tcy_Knda"),  # Tulu
    num_beams=4, max_new_tokens=256,
)
print(tokenizer.batch_decode(out, skip_special_tokens=True)[0])

Citation

Paper under anonymous review. Dataset: randomDude26/mi_kh_tulu_dataset.