randomDude26/nllb_khasi_sft

NLLB-200-1.3B — per-pair SFT for English → Khasi

NLLB-200-1.3B fine-tuned on English → Khasi alone, rather than jointly with the other three language pairs.

This is the counterpart to randomDude26/nllb_mktg_sft, which is trained jointly on Mizo, Khasi, Tulu and Gondi. For Khasi, the joint adapter collapses and this one does not:

Adapter	eng→khasi BLEU	eng→khasi chrF++
Joint (nllb_mktg_sft)	5.76	24.63
Per-pair (this model)	21.50	42.67

A 3.7× BLEU gain from removing the other three languages. Khasi is Austroasiatic and has no relative among NLLB's supported languages, so it appears to be crowded out during joint training — while Mizo, Tulu and Gondi all benefit from it. This is the main reason the model exists.

A LoRA adapter — load it on top of facebook/nllb-200-1.3B.

Results

Decontaminated eng→khasi test set (n = 800). BLEU / chrF++ via sacrebleu (BLEU(effective_order=True), CHRF(word_order=2)).

System	BLEU	chrF++
NLLB-200 zero-shot*	4.50	21.42
NLLB joint SFT	5.76	24.63
mBART-50 joint SFT	19.59	38.96
NLLB per-pair SFT (this model)	21.50	42.67
Gemma-3-4B CPT+SFT	34.49	58.74

* base NLLB has no kha_Latn token; the zero-shot number forces eng_Latn as a proxy and is not a native translation.

Checkpoint used for these scores: nllb-sft-en-khasi-20260521_100238/final_adapter

Note that a decoder-only model (Gemma) still beats this by a wide margin on Khasi. Per-pair training rescues NLLB from collapse but does not close the gap.

Other directions

This adapter is English→Khasi only. It was trained on nothing else and degrades badly if pushed at other pairs:

Direction	BLEU
eng→khasi	21.50
eng→mizo	4.88
eng→tulu	1.19
hi→gondi	0.44

Use nllb_mktg_sft for Mizo, Tulu or Gondi.

Training data

English–Khasi split of randomDude26/mi_kh_tulu_dataset:

Split	Sentences
train	23,358
validation	482
test	800

Validation and test are decontaminated: no exact or near duplicate of any test/val sentence appears in training (exact pair, exact source, exact target, ≥0.90 character similarity, ≥0.80 4-gram containment, ≥0.80 token Jaccard).

Configuration

LoRA

rank r	64
lora_alpha	128
lora_dropout	0.1
target modules	q_proj, k_proj, v_proj, out_proj, fc1, fc2

A kha_Latn language token was added to the tokenizer — base NLLB-200 has no Khasi token. The trained embedding for it ships inside the adapter's lm_head (NLLB ties input embeddings and the LM head), so the adapter is self-contained.

Training

max steps	10,000
batch size	24 (per device), grad-accum 1
learning rate	3e-4, cosine, 200 warmup steps
weight decay	0.01
max sequence length	256
precision	bf16
decoding	beam search, 4 beams

Usage

import torch
from transformers import AutoTokenizer
from transformers.models.m2m_100 import M2M100ForConditionalGeneration
from peft import PeftModel

REPO = "randomDude26/nllb_khasi_sft"

# the tokenizer in this repo carries the added kha_Latn token
tokenizer = AutoTokenizer.from_pretrained(REPO)

base = M2M100ForConditionalGeneration.from_pretrained(
    "facebook/nllb-200-1.3B", dtype=torch.bfloat16
)
base.resize_token_embeddings(len(tokenizer), mean_resizing=False)

model = PeftModel.from_pretrained(base, REPO).merge_and_unload().cuda().eval()

tokenizer.src_lang = "eng_Latn"
inputs = tokenizer("The temple is very old.", return_tensors="pt").to("cuda")

out = model.generate(
    **inputs,
    forced_bos_token_id=tokenizer.convert_tokens_to_ids("kha_Latn"),
    num_beams=4, max_new_tokens=256,
)
print(tokenizer.batch_decode(out, skip_special_tokens=True)[0])

Citation

Paper under anonymous review. Dataset: randomDude26/mi_kh_tulu_dataset.