randomDude26/mbart_mktg_sft

mBART-50 — joint SFT for Mizo, Khasi, Tulu, Gondi

mBART-50 supervised fine-tuned jointly on four low-resource Indian language pairs, with three new language tokens added to the tokenizer.

A LoRA adapter — load it on top of the base model facebook/mbart-large-50.

Training data

Fine-tuned jointly on four low-resource pairs from randomDude26/mi_kh_tulu_dataset:

Pair	Train	Val	Test
en-mizo (English → Mizo, lus)	49,496	464	800
en-khasi (English → Khasi, kha)	22,358	482	800
en-tulu (English → Tulu, tcy)	16,218	488	800
hi-gondi (Hindi → Gondi, gon)	15,500	389	737

The validation and test splits are decontaminated: no exact or near duplicate of any test/val sentence appears in training (exact pair, exact source, exact target, ≥0.90 character similarity, ≥0.80 4-gram containment, ≥0.80 token Jaccard). All scores below are on those decontaminated test sets.

Gondi has 737 test sentences rather than 800 — after decontamination only 737 clean sentences remain.

Results

Decontaminated test sets. BLEU and chrF++ via sacrebleu (BLEU(effective_order=True), CHRF(word_order=2)).

Direction	n	BLEU	chrF++
eng→mizo	800	11.79	31.00
eng→khasi	800	19.59	38.96
eng→tulu	800	26.93	53.95
hi→gondi	737	6.45	31.03

Checkpoint used for these scores: mbart-sft-joint-upsample-20260525_223852/final_adapter

Against the zero/few-shot baselines (BLEU)

Direction	NLLB 0-shot	this model
eng→mizo	20.40	11.79
eng→khasi	4.50	19.59
eng→tulu	8.51	26.93
hi→gondi	2.22	6.45

Configuration

LoRA

rank r	64
lora_alpha	128
lora_dropout	0.1
target modules	q_proj, k_proj, v_proj, out_proj, fc1, fc2
modules_to_save	embed_tokens, lm_head

The embedding matrix and LM head are trained in full (modules_to_save) because three new language tokens — kha_Latn, tcy_Knda, gon_Deva — were added to the tokenizer. None of the four target languages is native to mBART-50.

Training

max steps	10,000
batch size	24 (per device), grad-accum 1
learning rate	3e-4, cosine, 200 warmup steps
weight decay	0.01
max sequence length	256
precision	bf16
decoding	beam search, 4 beams

Trained jointly on all four pairs, with upsampling of the lower-resource pairs.

Usage

import torch
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
from peft import PeftModel

REPO = "randomDude26/mbart_mktg_sft"

tokenizer = AutoTokenizer.from_pretrained(REPO)
base = AutoModelForSeq2SeqLM.from_pretrained(
    "facebook/mbart-large-50", dtype=torch.bfloat16
)
base.resize_token_embeddings(len(tokenizer), mean_resizing=False)

model = PeftModel.from_pretrained(base, REPO).merge_and_unload().cuda().eval()

tokenizer.src_lang = "en_XX"
inputs = tokenizer("The temple is very old.", return_tensors="pt").to("cuda")
out = model.generate(
    **inputs,
    forced_bos_token_id=tokenizer.convert_tokens_to_ids("tcy_Knda"),
    num_beams=4, max_new_tokens=256,
)
print(tokenizer.batch_decode(out, skip_special_tokens=True)[0])

Citation

Paper under anonymous review. Dataset: randomDude26/mi_kh_tulu_dataset.