gemma4-e2b-vaani-t2t

A full fine-tune of google/gemma-4-E2B-it for translation between English and the 22 scheduled languages of India — 44 directions in a single model.

Scores 50.86 chrF++ averaged over the 44 directions of IN22-Gen, compared to Gemma4-E2B-IT's score of 24.96.

Layout

infer.py             inference (source language, target language, text -- that's it)
prompt_contract.py   the exact prompt and trained language set, no dependencies
README.md
config.json, model.safetensors, tokenizer.json, ...   weights, config and tokenizer

Quick start

Same transformers usage as the base model (google/gemma-4-E2B-it) — just this repo's id, plus the one fixed prompt below.

from transformers import AutoProcessor, AutoModelForMultimodalLM

MODEL_ID = "ARTPARK-IISc/gemma4-e2b-vaani-t2t"

processor = AutoProcessor.from_pretrained(MODEL_ID)
model = AutoModelForMultimodalLM.from_pretrained(
    MODEL_ID,
    dtype="auto",
    device_map="auto",
    attn_implementation="eager",  # matches training -- see below
)

T2T — translate text between English and any of the 22 languages:

messages = [{"role": "user", "content": [
    {"type": "text", "text": "Translate the following text from Hindi to English.\n\nनमस्ते, आप कैसे हैं?"},
]}]

inputs = processor.apply_chat_template(
    [messages], tokenize=True, return_dict=True, return_tensors="pt",
    add_generation_prompt=True,
).to(model.device)
input_len = inputs["input_ids"].shape[-1]

outputs = model.generate(**inputs, max_new_tokens=384, num_beams=5)
response = processor.tokenizer.decode(outputs[0][input_len:], skip_special_tokens=True)
print(response)

A ready-made infer.py/prompt_contract.py pair is also included, wrapping the above into an Infer class (load once, call .t2t() repeatedly) plus a CLI:

from infer import Infer
m = Infer()
m.t2t("The weather is nice today.", "English", "Hindi")

python infer.py --src-lang English --tgt-lang Hindi --text "The weather is nice today."

Languages may be given as hi, hin_Deva or Hindi. English must be on one side — Indic-to-Indic was never trained. Needs ~11 GB of VRAM in bf16.

The prompt is part of the model

The instruction this checkpoint was trained on, verbatim:

Translate the following text from {source language} to {target language}.

{text}

Language names are the English spellings in prompt_contract.py (Hindi, Santali, Konkani, …) — two newlines between instruction and text, and the period after the target language is part of the string.

No system turn, no few-shot examples, no task prefix.

prompt_contract.py holds all of this in importable form — T2T_PROMPT, LANG_NAMES, build_prompt_t2t(), and the fully rendered strings as RENDERED_INFERENCE / RENDERED_TRAINING:

from prompt_contract import build_prompt_t2t

build_prompt_t2t("en", "hi", "The weather is nice today.")
# 'Translate the following text from English to Hindi.\n\nThe weather is nice today.'

Run python prompt_contract.py to print the contract with no model loaded.

Supported languages

Assamese as, Bengali bn, Bodo brx, Dogri doi, Gujarati gu, Hindi hi, Kannada kn, Kashmiri kas (Arabic script), Konkani gom, Maithili mai, Malayalam ml, Manipuri mni (Meitei script), Marathi mr, Nepali ne, Odia or, Punjabi pa, Sanskrit sa, Santali sat (Ol Chiki), Sindhi sd (Devanagari), Tamil ta, Telugu te, Urdu ur — each paired with English en.

Both FLORES codes (hin_Deva) and short codes (hi) are accepted.

Results

chrF++ on IN22-Gen (1024 sentences per direction), scored with the IndicEvaluator harness — IndicNLP normalisation plus trivial_tokenize on both hypotheses and references, then sacrebleu CHRF with word_order=2. Beam 5.

	en→X (22)	X→en (22)	all 44
google/gemma-4-E2B-it (untuned)	18.35	31.56	24.96
gemma4-e2b-vaani-t2t	47.54	54.18	50.86
gain	+29.19	+22.62	+25.90

Four of the 22 target languages — Dogri, Bodo, Manipuri, Santali — score close to zero before fine-tuning, meaning the base model cannot write them. Those are the languages this model exists for.

One caveat, in the model's own terms. X→en gains less than en→X (+22.62 against +29.19) despite starting from a much stronger base (31.56 against 18.35). The failure mode is paraphrase rather than translation: the English output is fluent and accurate but restructures the sentence, which chrF++ penalises heavily. It is not a formatting or prompt problem — Dogri, on identical data, gains far more on en→doi than on doi→en, and no prompt variant improved it.

Per language, both directions

language	code	en→X base	en→X	X→en base	X→en
Assamese	asm_Beng	21.71	45.5	38.11	56.5
Bengali	ben_Beng	24.81	44.0	34.48	58.4
Bodo	brx_Deva	1.25	51.1	19.29	50.3
Dogri	doi_Deva	3.09	57.8	29.25	54.2
Gujarati	guj_Gujr	27.44	49.1	37.84	60.6
Hindi	hin_Deva	30.33	54.1	35.08	61.1
Kannada	kan_Knda	26.56	48.0	38.16	57.9
Kashmiri	kas_Arab	6.55	41.4	25.36	48.8
Konkani	gom_Deva	12.96	45.7	26.08	45.4
Maithili	mai_Deva	15.12	49.3	32.85	54.1
Malayalam	mal_Mlym	26.67	47.4	35.03	55.4
Manipuri	mni_Mtei	0.88	48.0	13.55	45.1
Marathi	mar_Deva	26.50	46.8	34.98	58.3
Nepali	npi_Deva	25.25	48.4	35.41	59.8
Odia	ory_Orya	24.43	45.7	42.02	57.0
Punjabi	pan_Guru	25.55	45.9	36.44	57.9
Sanskrit	san_Deva	14.75	40.7	25.14	45.2
Santali	sat_Olck	0.57	38.7	22.29	42.3
Sindhi	snd_Deva	1.11	38.4	25.82	48.9
Tamil	tam_Taml	28.97	47.5	33.48	54.2
Telugu	tel_Telu	27.86	48.8	36.22	58.3
Urdu	urd_Arab	31.39	63.6	37.40	62.2
mean (22)		18.35	47.54	31.56	54.18

Limitations

• English-pivot only. Every training pair had English on one side. Indic↔Indic is untrained; translate.py rejects it rather than producing something plausible and wrong.
• Sentence-level. Trained on sentence pairs at 512 tokens. Longer passages should be split; document-level coherence was never trained.
• Script-specific. Kashmiri in Arabic script, Manipuri in Meitei, Sindhi in Devanagari. The alternative scripts (kas_Deva, mni_Beng, snd_Arab) were excluded from training and are not supported.
• X→en paraphrases, as described above.
• Unused modalities. The checkpoint keeps Gemma-4's vision and audio towers (0.48B of 5.10B parameters), frozen throughout training and untouched by fine-tuning. They are dead weight for translation but are retained so the checkpoint loads with the stock Gemma4ForConditionalGeneration class.

License

Derived from google/gemma-4-E2B-it and therefore governed by the Gemma Terms of Use; the prohibited-use policy applies to this model as it does to the base. Training data is BPCC from AI4Bharat — check its own licence terms for your use case.

Acknowledgements

Trained with compute awarded by the Gemma Academic Program, supported by google.org. We thank Google for the award.