Hugging Face | GitHub | LinkedIn | WhatsApp | Portfolio

Author: Md. Emon Hasan

bangla-gemma-pro

Perplexity on held-out Bangla falls from 24.30 to 12.70, a 48% reduction. This is the headline number: it is the only measure reported here without a chance floor, so the gain reflects a real improvement in the model's language modeling of Bangla rather than movement within a bounded accuracy range.

A complete fine-tune of google/gemma-3-1b-it for Bangla — all 999,885,952 parameters, trained on a free Kaggle T4.

Not LoRA. Not an adapter. Every weight in the model was updated, on 32,037 natively-written Bangla instruction pairs from Bangla-Instruct, inside a 15 GB consumer GPU with 14.90 GB peak usage — 99.3% of what the card has.

This is v1 of an ongoing effort to build a small, open, genuinely Bangla-first model that anyone can run. What follows is every number from that run, including the ones that did not go the way I wanted.

What this run established

Gemma-3's tokenizer handles Bangla unusually well. Measured, not assumed:

	tokens per word
Bangla	1.42
English	1.30

Bangla costs only 1.1× what English does. Most multilingual models fragment Bangla far worse than this, and the penalty compounds: every extra token is context you cannot use and latency you pay for. Gemma-3's 262k vocabulary is a real asset for Bangla work, and that finding transfers to anyone choosing a base model for this language.

A 1B full fine-tune fits on free hardware. adamw_8bit keeps the optimizer state near 2 GB where plain AdamW would need 8 GB, and gradient checkpointing absorbs the activations. The run finished in 150 minutes at a peak of 14.90 GB. Nobody needs an A100 to do this.

The model learned structure. Asked to explain why water boils, the base model answers in loose prose; this model answers with headings and numbered steps, the register its training data uses. That is the fine-tune doing exactly what supervised instruction tuning does.

Results

Perplexity on 200 held-out Bangla responses the model never saw during training — the clearest signal available, since it has no chance floor:

	perplexity
gemma-3-1b-it	24.30
bangla-gemma-pro	12.70

The fine-tune fits held-out Bangla better than its base.

Five Bangla multiple-choice benchmarks were planned; 3 loaded. Scoring is length-normalised log-likelihood — each option's average log probability, highest wins. No generation, no judge, no string matching.

Benchmark	Random chance	gemma-3-1b-it	bangla-gemma-pro	Δ
bangla-mmlu	0.25	0.317	0.283	-0.033
piqa-bn	0.50	0.510	0.563	+0.053
boolq-bn	0.50	0.547	0.523	-0.023
average	—	0.458	0.457	-0.001

The chance column is there because it changes how the table reads. At 0.8B–1B scale, MMLU-style benchmarks barely separate models: both this model and its base sit close to guessing on several of them. They are reported because they were run, not because they settle anything. The perplexity figure and the samples below carry more information.

Sample generations

Prompt	gemma-3-1b-it	bangla-gemma-pro
বাংলাদেশের মুক্তিযুদ্ধ সম্পর্কে তিনটি বাক্যে লিখুন।	অবশ্যই! বাংলাদেশের মুক্তিযুদ্ধের ইতিহাস নিচে তিনটি বাক্যে তুলে ধরা হলো: বাংলাদেশের মুক্তিযুদ্ধ	বাংলাদেশের মুক্তিযুদ্ধ একটি গুরুত্বপূর্ণ ঐতিহাসিক ঘটনা, যা ১৯৫২ সালের ২৩শে আগস্ট থেকে শুরু হয়।
রবীন্দ্রনাথ ঠাকুর কে ছিলেন?	রবীন্দ্রনাথ ঠাকুর ছিলেন একজন বিখ্যাত বাঙালি কবি, ঔপন্যাসিক, নাট্যকার, দার্শনিক, সংগীতজ্ঞ, এবং স	রবীন্দ্রনাথ ঠাকুর ছিলেন একটি বিখ্যাত বাংলা কবি, ঔপন্যাসিক, নাট্যকার, দার্শনিক, চিন্তাবিদ এবং সম
একটি ছোট গল্প লিখুন যেখানে একজন কৃষক বৃষ্টির জন্য অপেক্	বৃষ্টির জন্য অপেক্ষা করা একজন কৃষকের গল্প: সকালটা শুরু হয়েছিল পাখির কলকাকলি আর ধুলোর আঘাতেই। রফ	বৃষ্টির জন্য অপেক্ষা করা একটি কৃষকের গল্প: সকাল শুরু হয় এক অস্পষ্ট আলোতে। রোদ-রোদ, রোদ-রোদ, রো
পানি ফুটলে কী ঘটে, সহজ ভাষায় ব্যাখ্যা করুন।	পানি ফুটলে যা ঘটে, তা হলো: ১. তাপ দেওয়া: যখন পানি ফুটিয়ে নিলে, সেটির তাপমাত্রা বেড়ে যায়। এই তাপ	পানি ফুটলে কিছু বিশেষ প্রক্রিয়া ঘটে। নিচে ধাপে ধাপে এটি ব্যাখ্যা করা হলো: ### ১. তাপের প্রভাব

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
 
tok = AutoTokenizer.from_pretrained("emonHasan01/bangla-gemma-pro")
model = AutoModelForCausalLM.from_pretrained("emonHasan01/bangla-gemma-pro", dtype=torch.float32,
                                             device_map="auto")
 
messages = [
    {"role": "system", "content": "আপনি একজন সহায়ক সহকারী। সবসময় স্পষ্ট ও সঠিক বাংলায় উত্তর দিন।"},
    {"role": "user", "content": "একটি ছোট গল্প লিখুন যেখানে একজন কৃষক বৃষ্টির জন্য অপেক্ষা করছেন।"},
]
prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tok(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)
out = model.generate(**inputs, max_new_tokens=256, do_sample=True,
                     temperature=0.7, top_p=0.9, repetition_penalty=1.1)
print(tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Keep repetition_penalty=1.1 — greedy decoding occasionally loops on this model. The Bangla system prompt was in every training example and output degrades without it.

llama.cpp

File	Size
bangla-gemma-pro.Q4_K_M.gguf	1.01 GB
bangla-gemma-pro.Q8_0.gguf	1.39 GB

llama-cli -hf emonHasan01/bangla-gemma-pro-GGUF:Q4_K_M -p "রবীন্দ্রনাথ ঠাকুর কে ছিলেন?" -n 256

Training

Base model	google/gemma-3-1b-it
Method	full fine-tune, 999,885,952 of 999,885,952 parameters (100%)
Framework	Unsloth
Data	Bangla-Instruct, 32,037 of 342,391 pairs within 512 tokens
Learning rate	1e-05, cosine, 40 warmup steps
Batch	2 × 8 accumulation = 16 effective
Steps	500
Optimizer	adamw_8bit, max_grad_norm=1.0
Precision	fp32 (fp16 unsupported for Gemma-3)
Peak VRAM	14.90 GB of 15.0
Hardware	Tesla T4, free Kaggle tier
Training time	150 minutes
Final loss	1.9392

Decisions behind those numbers

Full fine-tuning instead of LoRA, deliberately. An adapter learns a low-rank correction; teaching a model a language is not low-rank work. At 1B the full tune fits free hardware, so there was no reason to settle for less.

Learning rate 1e-05, an order of magnitude below LoRA's. Every weight moves in a full fine-tune. At LoRA's 2e-4 the multilingual ability the base arrived with would be destroyed in the first few hundred steps.

Natively-written Bangla, not translated Alpaca. Most Bangla fine-tunes train on machine-translated English instruction sets, which carry English sentence structure and translation artefacts into the model. Bangla-Instruct was generated in Bangla.

Loss on the assistant turn only. Prompt tokens are masked to -100 so the gradient is not spent regenerating the question — 67% of each sequence is supervised.

Limitations

Stated plainly, because a card that hides these is worth less than one that does not.

• It gets facts wrong. Dates, numbers and named people are unreliable — it misstated Rabindranath Tagore's birth date in testing. A 1B model has limited factual capacity in any language, and Bangla training data does not change that.
• Occasional repetition under greedy decoding; repetition_penalty=1.1 handles it.
• Grammatical slips, particularly Bangla classifiers (একজন / একটি).
• Multiple-choice benchmarks are near their chance floor at this scale, as the table shows. They do not demonstrate much either way.
• 300 items per benchmark, not full test sets, to fit free GPU hours. Differences under roughly 0.03 are noise.
• English ability after fine-tuning was not measured. Full fine-tuning on one language can degrade others.
• No safety, bias or cultural-sensitivity evaluation, which matters for Bangla content touching religion, politics or region.

Use it for drafting, rephrasing and structure. Verify anything factual.

Roadmap

The gap between this and a genuinely useful Bangla model is known, and none of it is mysterious:

1. Continual pretraining on Bangla text before instruction tuning. This is the single biggest lever. TigerLLM reached its results by pretraining on a 10M-token Bangla corpus first; instruction tuning alone adapts form, not knowledge.
2. Longer sequences. A 512-token window kept 32,037 of 40,000 sampled pairs. At 1024 or 2048 far more of the corpus becomes usable.
3. More steps on more data. 500 steps saw about 8,000 examples out of 342,391 available.
4. A benchmark that discriminates at this scale, plus human evaluation of generation quality, which is what actually matters for a writing assistant.
5. A 4B variant once the recipe is settled, where factual capacity stops being the binding constraint.

Reproducing

One Kaggle notebook on a free T4 produced the baseline, the training, every chart and this card. Raw numbers in results.json.

Citation

@inproceedings{raihan-zampieri-2025-tigerllm,
  title     = {{TigerLLM} - A Family of {B}angla Large Language Models},
  author    = {Raihan, Nishat and Zampieri, Marcos},
  booktitle = {Proceedings of ACL}, year = {2025}
}
@software{unsloth,
  title = {Unsloth}, author = {Daniel Han and Michael Han and Unsloth team},
  year = {2023}, url = {https://github.com/unslothai/unsloth}
}