BERT-mini

A BERT-Mini model (4 layers, hidden size 256, 4 attention heads, 11.6 M parameters) pre-trained from scratch on Telugu with Google's original BERT recipe. It is the second member of the Google's Recipes collection, which reproduces the compact "BERT miniatures" of Turc et al. (2019) for Telugu, size by size, without changing the recipe. The smaller sibling is BERT-tiny.

It is intended as a compact Telugu encoder: a baseline for fine-tuning, a step in a scaling or distillation study with the other sizes in the collection, or a lightweight masked language model for scoring and ranking Telugu text.

Model details

Architecture	BertForPreTraining (masked LM head + next-sentence head)
Size	BERT-Mini: 4 layers, hidden 256, 4 heads, feed-forward 1024
Parameters	11,581,440
Max sequence length	512
Vocabulary	32,000 WordPiece tokens, cased, learned on the pre-training text
Precision of the released weights	float32 (model.safetensors, 46.6 MB)
Language	Telugu (te)
Script	Telugu script, NFC-normalised, no lower-casing or accent stripping
Trained by	bammera-pothana
Training date	9 November 2026

The tokenizer is a standard BertTokenizerFast. Lower-casing, accent stripping and CJK handling are off: stripping accents would destroy Telugu vowel signs (మాత్రలు), so the tokenizer must be loaded from this repository rather than configured by hand. The vocabulary was learned afresh for this model and is not identical to the BERT-tiny vocabulary, so masked-LM losses of the two models are only roughly comparable; accuracies compare more fairly.

Training recipe

The recipe follows Devlin et al. (2018), as used for the BERT miniatures of Turc et al. (2019), with no modifications other than the Telugu vocabulary:

Setting	Value
Objectives	Masked LM (15 % of WordPiece tokens: 80 % [MASK], 10 % random token, 10 % unchanged) + next-sentence prediction
Masking	Token-level (Google's original BERT), dynamic, re-sampled every batch
Total steps	1,000,000 at a global batch of 256 sequences, one continuous schedule
Phase 1	900,000 steps at 128 tokens
Phase 2	100,000 steps at 512 tokens, optimizer state carried over from phase 1
Optimizer	AdamW, lr 1e-4, β₁ 0.9, β₂ 0.999, ε 1e-6, weight decay 0.01 (not on bias / LayerNorm)
Schedule	Linear warm-up over the first 10,000 steps, then linear decay to 0 at step 1,000,000
Gradient clipping	1.0
Dropout	0.1 (hidden and attention)
Activation	GELU
Early stopping	Off; the checkpoint with the best held-out MLM loss is released
Mixed precision	float16 autocast with loss scaling; weights kept in float32
Hardware	4 × NVIDIA GeForce RTX 2080 Ti (11 GB)
Wall-clock	14.9 h for phase 1 (0.06 s/step) + 6.8 h for phase 2 (0.24 s/step)

Training examples were built the way Google's create_pretraining_data.py builds them: [CLS] A [SEP] B [SEP], where B is the true continuation half of the time and a segment from a random other document otherwise; 10 % of examples use a shorter random target length. A held-out set of documents was kept aside before the vocabulary was learned and was never used for training.

Evaluation

Held-out pre-training text

Held-out documents, masked with a fixed random seed so that numbers are comparable across checkpoints. MLM accuracy is top-1 accuracy at masked positions.

Checkpoint	MLM loss	MLM perplexity	MLM accuracy	NSP accuracy
End of phase 1 (step 900 k, 128 tokens)	3.475	32.3	41.5 %	94.6 %
Released: end of phase 2 (step 1 M, 512 tokens)	3.418	30.5	42.4 %	95.5 %

Held-out MLM loss had converged by the end of phase 2 (3.432 at step 60 k → 3.418 at step 100 k). The per-phase evaluation files are in the training/ folder of this repository.

Clean typed Telugu outside the pre-training text

Two fixed samples of 3,000 paragraphs each, cut into 128-token pieces, 15 % of tokens masked with a fixed seed. Neither set was used in pre-training. BERT-tiny from the same collection is shown for scale.

Model	Wikisource (literary): loss / accuracy	News (modern): loss / accuracy
BERT-mini (this model)	3.61 / 39.4 %	3.33 / 43.8 %
BERT-tiny (2 layers, 128 hidden)	4.55 / 29.7 %	4.73 / 28.7 %

Doubling the depth and width over BERT-tiny lifts masked-token accuracy by about 10 points on literary text and 15 points on modern news text.

Downstream (fine-tuning) results for this model will be added to this card as they become available.

How to use

from transformers import pipeline

fill = pipeline("fill-mask", model="bammera-pothana/BERT-mini")
fill("నేను [MASK] వెళ్తున్నాను.")
# [{'token_str': 'యింటికి', ...}, {'token_str': 'ఇంటికి', ...}, {'token_str': 'హైదరాబాదు', ...}, ...]

As an encoder for fine-tuning:

from transformers import AutoTokenizer, AutoModel, AutoModelForSequenceClassification

tok = AutoTokenizer.from_pretrained("bammera-pothana/BERT-mini")
enc = AutoModel.from_pretrained("bammera-pothana/BERT-mini")           # hidden states, pooler
clf = AutoModelForSequenceClassification.from_pretrained("bammera-pothana/BERT-mini", num_labels=3)

Loading with AutoModelForMaskedLM discards the pooler and next-sentence head, which is expected and harmless. The full pre-training heads are available through BertForPreTraining.

Sample predictions from the released checkpoint:

Input	Top predictions
నేను [MASK] వెళ్తున్నాను.	యింటికి (0.05), ఇంటికి, హైదరాబాదు, ఆఫీసుకు, కాలేజీకి
ఆంధ్ర [MASK] పత్రిక	##జ్యోతి (0.29, as in ఆంధ్రజ్యోతి), సచిత్ర, రైతు, ##కేసరి, పత్రిక
తెలుగు [MASK] భాష	##భాష (0.11, as in తెలుగుభాష), తెలుగు, భాష, కన్నడ, భాషా

Limitations and bias

• Compact model. With 4 layers and hidden size 256, BERT-Mini is clearly stronger than BERT-tiny but still well below BERT-Base on downstream tasks. Use it where speed or size matters, or as a point on a scaling curve.
• Domain. Like any pre-trained language model, it reflects the register, spelling conventions and viewpoints of the text it was trained on, and may perform worse on text from other domains. The sample predictions above show that it has learned older spellings such as యింటికి alongside modern ones.
• Pre-training only. No instruction tuning, no safety tuning. Outputs are token probabilities, not answers.

Repository contents

File	Purpose
config.json, model.safetensors	Model (architecture BertForPreTraining)
tokenizer.json, tokenizer_config.json, special_tokens_map.json, vocab.txt	WordPiece tokenizer
training/eval_phase1_128.json, training/eval_phase2_512.json	Held-out metrics at the end of each phase

Citation

If you use this model, please cite the original BERT and BERT-miniatures papers together with this repository:

@misc{bammera-pothana-bert-mini-2026,
  title  = {BERT-mini: a Telugu BERT-Mini model pre-trained with Google's original recipe},
  author = {bammera-pothana},
  year   = {2026},
  url    = {https://huggingface.co/bammera-pothana/BERT-mini}
}

@article{turc2019,
  title   = {Well-Read Students Learn Better: On the Importance of Pre-training Compact Models},
  author  = {Turc, Iulia and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina},
  journal = {arXiv preprint arXiv:1908.08962},
  year    = {2019}
}

@inproceedings{devlin2019,
  title     = {{BERT}: Pre-training of Deep Bidirectional Transformers for Language Understanding},
  author    = {Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina},
  booktitle = {NAACL-HLT},
  year      = {2019}
}