BERT-tiny

A BERT-Tiny model (2 layers, hidden size 128, 2 attention heads, 4.6 M parameters) pre-trained from scratch on Telugu with Google's original BERT recipe. It is the smallest member of the Google's Recipes collection, which reproduces the compact "BERT miniatures" of Turc et al. (2019) for Telugu, size by size, without changing the recipe.

It is intended as a fast, small Telugu encoder: a baseline for fine-tuning, the small end of a scaling or distillation study with the larger sizes in the collection, or a lightweight masked language model for scoring and ranking Telugu text.

Model details

Architecture	BertForPreTraining (masked LM head + next-sentence head)
Size	BERT-Tiny: 2 layers, hidden 128, 2 heads, feed-forward 512
Parameters	4,607,360 (embeddings are the majority: 32,000 × 128)
Max sequence length	512
Vocabulary	32,000 WordPiece tokens, cased, learned on the pre-training text
Precision of the released weights	float32 (model.safetensors, 18.5 MB)
Language	Telugu (te)
Script	Telugu script, NFC-normalised, no lower-casing or accent stripping
Trained by	bammera-pothana
Training dates	3 – 4 November 2026

The tokenizer is a standard BertTokenizerFast. Lower-casing, accent stripping and CJK handling are off: stripping accents would destroy Telugu vowel signs (మాత్రలు), so the tokenizer must be loaded from this repository rather than configured by hand.

Training recipe

The recipe follows Devlin et al. (2018), as used for the BERT miniatures of Turc et al. (2019), with no modifications other than the Telugu vocabulary:

Setting	Value
Objectives	Masked LM (15 % of WordPiece tokens: 80 % [MASK], 10 % random token, 10 % unchanged) + next-sentence prediction
Masking	Token-level (Google's original BERT), dynamic, re-sampled every batch
Total steps	1,000,000 at a global batch of 256 sequences, one continuous schedule
Phase 1	900,000 steps at 128 tokens
Phase 2	100,000 steps at 512 tokens, optimizer state carried over from phase 1
Optimizer	AdamW, lr 1e-4, β₁ 0.9, β₂ 0.999, ε 1e-6, weight decay 0.01 (not on bias / LayerNorm)
Schedule	Linear warm-up over the first 10,000 steps, then linear decay to 0 at step 1,000,000
Gradient clipping	1.0
Dropout	0.1 (hidden and attention)
Activation	GELU
Early stopping	Off; the checkpoint with the best held-out MLM loss is released
Mixed precision	float16 autocast with loss scaling; weights kept in float32
Hardware	4 × NVIDIA GeForce RTX 2080 Ti (11 GB)
Wall-clock	about 10 h (phase 1) + 4 h (phase 2)

Training examples were built the way Google's create_pretraining_data.py builds them: [CLS] A [SEP] B [SEP], where B is the true continuation half of the time and a segment from a random other document otherwise; 10 % of examples use a shorter random target length. A held-out set of documents was kept aside before the vocabulary was learned and was never used for training.

Evaluation

Held-out documents, masked with a fixed random seed so that numbers are comparable across checkpoints. MLM accuracy is top-1 accuracy at masked positions.

Checkpoint	MLM loss	MLM perplexity	MLM accuracy	NSP accuracy
End of phase 1 (step 900 k, 128 tokens)	4.465	87.0	30.9 %	88.6 %
Released: end of phase 2 (step 1 M, 512 tokens)	4.535	93.2	30.2 %	91.8 %

The slightly higher MLM loss after phase 2 is expected: the 512-token held-out set is a different, harder sample (longer segments, more random-pair examples), while NSP improves with the longer context. Held-out MLM loss was still decreasing slowly at the end of phase 1 and had converged in phase 2 (4.536 at step 60 k → 4.535 at step 100 k). The per-phase evaluation files are in the training/ folder of this repository.

Downstream (fine-tuning) results for this model will be added to this card as they become available.

How to use

from transformers import pipeline

fill = pipeline("fill-mask", model="bammera-pothana/BERT-tiny")
fill("ఆంధ్ర [MASK] పత్రిక")
# [{'token_str': 'సచిత్ర', 'score': 0.48}, {'token_str': 'ఆంధ్ర', ...}, ...]

As an encoder for fine-tuning:

from transformers import AutoTokenizer, AutoModel, AutoModelForSequenceClassification

tok = AutoTokenizer.from_pretrained("bammera-pothana/BERT-tiny")
enc = AutoModel.from_pretrained("bammera-pothana/BERT-tiny")           # hidden states, pooler
clf = AutoModelForSequenceClassification.from_pretrained("bammera-pothana/BERT-tiny", num_labels=3)

Loading with AutoModelForMaskedLM discards the pooler and next-sentence head, which is expected and harmless. The full pre-training heads are available through BertForPreTraining.

Sample predictions from the released checkpoint:

Input	Top predictions
నేను [MASK] వెళ్తున్నాను.	ఇంటికి (0.13), నేను, యింటికి, తిరిగి, మద్రాసు
ఆంధ్ర [MASK] పత్రిక	సచిత్ర (0.48), ఆంధ్ర, పత్రిక, వార, వారపత్రిక
తెలుగు [MASK] భాష	తెలుగు (0.33), భాష, కన్నడ, భాషా, హిందీ

Limitations and bias

• Small capacity. With 2 layers and hidden size 128, BERT-Tiny is far weaker than BERT-Base on every downstream task. Use it where speed or size matters, or as the small end of a scaling study, not as a strong general-purpose encoder.
• Domain. Like any pre-trained language model, it reflects the register, spelling conventions and viewpoints of the text it was trained on, and may perform worse on text from other domains.
• Pre-training only. No instruction tuning, no safety tuning. Outputs are token probabilities, not answers.

Repository contents

File	Purpose
config.json, model.safetensors	Model (architecture BertForPreTraining)
tokenizer.json, tokenizer_config.json, special_tokens_map.json, vocab.txt	WordPiece tokenizer
training/eval_phase1_128.json, training/eval_phase2_512.json	Held-out metrics at the end of each phase

Citation

If you use this model, please cite the original BERT and BERT-miniatures papers together with this repository:

@misc{bammera-pothana-bert-tiny-2026,
  title  = {BERT-tiny: a Telugu BERT-Tiny model pre-trained with Google's original recipe},
  author = {bammera-pothana},
  year   = {2026},
  url    = {https://huggingface.co/bammera-pothana/BERT-tiny}
}

@article{turc2019,
  title   = {Well-Read Students Learn Better: On the Importance of Pre-training Compact Models},
  author  = {Turc, Iulia and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina},
  journal = {arXiv preprint arXiv:1908.08962},
  year    = {2019}
}

@inproceedings{devlin2019,
  title     = {{BERT}: Pre-training of Deep Bidirectional Transformers for Language Understanding},
  author    = {Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina},
  booktitle = {NAACL-HLT},
  year      = {2019}
}