bert-tiny (mirror)

A 2-layer, 128-hidden BERT — about 4.4M parameters / 17 MB. Small enough to fine-tune on a laptop CPU in minutes, which makes it the go-to model for smoke tests, CI pipelines, unit tests for training code, and edge deployment.

This is a mirror. The weights and tokenizer files here are an unmodified copy of prajjwal1/bert-tiny, re-hosted on this profile for reproducibility and convenience. All credit for the original work belongs to its authors. The upstream license (mit) is preserved and applies to this copy. If you need the canonical version, please use the upstream repository.

Specs

Layers	2
Hidden size	128
Attention heads	2
Parameters	~4.4M
Vocab	30,522 (uncased WordPiece)
Disk	~17 MB

Usage

from transformers import AutoTokenizer, AutoModel

tok = AutoTokenizer.from_pretrained("priyaganesh2050/bert-tiny")
model = AutoModel.from_pretrained("priyaganesh2050/bert-tiny")

out = model(**tok("A tiny BERT for fast experiments.", return_tensors="pt"))
print(out.last_hidden_state.shape)   # torch.Size([1, 9, 128])

Fine-tuning for classification:

from transformers import AutoModelForSequenceClassification

model = AutoModelForSequenceClassification.from_pretrained("priyaganesh2050/bert-tiny", num_labels=2)

When to use this

• Good for: CI/CD tests of training loops, hyperparameter search, teaching, edge/mobile, latency-critical baselines.
• Not good for: accuracy-sensitive production NLP. A 2-layer model gives up a lot of quality versus bert-base. Use it as a baseline, then scale up.

Citation

The tiny BERT variants come from the well-read-students line of work:

@misc{turc2019,
  title  = {Well-Read Students Learn Better: On the Importance of Pre-training Compact Models},
  author = {Turc, Iulia and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina},
  year   = {2019},
  eprint = {1908.08962},
  archivePrefix = {arXiv}
}