---
license: mit
library_name: transformers
pipeline_tag: fill-mask
base_model: prajjwal1/bert-tiny
language:
- en
tags:
- bert
- tiny
- lightweight
- edge
- cpu
- text-embedding
---

# bert-tiny (mirror)

A 2-layer, 128-hidden BERT — about **4.4M parameters / 17 MB**. Small enough to fine-tune on a
laptop CPU in minutes, which makes it the go-to model for smoke tests, CI pipelines, unit tests
for training code, and edge deployment.

> [!NOTE]
> **This is a mirror.** The weights and tokenizer files here are an unmodified copy of
> [`prajjwal1/bert-tiny`](https://huggingface.co/prajjwal1/bert-tiny), re-hosted on this profile for reproducibility and
> convenience. All credit for the original work belongs to its authors. The upstream license
> (`mit`) is preserved and applies to this copy. If you need the canonical version, please
> use the upstream repository.

## Specs

| | |
|---|---|
| Layers | 2 |
| Hidden size | 128 |
| Attention heads | 2 |
| Parameters | ~4.4M |
| Vocab | 30,522 (uncased WordPiece) |
| Disk | ~17 MB |

## Usage

```python
from transformers import AutoTokenizer, AutoModel

tok = AutoTokenizer.from_pretrained("priyaganesh2050/bert-tiny")
model = AutoModel.from_pretrained("priyaganesh2050/bert-tiny")

out = model(**tok("A tiny BERT for fast experiments.", return_tensors="pt"))
print(out.last_hidden_state.shape)   # torch.Size([1, 9, 128])
```

Fine-tuning for classification:

```python
from transformers import AutoModelForSequenceClassification

model = AutoModelForSequenceClassification.from_pretrained("priyaganesh2050/bert-tiny", num_labels=2)
```

## When to use this

- **Good for:** CI/CD tests of training loops, hyperparameter search, teaching, edge/mobile,
  latency-critical baselines.
- **Not good for:** accuracy-sensitive production NLP. A 2-layer model gives up a lot of quality
  versus `bert-base`. Use it as a baseline, then scale up.

## Citation

The tiny BERT variants come from the well-read-students line of work:

```bibtex
@misc{turc2019,
  title  = {Well-Read Students Learn Better: On the Importance of Pre-training Compact Models},
  author = {Turc, Iulia and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina},
  year   = {2019},
  eprint = {1908.08962},
  archivePrefix = {arXiv}
}
```