File size: 2,347 Bytes
74ca936 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 | ---
license: mit
library_name: transformers
pipeline_tag: fill-mask
base_model: prajjwal1/bert-tiny
language:
- en
tags:
- bert
- tiny
- lightweight
- edge
- cpu
- text-embedding
---
# bert-tiny (mirror)
A 2-layer, 128-hidden BERT — about **4.4M parameters / 17 MB**. Small enough to fine-tune on a
laptop CPU in minutes, which makes it the go-to model for smoke tests, CI pipelines, unit tests
for training code, and edge deployment.
> [!NOTE]
> **This is a mirror.** The weights and tokenizer files here are an unmodified copy of
> [`prajjwal1/bert-tiny`](https://huggingface.co/prajjwal1/bert-tiny), re-hosted on this profile for reproducibility and
> convenience. All credit for the original work belongs to its authors. The upstream license
> (`mit`) is preserved and applies to this copy. If you need the canonical version, please
> use the upstream repository.
## Specs
| | |
|---|---|
| Layers | 2 |
| Hidden size | 128 |
| Attention heads | 2 |
| Parameters | ~4.4M |
| Vocab | 30,522 (uncased WordPiece) |
| Disk | ~17 MB |
## Usage
```python
from transformers import AutoTokenizer, AutoModel
tok = AutoTokenizer.from_pretrained("priyaganesh2050/bert-tiny")
model = AutoModel.from_pretrained("priyaganesh2050/bert-tiny")
out = model(**tok("A tiny BERT for fast experiments.", return_tensors="pt"))
print(out.last_hidden_state.shape) # torch.Size([1, 9, 128])
```
Fine-tuning for classification:
```python
from transformers import AutoModelForSequenceClassification
model = AutoModelForSequenceClassification.from_pretrained("priyaganesh2050/bert-tiny", num_labels=2)
```
## When to use this
- **Good for:** CI/CD tests of training loops, hyperparameter search, teaching, edge/mobile,
latency-critical baselines.
- **Not good for:** accuracy-sensitive production NLP. A 2-layer model gives up a lot of quality
versus `bert-base`. Use it as a baseline, then scale up.
## Citation
The tiny BERT variants come from the well-read-students line of work:
```bibtex
@misc{turc2019,
title = {Well-Read Students Learn Better: On the Importance of Pre-training Compact Models},
author = {Turc, Iulia and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina},
year = {2019},
eprint = {1908.08962},
archivePrefix = {arXiv}
}
```
|