| ---
|
| license: mit
|
| library_name: transformers
|
| pipeline_tag: fill-mask
|
| base_model: prajjwal1/bert-tiny
|
| language:
|
| - en
|
| tags:
|
| - bert
|
| - tiny
|
| - lightweight
|
| - edge
|
| - cpu
|
| - text-embedding
|
| ---
|
|
|
| # bert-tiny (mirror)
|
|
|
| A 2-layer, 128-hidden BERT — about **4.4M parameters / 17 MB**. Small enough to fine-tune on a
|
| laptop CPU in minutes, which makes it the go-to model for smoke tests, CI pipelines, unit tests
|
| for training code, and edge deployment.
|
|
|
| > [!NOTE]
|
| > **This is a mirror.** The weights and tokenizer files here are an unmodified copy of
|
| > [`prajjwal1/bert-tiny`](https://huggingface.co/prajjwal1/bert-tiny), re-hosted on this profile for reproducibility and
|
| > convenience. All credit for the original work belongs to its authors. The upstream license
|
| > (`mit`) is preserved and applies to this copy. If you need the canonical version, please
|
| > use the upstream repository.
|
|
|
| ## Specs
|
|
|
| | | |
|
| |---|---|
|
| | Layers | 2 |
|
| | Hidden size | 128 |
|
| | Attention heads | 2 |
|
| | Parameters | ~4.4M |
|
| | Vocab | 30,522 (uncased WordPiece) |
|
| | Disk | ~17 MB |
|
|
|
| ## Usage
|
|
|
| ```python
|
| from transformers import AutoTokenizer, AutoModel
|
|
|
| tok = AutoTokenizer.from_pretrained("priyaganesh2050/bert-tiny")
|
| model = AutoModel.from_pretrained("priyaganesh2050/bert-tiny")
|
|
|
| out = model(**tok("A tiny BERT for fast experiments.", return_tensors="pt"))
|
| print(out.last_hidden_state.shape) # torch.Size([1, 9, 128])
|
| ```
|
|
|
| Fine-tuning for classification:
|
|
|
| ```python
|
| from transformers import AutoModelForSequenceClassification
|
|
|
| model = AutoModelForSequenceClassification.from_pretrained("priyaganesh2050/bert-tiny", num_labels=2)
|
| ```
|
|
|
| ## When to use this
|
|
|
| - **Good for:** CI/CD tests of training loops, hyperparameter search, teaching, edge/mobile,
|
| latency-critical baselines.
|
| - **Not good for:** accuracy-sensitive production NLP. A 2-layer model gives up a lot of quality
|
| versus `bert-base`. Use it as a baseline, then scale up.
|
|
|
| ## Citation
|
|
|
| The tiny BERT variants come from the well-read-students line of work:
|
|
|
| ```bibtex
|
| @misc{turc2019,
|
| title = {Well-Read Students Learn Better: On the Importance of Pre-training Compact Models},
|
| author = {Turc, Iulia and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina},
|
| year = {2019},
|
| eprint = {1908.08962},
|
| archivePrefix = {arXiv}
|
| }
|
| ```
|
|
|
