| --- |
| license: mit |
| datasets: |
| - roneneldan/TinyStories |
| tags: |
| - tokenizer |
| - bpe |
| --- |
| |
| # Complete Suite of BPE Tokenizers for TinyStories |
|
|
| This repository contains a full suite of custom **BPE tokenizers** trained from scratch on **1,000,000 stories** (about 200M words) from the `roneneldan/TinyStories` dataset. |
|
|
| Optimizing vocabulary size is critical when training Small Language Models (SLMs) on resource-constrained hardware like Mac M1 (8GB RAM). This benchmark provides data-driven evidence for choosing the right vocabulary size. |
|
|
| ## Benchmark Results (Tested on 5,000 Validation Stories) |
|
|
|
|
| | Vocab Size | Total Tokens | Compression Ratio | Avg Tokens/Story | Min Tokens/Story | Max Tokens/Story | Speed (Tokens/sec) | |
| |------------|--------------|-------------------|------------------|------------------|------------------|--------------------| |
| | 1024 | 2,700,793 | 3.07 | 270.1 | 15 | 1399 | ~554,000 | |
| | 1536 | 2,420,572 | 3.42 | 242.1 | 15 | 1288 | ~508,000 | |
| | 2048 | 2,284,150 | 3.63 | 228.4 | 15 | 1243 | ~511,000 | |
| | 2560 | 2,205,438 | 3.76 | 220.5 | 15 | 1224 | ~483,000 | |
| | 3072 | 2,154,156 | 3.85 | 215.4 | 15 | 1135 | ~480,000 | |
| | 4096 | 2,086,877 | 3.97 | 208.7 | 15 | 1117 | ~468,000 | |
| | 8192 | 2,001,284 | 4.14 | 200.1 | 15 | 1079 | ~444,000 | |
| | 16384 | 1,984,990 | 4.17 | 198.5 | 15 | 1067 | ~443,000 | |
| | 50257 | 1,980,834 | 4.18 | 198.1 | 15 | 1067 | ~354,000 | |
|
|
| ## Key Insights |
| 1. **The 1536-2048 Sweet Spot:** Moving from 1024 to 1536/2048 gives the sharpest increase in compression efficiency (**3.07 → 3.63**). For lightweight models, this range offers the best trade-off between sequence compression and embedding matrix size. |
| 2. **The 4096 Diminishing Returns:** A vocabulary size of 4096 captures almost all structure needed for TinyStories (compression ratio **3.97**). |
| 3. **The 50257 Overkill:** The standard GPT-2 vocabulary (50,257) is massive overkill. Increasing vocabulary size by 12x (compared to 4096) saves a mere **10 tokens per story** on average. For TinyStories, larger vocabularies just bloat the model's weights without giving any real context-window benefits. |
|
|
| ## How to use |
|
|
| Since all files are stored in the root directory, you can load any specific tokenizer directly by specifying the file name: |
|
|
| ```python |
| from tokenizers import Tokenizer |
| |
| # Load the optimal 2048 tokenizer |
| tokenizer = Tokenizer.from_pretrained( |
| "morginalium/tinystories-tokenizers", |
| filename="tokenizer_2048.json" |
| ) |
| |
| encoded = tokenizer.encode("Once upon a time, there was a little pup.") |
| print(encoded.ids) |
| ``` |
|
|
