Update README.md

README.md

@@ -1,3 +1,62 @@
 ---
 license: mit
+datasets:
+- roneneldan/TinyStories
+tags:
+- tokenizer
+- bpe
 ---
+
+---
+license: mit
+datasets:
+- roneneldan/TinyStories
+language:
+- en
+tags:
+- tokenizer
+- bpe
+---
+
+# Complete Suite of BPE Tokenizers for TinyStories
+
+This repository contains a full suite of custom **BPE tokenizers** trained from scratch on **1,000,000 stories** (about 200M words) from the `roneneldan/TinyStories` dataset. 
+
+Optimizing vocabulary size is critical when training Small Language Models (SLMs) on resource-constrained hardware like Mac M1 (8GB RAM). This benchmark provides data-driven evidence for choosing the right vocabulary size.
+
+## Benchmark Results (Tested on 5,000 Validation Stories)
+
+
+| Vocab Size | Total Tokens | Compression Ratio | Avg Tokens/Story | Min Tokens/Story | Max Tokens/Story | Speed (Tokens/sec) |
+|------------|--------------|-------------------|------------------|------------------|------------------|--------------------|
+| 1024       | 2,700,793    | 3.07              | 270.1            | 15               | 1399             | ~554,000           |
+| 1536       | 2,420,572    | 3.42              | 242.1            | 15               | 1288             | ~508,000           |
+| 2048       | 2,284,150    | 3.63              | 228.4            | 15               | 1243             | ~511,000           |
+| 2560       | 2,205,438    | 3.76              | 220.5            | 15               | 1224             | ~483,000           |
+| 3072       | 2,154,156    | 3.85              | 215.4            | 15               | 1135             | ~480,000           |
+| 4096       | 2,086,877    | 3.97              | 208.7            | 15               | 1117             | ~468,000           |
+| 8192       | 2,001,284    | 4.14              | 200.1            | 15               | 1079             | ~444,000           |
+| 16384      | 1,984,990    | 4.17              | 198.5            | 15               | 1067             | ~443,000           |
+| 50257      | 1,980,834    | 4.18              | 198.1            | 15               | 1067             | ~354,000           |
+
+## Key Insights
+1. **The 1536-2048 Sweet Spot:** Moving from 1024 to 1536/2048 gives the sharpest increase in compression efficiency (**3.07 → 3.63**). For lightweight models, this range offers the best trade-off between sequence compression and embedding matrix size.
+2. **The 4096 Diminishing Returns:** A vocabulary size of 4096 captures almost all structure needed for TinyStories (compression ratio **3.97**).
+3. **The 50257 Overkill:** The standard GPT-2 vocabulary (50,257) is massive overkill. Increasing vocabulary size by 12x (compared to 4096) saves a mere **10 tokens per story** on average. For TinyStories, larger vocabularies just bloat the model's weights without giving any real context-window benefits.
+
+## How to use
+
+Since all files are stored in the root directory, you can load any specific tokenizer directly by specifying the file name:
+
+```python
+from tokenizers import Tokenizer
+
+# Load the optimal 2048 tokenizer
+tokenizer = Tokenizer.from_pretrained(
+    "morginalium/tinystories-tokenizers", 
+    filename="tokenizer_2048.json"
+)
+
+encoded = tokenizer.encode("Once upon a time, there was a little pup.")
+print(encoded.ids)
+```