File size: 2,926 Bytes
831b40b | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 | # Tokenizer Inspector
A small CLI tool for SLM builders: inspect and compare Hugging Face tokenizers before you commit one to a training run.
## What it tells you
- **Vocab size** and token type (byte-level BPE, SentencePiece Unigram, WordPiece, etc.)
- **Special tokens** — what's added on top of the learned vocab
- **Chars/token** on your test corpus — higher means more compact (fewer tokens to train on)
- **Vocab utilization** — what fraction of the vocab your data actually touches
- **Roundtrip check** — does `decode(encode(text))` give you back the same text?
- **Top tokens** — what the tokenizer is actually producing for your data
- **Side-by-side comparison** of two or more tokenizers on the same corpus
## Install
```bash
pip install tokenizers huggingface_hub
```
## Usage
```bash
# Inspect a single tokenizer by repo ID
python3 tokenizer_inspector.py gpt2
# Compare two tokenizers side by side
python3 tokenizer_inspector.py gpt2 HuggingFaceTB/SmolLM2-135M --compare
# Tokenize your own text
python3 tokenizer_inspector.py gpt2 --text "Hello, world! This is a test."
# Show the first 30 tokens produced
python3 tokenizer_inspector.py gpt2 --show-tokens
# Use a local tokenizer.json file
python3 tokenizer_inspector.py ./my_tokenizer.json
# JSON output (for scripting)
python3 tokenizer_inspector.py gpt2 --json
```
## Example output
```
============================================================
COMPARISON (2 tokenizers)
============================================================
Metric gpt2 HuggingFaceTB/Smol
-------------------- -------------------- --------------------
Vocab size 50,257 49,152
Token type byte-level BPE byte-level BPE
Special tokens 1 17
Chars/token 2.6 2.72
Corpus tokens 116 111
Vocab util 0.13% 0.12%
Longest tok 128 81
Roundtrip ok ok
```
## When to use this
- **Before choosing a tokenizer for training**: compare chars/token on a sample of your actual data, not just English prose
- **Debugging a bad tokenizer**: if roundtrip fails, your tokenizer is lossy
- **Vocab sizing**: if vocab utilization is very low (<1%), your vocab is oversized for your data
- **Special token audit**: see exactly what control tokens are in the vocab
## Notes
- The default corpus is mixed English + Python code. Use `--corpus` or `--text` to test on your own data.
- Chars/token is corpus-dependent. A tokenizer that's great for code may be worse for Turkish, and vice versa.
- This tool reads `tokenizer.json` (the fast-rs format). For SentencePiece `.model` files, it falls back to that format. |