-
-
Loading tokenizer...
- This may take a moment for large vocabularies
+
+
+
-
+
+
+
+
+ ✎
+ Input Text
+
+
+
+
+
+
+
+ 0 characters
- ⚠️
- Error message
-
+
+
-
-
-
-
-
-
-
- 🔤
- TokenViz
-- Universal tokenizer visualization for any HuggingFace model. - See exactly how LLMs break down text into tokens, IDs, and bytes — all in your browser with zero GPU required. -
-
-
- 🤖 Select Model
-
-
-
-
-
-
-
-
+
+
+
+
-
-
- Tokens
+ —
+ no model loaded
-
-
- 📊
- Vocabulary Size
- -
-
-
- 🔢
- Token Count
- -
-
-
- 📏
- Char / Token Ratio
- -
-
-
+ ⚡
- Model Type
- -
-
+
-
-
- Characters
+ —
+ total input
-
+
+
+
-
-
-
- ✏️ Input Text
-
-
-
-
- Chars:
- 0
-
-
- Words:
- 0
-
-
-
-
-
-
-
-
-
-
+
+
-
-
- Words
+ —
+ approx
-
+
+
+
+
+
-
-
-
-
-
-
-
-
+
-
- 🔍
- Enter text above to visualize tokenization
- Select a model and start typing to see the magic happen
-
+
-
-
- Chars / Token
+ —
+ efficiency
- 💡 How it works: This app uses
+
+
+ @huggingface/transformers (v3.5.0) to load tokenizer files directly from the HuggingFace Hub in your browser.
- It downloads tokenizer.json and tokenizer_config.json and runs tokenization entirely client-side with WebAssembly — no GPU or server required.
- Works with BPE, WordPiece, Unigram, and SentencePiece tokenizers from any model.
+
+
-
-
+
+
+
+
+
-
+
+
+
+
+ ⬡
+ Select a model above and type something
to see tokenization in action
+
+
+
+
+
+
+
+
\ No newline at end of file
Loading Tokenizer
+ Downloading tokenizer files from Hugging Face Hub…
This may take a moment on first load. Files are cached in your browser.
+ This may take a moment on first load. Files are cached in your browser.
+
+
+
+