TokenViz

Universal tokenizer visualization for any HuggingFace model. See exactly how LLMs break down text into tokens, IDs, and bytes — all in your browser with zero GPU required.

🤖 Select Model

✏️ Input Text

Chars: 0

Words: 0

🔍

Enter text above to visualize tokenization

Select a model and start typing to see the magic happen

💡 How it works: This app uses @huggingface/transformers (v3.5.0) to load tokenizer files directly from the HuggingFace Hub in your browser. It downloads tokenizer.json and tokenizer_config.json and runs tokenization entirely client-side with WebAssembly — no GPU or server required. Works with BPE, WordPiece, Unigram, and SentencePiece tokenizers from any model.