README.md

metadata

title: Token Visualizer
emoji: 🖋️
colorFrom: purple
colorTo: blue
sdk: docker
pinned: false
license: mit

Token Visualizer

A web app to visualize text tokenization by any LLM hosted on HuggingFace. Inspired by OpenAI's tokenizer tool.

Features

• Load any tokenizer from Hugging Face Hub by model ID (e.g., openai-community/gpt2)
• Color-coded token visualization
• Hover to see token IDs
• Character and token count statistics
• Popular model suggestions

Usage

1. Enter a Hugging Face model ID (e.g., meta-llama/Llama-2-7b-hf)
2. Type or paste text you want to tokenize
3. Click "Tokenize" to see the colorized output

Local Development

pip install -r requirements.txt
python app.py

Then open http://localhost:7860 in your browser.