README.md

---
title: Token Visualizer
emoji: 🖋️
colorFrom: purple
colorTo: blue
sdk: docker
pinned: false
license: mit
---

# Token Visualizer

A web app to visualize text tokenization by any LLM hosted on HuggingFace. Inspired by OpenAI's tokenizer tool.

## Features

- Load any tokenizer from Hugging Face Hub by model ID (e.g., `openai-community/gpt2`)
- Color-coded token visualization
- Hover to see token IDs
- Character and token count statistics
- Popular model suggestions

## Usage

1. Enter a Hugging Face model ID (e.g., `meta-llama/Llama-2-7b-hf`)
2. Type or paste text you want to tokenize
3. Click "Tokenize" to see the colorized output

## Local Development

```bash
pip install -r requirements.txt
python app.py
```

Then open http://localhost:7860 in your browser.