File size: 506 Bytes
090668b
 
0285034
 
090668b
 
0285034
090668b
 
0285034
 
090668b
 
0285034
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
---
title: Tokenizer Visualizer
emoji: πŸ”€
colorFrom: blue
colorTo: purple
sdk: gradio
sdk_version: "4.44.0"
app_file: app.py
pinned: false
license: apache-2.0
tags: [tokenizer, visualization, mcp, dispatchai]
---

# πŸ”€ dispatchAI Tokenizer Visualizer

**MCP Server** β€” visualize how tokenizers split text.

## Tools
- `tokenize_text(text, method)` β€” Tokenize with chosen method
- `compare_methods(text)` β€” Compare token counts across methods

πŸš€ [dispatchAI](https://huggingface.co/dispatchAI)