| --- |
| title: The Tokenizer Playground |
| emoji: 📝 |
| colorFrom: indigo |
| colorTo: purple |
| sdk: static |
| pinned: false |
| short_description: Experiment with and compare different tokenizers |
| models: |
| - Xenova/gpt-4 |
| - Xenova/text-davinci-003 |
| - Xenova/gpt-3 |
| - Xenova/grok-1-tokenizer |
| - Xenova/claude-tokenizer |
| - Xenova/mistral-tokenizer |
| - Xenova/gemma-tokenizer |
| - Xenova/llama-tokenizer |
| - Xenova/c4ai-command-r-v01-tokenizer |
| - Xenova/t5-small |
| - Xenova/bert-base-cased |
| - xai-org/grok-1 |
| --- |
| |
| Link to source code: https://github.com/xenova/transformers.js/tree/main/examples/tokenizer-playground. |
