README.md

metadata

title: Claude Tokenizer
emoji: 🔤
colorFrom: purple
colorTo: green
sdk: gradio
sdk_version: 6.24.0
python_version: '3.12'
app_file: app.py
pinned: false
license: mit
short_description: How Claude tokenizes text, next to GPT and open models
tags:
  - tokenizer
  - tokenization
  - multilingual
  - claude

Claude's tokenizer, side by side

Paste text — or point at any public dataset — and see how each tokenizer segments it, what the count is, and how many characters each tokenizer buys per token.

Source	github.com/sanderland/ctok
Python package	pip install ctok — counts Claude tokens offline, no API call
The research	On the biology of Claude's tokenizer

What is compared

• Claude, via ctok: v3 (Claude 3 through Opus 4.6) and v5 (Opus 4.7 onward). v4.7 and v5 share a vocabulary, so on text-only counts they are one tokenizer.
• OpenAI, via tiktoken: cl100k_base and o200k_base.
• Anything on the Hub with a tokenizer.json, via tokenizers — type a repo id into the dropdown. Gated repos need an HF_TOKEN secret on the Space.

The defaults are DeepSeek V4 Flash, o200k_base, and Claude v5.

Reading the numbers

Counts are text-only, without the fixed frame a real one-message API request adds on top.

ctok reconstructs the vocabulary rather than reading it, so its vocab figure is the number of pieces with a membership witness, not a published vocabulary size.

Claude rows carry the notation from the write-up: ^ and $ for the start and end of a word, ↑ for a capitalised word, ⇪ for an all-caps one, and 0xNN for a UTF-8 byte the vocabulary has no piece for. A � marks a token covering only part of a character.

Datasets

The dataset tab streams rows, so it never downloads a full corpus. Pick a dataset id, a config, and a text column, then compare over the first N rows. google/wmt24pp is a good starting point: the same English source translated into 55 languages, which makes the per-language cost differences easy to see.

Running it locally

pip install -r requirements.txt gradio==6.24.0
python app.py