README.md

---
title: Claude Tokenizer
emoji: 🔤
colorFrom: purple
colorTo: green
sdk: gradio
sdk_version: 6.24.0
python_version: '3.12'
app_file: app.py
pinned: false
license: mit
short_description: How Claude tokenizes text, next to GPT and open models
tags:
  - tokenizer
  - tokenization
  - multilingual
  - claude
---

# Claude's tokenizer, side by side

Paste text — or point at any public dataset — and see how each tokenizer segments it, what the
count is, and how many characters each tokenizer buys per token.

| | |
|---|---|
| **Source** | [github.com/sanderland/ctok](https://github.com/sanderland/ctok) |
| **Python package** | [`pip install ctok`](https://pypi.org/project/ctok/) — counts Claude tokens offline, no API call |
| **The research** | [On the biology of Claude's tokenizer](https://tokencontributions.substack.com/p/on-the-biology-of-claudes-tokenizer) |

## What is compared

- **Claude**, via [`ctok`](https://pypi.org/project/ctok/): v3 (Claude 3 through Opus 4.6) and v5
  (Opus 4.7 onward). v4.7 and v5 share a vocabulary, so on text-only counts they are one tokenizer.
- **OpenAI**, via [`tiktoken`](https://github.com/openai/tiktoken): `cl100k_base` and `o200k_base`.
- **Anything on the Hub** with a `tokenizer.json`, via
  [`tokenizers`](https://github.com/huggingface/tokenizers) — type a repo id into the dropdown.
  Gated repos need an `HF_TOKEN` secret on the Space.

The defaults are DeepSeek V4 Flash, `o200k_base`, and Claude v5.

## Reading the numbers

Counts are text-only, without the fixed frame a real one-message API request adds on top.

`ctok` reconstructs the vocabulary rather than reading it, so its **vocab** figure is the number of
pieces with a membership witness, not a published vocabulary size.

Claude rows carry the notation from the write-up: `^` and `$` for the start and end of a word, `↑`
for a capitalised word, `⇪` for an all-caps one, and `0xNN` for a UTF-8 byte the vocabulary has no
piece for. A `�` marks a token covering only part of a character.

## Datasets

The dataset tab streams rows, so it never downloads a full corpus. Pick a dataset id, a config, and
a text column, then compare over the first *N* rows. `google/wmt24pp` is a good starting point: the
same English source translated into 55 languages, which makes the per-language cost differences
easy to see.

## Running it locally

```bash
pip install -r requirements.txt gradio==6.24.0
python app.py
```