| --- |
| license: apache-2.0 |
| language: |
| - en |
| - zh |
| library_name: tokenizers |
| tags: |
| - qwen |
| - qwen-image |
| - tokenizer |
| - mlx |
| - sceneworks |
| --- |
| |
| # Qwen-Image fast tokenizer (`tokenizer.json`) |
|
|
| A **derived artifact** for running [Qwen/Qwen-Image](https://huggingface.co/Qwen/Qwen-Image) |
| on the native-Rust/MLX **mlx-gen** engine (SceneWorks). |
|
|
| ## Why this exists |
|
|
| `Qwen/Qwen-Image` ships its Qwen2 BPE tokenizer as **`vocab.json` + `merges.txt` only** — there is |
| **no fast `tokenizer.json`** in the upstream repo (the Python fork builds the fast tokenizer at |
| runtime via `transformers`). The Rust engine's tokenizer loader (`mlx_gen::TextTokenizer`, consumed by |
| the `qwen-image` provider's `load_tokenizer`) reads the HF `tokenizers` **fast** serialization, so it |
| needs a `tokenizer.json`. |
|
|
| This repo hosts that derived `tokenizer.json` so SceneWorks model-install can overlay it onto the |
| upstream Qwen-Image snapshot (instead of running a Python `vocab.json`+`merges.txt`→fast conversion at |
| install time on every machine — the desktop Mac bundle ships no Python). See SceneWorks sc-6570; this |
| mirrors the [Kolors fast-tokenizer overlay](https://huggingface.co/SceneWorks/kolors-chatglm3-tokenizer) |
| (sc-4764). |
|
|
| > Note: `Qwen/Qwen-Image-Edit-2511` already ships its own `tokenizer.json` upstream, so only the |
| > base text-to-image `Qwen/Qwen-Image` repo needs this overlay. |
|
|
| ## How it was built |
|
|
| Materialized by `tools/build_qwen_tokenizer.py` (mlx-gen): loads the Qwen2 tokenizer with |
| `transformers.AutoTokenizer.from_pretrained` (the fast path) and writes `backend_tokenizer.save(...)`. |
| The result is the **byte-identical** fast tokenizer the fork builds at runtime — same vocab, merges, |
| NFC + ByteLevel pipeline, and special tokens. |
|
|
| **Validation:** fast-tokenizer ids == the fork's runtime `transformers` tokenizer across an |
| EN + EN-long + CN + mixed CN/EN/numeric/punct + empty(negative-prompt) battery — **0 mismatches**. |
| `vocab_size` 151665, pad token id 151643 (`<|endoftext|>`). |
|
|
| ## Files |
|
|
| - `tokenizer.json` — the derived fast tokenizer (the file the Rust engine needs). |
| - `vocab.json`, `merges.txt`, `tokenizer_config.json`, `added_tokens.json`, `special_tokens_map.json` — |
| the upstream slow-tokenizer source files (provenance / reproducibility). |
|
|
| ## License & provenance |
|
|
| Derived from the Qwen2 tokenizer shipped with [Qwen/Qwen-Image](https://huggingface.co/Qwen/Qwen-Image) |
| (Apache-2.0). This repo redistributes only the tokenizer (no model weights) for engine interoperability. |
|
|
