---
license: other
license_name: kolors
license_link: https://huggingface.co/Kwai-Kolors/Kolors-diffusers/blob/main/MODEL_LICENSE
language:
- en
- zh
library_name: tokenizers
tags:
- kolors
- chatglm3
- tokenizer
- mlx
- sceneworks
---

# Kolors ChatGLM3 fast tokenizer (`tokenizer.json`)

A **derived artifact** for running [Kwai-Kolors/Kolors-diffusers](https://huggingface.co/Kwai-Kolors/Kolors-diffusers)
on the native-Rust/MLX **mlx-gen** engine (SceneWorks, epic 3090).

## Why this exists

Kolors conditions on **ChatGLM3-6B**, which ships **only a slow SentencePiece tokenizer**
(`tokenizer.model` + a custom `ChatGLMTokenizer(PreTrainedTokenizer)`) — there is **no fast
`tokenizer.json`** in the upstream repo. The Rust engine's tokenizer loader (`mlx_gen::TextTokenizer`,
consumed by the Kolors generator and the Kolors LoRA/LoKr trainer) reads the HF `tokenizers` **fast**
serialization, so it needs a `tokenizer.json`.

This repo hosts that derived `tokenizer.json` so SceneWorks model-install can overlay it onto the
upstream Kolors-diffusers snapshot (instead of running a Python SentencePiece→fast conversion at
install time on every machine — a Python-eradication consideration, epic 3482).

## How it was built

Materialized by `tools/build_kolors_tokenizer.py` (mlx-gen): converts the ChatGLM3 SP model to a fast
`tokenizer.json` via transformers' SP converter. The fast tokenizer reproduces the SP **content** ids
exactly; it adds **no** special tokens — the ChatGLM `[gMASK]` (64790) / `sop` (64792) prefix, left-pad,
and `position_ids` are applied by the Rust `KolorsTokenizer` wrapper (matching
`build_inputs_with_special_tokens` + `_pad`, `max_length=256`).

**Validation:** fast-tokenizer content ids == `sp_model.encode(text)` across an EN + EN-long(truncation)
+ CN + mixed CN/EN + empty(negative-prompt) battery; special-token ids asserted
(`[gMASK]`=64790, `sop`=64792, pad=unk=0).

## Files

- `tokenizer.json` — the derived fast tokenizer (the file the Rust engine needs).
- `tokenizer.model` — the upstream ChatGLM3 SentencePiece model (provenance / reproducibility).
- `tokenizer_config.json` — the upstream tokenizer config.

## License & provenance

Derived from the ChatGLM3 tokenizer shipped with Kwai-Kolors/Kolors-diffusers. Use is governed by the
upstream [Kolors model license](https://huggingface.co/Kwai-Kolors/Kolors-diffusers/blob/main/MODEL_LICENSE)
and the [ChatGLM3-6B license](https://huggingface.co/THUDM/chatglm3-6b). This repo redistributes only the
tokenizer (no model weights) for engine interoperability.