| license: other | |
| license_name: modified-mit | |
| license_link: https://huggingface.co/moonshotai/Kimi-K2.6/blob/main/LICENSE | |
| # Kimi-K2.6 fast tokenizer (no custom code) | |
| Faithful conversion of the `moonshotai/Kimi-K2.6` tiktoken tokenizer to a HF fast tokenizer | |
| (`PreTrainedTokenizerFast`), loadable **without `trust_remote_code`**. Built with | |
| `transformers.convert_slow_tokenizer.TikTokenConverter` from `tiktoken.model` | |
| (163584 base ranks + 256 special tokens, ids preserved). Verified token-for-token identical | |
| to the reference tokenizer on code/CJK/special-token/long-text samples. | |
| Purpose: benchmarking harnesses (genai-bench etc.) that cannot enable remote code. | |
| Maintained by Baseten FDE; not an official Moonshot AI artifact. | |