metadata
license: other
license_name: modified-mit
license_link: https://huggingface.co/moonshotai/Kimi-K2.6/blob/main/LICENSE
Kimi-K2.6 fast tokenizer (no custom code)
Faithful conversion of the moonshotai/Kimi-K2.6 tiktoken tokenizer to a HF fast tokenizer
(PreTrainedTokenizerFast), loadable without trust_remote_code. Built with
transformers.convert_slow_tokenizer.TikTokenConverter from tiktoken.model
(163584 base ranks + 256 special tokens, ids preserved). Verified token-for-token identical
to the reference tokenizer on code/CJK/special-token/long-text samples.
Purpose: benchmarking harnesses (genai-bench etc.) that cannot enable remote code. Maintained by Baseten FDE; not an official Moonshot AI artifact.