File size: 5,051 Bytes
63fb1ea c9229b5 63fb1ea c9229b5 63fb1ea c9229b5 63fb1ea c9229b5 63fb1ea c9229b5 62be6b8 c9229b5 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 | # Baseten Tokenizer for Kimi K3
High-performance, Rust-backed BPE tokenization for inference. This repo contains the `tokenizer.json`.
The Python package is named `basetenkenizer`.
## Fast tokenization for Kimi K3
## Install
```bash
pip install basetenkenizer
```
Loading by Hugging Face model ID downloads `tokenizer.json` on first use and
then uses the local Hugging Face cache. Private or gated models use the usual
`HF_TOKEN` environment variable.
## Kimi K3
Kimi K3 uses XTML and a Python chat renderer rather than a Jinja chat
template. Raw text can still be encoded normally:
```python
from basetenkenizer import Tokenizer
tokenizer = Tokenizer.from_model("baseten/kimi-k3-tokenizer")
ids = tokenizer.encode("Hello from Kimi K3").ids
```
For rendered chat, pass the official Kimi K3 renderer's `EncodeSegment`
objects to `encode_segments` as `(segment.text, segment.allow_special)`. A
minimal user message followed by an assistant-generation prefix looks like
this:
```python
user_message = "Explain speculative decoding."
segments = [
("<|open|>", True),
("message", False),
(" role", False),
('="', False),
("user", False),
('"', False),
("<|sep|>", True),
(user_message, False),
("<|close|>", True),
("message", False),
("<|sep|>", True),
("<|end_of_msg|>", True),
("<|open|>", True),
("message", False),
(" role", False),
('="', False),
("assistant", False),
('"', False),
("<|sep|>", True),
("<|open|>", True),
("response", False),
("<|sep|>", True),
]
encoding = tokenizer.encode_segments(segments)
input_ids = encoding.ids
```
Use Kimi K3's official `build_chat_segments` renderer for production
conversations involving tools, images, reasoning content, response schemas,
or multiple message types. Do not concatenate its segments before encoding:
doing so loses the special-token safety boundary.
## Why `encode_segments`?
Each segment is a `(text, allow_special)` pair:
- `allow_special=True` recognizes tokenizer control tokens emitted by a trusted
chat renderer.
- `allow_special=False` treats control-token-looking strings in user, tool, or
attribute content as ordinary text.
`tiktoken_safe=True` is the default. It reproduces the chunk boundaries used
by legacy tiktoken tokenizers, including on very long inputs, so token IDs stay
compatible. Set it to `False` only when exact tiktoken parity is not required
and whole-segment BPE encoding is intentional.
Post-processing, truncation, and padding are applied after all segment IDs
have been joined.
## Encode text with basetenkenizer
```python
from basetenkenizer import Tokenizer
tokenizer = Tokenizer.from_model("deepseek-ai/DeepSeek-V3.2")
encoding = tokenizer.encode(
"A very long prompt that is now much faster.",
add_special_tokens=False,
)
print(encoding.ids)
print(tokenizer.decode(encoding.ids))
```
`Tokenizer.from_file("tokenizer.json")` loads a local tokenizer. Encoding
objects expose `ids`, `attention_mask`, `type_ids`, and
`special_tokens_mask`; selected fields can be moved into NumPy arrays with
`encoding.into_numpy(...)`.
## Kimi K2.7 Code with basetenkenizer
Load the published tokenizer directly from its model repository:
```python
from basetenkenizer import Tokenizer
tokenizer = Tokenizer.from_model("moonshotai/Kimi-K2.7-Code")
ids = tokenizer.encode("def hello():\n return 'world'").ids
```
For a chat prompt, preserve the boundary between template control tokens and
untrusted message content with `encode_segments`:
```python
user_message = "Write a Python HTTP server."
segments = [
("<|im_user|>user<|im_middle|>", True),
(user_message, False),
("<|im_end|>", True),
("<|im_assistant|>assistant<|im_middle|><think>", True),
]
encoding = tokenizer.encode_segments(segments)
input_ids = encoding.ids
```
This is the minimal Kimi K2.7 Code user/assistant shape. Applications using
system messages, tools, images, or existing assistant messages should render
the complete official model template and retain the same control-text versus
message-text boundaries.
## Use with Transformers
Call `patch_transformers` before loading a tokenizer:
```python
import basetenkenizer
basetenkenizer.patch_transformers()
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("openai/gpt-oss-120b")
tokens = tokenizer("Hello, world!")
```
Transformers v4 and v5 are supported. Pass
`patch_transformers(apply_chat_template=True)` to also use the native renderer
for supported render-only `apply_chat_template(..., tokenize=False)` calls;
unsupported templates automatically fall back to Transformers.
Baseten Tokenizer is focused on inference and does not implement every Hugging
Face Tokenizers training or alignment feature.
## License
The `basetenkenizer` package is licensed under the MIT License.
[View `basetenkenizer` on PyPI](https://pypi.org/project/basetenkenizer/)
|