Update README.md

README.md

@@ -1,9 +1,9 @@
-# Baseten Tokenizer
+# Baseten Tokenizer for Kimi K3
 
-High-performance, Rust-backed BPE tokenization for inference. The Python
-package is named `basetenkenizer`.
+High-performance, Rust-backed BPE tokenization for inference. This repo contains the `tokenizer.json`.
+The Python package is named `basetenkenizer`.
 
-## Performance
+## Fast tokenization for Kimi K3
 
 ![Kimi K3 tokenization latency comparison](assets/basetenkenizer_vs_gigatokenfastokenstiktoken.png)
 
@@ -17,59 +17,6 @@ Loading by Hugging Face model ID downloads `tokenizer.json` on first use and
 then uses the local Hugging Face cache. Private or gated models use the usual
 `HF_TOKEN` environment variable.
 
-## Encode text
-
-```python
-from basetenkenizer import Tokenizer
-
-tokenizer = Tokenizer.from_model("deepseek-ai/DeepSeek-V3.2")
-encoding = tokenizer.encode(
-    "A very long prompt that is now much faster.",
-    add_special_tokens=False,
-)
-
-print(encoding.ids)
-print(tokenizer.decode(encoding.ids))
-```
-
-`Tokenizer.from_file("tokenizer.json")` loads a local tokenizer. Encoding
-objects expose `ids`, `attention_mask`, `type_ids`, and
-`special_tokens_mask`; selected fields can be moved into NumPy arrays with
-`encoding.into_numpy(...)`.
-
-## Kimi K2.7 Code
-
-Load the published tokenizer directly from its model repository:
-
-```python
-from basetenkenizer import Tokenizer
-
-tokenizer = Tokenizer.from_model("moonshotai/Kimi-K2.7-Code")
-ids = tokenizer.encode("def hello():\n    return 'world'").ids
-```
-
-For a chat prompt, preserve the boundary between template control tokens and
-untrusted message content with `encode_segments`:
-
-```python
-user_message = "Write a Python HTTP server."
-
-segments = [
-    ("<|im_user|>user<|im_middle|>", True),
-    (user_message, False),
-    ("<|im_end|>", True),
-    ("<|im_assistant|>assistant<|im_middle|><think>", True),
-]
-
-encoding = tokenizer.encode_segments(segments)
-input_ids = encoding.ids
-```
-
-This is the minimal Kimi K2.7 Code user/assistant shape. Applications using
-system messages, tools, images, or existing assistant messages should render
-the complete official model template and retain the same control-text versus
-message-text boundaries.
-
 ## Kimi K3
 
 Kimi K3 uses XTML and a Python chat renderer rather than a Jinja chat
@@ -78,7 +25,7 @@ template. Raw text can still be encoded normally:
 ```python
 from basetenkenizer import Tokenizer
 
-tokenizer = Tokenizer.from_model("moonshotai/Kimi-K3")
+tokenizer = Tokenizer.from_model("baseten/kimi-k3-tokenizer")
 ids = tokenizer.encode("Hello from Kimi K3").ids
 ```
 
@@ -141,6 +88,59 @@ and whole-segment BPE encoding is intentional.
 Post-processing, truncation, and padding are applied after all segment IDs
 have been joined.
 
+## Encode text with basetenkenizer
+
+```python
+from basetenkenizer import Tokenizer
+
+tokenizer = Tokenizer.from_model("deepseek-ai/DeepSeek-V3.2")
+encoding = tokenizer.encode(
+    "A very long prompt that is now much faster.",
+    add_special_tokens=False,
+)
+
+print(encoding.ids)
+print(tokenizer.decode(encoding.ids))
+```
+
+`Tokenizer.from_file("tokenizer.json")` loads a local tokenizer. Encoding
+objects expose `ids`, `attention_mask`, `type_ids`, and
+`special_tokens_mask`; selected fields can be moved into NumPy arrays with
+`encoding.into_numpy(...)`.
+
+## Kimi K2.7 Code with basetenkenizer
+
+Load the published tokenizer directly from its model repository:
+
+```python
+from basetenkenizer import Tokenizer
+
+tokenizer = Tokenizer.from_model("moonshotai/Kimi-K2.7-Code")
+ids = tokenizer.encode("def hello():\n    return 'world'").ids
+```
+
+For a chat prompt, preserve the boundary between template control tokens and
+untrusted message content with `encode_segments`:
+
+```python
+user_message = "Write a Python HTTP server."
+
+segments = [
+    ("<|im_user|>user<|im_middle|>", True),
+    (user_message, False),
+    ("<|im_end|>", True),
+    ("<|im_assistant|>assistant<|im_middle|><think>", True),
+]
+
+encoding = tokenizer.encode_segments(segments)
+input_ids = encoding.ids
+```
+
+This is the minimal Kimi K2.7 Code user/assistant shape. Applications using
+system messages, tools, images, or existing assistant messages should render
+the complete official model template and retain the same control-text versus
+message-text boundaries.
+
 ## Use with Transformers
 
 Call `patch_transformers` before loading a tokenizer: