v0: tokenizer trained on pretraining corpus

README.md

@@ -1,78 +1,32 @@
 ---
 license: apache-2.0
-language:
-- en
 tags:
-- tokenizer
-- code
-- bpe
+  - merlin
+  - tokenizer
 ---
 
 # merlin-tokenizer-v0
 
-BPE tokenizer for [Merlin](https://github.com/tsuberim/merlin) — a small LM purpose-built for agentic coding on Apple Silicon.
+BPE tokenizer for [Merlin](https://github.com/tsuberim/mllm), trained on Python, Bash, Markdown,
+Stack Exchange Q&A, GitHub commits/issues, and tldr-pages.
 
-**v0** — trained on Python, Bash, and Markdown from The Stack dedup. Will be retrained clean (v1) after agentic traces are generated.
+## Vocab
 
-## Specs
-
-| Property | Value |
-|---|---|
-| Algorithm | BPE (byte-level) |
-| Vocab size | 32,016 |
-| BPE tokens | 32,000 |
-| Special tokens | 16 |
-| Backend | HuggingFace `tokenizers` (Rust) |
-| Compression | ~3.57 chars/token on Python |
-
-## Special tokens
-
-Legacy tokens (original BPE training, IDs 0–13):
-
-| ID | Token |
-|---|---|
-| 0 | `<\|bos\|>` |
-| 1 | `<\|eos\|>` |
-| 2 | `<\|pad\|>` |
-| 3 | `<\|unk\|>` |
-| 4 | `<\|user\|>` |
-| 5 | `<\|assistant\|>` |
-| 6 | `<\|tool_call\|>` |
-| 7 | `<\|end_tool_call\|>` |
-| 8 | `<\|tool_result\|>` |
-| 9 | `<\|sep\|>` |
-| 10 | `<\|end\|>` |
-| 11 | `<\|python\|>` |
-| 12 | `<\|bash\|>` |
-| 13 | `<\|markdown\|>` |
-
-Agent protocol tokens (patched in, IDs 32000–32015):
-
-| ID | Token | Role |
-|---|---|---|
-| 32000 | `<\|task\|>` | task open |
-| 32001 | `<\|/task\|>` | task close |
-| 32002 | `<\|think\|>` | thinking open |
-| 32003 | `<\|/think\|>` | thinking close |
-| 32004 | `<\|/tool_call\|>` | tool call close |
-| 32005 | `<\|/tool_result\|>` | tool result close |
-| 32006 | `<\|spawn\|>` | spawn agent open |
-| 32007 | `<\|/spawn\|>` | spawn agent close |
-| 32008 | `<\|agent_id\|>` | agent ID open |
-| 32009 | `<\|/agent_id\|>` | agent ID close |
-| 32010 | `<\|wait\|>` | wait open |
-| 32011 | `<\|/wait\|>` | wait close |
-| 32012 | `<\|wait_result\|>` | wait result open |
-| 32013 | `<\|/wait_result\|>` | wait result close |
-| 32014 | `<\|done\|>` | done open |
-| 32015 | `<\|/done\|>` | done close |
+- 32,016 tokens total: 32,000 BPE + 16 special tokens
+- Special tokens:
+  - IDs 0–13: legacy slots (unused)
+  - ID 32000: `<|bos|>`
+  - ID 32001: `<|eos|>`
+  - ID 32002–32013: agent protocol tokens (`<|tool_call|>`, `<|/tool_call|>`, etc.)
+  - ID 32014: `<|done|>`
+  - ID 32015: `<|pad|>`
 
 ## Usage
 
 ```python
 from tokenizers import Tokenizer
-
-tok = Tokenizer.from_pretrained("tsuberim/merlin-tokenizer-v0")
+tok = Tokenizer.from_file("tokenizer.json")
 ids = tok.encode("def hello(): pass").ids
-text = tok.decode(ids, skip_special_tokens=False)
 ```
+
+> **Note:** Will be retrained after agentic trace generation. This is v0 — trained on pretraining corpus only.