| license: apache-2.0 | |
| tags: | |
| - merlin | |
| - tokenizer | |
| # merlin-tokenizer-v0 | |
| BPE tokenizer for [Merlin](https://github.com/tsuberim/mllm), trained on Python, Bash, Markdown, | |
| Stack Exchange Q&A, GitHub commits/issues, and tldr-pages. | |
| ## Vocab | |
| - 32,016 tokens total: 32,000 BPE + 16 special tokens | |
| - Special tokens: | |
| - IDs 0–13: legacy slots (unused) | |
| - ID 32000: `<|bos|>` | |
| - ID 32001: `<|eos|>` | |
| - ID 32002–32013: agent protocol tokens (`<|tool_call|>`, `<|/tool_call|>`, etc.) | |
| - ID 32014: `<|done|>` | |
| - ID 32015: `<|pad|>` | |
| ## Usage | |
| ```python | |
| from tokenizers import Tokenizer | |
| tok = Tokenizer.from_file("tokenizer.json") | |
| ids = tok.encode("def hello(): pass").ids | |
| ``` | |
| > **Note:** Will be retrained after agentic trace generation. This is v0 — trained on pretraining corpus only. | |