Create README.md

README.md

@@ -0,0 +1,67 @@
+---
+language: en
+tags:
+  - tokenizer
+  - bpe
+  - shell
+  - code
+  - chatml
+license: apache-2.0
+datasets:
+  - bigcode/the-stack-dedup
+  - m-a-p/CodeFeedback-Filtered-Instruction
+  - HuggingFaceH4/helpful-instructions
+  - HuggingFaceFW/fineweb
+  - Magpie-Align/Magpie-Reasoning-150K
+---
+
+# DwarfGoToken
+
+A compact **BPE tokenizer** (8,192 tokens) designed for tiny language models that need to understand **shell commands, code snippets, and ChatML-formatted conversations**. Built on top of a custom Go pre‑tokenizer that keeps critical shell tokens (`grep`, `chmod`, `2>&1`, `-rf`, …) atomic, avoiding the fragmentation that kills performance on CPU-bound inference.
+
+## Why 8,192 tokens?
+
+For a small LM (<20M parameters), a large vocabulary (e.g., 64K) wastes the majority of the model’s parameters on the embedding matrix. With `d_model=256`, the embedding here accounts for only **2.1M parameters (~14%)** — the rest goes into the transformer layers, where it matters most.
+
+## Corpus
+
+| Source | Domain | Lines |
+|--------|--------|-------|
+| `bigcode/the-stack-dedup/shell` | Shell | 1,500,000 |
+| `bigcode/the-stack-dedup/batchfile` | Batch | 500,000 |
+| `bigcode/the-stack-dedup/python` | Python | 1,000,000 |
+| `bigcode/the-stack-dedup/c` | C | 500,000 |
+| `m-a-p/CodeFeedback-Filtered-Instruction` | Code+Instructions | 200,000 |
+| `HuggingFaceH4/helpful-instructions` | English instructions | 150,000 |
+| `HuggingFaceFW/fineweb/sample-10BT` | Web English | 300,000 |
+| `Magpie-Align/Magpie-Reasoning-150K` | Chain-of-Thought | 200,000 |
+
+**Total:** 4,251,427 lines (3.5 GB) — 47% Shell, 40% Code, 9.5% EN, 3.5% CoT.
+
+## Special tokens (all atomic)
+
+`<s>`, `</s>`, `<unk>`, `<pad>`, `<|system|>`, `<|user|>`, `<|assistant|>`, `<|end|>`, `<|thinking|>`, `<|/thinking|>`, plus **54 Go‑pre‑tokenizer tokens** (e.g., `grep`, `chmod`, `2>&1`, `&&`, `>>`, `-rf`, `--help`).
+
+## Quick test
+
+```python
+from transformers import AutoTokenizer
+tok = AutoTokenizer.from_pretrained("ThingAI/DwarfGoToken")
+
+# Shell commands stay atomic
+tok.tokenize("find /var/log -name '*.gz' | xargs rm -rf")
+# → ['find', '/', 'var', '/', 'log', '-n', 'ame', "'", '*.', 'gz', "'", '|', 'xargs', 'rm', '-rf']
+
+# ChatML template
+tok.tokenize("<|user|>\nCosa fa grep?\n<|end|>\n<|assistant|>\n...")
+# → ['<|user|>', 'C', 'os', 'a', 'fa', 'grep', '?', '<|end|>', '<|assistant|>', '...']
+```
+## Usage
+```python
+from transformers import AutoTokenizer
+tokenizer = AutoTokenizer.from_pretrained("ThingAI/DwarfGoToken")
+```
+## Intended use
+*This tokenizer was built to pair with tiny LMs (~10–20M parameters) specialised in command‑line assistance, shell scripting, or code generation. It’s the companion of the Dwarf model family by ThingsAI.*
+## License
+**Apache 2.0** — *use it, modify it, ship it.*