ThingsAI commited on
Commit
93413d4
·
verified ·
1 Parent(s): 0d314c9

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +0 -15
README.md CHANGED
@@ -33,21 +33,6 @@ Building efficient, specialist Small Language Models that run on consumer hardwa
33
  * **Quark-Mod**
34
  Multi-label content moderation model. 9 categories: toxic, severe_toxic, obscene, threat, insult, identity_hate, cyberbullying, hate_speech, offensive.
35
 
36
- ## Tokenizers
37
-
38
- * **DwarfGoToken**
39
- An 8202-token BPE tokenizer built for shell/bash. Uses ByteLevel pre-tokenization with syntax-aware protected tokens for shell operators (2>&1, &&, >>, ||) that would otherwise be split by standard BPE. Built on a 51MB corpus of shell, Python, C, and English text. Two critical bugs were found and fixed during Dwarf-15M training: space loss from incorrect pre-tokenizer configuration, and short shell keywords (fi, do, if) matching as substrings inside English words.
40
-
41
- * **GoToken**
42
- A BPE tokenizer written in Rust with Python bindings via PyO3. Published on crates.io and PyPI. Provides syntax-aware pre-tokenization for shell/bash patterns. Used as the foundation for DwarfGoToken.
43
-
44
- ## What We Focus On
45
-
46
- * **Specialist over generalist:** A 15M model can't do everything, but it can excel at one thing. Dwarf targets shell/bash; future models will target math/physics.
47
- * **Honest failure documentation:** When something doesn't work (72M vocabulary problem, zero-shot CoT degradation, tokenizer bugs), we publish the failure and what we learned.
48
- * **Extreme overtraining for small models:** Following the Phi/SmolLM philosophy — small models need more tokens per parameter, not fewer. Dwarf trains at 2500:1, 125x beyond Chinchilla optimal.
49
- * **Custom tooling from scratch:** Tokenizers (gotoken, DwarfGoToken), training scripts with multi-source streaming, and inference tools — all built in-house, all open.
50
- * **Consumer hardware:** Everything runs on an RTX 3070 (8GB) or equivalent. No datacenter required.
51
 
52
  ## Links
53
 
 
33
  * **Quark-Mod**
34
  Multi-label content moderation model. 9 categories: toxic, severe_toxic, obscene, threat, insult, identity_hate, cyberbullying, hate_speech, offensive.
35
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
36
 
37
  ## Links
38