Spaces:
Configuration error
Configuration error
Update README.md
Browse files
README.md
CHANGED
|
@@ -33,21 +33,6 @@ Building efficient, specialist Small Language Models that run on consumer hardwa
|
|
| 33 |
* **Quark-Mod**
|
| 34 |
Multi-label content moderation model. 9 categories: toxic, severe_toxic, obscene, threat, insult, identity_hate, cyberbullying, hate_speech, offensive.
|
| 35 |
|
| 36 |
-
## Tokenizers
|
| 37 |
-
|
| 38 |
-
* **DwarfGoToken**
|
| 39 |
-
An 8202-token BPE tokenizer built for shell/bash. Uses ByteLevel pre-tokenization with syntax-aware protected tokens for shell operators (2>&1, &&, >>, ||) that would otherwise be split by standard BPE. Built on a 51MB corpus of shell, Python, C, and English text. Two critical bugs were found and fixed during Dwarf-15M training: space loss from incorrect pre-tokenizer configuration, and short shell keywords (fi, do, if) matching as substrings inside English words.
|
| 40 |
-
|
| 41 |
-
* **GoToken**
|
| 42 |
-
A BPE tokenizer written in Rust with Python bindings via PyO3. Published on crates.io and PyPI. Provides syntax-aware pre-tokenization for shell/bash patterns. Used as the foundation for DwarfGoToken.
|
| 43 |
-
|
| 44 |
-
## What We Focus On
|
| 45 |
-
|
| 46 |
-
* **Specialist over generalist:** A 15M model can't do everything, but it can excel at one thing. Dwarf targets shell/bash; future models will target math/physics.
|
| 47 |
-
* **Honest failure documentation:** When something doesn't work (72M vocabulary problem, zero-shot CoT degradation, tokenizer bugs), we publish the failure and what we learned.
|
| 48 |
-
* **Extreme overtraining for small models:** Following the Phi/SmolLM philosophy — small models need more tokens per parameter, not fewer. Dwarf trains at 2500:1, 125x beyond Chinchilla optimal.
|
| 49 |
-
* **Custom tooling from scratch:** Tokenizers (gotoken, DwarfGoToken), training scripts with multi-source streaming, and inference tools — all built in-house, all open.
|
| 50 |
-
* **Consumer hardware:** Everything runs on an RTX 3070 (8GB) or equivalent. No datacenter required.
|
| 51 |
|
| 52 |
## Links
|
| 53 |
|
|
|
|
| 33 |
* **Quark-Mod**
|
| 34 |
Multi-label content moderation model. 9 categories: toxic, severe_toxic, obscene, threat, insult, identity_hate, cyberbullying, hate_speech, offensive.
|
| 35 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 36 |
|
| 37 |
## Links
|
| 38 |
|