small-lm-lab tokenizers
The two tokenizers used by small-lm-lab, a study of training small language
models with modern methods on free compute (repository link added at release). This
folder is both part of the project repository (configs/tokenizers/) and the
content of the public Hugging Face repo adityaansh419/small-lm-lab-tokenizers.
| Folder | What it is | Vocabulary |
|---|---|---|
own-16384/ |
The project's own byte-level BPE tokenizer | 16,384 |
smol2/ |
Hugging Face's SmolLM2 tokenizer, unmodified, needed for distillation from SmolLM2 teachers | 49,152 |
Each folder has tokenizer.json (load with tokenizers.Tokenizer.from_file)
and token_bytes.npy: the UTF-8 byte length of every token (0 for special
tokens), used to compute bits per byte. manifest.yaml lists every file's
SHA-256; jobs verify them before use.
own-16384
- Algorithm: byte-level BPE (all 256 bytes in the vocabulary, so any text is encodable and decoding is lossless).
- Pre-tokenization: the GPT-4 / Llama 3 split pattern, except that numbers are split into single digits.
- Special tokens (ids 0β7):
<|endoftext|>(end of document),<|im_start|>,<|im_end|>(ChatML turns),<|pad|>,<|reserved_0|>β¦<|reserved_3|>. - Training data: 1.2 GB of text in the project's real-web mix: 57%
FineWeb-Edu (ODC-By 1.0), 38% DCLM-Baseline (CC-BY-4.0), 5% FineMath-4+
(ODC-By 1.0), pinned revisions in the project's
configs/data_sources.yaml. - Why 16,384: least training compute per byte of text among 4Kβ32K
candidates for 35β75M-parameter models (measurements in the project's
docs/DECISIONS.md, D-019). - License: MIT (this tokenizer); the training data's licenses require attribution, given above.
smol2
The tokenizer of HuggingFaceTB/SmolLM2-135M at revision
93efa2f097d58c2a74874c7e644dbc9b0cee75a2, redistributed unmodified under its
Apache-2.0 license. Note: its vocabulary has no tokens for 21 byte values
(including six ASCII control characters), so text must be cleaned of control
characters before encoding to stay lossless (project decision D-021).