small-lm-lab tokenizers

The two tokenizers used by small-lm-lab, a study of training small language models with modern methods on free compute (repository link added at release). This folder is both part of the project repository (configs/tokenizers/) and the content of the public Hugging Face repo adityaansh419/small-lm-lab-tokenizers.

Folder What it is Vocabulary
own-16384/ The project's own byte-level BPE tokenizer 16,384
smol2/ Hugging Face's SmolLM2 tokenizer, unmodified, needed for distillation from SmolLM2 teachers 49,152

Each folder has tokenizer.json (load with tokenizers.Tokenizer.from_file) and token_bytes.npy: the UTF-8 byte length of every token (0 for special tokens), used to compute bits per byte. manifest.yaml lists every file's SHA-256; jobs verify them before use.

own-16384

  • Algorithm: byte-level BPE (all 256 bytes in the vocabulary, so any text is encodable and decoding is lossless).
  • Pre-tokenization: the GPT-4 / Llama 3 split pattern, except that numbers are split into single digits.
  • Special tokens (ids 0–7): <|endoftext|> (end of document), <|im_start|>, <|im_end|> (ChatML turns), <|pad|>, <|reserved_0|> … <|reserved_3|>.
  • Training data: 1.2 GB of text in the project's real-web mix: 57% FineWeb-Edu (ODC-By 1.0), 38% DCLM-Baseline (CC-BY-4.0), 5% FineMath-4+ (ODC-By 1.0), pinned revisions in the project's configs/data_sources.yaml.
  • Why 16,384: least training compute per byte of text among 4K–32K candidates for 35–75M-parameter models (measurements in the project's docs/DECISIONS.md, D-019).
  • License: MIT (this tokenizer); the training data's licenses require attribution, given above.

smol2

The tokenizer of HuggingFaceTB/SmolLM2-135M at revision 93efa2f097d58c2a74874c7e644dbc9b0cee75a2, redistributed unmodified under its Apache-2.0 license. Note: its vocabulary has no tokens for 21 byte values (including six ASCII control characters), so text must be cleaned of control characters before encoding to stay lossless (project decision D-021).

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support