| tokenizer-lm |
| Copyright 2026 Clara Meister |
|
|
| This product includes software developed as part of the tokenizer-lm study, |
| licensed under the Apache License, Version 2.0. See LICENSE. |
|
|
| -------------------------------------------------------------------------------- |
| Third-party components |
| -------------------------------------------------------------------------------- |
|
|
| nanochat |
| https://github.com/karpathy/nanochat |
| MIT License, Copyright (c) 2025 Andrej Karpathy |
|
|
| The model architecture, optimizer and dataloader come from nanochat. This project |
| uses a patched copy (https://github.com/cimeister/nanochat, branch |
| tokenizer-lm-patches) that adds a configurable value-embedding dimension, a |
| configurable logit soft cap, and a configurable learning-rate width exponent for |
| muP. The released model class scripts/hf_release/nanochat_model.py reimplements that |
| architecture for the HuggingFace transformers interface. The full MIT license text |
| is at nanochat/LICENSE. |
|
|
| -------------------------------------------------------------------------------- |
| Training data |
| -------------------------------------------------------------------------------- |
|
|
| The released model weights were trained on a mixture of: |
|
|
| FineWeb-Edu HuggingFaceFW/fineweb-edu ODC-By |
| FineWeb-2 (filtered) HuggingFaceFW/fineweb-2 ODC-By |
| FineMath 4plus HuggingFaceTB/finemath ODC-By |
| StarCoderData bigcode/starcoderdata listed as "other" |
|
|
| Licenses as stated on each dataset's Hugging Face card, read 2026-08-17. The |
| multilingual and code portions were used through quality-filtered derivatives |
| prepared inside the Swiss AI initiative; see EXPERIMENTAL_SETUP.md for the exact |
| paths and filters. |
|
|
| -------------------------------------------------------------------------------- |
| Tokenizers not redistributed |
| -------------------------------------------------------------------------------- |
|
|
| Two off-the-shelf tokenizers were used as reference points and are cited rather than |
| redistributed: the Apertus tokenizer (swiss-ai/Apertus-70B-2509, the same artifact as |
| Mistral-Nemo's tokenizer) and the Llama-3 tokenizer (recorded at training time as |
| NousResearch/Meta-Llama-3-8B). In the names of the tokenizers this project trained, |
| "gpt4o", "claude" and "gpt2" refer to a borrowed pretokenization regular expression, |
| not to those vendors' tokenizers. |
|
|
