Yujivus/nanochat-tokenizer-digit-32k

nanochat digit-level 32K tokenizer

A nanochat-compatible 32,768-token RustBPE/tiktoken tokenizer trained on the first 8 training shards of Yujivus/nanochat-climbmix-arithmetic-base10 (source revision 61e1951a802b0d5a17b69bf712407e7ec0acb411).

The only nanochat code change is the numeric pre-tokenization rule:

\p{N}{1,2}  ->  \p{N}{1}

Therefore decimal strings are tokenized digit by digit, e.g. 189345 -> 1|8|9|3|4|5. The same tokenizer can be used for base-10 and base-6 experiments.

Compatibility

• nanochat commit: 92d63d4e8bb4df75c3b71618f31ddde2378b2bcd
• vocab size: 32768
• tokenizer training text cap: 2,000,000,000 characters
• files expected by nanochat: tokenizer.pkl, token_bytes.pt

tokenizer.pkl is a Python pickle. Load only artifacts from sources you trust and use the pinned nanochat commit above. See audit.json, manifest.json, and the included patch.

RunPod placement

Download the repository contents into $NANOCHAT_BASE_DIR/tokenizer/, then skip python -m scripts.tok_train and start nanochat training.

Provenance

• nanochat: https://github.com/karpathy/nanochat
• mixed dataset: https://huggingface.co/datasets/Yujivus/nanochat-climbmix-arithmetic-base10
• upstream ClimbMix licensing is conservatively carried over as CC BY-NC 4.0.