sh-lee/nanochat-d12-base

nanochat d12 base (FineWeb-Edu, 3.7B tokens)

A small base language model trained with nanochat at commit d6d86cb (October 2025) plus a two-line data-loader patch (below). It continues text; it is not a chat model and was not fine-tuned or aligned. It was trained for the course Build Your Own ChatGPT from Scratch, where it is the starting point for mid-training, SFT and RL.

Model

Architecture	nanochat GPT, depth 12, width 768, 6 heads (no grouped KV), RoPE, QK norm, ReLU² MLP, RMSNorm, untied embeddings
Parameters	185,597,952
Context	2,048 tokens
Tokenizer	nanochat rustbpe/tiktoken BPE, vocabulary 65,536 (tokenizer/)
Training data	FineWeb-Edu (nanochat's karpathy/fineweb-edu-100b-shuffle shards), 3,711,959,040 tokens (20 tokens per parameter)
Training	7,080 steps × 524,288 tokens, Muon + AdamW, bf16, 2× A100 80GB PCIe, about 3.6 hours

Evaluation (from the training run's own logs)

Metric	Value
Validation bits per byte (FineWeb-Edu)	0.9149
CORE (22 tasks, centered so random = 0)	0.1240

For scale: the CORE of GPT-2 small is about 0.11 and of GPT-2 XL about 0.26 by nanochat's own eval bundle. Expect fluent-looking but often wrong text, repetition, and near-chance results on many knowledge tasks.

Files

base_checkpoints/d12/model_007080.pt    # PyTorch state dict as saved by nanochat (token embedding bf16, the rest fp32)
base_checkpoints/d12/meta_007080.json   # model config and training settings
tokenizer/tokenizer.pkl                  # nanochat tokenizer (a pickle: load only from sources you trust)
tokenizer/token_bytes.pt
course_patch.diff                        # the two-line data-loader fix below

Use with nanochat

git clone https://github.com/karpathy/nanochat && cd nanochat
huggingface-cli download sh-lee/nanochat-d12-base --local-dir ../cache
git checkout d6d86cb && git apply ../cache/course_patch.diff     # the patch below
export NANOCHAT_BASE_DIR=$(cd ../cache && pwd)
python -m scripts.base_eval            # or continue with scripts.mid_train, scripts.chat_sft

The data-loader patch

In the version above, the pretraining and mid-training loaders copy their targets from a reused pinned buffer with non_blocking=True; when the GPU runs behind the CPU, the buffer can be refilled before the copy runs and some micro-batches train on the wrong targets. On our hardware this cost about 6% in validation bits per byte (800-step A/B, same speed). This checkpoint was trained with the fix:

diff --git a/nanochat/dataloader.py b/nanochat/dataloader.py
index c1636b1..05b149b 100644
--- a/nanochat/dataloader.py
+++ b/nanochat/dataloader.py
@@ -42,7 +42,7 @@ def tokenizing_distributed_data_loader(B, T, split, tokenizer_threads=4, tokeniz
             scratch[i] = token_buffer.popleft()
         # Create the inputs/targets as 1D tensors
         inputs_cpu = scratch[:-1].to(dtype=torch.int32)
-        targets_cpu = scratch[1:]
+        targets_cpu = scratch[1:].clone()  # course patch: scratch is refilled before the async copy runs
         # Reshape to 2D and move to GPU async
         inputs = inputs_cpu.view(B, T).to(device="cuda", dtype=torch.int32, non_blocking=True)
         targets = targets_cpu.view(B, T).to(device="cuda", dtype=torch.int64, non_blocking=True)
diff --git a/scripts/mid_train.py b/scripts/mid_train.py
index 90ab954..aa5bc55 100644
--- a/scripts/mid_train.py
+++ b/scripts/mid_train.py
@@ -128,7 +128,7 @@ def mid_data_generator(split):
         for i in range(needed_tokens):
             scratch[i] = token_buffer.popleft()
         inputs_cpu = scratch[:-1].to(dtype=torch.int32)
-        targets_cpu = scratch[1:]
+        targets_cpu = scratch[1:].clone()  # course patch: scratch is refilled before the async copy runs
         inputs = inputs_cpu.view(device_batch_size, max_seq_len).to(device=device, dtype=torch.int32, non_blocking=True)
         targets = targets_cpu.view(device_batch_size, max_seq_len).to(device=device, dtype=torch.int64, non_blocking=True)
         if split == "train":

Licence and attribution

The weights are released under the MIT licence, like nanochat itself. The training data is FineWeb-Edu (ODC-By 1.0; also subject to Common Crawl's terms of use); please keep this attribution when you redistribute the model or derivatives. Not affiliated with or endorsed by Andrej Karpathy or Hugging Face.