WillyGPT

A 1.68B-parameter chat model trained from scratch — pretraining, supervised finetuning (SFT), and GRPO reinforcement learning — on the nanochat recipe (MIT). Built & trained by William Yates on a single 8×H100 node.

Base pretraining clears the GPT-2 CORE reference (0.3016 vs 0.2565). Two checkpoints are released: an SFT generalist (default) and an RL math-specialist.

• Full recipe, training scripts, and reproduction: github.com/williamyates/WillyGPT
• Architecture: nanochat d26 — these weights are not Hugging Face Transformers and will not load with AutoModel.from_pretrained. Run them with the nanochat code (below).

Files in this repo

Path	Checkpoint	Use
chatsft_checkpoints/d26/model_000483.pt	SFT (default)	Best all-rounder; chat, identity, spelling
chatrl_checkpoints/d26/model_000466.pt	RL	GSM8K-specialized; see decoding note below
tokenizer/	tokenizer.pkl, token_bytes.pt	32,768-token BPE

Download & run

# 1. get nanochat
git clone https://github.com/karpathy/nanochat.git && cd nanochat
uv sync --extra gpu        # or: uv sync --extra cpu   (Mac/MPS, slow but works)

# 2. download these weights into nanochat's cache (structure is preserved)
export NANOCHAT_BASE_DIR=$PWD
hf download williamyates/WillyGPT --local-dir "$NANOCHAT_BASE_DIR/.cache/nanochat"

# 3. chat
python -m scripts.chat_cli -i sft -p "Who are you?"       # SFT: generalist (recommended)
python -m scripts.chat_cli -i rl  -p "What is 17 * 23?"   # RL: math specialist
python -m scripts.chat_web                                 # browser UI

A 1.68B model runs (slowly) on Apple Silicon via MPS, or on any single modern GPU.

Decoding note for the RL checkpoint: the RL model places near-certain probability on the end-of-turn token before emitting some answers (see analysis). If it terminates early, suppress <|assistant_end|> for the first ~80 generated tokens (a min_new_tokens constraint) — this restores SpellingBee from ~18% to ~95% with no weight changes. The SFT checkpoint does not need this.

Model specification

Architecture	GPT-style decoder transformer (nanochat d26)
Parameters	1.68B (189 weight tensors)
Layers / heads	26 layers · 13 attention heads · 13 KV heads · head dim 128
Model dimension	1664
Context length	2048
Attention	Sliding-window (SSSL), Flash Attention 3
Precision	fp8 training (Hopper / H100)
Tokenizer	32,768-token BPE, GPT-4 style
Training data	NVIDIA ClimbMix (pretrain); SmolTalk + MMLU + GSM8K + custom identity set (SFT); GSM8K (RL)

Evaluation

Base on the pretraining CORE benchmark; SFT and RL on the nanochat chat suite. Accuracies (0–1) except the CORE/ChatCORE composites.

Metric	Base	SFT	RL	Δ SFT→RL
CORE (pretrain)	0.3016	—	—	—
ChatCORE	—	0.3982	0.2639	−13.4 pt
GSM8K	—	0.1099	0.1729	+6.3 pt
SpellingBee	—	0.9961	0.1836	−81.3 pt
HumanEval	—	0.1524	0.0915	−6.1 pt
MMLU	—	0.3831	0.3797	−0.3 pt
ARC-Easy	—	0.6843	0.6911	+0.7 pt
ARC-Challenge	—	0.5307	0.5307	±0.0

CORE 0.3016 exceeds the GPT-2 reference of 0.2565. The SFT checkpoint is the default; RL is a math-specialized variant.

External comparison (directional — methodologies differ)

Model	CORE	MMLU	ARC-E	ARC-C	GSM8K
WillyGPT · 1.68B · '26	0.30	38.3	68.4	53.1	11.0→17.3
GPT-2 XL · 1.5B · '19	~0.26	~26	—	~30	~0
nanochat d20 ($100) · 561M	0.22	31.5	38.8	28.1	4.6→7.6
Qwen2.5-1.5B · '24	—	59.8	79.1	53.4	68.5
SmolLM2-1.7B · '24	—	51.9	77.8	50.3	47.7

At identical parameter count WillyGPT is well ahead of GPT-2 XL; the gap to 2024 1.5B models is specific, not global (ARC-Challenge is level with Qwen2.5-1.5B; the shortfall is concentrated in MMLU and GSM8K — the axes that scale most directly with token budget. Qwen2.5 trained on ~18T tokens vs WillyGPT's 11B).

SFT vs. RL analysis (summary)

GRPO improved its single optimization target (GSM8K, +6.3 pt / +57% relative) and left the rest flat or lower — a specialization tax. A logit-level investigation locates a large part of the regression in when the model stops generating, not in the underlying skills:

• SpellingBee is masked. Raw spelling is identical between SFT and RL (23/24). RL changed termination: P(<|assistant_end|>) at the answer position goes 0.000 → 0.999. Suppressing that token for ~80 decode steps recovers ~95% accuracy, inference-only.
• HumanEval is mixed — roughly half harness/extraction artifact, half real RL mode-collapse (degenerate repetition).
• Identity robustness is thin. Both checkpoints answer "Who are you?" correctly cold; mid-conversation the RL checkpoint can confabulate a false (DeepMind/AlphaGo) origin, while SFT holds. Mitigate with a system prompt.
• Smear (falsified hypothesis) An initial weight-diff hypothesis (RL altered "smear" parameters) was falsified by a dose-response ablation — largest relative weight change ≠ functional importance.

Full mechanistic writeup and recommendations: the GitHub README.

Limitations

This is a sub-2B model trained on 11B tokens. It is not a frontier model and confidently hallucinates specific facts (e.g. inventing place names). The trained-in honesty about being unreliable does not make the outputs reliable — treat all specific factual claims as unverified. Knowledge breadth (MMLU) and multi-step math (GSM8K) are the weakest areas, as expected from the token budget.

Provenance

Both checkpoints were verified bit-for-bit (SHA-256) between the GPU node and a laptop before teardown, deserialize cleanly on CPU/MPS (189 tensors, ~1.68B params), and run locally on a 32 GB M4 MacBook Air via MPS.

Credits & license

MIT-licensed; a derivative of nanochat by Andrej Karpathy (also MIT). The nanochat framework — architecture, training scripts, tokenizer, and eval harness — is Karpathy's work. This release adds the d26 configuration, the identity layer, the GRPO stage, and the accompanying analysis. Pretraining data: NVIDIA ClimbMix. SFT data: SmolTalk, MMLU, GSM8K, plus a custom identity set.