README.md

---
license: mit
language:
- en
pipeline_tag: text-generation
tags:
- nanochat
- gpt
- from-scratch
- chat
- reinforcement-learning
- grpo
- mechanistic-interpretability
- llm
datasets:
- nvidia/Nemotron-ClimbMix
---

# WillyGPT

A 1.68B-parameter chat model trained from scratch — pretraining, supervised finetuning (SFT), and GRPO reinforcement learning — on the [nanochat](https://github.com/karpathy/nanochat) recipe (MIT). Built & trained by William Yates on a single 8×H100 node.

Base pretraining clears the GPT-2 CORE reference (0.3016 vs 0.2565). Two checkpoints are released: an **SFT generalist (default)** and an **RL math-specialist**.

- **Full recipe, training scripts, and reproduction:** [github.com/williamyates/WillyGPT](https://github.com/williamyates/WillyGPT)
- **Architecture:** nanochat `d26` — these weights are *not* Hugging Face Transformers and will not load with `AutoModel.from_pretrained`. Run them with the nanochat code (below).

## Files in this repo

| Path | Checkpoint | Use |
|---|---|---|
| `chatsft_checkpoints/d26/model_000483.pt` | **SFT** (default) | Best all-rounder; chat, identity, spelling |
| `chatrl_checkpoints/d26/model_000466.pt` | **RL** | GSM8K-specialized; see decoding note below |
| `tokenizer/` | `tokenizer.pkl`, `token_bytes.pt` | 32,768-token BPE |

## Download & run

```bash
# 1. get nanochat
git clone https://github.com/karpathy/nanochat.git && cd nanochat
uv sync --extra gpu        # or: uv sync --extra cpu   (Mac/MPS, slow but works)

# 2. download these weights into nanochat's cache (structure is preserved)
export NANOCHAT_BASE_DIR=$PWD
hf download williamyates/WillyGPT --local-dir "$NANOCHAT_BASE_DIR/.cache/nanochat"

# 3. chat
python -m scripts.chat_cli -i sft -p "Who are you?"       # SFT: generalist (recommended)
python -m scripts.chat_cli -i rl  -p "What is 17 * 23?"   # RL: math specialist
python -m scripts.chat_web                                 # browser UI
```

A 1.68B model runs (slowly) on Apple Silicon via MPS, or on any single modern GPU.

**Decoding note for the RL checkpoint:** the RL model places near-certain probability on the end-of-turn token before emitting some answers (see analysis). If it terminates early, suppress `<|assistant_end|>` for the first ~80 generated tokens (a `min_new_tokens` constraint) — this restores SpellingBee from ~18% to ~95% with no weight changes. The SFT checkpoint does not need this.

## Model specification

| | |
|---|---|
| Architecture | GPT-style decoder transformer (nanochat `d26`) |
| Parameters | 1.68B (189 weight tensors) |
| Layers / heads | 26 layers · 13 attention heads · 13 KV heads · head dim 128 |
| Model dimension | 1664 |
| Context length | 2048 |
| Attention | Sliding-window (SSSL), Flash Attention 3 |
| Precision | fp8 training (Hopper / H100) |
| Tokenizer | 32,768-token BPE, GPT-4 style |
| Training data | NVIDIA ClimbMix (pretrain); SmolTalk + MMLU + GSM8K + custom identity set (SFT); GSM8K (RL) |

## Evaluation

Base on the pretraining CORE benchmark; SFT and RL on the nanochat chat suite. Accuracies (0–1) except the CORE/ChatCORE composites.

| Metric | Base | SFT | RL | Δ SFT→RL |
|---|---:|---:|---:|---:|
| CORE (pretrain) | 0.3016 | — | — | — |
| ChatCORE | — | 0.3982 | 0.2639 | −13.4 pt |
| GSM8K | — | 0.1099 | 0.1729 | +6.3 pt |
| SpellingBee | — | 0.9961 | 0.1836 | −81.3 pt |
| HumanEval | — | 0.1524 | 0.0915 | −6.1 pt |
| MMLU | — | 0.3831 | 0.3797 | −0.3 pt |
| ARC-Easy | — | 0.6843 | 0.6911 | +0.7 pt |
| ARC-Challenge | — | 0.5307 | 0.5307 | ±0.0 |

CORE 0.3016 exceeds the GPT-2 reference of 0.2565. **The SFT checkpoint is the default**; RL is a math-specialized variant.

### External comparison (directional — methodologies differ)

| Model | CORE | MMLU | ARC-E | ARC-C | GSM8K |
|---|---:|---:|---:|---:|---:|
| WillyGPT · 1.68B · '26 | 0.30 | 38.3 | 68.4 | 53.1 | 11.0→17.3 |
| GPT-2 XL · 1.5B · '19 | ~0.26 | ~26 | — | ~30 | ~0 |
| nanochat d20 ($100) · 561M | 0.22 | 31.5 | 38.8 | 28.1 | 4.6→7.6 |
| Qwen2.5-1.5B · '24 | — | 59.8 | 79.1 | 53.4 | 68.5 |
| SmolLM2-1.7B · '24 | — | 51.9 | 77.8 | 50.3 | 47.7 |

At identical parameter count WillyGPT is well ahead of GPT-2 XL; the gap to 2024 1.5B models is specific, not global (ARC-Challenge is level with Qwen2.5-1.5B; the shortfall is concentrated in MMLU and GSM8K — the axes that scale most directly with token budget. Qwen2.5 trained on ~18T tokens vs WillyGPT's 11B).

## SFT vs. RL analysis (summary)

GRPO improved its single optimization target (GSM8K, +6.3 pt / +57% relative) and left the rest flat or lower — a specialization tax. A logit-level investigation locates a large part of the regression in *when the model stops generating*, not in the underlying skills:

- **SpellingBee is masked.** Raw spelling is identical between SFT and RL (23/24). RL changed termination: `P(<|assistant_end|>)` at the answer position goes 0.000 → 0.999. Suppressing that token for ~80 decode steps recovers ~95% accuracy, inference-only.
- **HumanEval is mixed** — roughly half harness/extraction artifact, half real RL mode-collapse (degenerate repetition).
- **Identity robustness is thin.** Both checkpoints answer "Who are you?" correctly cold; mid-conversation the RL checkpoint can confabulate a false (DeepMind/AlphaGo) origin, while SFT holds. Mitigate with a system prompt.
- **Smear (falsified hypothesis)** An initial weight-diff hypothesis (RL altered "smear" parameters) was falsified by a dose-response ablation — largest relative weight change ≠ functional importance.

Full mechanistic writeup and recommendations: [the GitHub README](https://github.com/williamyates/WillyGPT).

## Limitations

This is a sub-2B model trained on 11B tokens. It is **not a frontier model** and **confidently hallucinates specific facts** (e.g. inventing place names). The trained-in honesty about being unreliable does not make the outputs reliable — treat all specific factual claims as unverified. Knowledge breadth (MMLU) and multi-step math (GSM8K) are the weakest areas, as expected from the token budget.

## Provenance

Both checkpoints were verified bit-for-bit (SHA-256) between the GPU node and a laptop before teardown, deserialize cleanly on CPU/MPS (189 tensors, ~1.68B params), and run locally on a 32 GB M4 MacBook Air via MPS.

## Credits & license

MIT-licensed; a derivative of [nanochat](https://github.com/karpathy/nanochat) by Andrej Karpathy (also MIT). The nanochat framework — architecture, training scripts, tokenizer, and eval harness — is Karpathy's work. This release adds the d26 configuration, the identity layer, the GRPO stage, and the accompanying analysis. Pretraining data: NVIDIA ClimbMix. SFT data: SmolTalk, MMLU, GSM8K, plus a custom identity set.