JohnEnev/modern-llm-v3-base

modern-llm V3 (base (pretrained))

A 672M-parameter decoder-only transformer, built and trained from scratch in raw PyTorch (no Hugging Face Trainer). This is the base (pretrained) checkpoint.

Part of a four-part series on building LLMs from scratch. Writeup: https://john463212.substack.com Code: https://github.com/JohnEnev/modern-llm

Architecture

Parameters	671,885,088
d_model	1536
Layers	24
Query heads	12
KV heads	3 (GQA 4:1)
Head dim	128
Attention	XSA (Exclusive Self-Attention)
QK-Norm	Yes
Position	RoPE
Norm	RMSNorm, pre-norm
Feed-forward	SwiGLU
Context length	1024
Vocab	50,304 (tiktoken GPT-2 BPE, padded)
Optimizer	AdamW + Muon
Tied embeddings	Yes

Training

• Data: FineWeb-Edu + Python + math, 30B tokens (~45 tokens/param)
• Hardware: 2x B200 (DDP)
• Base validation loss: 2.5885 (EMA), FineWeb-Edu

How to load

The weights ship with the model code (modeling/) so the repo is self-contained.

import torch, importlib.util, sys
from huggingface_hub import snapshot_download
from safetensors.torch import load_file

local = snapshot_download("JohnEnev/modern-llm-v3-base")
sys.path.insert(0, local)
from modeling.gpt import GPT, GPTConfig   # noqa: E402

# V3 config. These are NOT the GPTConfig defaults, so pass them explicitly.
config = GPTConfig(vocab_size=50304, d_model=1536, n_layers=24, n_heads=12, n_kv_heads=3, max_seq_len=1024, use_qk_norm=True, use_diff_attn=False, use_xsa=True, tie_weights=True)
model = GPT(config)

state = load_file(f"{local}/model.safetensors")
# strict=False because lm_head.weight is tied to the embedding and not stored.
model.load_state_dict(state, strict=False)
model.eval()

Related checkpoints

• modern-llm-v3-base — pretrained base
• modern-llm-v3-sft — instruction-tuned
• modern-llm-v3-grpo — further trained with GRPO

Note

Per the series' central finding, the SFT checkpoint is the recommended assistant: GRPO improved the arithmetic training curriculum but degraded general capability. See the writeup for the full comparison.