Commit History
-
clean up fragile code
Andrej Karpathy92d63d4 -
add inference benchmarking
Andrej Karpathyeb16d01 -
fix deprecated commit on torchao
Andrej Karpathyf527f76 -
nuke huggingface and its giant dependency footprint from orbit. datasets still todo
Andrej Karpathy4186540 -
delete spurious 'cute' bloat that doesn't matter to a serious research stack
Andrej Karpathy950a1dc -
remove a ton of UI bloat, we need to make nanochat small and sexy and minimal and forkable. code is ~free now, if you want UI get your favorite LLM to add it. nukes ~1000 LOC from orbit nice
Andrej Karpathyda32e1d -
Improve error handling for batch size alignment
AndrejUnverified signature6facdee -
Merge pull request #540 from svlandeg/fix/kernel
AndrejUnverified signatureUse HF's community kernel for FA3 for Ada and Ampere2fc63c9 -
Small doc fixes for consistency
AndrejUnverified signaturedbe6e2a -
fix token_bytes calculation to use raw token bytes instead of round-tripping through str, which corrupts tokens that are not valid standalone UTF-8 (e.g. bytes >= 0x80 became 3-byte U+FFFD). 193/32768 tokens were affected, so bpb of new runs will read ~0.05% higher than old runs. Thank you @ellenjxu #693
Andrej Karpathy +1: Claude Fable 5Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>2ce972a
-
rename fwe to climbmix
svlandegfbb50ad -
Merge branch 'master' into feat/small_fixes
svlandeg07e2103 -
delete the whole report thing i think it was a bad idea it just bloats the code. doing this subtracts 540 LOC
Andrej Karpathyf10bd75
-
add required -i to example chat_eval scripts
svlandegbc78a71
-
print CORE score in chat_eval
svlandegc2082b3
-
Merge branch 'master' into fix-batch-size-assertion
9f9ef95
-
Merge pull request #634 from 2bitbit/fix-docs-and-comments
AndrejUnverified signaturefix: correct minor typos in help text, README, and commentsa4ed966 -
Merge pull request #639 from mathieu-lacage/master
AndrejUnverified signatureVerified: both paths return identical data (99,842 rows), and all splits under subset='all' load cleanly.7b70f6b -
fix: use meta device in disable_fp8 to avoid VRAM spike (#616)
RoomWithOutRoof +1: RoomWithOutRoofUnverified signatureWhen swapping Float8Linear to Linear in disable_fp8 context manager, using device=fp8_module.weight.device directly allocates new tensors on GPU, causing unnecessary VRAM spike (~1GB for large models). This fix uses device='meta' to avoid physical memory allocation, then swaps in the weight tensor reference. This eliminates the unnecessary VRAM spike during evaluation phase. Fixes issue #592 Co-authored-by: RoomWithOutRoof <roomwithoutroof@sparklab.ai>47e983e
-
bump D:N ratio to 12 per recent scaling laws re-run
Andrej Karpathy1cd94d7
-
Merge branch 'master' into fix-batch-size-assertion
daba23c
-
Autoresearch round 2: smear, backout, and hyperparameter tuning
Andrej Karpathy +1: Claude Opus 4.6 (1M context)New architectural features: - Smear: mix previous token embedding into current position via learned gate, providing cheap bigram-like info (works in training + KV cache) - Backout: subtract learned fraction of mid-layer residual before logit projection to remove low-level features Hyperparameter tuning: - Muon momentum warmdown 0.97→0.90 during LR warmdown phase - Non-uniform per-layer init: resid_lambdas 1.15→1.05, x0_lambdas 0.20→0.05 - c_fc init scale 0.4x, QK norm scale 1.2, sliding window seq_len/4 - Speedrun data:params ratio reduced to 8 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>a825e63
-
MMLU main split is named auxiliary_train, not train
Mathieu Lacagea641b6c
-
fix: correct minor typos in help text, README, and comments
2bitbit2bb93b2
-
Merge branch 'master' into fix/kernel
svlandega14c399
-
Merge branch 'master' into fix-batch-size-assertion
0e5403e
-
All of these improvements were developed by Claude running autonomously over ~2 days using autoresearch. I didn't touch anything - incredible. All tuning was done on d12 but generalized easily to larger models (e.g. d24 in particular). This means we will also get a new "Time to GPT-2" Leaderboard entry, which I will push separately.
Andrej KarpathyOptimizer & schedule changes: - Increase unembedding LR 0.004 -> 0.008, weight decay 0.2 -> 0.28 - Per-group Adam betas and weight decay (instead of shared global betas) - Muon beta2 0.95 -> 0.9, momentum warmup target 0.95 -> 0.97 over 400 steps - Warmup: ratio-based -> absolute steps (default 40) - Warmdown ratio 0.5 -> 0.65, final LR fraction 0.0 -> 0.05 - Weight decay schedule: linear -> cosine decay - Polar express norm factor 1.02 -> 1.01 Architecture & init changes: - VE gate: channels 32 -> 12, scale range 2x -> 3x, init small positive - Add post-QK-norm scaling (q,k *= 1.15) for sharper attention - Embedding init std 1.0 -> 0.8, MLP c_fc init 0.5x smaller - RoPE base theta 10K -> 100K - Short attention window: seq_len/2 -> ~seq_len/3 (ceil to 128 tile) - Logit softcap 20 -> 156ed7d1d
-
Extend informative assertion message to chat_sft.py for consistency
f2899a1
-
Merge branch 'master' into fix-batch-size-assertion
28894e1
-
delete autocast, an unnecessary thorn in my side, manage dtypes directly
Andrej Karpathy1076f97 -
Ensure that inputs and targets are contiguous (#569)
Sofie Van LandeghemUnverified signature* call reshape instead of view in case the tensors are not contiguous * fix directly in data loader instead752abc8 -
big, breaking change but large upside: swap previous FineWeb-EDU dataset to NVIDIA ClimbMix dataset. Requires people to download the data shards. The upside is that training GPT-2 capablity model now only takes ~2 hours, down from 2.76 hours, so this is a huge win data-wise
Andrej Karpathy324e69c
-
Merge branch 'master' into fix-batch-size-assertion
6e9ef8f -
Restore completion-only loss masking in SFT dataloader (#582)
AnishUnverified signature* printing steps count * adding reply only loss for chat * using the mask by render_conversation function of tokeniser * undoing some changes * putting back the comment which got removed accidently, no functionality change83dccc2
-
Simplify batch size assertion message
998b8f8 -
Merge remote-tracking branch 'upstream/master' into fix-batch-size-assertion
d489a1f
-
Merge branch 'master' into fix/kernel
svlandega54691b
-
fix minor bug in fp8 application to skip tiny matmuls
Andrej Karpathybac5a35
-
tune the data mixture a bit, load optimizer by default when SFT. These were confirmed to be best settings from sweeps of sft
Andrej Karpathy1415fb7 -
a number of upgrades to SFT script to bring it up to date w.r.t. pretraining and tuning some of its kwargs based on sweeps
Andrej Karpathy77f8fb8 -
Add informative error message to batch size assertion
240a60f
-
Replace cryptic assertion with descriptive ValueError for batch size alignment
0f3b6a4
-
a number of upgrades to SFT script to bring it up to date w.r.t. pretraining and tuning some of its kwargs based on sweeps
Andrej Karpathy788dade
-
clarify that this is bf16 mfu we're talking about
Andrej Karpathy2f09686 -
delete torchao dependency, create our own exact API-matched version of Float8Linear, document it very well. for some poorly understood reason, the performance is not only ~identical but actually runs 3% faster. despite of it being significantly simpler and much less code. i don't fully understand why/how atm
Andrej Karpathye569b59
-
better comments/flow on all the hyperparameter transfer stuff, and change the WD scaling from my empirical 1/d^2 to a bit more principled version based on Tepoch. All of that theory is based on AdamW and could be suboptimal for Muon
Andrej Karpathyaeff095
-
nit: don't mutate args, create new var for total_batch_size
Andrej Karpathy2c062aa -
auto-calculate optimal batch size. the original setting of 0.5M was only optimal for d12, but d26 prefers 1M and so on
Andrej Karpathyf41dd3c
-
add fp8 training with torchao
Andrej Karpathy6079f78 -
small touchups to the eval script, re-order items etc, cosmetic
Andrej Karpathy8ebc14b