Commit History
-
fix deprecated commit on torchao
Andrej Karpathyf527f76 -
Improve error handling for batch size alignment
AndrejUnverified signature6facdee -
Merge pull request #540 from svlandeg/fix/kernel
AndrejUnverified signatureUse HF's community kernel for FA3 for Ada and Ampere2fc63c9
-
delete the whole report thing i think it was a bad idea it just bloats the code. doing this subtracts 540 LOC
Andrej Karpathyf10bd75
-
Merge branch 'master' into fix-batch-size-assertion
9f9ef95
-
fix: use meta device in disable_fp8 to avoid VRAM spike (#616)
RoomWithOutRoof +1: RoomWithOutRoofUnverified signatureWhen swapping Float8Linear to Linear in disable_fp8 context manager, using device=fp8_module.weight.device directly allocates new tensors on GPU, causing unnecessary VRAM spike (~1GB for large models). This fix uses device='meta' to avoid physical memory allocation, then swaps in the weight tensor reference. This eliminates the unnecessary VRAM spike during evaluation phase. Fixes issue #592 Co-authored-by: RoomWithOutRoof <roomwithoutroof@sparklab.ai>47e983e
-
bump D:N ratio to 12 per recent scaling laws re-run
Andrej Karpathy1cd94d7
-
Merge branch 'master' into fix-batch-size-assertion
daba23c
-
Autoresearch round 2: smear, backout, and hyperparameter tuning
Andrej Karpathy +1: Claude Opus 4.6 (1M context)New architectural features: - Smear: mix previous token embedding into current position via learned gate, providing cheap bigram-like info (works in training + KV cache) - Backout: subtract learned fraction of mid-layer residual before logit projection to remove low-level features Hyperparameter tuning: - Muon momentum warmdown 0.97→0.90 during LR warmdown phase - Non-uniform per-layer init: resid_lambdas 1.15→1.05, x0_lambdas 0.20→0.05 - c_fc init scale 0.4x, QK norm scale 1.2, sliding window seq_len/4 - Speedrun data:params ratio reduced to 8 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>a825e63
-
Merge branch 'master' into fix/kernel
svlandega14c399
-
Merge branch 'master' into fix-batch-size-assertion
0e5403e
-
All of these improvements were developed by Claude running autonomously over ~2 days using autoresearch. I didn't touch anything - incredible. All tuning was done on d12 but generalized easily to larger models (e.g. d24 in particular). This means we will also get a new "Time to GPT-2" Leaderboard entry, which I will push separately.
Andrej KarpathyOptimizer & schedule changes: - Increase unembedding LR 0.004 -> 0.008, weight decay 0.2 -> 0.28 - Per-group Adam betas and weight decay (instead of shared global betas) - Muon beta2 0.95 -> 0.9, momentum warmup target 0.95 -> 0.97 over 400 steps - Warmup: ratio-based -> absolute steps (default 40) - Warmdown ratio 0.5 -> 0.65, final LR fraction 0.0 -> 0.05 - Weight decay schedule: linear -> cosine decay - Polar express norm factor 1.02 -> 1.01 Architecture & init changes: - VE gate: channels 32 -> 12, scale range 2x -> 3x, init small positive - Add post-QK-norm scaling (q,k *= 1.15) for sharper attention - Embedding init std 1.0 -> 0.8, MLP c_fc init 0.5x smaller - RoPE base theta 10K -> 100K - Short attention window: seq_len/2 -> ~seq_len/3 (ceil to 128 tile) - Logit softcap 20 -> 156ed7d1d
-
Merge branch 'master' into fix-batch-size-assertion
28894e1
-
delete autocast, an unnecessary thorn in my side, manage dtypes directly
Andrej Karpathy1076f97 -
big, breaking change but large upside: swap previous FineWeb-EDU dataset to NVIDIA ClimbMix dataset. Requires people to download the data shards. The upside is that training GPT-2 capablity model now only takes ~2 hours, down from 2.76 hours, so this is a huge win data-wise
Andrej Karpathy324e69c
-
Simplify batch size assertion message
998b8f8 -
Merge remote-tracking branch 'upstream/master' into fix-batch-size-assertion
d489a1f
-
Merge branch 'master' into fix/kernel
svlandega54691b
-
fix minor bug in fp8 application to skip tiny matmuls
Andrej Karpathybac5a35
-
a number of upgrades to SFT script to bring it up to date w.r.t. pretraining and tuning some of its kwargs based on sweeps
Andrej Karpathy77f8fb8 -
Add informative error message to batch size assertion
240a60f
-
Replace cryptic assertion with descriptive ValueError for batch size alignment
0f3b6a4
-
a number of upgrades to SFT script to bring it up to date w.r.t. pretraining and tuning some of its kwargs based on sweeps
Andrej Karpathy788dade
-
clarify that this is bf16 mfu we're talking about
Andrej Karpathy2f09686 -
delete torchao dependency, create our own exact API-matched version of Float8Linear, document it very well. for some poorly understood reason, the performance is not only ~identical but actually runs 3% faster. despite of it being significantly simpler and much less code. i don't fully understand why/how atm
Andrej Karpathye569b59
-
better comments/flow on all the hyperparameter transfer stuff, and change the WD scaling from my empirical 1/d^2 to a bit more principled version based on Tepoch. All of that theory is based on AdamW and could be suboptimal for Muon
Andrej Karpathyaeff095
-
nit: don't mutate args, create new var for total_batch_size
Andrej Karpathy2c062aa -
auto-calculate optimal batch size. the original setting of 0.5M was only optimal for d12, but d26 prefers 1M and so on
Andrej Karpathyf41dd3c
-
add fp8 training with torchao
Andrej Karpathy6079f78
-
also update comments
svlandeg535f664 -
manually control the over-active garbage collector, save a small few minutes from a typical run
Andrej Karpathy07c4dd4
-
fix broken import sigh
Andrej Karpathy31b61d2
-
warmdown of 0.5 is slightly better:
Andrej Karpathy3c3a3d7
-
Combine AdamW and Muon into single MuonAdamW optimizer, cleaner, ty @chrisjmccormick for idea/help
Andrej Karpathy41bb2ea
-
add engram-lite, add log, tune scaling laws analysis scripts
Andrej Karpathyc8d93be
-
first version of engram following modded nanogpt style
Andrej Karpathy59e36cc
-
Merge branch 've'
Andrej Karpathya91743c -
resolve a crash for odd depths because FA3 needs head_dim % 8 == 0
Andrej Karpathycf5c9e5
-
optimal ratio is now around 4
Andrej Karpathy413e91a -
add detection of device to report more correct mfu for bf16
Andrej Karpathy2955650
-
implement flash attention 3 fallback to pytorch sdpa by touching as few lines of code as possible in main files and keeping all implementation to a single file. add tests. add helpful warning messages for the user.
Andrej Karpathy8203efa
-
oops legacy spurious line now
Andrej Karpathybdcc030 -
cleanly separate cpu and gpu sections
Andrej Karpathy255f8b9
-
fix buggy midtrain and update all kwargs to be idiomatic. that is, argparse uses dashes variables use underscores. the underscores are just a remnant of the previous Configurator object. This is the right way
Andrej Karpathy7312ec9 -
Big DataLoader refactor: BOS-aligned dataloaders with epoch tracking for pre/mid-training
Andrej KarpathyThe new DataLoader ensures that every token sequence in train/val batches has a BOS token at the beginning. Therefore, no token streams start abruptly in the middle of a document, which could be confusing for the model. Note that this changes the loss scale because there are fewer confusing tokens in the train/val batches. The main downside is that we now waste about 35% of tokens due to cropping. This is ok because we have a lot of data. See dev/LOG.md entry for this change for a lot more information.43c29dd
-
allow base_loss to report the loss of any arbitrary huggingface model similar to base_eval. had to change dataloader to be a lot better and just take tokenizer, not load the nanochat one. much better this way anyway
Andrej Karpathy21608ec
-
oops actually make SSSL the default window pattern
Andrej Karpathyb33e394 -
add alternating window size patterns for the GPT layers, following GPT-3. Experimented a bit and found the pattern SSSL to work well - 3 short, 1 long alternating. This is now the new default and the plots look quite a bit better on flops vs. bpb
Andrej Karpathyfbc1484 -
Add learnable lambdas that gate the residual connection and a skip connection to the input embeddings, solid bump to val_bpb
Andrej Karpathyaa530cd -
Big Muon optimizer changes inspired by latest of modded-nanogpt. Added Polar Express, Adafactor-style variance reduction, cautious weight decay, schedule weight decay linearly to ramp down to zero. Tuned optimum weight decay for multiple model sizes d8, d12, d16, d20 and found a scaling law with optimum wd \propto 1/channels^2, including it as default into code. --weight_decay of base_train is now default on and configured optimally according to all of these experiments. Solid bump to val_bpb observed as a result of these changes.
Andrej Karpathy2c4473d