Hugging Face's logo

ML-Intern-lab
/
nanochat

Model card Files
xet
 Community
nanochat / dev /LOG.md

Commit History

  1. tried and failed at DyT

    Andrej Karpathy
    dc54a1a

  1. bunch of ideas tried from openai/parameter-golf, all negative results for nanochat

    Andrej Karpathy
    4e1694c

  1. delete autocast, an unnecessary thorn in my side, manage dtypes directly

    Andrej Karpathy
    1076f97

  2. big, breaking change but large upside: swap previous FineWeb-EDU dataset to NVIDIA ClimbMix dataset. Requires people to download the data shards. The upside is that training GPT-2 capablity model now only takes ~2 hours, down from 2.76 hours, so this is a huge win data-wise

    Andrej Karpathy
    324e69c

  1. tune logit softcap?

    Andrej Karpathy
    aba30cb

  1. docs: fix typos in experiment log (#547)

    Dipesh Babu
    Unverified signature
    c7ba252

  1. document MoE exploration

    Andrej Karpathy
    2dffdc8

  1. report negative result on fineweb dataset

    Andrej Karpathy
    48804bf

  1. update dev log with recent

    Andrej Karpathy
    4a6e47b

  1. briefly mention batch ramp experimentation too, too weak to merge in my few attempts

    Andrej Karpathy
    e527521

  2. auto-calculate optimal batch size. the original setting of 0.5M was only optimal for d12, but d26 prefers 1M and so on

    Andrej Karpathy
    f41dd3c

  3. try and fail relu^2 -> swiglu

    Andrej Karpathy
    1144d18

  4. try and fail relu^2 -> swiglu

    Andrej Karpathy
    d63b7ab

  5. correctly reference NorMuon and fix misleading terms that i may have hastily ported over from modded-nanogpt

    Andrej Karpathy
    718e5e9

  1. quick experiments to log

    Andrej Karpathy
    d510b13

  2. add feb2 new leaderboard record from upgrading to fp8 training, +4.3% speedup to time to GPT-2

    Andrej Karpathy
    a67eba3

  1. tried muonh, appealing but didn't work out of the box

    Andrej Karpathy
    ebd4d9b

  1. revert engram, not seeing an improvement at larger scale

    Andrej Karpathy
    74554be

  2. Fix link to DeepSeek Engram paper (#470)⁠

    Sofie Van Landeghem
    Unverified signature
    d5418ea

  1. add engram-lite, add log, tune scaling laws analysis scripts

    Andrej Karpathy
    c8d93be

  1. 320 experiments just to tune the adam beta1 of x0 a little bit up from 0.8 to 0.96

    Andrej Karpathy
    85b3e95

  1. log for jan 17

    Andrej Karpathy
    d58fcd9

  1. brief update to log

    Andrej Karpathy
    1933e85

  1. also add to log about the FA3 changes

    Andrej Karpathy
    184d4c1

  2. update log with a bunch of attempts

    Andrej Karpathy
    fbf2bbe

  3. add negative result on olmo3 pretraining mix

    Andrej Karpathy
    747ed44

  1. add negative result about not allowing attention across BOS tokens. A lot more code complexity for basically no gain in performance

    Andrej Karpathy
    f92efce

  2. Big DataLoader refactor: BOS-aligned dataloaders with epoch tracking for pre/mid-training⁠

    Andrej Karpathy
    43c29dd

  3. validated that \p{N}{1,2} is the correct number of digits to group up to in the regex pattern of the GPT-4 tokenizer (2 down from 3), leading to the best val_bpb for 32K vocabs

    Andrej Karpathy
    64b48d0

  4. document my struggle with fp8 integration yesterday, it's not working like i thought it would and i suffered. one day i will return to continue the fight.

    Andrej Karpathy
    238353c

  1. record negative result on MTP

    Andrej Karpathy
    4610a83

  1. add alternating window size patterns for the GPT layers, following GPT-3. Experimented a bit and found the pattern SSSL to work well - 3 short, 1 long alternating. This is now the new default and the plots look quite a bit better on flops vs. bpb

    Andrej Karpathy
    fbc1484

  2. integrate Flash Attention 3. +9% tok_per_sec for d12 with ctx even as low as 2048 out of the box nice. also, ready to tune windows huge

    Andrej Karpathy
    2ff7d51

  3. Add learnable lambdas that gate the residual connection and a skip connection to the input embeddings, solid bump to val_bpb

    Andrej Karpathy
    aa530cd

  4. Big Muon optimizer changes inspired by latest of modded-nanogpt. Added Polar Express, Adafactor-style variance reduction, cautious weight decay, schedule weight decay linearly to ramp down to zero. Tuned optimum weight decay for multiple model sizes d8, d12, d16, d20 and found a scaling law with optimum wd \propto 1/channels^2, including it as default into code. --weight_decay of base_train is now default on and configured optimally according to all of these experiments. Solid bump to val_bpb observed as a result of these changes.

    Andrej Karpathy
    2c4473d

  1. delete grad_clip. appears to not be necessary at all. not only was it buggy because the clipping happened per gpu before grad synchronization, but it costs ~2% MFU, and it also doesn't even help. I tried deleting it a while ago and back then it did help. So I'm guessing that some hyperparameter tuning obviated the reason for it since then

    Andrej Karpathy
    061f83c