nanochat / dev

Commit History

Jul 3

19:22
delete datasets dependency bye

Andrej Karpathy

a5a3a33
18:59
nuke huggingface and its giant dependency footprint from orbit. datasets still todo

Andrej Karpathy

4186540
18:48
delete spurious 'cute' bloat that doesn't matter to a serious research stack

Andrej Karpathy

950a1dc
18:06
remove a ton of UI bloat, we need to make nanochat small and sexy and minimal and forkable. code is ~free now, if you want UI get your favorite LLM to add it. nukes ~1000 LOC from orbit nice

Andrej Karpathy

da32e1d

Jul 2

22:45
fix link

svlandeg

9ec05d9

May 5

03:17
tried and failed at DyT

Andrej Karpathy

dc54a1a

Mar 25

21:40
Merge pull request #595 from svlandeg/fix/typo⁠

Andrej

Unverified signature
7808dc7

Mar 24

22:13
bunch of ideas tried from openai/parameter-golf, all negative results for nanochat

Andrej Karpathy

4e1694c

Mar 18

16:01
Merge branch 'master' into fix/typo

svlandeg

dfe7d39

Mar 17

16:55
fix scaling laws scripts after the bigram embeddings were removed

Andrej Karpathy

5019acc

Mar 15

21:29
~1.5h :-)

svlandeg

51f42a4
21:18
Merge branch 'master' into fix/typo

svlandeg

02e865c

Mar 14

17:15
submit new time to GPT-2 leaderboard entry: 99 minutes

Andrej Karpathy

1b1cc3c

Mar 13

12:56
Merge branch 'master' into fix/typo

svlandeg

6405b26
12:46
we only need to wait 2h now!

svlandeg

1052d25

Mar 10

06:26
new leaderboard entry coming from improvements of autoresearch round 1, time to gpt-2 from 2.02 hours to 1.80 hours

Andrej Karpathy

f068604

Mar 4

23:55
delete autocast, an unnecessary thorn in my side, manage dtypes directly

Andrej Karpathy

1076f97
20:02
Document new Leaderboard entry congrats @ddudek for pointing out ClimbMix, time to GPT-2 is now 2.01 hours, down from 2.76 previously

Andrej Karpathy

4b40774
19:47
big, breaking change but large upside: swap previous FineWeb-EDU dataset to NVIDIA ClimbMix dataset. Requires people to download the data shards. The upside is that training GPT-2 capablity model now only takes ~2 hours, down from 2.76 hours, so this is a huge win data-wise

Andrej Karpathy

324e69c

Mar 3

17:24
document the legacy fineweb100b dataset and the new climbmix400b dataset

Andrej Karpathy

b07604e

Mar 2

18:19
tune logit softcap?

Andrej Karpathy

aba30cb

Feb 20

16:03
docs: fix typos in experiment log (#547)

Dipesh Babu

Unverified signature
c7ba252

Feb 19

02:53
document MoE exploration

Andrej Karpathy

2dffdc8

Feb 18

23:45
report negative result on fineweb dataset

Andrej Karpathy

48804bf

Feb 17

15:44
update dev log with recent

Andrej Karpathy

4a6e47b

Feb 5

22:21
briefly mention batch ramp experimentation too, too weak to merge in my few attempts

Andrej Karpathy

e527521
20:11
new leaderboard record via new auto-calculated optimal batch size. for d26 it is 1M, up from 0.5M that was default earlier

Andrej Karpathy

5fdd5cd
19:40
auto-calculate optimal batch size. the original setting of 0.5M was only optimal for d12, but d26 prefers 1M and so on

Andrej Karpathy

f41dd3c
18:12
Typo fixes (#480)⁠

Sofie Van Landeghem

Unverified signature
012da1a
02:42
try and fail relu^2 -> swiglu

Andrej Karpathy

1144d18
02:41
try and fail relu^2 -> swiglu

Andrej Karpathy

d63b7ab
01:39
correctly reference NorMuon and fix misleading terms that i may have hastily ported over from modded-nanogpt

Andrej Karpathy

718e5e9

Feb 3

23:21
quick experiments to log

Andrej Karpathy

d510b13
21:06
oops forgot to attach leaderboard file too

Andrej Karpathy

16b8ac7
20:54
add feb2 new leaderboard record from upgrading to fp8 training, +4.3% speedup to time to GPT-2

Andrej Karpathy

a67eba3

Feb 2

01:45
tune the synthetic data generation script. delete the king andrej stuff lol. also, upgrade to gemini 3

Andrej Karpathy

230d6cf

Jan 31

19:12
nuke midtraining from orbit, it's not as needed now that we have a BOS-aligned dataloader. Also change the README a lot. midtrianing is not yet fully properly erased across the board, but good enough for step 1

Andrej Karpathy

1ddaad1

Jan 29

19:01
tried muonh, appealing but didn't work out of the box

Andrej Karpathy

ebd4d9b

Jan 28

20:07
revert engram, not seeing an improvement at larger scale

Andrej Karpathy

74554be
16:31
Fix link to DeepSeek Engram paper (#470)⁠

Sofie Van Landeghem

Unverified signature
d5418ea

Jan 27

22:31
add engram-lite, add log, tune scaling laws analysis scripts

Andrej Karpathy

c8d93be

Jan 25

00:03
320 experiments just to tune the adam beta1 of x0 a little bit up from 0.8 to 0.96

Andrej Karpathy

85b3e95

Jan 18

15:27
something i've wanted to do for a while - move all .sh runs to their own directory so they don't pollute root dir

Andrej Karpathy

63bb583
03:01
log for jan 17

Andrej Karpathy

d58fcd9

Jan 17

20:27
update the CPU/MPS script to give reasonable results. The model can at least answer that Paris is the capital of France and knows that the sky is blue, for about 40 minutes of training on my macbook. Also fixed a bug that existed due to KVCache bfloat16 dtype assumption

karpathy

f9a7e0f
02:59
Fix WANDB_RUN parameter passing in runcpu.sh (#407)⁠

ykirpichev +1: svlandeg

Unverified signature
77a4690
00:25
brief update to log

Andrej Karpathy

1933e85

Jan 16

18:25
also add to log about the FA3 changes

Andrej Karpathy

184d4c1
02:21
update log with a bunch of attempts

Andrej Karpathy

fbf2bbe
00:43
add negative result on olmo3 pretraining mix

Andrej Karpathy

747ed44

Commit History

Jul 3

delete datasets dependency bye

nuke huggingface and its giant dependency footprint from orbit. datasets still todo

delete spurious 'cute' bloat that doesn't matter to a serious research stack

remove a ton of UI bloat, we need to make nanochat small and sexy and minimal and forkable. code is ~free now, if you want UI get your favorite LLM to add it. nukes ~1000 LOC from orbit nice

Jul 2

fix link

May 5

tried and failed at DyT

Mar 25

Merge pull request #595 from svlandeg/fix/typo⁠

Mar 24

bunch of ideas tried from openai/parameter-golf, all negative results for nanochat

Mar 18

Merge branch 'master' into fix/typo

Mar 17

fix scaling laws scripts after the bigram embeddings were removed

Mar 15

~1.5h :-)

Merge branch 'master' into fix/typo

Mar 14

submit new time to GPT-2 leaderboard entry: 99 minutes

Mar 13

Merge branch 'master' into fix/typo

we only need to wait 2h now!

Mar 10

new leaderboard entry coming from improvements of autoresearch round 1, time to gpt-2 from 2.02 hours to 1.80 hours

Mar 4

delete autocast, an unnecessary thorn in my side, manage dtypes directly

Document new Leaderboard entry congrats @ddudek for pointing out ClimbMix, time to GPT-2 is now 2.01 hours, down from 2.76 previously

big, breaking change but large upside: swap previous FineWeb-EDU dataset to NVIDIA ClimbMix dataset. Requires people to download the data shards. The upside is that training GPT-2 capablity model now only takes ~2 hours, down from 2.76 hours, so this is a huge win data-wise

Mar 3

document the legacy fineweb100b dataset and the new climbmix400b dataset

Mar 2

tune logit softcap?

Feb 20

docs: fix typos in experiment log (#547)

Feb 19

document MoE exploration

Feb 18

report negative result on fineweb dataset

Feb 17

update dev log with recent

Feb 5

briefly mention batch ramp experimentation too, too weak to merge in my few attempts

new leaderboard record via new auto-calculated optimal batch size. for d26 it is 1M, up from 0.5M that was default earlier

auto-calculate optimal batch size. the original setting of 0.5M was only optimal for d12, but d26 prefers 1M and so on

Typo fixes (#480)⁠

try and fail relu^2 -> swiglu

try and fail relu^2 -> swiglu

correctly reference NorMuon and fix misleading terms that i may have hastily ported over from modded-nanogpt

Feb 3

quick experiments to log

oops forgot to attach leaderboard file too

add feb2 new leaderboard record from upgrading to fp8 training, +4.3% speedup to time to GPT-2

Feb 2

tune the synthetic data generation script. delete the king andrej stuff lol. also, upgrade to gemini 3

Jan 31

nuke midtraining from orbit, it's not as needed now that we have a BOS-aligned dataloader. Also change the README a lot. midtrianing is not yet fully properly erased across the board, but good enough for step 1

Jan 29

tried muonh, appealing but didn't work out of the box

Jan 28

revert engram, not seeing an improvement at larger scale

Fix link to DeepSeek Engram paper (#470)⁠

Jan 27

add engram-lite, add log, tune scaling laws analysis scripts

Jan 25

320 experiments just to tune the adam beta1 of x0 a little bit up from 0.8 to 0.96

Jan 18

something i've wanted to do for a while - move all .sh runs to their own directory so they don't pollute root dir

log for jan 17

Jan 17

update the CPU/MPS script to give reasonable results. The model can at least answer that Paris is the capital of France and knows that the sky is blue, for about 40 minutes of training on my macbook. Also fixed a bug that existed due to KVCache bfloat16 dtype assumption

Fix WANDB_RUN parameter passing in runcpu.sh (#407)⁠

brief update to log

Jan 16

also add to log about the FA3 changes

update log with a bunch of attempts

add negative result on olmo3 pretraining mix