qgate-checkpoints / results /logs /train_dot_elementwise_seed1337.log
kolbrian's picture
results: phase 7-19d CSVs and summaries
605ff8d verified
Raw
History Blame Contribute Delete
6.54 kB
tokens per iteration will be: 4,096
defaulting to vocab_size of GPT-2 to 50304 (50257 rounded up for efficiency)
number of parameters: 123.59M
C:\Users\brian\documents\nanogpt\train.py:199: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
scaler = torch.cuda.amp.GradScaler(enabled=(dtype == 'float16'))
num decayed parameter tensors: 50, with 123,961,344 parameters
num non-decayed parameter tensors: 25, with 19,200 parameters
using fused AdamW: True
step 0: train loss 10.9795, val loss 10.9789
iter 0: loss 11.0007, time 18571.63ms, mfu -100.00%
step 250: train loss 6.9364, val loss 6.9696
saving checkpoint to out-openwebtext_dot_gate_shared
iter 250: loss 6.7257, time 21333.42ms, mfu 0.05%
step 500: train loss 6.4777, val loss 6.5451
saving checkpoint to out-openwebtext_dot_gate_shared
iter 500: loss 6.2618, time 21296.22ms, mfu 0.05%
step 750: train loss 6.3006, val loss 6.3182
saving checkpoint to out-openwebtext_dot_gate_shared
iter 750: loss 6.0881, time 21129.88ms, mfu 0.05%
step 1000: train loss 6.1729, val loss 6.1461
saving checkpoint to out-openwebtext_dot_gate_shared
iter 1000: loss 6.1423, time 21201.95ms, mfu 0.05%
step 1250: train loss 6.0707, val loss 6.0631
saving checkpoint to out-openwebtext_dot_gate_shared
iter 1250: loss 6.1124, time 21161.51ms, mfu 0.05%
step 1500: train loss 5.9830, val loss 5.9110
saving checkpoint to out-openwebtext_dot_gate_shared
iter 1500: loss 5.7030, time 21203.03ms, mfu 0.05%
step 1750: train loss 5.9064, val loss 5.8710
saving checkpoint to out-openwebtext_dot_gate_shared
iter 1750: loss 6.0679, time 21203.47ms, mfu 0.05%
step 2000: train loss 5.8000, val loss 5.7732
saving checkpoint to out-openwebtext_dot_gate_shared
iter 2000: loss 5.8516, time 21340.31ms, mfu 0.05%
step 2250: train loss 5.6919, val loss 5.6962
saving checkpoint to out-openwebtext_dot_gate_shared
iter 2250: loss 5.9000, time 21570.12ms, mfu 0.05%
step 2500: train loss 5.5901, val loss 5.5711
saving checkpoint to out-openwebtext_dot_gate_shared
iter 2500: loss 6.0801, time 21365.91ms, mfu 0.05%
step 2750: train loss 5.5034, val loss 5.5007
saving checkpoint to out-openwebtext_dot_gate_shared
iter 2750: loss 5.4930, time 21499.41ms, mfu 0.05%
step 3000: train loss 5.4336, val loss 5.4487
saving checkpoint to out-openwebtext_dot_gate_shared
iter 3000: loss 5.5091, time 21379.19ms, mfu 0.05%
step 3250: train loss 5.3468, val loss 5.3518
saving checkpoint to out-openwebtext_dot_gate_shared
iter 3250: loss 5.3241, time 21218.44ms, mfu 0.05%
step 3500: train loss 5.3555, val loss 5.3138
saving checkpoint to out-openwebtext_dot_gate_shared
iter 3500: loss 5.2600, time 21292.16ms, mfu 0.05%
step 3750: train loss 5.2604, val loss 5.2430
saving checkpoint to out-openwebtext_dot_gate_shared
iter 3750: loss 5.5489, time 21258.47ms, mfu 0.05%
step 4000: train loss 5.2137, val loss 5.2132
saving checkpoint to out-openwebtext_dot_gate_shared
iter 4000: loss 5.2934, time 21431.59ms, mfu 0.05%
step 4250: train loss 5.1521, val loss 5.1660
saving checkpoint to out-openwebtext_dot_gate_shared
iter 4250: loss 4.7351, time 21338.17ms, mfu 0.05%
step 4500: train loss 5.1111, val loss 5.1025
saving checkpoint to out-openwebtext_dot_gate_shared
iter 4500: loss 4.8370, time 21290.82ms, mfu 0.05%
step 4750: train loss 5.0548, val loss 5.0648
saving checkpoint to out-openwebtext_dot_gate_shared
iter 4750: loss 5.1715, time 21293.42ms, mfu 0.05%
step 5000: train loss 4.9836, val loss 5.0194
saving checkpoint to out-openwebtext_dot_gate_shared
iter 5000: loss 5.1628, time 21309.56ms, mfu 0.05%
step 5250: train loss 4.9793, val loss 4.9614
saving checkpoint to out-openwebtext_dot_gate_shared
iter 5250: loss 4.6227, time 21484.56ms, mfu 0.05%
step 5500: train loss 4.9490, val loss 4.9103
saving checkpoint to out-openwebtext_dot_gate_shared
iter 5500: loss 5.1721, time 21305.27ms, mfu 0.05%
step 5750: train loss 4.8919, val loss 4.8484
saving checkpoint to out-openwebtext_dot_gate_shared
iter 5750: loss 5.1753, time 21244.32ms, mfu 0.05%
step 6000: train loss 4.7978, val loss 4.8005
saving checkpoint to out-openwebtext_dot_gate_shared
iter 6000: loss 4.7117, time 21269.19ms, mfu 0.05%
step 6250: train loss 4.7806, val loss 4.8079
iter 6250: loss 4.6017, time 18312.38ms, mfu 0.05%
step 6500: train loss 4.7362, val loss 4.7719
saving checkpoint to out-openwebtext_dot_gate_shared
iter 6500: loss 4.9249, time 21388.15ms, mfu 0.05%
step 6750: train loss 4.7034, val loss 4.7044
saving checkpoint to out-openwebtext_dot_gate_shared
iter 6750: loss 4.8880, time 21301.28ms, mfu 0.05%
step 7000: train loss 4.6659, val loss 4.6621
saving checkpoint to out-openwebtext_dot_gate_shared
iter 7000: loss 4.5558, time 21242.25ms, mfu 0.05%
step 7250: train loss 4.6186, val loss 4.6468
saving checkpoint to out-openwebtext_dot_gate_shared
iter 7250: loss 4.7227, time 21070.07ms, mfu 0.05%
step 7500: train loss 4.6276, val loss 4.5832
saving checkpoint to out-openwebtext_dot_gate_shared
iter 7500: loss 4.5001, time 21465.43ms, mfu 0.05%
step 7750: train loss 4.5771, val loss 4.5701
saving checkpoint to out-openwebtext_dot_gate_shared
iter 7750: loss 4.7642, time 21323.07ms, mfu 0.05%
step 8000: train loss 4.5366, val loss 4.5635
saving checkpoint to out-openwebtext_dot_gate_shared
iter 8000: loss 4.5939, time 21334.04ms, mfu 0.05%
step 8250: train loss 4.5186, val loss 4.5473
saving checkpoint to out-openwebtext_dot_gate_shared
iter 8250: loss 4.3011, time 21412.39ms, mfu 0.05%
step 8500: train loss 4.5114, val loss 4.5233
saving checkpoint to out-openwebtext_dot_gate_shared
iter 8500: loss 4.2676, time 21286.35ms, mfu 0.05%
step 8750: train loss 4.4759, val loss 4.4773
saving checkpoint to out-openwebtext_dot_gate_shared
iter 8750: loss 4.4944, time 21323.93ms, mfu 0.05%
step 9000: train loss 4.4907, val loss 4.4935
iter 9000: loss 4.5788, time 18368.15ms, mfu 0.05%
step 9250: train loss 4.4502, val loss 4.4793
iter 9250: loss 4.5374, time 18283.96ms, mfu 0.05%
step 9500: train loss 4.4581, val loss 4.4678
saving checkpoint to out-openwebtext_dot_gate_shared
iter 9500: loss 4.4842, time 21401.30ms, mfu 0.05%
step 9750: train loss 4.4615, val loss 4.4374
saving checkpoint to out-openwebtext_dot_gate_shared
iter 9750: loss 4.3793, time 21161.72ms, mfu 0.05%
step 10000: train loss 4.4544, val loss 4.4806
iter 10000: loss 4.4831, time 18248.43ms, mfu 0.05%