| tokens per iteration will be: 4,096 | |
| defaulting to vocab_size of GPT-2 to 50304 (50257 rounded up for efficiency) | |
| number of parameters: 123.59M | |
| C:\Users\brian\documents\nanogpt\train.py:199: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead. | |
| scaler = torch.cuda.amp.GradScaler(enabled=(dtype == 'float16')) | |
| num decayed parameter tensors: 50, with 123,961,344 parameters | |
| num non-decayed parameter tensors: 25, with 19,200 parameters | |
| using fused AdamW: True | |
| step 0: train loss 10.9795, val loss 10.9789 | |
| iter 0: loss 11.0007, time 18571.63ms, mfu -100.00% | |
| step 250: train loss 6.9364, val loss 6.9696 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 250: loss 6.7257, time 21333.42ms, mfu 0.05% | |
| step 500: train loss 6.4777, val loss 6.5451 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 500: loss 6.2618, time 21296.22ms, mfu 0.05% | |
| step 750: train loss 6.3006, val loss 6.3182 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 750: loss 6.0881, time 21129.88ms, mfu 0.05% | |
| step 1000: train loss 6.1729, val loss 6.1461 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 1000: loss 6.1423, time 21201.95ms, mfu 0.05% | |
| step 1250: train loss 6.0707, val loss 6.0631 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 1250: loss 6.1124, time 21161.51ms, mfu 0.05% | |
| step 1500: train loss 5.9830, val loss 5.9110 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 1500: loss 5.7030, time 21203.03ms, mfu 0.05% | |
| step 1750: train loss 5.9064, val loss 5.8710 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 1750: loss 6.0679, time 21203.47ms, mfu 0.05% | |
| step 2000: train loss 5.8000, val loss 5.7732 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 2000: loss 5.8516, time 21340.31ms, mfu 0.05% | |
| step 2250: train loss 5.6919, val loss 5.6962 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 2250: loss 5.9000, time 21570.12ms, mfu 0.05% | |
| step 2500: train loss 5.5901, val loss 5.5711 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 2500: loss 6.0801, time 21365.91ms, mfu 0.05% | |
| step 2750: train loss 5.5034, val loss 5.5007 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 2750: loss 5.4930, time 21499.41ms, mfu 0.05% | |
| step 3000: train loss 5.4336, val loss 5.4487 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 3000: loss 5.5091, time 21379.19ms, mfu 0.05% | |
| step 3250: train loss 5.3468, val loss 5.3518 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 3250: loss 5.3241, time 21218.44ms, mfu 0.05% | |
| step 3500: train loss 5.3555, val loss 5.3138 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 3500: loss 5.2600, time 21292.16ms, mfu 0.05% | |
| step 3750: train loss 5.2604, val loss 5.2430 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 3750: loss 5.5489, time 21258.47ms, mfu 0.05% | |
| step 4000: train loss 5.2137, val loss 5.2132 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 4000: loss 5.2934, time 21431.59ms, mfu 0.05% | |
| step 4250: train loss 5.1521, val loss 5.1660 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 4250: loss 4.7351, time 21338.17ms, mfu 0.05% | |
| step 4500: train loss 5.1111, val loss 5.1025 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 4500: loss 4.8370, time 21290.82ms, mfu 0.05% | |
| step 4750: train loss 5.0548, val loss 5.0648 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 4750: loss 5.1715, time 21293.42ms, mfu 0.05% | |
| step 5000: train loss 4.9836, val loss 5.0194 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 5000: loss 5.1628, time 21309.56ms, mfu 0.05% | |
| step 5250: train loss 4.9793, val loss 4.9614 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 5250: loss 4.6227, time 21484.56ms, mfu 0.05% | |
| step 5500: train loss 4.9490, val loss 4.9103 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 5500: loss 5.1721, time 21305.27ms, mfu 0.05% | |
| step 5750: train loss 4.8919, val loss 4.8484 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 5750: loss 5.1753, time 21244.32ms, mfu 0.05% | |
| step 6000: train loss 4.7978, val loss 4.8005 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 6000: loss 4.7117, time 21269.19ms, mfu 0.05% | |
| step 6250: train loss 4.7806, val loss 4.8079 | |
| iter 6250: loss 4.6017, time 18312.38ms, mfu 0.05% | |
| step 6500: train loss 4.7362, val loss 4.7719 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 6500: loss 4.9249, time 21388.15ms, mfu 0.05% | |
| step 6750: train loss 4.7034, val loss 4.7044 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 6750: loss 4.8880, time 21301.28ms, mfu 0.05% | |
| step 7000: train loss 4.6659, val loss 4.6621 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 7000: loss 4.5558, time 21242.25ms, mfu 0.05% | |
| step 7250: train loss 4.6186, val loss 4.6468 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 7250: loss 4.7227, time 21070.07ms, mfu 0.05% | |
| step 7500: train loss 4.6276, val loss 4.5832 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 7500: loss 4.5001, time 21465.43ms, mfu 0.05% | |
| step 7750: train loss 4.5771, val loss 4.5701 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 7750: loss 4.7642, time 21323.07ms, mfu 0.05% | |
| step 8000: train loss 4.5366, val loss 4.5635 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 8000: loss 4.5939, time 21334.04ms, mfu 0.05% | |
| step 8250: train loss 4.5186, val loss 4.5473 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 8250: loss 4.3011, time 21412.39ms, mfu 0.05% | |
| step 8500: train loss 4.5114, val loss 4.5233 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 8500: loss 4.2676, time 21286.35ms, mfu 0.05% | |
| step 8750: train loss 4.4759, val loss 4.4773 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 8750: loss 4.4944, time 21323.93ms, mfu 0.05% | |
| step 9000: train loss 4.4907, val loss 4.4935 | |
| iter 9000: loss 4.5788, time 18368.15ms, mfu 0.05% | |
| step 9250: train loss 4.4502, val loss 4.4793 | |
| iter 9250: loss 4.5374, time 18283.96ms, mfu 0.05% | |
| step 9500: train loss 4.4581, val loss 4.4678 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 9500: loss 4.4842, time 21401.30ms, mfu 0.05% | |
| step 9750: train loss 4.4615, val loss 4.4374 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 9750: loss 4.3793, time 21161.72ms, mfu 0.05% | |
| step 10000: train loss 4.4544, val loss 4.4806 | |
| iter 10000: loss 4.4831, time 18248.43ms, mfu 0.05% | |