| tokens per iteration will be: 4,096 | |
| defaulting to vocab_size of GPT-2 to 50304 (50257 rounded up for efficiency) | |
| number of parameters: 123.59M | |
| C:\Users\brian\documents\nanogpt\train.py:199: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead. | |
| scaler = torch.cuda.amp.GradScaler(enabled=(dtype == 'float16')) | |
| num decayed parameter tensors: 50, with 123,961,344 parameters | |
| num non-decayed parameter tensors: 25, with 19,200 parameters | |
| using fused AdamW: True | |
| step 0: train loss 10.9682, val loss 10.9678 | |
| iter 0: loss 10.9624, time 19001.64ms, mfu -100.00% | |
| step 250: train loss 6.9875, val loss 6.9838 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 250: loss 7.0965, time 22530.73ms, mfu 0.05% | |
| step 500: train loss 6.5061, val loss 6.5111 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 500: loss 6.1544, time 22628.33ms, mfu 0.05% | |
| step 750: train loss 6.2891, val loss 6.3209 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 750: loss 6.6225, time 26305.43ms, mfu 0.05% | |
| step 1000: train loss 6.1594, val loss 6.1576 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 1000: loss 5.7977, time 24908.58ms, mfu 0.05% | |
| step 1250: train loss 6.0121, val loss 6.0310 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 1250: loss 6.5299, time 23927.30ms, mfu 0.05% | |
| step 1500: train loss 5.9694, val loss 5.9634 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 1500: loss 5.4807, time 25648.37ms, mfu 0.04% | |
| step 1750: train loss 5.8663, val loss 5.8788 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 1750: loss 6.1823, time 32322.77ms, mfu 0.04% | |
| step 2000: train loss 5.8149, val loss 5.8289 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 2000: loss 5.6224, time 28206.30ms, mfu 0.04% | |
| step 2250: train loss 5.6645, val loss 5.6771 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 2250: loss 5.7664, time 24650.11ms, mfu 0.04% | |
| step 2500: train loss 5.5932, val loss 5.5860 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 2500: loss 4.9037, time 22820.25ms, mfu 0.04% | |
| step 2750: train loss 5.5070, val loss 5.5058 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 2750: loss 5.5161, time 25611.20ms, mfu 0.04% | |
| step 3000: train loss 5.4293, val loss 5.4142 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 3000: loss 5.4806, time 29198.09ms, mfu 0.04% | |
| step 3250: train loss 5.3864, val loss 5.3429 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 3250: loss 5.5491, time 24008.33ms, mfu 0.04% | |
| step 3500: train loss 5.2909, val loss 5.3404 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 3500: loss 5.4112, time 25706.32ms, mfu 0.04% | |
| step 3750: train loss 5.2658, val loss 5.2893 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 3750: loss 5.3241, time 22767.72ms, mfu 0.04% | |
| step 4000: train loss 5.2127, val loss 5.2173 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 4000: loss 5.9016, time 20781.50ms, mfu 0.04% | |
| step 4250: train loss 5.1768, val loss 5.1609 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 4250: loss 4.8918, time 20488.61ms, mfu 0.04% | |
| step 4500: train loss 5.0815, val loss 5.1111 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 4500: loss 5.1913, time 19844.43ms, mfu 0.05% | |
| step 4750: train loss 5.0740, val loss 5.0802 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 4750: loss 5.1475, time 19913.09ms, mfu 0.05% | |
| step 5000: train loss 5.0227, val loss 4.9979 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 5000: loss 5.4350, time 19843.85ms, mfu 0.05% | |
| step 5250: train loss 4.9648, val loss 4.9642 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 5250: loss 5.1092, time 21647.50ms, mfu 0.05% | |
| step 5500: train loss 4.9513, val loss 4.9390 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 5500: loss 4.6933, time 20692.40ms, mfu 0.05% | |
| step 5750: train loss 4.8769, val loss 4.8809 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 5750: loss 5.1268, time 20457.43ms, mfu 0.05% | |
| step 6000: train loss 4.8167, val loss 4.8204 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 6000: loss 4.7500, time 19741.13ms, mfu 0.05% | |
| step 6250: train loss 4.7741, val loss 4.7863 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 6250: loss 4.4547, time 21809.88ms, mfu 0.05% | |
| step 6500: train loss 4.7652, val loss 4.7456 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 6500: loss 4.6225, time 21671.38ms, mfu 0.05% | |
| step 6750: train loss 4.7047, val loss 4.6923 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 6750: loss 4.6679, time 21874.99ms, mfu 0.05% | |
| step 7000: train loss 4.6549, val loss 4.6795 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 7000: loss 4.6344, time 21726.46ms, mfu 0.05% | |
| step 7250: train loss 4.6389, val loss 4.6345 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 7250: loss 4.8557, time 21668.56ms, mfu 0.05% | |
| step 7500: train loss 4.6335, val loss 4.6134 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 7500: loss 4.7470, time 21843.61ms, mfu 0.05% | |
| step 7750: train loss 4.6110, val loss 4.5860 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 7750: loss 4.7129, time 21326.06ms, mfu 0.05% | |
| step 8000: train loss 4.5669, val loss 4.5648 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 8000: loss 4.5233, time 21087.59ms, mfu 0.05% | |
| step 8250: train loss 4.5212, val loss 4.5407 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 8250: loss 4.3764, time 21365.32ms, mfu 0.05% | |
| step 8500: train loss 4.5304, val loss 4.5595 | |
| iter 8500: loss 4.4001, time 18279.96ms, mfu 0.05% | |
| step 8750: train loss 4.4846, val loss 4.5041 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 8750: loss 4.6658, time 21277.66ms, mfu 0.05% | |
| step 9000: train loss 4.4983, val loss 4.5177 | |
| iter 9000: loss 4.4957, time 18356.88ms, mfu 0.05% | |
| step 9250: train loss 4.4751, val loss 4.4660 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 9250: loss 4.5935, time 21322.21ms, mfu 0.05% | |
| step 9500: train loss 4.4470, val loss 4.4637 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 9500: loss 4.1646, time 21366.13ms, mfu 0.05% | |
| step 9750: train loss 4.4428, val loss 4.4574 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 9750: loss 4.7786, time 21184.27ms, mfu 0.05% | |
| step 10000: train loss 4.4404, val loss 4.4707 | |
| iter 10000: loss 4.4349, time 18273.80ms, mfu 0.05% | |