tokens per iteration will be: 4,096 defaulting to vocab_size of GPT-2 to 50304 (50257 rounded up for efficiency) number of parameters: 123.59M C:\Users\brian\documents\nanogpt\train.py:199: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead. scaler = torch.cuda.amp.GradScaler(enabled=(dtype == 'float16')) num decayed parameter tensors: 50, with 123,961,344 parameters num non-decayed parameter tensors: 25, with 19,200 parameters using fused AdamW: True step 0: train loss 10.9682, val loss 10.9678 iter 0: loss 10.9624, time 19001.64ms, mfu -100.00% step 250: train loss 6.9875, val loss 6.9838 saving checkpoint to out-openwebtext_dot_gate_shared iter 250: loss 7.0965, time 22530.73ms, mfu 0.05% step 500: train loss 6.5061, val loss 6.5111 saving checkpoint to out-openwebtext_dot_gate_shared iter 500: loss 6.1544, time 22628.33ms, mfu 0.05% step 750: train loss 6.2891, val loss 6.3209 saving checkpoint to out-openwebtext_dot_gate_shared iter 750: loss 6.6225, time 26305.43ms, mfu 0.05% step 1000: train loss 6.1594, val loss 6.1576 saving checkpoint to out-openwebtext_dot_gate_shared iter 1000: loss 5.7977, time 24908.58ms, mfu 0.05% step 1250: train loss 6.0121, val loss 6.0310 saving checkpoint to out-openwebtext_dot_gate_shared iter 1250: loss 6.5299, time 23927.30ms, mfu 0.05% step 1500: train loss 5.9694, val loss 5.9634 saving checkpoint to out-openwebtext_dot_gate_shared iter 1500: loss 5.4807, time 25648.37ms, mfu 0.04% step 1750: train loss 5.8663, val loss 5.8788 saving checkpoint to out-openwebtext_dot_gate_shared iter 1750: loss 6.1823, time 32322.77ms, mfu 0.04% step 2000: train loss 5.8149, val loss 5.8289 saving checkpoint to out-openwebtext_dot_gate_shared iter 2000: loss 5.6224, time 28206.30ms, mfu 0.04% step 2250: train loss 5.6645, val loss 5.6771 saving checkpoint to out-openwebtext_dot_gate_shared iter 2250: loss 5.7664, time 24650.11ms, mfu 0.04% step 2500: train loss 5.5932, val loss 5.5860 saving checkpoint to out-openwebtext_dot_gate_shared iter 2500: loss 4.9037, time 22820.25ms, mfu 0.04% step 2750: train loss 5.5070, val loss 5.5058 saving checkpoint to out-openwebtext_dot_gate_shared iter 2750: loss 5.5161, time 25611.20ms, mfu 0.04% step 3000: train loss 5.4293, val loss 5.4142 saving checkpoint to out-openwebtext_dot_gate_shared iter 3000: loss 5.4806, time 29198.09ms, mfu 0.04% step 3250: train loss 5.3864, val loss 5.3429 saving checkpoint to out-openwebtext_dot_gate_shared iter 3250: loss 5.5491, time 24008.33ms, mfu 0.04% step 3500: train loss 5.2909, val loss 5.3404 saving checkpoint to out-openwebtext_dot_gate_shared iter 3500: loss 5.4112, time 25706.32ms, mfu 0.04% step 3750: train loss 5.2658, val loss 5.2893 saving checkpoint to out-openwebtext_dot_gate_shared iter 3750: loss 5.3241, time 22767.72ms, mfu 0.04% step 4000: train loss 5.2127, val loss 5.2173 saving checkpoint to out-openwebtext_dot_gate_shared iter 4000: loss 5.9016, time 20781.50ms, mfu 0.04% step 4250: train loss 5.1768, val loss 5.1609 saving checkpoint to out-openwebtext_dot_gate_shared iter 4250: loss 4.8918, time 20488.61ms, mfu 0.04% step 4500: train loss 5.0815, val loss 5.1111 saving checkpoint to out-openwebtext_dot_gate_shared iter 4500: loss 5.1913, time 19844.43ms, mfu 0.05% step 4750: train loss 5.0740, val loss 5.0802 saving checkpoint to out-openwebtext_dot_gate_shared iter 4750: loss 5.1475, time 19913.09ms, mfu 0.05% step 5000: train loss 5.0227, val loss 4.9979 saving checkpoint to out-openwebtext_dot_gate_shared iter 5000: loss 5.4350, time 19843.85ms, mfu 0.05% step 5250: train loss 4.9648, val loss 4.9642 saving checkpoint to out-openwebtext_dot_gate_shared iter 5250: loss 5.1092, time 21647.50ms, mfu 0.05% step 5500: train loss 4.9513, val loss 4.9390 saving checkpoint to out-openwebtext_dot_gate_shared iter 5500: loss 4.6933, time 20692.40ms, mfu 0.05% step 5750: train loss 4.8769, val loss 4.8809 saving checkpoint to out-openwebtext_dot_gate_shared iter 5750: loss 5.1268, time 20457.43ms, mfu 0.05% step 6000: train loss 4.8167, val loss 4.8204 saving checkpoint to out-openwebtext_dot_gate_shared iter 6000: loss 4.7500, time 19741.13ms, mfu 0.05% step 6250: train loss 4.7741, val loss 4.7863 saving checkpoint to out-openwebtext_dot_gate_shared iter 6250: loss 4.4547, time 21809.88ms, mfu 0.05% step 6500: train loss 4.7652, val loss 4.7456 saving checkpoint to out-openwebtext_dot_gate_shared iter 6500: loss 4.6225, time 21671.38ms, mfu 0.05% step 6750: train loss 4.7047, val loss 4.6923 saving checkpoint to out-openwebtext_dot_gate_shared iter 6750: loss 4.6679, time 21874.99ms, mfu 0.05% step 7000: train loss 4.6549, val loss 4.6795 saving checkpoint to out-openwebtext_dot_gate_shared iter 7000: loss 4.6344, time 21726.46ms, mfu 0.05% step 7250: train loss 4.6389, val loss 4.6345 saving checkpoint to out-openwebtext_dot_gate_shared iter 7250: loss 4.8557, time 21668.56ms, mfu 0.05% step 7500: train loss 4.6335, val loss 4.6134 saving checkpoint to out-openwebtext_dot_gate_shared iter 7500: loss 4.7470, time 21843.61ms, mfu 0.05% step 7750: train loss 4.6110, val loss 4.5860 saving checkpoint to out-openwebtext_dot_gate_shared iter 7750: loss 4.7129, time 21326.06ms, mfu 0.05% step 8000: train loss 4.5669, val loss 4.5648 saving checkpoint to out-openwebtext_dot_gate_shared iter 8000: loss 4.5233, time 21087.59ms, mfu 0.05% step 8250: train loss 4.5212, val loss 4.5407 saving checkpoint to out-openwebtext_dot_gate_shared iter 8250: loss 4.3764, time 21365.32ms, mfu 0.05% step 8500: train loss 4.5304, val loss 4.5595 iter 8500: loss 4.4001, time 18279.96ms, mfu 0.05% step 8750: train loss 4.4846, val loss 4.5041 saving checkpoint to out-openwebtext_dot_gate_shared iter 8750: loss 4.6658, time 21277.66ms, mfu 0.05% step 9000: train loss 4.4983, val loss 4.5177 iter 9000: loss 4.4957, time 18356.88ms, mfu 0.05% step 9250: train loss 4.4751, val loss 4.4660 saving checkpoint to out-openwebtext_dot_gate_shared iter 9250: loss 4.5935, time 21322.21ms, mfu 0.05% step 9500: train loss 4.4470, val loss 4.4637 saving checkpoint to out-openwebtext_dot_gate_shared iter 9500: loss 4.1646, time 21366.13ms, mfu 0.05% step 9750: train loss 4.4428, val loss 4.4574 saving checkpoint to out-openwebtext_dot_gate_shared iter 9750: loss 4.7786, time 21184.27ms, mfu 0.05% step 10000: train loss 4.4404, val loss 4.4707 iter 10000: loss 4.4349, time 18273.80ms, mfu 0.05%