tokens per iteration will be: 4,096 defaulting to vocab_size of GPT-2 to 50304 (50257 rounded up for efficiency) number of parameters: 123.59M C:\Users\brian\documents\nanogpt\train.py:199: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead. scaler = torch.cuda.amp.GradScaler(enabled=(dtype == 'float16')) num decayed parameter tensors: 50, with 123,961,344 parameters num non-decayed parameter tensors: 25, with 19,200 parameters using fused AdamW: True step 0: train loss 10.9593, val loss 10.9583 iter 0: loss 10.9760, time 18616.92ms, mfu -100.00% step 250: train loss 6.9613, val loss 6.9070 saving checkpoint to out-openwebtext_dot_gate_shared iter 250: loss 6.6580, time 22311.69ms, mfu 0.05% step 500: train loss 6.4901, val loss 6.5665 saving checkpoint to out-openwebtext_dot_gate_shared iter 500: loss 6.5977, time 22371.65ms, mfu 0.05% step 750: train loss 6.3089, val loss 6.3140 saving checkpoint to out-openwebtext_dot_gate_shared iter 750: loss 5.6354, time 21354.18ms, mfu 0.05% step 1000: train loss 6.1910, val loss 6.1901 saving checkpoint to out-openwebtext_dot_gate_shared iter 1000: loss 6.4836, time 21357.54ms, mfu 0.05% step 1250: train loss 6.0133, val loss 6.0439 saving checkpoint to out-openwebtext_dot_gate_shared iter 1250: loss 5.7500, time 21198.50ms, mfu 0.05% step 1500: train loss 5.9849, val loss 5.9521 saving checkpoint to out-openwebtext_dot_gate_shared iter 1500: loss 5.7799, time 21473.31ms, mfu 0.05% step 1750: train loss 5.8593, val loss 5.8974 saving checkpoint to out-openwebtext_dot_gate_shared iter 1750: loss 6.4887, time 21373.76ms, mfu 0.05% step 2000: train loss 5.8044, val loss 5.8448 saving checkpoint to out-openwebtext_dot_gate_shared iter 2000: loss 5.7305, time 21305.85ms, mfu 0.05% step 2250: train loss 5.6502, val loss 5.7069 saving checkpoint to out-openwebtext_dot_gate_shared iter 2250: loss 6.1438, time 21288.79ms, mfu 0.05% step 2500: train loss 5.6104, val loss 5.6132 saving checkpoint to out-openwebtext_dot_gate_shared iter 2500: loss 5.8316, time 21250.85ms, mfu 0.05% step 2750: train loss 5.5629, val loss 5.5344 saving checkpoint to out-openwebtext_dot_gate_shared iter 2750: loss 6.0621, time 21221.10ms, mfu 0.05% step 3000: train loss 5.4816, val loss 5.4885 saving checkpoint to out-openwebtext_dot_gate_shared iter 3000: loss 6.1870, time 21188.14ms, mfu 0.05% step 3250: train loss 5.3774, val loss 5.3776 saving checkpoint to out-openwebtext_dot_gate_shared iter 3250: loss 5.3149, time 21333.74ms, mfu 0.05% step 3500: train loss 5.3088, val loss 5.3567 saving checkpoint to out-openwebtext_dot_gate_shared iter 3500: loss 5.3508, time 21179.13ms, mfu 0.05% step 3750: train loss 5.2780, val loss 5.2642 saving checkpoint to out-openwebtext_dot_gate_shared iter 3750: loss 5.0226, time 21351.70ms, mfu 0.05% step 4000: train loss 5.2210, val loss 5.2297 saving checkpoint to out-openwebtext_dot_gate_shared iter 4000: loss 5.1420, time 21436.75ms, mfu 0.05% step 4250: train loss 5.1917, val loss 5.2014 saving checkpoint to out-openwebtext_dot_gate_shared iter 4250: loss 5.2480, time 21301.86ms, mfu 0.05% step 4500: train loss 5.1140, val loss 5.1152 saving checkpoint to out-openwebtext_dot_gate_shared iter 4500: loss 5.1632, time 21265.97ms, mfu 0.05% step 4750: train loss 5.0912, val loss 5.0901 saving checkpoint to out-openwebtext_dot_gate_shared iter 4750: loss 4.9805, time 21115.88ms, mfu 0.05% step 5000: train loss 5.0096, val loss 5.0037 saving checkpoint to out-openwebtext_dot_gate_shared iter 5000: loss 5.1373, time 21249.09ms, mfu 0.05% step 5250: train loss 4.9955, val loss 4.9835 saving checkpoint to out-openwebtext_dot_gate_shared iter 5250: loss 5.1680, time 21113.54ms, mfu 0.05% step 5500: train loss 4.9145, val loss 4.9559 saving checkpoint to out-openwebtext_dot_gate_shared iter 5500: loss 4.8479, time 21343.74ms, mfu 0.05% step 5750: train loss 4.9378, val loss 4.9157 saving checkpoint to out-openwebtext_dot_gate_shared iter 5750: loss 5.2980, time 21215.62ms, mfu 0.05% step 6000: train loss 4.8847, val loss 4.8539 saving checkpoint to out-openwebtext_dot_gate_shared iter 6000: loss 4.9106, time 21354.24ms, mfu 0.05% step 6250: train loss 4.8053, val loss 4.8577 iter 6250: loss 5.0498, time 18422.93ms, mfu 0.05% step 6500: train loss 4.7705, val loss 4.7786 saving checkpoint to out-openwebtext_dot_gate_shared iter 6500: loss 4.6439, time 21349.20ms, mfu 0.05% step 6750: train loss 4.7284, val loss 4.7511 saving checkpoint to out-openwebtext_dot_gate_shared iter 6750: loss 4.4644, time 21331.62ms, mfu 0.05% step 7000: train loss 4.6923, val loss 4.6983 saving checkpoint to out-openwebtext_dot_gate_shared iter 7000: loss 4.6068, time 21215.54ms, mfu 0.05% step 7250: train loss 4.6634, val loss 4.6856 saving checkpoint to out-openwebtext_dot_gate_shared iter 7250: loss 4.2839, time 21796.60ms, mfu 0.05% step 7500: train loss 4.6686, val loss 4.6505 saving checkpoint to out-openwebtext_dot_gate_shared iter 7500: loss 4.6099, time 21891.26ms, mfu 0.05% step 7750: train loss 4.6168, val loss 4.6141 saving checkpoint to out-openwebtext_dot_gate_shared iter 7750: loss 4.7828, time 21872.66ms, mfu 0.05% step 8000: train loss 4.5929, val loss 4.5856 saving checkpoint to out-openwebtext_dot_gate_shared iter 8000: loss 4.2185, time 21387.06ms, mfu 0.05% step 8250: train loss 4.5486, val loss 4.5455 saving checkpoint to out-openwebtext_dot_gate_shared iter 8250: loss 4.3911, time 21357.12ms, mfu 0.05% step 8500: train loss 4.5357, val loss 4.5546 iter 8500: loss 4.3294, time 18294.71ms, mfu 0.05% step 8750: train loss 4.5091, val loss 4.5246 saving checkpoint to out-openwebtext_dot_gate_shared iter 8750: loss 4.3438, time 21379.42ms, mfu 0.05% step 9000: train loss 4.5249, val loss 4.5060 saving checkpoint to out-openwebtext_dot_gate_shared iter 9000: loss 4.2822, time 21386.57ms, mfu 0.05% step 9250: train loss 4.4734, val loss 4.4982 saving checkpoint to out-openwebtext_dot_gate_shared iter 9250: loss 5.0134, time 21317.64ms, mfu 0.05% step 9500: train loss 4.4701, val loss 4.4662 saving checkpoint to out-openwebtext_dot_gate_shared iter 9500: loss 4.9098, time 21371.72ms, mfu 0.05% step 9750: train loss 4.4939, val loss 4.5147 iter 9750: loss 4.4429, time 18282.87ms, mfu 0.05% step 10000: train loss 4.4780, val loss 4.4847 iter 10000: loss 4.6232, time 18338.30ms, mfu 0.05%