| tokens per iteration will be: 4,096 | |
| defaulting to vocab_size of GPT-2 to 50304 (50257 rounded up for efficiency) | |
| number of parameters: 123.59M | |
| C:\Users\brian\documents\nanogpt\train.py:199: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead. | |
| scaler = torch.cuda.amp.GradScaler(enabled=(dtype == 'float16')) | |
| num decayed parameter tensors: 50, with 123,961,344 parameters | |
| num non-decayed parameter tensors: 25, with 19,200 parameters | |
| using fused AdamW: True | |
| step 0: train loss 10.9593, val loss 10.9583 | |
| iter 0: loss 10.9760, time 18616.92ms, mfu -100.00% | |
| step 250: train loss 6.9613, val loss 6.9070 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 250: loss 6.6580, time 22311.69ms, mfu 0.05% | |
| step 500: train loss 6.4901, val loss 6.5665 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 500: loss 6.5977, time 22371.65ms, mfu 0.05% | |
| step 750: train loss 6.3089, val loss 6.3140 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 750: loss 5.6354, time 21354.18ms, mfu 0.05% | |
| step 1000: train loss 6.1910, val loss 6.1901 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 1000: loss 6.4836, time 21357.54ms, mfu 0.05% | |
| step 1250: train loss 6.0133, val loss 6.0439 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 1250: loss 5.7500, time 21198.50ms, mfu 0.05% | |
| step 1500: train loss 5.9849, val loss 5.9521 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 1500: loss 5.7799, time 21473.31ms, mfu 0.05% | |
| step 1750: train loss 5.8593, val loss 5.8974 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 1750: loss 6.4887, time 21373.76ms, mfu 0.05% | |
| step 2000: train loss 5.8044, val loss 5.8448 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 2000: loss 5.7305, time 21305.85ms, mfu 0.05% | |
| step 2250: train loss 5.6502, val loss 5.7069 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 2250: loss 6.1438, time 21288.79ms, mfu 0.05% | |
| step 2500: train loss 5.6104, val loss 5.6132 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 2500: loss 5.8316, time 21250.85ms, mfu 0.05% | |
| step 2750: train loss 5.5629, val loss 5.5344 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 2750: loss 6.0621, time 21221.10ms, mfu 0.05% | |
| step 3000: train loss 5.4816, val loss 5.4885 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 3000: loss 6.1870, time 21188.14ms, mfu 0.05% | |
| step 3250: train loss 5.3774, val loss 5.3776 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 3250: loss 5.3149, time 21333.74ms, mfu 0.05% | |
| step 3500: train loss 5.3088, val loss 5.3567 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 3500: loss 5.3508, time 21179.13ms, mfu 0.05% | |
| step 3750: train loss 5.2780, val loss 5.2642 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 3750: loss 5.0226, time 21351.70ms, mfu 0.05% | |
| step 4000: train loss 5.2210, val loss 5.2297 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 4000: loss 5.1420, time 21436.75ms, mfu 0.05% | |
| step 4250: train loss 5.1917, val loss 5.2014 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 4250: loss 5.2480, time 21301.86ms, mfu 0.05% | |
| step 4500: train loss 5.1140, val loss 5.1152 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 4500: loss 5.1632, time 21265.97ms, mfu 0.05% | |
| step 4750: train loss 5.0912, val loss 5.0901 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 4750: loss 4.9805, time 21115.88ms, mfu 0.05% | |
| step 5000: train loss 5.0096, val loss 5.0037 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 5000: loss 5.1373, time 21249.09ms, mfu 0.05% | |
| step 5250: train loss 4.9955, val loss 4.9835 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 5250: loss 5.1680, time 21113.54ms, mfu 0.05% | |
| step 5500: train loss 4.9145, val loss 4.9559 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 5500: loss 4.8479, time 21343.74ms, mfu 0.05% | |
| step 5750: train loss 4.9378, val loss 4.9157 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 5750: loss 5.2980, time 21215.62ms, mfu 0.05% | |
| step 6000: train loss 4.8847, val loss 4.8539 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 6000: loss 4.9106, time 21354.24ms, mfu 0.05% | |
| step 6250: train loss 4.8053, val loss 4.8577 | |
| iter 6250: loss 5.0498, time 18422.93ms, mfu 0.05% | |
| step 6500: train loss 4.7705, val loss 4.7786 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 6500: loss 4.6439, time 21349.20ms, mfu 0.05% | |
| step 6750: train loss 4.7284, val loss 4.7511 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 6750: loss 4.4644, time 21331.62ms, mfu 0.05% | |
| step 7000: train loss 4.6923, val loss 4.6983 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 7000: loss 4.6068, time 21215.54ms, mfu 0.05% | |
| step 7250: train loss 4.6634, val loss 4.6856 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 7250: loss 4.2839, time 21796.60ms, mfu 0.05% | |
| step 7500: train loss 4.6686, val loss 4.6505 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 7500: loss 4.6099, time 21891.26ms, mfu 0.05% | |
| step 7750: train loss 4.6168, val loss 4.6141 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 7750: loss 4.7828, time 21872.66ms, mfu 0.05% | |
| step 8000: train loss 4.5929, val loss 4.5856 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 8000: loss 4.2185, time 21387.06ms, mfu 0.05% | |
| step 8250: train loss 4.5486, val loss 4.5455 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 8250: loss 4.3911, time 21357.12ms, mfu 0.05% | |
| step 8500: train loss 4.5357, val loss 4.5546 | |
| iter 8500: loss 4.3294, time 18294.71ms, mfu 0.05% | |
| step 8750: train loss 4.5091, val loss 4.5246 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 8750: loss 4.3438, time 21379.42ms, mfu 0.05% | |
| step 9000: train loss 4.5249, val loss 4.5060 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 9000: loss 4.2822, time 21386.57ms, mfu 0.05% | |
| step 9250: train loss 4.4734, val loss 4.4982 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 9250: loss 5.0134, time 21317.64ms, mfu 0.05% | |
| step 9500: train loss 4.4701, val loss 4.4662 | |
| saving checkpoint to out-openwebtext_dot_gate_shared | |
| iter 9500: loss 4.9098, time 21371.72ms, mfu 0.05% | |
| step 9750: train loss 4.4939, val loss 4.5147 | |
| iter 9750: loss 4.4429, time 18282.87ms, mfu 0.05% | |
| step 10000: train loss 4.4780, val loss 4.4847 | |
| iter 10000: loss 4.6232, time 18338.30ms, mfu 0.05% | |