| Overriding config with config/train_wiki_gutenberg.py: |
| |
| |
| |
|
|
| out_dir = 'out-wiki-gutenberg-530m-1ep' |
| eval_interval = 2000 |
| eval_iters = 200 |
| log_interval = 100 |
| always_save_checkpoint = True |
|
|
| wandb_log = False |
| wandb_project = 'wiki_gutenberg' |
| wandb_run_name = '0.5b' |
|
|
| dataset = 'wiki_gutenberg' |
| gradient_accumulation_steps = 1 |
| batch_size = 256 |
| block_size = 256 |
|
|
| |
| n_layer = 6 |
| n_head = 6 |
| n_embd = 384 |
| dropout = 0.1 |
|
|
| learning_rate = 1e-3 |
| max_iters = 5000 |
| lr_decay_iters = 5000 |
| min_lr = 1e-4 |
| beta2 = 0.95 |
| warmup_iters = 100 |
|
|
| device = 'cpu' |
| compile = False |
|
|
| Overriding: device = cuda |
| Overriding: compile = True |
| Overriding: eval_interval = 2000 |
| Overriding: eval_iters = 200 |
| Overriding: log_interval = 100 |
| Overriding: always_save_checkpoint = True |
| Overriding: block_size = 256 |
| Overriding: batch_size = 256 |
| Overriding: gradient_accumulation_steps = 1 |
| Overriding: n_layer = 16 |
| Overriding: n_head = 24 |
| Overriding: n_embd = 1536 |
| Overriding: dropout = 0.1 |
| Overriding: learning_rate = 0.0012 |
| Overriding: min_lr = 0.00012 |
| Overriding: warmup_iters = 5000 |
| Overriding: beta2 = 0.95 |
| Overriding: max_iters = 137108 |
| Overriding: lr_decay_iters = 137108 |
| Overriding: out_dir = out-wiki-gutenberg-530m-1ep |
| tokens per iteration will be: 65,536 |
| Initializing a new model from scratch |
| defaulting to vocab_size of GPT-2 to 50304 (50257 rounded up for efficiency) |
| number of parameters: 530.30M |
| /root/nanoGPT/train.py:199: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead. |
| scaler = torch.cuda.amp.GradScaler(enabled=(dtype == 'float16')) |
| num decayed parameter tensors: 66, with 530,644,992 parameters |
| num non-decayed parameter tensors: 33, with 50,688 parameters |
| using fused AdamW: True |
| compiling the model... (takes a ~minute) |
| step 0: train loss 11.1036, val loss 11.1013 |
| iter 0: loss 11.1082, time 89170.69ms, mfu -100.00% |
| iter 100: loss 7.1267, time 437.34ms, mfu 156.45% |
| iter 200: loss 6.4479, time 442.17ms, mfu 156.28% |
| iter 300: loss 5.8774, time 442.33ms, mfu 156.12% |
| iter 400: loss 5.6738, time 440.82ms, mfu 156.03% |
| iter 500: loss 5.3547, time 442.74ms, mfu 155.88% |
| iter 600: loss 5.1492, time 441.76ms, mfu 155.78% |
| iter 700: loss 5.0054, time 441.16ms, mfu 155.71% |
| iter 800: loss 4.8405, time 442.20ms, mfu 155.61% |
| iter 900: loss 4.7264, time 441.09ms, mfu 155.56% |
| iter 1000: loss 4.5948, time 444.95ms, mfu 155.38% |
| iter 1100: loss 4.3694, time 442.62ms, mfu 155.30% |
| iter 1200: loss 4.3759, time 445.97ms, mfu 155.11% |
| iter 1300: loss 4.2373, time 441.98ms, mfu 155.08% |
| iter 1400: loss 4.1923, time 443.71ms, mfu 154.99% |
| iter 1500: loss 4.1416, time 446.06ms, mfu 154.83% |
| iter 1600: loss 4.0585, time 444.95ms, mfu 154.73% |
| iter 1700: loss 4.1305, time 442.71ms, mfu 154.71% |
| iter 1800: loss 4.0767, time 446.73ms, mfu 154.55% |
| iter 1900: loss 4.0259, time 442.95ms, mfu 154.55% |
| step 2000: train loss 3.9183, val loss 3.9155 |
| saving checkpoint to out-wiki-gutenberg-530m-1ep |
| iter 2000: loss 3.9544, time 62017.87ms, mfu 139.20% |
| iter 2100: loss 3.9057, time 446.44ms, mfu 140.61% |
| iter 2200: loss 3.8796, time 446.73ms, mfu 141.86% |
| iter 2300: loss 3.8484, time 447.67ms, mfu 142.96% |
| iter 2400: loss 3.9056, time 444.35ms, mfu 144.06% |
| iter 2500: loss 3.8618, time 446.88ms, mfu 144.97% |
| iter 2600: loss 3.8273, time 446.55ms, mfu 145.79% |
| iter 2700: loss 3.8333, time 440.68ms, mfu 146.74% |
| iter 2800: loss 3.8259, time 441.91ms, mfu 147.55% |
| iter 2900: loss 3.7795, time 448.62ms, mfu 148.04% |
| iter 3000: loss 3.7829, time 449.51ms, mfu 148.46% |
| iter 3100: loss 3.8019, time 444.32ms, mfu 149.01% |
| iter 3200: loss 3.8028, time 448.94ms, mfu 149.35% |
| iter 3300: loss 3.7317, time 445.18ms, mfu 149.79% |
| iter 3400: loss 3.6775, time 444.78ms, mfu 150.19% |
| iter 3500: loss 3.7106, time 443.69ms, mfu 150.59% |
| iter 3600: loss 3.7242, time 444.40ms, mfu 150.93% |
| iter 3700: loss 3.7189, time 447.56ms, mfu 151.12% |
| iter 3800: loss 3.6343, time 447.22ms, mfu 151.31% |
| iter 3900: loss 3.6177, time 446.55ms, mfu 151.50% |
| step 4000: train loss 3.6156, val loss 3.6195 |
| saving checkpoint to out-wiki-gutenberg-530m-1ep |
| iter 4000: loss 3.6261, time 66001.87ms, mfu 136.45% |
| iter 4100: loss 3.6635, time 439.91ms, mfu 138.36% |
| iter 4200: loss 3.6292, time 439.72ms, mfu 140.09% |
| iter 4300: loss 3.6484, time 442.83ms, mfu 141.53% |
| iter 4400: loss 3.6215, time 445.03ms, mfu 142.75% |
| iter 4500: loss 3.6225, time 442.49ms, mfu 143.94% |
| iter 4600: loss 3.6168, time 446.89ms, mfu 144.85% |
| iter 4700: loss 3.6826, time 443.07ms, mfu 145.81% |
| iter 4800: loss 3.5770, time 442.51ms, mfu 146.69% |
| iter 4900: loss 3.6001, time 444.92ms, mfu 147.40% |
| iter 5000: loss 3.6241, time 440.84ms, mfu 148.18% |
| iter 5100: loss 3.5333, time 447.42ms, mfu 148.66% |
| iter 5200: loss 3.6191, time 442.71ms, mfu 149.24% |
| iter 5300: loss 3.6189, time 448.83ms, mfu 149.56% |
| iter 5400: loss 3.6240, time 443.82ms, mfu 150.02% |
| iter 5500: loss 3.4967, time 445.14ms, mfu 150.39% |
| iter 5600: loss 3.4990, time 441.33ms, mfu 150.86% |
| iter 5700: loss 3.5285, time 442.22ms, mfu 151.24% |
| iter 5800: loss 3.5127, time 444.95ms, mfu 151.50% |
| iter 5900: loss 3.4877, time 442.02ms, mfu 151.82% |
| step 6000: train loss 3.4517, val loss 3.4570 |
| saving checkpoint to out-wiki-gutenberg-530m-1ep |
| iter 6000: loss 3.3989, time 65862.07ms, mfu 136.75% |
| iter 6100: loss 3.4812, time 439.08ms, mfu 138.65% |
| iter 6200: loss 3.5478, time 442.09ms, mfu 140.27% |
| iter 6300: loss 3.4423, time 442.49ms, mfu 141.70% |
| iter 6400: loss 3.4052, time 443.41ms, mfu 142.96% |
| iter 6500: loss 3.5069, time 442.02ms, mfu 144.14% |
| iter 6600: loss 3.3985, time 441.76ms, mfu 145.22% |
| iter 6700: loss 3.4148, time 446.59ms, mfu 146.02% |
| iter 6800: loss 3.4067, time 441.78ms, mfu 146.90% |
| iter 6900: loss 3.3920, time 442.39ms, mfu 147.68% |
| iter 7000: loss 3.3811, time 441.98ms, mfu 148.39% |
| iter 7100: loss 3.4921, time 445.73ms, mfu 148.90% |
| iter 7200: loss 3.4252, time 445.87ms, mfu 149.36% |
| iter 7300: loss 3.4486, time 442.58ms, mfu 149.88% |
| iter 7400: loss 3.4207, time 444.25ms, mfu 150.29% |
| iter 7500: loss 3.4161, time 444.91ms, mfu 150.64% |
| iter 7600: loss 3.3954, time 442.74ms, mfu 151.03% |
| iter 7700: loss 3.4477, time 441.99ms, mfu 151.41% |
| iter 7800: loss 3.3723, time 442.45ms, mfu 151.73% |
| iter 7900: loss 3.4532, time 443.68ms, mfu 151.98% |
| step 8000: train loss 3.3458, val loss 3.3517 |
| saving checkpoint to out-wiki-gutenberg-530m-1ep |
| iter 8000: loss 3.3585, time 65769.94ms, mfu 136.89% |
| iter 8100: loss 3.4348, time 443.89ms, mfu 138.61% |
| iter 8200: loss 3.3941, time 440.23ms, mfu 140.29% |
| iter 8300: loss 3.4014, time 443.97ms, mfu 141.67% |
| iter 8400: loss 3.4147, time 440.55ms, mfu 143.04% |
| iter 8500: loss 3.4033, time 442.57ms, mfu 144.19% |
| iter 8600: loss 3.3590, time 441.40ms, mfu 145.27% |
| iter 8700: loss 3.3398, time 441.06ms, mfu 146.26% |
| iter 8800: loss 3.3703, time 445.86ms, mfu 146.98% |
| iter 8900: loss 3.3603, time 444.12ms, mfu 147.69% |
| iter 9000: loss 3.3583, time 446.80ms, mfu 148.23% |
| iter 9100: loss 3.3942, time 441.60ms, mfu 148.90% |
| iter 9200: loss 3.4014, time 445.52ms, mfu 149.37% |
| iter 9300: loss 3.3763, time 442.88ms, mfu 149.88% |
| iter 9400: loss 3.3114, time 444.65ms, mfu 150.28% |
| iter 9500: loss 3.4656, time 444.21ms, mfu 150.66% |
| iter 9600: loss 3.2709, time 443.68ms, mfu 151.01% |
| iter 9700: loss 3.3287, time 441.56ms, mfu 151.41% |
| iter 9800: loss 3.3598, time 444.98ms, mfu 151.64% |
| iter 9900: loss 3.2859, time 441.92ms, mfu 151.96% |
| step 10000: train loss 3.2856, val loss 3.2934 |
| saving checkpoint to out-wiki-gutenberg-530m-1ep |
| iter 10000: loss 3.3140, time 65707.54ms, mfu 136.87% |
| iter 10100: loss 3.3347, time 440.20ms, mfu 138.72% |
| iter 10200: loss 3.3091, time 441.62ms, mfu 140.34% |
| iter 10300: loss 3.3442, time 444.88ms, mfu 141.69% |
| iter 10400: loss 3.4077, time 442.05ms, mfu 143.00% |
| iter 10500: loss 3.3773, time 444.11ms, mfu 144.10% |
| iter 10600: loss 3.3197, time 445.05ms, mfu 145.07% |
| iter 10700: loss 3.3867, time 445.74ms, mfu 145.91% |
| iter 10800: loss 3.3132, time 443.69ms, mfu 146.74% |
| iter 10900: loss 3.2219, time 442.60ms, mfu 147.53% |
| iter 11000: loss 3.3162, time 442.65ms, mfu 148.23% |
| iter 11100: loss 3.2492, time 443.92ms, mfu 148.82% |
| iter 11200: loss 3.2580, time 442.62ms, mfu 149.40% |
| iter 11300: loss 3.3678, time 442.80ms, mfu 149.91% |
| iter 11400: loss 3.3408, time 443.72ms, mfu 150.34% |
| iter 11500: loss 3.3912, time 441.13ms, mfu 150.81% |
| iter 11600: loss 3.2979, time 445.79ms, mfu 151.08% |
| iter 11700: loss 3.2721, time 443.37ms, mfu 151.40% |
| iter 11800: loss 3.2679, time 443.61ms, mfu 151.69% |
| iter 11900: loss 3.3370, time 440.43ms, mfu 152.05% |
| step 12000: train loss 3.2428, val loss 3.2486 |
| saving checkpoint to out-wiki-gutenberg-530m-1ep |
| iter 12000: loss 3.2503, time 65678.03ms, mfu 136.95% |
| iter 12100: loss 3.3662, time 438.84ms, mfu 138.85% |
| iter 12200: loss 3.2599, time 442.94ms, mfu 140.41% |
| iter 12300: loss 3.2895, time 442.09ms, mfu 141.85% |
| iter 12400: loss 3.2313, time 441.04ms, mfu 143.17% |
| iter 12500: loss 3.3367, time 441.28ms, mfu 144.36% |
| iter 12600: loss 3.2648, time 446.60ms, mfu 145.25% |
| iter 12700: loss 3.2849, time 447.12ms, mfu 146.02% |
| iter 12800: loss 3.2787, time 443.23ms, mfu 146.86% |
| iter 12900: loss 3.3005, time 445.94ms, mfu 147.52% |
| iter 13000: loss 3.2938, time 442.06ms, mfu 148.24% |
| iter 13100: loss 3.2868, time 447.35ms, mfu 148.71% |
| iter 13200: loss 3.2834, time 441.03ms, mfu 149.35% |
| iter 13300: loss 3.2777, time 440.04ms, mfu 149.97% |
| iter 13400: loss 3.2641, time 442.06ms, mfu 150.45% |
| iter 13500: loss 3.2625, time 443.11ms, mfu 150.84% |
| iter 13600: loss 3.2251, time 444.52ms, mfu 151.15% |
| iter 13700: loss 3.2294, time 444.16ms, mfu 151.44% |
| iter 13800: loss 3.2571, time 443.35ms, mfu 151.73% |
| iter 13900: loss 3.1879, time 443.60ms, mfu 151.98% |
| step 14000: train loss 3.2173, val loss 3.2256 |
| saving checkpoint to out-wiki-gutenberg-530m-1ep |
| iter 14000: loss 3.2374, time 65827.28ms, mfu 136.89% |
| iter 14100: loss 3.2433, time 441.60ms, mfu 138.69% |
| iter 14200: loss 3.2675, time 443.25ms, mfu 140.26% |
| iter 14300: loss 3.2100, time 441.87ms, mfu 141.72% |
| iter 14400: loss 3.2757, time 444.32ms, mfu 142.94% |
| iter 14500: loss 3.3211, time 443.98ms, mfu 144.06% |
| iter 14600: loss 3.2441, time 441.54ms, mfu 145.15% |
| iter 14700: loss 3.2467, time 440.77ms, mfu 146.16% |
| iter 14800: loss 3.3270, time 444.99ms, mfu 146.92% |
| iter 14900: loss 3.2327, time 442.88ms, mfu 147.67% |
| iter 15000: loss 3.2169, time 442.76ms, mfu 148.36% |
| iter 15100: loss 3.2549, time 440.97ms, mfu 149.04% |
| iter 15200: loss 3.2018, time 446.53ms, mfu 149.46% |
| iter 15300: loss 3.2698, time 442.74ms, mfu 149.97% |
| iter 15400: loss 3.2472, time 441.88ms, mfu 150.45% |
| iter 15500: loss 3.2258, time 445.46ms, mfu 150.77% |
| iter 15600: loss 3.1997, time 440.81ms, mfu 151.21% |
| iter 15700: loss 3.1950, time 443.81ms, mfu 151.51% |
| iter 15800: loss 3.3171, time 442.84ms, mfu 151.81% |
| iter 15900: loss 3.2615, time 445.16ms, mfu 152.00% |
| step 16000: train loss 3.1977, val loss 3.2048 |
| saving checkpoint to out-wiki-gutenberg-530m-1ep |
| iter 16000: loss 3.2691, time 65701.88ms, mfu 136.90% |
| iter 16100: loss 3.2874, time 441.43ms, mfu 138.71% |
| iter 16200: loss 3.1908, time 442.66ms, mfu 140.30% |
| iter 16300: loss 3.1912, time 443.20ms, mfu 141.70% |
| iter 16400: loss 3.2039, time 442.97ms, mfu 142.98% |
| iter 16500: loss 3.1949, time 440.96ms, mfu 144.20% |
| iter 16600: loss 3.2290, time 443.31ms, mfu 145.21% |
| iter 16700: loss 3.2693, time 445.50ms, mfu 146.05% |
| iter 16800: loss 3.2097, time 445.26ms, mfu 146.81% |
| iter 16900: loss 3.2467, time 442.98ms, mfu 147.57% |
| iter 17000: loss 3.2187, time 442.79ms, mfu 148.27% |
| iter 17100: loss 3.2884, time 443.43ms, mfu 148.87% |
| iter 17200: loss 3.2612, time 445.50ms, mfu 149.34% |
| iter 17300: loss 3.2020, time 442.55ms, mfu 149.87% |
| iter 17400: loss 3.2556, time 444.66ms, mfu 150.27% |
| iter 17500: loss 3.2447, time 442.91ms, mfu 150.69% |
| iter 17600: loss 3.2983, time 440.10ms, mfu 151.17% |
| iter 17700: loss 3.1734, time 444.52ms, mfu 151.44% |
| iter 17800: loss 3.2511, time 442.56ms, mfu 151.76% |
| iter 17900: loss 3.2137, time 445.42ms, mfu 151.94% |
| step 18000: train loss 3.1773, val loss 3.1889 |
| saving checkpoint to out-wiki-gutenberg-530m-1ep |
| iter 18000: loss 3.2322, time 65615.98ms, mfu 136.85% |
| iter 18100: loss 3.2220, time 440.36ms, mfu 138.71% |
| iter 18200: loss 3.3049, time 440.15ms, mfu 140.38% |
| iter 18300: loss 3.2544, time 446.33ms, mfu 141.67% |
| iter 18400: loss 3.2650, time 444.13ms, mfu 142.91% |
| iter 18500: loss 3.2118, time 441.45ms, mfu 144.12% |
| iter 18600: loss 3.1895, time 442.50ms, mfu 145.17% |
| iter 18700: loss 3.1941, time 443.30ms, mfu 146.09% |
| iter 18800: loss 3.2778, time 444.64ms, mfu 146.87% |
| iter 18900: loss 3.2339, time 441.31ms, mfu 147.68% |
| iter 19000: loss 3.1870, time 441.61ms, mfu 148.41% |
| iter 19100: loss 3.2831, time 443.72ms, mfu 148.99% |
| iter 19200: loss 3.2928, time 441.89ms, mfu 149.57% |
| iter 19300: loss 3.2986, time 445.25ms, mfu 149.98% |
| iter 19400: loss 3.2456, time 445.22ms, mfu 150.35% |
| iter 19500: loss 3.1557, time 445.29ms, mfu 150.68% |
| iter 19600: loss 3.1754, time 444.03ms, mfu 151.02% |
| iter 19700: loss 3.1769, time 443.85ms, mfu 151.33% |
| iter 19800: loss 3.2197, time 443.63ms, mfu 151.62% |
| iter 19900: loss 3.1796, time 441.84ms, mfu 151.95% |
| step 20000: train loss 3.1663, val loss 3.1786 |
| saving checkpoint to out-wiki-gutenberg-530m-1ep |
| iter 20000: loss 3.2533, time 65627.46ms, mfu 136.86% |
| iter 20100: loss 3.2156, time 440.87ms, mfu 138.69% |
| iter 20200: loss 3.2796, time 440.86ms, mfu 140.34% |
| iter 20300: loss 3.2432, time 439.73ms, mfu 141.87% |
| iter 20400: loss 3.2318, time 440.62ms, mfu 143.21% |
| iter 20500: loss 3.1981, time 440.47ms, mfu 144.42% |
| iter 20600: loss 3.2269, time 442.96ms, mfu 145.42% |
| iter 20700: loss 3.2127, time 441.25ms, mfu 146.39% |
| iter 20800: loss 3.2265, time 440.43ms, mfu 147.28% |
| iter 20900: loss 3.2139, time 444.35ms, mfu 147.95% |
| iter 21000: loss 3.2253, time 440.23ms, mfu 148.70% |
| iter 21100: loss 3.1392, time 436.57ms, mfu 149.50% |
| iter 21200: loss 3.2089, time 440.38ms, mfu 150.09% |
| iter 21300: loss 3.1923, time 439.31ms, mfu 150.65% |
| iter 21400: loss 3.2165, time 437.87ms, mfu 151.22% |
| iter 21500: loss 3.1946, time 440.78ms, mfu 151.62% |
| iter 21600: loss 3.2736, time 439.86ms, mfu 152.01% |
| iter 21700: loss 3.1535, time 437.99ms, mfu 152.43% |
| iter 21800: loss 3.2071, time 440.49ms, mfu 152.72% |
| iter 21900: loss 3.1913, time 441.04ms, mfu 152.96% |
| step 22000: train loss 3.1568, val loss 3.1641 |
| saving checkpoint to out-wiki-gutenberg-530m-1ep |
| iter 22000: loss 3.1692, time 65310.84ms, mfu 137.77% |
| iter 22100: loss 3.2188, time 440.52ms, mfu 139.52% |
| iter 22200: loss 3.1753, time 440.64ms, mfu 141.10% |
| iter 22300: loss 3.3106, time 437.58ms, mfu 142.63% |
| iter 22400: loss 3.1838, time 440.55ms, mfu 143.89% |
| iter 22500: loss 3.2047, time 441.35ms, mfu 145.01% |
| iter 22600: loss 3.2066, time 441.71ms, mfu 146.00% |
| iter 22700: loss 3.2010, time 440.30ms, mfu 146.94% |
| iter 22800: loss 3.1363, time 441.55ms, mfu 147.74% |
| iter 22900: loss 3.1682, time 441.18ms, mfu 148.47% |
| iter 23000: loss 3.2113, time 443.77ms, mfu 149.04% |
| iter 23100: loss 3.2214, time 442.58ms, mfu 149.60% |
| iter 23200: loss 3.1835, time 440.88ms, mfu 150.16% |
| iter 23300: loss 3.2083, time 441.10ms, mfu 150.65% |
| iter 23400: loss 3.2242, time 441.47ms, mfu 151.09% |
| iter 23500: loss 3.1649, time 441.47ms, mfu 151.48% |
| iter 23600: loss 3.1838, time 440.25ms, mfu 151.87% |
| iter 23700: loss 3.1758, time 440.56ms, mfu 152.21% |
| iter 23800: loss 3.2205, time 439.44ms, mfu 152.56% |
| iter 23900: loss 3.2537, time 438.36ms, mfu 152.91% |
| step 24000: train loss 3.1421, val loss 3.1553 |
| saving checkpoint to out-wiki-gutenberg-530m-1ep |
| iter 24000: loss 3.1881, time 65282.99ms, mfu 137.73% |
| iter 24100: loss 3.2003, time 439.85ms, mfu 139.51% |
| iter 24200: loss 3.2393, time 441.07ms, mfu 141.07% |
| iter 24300: loss 3.2287, time 438.57ms, mfu 142.56% |
| iter 24400: loss 3.2247, time 440.37ms, mfu 143.85% |
| iter 24500: loss 3.1927, time 440.19ms, mfu 145.00% |
| iter 24600: loss 3.2176, time 441.60ms, mfu 146.00% |
| iter 24700: loss 3.1701, time 440.06ms, mfu 146.95% |
| iter 24800: loss 3.2210, time 440.44ms, mfu 147.79% |
| iter 24900: loss 3.1681, time 438.30ms, mfu 148.62% |
| iter 25000: loss 3.2443, time 441.50ms, mfu 149.25% |
| iter 25100: loss 3.2527, time 442.08ms, mfu 149.80% |
| iter 25200: loss 3.1772, time 440.98ms, mfu 150.34% |
| iter 25300: loss 3.1544, time 440.34ms, mfu 150.84% |
| iter 25400: loss 3.1639, time 439.02ms, mfu 151.34% |
| iter 25500: loss 3.1479, time 440.53ms, mfu 151.74% |
| iter 25600: loss 3.2178, time 440.46ms, mfu 152.10% |
| iter 25700: loss 3.1651, time 442.39ms, mfu 152.36% |
| iter 25800: loss 3.1976, time 440.25ms, mfu 152.66% |
| iter 25900: loss 3.1531, time 441.09ms, mfu 152.91% |
| step 26000: train loss 3.1323, val loss 3.1432 |
| saving checkpoint to out-wiki-gutenberg-530m-1ep |
| iter 26000: loss 3.2047, time 65653.33ms, mfu 137.72% |
| iter 26100: loss 3.1503, time 441.37ms, mfu 139.45% |
| iter 26200: loss 3.2074, time 440.35ms, mfu 141.04% |
| iter 26300: loss 3.2216, time 440.95ms, mfu 142.46% |
| iter 26400: loss 3.1873, time 441.01ms, mfu 143.72% |
| iter 26500: loss 3.2559, time 441.12ms, mfu 144.86% |
| iter 26600: loss 3.1597, time 442.74ms, mfu 145.83% |
| iter 26700: loss 3.1844, time 441.18ms, mfu 146.76% |
| iter 26800: loss 3.1551, time 440.31ms, mfu 147.62% |
| iter 26900: loss 3.2272, time 440.56ms, mfu 148.39% |
| iter 27000: loss 3.1884, time 439.58ms, mfu 149.11% |
| iter 27100: loss 3.2449, time 439.77ms, mfu 149.76% |
| iter 27200: loss 3.2506, time 439.02ms, mfu 150.37% |
| iter 27300: loss 3.1501, time 438.69ms, mfu 150.93% |
| iter 27400: loss 3.1378, time 440.86ms, mfu 151.36% |
| iter 27500: loss 3.1627, time 440.59ms, mfu 151.75% |
| iter 27600: loss 3.1861, time 437.84ms, mfu 152.20% |
| iter 27700: loss 3.1644, time 438.50ms, mfu 152.58% |
| iter 27800: loss 3.1470, time 442.58ms, mfu 152.78% |
| iter 27900: loss 3.1677, time 440.85ms, mfu 153.03% |
| step 28000: train loss 3.1270, val loss 3.1334 |
| saving checkpoint to out-wiki-gutenberg-530m-1ep |
| iter 28000: loss 3.1931, time 65432.01ms, mfu 137.83% |
| iter 28100: loss 3.2114, time 437.68ms, mfu 139.68% |
| iter 28200: loss 3.2457, time 437.80ms, mfu 141.34% |
| iter 28300: loss 3.1986, time 439.94ms, mfu 142.76% |
| iter 28400: loss 3.1461, time 439.26ms, mfu 144.06% |
| iter 28500: loss 3.1284, time 442.32ms, mfu 145.12% |
| iter 28600: loss 3.2687, time 439.77ms, mfu 146.17% |
| iter 28700: loss 3.2131, time 440.94ms, mfu 147.07% |
| iter 28800: loss 3.1142, time 439.01ms, mfu 147.95% |
| iter 28900: loss 3.1451, time 441.84ms, mfu 148.64% |
| iter 29000: loss 3.1589, time 440.80ms, mfu 149.29% |
| iter 29100: loss 3.1450, time 443.06ms, mfu 149.81% |
| iter 29200: loss 3.1225, time 441.04ms, mfu 150.34% |
| iter 29300: loss 3.1363, time 440.94ms, mfu 150.82% |
| iter 29400: loss 3.1213, time 441.59ms, mfu 151.24% |
| iter 29500: loss 3.1392, time 441.64ms, mfu 151.60% |
| iter 29600: loss 3.1850, time 438.42ms, mfu 152.05% |
| iter 29700: loss 3.1586, time 440.93ms, mfu 152.36% |
| iter 29800: loss 3.2396, time 443.05ms, mfu 152.57% |
| iter 29900: loss 3.2580, time 434.33ms, mfu 153.07% |
| step 30000: train loss 3.1217, val loss 3.1208 |
| saving checkpoint to out-wiki-gutenberg-530m-1ep |
| iter 30000: loss 3.2306, time 65346.39ms, mfu 137.86% |
| iter 30100: loss 3.1761, time 440.21ms, mfu 139.62% |
| iter 30200: loss 3.1694, time 436.11ms, mfu 141.35% |
| iter 30300: loss 3.1369, time 438.80ms, mfu 142.80% |
| iter 30400: loss 3.2328, time 442.19ms, mfu 144.00% |
| iter 30500: loss 3.1111, time 439.89ms, mfu 145.15% |
| iter 30600: loss 3.1187, time 435.06ms, mfu 146.36% |
| iter 30700: loss 3.1222, time 438.36ms, mfu 147.33% |
| iter 30800: loss 3.1422, time 441.10ms, mfu 148.11% |
| iter 30900: loss 3.1692, time 442.65ms, mfu 148.76% |
| iter 31000: loss 3.1235, time 440.40ms, mfu 149.42% |
| iter 31100: loss 3.1903, time 437.54ms, mfu 150.11% |
| iter 31200: loss 3.1061, time 437.73ms, mfu 150.73% |
| iter 31300: loss 3.2265, time 440.03ms, mfu 151.21% |
| iter 31400: loss 3.1683, time 440.02ms, mfu 151.64% |
| iter 31500: loss 3.1792, time 440.73ms, mfu 152.00% |
| iter 31600: loss 3.1770, time 439.67ms, mfu 152.36% |
| iter 31700: loss 3.1152, time 440.78ms, mfu 152.65% |
| iter 31800: loss 3.1440, time 440.87ms, mfu 152.90% |
| iter 31900: loss 3.1648, time 443.14ms, mfu 153.05% |
| step 32000: train loss 3.1174, val loss 3.1209 |
| saving checkpoint to out-wiki-gutenberg-530m-1ep |
| iter 32000: loss 3.2021, time 65882.30ms, mfu 137.85% |
| iter 32100: loss 3.1142, time 440.49ms, mfu 139.60% |
| iter 32200: loss 3.1869, time 440.16ms, mfu 141.18% |
| iter 32300: loss 3.1862, time 441.76ms, mfu 142.55% |
| iter 32400: loss 3.0835, time 439.22ms, mfu 143.87% |
| iter 32500: loss 3.1186, time 441.53ms, mfu 144.98% |
| iter 32600: loss 3.1332, time 440.10ms, mfu 146.03% |
| iter 32700: loss 3.1653, time 441.09ms, mfu 146.94% |
| iter 32800: loss 3.0876, time 439.37ms, mfu 147.82% |
| iter 32900: loss 3.1725, time 440.65ms, mfu 148.56% |
| iter 33000: loss 3.1411, time 438.29ms, mfu 149.32% |
| iter 33100: loss 3.1240, time 440.70ms, mfu 149.91% |
| iter 33200: loss 3.1866, time 442.01ms, mfu 150.40% |
| iter 33300: loss 3.1937, time 441.39ms, mfu 150.86% |
| iter 33400: loss 3.1438, time 441.20ms, mfu 151.28% |
| iter 33500: loss 3.1431, time 438.93ms, mfu 151.74% |
| iter 33600: loss 3.1765, time 441.70ms, mfu 152.06% |
| iter 33700: loss 3.1790, time 439.30ms, mfu 152.43% |
| iter 33800: loss 3.1901, time 440.92ms, mfu 152.70% |
| iter 33900: loss 3.1656, time 441.90ms, mfu 152.92% |
| step 34000: train loss 3.1030, val loss 3.1095 |
| saving checkpoint to out-wiki-gutenberg-530m-1ep |
| iter 34000: loss 3.1539, time 65373.02ms, mfu 137.73% |
| iter 34100: loss 3.1435, time 441.38ms, mfu 139.46% |
| iter 34200: loss 3.1812, time 442.57ms, mfu 140.97% |
| iter 34300: loss 3.1066, time 442.16ms, mfu 142.35% |
| iter 34400: loss 3.1261, time 443.71ms, mfu 143.53% |
| iter 34500: loss 3.1057, time 440.65ms, mfu 144.71% |
| iter 34600: loss 3.2013, time 441.57ms, mfu 145.73% |
| iter 34700: loss 3.1375, time 440.28ms, mfu 146.70% |
| iter 34800: loss 3.1631, time 442.39ms, mfu 147.49% |
| iter 34900: loss 3.1176, time 440.82ms, mfu 148.27% |
| iter 35000: loss 3.0846, time 440.21ms, mfu 148.98% |
| iter 35100: loss 3.0308, time 437.86ms, mfu 149.71% |
| iter 35200: loss 3.0116, time 437.97ms, mfu 150.36% |
| iter 35300: loss 3.1386, time 439.07ms, mfu 150.91% |
| iter 35400: loss 3.2202, time 437.69ms, mfu 151.45% |
| iter 35500: loss 3.1263, time 438.35ms, mfu 151.91% |
| iter 35600: loss 3.1341, time 438.88ms, mfu 152.31% |
| iter 35700: loss 3.1480, time 439.99ms, mfu 152.63% |
| iter 35800: loss 3.1823, time 437.49ms, mfu 153.01% |
| iter 35900: loss 3.1703, time 437.42ms, mfu 153.35% |
| step 36000: train loss 3.0919, val loss 3.1089 |
| saving checkpoint to out-wiki-gutenberg-530m-1ep |
| iter 36000: loss 3.0796, time 65524.32ms, mfu 138.12% |
| iter 36100: loss 3.1635, time 440.45ms, mfu 139.84% |
| iter 36200: loss 3.1232, time 442.05ms, mfu 141.33% |
| iter 36300: loss 3.2023, time 442.16ms, mfu 142.67% |
| iter 36400: loss 3.0824, time 439.02ms, mfu 143.99% |
| iter 36500: loss 3.2438, time 441.87ms, mfu 145.08% |
| iter 36600: loss 3.0976, time 440.20ms, mfu 146.11% |
| iter 36700: loss 3.1506, time 438.43ms, mfu 147.11% |
| iter 36800: loss 3.1652, time 438.18ms, mfu 148.01% |
| iter 36900: loss 3.1624, time 440.07ms, mfu 148.76% |
| iter 37000: loss 3.1852, time 438.43ms, mfu 149.49% |
| iter 37100: loss 3.1130, time 440.50ms, mfu 150.07% |
| iter 37200: loss 3.1409, time 440.45ms, mfu 150.60% |
| iter 37300: loss 3.1261, time 439.34ms, mfu 151.11% |
| iter 37400: loss 3.1568, time 439.02ms, mfu 151.59% |
| iter 37500: loss 3.2170, time 439.44ms, mfu 152.00% |
| iter 37600: loss 3.1721, time 438.55ms, mfu 152.40% |
| iter 37700: loss 3.1254, time 439.89ms, mfu 152.71% |
| iter 37800: loss 3.2083, time 438.50ms, mfu 153.04% |
| iter 37900: loss 3.1972, time 439.99ms, mfu 153.29% |
| step 38000: train loss 3.0881, val loss 3.1060 |
| saving checkpoint to out-wiki-gutenberg-530m-1ep |
| iter 38000: loss 3.0752, time 65400.61ms, mfu 138.07% |
| iter 38100: loss 3.1241, time 439.69ms, mfu 139.82% |
| iter 38200: loss 3.1406, time 440.15ms, mfu 141.38% |
| iter 38300: loss 3.1196, time 441.75ms, mfu 142.73% |
| iter 38400: loss 3.0871, time 439.76ms, mfu 144.02% |
| iter 38500: loss 3.1002, time 439.26ms, mfu 145.19% |
| iter 38600: loss 3.1627, time 442.09ms, mfu 146.15% |
| iter 38700: loss 3.1359, time 440.17ms, mfu 147.08% |
| iter 38800: loss 3.1181, time 439.34ms, mfu 147.94% |
| iter 38900: loss 3.0825, time 439.96ms, mfu 148.70% |
| iter 39000: loss 3.1528, time 441.37ms, mfu 149.33% |
| iter 39100: loss 3.1497, time 438.43ms, mfu 150.01% |
| iter 39200: loss 3.1530, time 441.22ms, mfu 150.51% |
| iter 39300: loss 3.1517, time 441.30ms, mfu 150.97% |
| iter 39400: loss 3.1346, time 438.62ms, mfu 151.47% |
| iter 39500: loss 3.2056, time 440.11ms, mfu 151.87% |
| iter 39600: loss 3.1069, time 439.89ms, mfu 152.23% |
| iter 39700: loss 3.1575, time 441.61ms, mfu 152.50% |
| iter 39800: loss 3.1291, time 439.76ms, mfu 152.81% |
| iter 39900: loss 3.1384, time 440.10ms, mfu 153.08% |
| step 40000: train loss 3.0795, val loss 3.0926 |
| saving checkpoint to out-wiki-gutenberg-530m-1ep |
| iter 40000: loss 3.1666, time 65356.60ms, mfu 137.87% |
| iter 40100: loss 3.1513, time 440.58ms, mfu 139.62% |
| iter 40200: loss 3.1636, time 438.94ms, mfu 141.24% |
| iter 40300: loss 3.1182, time 441.61ms, mfu 142.61% |
| iter 40400: loss 3.2074, time 439.74ms, mfu 143.91% |
| iter 40500: loss 3.1014, time 440.35ms, mfu 145.06% |
| iter 40600: loss 3.0895, time 441.24ms, mfu 146.06% |
| iter 40700: loss 3.1138, time 440.28ms, mfu 146.99% |
| iter 40800: loss 3.0745, time 439.96ms, mfu 147.84% |
| iter 40900: loss 3.1401, time 442.18ms, mfu 148.53% |
| iter 41000: loss 3.0290, time 441.06ms, mfu 149.19% |
| iter 41100: loss 3.1356, time 441.09ms, mfu 149.79% |
| iter 41200: loss 3.1947, time 439.15ms, mfu 150.39% |
| iter 41300: loss 3.1699, time 440.93ms, mfu 150.87% |
| iter 41400: loss 3.1631, time 441.37ms, mfu 151.28% |
| iter 41500: loss 3.0933, time 438.71ms, mfu 151.75% |
| iter 41600: loss 3.1015, time 439.08ms, mfu 152.16% |
| iter 41700: loss 3.1541, time 439.18ms, mfu 152.52% |
| iter 41800: loss 3.1475, time 440.14ms, mfu 152.81% |
| iter 41900: loss 3.1103, time 438.70ms, mfu 153.13% |
| step 42000: train loss 3.0792, val loss 3.0862 |
| saving checkpoint to out-wiki-gutenberg-530m-1ep |
| iter 42000: loss 3.2236, time 65229.12ms, mfu 137.92% |
| iter 42100: loss 3.1352, time 441.86ms, mfu 139.61% |
| iter 42200: loss 3.1045, time 441.40ms, mfu 141.15% |
| iter 42300: loss 3.1690, time 442.42ms, mfu 142.50% |
| iter 42400: loss 3.1851, time 441.56ms, mfu 143.75% |
| iter 42500: loss 3.0816, time 441.01ms, mfu 144.89% |
| iter 42600: loss 3.1736, time 441.96ms, mfu 145.88% |
| iter 42700: loss 3.1924, time 441.26ms, mfu 146.80% |
| iter 42800: loss 3.1813, time 440.16ms, mfu 147.66% |
| iter 42900: loss 3.1326, time 441.00ms, mfu 148.41% |
| iter 43000: loss 3.1395, time 441.12ms, mfu 149.08% |
| iter 43100: loss 3.1630, time 440.65ms, mfu 149.70% |
| iter 43200: loss 3.1415, time 441.35ms, mfu 150.23% |
| iter 43300: loss 3.2113, time 440.30ms, mfu 150.75% |
| iter 43400: loss 3.1254, time 441.50ms, mfu 151.17% |
| iter 43500: loss 3.1059, time 441.31ms, mfu 151.56% |
| iter 43600: loss 3.1080, time 441.81ms, mfu 151.89% |
| iter 43700: loss 3.1519, time 441.20ms, mfu 152.21% |
| iter 43800: loss 3.1158, time 441.66ms, mfu 152.48% |
| iter 43900: loss 3.1154, time 439.62ms, mfu 152.79% |
| step 44000: train loss 3.0703, val loss 3.0790 |
| saving checkpoint to out-wiki-gutenberg-530m-1ep |
| iter 44000: loss 3.1420, time 65549.70ms, mfu 137.62% |
| iter 44100: loss 3.1244, time 442.45ms, mfu 139.32% |
| iter 44200: loss 3.0865, time 439.73ms, mfu 140.95% |
| iter 44300: loss 3.1255, time 440.76ms, mfu 142.38% |
| iter 44400: loss 3.1418, time 441.55ms, mfu 143.63% |
| iter 44500: loss 3.1720, time 442.80ms, mfu 144.72% |
| iter 44600: loss 3.1223, time 440.02ms, mfu 145.80% |
| iter 44700: loss 3.1375, time 443.06ms, mfu 146.66% |
| iter 44800: loss 3.1846, time 442.80ms, mfu 147.45% |
| iter 44900: loss 3.1498, time 442.65ms, mfu 148.16% |
| iter 45000: loss 3.0990, time 440.46ms, mfu 148.88% |
| iter 45100: loss 3.1634, time 440.51ms, mfu 149.52% |
| iter 45200: loss 3.0561, time 442.28ms, mfu 150.04% |
| iter 45300: loss 3.0795, time 440.67ms, mfu 150.56% |
| iter 45400: loss 3.1325, time 440.73ms, mfu 151.03% |
| iter 45500: loss 3.0683, time 440.33ms, mfu 151.47% |
| iter 45600: loss 3.0709, time 440.97ms, mfu 151.84% |
| iter 45700: loss 3.1183, time 440.89ms, mfu 152.17% |
| iter 45800: loss 3.0583, time 439.54ms, mfu 152.52% |
| iter 45900: loss 3.1016, time 442.65ms, mfu 152.72% |
| step 46000: train loss 3.0685, val loss 3.0794 |
| saving checkpoint to out-wiki-gutenberg-530m-1ep |
| iter 46000: loss 3.1274, time 65180.68ms, mfu 137.56% |
| iter 46100: loss 3.1506, time 437.55ms, mfu 139.44% |
| iter 46200: loss 3.1104, time 437.89ms, mfu 141.12% |
| iter 46300: loss 3.0399, time 437.23ms, mfu 142.66% |
| iter 46400: loss 3.1694, time 437.96ms, mfu 144.01% |
| iter 46500: loss 3.1508, time 440.17ms, mfu 145.16% |
| iter 46600: loss 3.0851, time 443.05ms, mfu 146.08% |
| iter 46700: loss 3.2149, time 438.40ms, mfu 147.08% |
| iter 46800: loss 3.1135, time 440.46ms, mfu 147.91% |
| iter 46900: loss 3.1074, time 440.79ms, mfu 148.64% |
| iter 47000: loss 3.1327, time 440.78ms, mfu 149.30% |
| iter 47100: loss 3.0934, time 441.38ms, mfu 149.87% |
| iter 47200: loss 3.1263, time 439.12ms, mfu 150.46% |
| iter 47300: loss 3.1413, time 439.19ms, mfu 151.00% |
| iter 47400: loss 3.1168, time 439.84ms, mfu 151.45% |
| iter 47500: loss 3.1493, time 438.65ms, mfu 151.90% |
| iter 47600: loss 3.1645, time 441.57ms, mfu 152.21% |
| iter 47700: loss 3.1114, time 442.51ms, mfu 152.45% |
| iter 47800: loss 3.1660, time 441.51ms, mfu 152.70% |
| iter 47900: loss 3.0851, time 441.90ms, mfu 152.92% |
| step 48000: train loss 3.0560, val loss 3.0647 |
| saving checkpoint to out-wiki-gutenberg-530m-1ep |
| iter 48000: loss 3.0687, time 65406.66ms, mfu 137.73% |
| iter 48100: loss 3.1452, time 441.18ms, mfu 139.46% |
| iter 48200: loss 3.1037, time 438.71ms, mfu 141.11% |
| iter 48300: loss 3.1458, time 439.62ms, mfu 142.57% |
| iter 48400: loss 3.0012, time 440.53ms, mfu 143.84% |
| iter 48500: loss 3.0982, time 441.51ms, mfu 144.95% |
| iter 48600: loss 3.0754, time 439.63ms, mfu 146.02% |
| iter 48700: loss 3.2024, time 440.63ms, mfu 146.95% |
| iter 48800: loss 3.0808, time 440.69ms, mfu 147.78% |
| iter 48900: loss 3.1253, time 438.94ms, mfu 148.59% |
| iter 49000: loss 3.0566, time 440.89ms, mfu 149.25% |
| iter 49100: loss 3.1114, time 439.45ms, mfu 149.89% |
| iter 49200: loss 3.0741, time 439.79ms, mfu 150.46% |
| iter 49300: loss 3.1150, time 441.63ms, mfu 150.91% |
| iter 49400: loss 3.1475, time 440.70ms, mfu 151.34% |
| iter 49500: loss 3.0570, time 440.60ms, mfu 151.74% |
| iter 49600: loss 3.1096, time 439.08ms, mfu 152.15% |
| iter 49700: loss 3.0753, time 440.17ms, mfu 152.47% |
| iter 49800: loss 3.1272, time 439.30ms, mfu 152.80% |
| iter 49900: loss 3.0495, time 441.76ms, mfu 153.01% |
| step 50000: train loss 3.0468, val loss 3.0641 |
| saving checkpoint to out-wiki-gutenberg-530m-1ep |
| iter 50000: loss 3.1086, time 65335.35ms, mfu 137.81% |
| iter 50100: loss 3.0666, time 440.85ms, mfu 139.55% |
| iter 50200: loss 3.1276, time 439.71ms, mfu 141.16% |
| iter 50300: loss 3.0820, time 438.84ms, mfu 142.63% |
| iter 50400: loss 3.0822, time 438.22ms, mfu 143.98% |
| iter 50500: loss 3.0515, time 439.91ms, mfu 145.14% |
| iter 50600: loss 3.0729, time 440.66ms, mfu 146.15% |
| iter 50700: loss 3.0959, time 440.80ms, mfu 147.06% |
| iter 50800: loss 3.1247, time 442.07ms, mfu 147.83% |
| iter 50900: loss 3.0912, time 441.28ms, mfu 148.55% |
| iter 51000: loss 3.0463, time 443.04ms, mfu 149.14% |
| iter 51100: loss 3.1229, time 439.78ms, mfu 149.78% |
| iter 51200: loss 3.0375, time 441.10ms, mfu 150.32% |
| iter 51300: loss 3.0534, time 440.89ms, mfu 150.80% |
| iter 51400: loss 3.1410, time 441.94ms, mfu 151.20% |
| iter 51500: loss 3.1354, time 441.35ms, mfu 151.59% |
| iter 51600: loss 3.1039, time 441.20ms, mfu 151.94% |
| iter 51700: loss 3.0724, time 439.71ms, mfu 152.30% |
| iter 51800: loss 3.0448, time 440.62ms, mfu 152.60% |
| iter 51900: loss 3.0713, time 440.05ms, mfu 152.89% |
| step 52000: train loss 3.0438, val loss 3.0566 |
| saving checkpoint to out-wiki-gutenberg-530m-1ep |
| iter 52000: loss 3.0551, time 65444.41ms, mfu 137.70% |
| iter 52100: loss 3.1701, time 438.14ms, mfu 139.55% |
| iter 52200: loss 3.1098, time 437.78ms, mfu 141.22% |
| iter 52300: loss 3.1294, time 441.31ms, mfu 142.61% |
| iter 52400: loss 3.0976, time 441.41ms, mfu 143.85% |
| iter 52500: loss 3.0292, time 440.10ms, mfu 145.01% |
| iter 52600: loss 3.0322, time 439.29ms, mfu 146.08% |
| iter 52700: loss 3.0833, time 440.24ms, mfu 147.02% |
| iter 52800: loss 3.0982, time 440.34ms, mfu 147.85% |
| iter 52900: loss 3.0738, time 440.22ms, mfu 148.61% |
| iter 53000: loss 3.1700, time 440.62ms, mfu 149.28% |
| iter 53100: loss 3.0691, time 439.03ms, mfu 149.93% |
| iter 53200: loss 3.0483, time 437.08ms, mfu 150.59% |
| iter 53300: loss 3.0580, time 436.85ms, mfu 151.20% |
| iter 53400: loss 3.1423, time 437.35ms, mfu 151.72% |
| iter 53500: loss 3.0353, time 437.62ms, mfu 152.18% |
| iter 53600: loss 3.0790, time 438.44ms, mfu 152.57% |
| iter 53700: loss 3.0630, time 439.01ms, mfu 152.90% |
| iter 53800: loss 3.0492, time 436.80ms, mfu 153.27% |
| iter 53900: loss 3.1117, time 439.91ms, mfu 153.50% |
| step 54000: train loss 3.0398, val loss 3.0534 |
| saving checkpoint to out-wiki-gutenberg-530m-1ep |
| iter 54000: loss 3.1341, time 65440.45ms, mfu 138.25% |
| iter 54100: loss 3.0274, time 439.48ms, mfu 140.00% |
| iter 54200: loss 3.0515, time 440.98ms, mfu 141.51% |
| iter 54300: loss 3.0363, time 440.06ms, mfu 142.91% |
| iter 54400: loss 3.1620, time 441.17ms, mfu 144.13% |
| iter 54500: loss 3.1331, time 440.40ms, mfu 145.25% |
| iter 54600: loss 3.1015, time 439.82ms, mfu 146.28% |
| iter 54700: loss 3.1023, time 439.37ms, mfu 147.23% |
| iter 54800: loss 3.0969, time 441.32ms, mfu 148.01% |
| iter 54900: loss 3.0559, time 440.49ms, mfu 148.74% |
| iter 55000: loss 3.0933, time 439.48ms, mfu 149.43% |
| iter 55100: loss 3.0770, time 440.10ms, mfu 150.04% |
| iter 55200: loss 3.0865, time 439.85ms, mfu 150.59% |
| iter 55300: loss 3.0621, time 438.69ms, mfu 151.13% |
| iter 55400: loss 3.0879, time 438.94ms, mfu 151.60% |
| iter 55500: loss 3.0152, time 440.65ms, mfu 151.97% |
| iter 55600: loss 3.0510, time 440.87ms, mfu 152.29% |
| iter 55700: loss 3.0697, time 442.77ms, mfu 152.51% |
| iter 55800: loss 3.0890, time 436.43ms, mfu 152.94% |
| iter 55900: loss 3.0304, time 440.71ms, mfu 153.17% |
| step 56000: train loss 3.0321, val loss 3.0409 |
| saving checkpoint to out-wiki-gutenberg-530m-1ep |
| iter 56000: loss 3.0638, time 65377.31ms, mfu 137.96% |
| iter 56100: loss 3.0231, time 439.57ms, mfu 139.73% |
| iter 56200: loss 3.0997, time 439.14ms, mfu 141.34% |
| iter 56300: loss 3.1754, time 437.52ms, mfu 142.84% |
| iter 56400: loss 3.0942, time 440.29ms, mfu 144.10% |
| iter 56500: loss 3.0885, time 438.94ms, mfu 145.27% |
| iter 56600: loss 3.0345, time 439.80ms, mfu 146.30% |
| iter 56700: loss 3.0617, time 440.04ms, mfu 147.22% |
| iter 56800: loss 3.1482, time 439.09ms, mfu 148.08% |
| iter 56900: loss 3.1352, time 439.00ms, mfu 148.86% |
| iter 57000: loss 3.0218, time 435.67ms, mfu 149.68% |
| iter 57100: loss 3.0228, time 439.07ms, mfu 150.29% |
| iter 57200: loss 3.0959, time 437.88ms, mfu 150.89% |
| iter 57300: loss 3.1141, time 439.84ms, mfu 151.36% |
| iter 57400: loss 3.1529, time 438.63ms, mfu 151.82% |
| iter 57500: loss 3.0456, time 439.39ms, mfu 152.21% |
| iter 57600: loss 3.0641, time 440.19ms, mfu 152.53% |
| iter 57700: loss 3.0254, time 440.62ms, mfu 152.81% |
| iter 57800: loss 3.1110, time 437.25ms, mfu 153.17% |
| iter 57900: loss 3.0839, time 440.61ms, mfu 153.39% |
| step 58000: train loss 3.0274, val loss 3.0375 |
| saving checkpoint to out-wiki-gutenberg-530m-1ep |
| iter 58000: loss 3.1437, time 65514.69ms, mfu 138.15% |
| iter 58100: loss 3.0870, time 438.90ms, mfu 139.93% |
| iter 58200: loss 3.0118, time 441.30ms, mfu 141.44% |
| iter 58300: loss 3.0542, time 440.42ms, mfu 142.83% |
| iter 58400: loss 3.0486, time 444.09ms, mfu 143.95% |
| iter 58500: loss 3.0095, time 440.40ms, mfu 145.09% |
| iter 58600: loss 3.0491, time 442.61ms, mfu 146.04% |
| iter 58700: loss 3.0548, time 440.76ms, mfu 146.96% |
| iter 58800: loss 3.0602, time 440.24ms, mfu 147.81% |
| iter 58900: loss 3.0851, time 442.68ms, mfu 148.48% |
| iter 59000: loss 3.1125, time 441.40ms, mfu 149.13% |
| iter 59100: loss 3.1091, time 440.05ms, mfu 149.77% |
| iter 59200: loss 3.0696, time 439.04ms, mfu 150.38% |
| iter 59300: loss 3.0043, time 441.42ms, mfu 150.84% |
| iter 59400: loss 3.0714, time 440.00ms, mfu 151.31% |
| iter 59500: loss 3.0090, time 441.06ms, mfu 151.69% |
| iter 59600: loss 3.0509, time 441.22ms, mfu 152.03% |
| iter 59700: loss 2.9839, time 439.31ms, mfu 152.40% |
| iter 59800: loss 3.0458, time 442.26ms, mfu 152.63% |
| iter 59900: loss 3.0372, time 439.29ms, mfu 152.94% |
| step 60000: train loss 3.0196, val loss 3.0302 |
| saving checkpoint to out-wiki-gutenberg-530m-1ep |
| iter 60000: loss 3.0854, time 65424.17ms, mfu 137.75% |
| iter 60100: loss 3.0641, time 436.63ms, mfu 139.65% |
| iter 60200: loss 3.0030, time 438.99ms, mfu 141.27% |
| iter 60300: loss 3.0349, time 439.89ms, mfu 142.69% |
| iter 60400: loss 3.0690, time 439.14ms, mfu 144.01% |
| iter 60500: loss 3.0861, time 439.68ms, mfu 145.17% |
| iter 60600: loss 3.0207, time 440.36ms, mfu 146.19% |
| iter 60700: loss 3.1289, time 440.17ms, mfu 147.11% |
| iter 60800: loss 3.0466, time 438.89ms, mfu 147.99% |
| iter 60900: loss 3.0868, time 440.52ms, mfu 148.72% |
| iter 61000: loss 3.0582, time 439.24ms, mfu 149.43% |
| iter 61100: loss 3.1366, time 438.59ms, mfu 150.08% |
| iter 61200: loss 3.0483, time 440.14ms, mfu 150.62% |
| iter 61300: loss 3.0071, time 438.43ms, mfu 151.16% |
| iter 61400: loss 3.0812, time 439.24ms, mfu 151.63% |
| iter 61500: loss 3.0668, time 439.06ms, mfu 152.05% |
| iter 61600: loss 3.1011, time 439.12ms, mfu 152.42% |
| iter 61700: loss 3.0613, time 439.92ms, mfu 152.73% |
| iter 61800: loss 3.0067, time 441.30ms, mfu 152.96% |
| iter 61900: loss 3.0275, time 439.99ms, mfu 153.22% |
| step 62000: train loss 3.0096, val loss 3.0239 |
| saving checkpoint to out-wiki-gutenberg-530m-1ep |
| iter 62000: loss 3.1109, time 65298.10ms, mfu 138.00% |
| iter 62100: loss 3.1113, time 439.43ms, mfu 139.77% |
| iter 62200: loss 3.0218, time 441.20ms, mfu 141.30% |
| iter 62300: loss 3.0534, time 439.83ms, mfu 142.73% |
| iter 62400: loss 3.0612, time 440.07ms, mfu 144.00% |
| iter 62500: loss 3.0278, time 439.53ms, mfu 145.17% |
| iter 62600: loss 3.1147, time 438.57ms, mfu 146.25% |
| iter 62700: loss 3.0488, time 439.77ms, mfu 147.19% |
| iter 62800: loss 3.0801, time 438.67ms, mfu 148.06% |
| iter 62900: loss 3.0712, time 440.71ms, mfu 148.78% |
| iter 63000: loss 3.1122, time 438.73ms, mfu 149.50% |
| iter 63100: loss 3.1121, time 438.09ms, mfu 150.17% |
| iter 63200: loss 3.0891, time 438.62ms, mfu 150.75% |
| iter 63300: loss 3.0539, time 439.92ms, mfu 151.23% |
| iter 63400: loss 3.0250, time 440.59ms, mfu 151.63% |
| iter 63500: loss 3.0018, time 441.27ms, mfu 151.98% |
| iter 63600: loss 3.0495, time 440.77ms, mfu 152.30% |
| iter 63700: loss 3.0530, time 440.74ms, mfu 152.60% |
| iter 63800: loss 3.0913, time 441.20ms, mfu 152.84% |
| iter 63900: loss 3.1316, time 440.95ms, mfu 153.08% |
| step 64000: train loss 3.0051, val loss 3.0152 |
| saving checkpoint to out-wiki-gutenberg-530m-1ep |
| iter 64000: loss 3.0311, time 65547.11ms, mfu 137.87% |
| iter 64100: loss 3.0031, time 438.93ms, mfu 139.67% |
| iter 64200: loss 3.0113, time 437.60ms, mfu 141.34% |
| iter 64300: loss 3.0274, time 438.37ms, mfu 142.82% |
| iter 64400: loss 2.9791, time 437.53ms, mfu 144.17% |
| iter 64500: loss 3.1084, time 440.40ms, mfu 145.29% |
| iter 64600: loss 3.0577, time 437.58ms, mfu 146.40% |
| iter 64700: loss 3.0359, time 437.50ms, mfu 147.40% |
| iter 64800: loss 3.0027, time 442.06ms, mfu 148.13% |
| iter 64900: loss 3.0366, time 440.09ms, mfu 148.87% |
| iter 65000: loss 3.0392, time 440.47ms, mfu 149.51% |
| iter 65100: loss 3.0057, time 441.29ms, mfu 150.07% |
| iter 65200: loss 3.0551, time 438.87ms, mfu 150.65% |
| iter 65300: loss 3.0577, time 442.30ms, mfu 151.05% |
| iter 65400: loss 2.9782, time 440.84ms, mfu 151.47% |
| iter 65500: loss 3.0549, time 440.91ms, mfu 151.84% |
| iter 65600: loss 3.0731, time 440.99ms, mfu 152.17% |
| iter 65700: loss 3.0597, time 440.05ms, mfu 152.50% |
| iter 65800: loss 3.0591, time 439.47ms, mfu 152.82% |
| iter 65900: loss 3.0704, time 440.06ms, mfu 153.09% |
| step 66000: train loss 2.9948, val loss 3.0059 |
| saving checkpoint to out-wiki-gutenberg-530m-1ep |
| iter 66000: loss 2.9691, time 65714.02ms, mfu 137.88% |
| iter 66100: loss 2.9972, time 438.43ms, mfu 139.70% |
| iter 66200: loss 3.1161, time 437.73ms, mfu 141.36% |
| iter 66300: loss 3.0164, time 438.58ms, mfu 142.83% |
| iter 66400: loss 3.0650, time 439.98ms, mfu 144.09% |
| iter 66500: loss 3.1190, time 438.58ms, mfu 145.28% |
| iter 66600: loss 3.0110, time 438.56ms, mfu 146.36% |
| iter 66700: loss 3.0292, time 439.49ms, mfu 147.29% |
| iter 66800: loss 3.0599, time 438.23ms, mfu 148.17% |
| iter 66900: loss 3.0774, time 438.24ms, mfu 148.97% |
| iter 67000: loss 3.0405, time 438.14ms, mfu 149.69% |
| iter 67100: loss 3.0034, time 438.40ms, mfu 150.33% |
| iter 67200: loss 3.0306, time 438.40ms, mfu 150.90% |
| iter 67300: loss 3.0122, time 437.84ms, mfu 151.44% |
| iter 67400: loss 3.0555, time 438.40ms, mfu 151.90% |
| iter 67500: loss 3.0834, time 438.13ms, mfu 152.33% |
| iter 67600: loss 3.0676, time 438.21ms, mfu 152.71% |
| iter 67700: loss 3.0356, time 438.19ms, mfu 153.05% |
| iter 67800: loss 3.0711, time 437.53ms, mfu 153.38% |
| iter 67900: loss 3.0364, time 440.20ms, mfu 153.59% |
| step 68000: train loss 2.9898, val loss 3.0041 |
| saving checkpoint to out-wiki-gutenberg-530m-1ep |
| iter 68000: loss 3.0174, time 65342.47ms, mfu 138.33% |
| iter 68100: loss 3.0396, time 442.89ms, mfu 139.95% |
| iter 68200: loss 3.0384, time 438.06ms, mfu 141.57% |
| iter 68300: loss 3.0651, time 437.81ms, mfu 143.04% |
| iter 68400: loss 3.0246, time 439.26ms, mfu 144.32% |
| iter 68500: loss 3.0212, time 439.19ms, mfu 145.46% |
| iter 68600: loss 3.0281, time 438.28ms, mfu 146.53% |
| iter 68700: loss 2.9349, time 440.20ms, mfu 147.42% |
| iter 68800: loss 3.0149, time 438.29ms, mfu 148.29% |
| iter 68900: loss 2.9418, time 439.87ms, mfu 149.01% |
| iter 69000: loss 2.9298, time 440.16ms, mfu 149.66% |
| iter 69100: loss 3.0403, time 439.74ms, mfu 150.25% |
| iter 69200: loss 3.0286, time 440.03ms, mfu 150.77% |
| iter 69300: loss 3.0987, time 437.97ms, mfu 151.32% |
| iter 69400: loss 3.0637, time 439.35ms, mfu 151.76% |
| iter 69500: loss 2.9498, time 442.28ms, mfu 152.05% |
| iter 69600: loss 3.1574, time 440.45ms, mfu 152.38% |
| iter 69700: loss 3.0499, time 439.77ms, mfu 152.70% |
| iter 69800: loss 3.0068, time 442.84ms, mfu 152.88% |
| iter 69900: loss 3.0940, time 441.11ms, mfu 153.11% |
| step 70000: train loss 2.9858, val loss 2.9930 |
| saving checkpoint to out-wiki-gutenberg-530m-1ep |
| iter 70000: loss 3.0232, time 65356.00ms, mfu 137.90% |
| iter 70100: loss 3.1126, time 434.98ms, mfu 139.84% |
| iter 70200: loss 3.0199, time 438.90ms, mfu 141.44% |
| iter 70300: loss 3.0699, time 438.10ms, mfu 142.92% |
| iter 70400: loss 3.0581, time 443.47ms, mfu 144.05% |
| iter 70500: loss 2.9961, time 441.07ms, mfu 145.16% |
| iter 70600: loss 3.0263, time 438.86ms, mfu 146.24% |
| iter 70700: loss 2.9721, time 440.47ms, mfu 147.15% |
| iter 70800: loss 3.0791, time 440.17ms, mfu 147.97% |
| iter 70900: loss 3.0591, time 440.57ms, mfu 148.71% |
| iter 71000: loss 3.0493, time 441.22ms, mfu 149.34% |
| iter 71100: loss 3.0477, time 440.28ms, mfu 149.95% |
| iter 71200: loss 2.9989, time 441.75ms, mfu 150.44% |
| iter 71300: loss 2.9560, time 440.06ms, mfu 150.95% |
| iter 71400: loss 3.0193, time 440.04ms, mfu 151.40% |
| iter 71500: loss 3.0037, time 440.68ms, mfu 151.79% |
| iter 71600: loss 2.9993, time 437.52ms, mfu 152.25% |
| iter 71700: loss 3.0268, time 440.95ms, mfu 152.54% |
| iter 71800: loss 2.9997, time 439.07ms, mfu 152.87% |
| iter 71900: loss 3.0690, time 440.83ms, mfu 153.10% |
| step 72000: train loss 2.9735, val loss 2.9862 |
| saving checkpoint to out-wiki-gutenberg-530m-1ep |
| iter 72000: loss 3.0948, time 65598.84ms, mfu 137.90% |
| iter 72100: loss 3.0025, time 439.80ms, mfu 139.66% |
| iter 72200: loss 3.0854, time 436.41ms, mfu 141.37% |
| iter 72300: loss 3.0258, time 439.93ms, mfu 142.79% |
| iter 72400: loss 3.0217, time 437.84ms, mfu 144.14% |
| iter 72500: loss 3.0330, time 439.45ms, mfu 145.29% |
| iter 72600: loss 3.0372, time 439.98ms, mfu 146.31% |
| iter 72700: loss 3.0036, time 439.52ms, mfu 147.25% |
| iter 72800: loss 3.0626, time 440.25ms, mfu 148.07% |
| iter 72900: loss 3.0605, time 437.06ms, mfu 148.91% |
| iter 73000: loss 2.9687, time 441.76ms, mfu 149.51% |
| iter 73100: loss 3.0497, time 438.09ms, mfu 150.18% |
| iter 73200: loss 3.0122, time 440.22ms, mfu 150.70% |
| iter 73300: loss 3.0715, time 439.92ms, mfu 151.19% |
| iter 73400: loss 3.0639, time 439.37ms, mfu 151.64% |
| iter 73500: loss 3.0497, time 438.90ms, mfu 152.06% |
| iter 73600: loss 3.0005, time 439.80ms, mfu 152.41% |
| iter 73700: loss 3.0588, time 439.21ms, mfu 152.75% |
| iter 73800: loss 3.0406, time 439.66ms, mfu 153.04% |
| iter 73900: loss 2.9904, time 438.64ms, mfu 153.33% |
| step 74000: train loss 2.9638, val loss 2.9808 |
| saving checkpoint to out-wiki-gutenberg-530m-1ep |
| iter 74000: loss 3.0316, time 65488.44ms, mfu 138.10% |
| iter 74100: loss 3.1043, time 437.11ms, mfu 139.95% |
| iter 74200: loss 2.9795, time 437.80ms, mfu 141.58% |
| iter 74300: loss 3.0361, time 437.77ms, mfu 143.05% |
| iter 74400: loss 2.9249, time 437.77ms, mfu 144.37% |
| iter 74500: loss 3.0242, time 437.82ms, mfu 145.56% |
| iter 74600: loss 2.9896, time 436.90ms, mfu 146.67% |
| iter 74700: loss 3.0461, time 437.00ms, mfu 147.66% |
| iter 74800: loss 2.9698, time 438.96ms, mfu 148.48% |
| iter 74900: loss 3.0505, time 437.60ms, mfu 149.27% |
| iter 75000: loss 3.0266, time 436.93ms, mfu 150.00% |
| iter 75100: loss 2.9543, time 438.34ms, mfu 150.61% |
| iter 75200: loss 3.0689, time 439.56ms, mfu 151.11% |
| iter 75300: loss 3.0272, time 437.55ms, mfu 151.64% |
| iter 75400: loss 3.0343, time 438.84ms, mfu 152.07% |
| iter 75500: loss 2.9909, time 438.34ms, mfu 152.47% |
| iter 75600: loss 2.9259, time 433.45ms, mfu 153.01% |
| iter 75700: loss 2.9975, time 437.86ms, mfu 153.33% |
| iter 75800: loss 2.9459, time 437.86ms, mfu 153.63% |
| iter 75900: loss 3.0009, time 437.00ms, mfu 153.92% |
| step 76000: train loss 2.9573, val loss 2.9723 |
| saving checkpoint to out-wiki-gutenberg-530m-1ep |
| iter 76000: loss 2.9639, time 65298.98ms, mfu 138.63% |
| iter 76100: loss 2.9857, time 437.93ms, mfu 140.39% |
| iter 76200: loss 2.9169, time 438.19ms, mfu 141.97% |
| iter 76300: loss 3.0461, time 438.74ms, mfu 143.37% |
| iter 76400: loss 3.0179, time 437.18ms, mfu 144.68% |
| iter 76500: loss 3.0035, time 437.48ms, mfu 145.85% |
| iter 76600: loss 2.9923, time 438.13ms, mfu 146.88% |
| iter 76700: loss 3.0297, time 438.27ms, mfu 147.81% |
| iter 76800: loss 2.9872, time 439.00ms, mfu 148.61% |
| iter 76900: loss 3.0313, time 438.51ms, mfu 149.35% |
| iter 77000: loss 2.9735, time 438.09ms, mfu 150.03% |
| iter 77100: loss 3.0751, time 436.92ms, mfu 150.69% |
| iter 77200: loss 2.9456, time 437.85ms, mfu 151.25% |
| iter 77300: loss 2.9293, time 438.58ms, mfu 151.72% |
| iter 77400: loss 2.9897, time 437.26ms, mfu 152.20% |
| iter 77500: loss 3.0412, time 437.71ms, mfu 152.61% |
| iter 77600: loss 3.0513, time 437.83ms, mfu 152.98% |
| iter 77700: loss 3.0317, time 438.25ms, mfu 153.29% |
| iter 77800: loss 3.0361, time 437.38ms, mfu 153.61% |
| iter 77900: loss 2.9983, time 438.15ms, mfu 153.86% |
| step 78000: train loss 2.9514, val loss 2.9628 |
| saving checkpoint to out-wiki-gutenberg-530m-1ep |
| iter 78000: loss 3.0739, time 65598.55ms, mfu 138.58% |
| iter 78100: loss 3.0574, time 437.45ms, mfu 140.36% |
| iter 78200: loss 2.9734, time 437.92ms, mfu 141.95% |
| iter 78300: loss 3.0352, time 439.09ms, mfu 143.34% |
| iter 78400: loss 2.9283, time 438.14ms, mfu 144.62% |
| iter 78500: loss 3.0011, time 437.72ms, mfu 145.79% |
| iter 78600: loss 2.9932, time 438.82ms, mfu 146.80% |
| iter 78700: loss 3.0094, time 436.45ms, mfu 147.80% |
| iter 78800: loss 2.9800, time 438.74ms, mfu 148.61% |
| iter 78900: loss 2.9542, time 440.57ms, mfu 149.28% |
| iter 79000: loss 2.9856, time 438.44ms, mfu 149.96% |
| iter 79100: loss 2.9302, time 439.43ms, mfu 150.53% |
| iter 79200: loss 2.9779, time 441.39ms, mfu 150.98% |
| iter 79300: loss 3.0164, time 440.53ms, mfu 151.41% |
| iter 79400: loss 2.9777, time 442.27ms, mfu 151.74% |
| iter 79500: loss 3.0307, time 440.36ms, mfu 152.11% |
| iter 79600: loss 3.0342, time 435.90ms, mfu 152.59% |
| iter 79700: loss 3.0273, time 440.87ms, mfu 152.85% |
| iter 79800: loss 3.0280, time 437.85ms, mfu 153.19% |
| iter 79900: loss 2.9910, time 439.48ms, mfu 153.44% |
| step 80000: train loss 2.9408, val loss 2.9569 |
| saving checkpoint to out-wiki-gutenberg-530m-1ep |
| iter 80000: loss 2.9765, time 65411.26ms, mfu 138.20% |
| iter 80100: loss 3.0190, time 439.59ms, mfu 139.95% |
| iter 80200: loss 2.9609, time 440.15ms, mfu 141.50% |
| iter 80300: loss 3.0224, time 439.79ms, mfu 142.90% |
| iter 80400: loss 3.0200, time 438.31ms, mfu 144.22% |
| iter 80500: loss 2.9863, time 440.80ms, mfu 145.32% |
| iter 80600: loss 2.9867, time 438.36ms, mfu 146.40% |
| iter 80700: loss 2.9357, time 438.45ms, mfu 147.36% |
| iter 80800: loss 2.9972, time 438.61ms, mfu 148.23% |
| iter 80900: loss 3.0588, time 439.56ms, mfu 148.97% |
| iter 81000: loss 3.0695, time 437.13ms, mfu 149.72% |
| iter 81100: loss 2.9412, time 439.88ms, mfu 150.31% |
| iter 81200: loss 3.0060, time 438.25ms, mfu 150.89% |
| iter 81300: loss 3.0475, time 440.86ms, mfu 151.32% |
| iter 81400: loss 3.0078, time 442.07ms, mfu 151.66% |
| iter 81500: loss 2.9458, time 440.70ms, mfu 152.02% |
| iter 81600: loss 3.0411, time 440.26ms, mfu 152.36% |
| iter 81700: loss 3.0199, time 442.20ms, mfu 152.60% |
| iter 81800: loss 2.9923, time 441.32ms, mfu 152.84% |
| iter 81900: loss 2.9766, time 442.32ms, mfu 153.03% |
| step 82000: train loss 2.9362, val loss 2.9501 |
| saving checkpoint to out-wiki-gutenberg-530m-1ep |
| iter 82000: loss 2.9465, time 65582.34ms, mfu 137.83% |
| iter 82100: loss 2.9394, time 439.73ms, mfu 139.60% |
| iter 82200: loss 3.0818, time 440.35ms, mfu 141.18% |
| iter 82300: loss 3.0278, time 436.19ms, mfu 142.75% |
| iter 82400: loss 2.9714, time 439.33ms, mfu 144.05% |
| iter 82500: loss 2.8894, time 438.94ms, mfu 145.23% |
| iter 82600: loss 2.9588, time 438.57ms, mfu 146.31% |
| iter 82700: loss 3.0096, time 439.47ms, mfu 147.25% |
| iter 82800: loss 2.9659, time 438.53ms, mfu 148.12% |
| iter 82900: loss 2.9508, time 441.04ms, mfu 148.83% |
| iter 83000: loss 3.0372, time 440.44ms, mfu 149.48% |
| iter 83100: loss 3.0274, time 441.16ms, mfu 150.04% |
| iter 83200: loss 2.9807, time 438.32ms, mfu 150.64% |
| iter 83300: loss 2.9766, time 438.18ms, mfu 151.19% |
| iter 83400: loss 2.9950, time 441.22ms, mfu 151.58% |
| iter 83500: loss 2.9684, time 440.90ms, mfu 151.94% |
| iter 83600: loss 3.0050, time 440.87ms, mfu 152.27% |
| iter 83700: loss 2.9657, time 441.82ms, mfu 152.53% |
| iter 83800: loss 2.9481, time 439.63ms, mfu 152.84% |
| iter 83900: loss 2.9321, time 439.32ms, mfu 153.13% |
| step 84000: train loss 2.9262, val loss 2.9444 |
| saving checkpoint to out-wiki-gutenberg-530m-1ep |
| iter 84000: loss 3.0083, time 65493.75ms, mfu 137.92% |
| iter 84100: loss 2.9745, time 441.36ms, mfu 139.63% |
| iter 84200: loss 2.9842, time 441.42ms, mfu 141.17% |
| iter 84300: loss 2.9390, time 441.32ms, mfu 142.55% |
| iter 84400: loss 2.9650, time 442.82ms, mfu 143.75% |
| iter 84500: loss 2.9894, time 441.60ms, mfu 144.87% |
| iter 84600: loss 3.0643, time 437.55ms, mfu 146.02% |
| iter 84700: loss 2.9741, time 441.67ms, mfu 146.91% |
| iter 84800: loss 2.9471, time 440.87ms, mfu 147.74% |
| iter 84900: loss 3.0156, time 439.49ms, mfu 148.53% |
| iter 85000: loss 2.9154, time 440.09ms, mfu 149.23% |
| iter 85100: loss 2.9607, time 441.57ms, mfu 149.80% |
| iter 85200: loss 3.0374, time 442.13ms, mfu 150.29% |
| iter 85300: loss 2.9413, time 438.47ms, mfu 150.87% |
| iter 85400: loss 2.9965, time 438.95ms, mfu 151.37% |
| iter 85500: loss 2.9431, time 439.23ms, mfu 151.81% |
| iter 85600: loss 2.9472, time 441.03ms, mfu 152.14% |
| iter 85700: loss 2.9410, time 441.67ms, mfu 152.42% |
| iter 85800: loss 2.9860, time 441.14ms, mfu 152.69% |
| iter 85900: loss 2.9026, time 437.76ms, mfu 153.05% |
| step 86000: train loss 2.9146, val loss 2.9330 |
| saving checkpoint to out-wiki-gutenberg-530m-1ep |
| iter 86000: loss 2.9082, time 65547.86ms, mfu 137.85% |
| iter 86100: loss 2.9350, time 438.56ms, mfu 139.66% |
| iter 86200: loss 2.9690, time 437.37ms, mfu 141.34% |
| iter 86300: loss 2.9492, time 437.77ms, mfu 142.84% |
| iter 86400: loss 2.9826, time 436.61ms, mfu 144.22% |
| iter 86500: loss 2.9921, time 439.23ms, mfu 145.38% |
| iter 86600: loss 2.8777, time 434.29ms, mfu 146.59% |
| iter 86700: loss 2.9166, time 438.70ms, mfu 147.53% |
| iter 86800: loss 2.9873, time 438.81ms, mfu 148.37% |
| iter 86900: loss 2.9912, time 438.29ms, mfu 149.14% |
| iter 87000: loss 2.9583, time 440.15ms, mfu 149.77% |
| iter 87100: loss 2.9258, time 441.78ms, mfu 150.28% |
| iter 87200: loss 2.9614, time 439.83ms, mfu 150.81% |
| iter 87300: loss 3.0091, time 440.73ms, mfu 151.26% |
| iter 87400: loss 2.9660, time 439.67ms, mfu 151.69% |
| iter 87500: loss 2.9393, time 438.90ms, mfu 152.11% |
| iter 87600: loss 2.9854, time 438.23ms, mfu 152.51% |
| iter 87700: loss 3.0175, time 439.13ms, mfu 152.84% |
| iter 87800: loss 2.9602, time 440.80ms, mfu 153.08% |
| iter 87900: loss 2.9994, time 441.11ms, mfu 153.28% |
| step 88000: train loss 2.9052, val loss 2.9267 |
| saving checkpoint to out-wiki-gutenberg-530m-1ep |
| iter 88000: loss 2.9541, time 65709.25ms, mfu 138.06% |
| iter 88100: loss 2.9144, time 440.60ms, mfu 139.78% |
| iter 88200: loss 2.9477, time 436.93ms, mfu 141.46% |
| iter 88300: loss 2.9622, time 440.49ms, mfu 142.85% |
| iter 88400: loss 2.9708, time 440.44ms, mfu 144.10% |
| iter 88500: loss 3.0075, time 441.13ms, mfu 145.20% |
| iter 88600: loss 2.9409, time 440.86ms, mfu 146.20% |
| iter 88700: loss 2.8936, time 440.64ms, mfu 147.11% |
| iter 88800: loss 2.9464, time 438.66ms, mfu 147.99% |
| iter 88900: loss 2.9645, time 438.04ms, mfu 148.81% |
| iter 89000: loss 2.9264, time 437.94ms, mfu 149.56% |
| iter 89100: loss 3.0051, time 438.51ms, mfu 150.20% |
| iter 89200: loss 2.9246, time 441.27ms, mfu 150.69% |
| iter 89300: loss 2.8754, time 441.59ms, mfu 151.11% |
| iter 89400: loss 2.9401, time 439.79ms, mfu 151.56% |
| iter 89500: loss 2.9174, time 439.98ms, mfu 151.95% |
| iter 89600: loss 2.9097, time 442.98ms, mfu 152.20% |
| iter 89700: loss 2.9532, time 441.70ms, mfu 152.47% |
| iter 89800: loss 2.9357, time 440.28ms, mfu 152.77% |
| iter 89900: loss 2.9332, time 440.80ms, mfu 153.01% |
| step 90000: train loss 2.9026, val loss 2.9190 |
| saving checkpoint to out-wiki-gutenberg-530m-1ep |
| iter 90000: loss 2.9688, time 65412.83ms, mfu 137.82% |
| iter 90100: loss 2.9655, time 441.77ms, mfu 139.52% |
| iter 90200: loss 2.9756, time 442.40ms, mfu 141.04% |
| iter 90300: loss 2.9108, time 442.07ms, mfu 142.41% |
| iter 90400: loss 2.9303, time 440.97ms, mfu 143.68% |
| iter 90500: loss 2.9480, time 442.29ms, mfu 144.79% |
| iter 90600: loss 2.9380, time 441.09ms, mfu 145.82% |
| iter 90700: loss 2.8538, time 440.57ms, mfu 146.77% |
| iter 90800: loss 2.9550, time 439.72ms, mfu 147.65% |
| iter 90900: loss 2.8678, time 442.74ms, mfu 148.34% |
| iter 91000: loss 2.9443, time 441.51ms, mfu 149.00% |
| iter 91100: loss 2.9807, time 441.07ms, mfu 149.61% |
| iter 91200: loss 2.9519, time 442.53ms, mfu 150.11% |
| iter 91300: loss 2.9118, time 443.00ms, mfu 150.55% |
| iter 91400: loss 2.8929, time 441.83ms, mfu 150.98% |
| iter 91500: loss 3.0036, time 441.27ms, mfu 151.39% |
| iter 91600: loss 2.8623, time 440.88ms, mfu 151.77% |
| iter 91700: loss 3.0222, time 440.24ms, mfu 152.13% |
| iter 91800: loss 2.9247, time 439.33ms, mfu 152.49% |
| iter 91900: loss 2.9979, time 441.10ms, mfu 152.75% |
| step 92000: train loss 2.8951, val loss 2.9064 |
| saving checkpoint to out-wiki-gutenberg-530m-1ep |
| iter 92000: loss 2.9599, time 65688.85ms, mfu 137.58% |
| iter 92100: loss 2.9686, time 439.00ms, mfu 139.41% |
| iter 92200: loss 2.9342, time 440.28ms, mfu 141.01% |
| iter 92300: loss 2.9535, time 436.45ms, mfu 142.58% |
| iter 92400: loss 2.9564, time 439.53ms, mfu 143.89% |
| iter 92500: loss 2.9179, time 438.95ms, mfu 145.09% |
| iter 92600: loss 2.9405, time 440.72ms, mfu 146.11% |
| iter 92700: loss 3.0298, time 440.05ms, mfu 147.04% |
| iter 92800: loss 2.9647, time 440.84ms, mfu 147.86% |
| iter 92900: loss 2.9731, time 440.29ms, mfu 148.61% |
| iter 93000: loss 2.8835, time 442.02ms, mfu 149.23% |
| iter 93100: loss 2.9397, time 440.14ms, mfu 149.85% |
| iter 93200: loss 2.9325, time 439.87ms, mfu 150.42% |
| iter 93300: loss 2.9509, time 440.69ms, mfu 150.91% |
| iter 93400: loss 2.9944, time 440.90ms, mfu 151.33% |
| iter 93500: loss 2.9185, time 440.65ms, mfu 151.73% |
| iter 93600: loss 2.9569, time 440.10ms, mfu 152.10% |
| iter 93700: loss 2.9348, time 440.70ms, mfu 152.42% |
| iter 93800: loss 2.9680, time 439.96ms, mfu 152.73% |
| iter 93900: loss 2.9438, time 441.06ms, mfu 152.97% |
| step 94000: train loss 2.8807, val loss 2.9015 |
| saving checkpoint to out-wiki-gutenberg-530m-1ep |
| iter 94000: loss 2.9329, time 65846.10ms, mfu 137.77% |
| iter 94100: loss 2.9846, time 441.87ms, mfu 139.48% |
| iter 94200: loss 2.9531, time 441.94ms, mfu 141.01% |
| iter 94300: loss 2.9500, time 440.87ms, mfu 142.43% |
| iter 94400: loss 2.9647, time 443.10ms, mfu 143.63% |
| iter 94500: loss 2.9319, time 441.18ms, mfu 144.78% |
| iter 94600: loss 2.9485, time 440.18ms, mfu 145.84% |
| iter 94700: loss 3.0074, time 441.72ms, mfu 146.75% |
| iter 94800: loss 2.9065, time 441.31ms, mfu 147.58% |
| iter 94900: loss 2.9364, time 440.77ms, mfu 148.34% |
| iter 95000: loss 2.9349, time 441.64ms, mfu 149.00% |
| iter 95100: loss 2.9150, time 438.78ms, mfu 149.69% |
| iter 95200: loss 2.9206, time 437.76ms, mfu 150.35% |
| iter 95300: loss 2.9083, time 438.51ms, mfu 150.92% |
| iter 95400: loss 2.9255, time 437.82ms, mfu 151.46% |
| iter 95500: loss 2.9246, time 438.02ms, mfu 151.93% |
| iter 95600: loss 3.0054, time 438.74ms, mfu 152.33% |
| iter 95700: loss 2.9140, time 442.86ms, mfu 152.55% |
| iter 95800: loss 2.9589, time 440.32ms, mfu 152.83% |
| iter 95900: loss 2.8783, time 439.88ms, mfu 153.10% |
| step 96000: train loss 2.8689, val loss 2.8914 |
| saving checkpoint to out-wiki-gutenberg-530m-1ep |
| iter 96000: loss 2.8810, time 65397.45ms, mfu 137.90% |
| iter 96100: loss 2.9268, time 441.61ms, mfu 139.60% |
| iter 96200: loss 2.9547, time 442.89ms, mfu 141.09% |
| iter 96300: loss 2.9126, time 441.65ms, mfu 142.47% |
| iter 96400: loss 2.9130, time 441.00ms, mfu 143.74% |
| iter 96500: loss 2.9455, time 439.69ms, mfu 144.93% |
| iter 96600: loss 2.9946, time 441.06ms, mfu 145.95% |
| iter 96700: loss 2.9099, time 441.26ms, mfu 146.86% |
| iter 96800: loss 2.9280, time 441.13ms, mfu 147.68% |
| iter 96900: loss 2.8698, time 441.52ms, mfu 148.41% |
| iter 97000: loss 2.9649, time 438.83ms, mfu 149.16% |
| iter 97100: loss 2.8705, time 434.39ms, mfu 150.00% |
| iter 97200: loss 2.8748, time 440.30ms, mfu 150.54% |
| iter 97300: loss 2.9543, time 439.79ms, mfu 151.04% |
| iter 97400: loss 2.9275, time 438.32ms, mfu 151.55% |
| iter 97500: loss 2.9206, time 442.56ms, mfu 151.85% |
| iter 97600: loss 2.9960, time 438.72ms, mfu 152.26% |
| iter 97700: loss 2.8706, time 441.10ms, mfu 152.55% |
| iter 97800: loss 2.9292, time 441.47ms, mfu 152.79% |
| iter 97900: loss 2.9589, time 439.12ms, mfu 153.09% |
| step 98000: train loss 2.8713, val loss 2.8859 |
| saving checkpoint to out-wiki-gutenberg-530m-1ep |
| iter 98000: loss 2.9433, time 65360.28ms, mfu 137.89% |
| iter 98100: loss 2.9236, time 437.82ms, mfu 139.73% |
| iter 98200: loss 2.9142, time 437.63ms, mfu 141.39% |
| iter 98300: loss 2.9327, time 439.98ms, mfu 142.80% |
| iter 98400: loss 2.8660, time 441.16ms, mfu 144.03% |
| iter 98500: loss 2.9172, time 438.12ms, mfu 145.24% |
| iter 98600: loss 2.8958, time 441.72ms, mfu 146.21% |
| iter 98700: loss 2.9251, time 438.77ms, mfu 147.18% |
| iter 98800: loss 3.0009, time 441.02ms, mfu 147.98% |
| iter 98900: loss 2.9627, time 440.88ms, mfu 148.70% |
| iter 99000: loss 2.9117, time 438.90ms, mfu 149.42% |
| iter 99100: loss 2.9761, time 442.32ms, mfu 149.94% |
| iter 99200: loss 2.9181, time 439.01ms, mfu 150.54% |
| iter 99300: loss 2.9122, time 441.91ms, mfu 150.96% |
| iter 99400: loss 2.9218, time 438.08ms, mfu 151.49% |
| iter 99500: loss 2.8803, time 439.53ms, mfu 151.90% |
| iter 99600: loss 2.9193, time 439.50ms, mfu 152.28% |
| iter 99700: loss 2.8830, time 441.20ms, mfu 152.56% |
| iter 99800: loss 2.8586, time 440.78ms, mfu 152.83% |
| iter 99900: loss 2.8837, time 442.08ms, mfu 153.02% |
| step 100000: train loss 2.8606, val loss 2.8797 |
| saving checkpoint to out-wiki-gutenberg-530m-1ep |
| iter 100000: loss 2.9445, time 65533.23ms, mfu 137.82% |
| iter 100100: loss 2.8718, time 439.55ms, mfu 139.61% |
| iter 100200: loss 2.8828, time 440.85ms, mfu 141.17% |
| iter 100300: loss 2.9010, time 440.61ms, mfu 142.58% |
| iter 100400: loss 2.9367, time 441.15ms, mfu 143.83% |
| iter 100500: loss 2.9438, time 440.24ms, mfu 144.99% |
| iter 100600: loss 2.8464, time 441.28ms, mfu 146.00% |
| iter 100700: loss 2.9042, time 440.06ms, mfu 146.94% |
| iter 100800: loss 2.9074, time 439.75ms, mfu 147.81% |
| iter 100900: loss 2.8091, time 441.73ms, mfu 148.52% |
| iter 101000: loss 2.8996, time 441.06ms, mfu 149.18% |
| iter 101100: loss 2.8789, time 439.35ms, mfu 149.83% |
| iter 101200: loss 2.8777, time 440.51ms, mfu 150.38% |
| iter 101300: loss 2.8334, time 440.83ms, mfu 150.86% |
| iter 101400: loss 2.8216, time 440.47ms, mfu 151.31% |
| iter 101500: loss 2.8976, time 442.36ms, mfu 151.65% |
| iter 101600: loss 2.8487, time 440.83ms, mfu 152.00% |
| iter 101700: loss 2.9260, time 439.90ms, mfu 152.36% |
| iter 101800: loss 2.8934, time 440.74ms, mfu 152.64% |
| iter 101900: loss 2.9018, time 438.32ms, mfu 152.99% |
| step 102000: train loss 2.8502, val loss 2.8641 |
| saving checkpoint to out-wiki-gutenberg-530m-1ep |
| iter 102000: loss 2.8634, time 65283.53ms, mfu 137.80% |
| iter 102100: loss 2.8315, time 440.46ms, mfu 139.55% |
| iter 102200: loss 2.8964, time 438.88ms, mfu 141.18% |
| iter 102300: loss 2.8236, time 438.63ms, mfu 142.66% |
| iter 102400: loss 2.8673, time 439.93ms, mfu 143.95% |
| iter 102500: loss 2.8717, time 440.87ms, mfu 145.08% |
| iter 102600: loss 2.9232, time 441.14ms, mfu 146.08% |
| iter 102700: loss 2.9135, time 438.75ms, mfu 147.06% |
| iter 102800: loss 2.8939, time 439.90ms, mfu 147.91% |
| iter 102900: loss 2.8452, time 440.20ms, mfu 148.66% |
| iter 103000: loss 2.9538, time 440.85ms, mfu 149.32% |
| iter 103100: loss 2.8981, time 440.72ms, mfu 149.91% |
| iter 103200: loss 2.8443, time 440.97ms, mfu 150.43% |
| iter 103300: loss 2.8879, time 440.18ms, mfu 150.93% |
| iter 103400: loss 2.9201, time 440.99ms, mfu 151.36% |
| iter 103500: loss 2.9173, time 441.14ms, mfu 151.73% |
| iter 103600: loss 2.9144, time 442.33ms, mfu 152.03% |
| iter 103700: loss 2.9383, time 435.56ms, mfu 152.53% |
| iter 103800: loss 2.8762, time 444.99ms, mfu 152.65% |
| iter 103900: loss 2.8043, time 437.58ms, mfu 153.02% |
| step 104000: train loss 2.8393, val loss 2.8539 |
| saving checkpoint to out-wiki-gutenberg-530m-1ep |
| iter 104000: loss 2.8683, time 65516.72ms, mfu 137.83% |
| iter 104100: loss 2.8933, time 436.07ms, mfu 139.73% |
| iter 104200: loss 2.9221, time 440.92ms, mfu 141.28% |
| iter 104300: loss 2.8598, time 441.44ms, mfu 142.65% |
| iter 104400: loss 2.8540, time 438.76ms, mfu 143.98% |
| iter 104500: loss 2.8742, time 441.23ms, mfu 145.09% |
| iter 104600: loss 2.8288, time 442.52ms, mfu 146.04% |
| iter 104700: loss 2.8813, time 440.89ms, mfu 146.95% |
| iter 104800: loss 2.8738, time 440.62ms, mfu 147.79% |
| iter 104900: loss 2.8902, time 439.91ms, mfu 148.56% |
| iter 105000: loss 2.8792, time 441.89ms, mfu 149.19% |
| iter 105100: loss 2.8769, time 441.90ms, mfu 149.75% |
| iter 105200: loss 2.8926, time 441.14ms, mfu 150.29% |
| iter 105300: loss 2.8640, time 438.58ms, mfu 150.86% |
| iter 105400: loss 2.8965, time 440.09ms, mfu 151.32% |
| iter 105500: loss 2.8628, time 443.68ms, mfu 151.61% |
| iter 105600: loss 2.9195, time 442.92ms, mfu 151.90% |
| iter 105700: loss 2.9042, time 440.99ms, mfu 152.22% |
| iter 105800: loss 2.8231, time 439.44ms, mfu 152.57% |
| iter 105900: loss 2.8982, time 442.37ms, mfu 152.78% |
| step 106000: train loss 2.8309, val loss 2.8478 |
| saving checkpoint to out-wiki-gutenberg-530m-1ep |
| iter 106000: loss 2.8691, time 65609.33ms, mfu 137.61% |
| iter 106100: loss 2.8959, time 440.65ms, mfu 139.37% |
| iter 106200: loss 2.9283, time 439.90ms, mfu 140.99% |
| iter 106300: loss 2.9250, time 440.80ms, mfu 142.41% |
| iter 106400: loss 2.8535, time 441.25ms, mfu 143.68% |
| iter 106500: loss 2.8930, time 440.56ms, mfu 144.84% |
| iter 106600: loss 2.9439, time 441.26ms, mfu 145.86% |
| iter 106700: loss 2.8818, time 440.05ms, mfu 146.82% |
| iter 106800: loss 2.8768, time 440.56ms, mfu 147.67% |
| iter 106900: loss 2.8946, time 441.53ms, mfu 148.40% |
| iter 107000: loss 2.9425, time 441.62ms, mfu 149.05% |
| iter 107100: loss 2.9048, time 439.78ms, mfu 149.71% |
| iter 107200: loss 2.9147, time 441.06ms, mfu 150.25% |
| iter 107300: loss 2.9014, time 439.32ms, mfu 150.80% |
| iter 107400: loss 2.8473, time 439.92ms, mfu 151.27% |
| iter 107500: loss 2.8262, time 442.30ms, mfu 151.61% |
| iter 107600: loss 2.8969, time 438.53ms, mfu 152.05% |
| iter 107700: loss 2.9108, time 441.37ms, mfu 152.35% |
| iter 107800: loss 2.8964, time 440.98ms, mfu 152.63% |
| iter 107900: loss 2.8772, time 441.58ms, mfu 152.86% |
| step 108000: train loss 2.8179, val loss 2.8391 |
| saving checkpoint to out-wiki-gutenberg-530m-1ep |
| iter 108000: loss 2.9332, time 65294.80ms, mfu 137.68% |
| iter 108100: loss 2.9012, time 440.60ms, mfu 139.44% |
| iter 108200: loss 2.9023, time 440.68ms, mfu 141.02% |
| iter 108300: loss 2.8371, time 440.97ms, mfu 142.44% |
| iter 108400: loss 2.8989, time 443.29ms, mfu 143.63% |
| iter 108500: loss 2.8963, time 439.92ms, mfu 144.82% |
| iter 108600: loss 2.8682, time 438.51ms, mfu 145.94% |
| iter 108700: loss 2.9415, time 440.50ms, mfu 146.88% |
| iter 108800: loss 2.9263, time 440.74ms, mfu 147.71% |
| iter 108900: loss 2.8211, time 440.43ms, mfu 148.48% |
| iter 109000: loss 2.8775, time 442.16ms, mfu 149.10% |
| iter 109100: loss 2.8797, time 441.23ms, mfu 149.70% |
| iter 109200: loss 2.8588, time 439.91ms, mfu 150.28% |
| iter 109300: loss 2.8618, time 439.47ms, mfu 150.82% |
| iter 109400: loss 2.8647, time 441.18ms, mfu 151.25% |
| iter 109500: loss 2.7807, time 440.38ms, mfu 151.66% |
| iter 109600: loss 2.8184, time 441.41ms, mfu 152.00% |
| iter 109700: loss 2.8834, time 441.13ms, mfu 152.31% |
| iter 109800: loss 2.9062, time 439.78ms, mfu 152.63% |
| iter 109900: loss 2.8404, time 439.23ms, mfu 152.95% |
| step 110000: train loss 2.8121, val loss 2.8344 |
| saving checkpoint to out-wiki-gutenberg-530m-1ep |
| iter 110000: loss 2.9468, time 65452.59ms, mfu 137.76% |
| iter 110100: loss 2.8952, time 440.41ms, mfu 139.52% |
| iter 110200: loss 2.8594, time 442.24ms, mfu 141.04% |
| iter 110300: loss 2.8343, time 441.45ms, mfu 142.43% |
| iter 110400: loss 2.8454, time 440.00ms, mfu 143.74% |
| iter 110500: loss 2.8636, time 441.09ms, mfu 144.88% |
| iter 110600: loss 2.8063, time 440.61ms, mfu 145.92% |
| iter 110700: loss 2.9544, time 443.86ms, mfu 146.74% |
| iter 110800: loss 2.8225, time 441.31ms, mfu 147.57% |
| iter 110900: loss 2.9179, time 442.33ms, mfu 148.28% |
| iter 111000: loss 2.8416, time 440.25ms, mfu 148.99% |
| iter 111100: loss 2.8758, time 442.93ms, mfu 149.54% |
| iter 111200: loss 2.8718, time 440.38ms, mfu 150.13% |
| iter 111300: loss 2.9011, time 439.00ms, mfu 150.70% |
| iter 111400: loss 2.8385, time 442.41ms, mfu 151.09% |
| iter 111500: loss 2.8232, time 444.76ms, mfu 151.37% |
| iter 111600: loss 2.9330, time 441.64ms, mfu 151.72% |
| iter 111700: loss 2.8223, time 440.51ms, mfu 152.08% |
| iter 111800: loss 2.8453, time 440.18ms, mfu 152.42% |
| iter 111900: loss 2.7992, time 440.96ms, mfu 152.69% |
| step 112000: train loss 2.8069, val loss 2.8247 |
| saving checkpoint to out-wiki-gutenberg-530m-1ep |
| iter 112000: loss 2.8589, time 65653.75ms, mfu 137.53% |
| iter 112100: loss 2.8063, time 433.34ms, mfu 139.56% |
| iter 112200: loss 2.8442, time 441.00ms, mfu 141.12% |
| iter 112300: loss 2.7892, time 435.95ms, mfu 142.70% |
| iter 112400: loss 2.8514, time 443.14ms, mfu 143.87% |
| iter 112500: loss 2.8236, time 442.51ms, mfu 144.95% |
| iter 112600: loss 2.8906, time 442.76ms, mfu 145.91% |
| iter 112700: loss 2.8318, time 439.14ms, mfu 146.90% |
| iter 112800: loss 2.8489, time 440.93ms, mfu 147.72% |
| iter 112900: loss 2.8398, time 440.04ms, mfu 148.50% |
| iter 113000: loss 2.8333, time 439.81ms, mfu 149.21% |
| iter 113100: loss 2.8870, time 439.47ms, mfu 149.86% |
| iter 113200: loss 2.8974, time 440.93ms, mfu 150.39% |
| iter 113300: loss 2.8274, time 440.48ms, mfu 150.88% |
| iter 113400: loss 2.9033, time 439.30ms, mfu 151.37% |
| iter 113500: loss 2.8073, time 441.36ms, mfu 151.73% |
| iter 113600: loss 2.8538, time 442.87ms, mfu 152.01% |
| iter 113700: loss 2.8755, time 439.71ms, mfu 152.37% |
| iter 113800: loss 2.8030, time 442.40ms, mfu 152.60% |
| iter 113900: loss 2.8522, time 437.92ms, mfu 152.96% |
| step 114000: train loss 2.7994, val loss 2.8177 |
| saving checkpoint to out-wiki-gutenberg-530m-1ep |
| iter 114000: loss 2.7186, time 65533.58ms, mfu 137.77% |
| iter 114100: loss 2.8597, time 443.73ms, mfu 139.41% |
| iter 114200: loss 2.7810, time 440.44ms, mfu 141.01% |
| iter 114300: loss 2.9057, time 439.41ms, mfu 142.48% |
| iter 114400: loss 2.8561, time 441.48ms, mfu 143.73% |
| iter 114500: loss 2.8427, time 440.72ms, mfu 144.88% |
| iter 114600: loss 2.8784, time 441.59ms, mfu 145.88% |
| iter 114700: loss 2.8019, time 442.30ms, mfu 146.77% |
| iter 114800: loss 2.8505, time 440.93ms, mfu 147.61% |
| iter 114900: loss 2.8134, time 440.00ms, mfu 148.40% |
| iter 115000: loss 2.8616, time 439.42ms, mfu 149.13% |
| iter 115100: loss 2.8605, time 441.81ms, mfu 149.70% |
| iter 115200: loss 2.8474, time 443.10ms, mfu 150.17% |
| iter 115300: loss 2.8767, time 439.68ms, mfu 150.72% |
| iter 115400: loss 2.8730, time 434.14ms, mfu 151.40% |
| iter 115500: loss 2.8667, time 441.00ms, mfu 151.78% |
| iter 115600: loss 2.7871, time 439.12ms, mfu 152.18% |
| iter 115700: loss 2.8599, time 441.86ms, mfu 152.45% |
| iter 115800: loss 2.8359, time 434.86ms, mfu 152.94% |
| iter 115900: loss 2.8446, time 444.20ms, mfu 153.05% |
| step 116000: train loss 2.7871, val loss 2.8135 |
| saving checkpoint to out-wiki-gutenberg-530m-1ep |
| iter 116000: loss 2.8704, time 65761.15ms, mfu 137.85% |
| iter 116100: loss 2.8968, time 438.61ms, mfu 139.66% |
| iter 116200: loss 2.8724, time 441.28ms, mfu 141.20% |
| iter 116300: loss 2.8824, time 441.08ms, mfu 142.59% |
| iter 116400: loss 2.8774, time 435.40ms, mfu 144.05% |
| iter 116500: loss 2.8419, time 441.45ms, mfu 145.14% |
| iter 116600: loss 2.7817, time 442.87ms, mfu 146.08% |
| iter 116700: loss 2.8297, time 442.16ms, mfu 146.94% |
| iter 116800: loss 2.7375, time 439.17ms, mfu 147.83% |
| iter 116900: loss 2.8947, time 440.89ms, mfu 148.56% |
| iter 117000: loss 2.8754, time 440.55ms, mfu 149.24% |
| iter 117100: loss 2.7665, time 439.41ms, mfu 149.89% |
| iter 117200: loss 2.7990, time 444.02ms, mfu 150.31% |
| iter 117300: loss 2.8387, time 441.11ms, mfu 150.79% |
| iter 117400: loss 2.8929, time 442.40ms, mfu 151.17% |
| iter 117500: loss 2.7976, time 442.49ms, mfu 151.52% |
| iter 117600: loss 2.8134, time 439.71ms, mfu 151.93% |
| iter 117700: loss 2.7726, time 440.54ms, mfu 152.27% |
| iter 117800: loss 2.8762, time 443.03ms, mfu 152.48% |
| iter 117900: loss 2.8190, time 440.43ms, mfu 152.77% |
| step 118000: train loss 2.7863, val loss 2.8032 |
| saving checkpoint to out-wiki-gutenberg-530m-1ep |
| iter 118000: loss 2.8672, time 65690.25ms, mfu 137.60% |
| iter 118100: loss 2.8376, time 443.01ms, mfu 139.28% |
| iter 118200: loss 2.8407, time 440.66ms, mfu 140.88% |
| iter 118300: loss 2.8197, time 442.26ms, mfu 142.26% |
| iter 118400: loss 2.7998, time 438.98ms, mfu 143.62% |
| iter 118500: loss 2.8570, time 440.67ms, mfu 144.79% |
| iter 118600: loss 2.8095, time 439.62ms, mfu 145.87% |
| iter 118700: loss 2.8174, time 443.19ms, mfu 146.72% |
| iter 118800: loss 2.8729, time 441.74ms, mfu 147.54% |
| iter 118900: loss 2.8126, time 438.70ms, mfu 148.38% |
| iter 119000: loss 2.7807, time 441.84ms, mfu 149.03% |
| iter 119100: loss 2.8243, time 441.92ms, mfu 149.61% |
| iter 119200: loss 2.8336, time 440.16ms, mfu 150.19% |
| iter 119300: loss 2.8085, time 439.77ms, mfu 150.73% |
| iter 119400: loss 2.8617, time 441.80ms, mfu 151.14% |
| iter 119500: loss 2.8478, time 439.28ms, mfu 151.61% |
| iter 119600: loss 2.8275, time 443.04ms, mfu 151.89% |
| iter 119700: loss 2.8272, time 441.46ms, mfu 152.20% |
| iter 119800: loss 2.8077, time 439.17ms, mfu 152.56% |
| iter 119900: loss 2.8324, time 440.91ms, mfu 152.82% |
| step 120000: train loss 2.7764, val loss 2.7948 |
| saving checkpoint to out-wiki-gutenberg-530m-1ep |
| iter 120000: loss 2.8556, time 65414.84ms, mfu 137.64% |
| iter 120100: loss 2.8183, time 439.73ms, mfu 139.44% |
| iter 120200: loss 2.7740, time 441.18ms, mfu 141.00% |
| iter 120300: loss 2.7821, time 440.20ms, mfu 142.45% |
| iter 120400: loss 2.7554, time 433.85ms, mfu 143.97% |
| iter 120500: loss 2.8011, time 441.45ms, mfu 145.07% |
| iter 120600: loss 2.8431, time 444.78ms, mfu 145.95% |
| iter 120700: loss 2.8372, time 441.21ms, mfu 146.86% |
| iter 120800: loss 2.8175, time 432.67ms, mfu 147.99% |
| iter 120900: loss 2.7497, time 442.54ms, mfu 148.65% |
| iter 121000: loss 2.7956, time 440.99ms, mfu 149.30% |
| iter 121100: loss 2.8291, time 440.01ms, mfu 149.92% |
| iter 121200: loss 2.7597, time 439.76ms, mfu 150.49% |
| iter 121300: loss 2.7891, time 442.61ms, mfu 150.90% |
| iter 121400: loss 2.8242, time 439.47ms, mfu 151.38% |
| iter 121500: loss 2.8993, time 438.32ms, mfu 151.85% |
| iter 121600: loss 2.8140, time 438.95ms, mfu 152.25% |
| iter 121700: loss 2.7881, time 439.68ms, mfu 152.59% |
| iter 121800: loss 2.7911, time 441.33ms, mfu 152.83% |
| iter 121900: loss 2.8824, time 441.26ms, mfu 153.05% |
| step 122000: train loss 2.7670, val loss 2.7888 |
| saving checkpoint to out-wiki-gutenberg-530m-1ep |
| iter 122000: loss 2.8614, time 65505.33ms, mfu 137.85% |
| iter 122100: loss 2.8434, time 438.94ms, mfu 139.66% |
| iter 122200: loss 2.7899, time 439.73ms, mfu 141.25% |
| iter 122300: loss 2.8125, time 440.51ms, mfu 142.66% |
| iter 122400: loss 2.7955, time 441.64ms, mfu 143.88% |
| iter 122500: loss 2.7594, time 444.32ms, mfu 144.89% |
| iter 122600: loss 2.8907, time 443.27ms, mfu 145.84% |
| iter 122700: loss 2.8648, time 440.72ms, mfu 146.78% |
| iter 122800: loss 2.8149, time 441.06ms, mfu 147.61% |
| iter 122900: loss 2.8159, time 439.95ms, mfu 148.41% |
| iter 123000: loss 2.8114, time 445.07ms, mfu 148.94% |
| iter 123100: loss 2.8256, time 441.07ms, mfu 149.56% |
| iter 123200: loss 2.8304, time 443.19ms, mfu 150.04% |
| iter 123300: loss 2.7887, time 440.80ms, mfu 150.56% |
| iter 123400: loss 2.7741, time 440.07ms, mfu 151.05% |
| iter 123500: loss 2.8599, time 441.63ms, mfu 151.44% |
| iter 123600: loss 2.7530, time 442.85ms, mfu 151.74% |
| iter 123700: loss 2.7967, time 440.83ms, mfu 152.09% |
| iter 123800: loss 2.8116, time 441.59ms, mfu 152.37% |
| iter 123900: loss 2.8388, time 442.57ms, mfu 152.60% |
| step 124000: train loss 2.7536, val loss 2.7844 |
| saving checkpoint to out-wiki-gutenberg-530m-1ep |
| iter 124000: loss 2.7749, time 65548.67ms, mfu 137.44% |
| iter 124100: loss 2.8764, time 437.81ms, mfu 139.32% |
| iter 124200: loss 2.8399, time 439.51ms, mfu 140.96% |
| iter 124300: loss 2.8156, time 441.41ms, mfu 142.36% |
| iter 124400: loss 2.8278, time 436.19ms, mfu 143.81% |
| iter 124500: loss 2.7536, time 440.73ms, mfu 144.96% |
| iter 124600: loss 2.7980, time 438.64ms, mfu 146.06% |
| iter 124700: loss 2.8289, time 443.08ms, mfu 146.90% |
| iter 124800: loss 2.7836, time 440.77ms, mfu 147.73% |
| iter 124900: loss 2.7926, time 435.95ms, mfu 148.65% |
| iter 125000: loss 2.8268, time 441.68ms, mfu 149.28% |
| iter 125100: loss 2.7899, time 439.24ms, mfu 149.93% |
| iter 125200: loss 2.7296, time 439.46ms, mfu 150.50% |
| iter 125300: loss 2.8671, time 440.25ms, mfu 150.99% |
| iter 125400: loss 2.8313, time 442.21ms, mfu 151.37% |
| iter 125500: loss 2.8739, time 440.14ms, mfu 151.77% |
| iter 125600: loss 2.8147, time 441.52ms, mfu 152.09% |
| iter 125700: loss 2.8039, time 440.30ms, mfu 152.42% |
| iter 125800: loss 2.8488, time 438.91ms, mfu 152.77% |
| iter 125900: loss 2.7709, time 439.64ms, mfu 153.06% |
| step 126000: train loss 2.7582, val loss 2.7787 |
| saving checkpoint to out-wiki-gutenberg-530m-1ep |
| iter 126000: loss 2.8225, time 65705.47ms, mfu 137.85% |
| iter 126100: loss 2.7964, time 439.59ms, mfu 139.63% |
| iter 126200: loss 2.7949, time 434.45ms, mfu 141.42% |
| iter 126300: loss 2.7715, time 441.63ms, mfu 142.77% |
| iter 126400: loss 2.8160, time 441.32ms, mfu 144.00% |
| iter 126500: loss 2.8389, time 441.04ms, mfu 145.11% |
| iter 126600: loss 2.7874, time 442.28ms, mfu 146.07% |
| iter 126700: loss 2.7688, time 440.45ms, mfu 147.00% |
| iter 126800: loss 2.7711, time 442.12ms, mfu 147.77% |
| iter 126900: loss 2.7036, time 441.05ms, mfu 148.51% |
| iter 127000: loss 2.8151, time 440.12ms, mfu 149.20% |
| iter 127100: loss 2.8384, time 441.08ms, mfu 149.79% |
| iter 127200: loss 2.7610, time 440.01ms, mfu 150.36% |
| iter 127300: loss 2.7682, time 441.71ms, mfu 150.82% |
| iter 127400: loss 2.8408, time 440.31ms, mfu 151.28% |
| iter 127500: loss 2.8193, time 442.28ms, mfu 151.62% |
| iter 127600: loss 2.8509, time 442.31ms, mfu 151.92% |
| iter 127700: loss 2.8121, time 440.67ms, mfu 152.26% |
| iter 127800: loss 2.7416, time 442.17ms, mfu 152.51% |
| iter 127900: loss 2.8069, time 444.85ms, mfu 152.64% |
| step 128000: train loss 2.7548, val loss 2.7764 |
| saving checkpoint to out-wiki-gutenberg-530m-1ep |
| iter 128000: loss 2.7691, time 65564.38ms, mfu 137.48% |
| iter 128100: loss 2.7954, time 440.80ms, mfu 139.25% |
| iter 128200: loss 2.8105, time 440.43ms, mfu 140.86% |
| iter 128300: loss 2.7383, time 441.48ms, mfu 142.27% |
| iter 128400: loss 2.8281, time 441.35ms, mfu 143.55% |
| iter 128500: loss 2.7763, time 443.87ms, mfu 144.61% |
| iter 128600: loss 2.8198, time 441.87ms, mfu 145.63% |
| iter 128700: loss 2.8018, time 439.97ms, mfu 146.62% |
| iter 128800: loss 2.8550, time 443.92ms, mfu 147.37% |
| iter 128900: loss 2.7483, time 441.07ms, mfu 148.15% |
| iter 129000: loss 2.7486, time 442.83ms, mfu 148.78% |
| iter 129100: loss 2.8854, time 442.26ms, mfu 149.37% |
| iter 129200: loss 2.7922, time 443.30ms, mfu 149.87% |
| iter 129300: loss 2.7137, time 446.38ms, mfu 150.21% |
| iter 129400: loss 2.8317, time 439.50ms, mfu 150.76% |
| iter 129500: loss 2.7223, time 441.18ms, mfu 151.19% |
| iter 129600: loss 2.8584, time 439.49ms, mfu 151.64% |
| iter 129700: loss 2.8009, time 442.30ms, mfu 151.95% |
| iter 129800: loss 2.9103, time 441.96ms, mfu 152.23% |
| iter 129900: loss 2.7666, time 442.79ms, mfu 152.46% |
| step 130000: train loss 2.7483, val loss 2.7697 |
| saving checkpoint to out-wiki-gutenberg-530m-1ep |
| iter 130000: loss 2.8124, time 65799.62ms, mfu 137.32% |
| iter 130100: loss 2.8043, time 441.01ms, mfu 139.10% |
| iter 130200: loss 2.7715, time 440.01ms, mfu 140.74% |
| iter 130300: loss 2.8107, time 439.34ms, mfu 142.24% |
| iter 130400: loss 2.7556, time 442.41ms, mfu 143.48% |
| iter 130500: loss 2.7954, time 441.54ms, mfu 144.63% |
| iter 130600: loss 2.8161, time 440.13ms, mfu 145.71% |
| iter 130700: loss 2.7706, time 441.41ms, mfu 146.64% |
| iter 130800: loss 2.8254, time 440.91ms, mfu 147.49% |
| iter 130900: loss 2.7722, time 444.19ms, mfu 148.15% |
| iter 131000: loss 2.7176, time 441.95ms, mfu 148.82% |
| iter 131100: loss 2.7961, time 439.31ms, mfu 149.51% |
| iter 131200: loss 2.7558, time 440.95ms, mfu 150.07% |
| iter 131300: loss 2.8392, time 441.34ms, mfu 150.57% |
| iter 131400: loss 2.8425, time 442.77ms, mfu 150.97% |
| iter 131500: loss 2.7823, time 439.03ms, mfu 151.45% |
| iter 131600: loss 2.6663, time 438.23ms, mfu 151.92% |
| iter 131700: loss 2.7957, time 443.53ms, mfu 152.15% |
| iter 131800: loss 2.7258, time 443.30ms, mfu 152.37% |
| iter 131900: loss 2.8340, time 441.09ms, mfu 152.65% |
| step 132000: train loss 2.7441, val loss 2.7680 |
| saving checkpoint to out-wiki-gutenberg-530m-1ep |
| iter 132000: loss 2.7797, time 65457.70ms, mfu 137.49% |
| iter 132100: loss 2.8091, time 441.48ms, mfu 139.24% |
| iter 132200: loss 2.7777, time 441.62ms, mfu 140.81% |
| iter 132300: loss 2.7519, time 442.63ms, mfu 142.18% |
| iter 132400: loss 2.8004, time 440.96ms, mfu 143.48% |
| iter 132500: loss 2.7831, time 439.82ms, mfu 144.69% |
| iter 132600: loss 2.7491, time 442.04ms, mfu 145.70% |
| iter 132700: loss 2.7711, time 441.93ms, mfu 146.61% |
| iter 132800: loss 2.7886, time 441.62ms, mfu 147.44% |
| iter 132900: loss 2.7077, time 438.57ms, mfu 148.30% |
| iter 133000: loss 2.7744, time 442.46ms, mfu 148.93% |
| iter 133100: loss 2.8471, time 443.40ms, mfu 149.47% |
| iter 133200: loss 2.7616, time 439.65ms, mfu 150.09% |
| iter 133300: loss 2.8758, time 440.86ms, mfu 150.60% |
| iter 133400: loss 2.7872, time 440.40ms, mfu 151.07% |
| iter 133500: loss 2.8764, time 439.62ms, mfu 151.53% |
| iter 133600: loss 2.7800, time 441.22ms, mfu 151.88% |
| iter 133700: loss 2.8041, time 444.59ms, mfu 152.08% |
| iter 133800: loss 2.8194, time 441.35ms, mfu 152.38% |
| iter 133900: loss 2.7547, time 439.98ms, mfu 152.69% |
| step 134000: train loss 2.7390, val loss 2.7650 |
| saving checkpoint to out-wiki-gutenberg-530m-1ep |
| iter 134000: loss 2.7572, time 65663.64ms, mfu 137.53% |
| iter 134100: loss 2.7551, time 440.54ms, mfu 139.31% |
| iter 134200: loss 2.7672, time 439.60ms, mfu 140.94% |
| iter 134300: loss 2.8065, time 442.56ms, mfu 142.31% |
| iter 134400: loss 2.7587, time 441.61ms, mfu 143.57% |
| iter 134500: loss 2.7616, time 441.45ms, mfu 144.71% |
| iter 134600: loss 2.7910, time 435.63ms, mfu 145.94% |
| iter 134700: loss 2.8514, time 440.35ms, mfu 146.89% |
| iter 134800: loss 2.7414, time 441.74ms, mfu 147.69% |
| iter 134900: loss 2.8629, time 440.16ms, mfu 148.46% |
| iter 135000: loss 2.8098, time 439.35ms, mfu 149.19% |
| iter 135100: loss 2.8385, time 443.71ms, mfu 149.69% |
| iter 135200: loss 2.7048, time 440.65ms, mfu 150.25% |
| iter 135300: loss 2.7254, time 444.26ms, mfu 150.63% |
| iter 135400: loss 2.6914, time 441.42ms, mfu 151.06% |
| iter 135500: loss 2.7797, time 441.67ms, mfu 151.45% |
| iter 135600: loss 2.8389, time 442.64ms, mfu 151.76% |
| iter 135700: loss 2.8208, time 441.67ms, mfu 152.08% |
| iter 135800: loss 2.7976, time 441.08ms, mfu 152.38% |
| iter 135900: loss 2.7482, time 440.82ms, mfu 152.66% |
| step 136000: train loss 2.7352, val loss 2.7654 |
| saving checkpoint to out-wiki-gutenberg-530m-1ep |
| iter 136000: loss 2.7647, time 65743.26ms, mfu 137.50% |
| iter 136100: loss 2.8031, time 440.62ms, mfu 139.28% |
| iter 136200: loss 2.7676, time 440.96ms, mfu 140.87% |
| iter 136300: loss 2.8472, time 441.35ms, mfu 142.28% |
| iter 136400: loss 2.7656, time 437.28ms, mfu 143.70% |
| iter 136500: loss 2.8012, time 441.28ms, mfu 144.84% |
|
|
