train.log

Overriding config with config/train_wiki_gutenberg.py:
# train a 0.5B (16L/1536) gpt2-bpe model on the combined wikipedia+gutenberg
# corpus (~9B tok). The capacity-scaling run: 2x params vs our 16L/1024 (253M),
# Chinchilla-near at 1 epoch (~17 tok/param). Overridden by launch flags.

out_dir = 'out-wiki-gutenberg-530m-1ep'
eval_interval = 2000
eval_iters = 200
log_interval = 100
always_save_checkpoint = True   # local resume granularity; hf_backup.py mirrors to the Hub

wandb_log = False
wandb_project = 'wiki_gutenberg'
wandb_run_name = '0.5b'

dataset = 'wiki_gutenberg'
gradient_accumulation_steps = 1
batch_size = 256
block_size = 256

# baby GPT model :)  (overridden by launch flags -> 16L/1536/24head ~530M)
n_layer = 6
n_head = 6
n_embd = 384
dropout = 0.1

learning_rate = 1e-3
max_iters = 5000
lr_decay_iters = 5000
min_lr = 1e-4
beta2 = 0.95
warmup_iters = 100

device = 'cpu'
compile = False

Overriding: device = cuda
Overriding: compile = True
Overriding: eval_interval = 2000
Overriding: eval_iters = 200
Overriding: log_interval = 100
Overriding: always_save_checkpoint = True
Overriding: block_size = 256
Overriding: batch_size = 256
Overriding: gradient_accumulation_steps = 1
Overriding: n_layer = 16
Overriding: n_head = 24
Overriding: n_embd = 1536
Overriding: dropout = 0.1
Overriding: learning_rate = 0.0012
Overriding: min_lr = 0.00012
Overriding: warmup_iters = 5000
Overriding: beta2 = 0.95
Overriding: max_iters = 137108
Overriding: lr_decay_iters = 137108
Overriding: out_dir = out-wiki-gutenberg-530m-1ep
tokens per iteration will be: 65,536
Initializing a new model from scratch
defaulting to vocab_size of GPT-2 to 50304 (50257 rounded up for efficiency)
number of parameters: 530.30M
/root/nanoGPT/train.py:199: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
  scaler = torch.cuda.amp.GradScaler(enabled=(dtype == 'float16'))
num decayed parameter tensors: 66, with 530,644,992 parameters
num non-decayed parameter tensors: 33, with 50,688 parameters
using fused AdamW: True
compiling the model... (takes a ~minute)
step 0: train loss 11.1036, val loss 11.1013
iter 0: loss 11.1082, time 89170.69ms, mfu -100.00%
iter 100: loss 7.1267, time 437.34ms, mfu 156.45%
iter 200: loss 6.4479, time 442.17ms, mfu 156.28%
iter 300: loss 5.8774, time 442.33ms, mfu 156.12%
iter 400: loss 5.6738, time 440.82ms, mfu 156.03%
iter 500: loss 5.3547, time 442.74ms, mfu 155.88%
iter 600: loss 5.1492, time 441.76ms, mfu 155.78%
iter 700: loss 5.0054, time 441.16ms, mfu 155.71%
iter 800: loss 4.8405, time 442.20ms, mfu 155.61%
iter 900: loss 4.7264, time 441.09ms, mfu 155.56%
iter 1000: loss 4.5948, time 444.95ms, mfu 155.38%
iter 1100: loss 4.3694, time 442.62ms, mfu 155.30%
iter 1200: loss 4.3759, time 445.97ms, mfu 155.11%
iter 1300: loss 4.2373, time 441.98ms, mfu 155.08%
iter 1400: loss 4.1923, time 443.71ms, mfu 154.99%
iter 1500: loss 4.1416, time 446.06ms, mfu 154.83%
iter 1600: loss 4.0585, time 444.95ms, mfu 154.73%
iter 1700: loss 4.1305, time 442.71ms, mfu 154.71%
iter 1800: loss 4.0767, time 446.73ms, mfu 154.55%
iter 1900: loss 4.0259, time 442.95ms, mfu 154.55%
step 2000: train loss 3.9183, val loss 3.9155
saving checkpoint to out-wiki-gutenberg-530m-1ep
iter 2000: loss 3.9544, time 62017.87ms, mfu 139.20%
iter 2100: loss 3.9057, time 446.44ms, mfu 140.61%
iter 2200: loss 3.8796, time 446.73ms, mfu 141.86%
iter 2300: loss 3.8484, time 447.67ms, mfu 142.96%
iter 2400: loss 3.9056, time 444.35ms, mfu 144.06%
iter 2500: loss 3.8618, time 446.88ms, mfu 144.97%
iter 2600: loss 3.8273, time 446.55ms, mfu 145.79%
iter 2700: loss 3.8333, time 440.68ms, mfu 146.74%
iter 2800: loss 3.8259, time 441.91ms, mfu 147.55%
iter 2900: loss 3.7795, time 448.62ms, mfu 148.04%
iter 3000: loss 3.7829, time 449.51ms, mfu 148.46%
iter 3100: loss 3.8019, time 444.32ms, mfu 149.01%
iter 3200: loss 3.8028, time 448.94ms, mfu 149.35%
iter 3300: loss 3.7317, time 445.18ms, mfu 149.79%
iter 3400: loss 3.6775, time 444.78ms, mfu 150.19%
iter 3500: loss 3.7106, time 443.69ms, mfu 150.59%
iter 3600: loss 3.7242, time 444.40ms, mfu 150.93%
iter 3700: loss 3.7189, time 447.56ms, mfu 151.12%
iter 3800: loss 3.6343, time 447.22ms, mfu 151.31%
iter 3900: loss 3.6177, time 446.55ms, mfu 151.50%
step 4000: train loss 3.6156, val loss 3.6195
saving checkpoint to out-wiki-gutenberg-530m-1ep
iter 4000: loss 3.6261, time 66001.87ms, mfu 136.45%
iter 4100: loss 3.6635, time 439.91ms, mfu 138.36%
iter 4200: loss 3.6292, time 439.72ms, mfu 140.09%
iter 4300: loss 3.6484, time 442.83ms, mfu 141.53%
iter 4400: loss 3.6215, time 445.03ms, mfu 142.75%
iter 4500: loss 3.6225, time 442.49ms, mfu 143.94%
iter 4600: loss 3.6168, time 446.89ms, mfu 144.85%
iter 4700: loss 3.6826, time 443.07ms, mfu 145.81%
iter 4800: loss 3.5770, time 442.51ms, mfu 146.69%
iter 4900: loss 3.6001, time 444.92ms, mfu 147.40%
iter 5000: loss 3.6241, time 440.84ms, mfu 148.18%
iter 5100: loss 3.5333, time 447.42ms, mfu 148.66%
iter 5200: loss 3.6191, time 442.71ms, mfu 149.24%
iter 5300: loss 3.6189, time 448.83ms, mfu 149.56%
iter 5400: loss 3.6240, time 443.82ms, mfu 150.02%
iter 5500: loss 3.4967, time 445.14ms, mfu 150.39%
iter 5600: loss 3.4990, time 441.33ms, mfu 150.86%
iter 5700: loss 3.5285, time 442.22ms, mfu 151.24%
iter 5800: loss 3.5127, time 444.95ms, mfu 151.50%
iter 5900: loss 3.4877, time 442.02ms, mfu 151.82%
step 6000: train loss 3.4517, val loss 3.4570
saving checkpoint to out-wiki-gutenberg-530m-1ep
iter 6000: loss 3.3989, time 65862.07ms, mfu 136.75%
iter 6100: loss 3.4812, time 439.08ms, mfu 138.65%
iter 6200: loss 3.5478, time 442.09ms, mfu 140.27%
iter 6300: loss 3.4423, time 442.49ms, mfu 141.70%
iter 6400: loss 3.4052, time 443.41ms, mfu 142.96%
iter 6500: loss 3.5069, time 442.02ms, mfu 144.14%
iter 6600: loss 3.3985, time 441.76ms, mfu 145.22%
iter 6700: loss 3.4148, time 446.59ms, mfu 146.02%
iter 6800: loss 3.4067, time 441.78ms, mfu 146.90%
iter 6900: loss 3.3920, time 442.39ms, mfu 147.68%
iter 7000: loss 3.3811, time 441.98ms, mfu 148.39%
iter 7100: loss 3.4921, time 445.73ms, mfu 148.90%
iter 7200: loss 3.4252, time 445.87ms, mfu 149.36%
iter 7300: loss 3.4486, time 442.58ms, mfu 149.88%
iter 7400: loss 3.4207, time 444.25ms, mfu 150.29%
iter 7500: loss 3.4161, time 444.91ms, mfu 150.64%
iter 7600: loss 3.3954, time 442.74ms, mfu 151.03%
iter 7700: loss 3.4477, time 441.99ms, mfu 151.41%
iter 7800: loss 3.3723, time 442.45ms, mfu 151.73%
iter 7900: loss 3.4532, time 443.68ms, mfu 151.98%
step 8000: train loss 3.3458, val loss 3.3517
saving checkpoint to out-wiki-gutenberg-530m-1ep
iter 8000: loss 3.3585, time 65769.94ms, mfu 136.89%
iter 8100: loss 3.4348, time 443.89ms, mfu 138.61%
iter 8200: loss 3.3941, time 440.23ms, mfu 140.29%
iter 8300: loss 3.4014, time 443.97ms, mfu 141.67%
iter 8400: loss 3.4147, time 440.55ms, mfu 143.04%
iter 8500: loss 3.4033, time 442.57ms, mfu 144.19%
iter 8600: loss 3.3590, time 441.40ms, mfu 145.27%
iter 8700: loss 3.3398, time 441.06ms, mfu 146.26%
iter 8800: loss 3.3703, time 445.86ms, mfu 146.98%
iter 8900: loss 3.3603, time 444.12ms, mfu 147.69%
iter 9000: loss 3.3583, time 446.80ms, mfu 148.23%
iter 9100: loss 3.3942, time 441.60ms, mfu 148.90%
iter 9200: loss 3.4014, time 445.52ms, mfu 149.37%
iter 9300: loss 3.3763, time 442.88ms, mfu 149.88%
iter 9400: loss 3.3114, time 444.65ms, mfu 150.28%
iter 9500: loss 3.4656, time 444.21ms, mfu 150.66%
iter 9600: loss 3.2709, time 443.68ms, mfu 151.01%
iter 9700: loss 3.3287, time 441.56ms, mfu 151.41%
iter 9800: loss 3.3598, time 444.98ms, mfu 151.64%
iter 9900: loss 3.2859, time 441.92ms, mfu 151.96%
step 10000: train loss 3.2856, val loss 3.2934
saving checkpoint to out-wiki-gutenberg-530m-1ep
iter 10000: loss 3.3140, time 65707.54ms, mfu 136.87%
iter 10100: loss 3.3347, time 440.20ms, mfu 138.72%
iter 10200: loss 3.3091, time 441.62ms, mfu 140.34%
iter 10300: loss 3.3442, time 444.88ms, mfu 141.69%
iter 10400: loss 3.4077, time 442.05ms, mfu 143.00%
iter 10500: loss 3.3773, time 444.11ms, mfu 144.10%
iter 10600: loss 3.3197, time 445.05ms, mfu 145.07%
iter 10700: loss 3.3867, time 445.74ms, mfu 145.91%
iter 10800: loss 3.3132, time 443.69ms, mfu 146.74%
iter 10900: loss 3.2219, time 442.60ms, mfu 147.53%
iter 11000: loss 3.3162, time 442.65ms, mfu 148.23%
iter 11100: loss 3.2492, time 443.92ms, mfu 148.82%
iter 11200: loss 3.2580, time 442.62ms, mfu 149.40%
iter 11300: loss 3.3678, time 442.80ms, mfu 149.91%
iter 11400: loss 3.3408, time 443.72ms, mfu 150.34%
iter 11500: loss 3.3912, time 441.13ms, mfu 150.81%
iter 11600: loss 3.2979, time 445.79ms, mfu 151.08%
iter 11700: loss 3.2721, time 443.37ms, mfu 151.40%
iter 11800: loss 3.2679, time 443.61ms, mfu 151.69%
iter 11900: loss 3.3370, time 440.43ms, mfu 152.05%
step 12000: train loss 3.2428, val loss 3.2486
saving checkpoint to out-wiki-gutenberg-530m-1ep
iter 12000: loss 3.2503, time 65678.03ms, mfu 136.95%
iter 12100: loss 3.3662, time 438.84ms, mfu 138.85%
iter 12200: loss 3.2599, time 442.94ms, mfu 140.41%
iter 12300: loss 3.2895, time 442.09ms, mfu 141.85%
iter 12400: loss 3.2313, time 441.04ms, mfu 143.17%
iter 12500: loss 3.3367, time 441.28ms, mfu 144.36%
iter 12600: loss 3.2648, time 446.60ms, mfu 145.25%
iter 12700: loss 3.2849, time 447.12ms, mfu 146.02%
iter 12800: loss 3.2787, time 443.23ms, mfu 146.86%
iter 12900: loss 3.3005, time 445.94ms, mfu 147.52%
iter 13000: loss 3.2938, time 442.06ms, mfu 148.24%
iter 13100: loss 3.2868, time 447.35ms, mfu 148.71%
iter 13200: loss 3.2834, time 441.03ms, mfu 149.35%
iter 13300: loss 3.2777, time 440.04ms, mfu 149.97%
iter 13400: loss 3.2641, time 442.06ms, mfu 150.45%
iter 13500: loss 3.2625, time 443.11ms, mfu 150.84%
iter 13600: loss 3.2251, time 444.52ms, mfu 151.15%
iter 13700: loss 3.2294, time 444.16ms, mfu 151.44%
iter 13800: loss 3.2571, time 443.35ms, mfu 151.73%
iter 13900: loss 3.1879, time 443.60ms, mfu 151.98%
step 14000: train loss 3.2173, val loss 3.2256
saving checkpoint to out-wiki-gutenberg-530m-1ep
iter 14000: loss 3.2374, time 65827.28ms, mfu 136.89%
iter 14100: loss 3.2433, time 441.60ms, mfu 138.69%
iter 14200: loss 3.2675, time 443.25ms, mfu 140.26%
iter 14300: loss 3.2100, time 441.87ms, mfu 141.72%
iter 14400: loss 3.2757, time 444.32ms, mfu 142.94%
iter 14500: loss 3.3211, time 443.98ms, mfu 144.06%
iter 14600: loss 3.2441, time 441.54ms, mfu 145.15%
iter 14700: loss 3.2467, time 440.77ms, mfu 146.16%
iter 14800: loss 3.3270, time 444.99ms, mfu 146.92%
iter 14900: loss 3.2327, time 442.88ms, mfu 147.67%
iter 15000: loss 3.2169, time 442.76ms, mfu 148.36%
iter 15100: loss 3.2549, time 440.97ms, mfu 149.04%
iter 15200: loss 3.2018, time 446.53ms, mfu 149.46%
iter 15300: loss 3.2698, time 442.74ms, mfu 149.97%
iter 15400: loss 3.2472, time 441.88ms, mfu 150.45%
iter 15500: loss 3.2258, time 445.46ms, mfu 150.77%
iter 15600: loss 3.1997, time 440.81ms, mfu 151.21%
iter 15700: loss 3.1950, time 443.81ms, mfu 151.51%
iter 15800: loss 3.3171, time 442.84ms, mfu 151.81%
iter 15900: loss 3.2615, time 445.16ms, mfu 152.00%
step 16000: train loss 3.1977, val loss 3.2048
saving checkpoint to out-wiki-gutenberg-530m-1ep
iter 16000: loss 3.2691, time 65701.88ms, mfu 136.90%
iter 16100: loss 3.2874, time 441.43ms, mfu 138.71%
iter 16200: loss 3.1908, time 442.66ms, mfu 140.30%
iter 16300: loss 3.1912, time 443.20ms, mfu 141.70%
iter 16400: loss 3.2039, time 442.97ms, mfu 142.98%
iter 16500: loss 3.1949, time 440.96ms, mfu 144.20%
iter 16600: loss 3.2290, time 443.31ms, mfu 145.21%
iter 16700: loss 3.2693, time 445.50ms, mfu 146.05%
iter 16800: loss 3.2097, time 445.26ms, mfu 146.81%
iter 16900: loss 3.2467, time 442.98ms, mfu 147.57%
iter 17000: loss 3.2187, time 442.79ms, mfu 148.27%
iter 17100: loss 3.2884, time 443.43ms, mfu 148.87%
iter 17200: loss 3.2612, time 445.50ms, mfu 149.34%
iter 17300: loss 3.2020, time 442.55ms, mfu 149.87%
iter 17400: loss 3.2556, time 444.66ms, mfu 150.27%
iter 17500: loss 3.2447, time 442.91ms, mfu 150.69%
iter 17600: loss 3.2983, time 440.10ms, mfu 151.17%
iter 17700: loss 3.1734, time 444.52ms, mfu 151.44%
iter 17800: loss 3.2511, time 442.56ms, mfu 151.76%
iter 17900: loss 3.2137, time 445.42ms, mfu 151.94%
step 18000: train loss 3.1773, val loss 3.1889
saving checkpoint to out-wiki-gutenberg-530m-1ep
iter 18000: loss 3.2322, time 65615.98ms, mfu 136.85%
iter 18100: loss 3.2220, time 440.36ms, mfu 138.71%
iter 18200: loss 3.3049, time 440.15ms, mfu 140.38%
iter 18300: loss 3.2544, time 446.33ms, mfu 141.67%
iter 18400: loss 3.2650, time 444.13ms, mfu 142.91%
iter 18500: loss 3.2118, time 441.45ms, mfu 144.12%
iter 18600: loss 3.1895, time 442.50ms, mfu 145.17%
iter 18700: loss 3.1941, time 443.30ms, mfu 146.09%
iter 18800: loss 3.2778, time 444.64ms, mfu 146.87%
iter 18900: loss 3.2339, time 441.31ms, mfu 147.68%
iter 19000: loss 3.1870, time 441.61ms, mfu 148.41%
iter 19100: loss 3.2831, time 443.72ms, mfu 148.99%
iter 19200: loss 3.2928, time 441.89ms, mfu 149.57%
iter 19300: loss 3.2986, time 445.25ms, mfu 149.98%
iter 19400: loss 3.2456, time 445.22ms, mfu 150.35%
iter 19500: loss 3.1557, time 445.29ms, mfu 150.68%
iter 19600: loss 3.1754, time 444.03ms, mfu 151.02%
iter 19700: loss 3.1769, time 443.85ms, mfu 151.33%
iter 19800: loss 3.2197, time 443.63ms, mfu 151.62%
iter 19900: loss 3.1796, time 441.84ms, mfu 151.95%
step 20000: train loss 3.1663, val loss 3.1786
saving checkpoint to out-wiki-gutenberg-530m-1ep
iter 20000: loss 3.2533, time 65627.46ms, mfu 136.86%
iter 20100: loss 3.2156, time 440.87ms, mfu 138.69%
iter 20200: loss 3.2796, time 440.86ms, mfu 140.34%
iter 20300: loss 3.2432, time 439.73ms, mfu 141.87%
iter 20400: loss 3.2318, time 440.62ms, mfu 143.21%
iter 20500: loss 3.1981, time 440.47ms, mfu 144.42%
iter 20600: loss 3.2269, time 442.96ms, mfu 145.42%
iter 20700: loss 3.2127, time 441.25ms, mfu 146.39%
iter 20800: loss 3.2265, time 440.43ms, mfu 147.28%
iter 20900: loss 3.2139, time 444.35ms, mfu 147.95%
iter 21000: loss 3.2253, time 440.23ms, mfu 148.70%
iter 21100: loss 3.1392, time 436.57ms, mfu 149.50%
iter 21200: loss 3.2089, time 440.38ms, mfu 150.09%
iter 21300: loss 3.1923, time 439.31ms, mfu 150.65%
iter 21400: loss 3.2165, time 437.87ms, mfu 151.22%
iter 21500: loss 3.1946, time 440.78ms, mfu 151.62%
iter 21600: loss 3.2736, time 439.86ms, mfu 152.01%
iter 21700: loss 3.1535, time 437.99ms, mfu 152.43%
iter 21800: loss 3.2071, time 440.49ms, mfu 152.72%
iter 21900: loss 3.1913, time 441.04ms, mfu 152.96%
step 22000: train loss 3.1568, val loss 3.1641
saving checkpoint to out-wiki-gutenberg-530m-1ep
iter 22000: loss 3.1692, time 65310.84ms, mfu 137.77%
iter 22100: loss 3.2188, time 440.52ms, mfu 139.52%
iter 22200: loss 3.1753, time 440.64ms, mfu 141.10%
iter 22300: loss 3.3106, time 437.58ms, mfu 142.63%
iter 22400: loss 3.1838, time 440.55ms, mfu 143.89%
iter 22500: loss 3.2047, time 441.35ms, mfu 145.01%
iter 22600: loss 3.2066, time 441.71ms, mfu 146.00%
iter 22700: loss 3.2010, time 440.30ms, mfu 146.94%
iter 22800: loss 3.1363, time 441.55ms, mfu 147.74%
iter 22900: loss 3.1682, time 441.18ms, mfu 148.47%
iter 23000: loss 3.2113, time 443.77ms, mfu 149.04%
iter 23100: loss 3.2214, time 442.58ms, mfu 149.60%
iter 23200: loss 3.1835, time 440.88ms, mfu 150.16%
iter 23300: loss 3.2083, time 441.10ms, mfu 150.65%
iter 23400: loss 3.2242, time 441.47ms, mfu 151.09%
iter 23500: loss 3.1649, time 441.47ms, mfu 151.48%
iter 23600: loss 3.1838, time 440.25ms, mfu 151.87%
iter 23700: loss 3.1758, time 440.56ms, mfu 152.21%
iter 23800: loss 3.2205, time 439.44ms, mfu 152.56%
iter 23900: loss 3.2537, time 438.36ms, mfu 152.91%
step 24000: train loss 3.1421, val loss 3.1553
saving checkpoint to out-wiki-gutenberg-530m-1ep
iter 24000: loss 3.1881, time 65282.99ms, mfu 137.73%
iter 24100: loss 3.2003, time 439.85ms, mfu 139.51%
iter 24200: loss 3.2393, time 441.07ms, mfu 141.07%
iter 24300: loss 3.2287, time 438.57ms, mfu 142.56%
iter 24400: loss 3.2247, time 440.37ms, mfu 143.85%
iter 24500: loss 3.1927, time 440.19ms, mfu 145.00%
iter 24600: loss 3.2176, time 441.60ms, mfu 146.00%
iter 24700: loss 3.1701, time 440.06ms, mfu 146.95%
iter 24800: loss 3.2210, time 440.44ms, mfu 147.79%
iter 24900: loss 3.1681, time 438.30ms, mfu 148.62%
iter 25000: loss 3.2443, time 441.50ms, mfu 149.25%
iter 25100: loss 3.2527, time 442.08ms, mfu 149.80%
iter 25200: loss 3.1772, time 440.98ms, mfu 150.34%
iter 25300: loss 3.1544, time 440.34ms, mfu 150.84%
iter 25400: loss 3.1639, time 439.02ms, mfu 151.34%
iter 25500: loss 3.1479, time 440.53ms, mfu 151.74%
iter 25600: loss 3.2178, time 440.46ms, mfu 152.10%
iter 25700: loss 3.1651, time 442.39ms, mfu 152.36%
iter 25800: loss 3.1976, time 440.25ms, mfu 152.66%
iter 25900: loss 3.1531, time 441.09ms, mfu 152.91%
step 26000: train loss 3.1323, val loss 3.1432
saving checkpoint to out-wiki-gutenberg-530m-1ep
iter 26000: loss 3.2047, time 65653.33ms, mfu 137.72%
iter 26100: loss 3.1503, time 441.37ms, mfu 139.45%
iter 26200: loss 3.2074, time 440.35ms, mfu 141.04%
iter 26300: loss 3.2216, time 440.95ms, mfu 142.46%
iter 26400: loss 3.1873, time 441.01ms, mfu 143.72%
iter 26500: loss 3.2559, time 441.12ms, mfu 144.86%
iter 26600: loss 3.1597, time 442.74ms, mfu 145.83%
iter 26700: loss 3.1844, time 441.18ms, mfu 146.76%
iter 26800: loss 3.1551, time 440.31ms, mfu 147.62%
iter 26900: loss 3.2272, time 440.56ms, mfu 148.39%
iter 27000: loss 3.1884, time 439.58ms, mfu 149.11%
iter 27100: loss 3.2449, time 439.77ms, mfu 149.76%
iter 27200: loss 3.2506, time 439.02ms, mfu 150.37%
iter 27300: loss 3.1501, time 438.69ms, mfu 150.93%
iter 27400: loss 3.1378, time 440.86ms, mfu 151.36%
iter 27500: loss 3.1627, time 440.59ms, mfu 151.75%
iter 27600: loss 3.1861, time 437.84ms, mfu 152.20%
iter 27700: loss 3.1644, time 438.50ms, mfu 152.58%
iter 27800: loss 3.1470, time 442.58ms, mfu 152.78%
iter 27900: loss 3.1677, time 440.85ms, mfu 153.03%
step 28000: train loss 3.1270, val loss 3.1334
saving checkpoint to out-wiki-gutenberg-530m-1ep
iter 28000: loss 3.1931, time 65432.01ms, mfu 137.83%
iter 28100: loss 3.2114, time 437.68ms, mfu 139.68%
iter 28200: loss 3.2457, time 437.80ms, mfu 141.34%
iter 28300: loss 3.1986, time 439.94ms, mfu 142.76%
iter 28400: loss 3.1461, time 439.26ms, mfu 144.06%
iter 28500: loss 3.1284, time 442.32ms, mfu 145.12%
iter 28600: loss 3.2687, time 439.77ms, mfu 146.17%
iter 28700: loss 3.2131, time 440.94ms, mfu 147.07%
iter 28800: loss 3.1142, time 439.01ms, mfu 147.95%
iter 28900: loss 3.1451, time 441.84ms, mfu 148.64%
iter 29000: loss 3.1589, time 440.80ms, mfu 149.29%
iter 29100: loss 3.1450, time 443.06ms, mfu 149.81%
iter 29200: loss 3.1225, time 441.04ms, mfu 150.34%
iter 29300: loss 3.1363, time 440.94ms, mfu 150.82%
iter 29400: loss 3.1213, time 441.59ms, mfu 151.24%
iter 29500: loss 3.1392, time 441.64ms, mfu 151.60%
iter 29600: loss 3.1850, time 438.42ms, mfu 152.05%
iter 29700: loss 3.1586, time 440.93ms, mfu 152.36%
iter 29800: loss 3.2396, time 443.05ms, mfu 152.57%
iter 29900: loss 3.2580, time 434.33ms, mfu 153.07%
step 30000: train loss 3.1217, val loss 3.1208
saving checkpoint to out-wiki-gutenberg-530m-1ep
iter 30000: loss 3.2306, time 65346.39ms, mfu 137.86%
iter 30100: loss 3.1761, time 440.21ms, mfu 139.62%
iter 30200: loss 3.1694, time 436.11ms, mfu 141.35%
iter 30300: loss 3.1369, time 438.80ms, mfu 142.80%
iter 30400: loss 3.2328, time 442.19ms, mfu 144.00%
iter 30500: loss 3.1111, time 439.89ms, mfu 145.15%
iter 30600: loss 3.1187, time 435.06ms, mfu 146.36%
iter 30700: loss 3.1222, time 438.36ms, mfu 147.33%
iter 30800: loss 3.1422, time 441.10ms, mfu 148.11%
iter 30900: loss 3.1692, time 442.65ms, mfu 148.76%
iter 31000: loss 3.1235, time 440.40ms, mfu 149.42%
iter 31100: loss 3.1903, time 437.54ms, mfu 150.11%
iter 31200: loss 3.1061, time 437.73ms, mfu 150.73%
iter 31300: loss 3.2265, time 440.03ms, mfu 151.21%
iter 31400: loss 3.1683, time 440.02ms, mfu 151.64%
iter 31500: loss 3.1792, time 440.73ms, mfu 152.00%
iter 31600: loss 3.1770, time 439.67ms, mfu 152.36%
iter 31700: loss 3.1152, time 440.78ms, mfu 152.65%
iter 31800: loss 3.1440, time 440.87ms, mfu 152.90%
iter 31900: loss 3.1648, time 443.14ms, mfu 153.05%
step 32000: train loss 3.1174, val loss 3.1209
saving checkpoint to out-wiki-gutenberg-530m-1ep
iter 32000: loss 3.2021, time 65882.30ms, mfu 137.85%
iter 32100: loss 3.1142, time 440.49ms, mfu 139.60%
iter 32200: loss 3.1869, time 440.16ms, mfu 141.18%
iter 32300: loss 3.1862, time 441.76ms, mfu 142.55%
iter 32400: loss 3.0835, time 439.22ms, mfu 143.87%
iter 32500: loss 3.1186, time 441.53ms, mfu 144.98%
iter 32600: loss 3.1332, time 440.10ms, mfu 146.03%
iter 32700: loss 3.1653, time 441.09ms, mfu 146.94%
iter 32800: loss 3.0876, time 439.37ms, mfu 147.82%
iter 32900: loss 3.1725, time 440.65ms, mfu 148.56%
iter 33000: loss 3.1411, time 438.29ms, mfu 149.32%
iter 33100: loss 3.1240, time 440.70ms, mfu 149.91%
iter 33200: loss 3.1866, time 442.01ms, mfu 150.40%
iter 33300: loss 3.1937, time 441.39ms, mfu 150.86%
iter 33400: loss 3.1438, time 441.20ms, mfu 151.28%
iter 33500: loss 3.1431, time 438.93ms, mfu 151.74%
iter 33600: loss 3.1765, time 441.70ms, mfu 152.06%
iter 33700: loss 3.1790, time 439.30ms, mfu 152.43%
iter 33800: loss 3.1901, time 440.92ms, mfu 152.70%
iter 33900: loss 3.1656, time 441.90ms, mfu 152.92%
step 34000: train loss 3.1030, val loss 3.1095
saving checkpoint to out-wiki-gutenberg-530m-1ep
iter 34000: loss 3.1539, time 65373.02ms, mfu 137.73%
iter 34100: loss 3.1435, time 441.38ms, mfu 139.46%
iter 34200: loss 3.1812, time 442.57ms, mfu 140.97%
iter 34300: loss 3.1066, time 442.16ms, mfu 142.35%
iter 34400: loss 3.1261, time 443.71ms, mfu 143.53%
iter 34500: loss 3.1057, time 440.65ms, mfu 144.71%
iter 34600: loss 3.2013, time 441.57ms, mfu 145.73%
iter 34700: loss 3.1375, time 440.28ms, mfu 146.70%
iter 34800: loss 3.1631, time 442.39ms, mfu 147.49%
iter 34900: loss 3.1176, time 440.82ms, mfu 148.27%
iter 35000: loss 3.0846, time 440.21ms, mfu 148.98%
iter 35100: loss 3.0308, time 437.86ms, mfu 149.71%
iter 35200: loss 3.0116, time 437.97ms, mfu 150.36%
iter 35300: loss 3.1386, time 439.07ms, mfu 150.91%
iter 35400: loss 3.2202, time 437.69ms, mfu 151.45%
iter 35500: loss 3.1263, time 438.35ms, mfu 151.91%
iter 35600: loss 3.1341, time 438.88ms, mfu 152.31%
iter 35700: loss 3.1480, time 439.99ms, mfu 152.63%
iter 35800: loss 3.1823, time 437.49ms, mfu 153.01%
iter 35900: loss 3.1703, time 437.42ms, mfu 153.35%
step 36000: train loss 3.0919, val loss 3.1089
saving checkpoint to out-wiki-gutenberg-530m-1ep
iter 36000: loss 3.0796, time 65524.32ms, mfu 138.12%
iter 36100: loss 3.1635, time 440.45ms, mfu 139.84%
iter 36200: loss 3.1232, time 442.05ms, mfu 141.33%
iter 36300: loss 3.2023, time 442.16ms, mfu 142.67%
iter 36400: loss 3.0824, time 439.02ms, mfu 143.99%
iter 36500: loss 3.2438, time 441.87ms, mfu 145.08%
iter 36600: loss 3.0976, time 440.20ms, mfu 146.11%
iter 36700: loss 3.1506, time 438.43ms, mfu 147.11%
iter 36800: loss 3.1652, time 438.18ms, mfu 148.01%
iter 36900: loss 3.1624, time 440.07ms, mfu 148.76%
iter 37000: loss 3.1852, time 438.43ms, mfu 149.49%
iter 37100: loss 3.1130, time 440.50ms, mfu 150.07%
iter 37200: loss 3.1409, time 440.45ms, mfu 150.60%
iter 37300: loss 3.1261, time 439.34ms, mfu 151.11%
iter 37400: loss 3.1568, time 439.02ms, mfu 151.59%
iter 37500: loss 3.2170, time 439.44ms, mfu 152.00%
iter 37600: loss 3.1721, time 438.55ms, mfu 152.40%
iter 37700: loss 3.1254, time 439.89ms, mfu 152.71%
iter 37800: loss 3.2083, time 438.50ms, mfu 153.04%
iter 37900: loss 3.1972, time 439.99ms, mfu 153.29%
step 38000: train loss 3.0881, val loss 3.1060
saving checkpoint to out-wiki-gutenberg-530m-1ep
iter 38000: loss 3.0752, time 65400.61ms, mfu 138.07%
iter 38100: loss 3.1241, time 439.69ms, mfu 139.82%
iter 38200: loss 3.1406, time 440.15ms, mfu 141.38%
iter 38300: loss 3.1196, time 441.75ms, mfu 142.73%
iter 38400: loss 3.0871, time 439.76ms, mfu 144.02%
iter 38500: loss 3.1002, time 439.26ms, mfu 145.19%
iter 38600: loss 3.1627, time 442.09ms, mfu 146.15%
iter 38700: loss 3.1359, time 440.17ms, mfu 147.08%
iter 38800: loss 3.1181, time 439.34ms, mfu 147.94%
iter 38900: loss 3.0825, time 439.96ms, mfu 148.70%
iter 39000: loss 3.1528, time 441.37ms, mfu 149.33%
iter 39100: loss 3.1497, time 438.43ms, mfu 150.01%
iter 39200: loss 3.1530, time 441.22ms, mfu 150.51%
iter 39300: loss 3.1517, time 441.30ms, mfu 150.97%
iter 39400: loss 3.1346, time 438.62ms, mfu 151.47%
iter 39500: loss 3.2056, time 440.11ms, mfu 151.87%
iter 39600: loss 3.1069, time 439.89ms, mfu 152.23%
iter 39700: loss 3.1575, time 441.61ms, mfu 152.50%
iter 39800: loss 3.1291, time 439.76ms, mfu 152.81%
iter 39900: loss 3.1384, time 440.10ms, mfu 153.08%
step 40000: train loss 3.0795, val loss 3.0926
saving checkpoint to out-wiki-gutenberg-530m-1ep
iter 40000: loss 3.1666, time 65356.60ms, mfu 137.87%
iter 40100: loss 3.1513, time 440.58ms, mfu 139.62%
iter 40200: loss 3.1636, time 438.94ms, mfu 141.24%
iter 40300: loss 3.1182, time 441.61ms, mfu 142.61%
iter 40400: loss 3.2074, time 439.74ms, mfu 143.91%
iter 40500: loss 3.1014, time 440.35ms, mfu 145.06%
iter 40600: loss 3.0895, time 441.24ms, mfu 146.06%
iter 40700: loss 3.1138, time 440.28ms, mfu 146.99%
iter 40800: loss 3.0745, time 439.96ms, mfu 147.84%
iter 40900: loss 3.1401, time 442.18ms, mfu 148.53%
iter 41000: loss 3.0290, time 441.06ms, mfu 149.19%
iter 41100: loss 3.1356, time 441.09ms, mfu 149.79%
iter 41200: loss 3.1947, time 439.15ms, mfu 150.39%
iter 41300: loss 3.1699, time 440.93ms, mfu 150.87%
iter 41400: loss 3.1631, time 441.37ms, mfu 151.28%
iter 41500: loss 3.0933, time 438.71ms, mfu 151.75%
iter 41600: loss 3.1015, time 439.08ms, mfu 152.16%
iter 41700: loss 3.1541, time 439.18ms, mfu 152.52%
iter 41800: loss 3.1475, time 440.14ms, mfu 152.81%
iter 41900: loss 3.1103, time 438.70ms, mfu 153.13%
step 42000: train loss 3.0792, val loss 3.0862
saving checkpoint to out-wiki-gutenberg-530m-1ep
iter 42000: loss 3.2236, time 65229.12ms, mfu 137.92%
iter 42100: loss 3.1352, time 441.86ms, mfu 139.61%
iter 42200: loss 3.1045, time 441.40ms, mfu 141.15%
iter 42300: loss 3.1690, time 442.42ms, mfu 142.50%
iter 42400: loss 3.1851, time 441.56ms, mfu 143.75%
iter 42500: loss 3.0816, time 441.01ms, mfu 144.89%
iter 42600: loss 3.1736, time 441.96ms, mfu 145.88%
iter 42700: loss 3.1924, time 441.26ms, mfu 146.80%
iter 42800: loss 3.1813, time 440.16ms, mfu 147.66%
iter 42900: loss 3.1326, time 441.00ms, mfu 148.41%
iter 43000: loss 3.1395, time 441.12ms, mfu 149.08%
iter 43100: loss 3.1630, time 440.65ms, mfu 149.70%
iter 43200: loss 3.1415, time 441.35ms, mfu 150.23%
iter 43300: loss 3.2113, time 440.30ms, mfu 150.75%
iter 43400: loss 3.1254, time 441.50ms, mfu 151.17%
iter 43500: loss 3.1059, time 441.31ms, mfu 151.56%
iter 43600: loss 3.1080, time 441.81ms, mfu 151.89%
iter 43700: loss 3.1519, time 441.20ms, mfu 152.21%
iter 43800: loss 3.1158, time 441.66ms, mfu 152.48%
iter 43900: loss 3.1154, time 439.62ms, mfu 152.79%
step 44000: train loss 3.0703, val loss 3.0790
saving checkpoint to out-wiki-gutenberg-530m-1ep
iter 44000: loss 3.1420, time 65549.70ms, mfu 137.62%
iter 44100: loss 3.1244, time 442.45ms, mfu 139.32%
iter 44200: loss 3.0865, time 439.73ms, mfu 140.95%
iter 44300: loss 3.1255, time 440.76ms, mfu 142.38%
iter 44400: loss 3.1418, time 441.55ms, mfu 143.63%
iter 44500: loss 3.1720, time 442.80ms, mfu 144.72%
iter 44600: loss 3.1223, time 440.02ms, mfu 145.80%
iter 44700: loss 3.1375, time 443.06ms, mfu 146.66%
iter 44800: loss 3.1846, time 442.80ms, mfu 147.45%
iter 44900: loss 3.1498, time 442.65ms, mfu 148.16%
iter 45000: loss 3.0990, time 440.46ms, mfu 148.88%
iter 45100: loss 3.1634, time 440.51ms, mfu 149.52%
iter 45200: loss 3.0561, time 442.28ms, mfu 150.04%
iter 45300: loss 3.0795, time 440.67ms, mfu 150.56%
iter 45400: loss 3.1325, time 440.73ms, mfu 151.03%
iter 45500: loss 3.0683, time 440.33ms, mfu 151.47%
iter 45600: loss 3.0709, time 440.97ms, mfu 151.84%
iter 45700: loss 3.1183, time 440.89ms, mfu 152.17%
iter 45800: loss 3.0583, time 439.54ms, mfu 152.52%
iter 45900: loss 3.1016, time 442.65ms, mfu 152.72%
step 46000: train loss 3.0685, val loss 3.0794
saving checkpoint to out-wiki-gutenberg-530m-1ep
iter 46000: loss 3.1274, time 65180.68ms, mfu 137.56%
iter 46100: loss 3.1506, time 437.55ms, mfu 139.44%
iter 46200: loss 3.1104, time 437.89ms, mfu 141.12%
iter 46300: loss 3.0399, time 437.23ms, mfu 142.66%
iter 46400: loss 3.1694, time 437.96ms, mfu 144.01%
iter 46500: loss 3.1508, time 440.17ms, mfu 145.16%
iter 46600: loss 3.0851, time 443.05ms, mfu 146.08%
iter 46700: loss 3.2149, time 438.40ms, mfu 147.08%
iter 46800: loss 3.1135, time 440.46ms, mfu 147.91%
iter 46900: loss 3.1074, time 440.79ms, mfu 148.64%
iter 47000: loss 3.1327, time 440.78ms, mfu 149.30%
iter 47100: loss 3.0934, time 441.38ms, mfu 149.87%
iter 47200: loss 3.1263, time 439.12ms, mfu 150.46%
iter 47300: loss 3.1413, time 439.19ms, mfu 151.00%
iter 47400: loss 3.1168, time 439.84ms, mfu 151.45%
iter 47500: loss 3.1493, time 438.65ms, mfu 151.90%
iter 47600: loss 3.1645, time 441.57ms, mfu 152.21%
iter 47700: loss 3.1114, time 442.51ms, mfu 152.45%
iter 47800: loss 3.1660, time 441.51ms, mfu 152.70%
iter 47900: loss 3.0851, time 441.90ms, mfu 152.92%
step 48000: train loss 3.0560, val loss 3.0647
saving checkpoint to out-wiki-gutenberg-530m-1ep
iter 48000: loss 3.0687, time 65406.66ms, mfu 137.73%
iter 48100: loss 3.1452, time 441.18ms, mfu 139.46%
iter 48200: loss 3.1037, time 438.71ms, mfu 141.11%
iter 48300: loss 3.1458, time 439.62ms, mfu 142.57%
iter 48400: loss 3.0012, time 440.53ms, mfu 143.84%
iter 48500: loss 3.0982, time 441.51ms, mfu 144.95%
iter 48600: loss 3.0754, time 439.63ms, mfu 146.02%
iter 48700: loss 3.2024, time 440.63ms, mfu 146.95%
iter 48800: loss 3.0808, time 440.69ms, mfu 147.78%
iter 48900: loss 3.1253, time 438.94ms, mfu 148.59%
iter 49000: loss 3.0566, time 440.89ms, mfu 149.25%
iter 49100: loss 3.1114, time 439.45ms, mfu 149.89%
iter 49200: loss 3.0741, time 439.79ms, mfu 150.46%
iter 49300: loss 3.1150, time 441.63ms, mfu 150.91%
iter 49400: loss 3.1475, time 440.70ms, mfu 151.34%
iter 49500: loss 3.0570, time 440.60ms, mfu 151.74%
iter 49600: loss 3.1096, time 439.08ms, mfu 152.15%
iter 49700: loss 3.0753, time 440.17ms, mfu 152.47%
iter 49800: loss 3.1272, time 439.30ms, mfu 152.80%
iter 49900: loss 3.0495, time 441.76ms, mfu 153.01%
step 50000: train loss 3.0468, val loss 3.0641
saving checkpoint to out-wiki-gutenberg-530m-1ep
iter 50000: loss 3.1086, time 65335.35ms, mfu 137.81%
iter 50100: loss 3.0666, time 440.85ms, mfu 139.55%
iter 50200: loss 3.1276, time 439.71ms, mfu 141.16%
iter 50300: loss 3.0820, time 438.84ms, mfu 142.63%
iter 50400: loss 3.0822, time 438.22ms, mfu 143.98%
iter 50500: loss 3.0515, time 439.91ms, mfu 145.14%
iter 50600: loss 3.0729, time 440.66ms, mfu 146.15%
iter 50700: loss 3.0959, time 440.80ms, mfu 147.06%
iter 50800: loss 3.1247, time 442.07ms, mfu 147.83%
iter 50900: loss 3.0912, time 441.28ms, mfu 148.55%
iter 51000: loss 3.0463, time 443.04ms, mfu 149.14%
iter 51100: loss 3.1229, time 439.78ms, mfu 149.78%
iter 51200: loss 3.0375, time 441.10ms, mfu 150.32%
iter 51300: loss 3.0534, time 440.89ms, mfu 150.80%
iter 51400: loss 3.1410, time 441.94ms, mfu 151.20%
iter 51500: loss 3.1354, time 441.35ms, mfu 151.59%
iter 51600: loss 3.1039, time 441.20ms, mfu 151.94%
iter 51700: loss 3.0724, time 439.71ms, mfu 152.30%
iter 51800: loss 3.0448, time 440.62ms, mfu 152.60%
iter 51900: loss 3.0713, time 440.05ms, mfu 152.89%
step 52000: train loss 3.0438, val loss 3.0566
saving checkpoint to out-wiki-gutenberg-530m-1ep
iter 52000: loss 3.0551, time 65444.41ms, mfu 137.70%
iter 52100: loss 3.1701, time 438.14ms, mfu 139.55%
iter 52200: loss 3.1098, time 437.78ms, mfu 141.22%
iter 52300: loss 3.1294, time 441.31ms, mfu 142.61%
iter 52400: loss 3.0976, time 441.41ms, mfu 143.85%
iter 52500: loss 3.0292, time 440.10ms, mfu 145.01%
iter 52600: loss 3.0322, time 439.29ms, mfu 146.08%
iter 52700: loss 3.0833, time 440.24ms, mfu 147.02%
iter 52800: loss 3.0982, time 440.34ms, mfu 147.85%
iter 52900: loss 3.0738, time 440.22ms, mfu 148.61%
iter 53000: loss 3.1700, time 440.62ms, mfu 149.28%
iter 53100: loss 3.0691, time 439.03ms, mfu 149.93%
iter 53200: loss 3.0483, time 437.08ms, mfu 150.59%
iter 53300: loss 3.0580, time 436.85ms, mfu 151.20%
iter 53400: loss 3.1423, time 437.35ms, mfu 151.72%
iter 53500: loss 3.0353, time 437.62ms, mfu 152.18%
iter 53600: loss 3.0790, time 438.44ms, mfu 152.57%
iter 53700: loss 3.0630, time 439.01ms, mfu 152.90%
iter 53800: loss 3.0492, time 436.80ms, mfu 153.27%
iter 53900: loss 3.1117, time 439.91ms, mfu 153.50%
step 54000: train loss 3.0398, val loss 3.0534
saving checkpoint to out-wiki-gutenberg-530m-1ep
iter 54000: loss 3.1341, time 65440.45ms, mfu 138.25%
iter 54100: loss 3.0274, time 439.48ms, mfu 140.00%
iter 54200: loss 3.0515, time 440.98ms, mfu 141.51%
iter 54300: loss 3.0363, time 440.06ms, mfu 142.91%
iter 54400: loss 3.1620, time 441.17ms, mfu 144.13%
iter 54500: loss 3.1331, time 440.40ms, mfu 145.25%
iter 54600: loss 3.1015, time 439.82ms, mfu 146.28%
iter 54700: loss 3.1023, time 439.37ms, mfu 147.23%
iter 54800: loss 3.0969, time 441.32ms, mfu 148.01%
iter 54900: loss 3.0559, time 440.49ms, mfu 148.74%
iter 55000: loss 3.0933, time 439.48ms, mfu 149.43%
iter 55100: loss 3.0770, time 440.10ms, mfu 150.04%
iter 55200: loss 3.0865, time 439.85ms, mfu 150.59%
iter 55300: loss 3.0621, time 438.69ms, mfu 151.13%
iter 55400: loss 3.0879, time 438.94ms, mfu 151.60%
iter 55500: loss 3.0152, time 440.65ms, mfu 151.97%
iter 55600: loss 3.0510, time 440.87ms, mfu 152.29%
iter 55700: loss 3.0697, time 442.77ms, mfu 152.51%
iter 55800: loss 3.0890, time 436.43ms, mfu 152.94%
iter 55900: loss 3.0304, time 440.71ms, mfu 153.17%
step 56000: train loss 3.0321, val loss 3.0409
saving checkpoint to out-wiki-gutenberg-530m-1ep
iter 56000: loss 3.0638, time 65377.31ms, mfu 137.96%
iter 56100: loss 3.0231, time 439.57ms, mfu 139.73%
iter 56200: loss 3.0997, time 439.14ms, mfu 141.34%
iter 56300: loss 3.1754, time 437.52ms, mfu 142.84%
iter 56400: loss 3.0942, time 440.29ms, mfu 144.10%
iter 56500: loss 3.0885, time 438.94ms, mfu 145.27%
iter 56600: loss 3.0345, time 439.80ms, mfu 146.30%
iter 56700: loss 3.0617, time 440.04ms, mfu 147.22%
iter 56800: loss 3.1482, time 439.09ms, mfu 148.08%
iter 56900: loss 3.1352, time 439.00ms, mfu 148.86%
iter 57000: loss 3.0218, time 435.67ms, mfu 149.68%
iter 57100: loss 3.0228, time 439.07ms, mfu 150.29%
iter 57200: loss 3.0959, time 437.88ms, mfu 150.89%
iter 57300: loss 3.1141, time 439.84ms, mfu 151.36%
iter 57400: loss 3.1529, time 438.63ms, mfu 151.82%
iter 57500: loss 3.0456, time 439.39ms, mfu 152.21%
iter 57600: loss 3.0641, time 440.19ms, mfu 152.53%
iter 57700: loss 3.0254, time 440.62ms, mfu 152.81%
iter 57800: loss 3.1110, time 437.25ms, mfu 153.17%
iter 57900: loss 3.0839, time 440.61ms, mfu 153.39%
step 58000: train loss 3.0274, val loss 3.0375
saving checkpoint to out-wiki-gutenberg-530m-1ep
iter 58000: loss 3.1437, time 65514.69ms, mfu 138.15%
iter 58100: loss 3.0870, time 438.90ms, mfu 139.93%
iter 58200: loss 3.0118, time 441.30ms, mfu 141.44%
iter 58300: loss 3.0542, time 440.42ms, mfu 142.83%
iter 58400: loss 3.0486, time 444.09ms, mfu 143.95%
iter 58500: loss 3.0095, time 440.40ms, mfu 145.09%
iter 58600: loss 3.0491, time 442.61ms, mfu 146.04%
iter 58700: loss 3.0548, time 440.76ms, mfu 146.96%
iter 58800: loss 3.0602, time 440.24ms, mfu 147.81%
iter 58900: loss 3.0851, time 442.68ms, mfu 148.48%
iter 59000: loss 3.1125, time 441.40ms, mfu 149.13%
iter 59100: loss 3.1091, time 440.05ms, mfu 149.77%
iter 59200: loss 3.0696, time 439.04ms, mfu 150.38%
iter 59300: loss 3.0043, time 441.42ms, mfu 150.84%
iter 59400: loss 3.0714, time 440.00ms, mfu 151.31%
iter 59500: loss 3.0090, time 441.06ms, mfu 151.69%
iter 59600: loss 3.0509, time 441.22ms, mfu 152.03%
iter 59700: loss 2.9839, time 439.31ms, mfu 152.40%
iter 59800: loss 3.0458, time 442.26ms, mfu 152.63%
iter 59900: loss 3.0372, time 439.29ms, mfu 152.94%
step 60000: train loss 3.0196, val loss 3.0302
saving checkpoint to out-wiki-gutenberg-530m-1ep
iter 60000: loss 3.0854, time 65424.17ms, mfu 137.75%
iter 60100: loss 3.0641, time 436.63ms, mfu 139.65%
iter 60200: loss 3.0030, time 438.99ms, mfu 141.27%
iter 60300: loss 3.0349, time 439.89ms, mfu 142.69%
iter 60400: loss 3.0690, time 439.14ms, mfu 144.01%
iter 60500: loss 3.0861, time 439.68ms, mfu 145.17%
iter 60600: loss 3.0207, time 440.36ms, mfu 146.19%
iter 60700: loss 3.1289, time 440.17ms, mfu 147.11%
iter 60800: loss 3.0466, time 438.89ms, mfu 147.99%
iter 60900: loss 3.0868, time 440.52ms, mfu 148.72%
iter 61000: loss 3.0582, time 439.24ms, mfu 149.43%
iter 61100: loss 3.1366, time 438.59ms, mfu 150.08%
iter 61200: loss 3.0483, time 440.14ms, mfu 150.62%
iter 61300: loss 3.0071, time 438.43ms, mfu 151.16%
iter 61400: loss 3.0812, time 439.24ms, mfu 151.63%
iter 61500: loss 3.0668, time 439.06ms, mfu 152.05%
iter 61600: loss 3.1011, time 439.12ms, mfu 152.42%
iter 61700: loss 3.0613, time 439.92ms, mfu 152.73%
iter 61800: loss 3.0067, time 441.30ms, mfu 152.96%
iter 61900: loss 3.0275, time 439.99ms, mfu 153.22%
step 62000: train loss 3.0096, val loss 3.0239
saving checkpoint to out-wiki-gutenberg-530m-1ep
iter 62000: loss 3.1109, time 65298.10ms, mfu 138.00%
iter 62100: loss 3.1113, time 439.43ms, mfu 139.77%
iter 62200: loss 3.0218, time 441.20ms, mfu 141.30%
iter 62300: loss 3.0534, time 439.83ms, mfu 142.73%
iter 62400: loss 3.0612, time 440.07ms, mfu 144.00%
iter 62500: loss 3.0278, time 439.53ms, mfu 145.17%
iter 62600: loss 3.1147, time 438.57ms, mfu 146.25%
iter 62700: loss 3.0488, time 439.77ms, mfu 147.19%
iter 62800: loss 3.0801, time 438.67ms, mfu 148.06%
iter 62900: loss 3.0712, time 440.71ms, mfu 148.78%
iter 63000: loss 3.1122, time 438.73ms, mfu 149.50%
iter 63100: loss 3.1121, time 438.09ms, mfu 150.17%
iter 63200: loss 3.0891, time 438.62ms, mfu 150.75%
iter 63300: loss 3.0539, time 439.92ms, mfu 151.23%
iter 63400: loss 3.0250, time 440.59ms, mfu 151.63%
iter 63500: loss 3.0018, time 441.27ms, mfu 151.98%
iter 63600: loss 3.0495, time 440.77ms, mfu 152.30%
iter 63700: loss 3.0530, time 440.74ms, mfu 152.60%
iter 63800: loss 3.0913, time 441.20ms, mfu 152.84%
iter 63900: loss 3.1316, time 440.95ms, mfu 153.08%
step 64000: train loss 3.0051, val loss 3.0152
saving checkpoint to out-wiki-gutenberg-530m-1ep
iter 64000: loss 3.0311, time 65547.11ms, mfu 137.87%
iter 64100: loss 3.0031, time 438.93ms, mfu 139.67%
iter 64200: loss 3.0113, time 437.60ms, mfu 141.34%
iter 64300: loss 3.0274, time 438.37ms, mfu 142.82%
iter 64400: loss 2.9791, time 437.53ms, mfu 144.17%
iter 64500: loss 3.1084, time 440.40ms, mfu 145.29%
iter 64600: loss 3.0577, time 437.58ms, mfu 146.40%
iter 64700: loss 3.0359, time 437.50ms, mfu 147.40%
iter 64800: loss 3.0027, time 442.06ms, mfu 148.13%
iter 64900: loss 3.0366, time 440.09ms, mfu 148.87%
iter 65000: loss 3.0392, time 440.47ms, mfu 149.51%
iter 65100: loss 3.0057, time 441.29ms, mfu 150.07%
iter 65200: loss 3.0551, time 438.87ms, mfu 150.65%
iter 65300: loss 3.0577, time 442.30ms, mfu 151.05%
iter 65400: loss 2.9782, time 440.84ms, mfu 151.47%
iter 65500: loss 3.0549, time 440.91ms, mfu 151.84%
iter 65600: loss 3.0731, time 440.99ms, mfu 152.17%
iter 65700: loss 3.0597, time 440.05ms, mfu 152.50%
iter 65800: loss 3.0591, time 439.47ms, mfu 152.82%
iter 65900: loss 3.0704, time 440.06ms, mfu 153.09%
step 66000: train loss 2.9948, val loss 3.0059
saving checkpoint to out-wiki-gutenberg-530m-1ep
iter 66000: loss 2.9691, time 65714.02ms, mfu 137.88%
iter 66100: loss 2.9972, time 438.43ms, mfu 139.70%
iter 66200: loss 3.1161, time 437.73ms, mfu 141.36%
iter 66300: loss 3.0164, time 438.58ms, mfu 142.83%
iter 66400: loss 3.0650, time 439.98ms, mfu 144.09%
iter 66500: loss 3.1190, time 438.58ms, mfu 145.28%
iter 66600: loss 3.0110, time 438.56ms, mfu 146.36%
iter 66700: loss 3.0292, time 439.49ms, mfu 147.29%
iter 66800: loss 3.0599, time 438.23ms, mfu 148.17%
iter 66900: loss 3.0774, time 438.24ms, mfu 148.97%
iter 67000: loss 3.0405, time 438.14ms, mfu 149.69%
iter 67100: loss 3.0034, time 438.40ms, mfu 150.33%
iter 67200: loss 3.0306, time 438.40ms, mfu 150.90%
iter 67300: loss 3.0122, time 437.84ms, mfu 151.44%
iter 67400: loss 3.0555, time 438.40ms, mfu 151.90%
iter 67500: loss 3.0834, time 438.13ms, mfu 152.33%
iter 67600: loss 3.0676, time 438.21ms, mfu 152.71%
iter 67700: loss 3.0356, time 438.19ms, mfu 153.05%
iter 67800: loss 3.0711, time 437.53ms, mfu 153.38%
iter 67900: loss 3.0364, time 440.20ms, mfu 153.59%
step 68000: train loss 2.9898, val loss 3.0041
saving checkpoint to out-wiki-gutenberg-530m-1ep
iter 68000: loss 3.0174, time 65342.47ms, mfu 138.33%
iter 68100: loss 3.0396, time 442.89ms, mfu 139.95%
iter 68200: loss 3.0384, time 438.06ms, mfu 141.57%
iter 68300: loss 3.0651, time 437.81ms, mfu 143.04%
iter 68400: loss 3.0246, time 439.26ms, mfu 144.32%
iter 68500: loss 3.0212, time 439.19ms, mfu 145.46%
iter 68600: loss 3.0281, time 438.28ms, mfu 146.53%
iter 68700: loss 2.9349, time 440.20ms, mfu 147.42%
iter 68800: loss 3.0149, time 438.29ms, mfu 148.29%
iter 68900: loss 2.9418, time 439.87ms, mfu 149.01%
iter 69000: loss 2.9298, time 440.16ms, mfu 149.66%
iter 69100: loss 3.0403, time 439.74ms, mfu 150.25%
iter 69200: loss 3.0286, time 440.03ms, mfu 150.77%
iter 69300: loss 3.0987, time 437.97ms, mfu 151.32%
iter 69400: loss 3.0637, time 439.35ms, mfu 151.76%
iter 69500: loss 2.9498, time 442.28ms, mfu 152.05%
iter 69600: loss 3.1574, time 440.45ms, mfu 152.38%
iter 69700: loss 3.0499, time 439.77ms, mfu 152.70%
iter 69800: loss 3.0068, time 442.84ms, mfu 152.88%
iter 69900: loss 3.0940, time 441.11ms, mfu 153.11%
step 70000: train loss 2.9858, val loss 2.9930
saving checkpoint to out-wiki-gutenberg-530m-1ep
iter 70000: loss 3.0232, time 65356.00ms, mfu 137.90%
iter 70100: loss 3.1126, time 434.98ms, mfu 139.84%
iter 70200: loss 3.0199, time 438.90ms, mfu 141.44%
iter 70300: loss 3.0699, time 438.10ms, mfu 142.92%
iter 70400: loss 3.0581, time 443.47ms, mfu 144.05%
iter 70500: loss 2.9961, time 441.07ms, mfu 145.16%
iter 70600: loss 3.0263, time 438.86ms, mfu 146.24%
iter 70700: loss 2.9721, time 440.47ms, mfu 147.15%
iter 70800: loss 3.0791, time 440.17ms, mfu 147.97%
iter 70900: loss 3.0591, time 440.57ms, mfu 148.71%
iter 71000: loss 3.0493, time 441.22ms, mfu 149.34%
iter 71100: loss 3.0477, time 440.28ms, mfu 149.95%
iter 71200: loss 2.9989, time 441.75ms, mfu 150.44%
iter 71300: loss 2.9560, time 440.06ms, mfu 150.95%
iter 71400: loss 3.0193, time 440.04ms, mfu 151.40%
iter 71500: loss 3.0037, time 440.68ms, mfu 151.79%
iter 71600: loss 2.9993, time 437.52ms, mfu 152.25%
iter 71700: loss 3.0268, time 440.95ms, mfu 152.54%
iter 71800: loss 2.9997, time 439.07ms, mfu 152.87%
iter 71900: loss 3.0690, time 440.83ms, mfu 153.10%
step 72000: train loss 2.9735, val loss 2.9862
saving checkpoint to out-wiki-gutenberg-530m-1ep
iter 72000: loss 3.0948, time 65598.84ms, mfu 137.90%
iter 72100: loss 3.0025, time 439.80ms, mfu 139.66%
iter 72200: loss 3.0854, time 436.41ms, mfu 141.37%
iter 72300: loss 3.0258, time 439.93ms, mfu 142.79%
iter 72400: loss 3.0217, time 437.84ms, mfu 144.14%
iter 72500: loss 3.0330, time 439.45ms, mfu 145.29%
iter 72600: loss 3.0372, time 439.98ms, mfu 146.31%
iter 72700: loss 3.0036, time 439.52ms, mfu 147.25%
iter 72800: loss 3.0626, time 440.25ms, mfu 148.07%
iter 72900: loss 3.0605, time 437.06ms, mfu 148.91%
iter 73000: loss 2.9687, time 441.76ms, mfu 149.51%
iter 73100: loss 3.0497, time 438.09ms, mfu 150.18%
iter 73200: loss 3.0122, time 440.22ms, mfu 150.70%
iter 73300: loss 3.0715, time 439.92ms, mfu 151.19%
iter 73400: loss 3.0639, time 439.37ms, mfu 151.64%
iter 73500: loss 3.0497, time 438.90ms, mfu 152.06%
iter 73600: loss 3.0005, time 439.80ms, mfu 152.41%
iter 73700: loss 3.0588, time 439.21ms, mfu 152.75%
iter 73800: loss 3.0406, time 439.66ms, mfu 153.04%
iter 73900: loss 2.9904, time 438.64ms, mfu 153.33%
step 74000: train loss 2.9638, val loss 2.9808
saving checkpoint to out-wiki-gutenberg-530m-1ep
iter 74000: loss 3.0316, time 65488.44ms, mfu 138.10%
iter 74100: loss 3.1043, time 437.11ms, mfu 139.95%
iter 74200: loss 2.9795, time 437.80ms, mfu 141.58%
iter 74300: loss 3.0361, time 437.77ms, mfu 143.05%
iter 74400: loss 2.9249, time 437.77ms, mfu 144.37%
iter 74500: loss 3.0242, time 437.82ms, mfu 145.56%
iter 74600: loss 2.9896, time 436.90ms, mfu 146.67%
iter 74700: loss 3.0461, time 437.00ms, mfu 147.66%
iter 74800: loss 2.9698, time 438.96ms, mfu 148.48%
iter 74900: loss 3.0505, time 437.60ms, mfu 149.27%
iter 75000: loss 3.0266, time 436.93ms, mfu 150.00%
iter 75100: loss 2.9543, time 438.34ms, mfu 150.61%
iter 75200: loss 3.0689, time 439.56ms, mfu 151.11%
iter 75300: loss 3.0272, time 437.55ms, mfu 151.64%
iter 75400: loss 3.0343, time 438.84ms, mfu 152.07%
iter 75500: loss 2.9909, time 438.34ms, mfu 152.47%
iter 75600: loss 2.9259, time 433.45ms, mfu 153.01%
iter 75700: loss 2.9975, time 437.86ms, mfu 153.33%
iter 75800: loss 2.9459, time 437.86ms, mfu 153.63%
iter 75900: loss 3.0009, time 437.00ms, mfu 153.92%
step 76000: train loss 2.9573, val loss 2.9723
saving checkpoint to out-wiki-gutenberg-530m-1ep
iter 76000: loss 2.9639, time 65298.98ms, mfu 138.63%
iter 76100: loss 2.9857, time 437.93ms, mfu 140.39%
iter 76200: loss 2.9169, time 438.19ms, mfu 141.97%
iter 76300: loss 3.0461, time 438.74ms, mfu 143.37%
iter 76400: loss 3.0179, time 437.18ms, mfu 144.68%
iter 76500: loss 3.0035, time 437.48ms, mfu 145.85%
iter 76600: loss 2.9923, time 438.13ms, mfu 146.88%
iter 76700: loss 3.0297, time 438.27ms, mfu 147.81%
iter 76800: loss 2.9872, time 439.00ms, mfu 148.61%
iter 76900: loss 3.0313, time 438.51ms, mfu 149.35%
iter 77000: loss 2.9735, time 438.09ms, mfu 150.03%
iter 77100: loss 3.0751, time 436.92ms, mfu 150.69%
iter 77200: loss 2.9456, time 437.85ms, mfu 151.25%
iter 77300: loss 2.9293, time 438.58ms, mfu 151.72%
iter 77400: loss 2.9897, time 437.26ms, mfu 152.20%
iter 77500: loss 3.0412, time 437.71ms, mfu 152.61%
iter 77600: loss 3.0513, time 437.83ms, mfu 152.98%
iter 77700: loss 3.0317, time 438.25ms, mfu 153.29%
iter 77800: loss 3.0361, time 437.38ms, mfu 153.61%
iter 77900: loss 2.9983, time 438.15ms, mfu 153.86%
step 78000: train loss 2.9514, val loss 2.9628
saving checkpoint to out-wiki-gutenberg-530m-1ep
iter 78000: loss 3.0739, time 65598.55ms, mfu 138.58%
iter 78100: loss 3.0574, time 437.45ms, mfu 140.36%
iter 78200: loss 2.9734, time 437.92ms, mfu 141.95%
iter 78300: loss 3.0352, time 439.09ms, mfu 143.34%
iter 78400: loss 2.9283, time 438.14ms, mfu 144.62%
iter 78500: loss 3.0011, time 437.72ms, mfu 145.79%
iter 78600: loss 2.9932, time 438.82ms, mfu 146.80%
iter 78700: loss 3.0094, time 436.45ms, mfu 147.80%
iter 78800: loss 2.9800, time 438.74ms, mfu 148.61%
iter 78900: loss 2.9542, time 440.57ms, mfu 149.28%
iter 79000: loss 2.9856, time 438.44ms, mfu 149.96%
iter 79100: loss 2.9302, time 439.43ms, mfu 150.53%
iter 79200: loss 2.9779, time 441.39ms, mfu 150.98%
iter 79300: loss 3.0164, time 440.53ms, mfu 151.41%
iter 79400: loss 2.9777, time 442.27ms, mfu 151.74%
iter 79500: loss 3.0307, time 440.36ms, mfu 152.11%
iter 79600: loss 3.0342, time 435.90ms, mfu 152.59%
iter 79700: loss 3.0273, time 440.87ms, mfu 152.85%
iter 79800: loss 3.0280, time 437.85ms, mfu 153.19%
iter 79900: loss 2.9910, time 439.48ms, mfu 153.44%
step 80000: train loss 2.9408, val loss 2.9569
saving checkpoint to out-wiki-gutenberg-530m-1ep
iter 80000: loss 2.9765, time 65411.26ms, mfu 138.20%
iter 80100: loss 3.0190, time 439.59ms, mfu 139.95%
iter 80200: loss 2.9609, time 440.15ms, mfu 141.50%
iter 80300: loss 3.0224, time 439.79ms, mfu 142.90%
iter 80400: loss 3.0200, time 438.31ms, mfu 144.22%
iter 80500: loss 2.9863, time 440.80ms, mfu 145.32%
iter 80600: loss 2.9867, time 438.36ms, mfu 146.40%
iter 80700: loss 2.9357, time 438.45ms, mfu 147.36%
iter 80800: loss 2.9972, time 438.61ms, mfu 148.23%
iter 80900: loss 3.0588, time 439.56ms, mfu 148.97%
iter 81000: loss 3.0695, time 437.13ms, mfu 149.72%
iter 81100: loss 2.9412, time 439.88ms, mfu 150.31%
iter 81200: loss 3.0060, time 438.25ms, mfu 150.89%
iter 81300: loss 3.0475, time 440.86ms, mfu 151.32%
iter 81400: loss 3.0078, time 442.07ms, mfu 151.66%
iter 81500: loss 2.9458, time 440.70ms, mfu 152.02%
iter 81600: loss 3.0411, time 440.26ms, mfu 152.36%
iter 81700: loss 3.0199, time 442.20ms, mfu 152.60%
iter 81800: loss 2.9923, time 441.32ms, mfu 152.84%
iter 81900: loss 2.9766, time 442.32ms, mfu 153.03%
step 82000: train loss 2.9362, val loss 2.9501
saving checkpoint to out-wiki-gutenberg-530m-1ep
iter 82000: loss 2.9465, time 65582.34ms, mfu 137.83%
iter 82100: loss 2.9394, time 439.73ms, mfu 139.60%
iter 82200: loss 3.0818, time 440.35ms, mfu 141.18%
iter 82300: loss 3.0278, time 436.19ms, mfu 142.75%
iter 82400: loss 2.9714, time 439.33ms, mfu 144.05%
iter 82500: loss 2.8894, time 438.94ms, mfu 145.23%
iter 82600: loss 2.9588, time 438.57ms, mfu 146.31%
iter 82700: loss 3.0096, time 439.47ms, mfu 147.25%
iter 82800: loss 2.9659, time 438.53ms, mfu 148.12%
iter 82900: loss 2.9508, time 441.04ms, mfu 148.83%
iter 83000: loss 3.0372, time 440.44ms, mfu 149.48%
iter 83100: loss 3.0274, time 441.16ms, mfu 150.04%
iter 83200: loss 2.9807, time 438.32ms, mfu 150.64%
iter 83300: loss 2.9766, time 438.18ms, mfu 151.19%
iter 83400: loss 2.9950, time 441.22ms, mfu 151.58%
iter 83500: loss 2.9684, time 440.90ms, mfu 151.94%
iter 83600: loss 3.0050, time 440.87ms, mfu 152.27%
iter 83700: loss 2.9657, time 441.82ms, mfu 152.53%
iter 83800: loss 2.9481, time 439.63ms, mfu 152.84%
iter 83900: loss 2.9321, time 439.32ms, mfu 153.13%
step 84000: train loss 2.9262, val loss 2.9444
saving checkpoint to out-wiki-gutenberg-530m-1ep
iter 84000: loss 3.0083, time 65493.75ms, mfu 137.92%
iter 84100: loss 2.9745, time 441.36ms, mfu 139.63%
iter 84200: loss 2.9842, time 441.42ms, mfu 141.17%
iter 84300: loss 2.9390, time 441.32ms, mfu 142.55%
iter 84400: loss 2.9650, time 442.82ms, mfu 143.75%
iter 84500: loss 2.9894, time 441.60ms, mfu 144.87%
iter 84600: loss 3.0643, time 437.55ms, mfu 146.02%
iter 84700: loss 2.9741, time 441.67ms, mfu 146.91%
iter 84800: loss 2.9471, time 440.87ms, mfu 147.74%
iter 84900: loss 3.0156, time 439.49ms, mfu 148.53%
iter 85000: loss 2.9154, time 440.09ms, mfu 149.23%
iter 85100: loss 2.9607, time 441.57ms, mfu 149.80%
iter 85200: loss 3.0374, time 442.13ms, mfu 150.29%
iter 85300: loss 2.9413, time 438.47ms, mfu 150.87%
iter 85400: loss 2.9965, time 438.95ms, mfu 151.37%
iter 85500: loss 2.9431, time 439.23ms, mfu 151.81%
iter 85600: loss 2.9472, time 441.03ms, mfu 152.14%
iter 85700: loss 2.9410, time 441.67ms, mfu 152.42%
iter 85800: loss 2.9860, time 441.14ms, mfu 152.69%
iter 85900: loss 2.9026, time 437.76ms, mfu 153.05%
step 86000: train loss 2.9146, val loss 2.9330
saving checkpoint to out-wiki-gutenberg-530m-1ep
iter 86000: loss 2.9082, time 65547.86ms, mfu 137.85%
iter 86100: loss 2.9350, time 438.56ms, mfu 139.66%
iter 86200: loss 2.9690, time 437.37ms, mfu 141.34%
iter 86300: loss 2.9492, time 437.77ms, mfu 142.84%
iter 86400: loss 2.9826, time 436.61ms, mfu 144.22%
iter 86500: loss 2.9921, time 439.23ms, mfu 145.38%
iter 86600: loss 2.8777, time 434.29ms, mfu 146.59%
iter 86700: loss 2.9166, time 438.70ms, mfu 147.53%
iter 86800: loss 2.9873, time 438.81ms, mfu 148.37%
iter 86900: loss 2.9912, time 438.29ms, mfu 149.14%
iter 87000: loss 2.9583, time 440.15ms, mfu 149.77%
iter 87100: loss 2.9258, time 441.78ms, mfu 150.28%
iter 87200: loss 2.9614, time 439.83ms, mfu 150.81%
iter 87300: loss 3.0091, time 440.73ms, mfu 151.26%
iter 87400: loss 2.9660, time 439.67ms, mfu 151.69%
iter 87500: loss 2.9393, time 438.90ms, mfu 152.11%
iter 87600: loss 2.9854, time 438.23ms, mfu 152.51%
iter 87700: loss 3.0175, time 439.13ms, mfu 152.84%
iter 87800: loss 2.9602, time 440.80ms, mfu 153.08%
iter 87900: loss 2.9994, time 441.11ms, mfu 153.28%
step 88000: train loss 2.9052, val loss 2.9267
saving checkpoint to out-wiki-gutenberg-530m-1ep
iter 88000: loss 2.9541, time 65709.25ms, mfu 138.06%
iter 88100: loss 2.9144, time 440.60ms, mfu 139.78%
iter 88200: loss 2.9477, time 436.93ms, mfu 141.46%
iter 88300: loss 2.9622, time 440.49ms, mfu 142.85%
iter 88400: loss 2.9708, time 440.44ms, mfu 144.10%
iter 88500: loss 3.0075, time 441.13ms, mfu 145.20%
iter 88600: loss 2.9409, time 440.86ms, mfu 146.20%
iter 88700: loss 2.8936, time 440.64ms, mfu 147.11%
iter 88800: loss 2.9464, time 438.66ms, mfu 147.99%
iter 88900: loss 2.9645, time 438.04ms, mfu 148.81%
iter 89000: loss 2.9264, time 437.94ms, mfu 149.56%
iter 89100: loss 3.0051, time 438.51ms, mfu 150.20%
iter 89200: loss 2.9246, time 441.27ms, mfu 150.69%
iter 89300: loss 2.8754, time 441.59ms, mfu 151.11%
iter 89400: loss 2.9401, time 439.79ms, mfu 151.56%
iter 89500: loss 2.9174, time 439.98ms, mfu 151.95%
iter 89600: loss 2.9097, time 442.98ms, mfu 152.20%
iter 89700: loss 2.9532, time 441.70ms, mfu 152.47%
iter 89800: loss 2.9357, time 440.28ms, mfu 152.77%
iter 89900: loss 2.9332, time 440.80ms, mfu 153.01%
step 90000: train loss 2.9026, val loss 2.9190
saving checkpoint to out-wiki-gutenberg-530m-1ep
iter 90000: loss 2.9688, time 65412.83ms, mfu 137.82%
iter 90100: loss 2.9655, time 441.77ms, mfu 139.52%
iter 90200: loss 2.9756, time 442.40ms, mfu 141.04%
iter 90300: loss 2.9108, time 442.07ms, mfu 142.41%
iter 90400: loss 2.9303, time 440.97ms, mfu 143.68%
iter 90500: loss 2.9480, time 442.29ms, mfu 144.79%
iter 90600: loss 2.9380, time 441.09ms, mfu 145.82%
iter 90700: loss 2.8538, time 440.57ms, mfu 146.77%
iter 90800: loss 2.9550, time 439.72ms, mfu 147.65%
iter 90900: loss 2.8678, time 442.74ms, mfu 148.34%
iter 91000: loss 2.9443, time 441.51ms, mfu 149.00%
iter 91100: loss 2.9807, time 441.07ms, mfu 149.61%
iter 91200: loss 2.9519, time 442.53ms, mfu 150.11%
iter 91300: loss 2.9118, time 443.00ms, mfu 150.55%
iter 91400: loss 2.8929, time 441.83ms, mfu 150.98%
iter 91500: loss 3.0036, time 441.27ms, mfu 151.39%
iter 91600: loss 2.8623, time 440.88ms, mfu 151.77%
iter 91700: loss 3.0222, time 440.24ms, mfu 152.13%
iter 91800: loss 2.9247, time 439.33ms, mfu 152.49%
iter 91900: loss 2.9979, time 441.10ms, mfu 152.75%
step 92000: train loss 2.8951, val loss 2.9064
saving checkpoint to out-wiki-gutenberg-530m-1ep
iter 92000: loss 2.9599, time 65688.85ms, mfu 137.58%
iter 92100: loss 2.9686, time 439.00ms, mfu 139.41%
iter 92200: loss 2.9342, time 440.28ms, mfu 141.01%
iter 92300: loss 2.9535, time 436.45ms, mfu 142.58%
iter 92400: loss 2.9564, time 439.53ms, mfu 143.89%
iter 92500: loss 2.9179, time 438.95ms, mfu 145.09%
iter 92600: loss 2.9405, time 440.72ms, mfu 146.11%
iter 92700: loss 3.0298, time 440.05ms, mfu 147.04%
iter 92800: loss 2.9647, time 440.84ms, mfu 147.86%
iter 92900: loss 2.9731, time 440.29ms, mfu 148.61%
iter 93000: loss 2.8835, time 442.02ms, mfu 149.23%
iter 93100: loss 2.9397, time 440.14ms, mfu 149.85%
iter 93200: loss 2.9325, time 439.87ms, mfu 150.42%
iter 93300: loss 2.9509, time 440.69ms, mfu 150.91%
iter 93400: loss 2.9944, time 440.90ms, mfu 151.33%
iter 93500: loss 2.9185, time 440.65ms, mfu 151.73%
iter 93600: loss 2.9569, time 440.10ms, mfu 152.10%
iter 93700: loss 2.9348, time 440.70ms, mfu 152.42%
iter 93800: loss 2.9680, time 439.96ms, mfu 152.73%
iter 93900: loss 2.9438, time 441.06ms, mfu 152.97%
step 94000: train loss 2.8807, val loss 2.9015
saving checkpoint to out-wiki-gutenberg-530m-1ep
iter 94000: loss 2.9329, time 65846.10ms, mfu 137.77%
iter 94100: loss 2.9846, time 441.87ms, mfu 139.48%
iter 94200: loss 2.9531, time 441.94ms, mfu 141.01%
iter 94300: loss 2.9500, time 440.87ms, mfu 142.43%
iter 94400: loss 2.9647, time 443.10ms, mfu 143.63%
iter 94500: loss 2.9319, time 441.18ms, mfu 144.78%
iter 94600: loss 2.9485, time 440.18ms, mfu 145.84%
iter 94700: loss 3.0074, time 441.72ms, mfu 146.75%
iter 94800: loss 2.9065, time 441.31ms, mfu 147.58%
iter 94900: loss 2.9364, time 440.77ms, mfu 148.34%
iter 95000: loss 2.9349, time 441.64ms, mfu 149.00%
iter 95100: loss 2.9150, time 438.78ms, mfu 149.69%
iter 95200: loss 2.9206, time 437.76ms, mfu 150.35%
iter 95300: loss 2.9083, time 438.51ms, mfu 150.92%
iter 95400: loss 2.9255, time 437.82ms, mfu 151.46%
iter 95500: loss 2.9246, time 438.02ms, mfu 151.93%
iter 95600: loss 3.0054, time 438.74ms, mfu 152.33%
iter 95700: loss 2.9140, time 442.86ms, mfu 152.55%
iter 95800: loss 2.9589, time 440.32ms, mfu 152.83%
iter 95900: loss 2.8783, time 439.88ms, mfu 153.10%
step 96000: train loss 2.8689, val loss 2.8914
saving checkpoint to out-wiki-gutenberg-530m-1ep
iter 96000: loss 2.8810, time 65397.45ms, mfu 137.90%
iter 96100: loss 2.9268, time 441.61ms, mfu 139.60%
iter 96200: loss 2.9547, time 442.89ms, mfu 141.09%
iter 96300: loss 2.9126, time 441.65ms, mfu 142.47%
iter 96400: loss 2.9130, time 441.00ms, mfu 143.74%
iter 96500: loss 2.9455, time 439.69ms, mfu 144.93%
iter 96600: loss 2.9946, time 441.06ms, mfu 145.95%
iter 96700: loss 2.9099, time 441.26ms, mfu 146.86%
iter 96800: loss 2.9280, time 441.13ms, mfu 147.68%
iter 96900: loss 2.8698, time 441.52ms, mfu 148.41%
iter 97000: loss 2.9649, time 438.83ms, mfu 149.16%
iter 97100: loss 2.8705, time 434.39ms, mfu 150.00%
iter 97200: loss 2.8748, time 440.30ms, mfu 150.54%
iter 97300: loss 2.9543, time 439.79ms, mfu 151.04%
iter 97400: loss 2.9275, time 438.32ms, mfu 151.55%
iter 97500: loss 2.9206, time 442.56ms, mfu 151.85%
iter 97600: loss 2.9960, time 438.72ms, mfu 152.26%
iter 97700: loss 2.8706, time 441.10ms, mfu 152.55%
iter 97800: loss 2.9292, time 441.47ms, mfu 152.79%
iter 97900: loss 2.9589, time 439.12ms, mfu 153.09%
step 98000: train loss 2.8713, val loss 2.8859
saving checkpoint to out-wiki-gutenberg-530m-1ep
iter 98000: loss 2.9433, time 65360.28ms, mfu 137.89%
iter 98100: loss 2.9236, time 437.82ms, mfu 139.73%
iter 98200: loss 2.9142, time 437.63ms, mfu 141.39%
iter 98300: loss 2.9327, time 439.98ms, mfu 142.80%
iter 98400: loss 2.8660, time 441.16ms, mfu 144.03%
iter 98500: loss 2.9172, time 438.12ms, mfu 145.24%
iter 98600: loss 2.8958, time 441.72ms, mfu 146.21%
iter 98700: loss 2.9251, time 438.77ms, mfu 147.18%
iter 98800: loss 3.0009, time 441.02ms, mfu 147.98%
iter 98900: loss 2.9627, time 440.88ms, mfu 148.70%
iter 99000: loss 2.9117, time 438.90ms, mfu 149.42%
iter 99100: loss 2.9761, time 442.32ms, mfu 149.94%
iter 99200: loss 2.9181, time 439.01ms, mfu 150.54%
iter 99300: loss 2.9122, time 441.91ms, mfu 150.96%
iter 99400: loss 2.9218, time 438.08ms, mfu 151.49%
iter 99500: loss 2.8803, time 439.53ms, mfu 151.90%
iter 99600: loss 2.9193, time 439.50ms, mfu 152.28%
iter 99700: loss 2.8830, time 441.20ms, mfu 152.56%
iter 99800: loss 2.8586, time 440.78ms, mfu 152.83%
iter 99900: loss 2.8837, time 442.08ms, mfu 153.02%
step 100000: train loss 2.8606, val loss 2.8797
saving checkpoint to out-wiki-gutenberg-530m-1ep
iter 100000: loss 2.9445, time 65533.23ms, mfu 137.82%
iter 100100: loss 2.8718, time 439.55ms, mfu 139.61%
iter 100200: loss 2.8828, time 440.85ms, mfu 141.17%
iter 100300: loss 2.9010, time 440.61ms, mfu 142.58%
iter 100400: loss 2.9367, time 441.15ms, mfu 143.83%
iter 100500: loss 2.9438, time 440.24ms, mfu 144.99%
iter 100600: loss 2.8464, time 441.28ms, mfu 146.00%
iter 100700: loss 2.9042, time 440.06ms, mfu 146.94%
iter 100800: loss 2.9074, time 439.75ms, mfu 147.81%
iter 100900: loss 2.8091, time 441.73ms, mfu 148.52%
iter 101000: loss 2.8996, time 441.06ms, mfu 149.18%
iter 101100: loss 2.8789, time 439.35ms, mfu 149.83%
iter 101200: loss 2.8777, time 440.51ms, mfu 150.38%
iter 101300: loss 2.8334, time 440.83ms, mfu 150.86%
iter 101400: loss 2.8216, time 440.47ms, mfu 151.31%
iter 101500: loss 2.8976, time 442.36ms, mfu 151.65%
iter 101600: loss 2.8487, time 440.83ms, mfu 152.00%
iter 101700: loss 2.9260, time 439.90ms, mfu 152.36%
iter 101800: loss 2.8934, time 440.74ms, mfu 152.64%
iter 101900: loss 2.9018, time 438.32ms, mfu 152.99%
step 102000: train loss 2.8502, val loss 2.8641
saving checkpoint to out-wiki-gutenberg-530m-1ep
iter 102000: loss 2.8634, time 65283.53ms, mfu 137.80%
iter 102100: loss 2.8315, time 440.46ms, mfu 139.55%
iter 102200: loss 2.8964, time 438.88ms, mfu 141.18%
iter 102300: loss 2.8236, time 438.63ms, mfu 142.66%
iter 102400: loss 2.8673, time 439.93ms, mfu 143.95%
iter 102500: loss 2.8717, time 440.87ms, mfu 145.08%
iter 102600: loss 2.9232, time 441.14ms, mfu 146.08%
iter 102700: loss 2.9135, time 438.75ms, mfu 147.06%
iter 102800: loss 2.8939, time 439.90ms, mfu 147.91%
iter 102900: loss 2.8452, time 440.20ms, mfu 148.66%
iter 103000: loss 2.9538, time 440.85ms, mfu 149.32%
iter 103100: loss 2.8981, time 440.72ms, mfu 149.91%
iter 103200: loss 2.8443, time 440.97ms, mfu 150.43%
iter 103300: loss 2.8879, time 440.18ms, mfu 150.93%
iter 103400: loss 2.9201, time 440.99ms, mfu 151.36%
iter 103500: loss 2.9173, time 441.14ms, mfu 151.73%
iter 103600: loss 2.9144, time 442.33ms, mfu 152.03%
iter 103700: loss 2.9383, time 435.56ms, mfu 152.53%
iter 103800: loss 2.8762, time 444.99ms, mfu 152.65%
iter 103900: loss 2.8043, time 437.58ms, mfu 153.02%
step 104000: train loss 2.8393, val loss 2.8539
saving checkpoint to out-wiki-gutenberg-530m-1ep
iter 104000: loss 2.8683, time 65516.72ms, mfu 137.83%
iter 104100: loss 2.8933, time 436.07ms, mfu 139.73%
iter 104200: loss 2.9221, time 440.92ms, mfu 141.28%
iter 104300: loss 2.8598, time 441.44ms, mfu 142.65%
iter 104400: loss 2.8540, time 438.76ms, mfu 143.98%
iter 104500: loss 2.8742, time 441.23ms, mfu 145.09%
iter 104600: loss 2.8288, time 442.52ms, mfu 146.04%
iter 104700: loss 2.8813, time 440.89ms, mfu 146.95%
iter 104800: loss 2.8738, time 440.62ms, mfu 147.79%
iter 104900: loss 2.8902, time 439.91ms, mfu 148.56%
iter 105000: loss 2.8792, time 441.89ms, mfu 149.19%
iter 105100: loss 2.8769, time 441.90ms, mfu 149.75%
iter 105200: loss 2.8926, time 441.14ms, mfu 150.29%
iter 105300: loss 2.8640, time 438.58ms, mfu 150.86%
iter 105400: loss 2.8965, time 440.09ms, mfu 151.32%
iter 105500: loss 2.8628, time 443.68ms, mfu 151.61%
iter 105600: loss 2.9195, time 442.92ms, mfu 151.90%
iter 105700: loss 2.9042, time 440.99ms, mfu 152.22%
iter 105800: loss 2.8231, time 439.44ms, mfu 152.57%
iter 105900: loss 2.8982, time 442.37ms, mfu 152.78%
step 106000: train loss 2.8309, val loss 2.8478
saving checkpoint to out-wiki-gutenberg-530m-1ep
iter 106000: loss 2.8691, time 65609.33ms, mfu 137.61%
iter 106100: loss 2.8959, time 440.65ms, mfu 139.37%
iter 106200: loss 2.9283, time 439.90ms, mfu 140.99%
iter 106300: loss 2.9250, time 440.80ms, mfu 142.41%
iter 106400: loss 2.8535, time 441.25ms, mfu 143.68%
iter 106500: loss 2.8930, time 440.56ms, mfu 144.84%
iter 106600: loss 2.9439, time 441.26ms, mfu 145.86%
iter 106700: loss 2.8818, time 440.05ms, mfu 146.82%
iter 106800: loss 2.8768, time 440.56ms, mfu 147.67%
iter 106900: loss 2.8946, time 441.53ms, mfu 148.40%
iter 107000: loss 2.9425, time 441.62ms, mfu 149.05%
iter 107100: loss 2.9048, time 439.78ms, mfu 149.71%
iter 107200: loss 2.9147, time 441.06ms, mfu 150.25%
iter 107300: loss 2.9014, time 439.32ms, mfu 150.80%
iter 107400: loss 2.8473, time 439.92ms, mfu 151.27%
iter 107500: loss 2.8262, time 442.30ms, mfu 151.61%
iter 107600: loss 2.8969, time 438.53ms, mfu 152.05%
iter 107700: loss 2.9108, time 441.37ms, mfu 152.35%
iter 107800: loss 2.8964, time 440.98ms, mfu 152.63%
iter 107900: loss 2.8772, time 441.58ms, mfu 152.86%
step 108000: train loss 2.8179, val loss 2.8391
saving checkpoint to out-wiki-gutenberg-530m-1ep
iter 108000: loss 2.9332, time 65294.80ms, mfu 137.68%
iter 108100: loss 2.9012, time 440.60ms, mfu 139.44%
iter 108200: loss 2.9023, time 440.68ms, mfu 141.02%
iter 108300: loss 2.8371, time 440.97ms, mfu 142.44%
iter 108400: loss 2.8989, time 443.29ms, mfu 143.63%
iter 108500: loss 2.8963, time 439.92ms, mfu 144.82%
iter 108600: loss 2.8682, time 438.51ms, mfu 145.94%
iter 108700: loss 2.9415, time 440.50ms, mfu 146.88%
iter 108800: loss 2.9263, time 440.74ms, mfu 147.71%
iter 108900: loss 2.8211, time 440.43ms, mfu 148.48%
iter 109000: loss 2.8775, time 442.16ms, mfu 149.10%
iter 109100: loss 2.8797, time 441.23ms, mfu 149.70%
iter 109200: loss 2.8588, time 439.91ms, mfu 150.28%
iter 109300: loss 2.8618, time 439.47ms, mfu 150.82%
iter 109400: loss 2.8647, time 441.18ms, mfu 151.25%
iter 109500: loss 2.7807, time 440.38ms, mfu 151.66%
iter 109600: loss 2.8184, time 441.41ms, mfu 152.00%
iter 109700: loss 2.8834, time 441.13ms, mfu 152.31%
iter 109800: loss 2.9062, time 439.78ms, mfu 152.63%
iter 109900: loss 2.8404, time 439.23ms, mfu 152.95%
step 110000: train loss 2.8121, val loss 2.8344
saving checkpoint to out-wiki-gutenberg-530m-1ep
iter 110000: loss 2.9468, time 65452.59ms, mfu 137.76%
iter 110100: loss 2.8952, time 440.41ms, mfu 139.52%
iter 110200: loss 2.8594, time 442.24ms, mfu 141.04%
iter 110300: loss 2.8343, time 441.45ms, mfu 142.43%
iter 110400: loss 2.8454, time 440.00ms, mfu 143.74%
iter 110500: loss 2.8636, time 441.09ms, mfu 144.88%
iter 110600: loss 2.8063, time 440.61ms, mfu 145.92%
iter 110700: loss 2.9544, time 443.86ms, mfu 146.74%
iter 110800: loss 2.8225, time 441.31ms, mfu 147.57%
iter 110900: loss 2.9179, time 442.33ms, mfu 148.28%
iter 111000: loss 2.8416, time 440.25ms, mfu 148.99%
iter 111100: loss 2.8758, time 442.93ms, mfu 149.54%
iter 111200: loss 2.8718, time 440.38ms, mfu 150.13%
iter 111300: loss 2.9011, time 439.00ms, mfu 150.70%
iter 111400: loss 2.8385, time 442.41ms, mfu 151.09%
iter 111500: loss 2.8232, time 444.76ms, mfu 151.37%
iter 111600: loss 2.9330, time 441.64ms, mfu 151.72%
iter 111700: loss 2.8223, time 440.51ms, mfu 152.08%
iter 111800: loss 2.8453, time 440.18ms, mfu 152.42%
iter 111900: loss 2.7992, time 440.96ms, mfu 152.69%
step 112000: train loss 2.8069, val loss 2.8247
saving checkpoint to out-wiki-gutenberg-530m-1ep
iter 112000: loss 2.8589, time 65653.75ms, mfu 137.53%
iter 112100: loss 2.8063, time 433.34ms, mfu 139.56%
iter 112200: loss 2.8442, time 441.00ms, mfu 141.12%
iter 112300: loss 2.7892, time 435.95ms, mfu 142.70%
iter 112400: loss 2.8514, time 443.14ms, mfu 143.87%
iter 112500: loss 2.8236, time 442.51ms, mfu 144.95%
iter 112600: loss 2.8906, time 442.76ms, mfu 145.91%
iter 112700: loss 2.8318, time 439.14ms, mfu 146.90%
iter 112800: loss 2.8489, time 440.93ms, mfu 147.72%
iter 112900: loss 2.8398, time 440.04ms, mfu 148.50%
iter 113000: loss 2.8333, time 439.81ms, mfu 149.21%
iter 113100: loss 2.8870, time 439.47ms, mfu 149.86%
iter 113200: loss 2.8974, time 440.93ms, mfu 150.39%
iter 113300: loss 2.8274, time 440.48ms, mfu 150.88%
iter 113400: loss 2.9033, time 439.30ms, mfu 151.37%
iter 113500: loss 2.8073, time 441.36ms, mfu 151.73%
iter 113600: loss 2.8538, time 442.87ms, mfu 152.01%
iter 113700: loss 2.8755, time 439.71ms, mfu 152.37%
iter 113800: loss 2.8030, time 442.40ms, mfu 152.60%
iter 113900: loss 2.8522, time 437.92ms, mfu 152.96%
step 114000: train loss 2.7994, val loss 2.8177
saving checkpoint to out-wiki-gutenberg-530m-1ep
iter 114000: loss 2.7186, time 65533.58ms, mfu 137.77%
iter 114100: loss 2.8597, time 443.73ms, mfu 139.41%
iter 114200: loss 2.7810, time 440.44ms, mfu 141.01%
iter 114300: loss 2.9057, time 439.41ms, mfu 142.48%
iter 114400: loss 2.8561, time 441.48ms, mfu 143.73%
iter 114500: loss 2.8427, time 440.72ms, mfu 144.88%
iter 114600: loss 2.8784, time 441.59ms, mfu 145.88%
iter 114700: loss 2.8019, time 442.30ms, mfu 146.77%
iter 114800: loss 2.8505, time 440.93ms, mfu 147.61%
iter 114900: loss 2.8134, time 440.00ms, mfu 148.40%
iter 115000: loss 2.8616, time 439.42ms, mfu 149.13%
iter 115100: loss 2.8605, time 441.81ms, mfu 149.70%
iter 115200: loss 2.8474, time 443.10ms, mfu 150.17%
iter 115300: loss 2.8767, time 439.68ms, mfu 150.72%
iter 115400: loss 2.8730, time 434.14ms, mfu 151.40%
iter 115500: loss 2.8667, time 441.00ms, mfu 151.78%
iter 115600: loss 2.7871, time 439.12ms, mfu 152.18%
iter 115700: loss 2.8599, time 441.86ms, mfu 152.45%
iter 115800: loss 2.8359, time 434.86ms, mfu 152.94%
iter 115900: loss 2.8446, time 444.20ms, mfu 153.05%
step 116000: train loss 2.7871, val loss 2.8135
saving checkpoint to out-wiki-gutenberg-530m-1ep
iter 116000: loss 2.8704, time 65761.15ms, mfu 137.85%
iter 116100: loss 2.8968, time 438.61ms, mfu 139.66%
iter 116200: loss 2.8724, time 441.28ms, mfu 141.20%
iter 116300: loss 2.8824, time 441.08ms, mfu 142.59%
iter 116400: loss 2.8774, time 435.40ms, mfu 144.05%
iter 116500: loss 2.8419, time 441.45ms, mfu 145.14%
iter 116600: loss 2.7817, time 442.87ms, mfu 146.08%
iter 116700: loss 2.8297, time 442.16ms, mfu 146.94%
iter 116800: loss 2.7375, time 439.17ms, mfu 147.83%
iter 116900: loss 2.8947, time 440.89ms, mfu 148.56%
iter 117000: loss 2.8754, time 440.55ms, mfu 149.24%
iter 117100: loss 2.7665, time 439.41ms, mfu 149.89%
iter 117200: loss 2.7990, time 444.02ms, mfu 150.31%
iter 117300: loss 2.8387, time 441.11ms, mfu 150.79%
iter 117400: loss 2.8929, time 442.40ms, mfu 151.17%
iter 117500: loss 2.7976, time 442.49ms, mfu 151.52%
iter 117600: loss 2.8134, time 439.71ms, mfu 151.93%
iter 117700: loss 2.7726, time 440.54ms, mfu 152.27%
iter 117800: loss 2.8762, time 443.03ms, mfu 152.48%
iter 117900: loss 2.8190, time 440.43ms, mfu 152.77%
step 118000: train loss 2.7863, val loss 2.8032
saving checkpoint to out-wiki-gutenberg-530m-1ep
iter 118000: loss 2.8672, time 65690.25ms, mfu 137.60%
iter 118100: loss 2.8376, time 443.01ms, mfu 139.28%
iter 118200: loss 2.8407, time 440.66ms, mfu 140.88%
iter 118300: loss 2.8197, time 442.26ms, mfu 142.26%
iter 118400: loss 2.7998, time 438.98ms, mfu 143.62%
iter 118500: loss 2.8570, time 440.67ms, mfu 144.79%
iter 118600: loss 2.8095, time 439.62ms, mfu 145.87%
iter 118700: loss 2.8174, time 443.19ms, mfu 146.72%
iter 118800: loss 2.8729, time 441.74ms, mfu 147.54%
iter 118900: loss 2.8126, time 438.70ms, mfu 148.38%
iter 119000: loss 2.7807, time 441.84ms, mfu 149.03%
iter 119100: loss 2.8243, time 441.92ms, mfu 149.61%
iter 119200: loss 2.8336, time 440.16ms, mfu 150.19%
iter 119300: loss 2.8085, time 439.77ms, mfu 150.73%
iter 119400: loss 2.8617, time 441.80ms, mfu 151.14%
iter 119500: loss 2.8478, time 439.28ms, mfu 151.61%
iter 119600: loss 2.8275, time 443.04ms, mfu 151.89%
iter 119700: loss 2.8272, time 441.46ms, mfu 152.20%
iter 119800: loss 2.8077, time 439.17ms, mfu 152.56%
iter 119900: loss 2.8324, time 440.91ms, mfu 152.82%
step 120000: train loss 2.7764, val loss 2.7948
saving checkpoint to out-wiki-gutenberg-530m-1ep
iter 120000: loss 2.8556, time 65414.84ms, mfu 137.64%
iter 120100: loss 2.8183, time 439.73ms, mfu 139.44%
iter 120200: loss 2.7740, time 441.18ms, mfu 141.00%
iter 120300: loss 2.7821, time 440.20ms, mfu 142.45%
iter 120400: loss 2.7554, time 433.85ms, mfu 143.97%
iter 120500: loss 2.8011, time 441.45ms, mfu 145.07%
iter 120600: loss 2.8431, time 444.78ms, mfu 145.95%
iter 120700: loss 2.8372, time 441.21ms, mfu 146.86%
iter 120800: loss 2.8175, time 432.67ms, mfu 147.99%
iter 120900: loss 2.7497, time 442.54ms, mfu 148.65%
iter 121000: loss 2.7956, time 440.99ms, mfu 149.30%
iter 121100: loss 2.8291, time 440.01ms, mfu 149.92%
iter 121200: loss 2.7597, time 439.76ms, mfu 150.49%
iter 121300: loss 2.7891, time 442.61ms, mfu 150.90%
iter 121400: loss 2.8242, time 439.47ms, mfu 151.38%
iter 121500: loss 2.8993, time 438.32ms, mfu 151.85%
iter 121600: loss 2.8140, time 438.95ms, mfu 152.25%
iter 121700: loss 2.7881, time 439.68ms, mfu 152.59%
iter 121800: loss 2.7911, time 441.33ms, mfu 152.83%
iter 121900: loss 2.8824, time 441.26ms, mfu 153.05%
step 122000: train loss 2.7670, val loss 2.7888
saving checkpoint to out-wiki-gutenberg-530m-1ep
iter 122000: loss 2.8614, time 65505.33ms, mfu 137.85%
iter 122100: loss 2.8434, time 438.94ms, mfu 139.66%
iter 122200: loss 2.7899, time 439.73ms, mfu 141.25%
iter 122300: loss 2.8125, time 440.51ms, mfu 142.66%
iter 122400: loss 2.7955, time 441.64ms, mfu 143.88%
iter 122500: loss 2.7594, time 444.32ms, mfu 144.89%
iter 122600: loss 2.8907, time 443.27ms, mfu 145.84%
iter 122700: loss 2.8648, time 440.72ms, mfu 146.78%
iter 122800: loss 2.8149, time 441.06ms, mfu 147.61%
iter 122900: loss 2.8159, time 439.95ms, mfu 148.41%
iter 123000: loss 2.8114, time 445.07ms, mfu 148.94%
iter 123100: loss 2.8256, time 441.07ms, mfu 149.56%
iter 123200: loss 2.8304, time 443.19ms, mfu 150.04%
iter 123300: loss 2.7887, time 440.80ms, mfu 150.56%
iter 123400: loss 2.7741, time 440.07ms, mfu 151.05%
iter 123500: loss 2.8599, time 441.63ms, mfu 151.44%
iter 123600: loss 2.7530, time 442.85ms, mfu 151.74%
iter 123700: loss 2.7967, time 440.83ms, mfu 152.09%
iter 123800: loss 2.8116, time 441.59ms, mfu 152.37%
iter 123900: loss 2.8388, time 442.57ms, mfu 152.60%
step 124000: train loss 2.7536, val loss 2.7844
saving checkpoint to out-wiki-gutenberg-530m-1ep
iter 124000: loss 2.7749, time 65548.67ms, mfu 137.44%
iter 124100: loss 2.8764, time 437.81ms, mfu 139.32%
iter 124200: loss 2.8399, time 439.51ms, mfu 140.96%
iter 124300: loss 2.8156, time 441.41ms, mfu 142.36%
iter 124400: loss 2.8278, time 436.19ms, mfu 143.81%
iter 124500: loss 2.7536, time 440.73ms, mfu 144.96%
iter 124600: loss 2.7980, time 438.64ms, mfu 146.06%
iter 124700: loss 2.8289, time 443.08ms, mfu 146.90%
iter 124800: loss 2.7836, time 440.77ms, mfu 147.73%
iter 124900: loss 2.7926, time 435.95ms, mfu 148.65%
iter 125000: loss 2.8268, time 441.68ms, mfu 149.28%
iter 125100: loss 2.7899, time 439.24ms, mfu 149.93%
iter 125200: loss 2.7296, time 439.46ms, mfu 150.50%
iter 125300: loss 2.8671, time 440.25ms, mfu 150.99%
iter 125400: loss 2.8313, time 442.21ms, mfu 151.37%
iter 125500: loss 2.8739, time 440.14ms, mfu 151.77%
iter 125600: loss 2.8147, time 441.52ms, mfu 152.09%
iter 125700: loss 2.8039, time 440.30ms, mfu 152.42%
iter 125800: loss 2.8488, time 438.91ms, mfu 152.77%
iter 125900: loss 2.7709, time 439.64ms, mfu 153.06%
step 126000: train loss 2.7582, val loss 2.7787
saving checkpoint to out-wiki-gutenberg-530m-1ep
iter 126000: loss 2.8225, time 65705.47ms, mfu 137.85%
iter 126100: loss 2.7964, time 439.59ms, mfu 139.63%
iter 126200: loss 2.7949, time 434.45ms, mfu 141.42%
iter 126300: loss 2.7715, time 441.63ms, mfu 142.77%
iter 126400: loss 2.8160, time 441.32ms, mfu 144.00%
iter 126500: loss 2.8389, time 441.04ms, mfu 145.11%
iter 126600: loss 2.7874, time 442.28ms, mfu 146.07%
iter 126700: loss 2.7688, time 440.45ms, mfu 147.00%
iter 126800: loss 2.7711, time 442.12ms, mfu 147.77%
iter 126900: loss 2.7036, time 441.05ms, mfu 148.51%
iter 127000: loss 2.8151, time 440.12ms, mfu 149.20%
iter 127100: loss 2.8384, time 441.08ms, mfu 149.79%
iter 127200: loss 2.7610, time 440.01ms, mfu 150.36%
iter 127300: loss 2.7682, time 441.71ms, mfu 150.82%
iter 127400: loss 2.8408, time 440.31ms, mfu 151.28%
iter 127500: loss 2.8193, time 442.28ms, mfu 151.62%
iter 127600: loss 2.8509, time 442.31ms, mfu 151.92%
iter 127700: loss 2.8121, time 440.67ms, mfu 152.26%
iter 127800: loss 2.7416, time 442.17ms, mfu 152.51%
iter 127900: loss 2.8069, time 444.85ms, mfu 152.64%
step 128000: train loss 2.7548, val loss 2.7764
saving checkpoint to out-wiki-gutenberg-530m-1ep
iter 128000: loss 2.7691, time 65564.38ms, mfu 137.48%
iter 128100: loss 2.7954, time 440.80ms, mfu 139.25%
iter 128200: loss 2.8105, time 440.43ms, mfu 140.86%
iter 128300: loss 2.7383, time 441.48ms, mfu 142.27%
iter 128400: loss 2.8281, time 441.35ms, mfu 143.55%
iter 128500: loss 2.7763, time 443.87ms, mfu 144.61%
iter 128600: loss 2.8198, time 441.87ms, mfu 145.63%
iter 128700: loss 2.8018, time 439.97ms, mfu 146.62%
iter 128800: loss 2.8550, time 443.92ms, mfu 147.37%
iter 128900: loss 2.7483, time 441.07ms, mfu 148.15%
iter 129000: loss 2.7486, time 442.83ms, mfu 148.78%
iter 129100: loss 2.8854, time 442.26ms, mfu 149.37%
iter 129200: loss 2.7922, time 443.30ms, mfu 149.87%
iter 129300: loss 2.7137, time 446.38ms, mfu 150.21%
iter 129400: loss 2.8317, time 439.50ms, mfu 150.76%
iter 129500: loss 2.7223, time 441.18ms, mfu 151.19%
iter 129600: loss 2.8584, time 439.49ms, mfu 151.64%
iter 129700: loss 2.8009, time 442.30ms, mfu 151.95%
iter 129800: loss 2.9103, time 441.96ms, mfu 152.23%
iter 129900: loss 2.7666, time 442.79ms, mfu 152.46%
step 130000: train loss 2.7483, val loss 2.7697
saving checkpoint to out-wiki-gutenberg-530m-1ep
iter 130000: loss 2.8124, time 65799.62ms, mfu 137.32%
iter 130100: loss 2.8043, time 441.01ms, mfu 139.10%
iter 130200: loss 2.7715, time 440.01ms, mfu 140.74%
iter 130300: loss 2.8107, time 439.34ms, mfu 142.24%
iter 130400: loss 2.7556, time 442.41ms, mfu 143.48%
iter 130500: loss 2.7954, time 441.54ms, mfu 144.63%
iter 130600: loss 2.8161, time 440.13ms, mfu 145.71%
iter 130700: loss 2.7706, time 441.41ms, mfu 146.64%
iter 130800: loss 2.8254, time 440.91ms, mfu 147.49%
iter 130900: loss 2.7722, time 444.19ms, mfu 148.15%
iter 131000: loss 2.7176, time 441.95ms, mfu 148.82%
iter 131100: loss 2.7961, time 439.31ms, mfu 149.51%
iter 131200: loss 2.7558, time 440.95ms, mfu 150.07%
iter 131300: loss 2.8392, time 441.34ms, mfu 150.57%
iter 131400: loss 2.8425, time 442.77ms, mfu 150.97%
iter 131500: loss 2.7823, time 439.03ms, mfu 151.45%
iter 131600: loss 2.6663, time 438.23ms, mfu 151.92%
iter 131700: loss 2.7957, time 443.53ms, mfu 152.15%
iter 131800: loss 2.7258, time 443.30ms, mfu 152.37%
iter 131900: loss 2.8340, time 441.09ms, mfu 152.65%
step 132000: train loss 2.7441, val loss 2.7680
saving checkpoint to out-wiki-gutenberg-530m-1ep
iter 132000: loss 2.7797, time 65457.70ms, mfu 137.49%
iter 132100: loss 2.8091, time 441.48ms, mfu 139.24%
iter 132200: loss 2.7777, time 441.62ms, mfu 140.81%
iter 132300: loss 2.7519, time 442.63ms, mfu 142.18%
iter 132400: loss 2.8004, time 440.96ms, mfu 143.48%
iter 132500: loss 2.7831, time 439.82ms, mfu 144.69%
iter 132600: loss 2.7491, time 442.04ms, mfu 145.70%
iter 132700: loss 2.7711, time 441.93ms, mfu 146.61%
iter 132800: loss 2.7886, time 441.62ms, mfu 147.44%
iter 132900: loss 2.7077, time 438.57ms, mfu 148.30%
iter 133000: loss 2.7744, time 442.46ms, mfu 148.93%
iter 133100: loss 2.8471, time 443.40ms, mfu 149.47%
iter 133200: loss 2.7616, time 439.65ms, mfu 150.09%
iter 133300: loss 2.8758, time 440.86ms, mfu 150.60%
iter 133400: loss 2.7872, time 440.40ms, mfu 151.07%
iter 133500: loss 2.8764, time 439.62ms, mfu 151.53%
iter 133600: loss 2.7800, time 441.22ms, mfu 151.88%
iter 133700: loss 2.8041, time 444.59ms, mfu 152.08%
iter 133800: loss 2.8194, time 441.35ms, mfu 152.38%
iter 133900: loss 2.7547, time 439.98ms, mfu 152.69%
step 134000: train loss 2.7390, val loss 2.7650
saving checkpoint to out-wiki-gutenberg-530m-1ep
iter 134000: loss 2.7572, time 65663.64ms, mfu 137.53%
iter 134100: loss 2.7551, time 440.54ms, mfu 139.31%
iter 134200: loss 2.7672, time 439.60ms, mfu 140.94%
iter 134300: loss 2.8065, time 442.56ms, mfu 142.31%
iter 134400: loss 2.7587, time 441.61ms, mfu 143.57%
iter 134500: loss 2.7616, time 441.45ms, mfu 144.71%
iter 134600: loss 2.7910, time 435.63ms, mfu 145.94%
iter 134700: loss 2.8514, time 440.35ms, mfu 146.89%
iter 134800: loss 2.7414, time 441.74ms, mfu 147.69%
iter 134900: loss 2.8629, time 440.16ms, mfu 148.46%
iter 135000: loss 2.8098, time 439.35ms, mfu 149.19%
iter 135100: loss 2.8385, time 443.71ms, mfu 149.69%
iter 135200: loss 2.7048, time 440.65ms, mfu 150.25%
iter 135300: loss 2.7254, time 444.26ms, mfu 150.63%
iter 135400: loss 2.6914, time 441.42ms, mfu 151.06%
iter 135500: loss 2.7797, time 441.67ms, mfu 151.45%
iter 135600: loss 2.8389, time 442.64ms, mfu 151.76%
iter 135700: loss 2.8208, time 441.67ms, mfu 152.08%
iter 135800: loss 2.7976, time 441.08ms, mfu 152.38%
iter 135900: loss 2.7482, time 440.82ms, mfu 152.66%
step 136000: train loss 2.7352, val loss 2.7654
saving checkpoint to out-wiki-gutenberg-530m-1ep
iter 136000: loss 2.7647, time 65743.26ms, mfu 137.50%
iter 136100: loss 2.8031, time 440.62ms, mfu 139.28%
iter 136200: loss 2.7676, time 440.96ms, mfu 140.87%
iter 136300: loss 2.8472, time 441.35ms, mfu 142.28%
iter 136400: loss 2.7656, time 437.28ms, mfu 143.70%
iter 136500: loss 2.8012, time 441.28ms, mfu 144.84%