File size: 97,782 Bytes
4deb0f5 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 |
*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
*****************************************
train: 27839 of 27839 rows fit in 3072 tokens
val: 300 of 300 rows fit in 3072 tokens
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00, 130.07it/s]
train: 27839 of 27839 rows fit in 3072 tokens
trainable params: 40,370,176 || all params: 7,655,986,688 || trainable%: 0.5273
val: 300 of 300 rows fit in 3072 tokens
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00, 128.48it/s]
trainable params: 40,370,176 || all params: 7,655,986,688 || trainable%: 0.5273
0%| | 0/870 [00:00<?, ?it/s]`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
0%| | 1/870 [00:12<2:54:06, 12.02s/it]
0%| | 2/870 [00:21<2:35:52, 10.78s/it]
0%| | 3/870 [00:31<2:25:50, 10.09s/it]
0%| | 4/870 [00:40<2:20:18, 9.72s/it]
1%| | 5/870 [00:49<2:15:37, 9.41s/it]
1%| | 6/870 [00:57<2:10:20, 9.05s/it]
1%| | 7/870 [01:05<2:06:14, 8.78s/it]
1%| | 8/870 [01:13<2:02:41, 8.54s/it]
1%| | 9/870 [01:22<2:02:41, 8.55s/it]
1%| | 10/870 [01:29<1:57:18, 8.18s/it]
{'loss': 0.2436, 'grad_norm': 0.23789972066879272, 'learning_rate': 6.666666666666667e-05, 'epoch': 0.01}
1%| | 10/870 [01:29<1:57:18, 8.18s/it]
1%|▏ | 11/870 [01:37<1:54:25, 7.99s/it]
1%|▏ | 12/870 [01:44<1:51:11, 7.78s/it]
1%|▏ | 13/870 [01:51<1:47:37, 7.53s/it]
2%|▏ | 14/870 [01:58<1:46:39, 7.48s/it]
2%|▏ | 15/870 [02:05<1:43:51, 7.29s/it]
2%|▏ | 16/870 [02:12<1:40:38, 7.07s/it]
2%|▏ | 17/870 [02:18<1:36:44, 6.80s/it]
2%|▏ | 18/870 [02:24<1:34:07, 6.63s/it]
2%|▏ | 19/870 [02:31<1:32:42, 6.54s/it]
2%|▏ | 20/870 [02:36<1:29:24, 6.31s/it]
{'loss': 0.0853, 'grad_norm': 0.12851466238498688, 'learning_rate': 0.00014074074074074076, 'epoch': 0.02}
2%|▏ | 20/870 [02:36<1:29:24, 6.31s/it]
2%|▏ | 21/870 [02:42<1:26:33, 6.12s/it]
3%|▎ | 22/870 [02:47<1:23:38, 5.92s/it]
3%|▎ | 23/870 [02:53<1:20:05, 5.67s/it]
3%|▎ | 24/870 [02:57<1:15:37, 5.36s/it]
3%|▎ | 25/870 [03:00<1:05:59, 4.69s/it]
3%|▎ | 26/870 [03:12<1:35:34, 6.79s/it]
3%|▎ | 27/870 [03:22<1:47:13, 7.63s/it]
3%|▎ | 28/870 [03:31<1:55:34, 8.24s/it]
3%|▎ | 29/870 [03:40<1:59:03, 8.49s/it]
3%|▎ | 30/870 [03:49<1:59:05, 8.51s/it]
{'loss': 0.0627, 'grad_norm': 0.16485165059566498, 'learning_rate': 0.0001999972223815946, 'epoch': 0.03}
3%|▎ | 30/870 [03:49<1:59:05, 8.51s/it]
4%|▎ | 31/870 [03:57<1:58:25, 8.47s/it]
4%|▎ | 32/870 [04:05<1:56:34, 8.35s/it]
4%|▍ | 33/870 [04:13<1:55:11, 8.26s/it]
4%|▍ | 34/870 [04:21<1:53:30, 8.15s/it]
4%|▍ | 35/870 [04:29<1:50:43, 7.96s/it]
4%|▍ | 36/870 [04:36<1:49:00, 7.84s/it]
4%|▍ | 37/870 [04:44<1:46:33, 7.68s/it]
4%|▍ | 38/870 [04:51<1:45:18, 7.59s/it]
4%|▍ | 39/870 [04:58<1:42:40, 7.41s/it]
5%|▍ | 40/870 [05:05<1:39:36, 7.20s/it]
{'loss': 0.048, 'grad_norm': 0.08059470355510712, 'learning_rate': 0.00019990002193828922, 'epoch': 0.05}
5%|▍ | 40/870 [05:05<1:39:36, 7.20s/it]
5%|▍ | 41/870 [05:11<1:36:34, 6.99s/it]
5%|▍ | 42/870 [05:17<1:33:16, 6.76s/it]
5%|▍ | 43/870 [05:24<1:32:19, 6.70s/it]
5%|▌ | 44/870 [05:30<1:29:42, 6.52s/it]
5%|▌ | 45/870 [05:36<1:27:24, 6.36s/it]
5%|▌ | 46/870 [05:42<1:24:36, 6.16s/it]
5%|▌ | 47/870 [05:47<1:22:03, 5.98s/it]
6%|▌ | 48/870 [05:53<1:20:04, 5.85s/it]
6%|▌ | 49/870 [05:57<1:14:49, 5.47s/it]
6%|▌ | 50/870 [06:01<1:05:50, 4.82s/it]
{'loss': 0.0421, 'grad_norm': 0.10289835184812546, 'learning_rate': 0.0001996640948394666, 'epoch': 0.06}
6%|▌ | 50/870 [06:01<1:05:50, 4.82s/it]
6%|▌ | 51/870 [06:12<1:33:52, 6.88s/it]
6%|▌ | 52/870 [06:23<1:47:31, 7.89s/it]
6%|▌ | 53/870 [06:32<1:55:00, 8.45s/it]
6%|▌ | 54/870 [06:42<1:58:20, 8.70s/it]
6%|▋ | 55/870 [06:51<1:58:49, 8.75s/it]
6%|▋ | 56/870 [06:59<1:57:23, 8.65s/it]
7%|▋ | 57/870 [07:08<1:56:38, 8.61s/it]
7%|▋ | 58/870 [07:15<1:53:18, 8.37s/it]
7%|▋ | 59/870 [07:23<1:51:14, 8.23s/it]
7%|▋ | 60/870 [07:31<1:48:41, 8.05s/it]
{'loss': 0.05, 'grad_norm': 0.10482244938611984, 'learning_rate': 0.00019928976870644988, 'epoch': 0.07}
7%|▋ | 60/870 [07:31<1:48:41, 8.05s/it]
7%|▋ | 61/870 [07:38<1:45:49, 7.85s/it]
7%|▋ | 62/870 [07:46<1:44:05, 7.73s/it]
7%|▋ | 63/870 [07:53<1:41:06, 7.52s/it]
7%|▋ | 64/870 [08:00<1:38:54, 7.36s/it]
7%|▋ | 65/870 [08:07<1:36:39, 7.20s/it]
8%|▊ | 66/870 [08:13<1:35:02, 7.09s/it]
8%|▊ | 67/870 [08:20<1:32:27, 6.91s/it]
8%|▊ | 68/870 [08:26<1:29:09, 6.67s/it]
8%|▊ | 69/870 [08:32<1:26:27, 6.48s/it]
8%|▊ | 70/870 [08:38<1:24:14, 6.32s/it]
{'loss': 0.0377, 'grad_norm': 0.10887732356786728, 'learning_rate': 0.00019877756334905984, 'epoch': 0.08}
8%|▊ | 70/870 [08:38<1:24:14, 6.32s/it]
8%|▊ | 71/870 [08:44<1:21:37, 6.13s/it]
8%|▊ | 72/870 [08:49<1:20:08, 6.03s/it]
8%|▊ | 73/870 [08:55<1:16:25, 5.75s/it]
9%|▊ | 74/870 [08:59<1:12:26, 5.46s/it]
9%|▊ | 75/870 [09:03<1:03:50, 4.82s/it]
9%|▊ | 76/870 [09:15<1:32:44, 7.01s/it]
9%|▉ | 77/870 [09:25<1:47:17, 8.12s/it]
9%|▉ | 78/870 [09:35<1:52:17, 8.51s/it]
9%|▉ | 79/870 [09:44<1:55:33, 8.76s/it]
9%|▉ | 80/870 [09:53<1:57:10, 8.90s/it]
{'loss': 0.0427, 'grad_norm': 0.10280067473649979, 'learning_rate': 0.00019812819004377862, 'epoch': 0.09}
9%|▉ | 80/870 [09:53<1:57:10, 8.90s/it]
9%|▉ | 81/870 [10:02<1:55:32, 8.79s/it]
9%|▉ | 82/870 [10:11<1:54:32, 8.72s/it]
10%|▉ | 83/870 [10:19<1:52:27, 8.57s/it]
10%|▉ | 84/870 [10:27<1:49:44, 8.38s/it]
10%|▉ | 85/870 [10:35<1:47:23, 8.21s/it]
10%|▉ | 86/870 [10:42<1:45:50, 8.10s/it]
10%|█ | 87/870 [10:50<1:42:51, 7.88s/it]
10%|█ | 88/870 [10:57<1:39:21, 7.62s/it]
10%|█ | 89/870 [11:04<1:36:59, 7.45s/it]
10%|█ | 90/870 [11:11<1:34:03, 7.23s/it]
{'loss': 0.0375, 'grad_norm': 0.1345912218093872, 'learning_rate': 0.0001973425505460321, 'epoch': 0.1}
10%|█ | 90/870 [11:11<1:34:03, 7.23s/it]
10%|█ | 91/870 [11:18<1:33:21, 7.19s/it]
11%|█ | 92/870 [11:24<1:29:23, 6.89s/it]
11%|█ | 93/870 [11:30<1:26:27, 6.68s/it]
11%|█ | 94/870 [11:36<1:24:23, 6.52s/it]
11%|█ | 95/870 [11:42<1:21:49, 6.34s/it]
11%|█ | 96/870 [11:48<1:20:51, 6.27s/it]
11%|█ | 97/870 [11:54<1:17:53, 6.05s/it]
11%|█▏ | 98/870 [11:59<1:14:14, 5.77s/it]
11%|█▏ | 99/870 [12:03<1:08:35, 5.34s/it]
11%|█▏ | 100/870 [12:07<1:00:45, 4.73s/it]
{'loss': 0.0357, 'grad_norm': 0.3418385982513428, 'learning_rate': 0.00019642173583796267, 'epoch': 0.11}
11%|█▏ | 100/870 [12:07<1:00:45, 4.73s/it]
0%| | 0/38 [00:00<?, ?it/s][A
5%|▌ | 2/38 [00:00<00:13, 2.71it/s][A
8%|▊ | 3/38 [00:01<00:15, 2.22it/s][A
11%|█ | 4/38 [00:01<00:16, 2.09it/s][A
13%|█▎ | 5/38 [00:02<00:16, 2.01it/s][A
16%|█▌ | 6/38 [00:02<00:15, 2.00it/s][A
18%|█▊ | 7/38 [00:03<00:15, 2.05it/s][A
21%|██ | 8/38 [00:03<00:13, 2.17it/s][A
24%|██▎ | 9/38 [00:04<00:12, 2.34it/s][A
26%|██▋ | 10/38 [00:04<00:13, 2.05it/s][A
29%|██▉ | 11/38 [00:05<00:14, 1.82it/s][A
32%|███▏ | 12/38 [00:05<00:14, 1.80it/s][A
34%|███▍ | 13/38 [00:06<00:13, 1.87it/s][A
37%|███▋ | 14/38 [00:07<00:15, 1.54it/s][A
39%|███▉ | 15/38 [00:07<00:13, 1.68it/s][A
42%|████▏ | 16/38 [00:08<00:11, 1.85it/s][A
45%|████▍ | 17/38 [00:08<00:10, 2.05it/s][A
47%|████▋ | 18/38 [00:08<00:08, 2.37it/s][A
50%|█████ | 19/38 [00:09<00:08, 2.22it/s][A
53%|█████▎ | 20/38 [00:10<00:08, 2.01it/s][A
55%|█████▌ | 21/38 [00:10<00:08, 1.97it/s][A
58%|█████▊ | 22/38 [00:11<00:08, 1.93it/s][A
61%|██████ | 23/38 [00:11<00:07, 1.95it/s][A
63%|██████▎ | 24/38 [00:12<00:06, 2.02it/s][A
66%|██████▌ | 25/38 [00:12<00:06, 2.12it/s][A
68%|██████▊ | 26/38 [00:12<00:05, 2.22it/s][A
71%|███████ | 27/38 [00:13<00:04, 2.36it/s][A
74%|███████▎ | 28/38 [00:13<00:04, 2.05it/s][A
76%|███████▋ | 29/38 [00:14<00:04, 1.83it/s][A
79%|███████▉ | 30/38 [00:15<00:04, 1.79it/s][A
82%|████████▏ | 31/38 [00:15<00:03, 1.81it/s][A
84%|████████▍ | 32/38 [00:16<00:03, 1.86it/s][A
87%|████████▋ | 33/38 [00:16<00:02, 1.90it/s][A
89%|████████▉ | 34/38 [00:17<00:02, 1.97it/s][A
92%|█████████▏| 35/38 [00:17<00:01, 2.07it/s][A
95%|█████████▍| 36/38 [00:17<00:00, 2.21it/s][A
97%|█████████▋| 37/38 [00:18<00:00, 2.05it/s][A
100%|██████████| 38/38 [00:19<00:00, 1.98it/s][A
[A{'eval_loss': 0.052477214485406876, 'eval_runtime': 20.6792, 'eval_samples_per_second': 14.507, 'eval_steps_per_second': 1.838, 'epoch': 0.11}
11%|█▏ | 100/870 [12:27<1:00:45, 4.73s/it]
100%|██████████| 38/38 [00:19<00:00, 1.98it/s][A
[A
12%|█▏ | 101/870 [12:40<2:49:23, 13.22s/it]
12%|█▏ | 102/870 [12:50<2:37:13, 12.28s/it]
12%|█▏ | 103/870 [12:59<2:26:46, 11.48s/it]
12%|█▏ | 104/870 [13:09<2:19:54, 10.96s/it]
12%|█▏ | 105/870 [13:18<2:12:26, 10.39s/it]
12%|█▏ | 106/870 [13:27<2:05:02, 9.82s/it]
12%|█▏ | 107/870 [13:35<1:59:00, 9.36s/it]
12%|█▏ | 108/870 [13:43<1:54:14, 9.00s/it]
13%|█▎ | 109/870 [13:51<1:50:00, 8.67s/it]
13%|█▎ | 110/870 [13:59<1:47:06, 8.46s/it]
{'loss': 0.047, 'grad_norm': 0.07799140363931656, 'learning_rate': 0.00019536702461343168, 'epoch': 0.13}
13%|█▎ | 110/870 [13:59<1:47:06, 8.46s/it]
13%|█▎ | 111/870 [14:06<1:43:48, 8.21s/it]
13%|█▎ | 112/870 [14:14<1:40:26, 7.95s/it]
13%|█▎ | 113/870 [14:21<1:37:19, 7.71s/it]
13%|█▎ | 114/870 [14:28<1:34:16, 7.48s/it]
13%|█▎ | 115/870 [14:35<1:32:24, 7.34s/it]
13%|█▎ | 116/870 [14:42<1:29:43, 7.14s/it]
13%|█▎ | 117/870 [14:48<1:26:33, 6.90s/it]
14%|█▎ | 118/870 [14:54<1:23:41, 6.68s/it]
14%|█▎ | 119/870 [15:01<1:22:48, 6.62s/it]
14%|█▍ | 120/870 [15:06<1:19:57, 6.40s/it]
{'loss': 0.0327, 'grad_norm': 0.1051594689488411, 'learning_rate': 0.00019417988150235494, 'epoch': 0.14}
14%|█▍ | 120/870 [15:06<1:19:57, 6.40s/it]
14%|█▍ | 121/870 [15:12<1:17:24, 6.20s/it]
14%|█▍ | 122/870 [15:18<1:14:53, 6.01s/it]
14%|█▍ | 123/870 [15:23<1:11:52, 5.77s/it]
14%|█▍ | 124/870 [15:28<1:08:24, 5.50s/it]
14%|█▍ | 125/870 [15:31<59:10, 4.77s/it]
14%|█▍ | 126/870 [15:43<1:24:39, 6.83s/it]
15%|█▍ | 127/870 [15:53<1:36:16, 7.77s/it]
15%|█▍ | 128/870 [16:02<1:42:15, 8.27s/it]
15%|█▍ | 129/870 [16:11<1:46:32, 8.63s/it]
15%|█▍ | 130/870 [16:20<1:46:36, 8.64s/it]
{'loss': 0.0418, 'grad_norm': 0.08026126027107239, 'learning_rate': 0.00019286195503683708, 'epoch': 0.15}
15%|█▍ | 130/870 [16:20<1:46:36, 8.64s/it]
15%|█▌ | 131/870 [16:28<1:45:15, 8.55s/it]
15%|█▌ | 132/870 [16:37<1:43:36, 8.42s/it]
15%|█▌ | 133/870 [16:44<1:39:39, 8.11s/it]
15%|█▌ | 134/870 [16:52<1:39:04, 8.08s/it]
16%|█▌ | 135/870 [17:00<1:37:09, 7.93s/it]
16%|█▌ | 136/870 [17:07<1:35:12, 7.78s/it]
16%|█▌ | 137/870 [17:14<1:31:46, 7.51s/it]
16%|█▌ | 138/870 [17:21<1:29:34, 7.34s/it]
16%|█▌ | 139/870 [17:28<1:28:47, 7.29s/it]
16%|█▌ | 140/870 [17:35<1:26:31, 7.11s/it]
{'loss': 0.037, 'grad_norm': 0.07206742465496063, 'learning_rate': 0.0001914150753619294, 'epoch': 0.16}
16%|█▌ | 140/870 [17:35<1:26:31, 7.11s/it]
16%|█▌ | 141/870 [17:41<1:24:18, 6.94s/it]
16%|█▋ | 142/870 [17:47<1:21:43, 6.74s/it]
16%|█▋ | 143/870 [17:54<1:19:23, 6.55s/it]
17%|█▋ | 144/870 [18:00<1:18:18, 6.47s/it]
17%|█▋ | 145/870 [18:06<1:16:04, 6.30s/it]
17%|█▋ | 146/870 [18:11<1:13:41, 6.11s/it]
17%|█▋ | 147/870 [18:17<1:11:05, 5.90s/it]
17%|█▋ | 148/870 [18:22<1:07:37, 5.62s/it]
17%|█▋ | 149/870 [18:26<1:03:45, 5.31s/it]
17%|█▋ | 150/870 [18:29<55:08, 4.60s/it]
{'loss': 0.033, 'grad_norm': 0.1900654435157776, 'learning_rate': 0.00018984125169419007, 'epoch': 0.17}
17%|█▋ | 150/870 [18:29<55:08, 4.60s/it]
17%|█▋ | 151/870 [18:41<1:21:17, 6.78s/it]
17%|█▋ | 152/870 [18:51<1:33:13, 7.79s/it]
18%|█▊ | 153/870 [19:01<1:40:06, 8.38s/it]
18%|█▊ | 154/870 [19:10<1:42:27, 8.59s/it]
18%|█▊ | 155/870 [19:19<1:41:57, 8.56s/it]
18%|█▊ | 156/870 [19:27<1:40:58, 8.49s/it]
18%|█▊ | 157/870 [19:35<1:39:13, 8.35s/it]
18%|█▊ | 158/870 [19:43<1:38:27, 8.30s/it]
18%|█▊ | 159/870 [19:51<1:36:54, 8.18s/it]
18%|█▊ | 160/870 [19:59<1:34:58, 8.03s/it]
{'loss': 0.0348, 'grad_norm': 0.11883541941642761, 'learning_rate': 0.00018814266953157556, 'epoch': 0.18}
18%|█▊ | 160/870 [19:59<1:34:58, 8.03s/it]
19%|█▊ | 161/870 [20:06<1:32:47, 7.85s/it]
19%|█▊ | 162/870 [20:13<1:30:13, 7.65s/it]
19%|█▊ | 163/870 [20:21<1:28:32, 7.51s/it]
19%|█▉ | 164/870 [20:27<1:26:09, 7.32s/it]
19%|█▉ | 165/870 [20:34<1:23:41, 7.12s/it]
19%|█▉ | 166/870 [20:40<1:20:47, 6.88s/it]
19%|█▉ | 167/870 [20:47<1:18:31, 6.70s/it]
19%|█▉ | 168/870 [20:53<1:17:24, 6.62s/it]
19%|█▉ | 169/870 [20:59<1:15:11, 6.44s/it]
20%|█▉ | 170/870 [21:05<1:12:58, 6.25s/it]
{'loss': 0.0296, 'grad_norm': 0.11700455099344254, 'learning_rate': 0.00018632168761853826, 'epoch': 0.2}
20%|█▉ | 170/870 [21:05<1:12:58, 6.25s/it]
20%|█▉ | 171/870 [21:11<1:10:43, 6.07s/it]
20%|█▉ | 172/870 [21:16<1:09:16, 5.95s/it]
20%|█▉ | 173/870 [21:21<1:06:06, 5.69s/it]
20%|██ | 174/870 [21:26<1:01:11, 5.27s/it]
20%|██ | 175/870 [21:29<53:16, 4.60s/it]
20%|██ | 176/870 [21:40<1:18:00, 6.74s/it]
20%|██ | 177/870 [21:51<1:29:45, 7.77s/it]
20%|██ | 178/870 [22:00<1:35:25, 8.27s/it]
21%|██ | 179/870 [22:09<1:38:36, 8.56s/it]
21%|██ | 180/870 [22:18<1:38:21, 8.55s/it]
{'loss': 0.0305, 'grad_norm': 0.15726420283317566, 'learning_rate': 0.00018438083467054443, 'epoch': 0.21}
21%|██ | 180/870 [22:18<1:38:21, 8.55s/it]
21%|██ | 181/870 [22:26<1:37:35, 8.50s/it]
21%|██ | 182/870 [22:35<1:37:03, 8.46s/it]
21%|██ | 183/870 [22:42<1:34:58, 8.29s/it]
21%|██ | 184/870 [22:50<1:33:56, 8.22s/it]
21%|██▏ | 185/870 [22:58<1:31:03, 7.98s/it]
21%|██▏ | 186/870 [23:06<1:29:59, 7.89s/it]
21%|██▏ | 187/870 [23:13<1:27:01, 7.65s/it]
22%|██▏ | 188/870 [23:20<1:24:43, 7.45s/it]
22%|██▏ | 189/870 [23:27<1:23:00, 7.31s/it]
22%|██▏ | 190/870 [23:33<1:20:43, 7.12s/it]
{'loss': 0.0319, 'grad_norm': 0.18836188316345215, 'learning_rate': 0.00018232280586256102, 'epoch': 0.22}
22%|██▏ | 190/870 [23:33<1:20:43, 7.12s/it]
22%|██▏ | 191/870 [23:40<1:19:15, 7.00s/it]
22%|██▏ | 192/870 [23:46<1:16:44, 6.79s/it]
22%|██▏ | 193/870 [23:52<1:14:11, 6.57s/it]
22%|██▏ | 194/870 [23:58<1:12:08, 6.40s/it]
22%|██▏ | 195/870 [24:04<1:09:33, 6.18s/it]
23%|██▎ | 196/870 [24:10<1:08:43, 6.12s/it]
23%|██▎ | 197/870 [24:15<1:05:52, 5.87s/it]
23%|██▎ | 198/870 [24:20<1:02:47, 5.61s/it]
23%|██▎ | 199/870 [24:25<58:23, 5.22s/it]
23%|██▎ | 200/870 [24:28<50:26, 4.52s/it]
{'loss': 0.0259, 'grad_norm': 0.08308509737253189, 'learning_rate': 0.0001801504590863877, 'epoch': 0.23}
23%|██▎ | 200/870 [24:28<50:26, 4.52s/it]
0%| | 0/38 [00:00<?, ?it/s][A
5%|▌ | 2/38 [00:00<00:12, 2.80it/s][A
8%|▊ | 3/38 [00:01<00:15, 2.25it/s][A
11%|█ | 4/38 [00:01<00:16, 2.09it/s][A
13%|█▎ | 5/38 [00:02<00:15, 2.10it/s][A
16%|█▌ | 6/38 [00:02<00:15, 2.11it/s][A
18%|█▊ | 7/38 [00:03<00:14, 2.12it/s][A
21%|██ | 8/38 [00:03<00:13, 2.23it/s][A
24%|██▎ | 9/38 [00:03<00:12, 2.39it/s][A
26%|██▋ | 10/38 [00:04<00:13, 2.12it/s][A
29%|██▉ | 11/38 [00:05<00:14, 1.90it/s][A
32%|███▏ | 12/38 [00:05<00:13, 1.88it/s][A
34%|███▍ | 13/38 [00:06<00:13, 1.90it/s][A
37%|███▋ | 14/38 [00:06<00:12, 1.91it/s][A
39%|███▉ | 15/38 [00:07<00:11, 1.99it/s][A
42%|████▏ | 16/38 [00:07<00:10, 2.11it/s][A
45%|████▍ | 17/38 [00:08<00:09, 2.27it/s][A
47%|████▋ | 18/38 [00:08<00:08, 2.44it/s][A
50%|█████ | 19/38 [00:08<00:08, 2.12it/s][A
53%|█████▎ | 20/38 [00:09<00:09, 1.90it/s][A
55%|█████▌ | 21/38 [00:10<00:09, 1.89it/s][A
58%|█████▊ | 22/38 [00:10<00:08, 1.84it/s][A
61%|██████ | 23/38 [00:11<00:08, 1.87it/s][A
63%|██████▎ | 24/38 [00:11<00:07, 1.90it/s][A
66%|██████▌ | 25/38 [00:12<00:06, 1.98it/s][A
68%|██████▊ | 26/38 [00:12<00:05, 2.08it/s][A
71%|███████ | 27/38 [00:13<00:04, 2.25it/s][A
74%|███████▎ | 28/38 [00:13<00:05, 1.99it/s][A
76%|███████▋ | 29/38 [00:14<00:05, 1.78it/s][A
79%|███████▉ | 30/38 [00:14<00:04, 1.77it/s][A
82%|████████▏ | 31/38 [00:15<00:03, 1.77it/s][A
84%|████████▍ | 32/38 [00:15<00:03, 1.82it/s][A
87%|████████▋ | 33/38 [00:16<00:02, 1.91it/s][A
89%|████████▉ | 34/38 [00:16<00:01, 2.05it/s][A
92%|█████████▏| 35/38 [00:17<00:01, 2.16it/s][A
95%|█████████▍| 36/38 [00:17<00:00, 2.38it/s][A
97%|█████████▋| 37/38 [00:18<00:00, 2.09it/s][A
100%|██████████| 38/38 [00:18<00:00, 1.94it/s][A
[A{'eval_loss': 0.02992151863873005, 'eval_runtime': 20.3901, 'eval_samples_per_second': 14.713, 'eval_steps_per_second': 1.864, 'epoch': 0.23}
23%|██▎ | 200/870 [24:48<50:26, 4.52s/it]
100%|██████████| 38/38 [00:19<00:00, 1.94it/s][A
[A
23%|██▎ | 201/870 [25:02<2:28:54, 13.36s/it]
23%|██▎ | 202/870 [25:12<2:18:46, 12.47s/it]
23%|██▎ | 203/870 [25:21<2:08:46, 11.58s/it]
23%|██▎ | 204/870 [25:31<2:00:47, 10.88s/it]
24%|██▎ | 205/870 [25:40<1:54:07, 10.30s/it]
24%|██▎ | 206/870 [25:48<1:47:43, 9.73s/it]
24%|██▍ | 207/870 [25:56<1:41:57, 9.23s/it]
24%|██▍ | 208/870 [26:04<1:37:56, 8.88s/it]
24%|██▍ | 209/870 [26:12<1:34:34, 8.58s/it]
24%|██▍ | 210/870 [26:20<1:31:57, 8.36s/it]
{'loss': 0.0313, 'grad_norm': 0.05260265991091728, 'learning_rate': 0.00017786681098203158, 'epoch': 0.24}
24%|██▍ | 210/870 [26:20<1:31:57, 8.36s/it]
24%|██▍ | 211/870 [26:27<1:28:32, 8.06s/it]
24%|██▍ | 212/870 [26:34<1:25:01, 7.75s/it]
24%|██▍ | 213/870 [26:41<1:22:38, 7.55s/it]
25%|██▍ | 214/870 [26:48<1:19:51, 7.30s/it]
25%|██▍ | 215/870 [26:55<1:19:00, 7.24s/it]
25%|██▍ | 216/870 [27:01<1:15:53, 6.96s/it]
25%|██▍ | 217/870 [27:08<1:13:05, 6.72s/it]
25%|██▌ | 218/870 [27:14<1:11:04, 6.54s/it]
25%|██▌ | 219/870 [27:20<1:08:29, 6.31s/it]
25%|██▌ | 220/870 [27:26<1:07:56, 6.27s/it]
{'loss': 0.0229, 'grad_norm': 0.0633602887392044, 'learning_rate': 0.00017547503274863496, 'epoch': 0.25}
25%|██▌ | 220/870 [27:26<1:07:56, 6.27s/it]
25%|██▌ | 221/870 [27:31<1:05:34, 6.06s/it]
26%|██▌ | 222/870 [27:37<1:03:39, 5.89s/it]
26%|██▌ | 223/870 [27:42<1:01:25, 5.70s/it]
26%|██▌ | 224/870 [27:47<58:18, 5.42s/it]
26%|██▌ | 225/870 [27:50<52:15, 4.86s/it]
26%|██▌ | 226/870 [28:02<1:13:42, 6.87s/it]
26%|██▌ | 227/870 [28:12<1:24:12, 7.86s/it]
26%|██▌ | 228/870 [28:22<1:29:24, 8.36s/it]
26%|██▋ | 229/870 [28:31<1:31:35, 8.57s/it]
26%|██▋ | 230/870 [28:39<1:31:50, 8.61s/it]
{'loss': 0.028, 'grad_norm': 0.11880059540271759, 'learning_rate': 0.00017297844574077446, 'epoch': 0.26}
26%|██▋ | 230/870 [28:39<1:31:50, 8.61s/it]
27%|██▋ | 231/870 [28:47<1:29:59, 8.45s/it]
27%|██▋ | 232/870 [28:56<1:28:52, 8.36s/it]
27%|██▋ | 233/870 [29:03<1:26:45, 8.17s/it]
27%|██▋ | 234/870 [29:11<1:25:40, 8.08s/it]
27%|██▋ | 235/870 [29:19<1:23:59, 7.94s/it]
27%|██▋ | 236/870 [29:26<1:22:17, 7.79s/it]
27%|██▋ | 237/870 [29:33<1:20:06, 7.59s/it]
27%|██▋ | 238/870 [29:40<1:17:46, 7.38s/it]
27%|██▋ | 239/870 [29:47<1:16:25, 7.27s/it]
28%|██▊ | 240/870 [29:54<1:14:52, 7.13s/it]
{'loss': 0.0264, 'grad_norm': 0.05497035011649132, 'learning_rate': 0.0001703805168562453, 'epoch': 0.28}
28%|██▊ | 240/870 [29:54<1:14:52, 7.13s/it]
28%|██▊ | 241/870 [30:01<1:12:49, 6.95s/it]
28%|██▊ | 242/870 [30:07<1:10:42, 6.76s/it]
28%|██▊ | 243/870 [30:13<1:08:38, 6.57s/it]
28%|██▊ | 244/870 [30:19<1:07:52, 6.51s/it]
28%|██▊ | 245/870 [30:25<1:05:45, 6.31s/it]
28%|██▊ | 246/870 [30:31<1:03:40, 6.12s/it]
28%|██▊ | 247/870 [30:36<1:01:43, 5.95s/it]
29%|██▊ | 248/870 [30:42<1:00:15, 5.81s/it]
29%|██▊ | 249/870 [30:47<56:18, 5.44s/it]
29%|██▊ | 250/870 [30:50<49:19, 4.77s/it]
{'loss': 0.0213, 'grad_norm': 0.07999792695045471, 'learning_rate': 0.00016768485372173697, 'epoch': 0.29}
29%|██▊ | 250/870 [30:50<49:19, 4.77s/it]
29%|██▉ | 251/870 [31:02<1:10:55, 6.87s/it]
29%|██▉ | 252/870 [31:11<1:20:12, 7.79s/it]
29%|██▉ | 253/870 [31:21<1:25:52, 8.35s/it]
29%|██▉ | 254/870 [31:30<1:28:35, 8.63s/it]
29%|██▉ | 255/870 [31:39<1:28:40, 8.65s/it]
29%|██▉ | 256/870 [31:48<1:27:49, 8.58s/it]
30%|██▉ | 257/870 [31:56<1:26:38, 8.48s/it]
30%|██▉ | 258/870 [32:04<1:25:45, 8.41s/it]
30%|██▉ | 259/870 [32:12<1:24:03, 8.25s/it]
30%|██▉ | 260/870 [32:20<1:22:20, 8.10s/it]
{'loss': 0.0284, 'grad_norm': 0.0921878069639206, 'learning_rate': 0.00016489519968308457, 'epoch': 0.3}
30%|██▉ | 260/870 [32:20<1:22:20, 8.10s/it]
30%|███ | 261/870 [32:27<1:20:35, 7.94s/it]
30%|███ | 262/870 [32:34<1:18:24, 7.74s/it]
30%|███ | 263/870 [32:42<1:17:16, 7.64s/it]
30%|███ | 264/870 [32:49<1:15:02, 7.43s/it]
30%|███ | 265/870 [32:56<1:13:07, 7.25s/it]
31%|███ | 266/870 [33:02<1:11:12, 7.07s/it]
31%|███ | 267/870 [33:09<1:09:14, 6.89s/it]
31%|███ | 268/870 [33:15<1:06:39, 6.64s/it]
31%|███ | 269/870 [33:21<1:04:38, 6.45s/it]
31%|███ | 270/870 [33:27<1:02:39, 6.27s/it]
{'loss': 0.0216, 'grad_norm': 0.057495057582855225, 'learning_rate': 0.00016201542860705336, 'epoch': 0.31}
31%|███ | 270/870 [33:27<1:02:39, 6.27s/it]
31%|███ | 271/870 [33:32<1:00:25, 6.05s/it]
31%|███▏ | 272/870 [33:38<59:24, 5.96s/it]
31%|███▏ | 273/870 [33:43<56:27, 5.67s/it]
31%|███▏ | 274/870 [33:47<52:26, 5.28s/it]
32%|███▏ | 275/870 [33:50<46:02, 4.64s/it]
32%|███▏ | 276/870 [34:02<1:06:43, 6.74s/it]
32%|███▏ | 277/870 [34:12<1:17:13, 7.81s/it]
32%|███▏ | 278/870 [34:22<1:22:07, 8.32s/it]
32%|███▏ | 279/870 [34:31<1:24:50, 8.61s/it]
32%|███▏ | 280/870 [34:40<1:25:32, 8.70s/it]
{'loss': 0.0235, 'grad_norm': 0.0719340518116951, 'learning_rate': 0.00015904953950187457, 'epoch': 0.32}
32%|███▏ | 280/870 [34:40<1:25:32, 8.70s/it]
32%|███▏ | 281/870 [34:49<1:24:48, 8.64s/it]
32%|███▏ | 282/870 [34:57<1:24:20, 8.61s/it]
33%|███▎ | 283/870 [35:05<1:23:02, 8.49s/it]
33%|███▎ | 284/870 [35:13<1:21:13, 8.32s/it]
33%|███▎ | 285/870 [35:21<1:19:05, 8.11s/it]
33%|███▎ | 286/870 [35:29<1:17:25, 7.95s/it]
33%|███▎ | 287/870 [35:36<1:16:30, 7.87s/it]
33%|███▎ | 288/870 [35:43<1:14:17, 7.66s/it]
33%|███▎ | 289/870 [35:50<1:12:23, 7.48s/it]
33%|███▎ | 290/870 [35:57<1:10:05, 7.25s/it]
{'loss': 0.0232, 'grad_norm': 0.11424289643764496, 'learning_rate': 0.00015600165096400307, 'epoch': 0.33}
33%|███▎ | 290/870 [35:57<1:10:05, 7.25s/it]
33%|███▎ | 291/870 [36:04<1:09:33, 7.21s/it]
34%|███▎ | 292/870 [36:11<1:07:12, 6.98s/it]
34%|███▎ | 293/870 [36:17<1:05:04, 6.77s/it]
34%|███▍ | 294/870 [36:23<1:02:47, 6.54s/it]
34%|███▍ | 295/870 [36:29<1:01:02, 6.37s/it]
34%|███▍ | 296/870 [36:35<59:47, 6.25s/it]
34%|███▍ | 297/870 [36:40<57:44, 6.05s/it]
34%|███▍ | 298/870 [36:46<55:17, 5.80s/it]
34%|███▍ | 299/870 [36:50<51:47, 5.44s/it]
34%|███▍ | 300/870 [36:54<45:18, 4.77s/it]
{'loss': 0.0185, 'grad_norm': 0.11561417579650879, 'learning_rate': 0.0001528759954588082, 'epoch': 0.34}
34%|███▍ | 300/870 [36:54<45:18, 4.77s/it]
0%| | 0/38 [00:00<?, ?it/s][A
5%|▌ | 2/38 [00:00<00:13, 2.64it/s][A
8%|▊ | 3/38 [00:01<00:16, 2.17it/s][A
11%|█ | 4/38 [00:01<00:16, 2.10it/s][A
13%|█▎ | 5/38 [00:02<00:16, 2.06it/s][A
16%|█▌ | 6/38 [00:02<00:15, 2.11it/s][A
18%|█▊ | 7/38 [00:03<00:14, 2.19it/s][A
21%|██ | 8/38 [00:03<00:13, 2.28it/s][A
24%|██▎ | 9/38 [00:03<00:11, 2.43it/s][A
26%|██▋ | 10/38 [00:04<00:13, 2.15it/s][A
29%|██▉ | 11/38 [00:05<00:13, 1.93it/s][A
32%|███▏ | 12/38 [00:05<00:13, 1.91it/s][A
34%|███▍ | 13/38 [00:06<00:13, 1.90it/s][A
37%|███▋ | 14/38 [00:06<00:12, 1.91it/s][A
39%|███▉ | 15/38 [00:07<00:11, 1.94it/s][A
42%|████▏ | 16/38 [00:07<00:10, 2.01it/s][A
45%|████▍ | 17/38 [00:08<00:10, 2.09it/s][A
47%|████▋ | 18/38 [00:08<00:08, 2.27it/s][A
50%|█████ | 19/38 [00:09<00:09, 2.00it/s][A
53%|█████▎ | 20/38 [00:09<00:09, 1.82it/s][A
55%|█████▌ | 21/38 [00:10<00:09, 1.82it/s][A
58%|█████▊ | 22/38 [00:10<00:08, 1.79it/s][A
61%|██████ | 23/38 [00:11<00:08, 1.83it/s][A
63%|██████▎ | 24/38 [00:11<00:07, 1.93it/s][A
66%|██████▌ | 25/38 [00:12<00:06, 2.02it/s][A
68%|██████▊ | 26/38 [00:12<00:05, 2.15it/s][A
71%|███████ | 27/38 [00:13<00:04, 2.36it/s][A
74%|███████▎ | 28/38 [00:13<00:04, 2.08it/s][A
76%|███████▋ | 29/38 [00:14<00:04, 1.86it/s][A
79%|███████▉ | 30/38 [00:14<00:04, 1.84it/s][A
82%|████████▏ | 31/38 [00:15<00:03, 1.83it/s][A
84%|████████▍ | 32/38 [00:15<00:03, 1.89it/s][A
87%|████████▋ | 33/38 [00:16<00:02, 1.95it/s][A
89%|████████▉ | 34/38 [00:16<00:01, 2.03it/s][A
92%|█████████▏| 35/38 [00:17<00:01, 2.15it/s][A
95%|█████████▍| 36/38 [00:17<00:00, 2.30it/s][A
97%|█████████▋| 37/38 [00:18<00:00, 2.08it/s][A
100%|██████████| 38/38 [00:18<00:00, 2.00it/s][A
[A{'eval_loss': 0.034754782915115356, 'eval_runtime': 20.413, 'eval_samples_per_second': 14.697, 'eval_steps_per_second': 1.862, 'epoch': 0.34}
34%|███▍ | 300/870 [37:14<45:18, 4.77s/it]
100%|██████████| 38/38 [00:19<00:00, 2.00it/s][A
[A
35%|███▍ | 301/870 [37:26<2:05:18, 13.21s/it]
35%|███▍ | 302/870 [37:37<1:57:23, 12.40s/it]
35%|███▍ | 303/870 [37:47<1:49:20, 11.57s/it]
35%|███▍ | 304/870 [37:56<1:42:43, 10.89s/it]
35%|███▌ | 305/870 [38:05<1:36:21, 10.23s/it]
35%|███▌ | 306/870 [38:13<1:31:37, 9.75s/it]
35%|███▌ | 307/870 [38:21<1:27:15, 9.30s/it]
35%|███▌ | 308/870 [38:29<1:22:56, 8.86s/it]
36%|███▌ | 309/870 [38:37<1:20:07, 8.57s/it]
36%|███▌ | 310/870 [38:45<1:17:43, 8.33s/it]
{'loss': 0.029, 'grad_norm': 0.08098062872886658, 'learning_rate': 0.00014967691344313996, 'epoch': 0.36}
36%|███▌ | 310/870 [38:45<1:17:43, 8.33s/it]
36%|███▌ | 311/870 [38:52<1:15:27, 8.10s/it]
36%|███▌ | 312/870 [39:00<1:13:11, 7.87s/it]
36%|███▌ | 313/870 [39:07<1:10:12, 7.56s/it]
36%|███▌ | 314/870 [39:14<1:08:31, 7.39s/it]
36%|███▌ | 315/870 [39:21<1:07:50, 7.33s/it]
36%|███▋ | 316/870 [39:28<1:06:09, 7.16s/it]
36%|███▋ | 317/870 [39:34<1:03:44, 6.92s/it]
37%|███▋ | 318/870 [39:40<1:01:39, 6.70s/it]
37%|███▋ | 319/870 [39:46<59:24, 6.47s/it]
37%|███▋ | 320/870 [39:52<58:45, 6.41s/it]
{'loss': 0.021, 'grad_norm': 0.07463961839675903, 'learning_rate': 0.00014640884733793216, 'epoch': 0.37}
37%|███▋ | 320/870 [39:52<58:45, 6.41s/it]
37%|███▋ | 321/870 [39:58<56:38, 6.19s/it]
37%|███▋ | 322/870 [40:03<54:26, 5.96s/it]
37%|███▋ | 323/870 [40:09<51:47, 5.68s/it]
37%|███▋ | 324/870 [40:13<49:31, 5.44s/it]
37%|███▋ | 325/870 [40:17<43:28, 4.79s/it]
37%|███▋ | 326/870 [40:29<1:02:56, 6.94s/it]
38%|███▊ | 327/870 [40:39<1:11:20, 7.88s/it]
38%|███▊ | 328/870 [40:48<1:15:38, 8.37s/it]
38%|███▊ | 329/870 [40:58<1:18:06, 8.66s/it]
38%|███▊ | 330/870 [41:06<1:17:32, 8.61s/it]
{'loss': 0.0247, 'grad_norm': 0.055279821157455444, 'learning_rate': 0.00014307633535921317, 'epoch': 0.38}
38%|███▊ | 330/870 [41:06<1:17:32, 8.61s/it]
38%|███▊ | 331/870 [41:14<1:16:28, 8.51s/it]
38%|███▊ | 332/870 [41:22<1:14:46, 8.34s/it]
38%|███▊ | 333/870 [41:30<1:13:23, 8.20s/it]
38%|███▊ | 334/870 [41:38<1:11:54, 8.05s/it]
39%|███▊ | 335/870 [41:45<1:09:31, 7.80s/it]
39%|███▊ | 336/870 [41:52<1:07:59, 7.64s/it]
39%|███▊ | 337/870 [41:59<1:06:32, 7.49s/it]
39%|███▉ | 338/870 [42:07<1:05:54, 7.43s/it]
39%|███▉ | 339/870 [42:13<1:03:53, 7.22s/it]
39%|███▉ | 340/870 [42:20<1:02:01, 7.02s/it]
{'loss': 0.0203, 'grad_norm': 0.18279875814914703, 'learning_rate': 0.00013968400521608969, 'epoch': 0.39}
39%|███▉ | 340/870 [42:20<1:02:01, 7.02s/it]
39%|███▉ | 341/870 [42:26<1:00:02, 6.81s/it]
39%|███▉ | 342/870 [42:33<58:24, 6.64s/it]
39%|███▉ | 343/870 [42:39<57:12, 6.51s/it]
40%|███▉ | 344/870 [42:45<55:41, 6.35s/it]
40%|███▉ | 345/870 [42:50<53:48, 6.15s/it]
40%|███▉ | 346/870 [42:56<52:08, 5.97s/it]
40%|███▉ | 347/870 [43:01<50:28, 5.79s/it]
40%|████ | 348/870 [43:07<48:51, 5.62s/it]
40%|████ | 349/870 [43:11<45:32, 5.25s/it]
40%|████ | 350/870 [43:14<40:32, 4.68s/it]
{'loss': 0.0151, 'grad_norm': 0.07882044464349747, 'learning_rate': 0.0001362365676844557, 'epoch': 0.4}
40%|████ | 350/870 [43:14<40:32, 4.68s/it]
40%|████ | 351/870 [43:26<59:04, 6.83s/it]
40%|████ | 352/870 [43:36<1:07:36, 7.83s/it]
41%|████ | 353/870 [43:46<1:12:04, 8.36s/it]
41%|████ | 354/870 [43:55<1:14:07, 8.62s/it]
41%|████ | 355/870 [44:04<1:13:45, 8.59s/it]
41%|████ | 356/870 [44:12<1:12:59, 8.52s/it]
41%|████ | 357/870 [44:20<1:12:12, 8.45s/it]
41%|████ | 358/870 [44:28<1:10:45, 8.29s/it]
41%|████▏ | 359/870 [44:36<1:09:07, 8.12s/it]
41%|████▏ | 360/870 [44:44<1:07:42, 7.97s/it]
{'loss': 0.0261, 'grad_norm': 0.07913503050804138, 'learning_rate': 0.00013273881006535074, 'epoch': 0.41}
41%|████▏ | 360/870 [44:44<1:07:42, 7.97s/it]
41%|████▏ | 361/870 [44:51<1:06:03, 7.79s/it]
42%|████▏ | 362/870 [44:59<1:05:27, 7.73s/it]
42%|████▏ | 363/870 [45:06<1:03:51, 7.56s/it]
42%|████▏ | 364/870 [45:13<1:02:27, 7.41s/it]
42%|████▏ | 365/870 [45:20<1:01:00, 7.25s/it]
42%|████▏ | 366/870 [45:26<59:41, 7.11s/it]
42%|████▏ | 367/870 [45:33<58:21, 6.96s/it]
42%|████▏ | 368/870 [45:39<56:28, 6.75s/it]
42%|████▏ | 369/870 [45:45<54:47, 6.56s/it]
43%|████▎ | 370/870 [45:51<53:11, 6.38s/it]
{'loss': 0.0176, 'grad_norm': 0.06874211877584457, 'learning_rate': 0.00012919558953705055, 'epoch': 0.43}
43%|████▎ | 370/870 [45:51<53:11, 6.38s/it]
43%|████▎ | 371/870 [45:57<52:04, 6.26s/it]
43%|████▎ | 372/870 [46:03<50:13, 6.05s/it]
43%|████▎ | 373/870 [46:08<48:06, 5.81s/it]
43%|████▎ | 374/870 [46:13<45:23, 5.49s/it]
43%|████▎ | 375/870 [46:16<40:06, 4.86s/it]
43%|████▎ | 376/870 [46:28<56:23, 6.85s/it]
43%|████▎ | 377/870 [46:38<1:04:00, 7.79s/it]
43%|████▎ | 378/870 [46:47<1:07:46, 8.26s/it]
44%|████▎ | 379/870 [46:56<1:09:55, 8.54s/it]
44%|████▎ | 380/870 [47:05<1:09:41, 8.53s/it]
{'loss': 0.0187, 'grad_norm': 0.06528666615486145, 'learning_rate': 0.0001256118264101228, 'epoch': 0.44}
44%|████▎ | 380/870 [47:05<1:09:41, 8.53s/it]
44%|████▍ | 381/870 [47:13<1:09:07, 8.48s/it]
44%|████▍ | 382/870 [47:21<1:07:39, 8.32s/it]
44%|████▍ | 383/870 [47:29<1:05:39, 8.09s/it]
44%|████▍ | 384/870 [47:36<1:04:41, 7.99s/it]
44%|████▍ | 385/870 [47:44<1:03:34, 7.86s/it]
44%|████▍ | 386/870 [47:51<1:02:18, 7.72s/it]
44%|████▍ | 387/870 [47:58<1:00:28, 7.51s/it]
45%|████▍ | 388/870 [48:05<58:38, 7.30s/it]
45%|████▍ | 389/870 [48:12<57:11, 7.13s/it]
45%|████▍ | 390/870 [48:19<56:59, 7.12s/it]
{'loss': 0.0158, 'grad_norm': 0.08248886466026306, 'learning_rate': 0.00012199249729481357, 'epoch': 0.45}
45%|████▍ | 390/870 [48:19<56:59, 7.12s/it]
45%|████▍ | 391/870 [48:26<55:25, 6.94s/it]
45%|████▌ | 392/870 [48:32<53:45, 6.75s/it]
45%|████▌ | 393/870 [48:38<52:17, 6.58s/it]
45%|████▌ | 394/870 [48:44<50:33, 6.37s/it]
45%|████▌ | 395/870 [48:50<50:11, 6.34s/it]
46%|████▌ | 396/870 [48:56<48:29, 6.14s/it]
46%|████▌ | 397/870 [49:01<46:42, 5.92s/it]
46%|████▌ | 398/870 [49:06<44:36, 5.67s/it]
46%|████▌ | 399/870 [49:11<41:11, 5.25s/it]
46%|████▌ | 400/870 [49:14<36:47, 4.70s/it]
{'loss': 0.0132, 'grad_norm': 0.24866768717765808, 'learning_rate': 0.00011834262819025326, 'epoch': 0.46}
46%|████▌ | 400/870 [49:14<36:47, 4.70s/it]
0%| | 0/38 [00:00<?, ?it/s][A
5%|▌ | 2/38 [00:00<00:12, 2.79it/s][A
8%|▊ | 3/38 [00:01<00:15, 2.28it/s][A
11%|█ | 4/38 [00:01<00:16, 2.03it/s][A
13%|█▎ | 5/38 [00:02<00:16, 2.00it/s][A
16%|█▌ | 6/38 [00:02<00:15, 2.05it/s][A
18%|█▊ | 7/38 [00:03<00:14, 2.10it/s][A
21%|██ | 8/38 [00:03<00:13, 2.22it/s][A
24%|██▎ | 9/38 [00:04<00:12, 2.40it/s][A
26%|██▋ | 10/38 [00:04<00:13, 2.09it/s][A
29%|██▉ | 11/38 [00:05<00:14, 1.86it/s][A
32%|███▏ | 12/38 [00:05<00:14, 1.82it/s][A
34%|███▍ | 13/38 [00:06<00:13, 1.82it/s][A
37%|███▋ | 14/38 [00:06<00:12, 1.87it/s][A
39%|███▉ | 15/38 [00:07<00:11, 1.94it/s][A
42%|████▏ | 16/38 [00:07<00:10, 2.02it/s][A
45%|████▍ | 17/38 [00:08<00:09, 2.15it/s][A
47%|████▋ | 18/38 [00:08<00:08, 2.25it/s][A
50%|█████ | 19/38 [00:09<00:09, 2.00it/s][A
53%|█████▎ | 20/38 [00:09<00:09, 1.83it/s][A
55%|█████▌ | 21/38 [00:10<00:09, 1.82it/s][A
58%|█████▊ | 22/38 [00:11<00:08, 1.80it/s][A
61%|██████ | 23/38 [00:11<00:08, 1.86it/s][A
63%|██████▎ | 24/38 [00:12<00:07, 1.90it/s][A
66%|██████▌ | 25/38 [00:12<00:06, 2.02it/s][A
68%|██████▊ | 26/38 [00:12<00:05, 2.14it/s][A
71%|███████ | 27/38 [00:13<00:04, 2.29it/s][A
74%|███████▎ | 28/38 [00:13<00:04, 2.02it/s][A
76%|███████▋ | 29/38 [00:14<00:04, 1.84it/s][A
79%|███████▉ | 30/38 [00:15<00:04, 1.83it/s][A
82%|████████▏ | 31/38 [00:15<00:03, 1.87it/s][A
84%|████████▍ | 32/38 [00:16<00:03, 1.94it/s][A
87%|████████▋ | 33/38 [00:16<00:02, 1.99it/s][A
89%|████████▉ | 34/38 [00:16<00:01, 2.09it/s][A
92%|█████████▏| 35/38 [00:17<00:01, 2.24it/s][A
95%|█████████▍| 36/38 [00:17<00:00, 2.52it/s][A
97%|█████████▋| 37/38 [00:18<00:00, 2.32it/s][A
100%|██████████| 38/38 [00:18<00:00, 2.19it/s][A
[A{'eval_loss': 0.018751109018921852, 'eval_runtime': 20.2674, 'eval_samples_per_second': 14.802, 'eval_steps_per_second': 1.875, 'epoch': 0.46}
46%|████▌ | 400/870 [49:34<36:47, 4.70s/it]
100%|██████████| 38/38 [00:19<00:00, 2.19it/s][A
[A
46%|████▌ | 401/870 [49:48<1:45:01, 13.44s/it]
46%|████▌ | 402/870 [49:58<1:37:21, 12.48s/it]
46%|████▋ | 403/870 [50:08<1:30:29, 11.63s/it]
46%|████▋ | 404/870 [50:17<1:25:28, 11.00s/it]
47%|████▋ | 405/870 [50:26<1:20:38, 10.40s/it]
47%|████▋ | 406/870 [50:35<1:15:28, 9.76s/it]
47%|████▋ | 407/870 [50:43<1:11:51, 9.31s/it]
47%|████▋ | 408/870 [50:51<1:08:52, 8.95s/it]
47%|████▋ | 409/870 [50:59<1:06:56, 8.71s/it]
47%|████▋ | 410/870 [51:07<1:04:35, 8.43s/it]
{'loss': 0.0212, 'grad_norm': 0.11699023842811584, 'learning_rate': 0.00011466728750507805, 'epoch': 0.47}
47%|████▋ | 410/870 [51:07<1:04:35, 8.43s/it]
47%|████▋ | 411/870 [51:14<1:02:03, 8.11s/it]
47%|████▋ | 412/870 [51:21<59:50, 7.84s/it]
47%|████▋ | 413/870 [51:29<58:02, 7.62s/it]
48%|████▊ | 414/870 [51:36<56:29, 7.43s/it]
48%|████▊ | 415/870 [51:42<54:59, 7.25s/it]
48%|████▊ | 416/870 [51:49<53:30, 7.07s/it]
48%|████▊ | 417/870 [51:55<51:06, 6.77s/it]
48%|████▊ | 418/870 [52:01<49:42, 6.60s/it]
48%|████▊ | 419/870 [52:08<49:15, 6.55s/it]
48%|████▊ | 420/870 [52:14<47:50, 6.38s/it]
{'loss': 0.0176, 'grad_norm': 0.08753076195716858, 'learning_rate': 0.00011097157901915953, 'epoch': 0.48}
48%|████▊ | 420/870 [52:14<47:50, 6.38s/it]
48%|████▊ | 421/870 [52:20<46:20, 6.19s/it]
49%|████▊ | 422/870 [52:25<44:49, 6.00s/it]
49%|████▊ | 423/870 [52:31<43:52, 5.89s/it]
49%|████▊ | 424/870 [52:35<41:00, 5.52s/it]
49%|████▉ | 425/870 [52:39<35:49, 4.83s/it]
49%|████▉ | 426/870 [52:51<52:27, 7.09s/it]
49%|████▉ | 427/870 [53:01<59:44, 8.09s/it]
49%|████▉ | 428/870 [53:11<1:03:26, 8.61s/it]
49%|████▉ | 429/870 [53:21<1:04:57, 8.84s/it]
49%|████▉ | 430/870 [53:30<1:05:14, 8.90s/it]
{'loss': 0.0226, 'grad_norm': 0.10365070402622223, 'learning_rate': 0.00010726063479621573, 'epoch': 0.49}
49%|████▉ | 430/870 [53:30<1:05:14, 8.90s/it]
50%|████▉ | 431/870 [53:38<1:04:13, 8.78s/it]
50%|████▉ | 432/870 [53:46<1:03:11, 8.66s/it]
50%|████▉ | 433/870 [53:54<1:01:32, 8.45s/it]
50%|████▉ | 434/870 [54:02<59:50, 8.23s/it]
50%|█████ | 435/870 [54:10<57:53, 7.99s/it]
50%|█████ | 436/870 [54:17<56:53, 7.87s/it]
50%|█████ | 437/870 [54:25<56:21, 7.81s/it]
50%|█████ | 438/870 [54:32<54:49, 7.61s/it]
50%|█████ | 439/870 [54:39<53:13, 7.41s/it]
51%|█████ | 440/870 [54:46<51:57, 7.25s/it]
{'loss': 0.0189, 'grad_norm': 0.18819937109947205, 'learning_rate': 0.00010353960805714566, 'epoch': 0.51}
51%|█████ | 440/870 [54:46<51:57, 7.25s/it]
51%|█████ | 441/870 [54:53<50:52, 7.12s/it]
51%|█████ | 442/870 [54:59<49:40, 6.96s/it]
51%|█████ | 443/870 [55:05<47:53, 6.73s/it]
51%|█████ | 444/870 [55:11<45:58, 6.48s/it]
51%|█████ | 445/870 [55:17<44:17, 6.25s/it]
51%|█████▏ | 446/870 [55:23<43:16, 6.12s/it]
51%|█████▏ | 447/870 [55:28<41:25, 5.88s/it]
51%|█████▏ | 448/870 [55:33<39:07, 5.56s/it]
52%|█████▏ | 449/870 [55:37<36:22, 5.18s/it]
52%|█████▏ | 450/870 [55:40<31:44, 4.53s/it]
{'loss': 0.0129, 'grad_norm': 0.10421279072761536, 'learning_rate': 9.98136660239837e-05, 'epoch': 0.52}
52%|█████▏ | 450/870 [55:40<31:44, 4.53s/it]
52%|█████▏ | 451/870 [55:53<47:53, 6.86s/it]
52%|█████▏ | 452/870 [56:03<54:37, 7.84s/it]
52%|█████▏ | 453/870 [56:12<57:45, 8.31s/it]
52%|█████▏ | 454/870 [56:21<59:09, 8.53s/it]
52%|█████▏ | 455/870 [56:30<58:58, 8.53s/it]
52%|█████▏ | 456/870 [56:38<58:56, 8.54s/it]
53%|█████▎ | 457/870 [56:46<58:00, 8.43s/it]
53%|█████▎ | 458/870 [56:54<56:47, 8.27s/it]
53%|█████▎ | 459/870 [57:02<55:12, 8.06s/it]
53%|█████▎ | 460/870 [57:09<54:01, 7.91s/it]
{'loss': 0.0224, 'grad_norm': 0.07066013664007187, 'learning_rate': 9.60879827444116e-05, 'epoch': 0.53}
53%|█████▎ | 460/870 [57:09<54:01, 7.91s/it]
53%|█████▎ | 461/870 [57:17<52:45, 7.74s/it]
53%|█████▎ | 462/870 [57:24<51:24, 7.56s/it]
53%|█████▎ | 463/870 [57:31<50:13, 7.40s/it]
53%|█████▎ | 464/870 [57:38<48:41, 7.20s/it]
53%|█████▎ | 465/870 [57:45<48:17, 7.15s/it]
54%|█████▎ | 466/870 [57:51<46:27, 6.90s/it]
54%|█████▎ | 467/870 [57:57<44:48, 6.67s/it]
54%|█████▍ | 468/870 [58:03<43:18, 6.46s/it]
54%|█████▍ | 469/870 [58:09<42:14, 6.32s/it]
54%|█████▍ | 470/870 [58:15<41:55, 6.29s/it]
{'loss': 0.0158, 'grad_norm': 0.04448763281106949, 'learning_rate': 9.236773190679163e-05, 'epoch': 0.54}
54%|█████▍ | 470/870 [58:15<41:55, 6.29s/it]
54%|█████▍ | 471/870 [58:21<40:20, 6.07s/it]
54%|█████▍ | 472/870 [58:26<38:49, 5.85s/it]
54%|█████▍ | 473/870 [58:32<37:52, 5.72s/it]
54%|█████▍ | 474/870 [58:36<35:57, 5.45s/it]
55%|█████▍ | 475/870 [58:40<31:08, 4.73s/it]
55%|█████▍ | 476/870 [58:52<46:56, 7.15s/it]
55%|█████▍ | 477/870 [59:03<53:25, 8.16s/it]
55%|█████▍ | 478/870 [59:13<56:14, 8.61s/it]
55%|█████▌ | 479/870 [59:22<57:48, 8.87s/it]
55%|█████▌ | 480/870 [59:31<57:23, 8.83s/it]
{'loss': 0.0216, 'grad_norm': 0.06225728988647461, 'learning_rate': 8.865807965569879e-05, 'epoch': 0.55}
55%|█████▌ | 480/870 [59:31<57:23, 8.83s/it]
55%|█████▌ | 481/870 [59:39<56:35, 8.73s/it]
55%|█████▌ | 482/870 [59:47<55:34, 8.59s/it]
56%|█████▌ | 483/870 [59:56<54:26, 8.44s/it]
56%|█████▌ | 484/870 [1:00:04<53:45, 8.36s/it]
56%|█████▌ | 485/870 [1:00:11<52:25, 8.17s/it]
56%|█████▌ | 486/870 [1:00:19<51:10, 8.00s/it]
56%|█████▌ | 487/870 [1:00:26<49:58, 7.83s/it]
56%|█████▌ | 488/870 [1:00:34<48:33, 7.63s/it]
56%|█████▌ | 489/870 [1:00:41<48:09, 7.58s/it]
56%|█████▋ | 490/870 [1:00:48<46:42, 7.38s/it]
{'loss': 0.022, 'grad_norm': 0.12085612863302231, 'learning_rate': 8.496417741792913e-05, 'epoch': 0.56}
56%|█████▋ | 490/870 [1:00:48<46:42, 7.38s/it]
56%|█████▋ | 491/870 [1:00:55<45:13, 7.16s/it]
57%|█████▋ | 492/870 [1:01:01<43:08, 6.85s/it]
57%|█████▋ | 493/870 [1:01:07<42:28, 6.76s/it]
57%|█████▋ | 494/870 [1:01:13<41:02, 6.55s/it]
57%|█████▋ | 495/870 [1:01:19<39:42, 6.35s/it]
57%|█████▋ | 496/870 [1:01:25<38:37, 6.20s/it]
57%|█████▋ | 497/870 [1:01:31<37:28, 6.03s/it]
57%|█████▋ | 498/870 [1:01:36<36:36, 5.90s/it]
57%|█████▋ | 499/870 [1:01:41<34:25, 5.57s/it]
57%|█████▋ | 500/870 [1:01:45<30:20, 4.92s/it]
{'loss': 0.0152, 'grad_norm': 0.04715719074010849, 'learning_rate': 8.129115474894523e-05, 'epoch': 0.57}
57%|█████▋ | 500/870 [1:01:45<30:20, 4.92s/it]
0%| | 0/38 [00:00<?, ?it/s][A
5%|▌ | 2/38 [00:00<00:13, 2.68it/s][A
8%|▊ | 3/38 [00:01<00:16, 2.09it/s][A
11%|█ | 4/38 [00:01<00:17, 1.98it/s][A
13%|█▎ | 5/38 [00:02<00:16, 1.98it/s][A
16%|█▌ | 6/38 [00:02<00:15, 2.03it/s][A
18%|█▊ | 7/38 [00:03<00:14, 2.09it/s][A
21%|██ | 8/38 [00:03<00:13, 2.18it/s][A
24%|██▎ | 9/38 [00:04<00:12, 2.33it/s][A
26%|██▋ | 10/38 [00:04<00:13, 2.04it/s][A
29%|██▉ | 11/38 [00:05<00:14, 1.84it/s][A
32%|███▏ | 12/38 [00:05<00:14, 1.83it/s][A
34%|███▍ | 13/38 [00:06<00:13, 1.84it/s][A
37%|███▋ | 14/38 [00:07<00:12, 1.87it/s][A
39%|███▉ | 15/38 [00:07<00:11, 1.95it/s][A
42%|████▏ | 16/38 [00:07<00:10, 2.04it/s][A
45%|████▍ | 17/38 [00:08<00:09, 2.16it/s][A
47%|████▋ | 18/38 [00:08<00:08, 2.31it/s][A
50%|█████ | 19/38 [00:09<00:09, 2.03it/s][A
53%|█████▎ | 20/38 [00:09<00:09, 1.85it/s][A
55%|█████▌ | 21/38 [00:10<00:09, 1.85it/s][A
58%|█████▊ | 22/38 [00:11<00:08, 1.81it/s][A
61%|██████ | 23/38 [00:11<00:08, 1.86it/s][A
63%|██████▎ | 24/38 [00:12<00:07, 1.95it/s][A
66%|██████▌ | 25/38 [00:12<00:06, 2.01it/s][A
68%|██████▊ | 26/38 [00:12<00:05, 2.14it/s][A
71%|███████ | 27/38 [00:13<00:04, 2.36it/s][A
74%|███████▎ | 28/38 [00:13<00:04, 2.07it/s][A
76%|███████▋ | 29/38 [00:14<00:04, 1.87it/s][A
79%|███████▉ | 30/38 [00:15<00:04, 1.87it/s][A
82%|████████▏ | 31/38 [00:15<00:03, 1.90it/s][A
84%|████████▍ | 32/38 [00:16<00:03, 1.93it/s][A
87%|████████▋ | 33/38 [00:16<00:02, 1.96it/s][A
89%|████████▉ | 34/38 [00:16<00:01, 2.03it/s][A
92%|█████████▏| 35/38 [00:17<00:01, 2.15it/s][A
95%|█████████▍| 36/38 [00:17<00:00, 2.37it/s][A
97%|█████████▋| 37/38 [00:18<00:00, 2.18it/s][A
100%|██████████| 38/38 [00:18<00:00, 2.07it/s][A
[A{'eval_loss': 0.01953456737101078, 'eval_runtime': 20.3994, 'eval_samples_per_second': 14.706, 'eval_steps_per_second': 1.863, 'epoch': 0.57}
57%|█████▋ | 500/870 [1:02:05<30:20, 4.92s/it]
100%|██████████| 38/38 [00:19<00:00, 2.07it/s][A
[A
58%|█████▊ | 501/870 [1:02:17<1:20:57, 13.16s/it]
58%|█████▊ | 502/870 [1:02:27<1:15:22, 12.29s/it]
58%|█████▊ | 503/870 [1:02:37<1:10:29, 11.53s/it]
58%|█████▊ | 504/870 [1:02:46<1:06:10, 10.85s/it]
58%|█████▊ | 505/870 [1:02:55<1:01:49, 10.16s/it]
58%|█████▊ | 506/870 [1:03:03<58:26, 9.63s/it]
58%|█████▊ | 507/870 [1:03:11<55:25, 9.16s/it]
58%|█████▊ | 508/870 [1:03:20<53:56, 8.94s/it]
59%|█████▊ | 509/870 [1:03:28<51:54, 8.63s/it]
59%|█████▊ | 510/870 [1:03:35<49:32, 8.26s/it]
{'loss': 0.0214, 'grad_norm': 0.08593538403511047, 'learning_rate': 7.764411220969414e-05, 'epoch': 0.59}
59%|█████▊ | 510/870 [1:03:35<49:32, 8.26s/it]
59%|█████▊ | 511/870 [1:03:43<48:11, 8.05s/it]
59%|█████▉ | 512/870 [1:03:50<46:57, 7.87s/it]
59%|█████▉ | 513/870 [1:03:57<45:31, 7.65s/it]
59%|█████▉ | 514/870 [1:04:04<44:12, 7.45s/it]
59%|█████▉ | 515/870 [1:04:11<42:45, 7.23s/it]
59%|█████▉ | 516/870 [1:04:17<41:36, 7.05s/it]
59%|█████▉ | 517/870 [1:04:24<41:08, 6.99s/it]
60%|█████▉ | 518/870 [1:04:30<39:29, 6.73s/it]
60%|█████▉ | 519/870 [1:04:36<37:58, 6.49s/it]
60%|█████▉ | 520/870 [1:04:42<36:58, 6.34s/it]
{'loss': 0.0174, 'grad_norm': 0.08397146314382553, 'learning_rate': 7.402811428368832e-05, 'epoch': 0.6}
60%|█████▉ | 520/870 [1:04:42<36:58, 6.34s/it]
60%|█████▉ | 521/870 [1:04:48<36:14, 6.23s/it]
60%|██████ | 522/870 [1:04:54<34:53, 6.02s/it]
60%|██████ | 523/870 [1:04:59<33:14, 5.75s/it]
60%|██████ | 524/870 [1:05:04<31:26, 5.45s/it]
60%|██████ | 525/870 [1:05:07<27:32, 4.79s/it]
60%|██████ | 526/870 [1:05:19<39:48, 6.94s/it]
61%|██████ | 527/870 [1:05:29<45:16, 7.92s/it]
61%|██████ | 528/870 [1:05:39<47:53, 8.40s/it]
61%|██████ | 529/870 [1:05:48<49:15, 8.67s/it]
61%|██████ | 530/870 [1:05:57<49:26, 8.73s/it]
{'loss': 0.0163, 'grad_norm': 0.07681377232074738, 'learning_rate': 7.044818234418576e-05, 'epoch': 0.61}
61%|██████ | 530/870 [1:05:57<49:26, 8.73s/it]
61%|██████ | 531/870 [1:06:06<49:16, 8.72s/it]
61%|██████ | 532/870 [1:06:14<48:22, 8.59s/it]
61%|██████▏ | 533/870 [1:06:22<47:14, 8.41s/it]
61%|██████▏ | 534/870 [1:06:30<46:11, 8.25s/it]
61%|██████▏ | 535/870 [1:06:37<45:00, 8.06s/it]
62%|██████▏ | 536/870 [1:06:45<44:25, 7.98s/it]
62%|██████▏ | 537/870 [1:06:52<42:59, 7.75s/it]
62%|██████▏ | 538/870 [1:06:59<41:32, 7.51s/it]
62%|██████▏ | 539/870 [1:07:06<40:22, 7.32s/it]
62%|██████▏ | 540/870 [1:07:13<39:39, 7.21s/it]
{'loss': 0.0135, 'grad_norm': 0.061246950179338455, 'learning_rate': 6.69092876812353e-05, 'epoch': 0.62}
62%|██████▏ | 540/870 [1:07:13<39:39, 7.21s/it]
62%|██████▏ | 541/870 [1:07:19<38:06, 6.95s/it]
62%|██████▏ | 542/870 [1:07:26<36:41, 6.71s/it]
62%|██████▏ | 543/870 [1:07:32<35:28, 6.51s/it]
63%|██████▎ | 544/870 [1:07:38<34:22, 6.33s/it]
63%|██████▎ | 545/870 [1:07:44<34:03, 6.29s/it]
63%|██████▎ | 546/870 [1:07:49<32:58, 6.11s/it]
63%|██████▎ | 547/870 [1:07:55<31:39, 5.88s/it]
63%|██████▎ | 548/870 [1:08:00<30:10, 5.62s/it]
63%|██████▎ | 549/870 [1:08:04<28:21, 5.30s/it]
63%|██████▎ | 550/870 [1:08:08<25:17, 4.74s/it]
{'loss': 0.0111, 'grad_norm': 0.1086721196770668, 'learning_rate': 6.341634459827052e-05, 'epoch': 0.63}
63%|██████▎ | 550/870 [1:08:08<25:17, 4.74s/it]
63%|██████▎ | 551/870 [1:08:20<36:24, 6.85s/it]
63%|██████▎ | 552/870 [1:08:30<41:18, 7.79s/it]
64%|██████▎ | 553/870 [1:08:39<43:59, 8.33s/it]
64%|██████▎ | 554/870 [1:08:48<45:18, 8.60s/it]
64%|██████▍ | 555/870 [1:08:57<45:41, 8.70s/it]
64%|██████▍ | 556/870 [1:09:06<45:01, 8.60s/it]
64%|██████▍ | 557/870 [1:09:14<44:17, 8.49s/it]
64%|██████▍ | 558/870 [1:09:22<43:17, 8.33s/it]
64%|██████▍ | 559/870 [1:09:30<42:35, 8.22s/it]
64%|██████▍ | 560/870 [1:09:37<41:04, 7.95s/it]
{'loss': 0.0185, 'grad_norm': 0.10877221822738647, 'learning_rate': 5.9974203587837986e-05, 'epoch': 0.64}
64%|██████▍ | 560/870 [1:09:37<41:04, 7.95s/it]
64%|██████▍ | 561/870 [1:09:44<40:02, 7.78s/it]
65%|██████▍ | 562/870 [1:09:52<38:46, 7.55s/it]
65%|██████▍ | 563/870 [1:09:59<37:56, 7.42s/it]
65%|██████▍ | 564/870 [1:10:06<37:29, 7.35s/it]
65%|██████▍ | 565/870 [1:10:12<36:07, 7.11s/it]
65%|██████▌ | 566/870 [1:10:19<35:20, 6.98s/it]
65%|██████▌ | 567/870 [1:10:25<34:13, 6.78s/it]
65%|██████▌ | 568/870 [1:10:32<33:19, 6.62s/it]
65%|██████▌ | 569/870 [1:10:38<32:50, 6.55s/it]
66%|██████▌ | 570/870 [1:10:44<31:47, 6.36s/it]
{'loss': 0.0134, 'grad_norm': 0.25607386231422424, 'learning_rate': 5.658764459593691e-05, 'epoch': 0.66}
66%|██████▌ | 570/870 [1:10:44<31:47, 6.36s/it]
66%|██████▌ | 571/870 [1:10:50<30:39, 6.15s/it]
66%|██████▌ | 572/870 [1:10:55<29:31, 5.94s/it]
66%|██████▌ | 573/870 [1:11:01<28:46, 5.81s/it]
66%|██████▌ | 574/870 [1:11:05<26:32, 5.38s/it]
66%|██████▌ | 575/870 [1:11:08<22:50, 4.65s/it]
66%|██████▌ | 576/870 [1:11:20<33:34, 6.85s/it]
66%|██████▋ | 577/870 [1:11:30<38:41, 7.92s/it]
66%|██████▋ | 578/870 [1:11:40<41:22, 8.50s/it]
67%|██████▋ | 579/870 [1:11:49<42:23, 8.74s/it]
67%|██████▋ | 580/870 [1:11:58<41:58, 8.68s/it]
{'loss': 0.0155, 'grad_norm': 0.06046198308467865, 'learning_rate': 5.3261370384323984e-05, 'epoch': 0.67}
67%|██████▋ | 580/870 [1:11:58<41:58, 8.68s/it]
67%|██████▋ | 581/870 [1:12:06<41:24, 8.60s/it]
67%|██████▋ | 582/870 [1:12:14<40:27, 8.43s/it]
67%|██████▋ | 583/870 [1:12:22<39:25, 8.24s/it]
67%|██████▋ | 584/870 [1:12:30<38:25, 8.06s/it]
67%|██████▋ | 585/870 [1:12:37<37:24, 7.88s/it]
67%|██████▋ | 586/870 [1:12:45<36:47, 7.77s/it]
67%|██████▋ | 587/870 [1:12:52<35:50, 7.60s/it]
68%|██████▊ | 588/870 [1:12:59<35:26, 7.54s/it]
68%|██████▊ | 589/870 [1:13:06<34:33, 7.38s/it]
68%|██████▊ | 590/870 [1:13:13<33:17, 7.13s/it]
{'loss': 0.0156, 'grad_norm': 0.08047638833522797, 'learning_rate': 5.000000000000002e-05, 'epoch': 0.68}
68%|██████▊ | 590/870 [1:13:13<33:17, 7.13s/it]
68%|██████▊ | 591/870 [1:13:19<32:17, 6.94s/it]
68%|██████▊ | 592/870 [1:13:26<31:17, 6.75s/it]
68%|██████▊ | 593/870 [1:13:32<30:55, 6.70s/it]
68%|██████▊ | 594/870 [1:13:38<30:00, 6.52s/it]
68%|██████▊ | 595/870 [1:13:44<29:08, 6.36s/it]
69%|██████▊ | 596/870 [1:13:50<28:06, 6.15s/it]
69%|██████▊ | 597/870 [1:13:56<27:26, 6.03s/it]
69%|██████▊ | 598/870 [1:14:01<26:04, 5.75s/it]
69%|██████▉ | 599/870 [1:14:05<24:20, 5.39s/it]
69%|██████▉ | 600/870 [1:14:09<21:06, 4.69s/it]
{'loss': 0.0118, 'grad_norm': 0.09806278347969055, 'learning_rate': 4.680806236094756e-05, 'epoch': 0.69}
69%|██████▉ | 600/870 [1:14:09<21:06, 4.69s/it]
0%| | 0/38 [00:00<?, ?it/s][A
5%|▌ | 2/38 [00:00<00:12, 2.78it/s][A
8%|▊ | 3/38 [00:01<00:15, 2.29it/s][A
11%|█ | 4/38 [00:01<00:16, 2.09it/s][A
13%|█▎ | 5/38 [00:02<00:16, 2.04it/s][A
16%|█▌ | 6/38 [00:02<00:15, 2.05it/s][A
18%|█▊ | 7/38 [00:03<00:14, 2.17it/s][A
21%|██ | 8/38 [00:03<00:13, 2.28it/s][A
24%|██▎ | 9/38 [00:03<00:11, 2.62it/s][A
26%|██▋ | 10/38 [00:04<00:12, 2.25it/s][A
29%|██▉ | 11/38 [00:05<00:13, 1.95it/s][A
32%|███▏ | 12/38 [00:05<00:13, 1.91it/s][A
34%|███▍ | 13/38 [00:06<00:13, 1.86it/s][A
37%|███▋ | 14/38 [00:06<00:12, 1.90it/s][A
39%|███▉ | 15/38 [00:07<00:11, 1.93it/s][A
42%|████▏ | 16/38 [00:07<00:10, 2.01it/s][A
45%|████▍ | 17/38 [00:08<00:09, 2.12it/s][A
47%|████▋ | 18/38 [00:08<00:08, 2.28it/s][A
50%|█████ | 19/38 [00:09<00:09, 2.02it/s][A
53%|█████▎ | 20/38 [00:09<00:09, 1.80it/s][A
55%|█████▌ | 21/38 [00:10<00:09, 1.78it/s][A
58%|█████▊ | 22/38 [00:10<00:08, 1.84it/s][A
61%|██████ | 23/38 [00:11<00:07, 1.89it/s][A
63%|██████▎ | 24/38 [00:11<00:07, 1.98it/s][A
66%|██████▌ | 25/38 [00:12<00:06, 2.05it/s][A
68%|██████▊ | 26/38 [00:12<00:05, 2.16it/s][A
71%|███████ | 27/38 [00:13<00:04, 2.26it/s][A
74%|███████▎ | 28/38 [00:13<00:04, 2.05it/s][A
76%|███████▋ | 29/38 [00:14<00:04, 1.87it/s][A
79%|███████▉ | 30/38 [00:14<00:04, 1.87it/s][A
82%|████████▏ | 31/38 [00:15<00:03, 1.88it/s][A
84%|████████▍ | 32/38 [00:15<00:03, 1.90it/s][A
87%|████████▋ | 33/38 [00:16<00:02, 1.98it/s][A
89%|████████▉ | 34/38 [00:16<00:01, 2.05it/s][A
92%|█████████▏| 35/38 [00:17<00:01, 2.17it/s][A
95%|█████████▍| 36/38 [00:17<00:00, 2.31it/s][A
97%|█████████▋| 37/38 [00:18<00:00, 2.14it/s][A
100%|██████████| 38/38 [00:18<00:00, 2.07it/s][A
[A{'eval_loss': 0.016160501167178154, 'eval_runtime': 20.2233, 'eval_samples_per_second': 14.834, 'eval_steps_per_second': 1.879, 'epoch': 0.69}
69%|██████▉ | 600/870 [1:14:29<21:06, 4.69s/it]
100%|██████████| 38/38 [00:19<00:00, 2.07it/s][A
[A
69%|██████▉ | 601/870 [1:14:42<59:36, 13.30s/it]
69%|██████▉ | 602/870 [1:14:53<55:49, 12.50s/it]
69%|██████▉ | 603/870 [1:15:02<51:40, 11.61s/it]
69%|██████▉ | 604/870 [1:15:11<48:18, 10.90s/it]
70%|██████▉ | 605/870 [1:15:20<44:58, 10.18s/it]
70%|██████▉ | 606/870 [1:15:28<42:06, 9.57s/it]
70%|██████▉ | 607/870 [1:15:37<40:32, 9.25s/it]
70%|██████▉ | 608/870 [1:15:45<38:50, 8.89s/it]
70%|███████ | 609/870 [1:15:52<37:12, 8.55s/it]
70%|███████ | 610/870 [1:16:00<35:35, 8.21s/it]
{'loss': 0.0154, 'grad_norm': 0.12618760764598846, 'learning_rate': 4.3689989967026935e-05, 'epoch': 0.7}
70%|███████ | 610/870 [1:16:00<35:35, 8.21s/it]
70%|███████ | 611/870 [1:16:07<34:39, 8.03s/it]
70%|███████ | 612/870 [1:16:15<33:25, 7.77s/it]
70%|███████ | 613/870 [1:16:21<32:03, 7.48s/it]
71%|███████ | 614/870 [1:16:28<31:09, 7.30s/it]
71%|███████ | 615/870 [1:16:35<30:24, 7.15s/it]
71%|███████ | 616/870 [1:16:42<29:59, 7.09s/it]
71%|███████ | 617/870 [1:16:48<28:44, 6.82s/it]
71%|███████ | 618/870 [1:16:54<27:50, 6.63s/it]
71%|███████ | 619/870 [1:17:00<26:40, 6.37s/it]
71%|███████▏ | 620/870 [1:17:06<25:58, 6.23s/it]
{'loss': 0.0105, 'grad_norm': 0.16422143578529358, 'learning_rate': 4.065011274476349e-05, 'epoch': 0.71}
71%|███████▏ | 620/870 [1:17:06<25:58, 6.23s/it]
71%|███████▏ | 621/870 [1:17:12<25:25, 6.13s/it]
71%|███████▏ | 622/870 [1:17:17<24:32, 5.94s/it]
72%|███████▏ | 623/870 [1:17:22<23:23, 5.68s/it]
72%|███████▏ | 624/870 [1:17:27<21:49, 5.32s/it]
72%|███████▏ | 625/870 [1:17:30<19:02, 4.66s/it]
72%|███████▏ | 626/870 [1:17:42<28:11, 6.93s/it]
72%|███████▏ | 627/870 [1:17:53<32:06, 7.93s/it]
72%|███████▏ | 628/870 [1:18:02<33:55, 8.41s/it]
72%|███████▏ | 629/870 [1:18:11<34:45, 8.65s/it]
72%|███████▏ | 630/870 [1:18:20<34:27, 8.62s/it]
{'loss': 0.0155, 'grad_norm': 0.3045821487903595, 'learning_rate': 3.769265203457375e-05, 'epoch': 0.72}
72%|███████▏ | 630/870 [1:18:20<34:27, 8.62s/it]
73%|███████▎ | 631/870 [1:18:28<34:21, 8.63s/it]
73%|███████▎ | 632/870 [1:18:37<33:45, 8.51s/it]
73%|███████▎ | 633/870 [1:18:45<33:00, 8.36s/it]
73%|███████▎ | 634/870 [1:18:52<32:06, 8.16s/it]
73%|███████▎ | 635/870 [1:19:00<31:38, 8.08s/it]
73%|███████▎ | 636/870 [1:19:08<30:40, 7.86s/it]
73%|███████▎ | 637/870 [1:19:15<29:34, 7.61s/it]
73%|███████▎ | 638/870 [1:19:22<28:31, 7.38s/it]
73%|███████▎ | 639/870 [1:19:28<27:52, 7.24s/it]
74%|███████▎ | 640/870 [1:19:35<27:14, 7.11s/it]
{'loss': 0.0158, 'grad_norm': 0.12457603216171265, 'learning_rate': 3.482171472878062e-05, 'epoch': 0.74}
74%|███████▎ | 640/870 [1:19:35<27:14, 7.11s/it]
74%|███████▎ | 641/870 [1:19:42<26:30, 6.94s/it]
74%|███████▍ | 642/870 [1:19:48<25:31, 6.72s/it]
74%|███████▍ | 643/870 [1:19:54<24:51, 6.57s/it]
74%|███████▍ | 644/870 [1:20:00<24:08, 6.41s/it]
74%|███████▍ | 645/870 [1:20:06<23:36, 6.30s/it]
74%|███████▍ | 646/870 [1:20:12<22:45, 6.10s/it]
74%|███████▍ | 647/870 [1:20:17<21:48, 5.87s/it]
74%|███████▍ | 648/870 [1:20:22<20:50, 5.64s/it]
75%|███████▍ | 649/870 [1:20:27<19:46, 5.37s/it]
75%|███████▍ | 650/870 [1:20:30<17:04, 4.66s/it]
{'loss': 0.0096, 'grad_norm': 0.04488776624202728, 'learning_rate': 3.20412875685569e-05, 'epoch': 0.75}
75%|███████▍ | 650/870 [1:20:30<17:04, 4.66s/it]
75%|███████▍ | 651/870 [1:20:42<25:05, 6.87s/it]
75%|███████▍ | 652/870 [1:20:52<28:44, 7.91s/it]
75%|███████▌ | 653/870 [1:21:02<30:26, 8.42s/it]
75%|███████▌ | 654/870 [1:21:12<31:38, 8.79s/it]
75%|███████▌ | 655/870 [1:21:20<31:12, 8.71s/it]
75%|███████▌ | 656/870 [1:21:29<30:39, 8.60s/it]
76%|███████▌ | 657/870 [1:21:37<30:06, 8.48s/it]
76%|███████▌ | 658/870 [1:21:45<29:23, 8.32s/it]
76%|███████▌ | 659/870 [1:21:53<28:58, 8.24s/it]
76%|███████▌ | 660/870 [1:22:00<28:09, 8.05s/it]
{'loss': 0.0164, 'grad_norm': 0.05114804208278656, 'learning_rate': 2.935523160771786e-05, 'epoch': 0.76}
76%|███████▌ | 660/870 [1:22:00<28:09, 8.05s/it]
76%|███████▌ | 661/870 [1:22:08<27:14, 7.82s/it]
76%|███████▌ | 662/870 [1:22:15<26:26, 7.63s/it]
76%|███████▌ | 663/870 [1:22:22<26:02, 7.55s/it]
76%|███████▋ | 664/870 [1:22:29<25:17, 7.37s/it]
76%|███████▋ | 665/870 [1:22:36<24:37, 7.21s/it]
77%|███████▋ | 666/870 [1:22:42<23:37, 6.95s/it]
77%|███████▋ | 667/870 [1:22:49<22:49, 6.75s/it]
77%|███████▋ | 668/870 [1:22:55<22:23, 6.65s/it]
77%|███████▋ | 669/870 [1:23:01<21:36, 6.45s/it]
77%|███████▋ | 670/870 [1:23:07<20:48, 6.24s/it]
{'loss': 0.0115, 'grad_norm': 0.03534635156393051, 'learning_rate': 2.6767276851049816e-05, 'epoch': 0.77}
77%|███████▋ | 670/870 [1:23:07<20:48, 6.24s/it]
77%|███████▋ | 671/870 [1:23:12<20:05, 6.06s/it]
77%|███████▋ | 672/870 [1:23:18<19:20, 5.86s/it]
77%|███████▋ | 673/870 [1:23:23<18:47, 5.72s/it]
77%|███████▋ | 674/870 [1:23:28<17:23, 5.32s/it]
78%|███████▊ | 675/870 [1:23:31<15:05, 4.64s/it]
78%|███████▊ | 676/870 [1:23:42<21:27, 6.64s/it]
78%|███████▊ | 677/870 [1:23:52<24:34, 7.64s/it]
78%|███████▊ | 678/870 [1:24:02<26:23, 8.25s/it]
78%|███████▊ | 679/870 [1:24:11<27:12, 8.55s/it]
78%|███████▊ | 680/870 [1:24:20<27:13, 8.60s/it]
{'loss': 0.0138, 'grad_norm': 0.06601171940565109, 'learning_rate': 2.4281017074620584e-05, 'epoch': 0.78}
78%|███████▊ | 680/870 [1:24:20<27:13, 8.60s/it]
78%|███████▊ | 681/870 [1:24:28<26:39, 8.46s/it]
78%|███████▊ | 682/870 [1:24:36<26:32, 8.47s/it]
79%|███████▊ | 683/870 [1:24:44<25:56, 8.33s/it]
79%|███████▊ | 684/870 [1:24:52<25:20, 8.18s/it]
79%|███████▊ | 685/870 [1:25:00<24:39, 8.00s/it]
79%|███████▉ | 686/870 [1:25:07<24:00, 7.83s/it]
79%|███████▉ | 687/870 [1:25:14<23:30, 7.71s/it]
79%|███████▉ | 688/870 [1:25:22<22:50, 7.53s/it]
79%|███████▉ | 689/870 [1:25:29<22:10, 7.35s/it]
79%|███████▉ | 690/870 [1:25:35<21:34, 7.19s/it]
{'loss': 0.0142, 'grad_norm': 0.10086585581302643, 'learning_rate': 2.1899904835264896e-05, 'epoch': 0.79}
79%|███████▉ | 690/870 [1:25:35<21:34, 7.19s/it]
79%|███████▉ | 691/870 [1:25:42<20:45, 6.96s/it]
80%|███████▉ | 692/870 [1:25:48<20:12, 6.81s/it]
80%|███████▉ | 693/870 [1:25:54<19:30, 6.61s/it]
80%|███████▉ | 694/870 [1:26:00<18:46, 6.40s/it]
80%|███████▉ | 695/870 [1:26:06<18:13, 6.25s/it]
80%|████████ | 696/870 [1:26:12<17:36, 6.07s/it]
80%|████████ | 697/870 [1:26:18<17:13, 5.97s/it]
80%|████████ | 698/870 [1:26:23<16:15, 5.67s/it]
80%|████████ | 699/870 [1:26:27<14:49, 5.20s/it]
80%|████████ | 700/870 [1:26:30<12:51, 4.54s/it]
{'loss': 0.0098, 'grad_norm': 0.11490526050329208, 'learning_rate': 1.962724667617436e-05, 'epoch': 0.8}
80%|████████ | 700/870 [1:26:30<12:51, 4.54s/it]
0%| | 0/38 [00:00<?, ?it/s][A
5%|▌ | 2/38 [00:00<00:13, 2.62it/s][A
8%|▊ | 3/38 [00:01<00:15, 2.20it/s][A
11%|█ | 4/38 [00:01<00:16, 2.08it/s][A
13%|█▎ | 5/38 [00:02<00:15, 2.08it/s][A
16%|█▌ | 6/38 [00:02<00:15, 2.10it/s][A
18%|█▊ | 7/38 [00:03<00:14, 2.18it/s][A
21%|██ | 8/38 [00:03<00:13, 2.27it/s][A
24%|██▎ | 9/38 [00:03<00:11, 2.44it/s][A
26%|██▋ | 10/38 [00:04<00:13, 2.10it/s][A
29%|██▉ | 11/38 [00:05<00:14, 1.82it/s][A
32%|███▏ | 12/38 [00:05<00:15, 1.73it/s][A
34%|███▍ | 13/38 [00:06<00:14, 1.76it/s][A
37%|███▋ | 14/38 [00:06<00:13, 1.83it/s][A
39%|███▉ | 15/38 [00:07<00:12, 1.91it/s][A
42%|████▏ | 16/38 [00:07<00:11, 1.99it/s][A
45%|████▍ | 17/38 [00:08<00:09, 2.12it/s][A
47%|████▋ | 18/38 [00:08<00:08, 2.28it/s][A
50%|█████ | 19/38 [00:09<00:09, 2.01it/s][A
53%|█████▎ | 20/38 [00:09<00:09, 1.83it/s][A
55%|█████▌ | 21/38 [00:10<00:09, 1.84it/s][A
58%|█████▊ | 22/38 [00:11<00:08, 1.82it/s][A
61%|██████ | 23/38 [00:11<00:08, 1.87it/s][A
63%|██████▎ | 24/38 [00:12<00:07, 1.91it/s][A
66%|██████▌ | 25/38 [00:12<00:06, 1.98it/s][A
68%|██████▊ | 26/38 [00:12<00:05, 2.08it/s][A
71%|███████ | 27/38 [00:13<00:04, 2.27it/s][A
74%|███████▎ | 28/38 [00:13<00:04, 2.02it/s][A
76%|███████▋ | 29/38 [00:14<00:04, 1.83it/s][A
79%|███████▉ | 30/38 [00:15<00:04, 1.82it/s][A
82%|████████▏ | 31/38 [00:15<00:03, 1.85it/s][A
84%|████████▍ | 32/38 [00:16<00:03, 1.89it/s][A
87%|████████▋ | 33/38 [00:16<00:02, 1.98it/s][A
89%|████████▉ | 34/38 [00:17<00:01, 2.10it/s][A
92%|█████████▏| 35/38 [00:17<00:01, 2.19it/s][A
95%|█████████▍| 36/38 [00:17<00:00, 2.41it/s][A
97%|█████████▋| 37/38 [00:18<00:00, 2.29it/s][A
100%|██████████| 38/38 [00:18<00:00, 2.17it/s][A
[A{'eval_loss': 0.012033017352223396, 'eval_runtime': 20.3491, 'eval_samples_per_second': 14.743, 'eval_steps_per_second': 1.867, 'epoch': 0.8}
80%|████████ | 700/870 [1:26:50<12:51, 4.54s/it]
100%|██████████| 38/38 [00:19<00:00, 2.17it/s][A
[A
81%|████████ | 701/870 [1:27:02<36:11, 12.85s/it]
81%|████████ | 702/870 [1:27:12<33:54, 12.11s/it]
81%|████████ | 703/870 [1:27:22<31:32, 11.33s/it]
81%|████████ | 704/870 [1:27:31<29:36, 10.70s/it]
81%|████████ | 705/870 [1:27:40<27:46, 10.10s/it]
81%|████████ | 706/870 [1:27:49<26:35, 9.73s/it]
81%|████████▏ | 707/870 [1:27:56<24:54, 9.17s/it]
81%|████████▏ | 708/870 [1:28:04<23:35, 8.74s/it]
81%|████████▏ | 709/870 [1:28:12<22:38, 8.44s/it]
82%|████████▏ | 710/870 [1:28:20<21:56, 8.23s/it]
{'loss': 0.0183, 'grad_norm': 0.17799951136112213, 'learning_rate': 1.746619853524991e-05, 'epoch': 0.82}
82%|████████▏ | 710/870 [1:28:20<21:56, 8.23s/it]
82%|████████▏ | 711/870 [1:28:27<21:28, 8.11s/it]
82%|████████▏ | 712/870 [1:28:35<20:45, 7.88s/it]
82%|████████▏ | 713/870 [1:28:42<20:01, 7.65s/it]
82%|████████▏ | 714/870 [1:28:49<19:20, 7.44s/it]
82%|████████▏ | 715/870 [1:28:56<18:37, 7.21s/it]
82%|████████▏ | 716/870 [1:29:02<18:17, 7.12s/it]
82%|████████▏ | 717/870 [1:29:09<17:27, 6.84s/it]
83%|████████▎ | 718/870 [1:29:15<16:50, 6.65s/it]
83%|████████▎ | 719/870 [1:29:21<16:15, 6.46s/it]
83%|████████▎ | 720/870 [1:29:27<15:59, 6.40s/it]
{'loss': 0.0125, 'grad_norm': 0.1625465303659439, 'learning_rate': 1.5419761362593332e-05, 'epoch': 0.83}
83%|████████▎ | 720/870 [1:29:27<15:59, 6.40s/it]
83%|████████▎ | 721/870 [1:29:33<15:22, 6.19s/it]
83%|████████▎ | 722/870 [1:29:38<14:46, 5.99s/it]
83%|████████▎ | 723/870 [1:29:44<14:05, 5.75s/it]
83%|████████▎ | 724/870 [1:29:48<13:07, 5.39s/it]
83%|████████▎ | 725/870 [1:29:52<11:38, 4.82s/it]
83%|████████▎ | 726/870 [1:30:04<16:53, 7.04s/it]
84%|████████▎ | 727/870 [1:30:14<19:02, 7.99s/it]
84%|████████▎ | 728/870 [1:30:23<19:58, 8.44s/it]
84%|████████▍ | 729/870 [1:30:33<20:23, 8.68s/it]
84%|████████▍ | 730/870 [1:30:41<20:18, 8.71s/it]
{'loss': 0.0111, 'grad_norm': 0.04061046242713928, 'learning_rate': 1.3490776953223105e-05, 'epoch': 0.84}
84%|████████▍ | 730/870 [1:30:41<20:18, 8.71s/it]
84%|████████▍ | 731/870 [1:30:50<19:56, 8.61s/it]
84%|████████▍ | 732/870 [1:30:58<19:24, 8.44s/it]
84%|████████▍ | 733/870 [1:31:06<18:50, 8.25s/it]
84%|████████▍ | 734/870 [1:31:14<18:31, 8.18s/it]
84%|████████▍ | 735/870 [1:31:21<18:03, 8.02s/it]
85%|████████▍ | 736/870 [1:31:29<17:35, 7.88s/it]
85%|████████▍ | 737/870 [1:31:36<16:54, 7.63s/it]
85%|████████▍ | 738/870 [1:31:43<16:26, 7.48s/it]
85%|████████▍ | 739/870 [1:31:50<16:04, 7.36s/it]
85%|████████▌ | 740/870 [1:31:57<15:30, 7.16s/it]
{'loss': 0.0102, 'grad_norm': 0.026591680943965912, 'learning_rate': 1.1681924000801791e-05, 'epoch': 0.85}
85%|████████▌ | 740/870 [1:31:57<15:30, 7.16s/it]
85%|████████▌ | 741/870 [1:32:03<14:58, 6.97s/it]
85%|████████▌ | 742/870 [1:32:10<14:21, 6.73s/it]
85%|████████▌ | 743/870 [1:32:16<13:55, 6.58s/it]
86%|████████▌ | 744/870 [1:32:22<13:39, 6.50s/it]
86%|████████▌ | 745/870 [1:32:28<13:12, 6.34s/it]
86%|████████▌ | 746/870 [1:32:34<12:41, 6.14s/it]
86%|████████▌ | 747/870 [1:32:39<12:12, 5.95s/it]
86%|████████▌ | 748/870 [1:32:45<11:39, 5.73s/it]
86%|████████▌ | 749/870 [1:32:49<11:03, 5.49s/it]
86%|████████▌ | 750/870 [1:32:53<09:33, 4.78s/it]
{'loss': 0.0098, 'grad_norm': 0.04456641525030136, 'learning_rate': 9.995714377854936e-06, 'epoch': 0.86}
86%|████████▌ | 750/870 [1:32:53<09:33, 4.78s/it]
86%|████████▋ | 751/870 [1:33:04<13:34, 6.85s/it]
86%|████████▋ | 752/870 [1:33:14<15:27, 7.86s/it]
87%|████████▋ | 753/870 [1:33:24<16:06, 8.26s/it]
87%|████████▋ | 754/870 [1:33:33<16:33, 8.56s/it]
87%|████████▋ | 755/870 [1:33:42<16:28, 8.60s/it]
87%|████████▋ | 756/870 [1:33:50<16:13, 8.54s/it]
87%|████████▋ | 757/870 [1:33:58<15:54, 8.45s/it]
87%|████████▋ | 758/870 [1:34:06<15:38, 8.38s/it]
87%|████████▋ | 759/870 [1:34:14<15:09, 8.19s/it]
87%|████████▋ | 760/870 [1:34:22<14:35, 7.96s/it]
{'loss': 0.0162, 'grad_norm': 0.19221937656402588, 'learning_rate': 8.434489647647092e-06, 'epoch': 0.87}
87%|████████▋ | 760/870 [1:34:22<14:35, 7.96s/it]
87%|████████▋ | 761/870 [1:34:29<14:05, 7.76s/it]
88%|████████▊ | 762/870 [1:34:36<13:42, 7.62s/it]
88%|████████▊ | 763/870 [1:34:43<13:19, 7.47s/it]
88%|████████▊ | 764/870 [1:34:50<12:50, 7.27s/it]
88%|████████▊ | 765/870 [1:34:57<12:29, 7.13s/it]
88%|████████▊ | 766/870 [1:35:04<12:07, 6.99s/it]
88%|████████▊ | 767/870 [1:35:10<11:51, 6.91s/it]
88%|████████▊ | 768/870 [1:35:17<11:23, 6.71s/it]
88%|████████▊ | 769/870 [1:35:23<10:56, 6.50s/it]
89%|████████▊ | 770/870 [1:35:28<10:31, 6.32s/it]
{'loss': 0.0099, 'grad_norm': 0.04924648255109787, 'learning_rate': 7.000417812558302e-06, 'epoch': 0.89}
89%|████████▊ | 770/870 [1:35:28<10:31, 6.32s/it]
89%|████████▊ | 771/870 [1:35:34<10:06, 6.13s/it]
89%|████████▊ | 772/870 [1:35:40<09:52, 6.04s/it]
89%|████████▉ | 773/870 [1:35:45<09:21, 5.79s/it]
89%|████████▉ | 774/870 [1:35:50<08:34, 5.36s/it]
89%|████████▉ | 775/870 [1:35:53<07:23, 4.67s/it]
89%|████████▉ | 776/870 [1:36:05<11:01, 7.04s/it]
89%|████████▉ | 777/870 [1:36:16<12:26, 8.03s/it]
89%|████████▉ | 778/870 [1:36:25<12:59, 8.48s/it]
90%|████████▉ | 779/870 [1:36:34<13:11, 8.70s/it]
90%|████████▉ | 780/870 [1:36:43<12:58, 8.65s/it]
{'loss': 0.0127, 'grad_norm': 0.09919004142284393, 'learning_rate': 5.695490303477124e-06, 'epoch': 0.9}
90%|████████▉ | 780/870 [1:36:43<12:58, 8.65s/it]
90%|████████▉ | 781/870 [1:36:51<12:48, 8.64s/it]
90%|████████▉ | 782/870 [1:36:59<12:24, 8.46s/it]
90%|█████████ | 783/870 [1:37:07<11:57, 8.24s/it]
90%|█████████ | 784/870 [1:37:15<11:39, 8.14s/it]
90%|█████████ | 785/870 [1:37:23<11:20, 8.00s/it]
90%|█████████ | 786/870 [1:37:31<11:07, 7.95s/it]
90%|█████████ | 787/870 [1:37:38<10:37, 7.68s/it]
91%|█████████ | 788/870 [1:37:45<10:12, 7.47s/it]
91%|█████████ | 789/870 [1:37:51<09:51, 7.30s/it]
91%|█████████ | 790/870 [1:37:58<09:30, 7.13s/it]
{'loss': 0.0111, 'grad_norm': 0.04783658683300018, 'learning_rate': 4.521519214390257e-06, 'epoch': 0.91}
91%|█████████ | 790/870 [1:37:58<09:30, 7.13s/it]
91%|█████████ | 791/870 [1:38:05<09:18, 7.07s/it]
91%|█████████ | 792/870 [1:38:11<08:53, 6.84s/it]
91%|█████████ | 793/870 [1:38:18<08:28, 6.61s/it]
91%|█████████▏| 794/870 [1:38:24<08:08, 6.43s/it]
91%|█████████▏| 795/870 [1:38:29<07:46, 6.23s/it]
91%|█████████▏| 796/870 [1:38:35<07:35, 6.16s/it]
92%|█████████▏| 797/870 [1:38:41<07:15, 5.97s/it]
92%|█████████▏| 798/870 [1:38:46<06:53, 5.74s/it]
92%|█████████▏| 799/870 [1:38:51<06:27, 5.46s/it]
92%|█████████▏| 800/870 [1:38:54<05:36, 4.80s/it]
{'loss': 0.0094, 'grad_norm': 0.11320256441831589, 'learning_rate': 3.4801347860093924e-06, 'epoch': 0.92}
92%|█████████▏| 800/870 [1:38:54<05:36, 4.80s/it]
0%| | 0/38 [00:00<?, ?it/s][A
5%|▌ | 2/38 [00:00<00:13, 2.71it/s][A
8%|▊ | 3/38 [00:01<00:15, 2.26it/s][A
11%|█ | 4/38 [00:01<00:15, 2.16it/s][A
13%|█▎ | 5/38 [00:02<00:15, 2.07it/s][A
16%|█▌ | 6/38 [00:02<00:15, 2.12it/s][A
18%|█▊ | 7/38 [00:03<00:14, 2.19it/s][A
21%|██ | 8/38 [00:03<00:13, 2.29it/s][A
24%|██▎ | 9/38 [00:03<00:12, 2.41it/s][A
26%|██▋ | 10/38 [00:04<00:13, 2.08it/s][A
29%|██▉ | 11/38 [00:05<00:14, 1.86it/s][A
32%|███▏ | 12/38 [00:05<00:14, 1.83it/s][A
34%|███▍ | 13/38 [00:06<00:13, 1.83it/s][A
37%|███▋ | 14/38 [00:06<00:12, 1.87it/s][A
39%|███▉ | 15/38 [00:07<00:11, 1.94it/s][A
42%|████▏ | 16/38 [00:07<00:11, 2.00it/s][A
45%|████▍ | 17/38 [00:08<00:09, 2.12it/s][A
47%|████▋ | 18/38 [00:08<00:08, 2.32it/s][A
50%|█████ | 19/38 [00:09<00:09, 2.10it/s][A
53%|█████▎ | 20/38 [00:09<00:09, 1.90it/s][A
55%|█████▌ | 21/38 [00:10<00:09, 1.87it/s][A
58%|█████▊ | 22/38 [00:10<00:08, 1.83it/s][A
61%|██████ | 23/38 [00:11<00:07, 1.88it/s][A
63%|██████▎ | 24/38 [00:11<00:07, 1.92it/s][A
66%|██████▌ | 25/38 [00:12<00:06, 1.98it/s][A
68%|██████▊ | 26/38 [00:12<00:05, 2.11it/s][A
71%|███████ | 27/38 [00:13<00:04, 2.26it/s][A
74%|███████▎ | 28/38 [00:13<00:04, 2.01it/s][A
76%|███████▋ | 29/38 [00:14<00:04, 1.81it/s][A
79%|███████▉ | 30/38 [00:14<00:04, 1.79it/s][A
82%|████████▏ | 31/38 [00:15<00:03, 1.80it/s][A
84%|████████▍ | 32/38 [00:16<00:03, 1.84it/s][A
87%|████████▋ | 33/38 [00:16<00:02, 1.90it/s][A
89%|████████▉ | 34/38 [00:16<00:02, 1.99it/s][A
92%|█████████▏| 35/38 [00:17<00:01, 2.13it/s][A
95%|█████████▍| 36/38 [00:17<00:00, 2.30it/s][A
97%|█████████▋| 37/38 [00:18<00:00, 2.23it/s][A
100%|██████████| 38/38 [00:18<00:00, 2.17it/s][A
[A{'eval_loss': 0.013403651304543018, 'eval_runtime': 20.315, 'eval_samples_per_second': 14.767, 'eval_steps_per_second': 1.871, 'epoch': 0.92}
92%|█████████▏| 800/870 [1:39:14<05:36, 4.80s/it]
100%|██████████| 38/38 [00:19<00:00, 2.17it/s][A
[A
92%|█████████▏| 801/870 [1:39:28<15:33, 13.53s/it]
92%|█████████▏| 802/870 [1:39:38<14:12, 12.54s/it]
92%|█████████▏| 803/870 [1:39:48<13:01, 11.66s/it]
92%|█████████▏| 804/870 [1:39:57<12:03, 10.96s/it]
93%|█████████▎| 805/870 [1:40:06<11:15, 10.40s/it]
93%|█████████▎| 806/870 [1:40:15<10:27, 9.80s/it]
93%|█████████▎| 807/870 [1:40:23<09:48, 9.34s/it]
93%|█████████▎| 808/870 [1:40:31<09:12, 8.91s/it]
93%|█████████▎| 809/870 [1:40:39<08:41, 8.55s/it]
93%|█████████▎| 810/870 [1:40:46<08:20, 8.35s/it]
{'loss': 0.016, 'grad_norm': 0.32974982261657715, 'learning_rate': 2.572783141929469e-06, 'epoch': 0.93}
93%|█████████▎| 810/870 [1:40:46<08:20, 8.35s/it]
93%|█████████▎| 811/870 [1:40:54<07:55, 8.05s/it]
93%|█████████▎| 812/870 [1:41:01<07:31, 7.79s/it]
93%|█████████▎| 813/870 [1:41:08<07:12, 7.58s/it]
94%|█████████▎| 814/870 [1:41:15<06:53, 7.38s/it]
94%|█████████▎| 815/870 [1:41:22<06:36, 7.21s/it]
94%|█████████▍| 816/870 [1:41:28<06:18, 7.01s/it]
94%|█████████▍| 817/870 [1:41:34<05:57, 6.75s/it]
94%|█████████▍| 818/870 [1:41:41<05:41, 6.57s/it]
94%|█████████▍| 819/870 [1:41:47<05:26, 6.39s/it]
94%|█████████▍| 820/870 [1:41:53<05:14, 6.28s/it]
{'loss': 0.0095, 'grad_norm': 0.24235671758651733, 'learning_rate': 1.8007242804619628e-06, 'epoch': 0.94}
94%|█████████▍| 820/870 [1:41:53<05:14, 6.28s/it]
94%|█████████▍| 821/870 [1:41:58<04:59, 6.11s/it]
94%|█████████▍| 822/870 [1:42:04<04:44, 5.93s/it]
95%|█████████▍| 823/870 [1:42:09<04:26, 5.68s/it]
95%|█████████▍| 824/870 [1:42:14<04:09, 5.42s/it]
95%|█████████▍| 825/870 [1:42:17<03:31, 4.70s/it]
95%|█████████▍| 826/870 [1:42:28<04:59, 6.81s/it]
95%|█████████▌| 827/870 [1:42:38<05:34, 7.77s/it]
95%|█████████▌| 828/870 [1:42:48<05:47, 8.27s/it]
95%|█████████▌| 829/870 [1:42:57<05:53, 8.63s/it]
95%|█████████▌| 830/870 [1:43:06<05:46, 8.66s/it]
{'loss': 0.01, 'grad_norm': 0.08694616705179214, 'learning_rate': 1.165030324932137e-06, 'epoch': 0.95}
95%|█████████▌| 830/870 [1:43:06<05:46, 8.66s/it]
96%|█████████▌| 831/870 [1:43:15<05:35, 8.61s/it]
96%|█████████▌| 832/870 [1:43:23<05:23, 8.51s/it]
96%|█████████▌| 833/870 [1:43:31<05:10, 8.38s/it]
96%|█████████▌| 834/870 [1:43:39<04:58, 8.30s/it]
96%|█████████▌| 835/870 [1:43:47<04:44, 8.12s/it]
96%|█████████▌| 836/870 [1:43:54<04:30, 7.94s/it]
96%|█████████▌| 837/870 [1:44:01<04:13, 7.69s/it]
96%|█████████▋| 838/870 [1:44:09<04:00, 7.52s/it]
96%|█████████▋| 839/870 [1:44:16<03:50, 7.43s/it]
97%|█████████▋| 840/870 [1:44:22<03:36, 7.21s/it]
{'loss': 0.0113, 'grad_norm': 0.09477345645427704, 'learning_rate': 6.665840348696994e-07, 'epoch': 0.97}
97%|█████████▋| 840/870 [1:44:22<03:36, 7.21s/it]
97%|█████████▋| 841/870 [1:44:29<03:22, 6.98s/it]
97%|█████████▋| 842/870 [1:44:35<03:09, 6.77s/it]
97%|█████████▋| 843/870 [1:44:42<03:00, 6.68s/it]
97%|█████████▋| 844/870 [1:44:48<02:48, 6.48s/it]
97%|█████████▋| 845/870 [1:44:54<02:37, 6.30s/it]
97%|█████████▋| 846/870 [1:44:59<02:26, 6.11s/it]
97%|█████████▋| 847/870 [1:45:05<02:15, 5.91s/it]
97%|█████████▋| 848/870 [1:45:10<02:07, 5.80s/it]
98%|█████████▊| 849/870 [1:45:15<01:54, 5.45s/it]
98%|█████████▊| 850/870 [1:45:18<01:35, 4.78s/it]
{'loss': 0.0086, 'grad_norm': 0.048642598092556, 'learning_rate': 3.060775801604354e-07, 'epoch': 0.98}
98%|█████████▊| 850/870 [1:45:18<01:35, 4.78s/it]
98%|█████████▊| 851/870 [1:45:29<02:08, 6.75s/it]
98%|█████████▊| 852/870 [1:45:39<02:16, 7.59s/it]
98%|█████████▊| 853/870 [1:45:49<02:19, 8.20s/it]
98%|█████████▊| 854/870 [1:45:57<02:12, 8.29s/it]
98%|█████████▊| 855/870 [1:46:05<02:04, 8.28s/it]
98%|█████████▊| 856/870 [1:46:13<01:54, 8.19s/it]
99%|█████████▊| 857/870 [1:46:21<01:45, 8.09s/it]
99%|█████████▊| 858/870 [1:46:29<01:36, 8.03s/it]
99%|█████████▊| 859/870 [1:46:36<01:26, 7.83s/it]
99%|█████████▉| 860/870 [1:46:43<01:16, 7.62s/it]
{'loss': 0.0148, 'grad_norm': 0.06935633718967438, 'learning_rate': 8.401157986109853e-08, 'epoch': 0.99}
99%|█████████▉| 860/870 [1:46:43<01:16, 7.62s/it]
99%|█████████▉| 861/870 [1:46:50<01:06, 7.37s/it]
99%|█████████▉| 862/870 [1:46:57<00:57, 7.23s/it]
99%|█████████▉| 863/870 [1:47:04<00:48, 6.99s/it]
99%|█████████▉| 864/870 [1:47:10<00:40, 6.75s/it]
99%|█████████▉| 865/870 [1:47:16<00:32, 6.53s/it]
100%|█████████▉| 866/870 [1:47:22<00:25, 6.34s/it]
100%|█████████▉| 867/870 [1:47:28<00:18, 6.23s/it]
100%|█████████▉| 868/870 [1:47:33<00:11, 5.96s/it]
100%|█████████▉| 869/870 [1:47:38<00:05, 5.62s/it]
100%|██████████| 870/870 [1:47:46<00:00, 6.37s/it]
{'loss': 0.0087, 'grad_norm': 0.06433246284723282, 'learning_rate': 6.944070123582336e-10, 'epoch': 1.0}
100%|██████████| 870/870 [1:47:46<00:00, 6.37s/it]
{'train_runtime': 6469.3899, 'train_samples_per_second': 4.303, 'train_steps_per_second': 0.134, 'train_loss': 0.024751038421159503, 'epoch': 1.0}
100%|██████████| 870/870 [1:47:47<00:00, 6.37s/it]
100%|██████████| 870/870 [1:47:47<00:00, 7.43s/it]
[rank0]:[W1024 16:20:21.677824557 ProcessGroupNCCL.cpp:1538] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
|