| nohup: ignoring input |
| Loading tokenizer from Qwen/Qwen3-1.7B |
| Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. |
| Loading multilingual SFT dataset from /scratch/nico/data/multilingual_v2_ci/sft_train.jsonl |
|
Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 14866 examples [00:00, 91805.32 examples/s]
Generating train split: 29663 examples [00:00, 86725.61 examples/s]
Generating train split: 44688 examples [00:00, 86618.75 examples/s]
Generating train split: 59498 examples [00:00, 87311.12 examples/s]
Generating train split: 74414 examples [00:00, 83028.84 examples/s]
Generating train split: 89455 examples [00:01, 80582.97 examples/s]
Generating train split: 93371 examples [00:01, 77753.12 examples/s] |
| Full dataset size: 93371 |
| Formatting examples with Qwen chat template... |
|
Map: 0%| | 0/93371 [00:00<?, ? examples/s]
Map: 0%| | 428/93371 [00:00<00:22, 4221.28 examples/s]
Map: 1%|▏ | 1000/93371 [00:00<00:18, 4998.83 examples/s]
Map: 2%|▍ | 1572/93371 [00:00<00:17, 5318.73 examples/s]
Map: 2%|▌ | 2124/93371 [00:00<00:16, 5396.03 examples/s]
Map: 3%|▋ | 2700/93371 [00:00<00:16, 5523.81 examples/s]
Map: 3%|▊ | 3257/93371 [00:00<00:16, 5535.80 examples/s]
Map: 4%|▉ | 3833/93371 [00:00<00:15, 5605.81 examples/s]
Map: 5%|█ | 4399/93371 [00:00<00:15, 5621.08 examples/s]
Map: 5%|█▏ | 4974/93371 [00:00<00:15, 5658.67 examples/s]
Map: 6%|█▍ | 5786/93371 [00:01<00:15, 5556.46 examples/s]
Map: 7%|█▋ | 6623/93371 [00:01<00:15, 5561.43 examples/s]
Map: 8%|█▊ | 7185/93371 [00:01<00:15, 5574.56 examples/s]
Map: 8%|█▉ | 7765/93371 [00:01<00:15, 5631.75 examples/s]
Map: 9%|██ | 8599/93371 [00:01<00:15, 5594.79 examples/s]
Map: 10%|██▎ | 9441/93371 [00:01<00:15, 5594.51 examples/s]
Map: 11%|██▎ | 10008/93371 [00:01<00:14, 5610.17 examples/s]
Map: 11%|██▍ | 10586/93371 [00:01<00:14, 5651.56 examples/s]
Map: 12%|██▋ | 11433/93371 [00:02<00:14, 5645.06 examples/s]
Map: 13%|██▊ | 12000/93371 [00:02<00:14, 5637.05 examples/s]
Map: 13%|██▉ | 12574/93371 [00:02<00:14, 5663.98 examples/s]
Map: 14%|███▏ | 13424/93371 [00:02<00:14, 5662.84 examples/s]
Map: 15%|███▎ | 14000/93371 [00:02<00:14, 5645.06 examples/s]
Map: 16%|███▍ | 14573/93371 [00:02<00:13, 5664.21 examples/s]
Map: 17%|███▋ | 15424/93371 [00:02<00:13, 5662.05 examples/s]
Map: 17%|███▊ | 16242/93371 [00:02<00:13, 5584.84 examples/s]
Map: 18%|███▉ | 16816/93371 [00:03<00:13, 5620.54 examples/s]
Map: 19%|████ | 17380/93371 [00:03<00:13, 5623.05 examples/s]
Map: 19%|████▏ | 17959/93371 [00:03<00:13, 5666.95 examples/s]
Map: 20%|████▍ | 18810/93371 [00:03<00:13, 5666.20 examples/s]
Map: 21%|████▌ | 19571/93371 [00:03<00:13, 5457.60 examples/s]
Map: 22%|████▋ | 20132/93371 [00:03<00:13, 5491.09 examples/s]
Map: 22%|████▉ | 20958/93371 [00:03<00:13, 5493.00 examples/s]
Map: 23%|█████ | 21521/93371 [00:03<00:13, 5524.92 examples/s]
Map: 24%|█████▏ | 22079/93371 [00:03<00:12, 5536.05 examples/s]
Map: 24%|█████▎ | 22657/93371 [00:04<00:12, 5599.09 examples/s]
Map: 25%|█████▌ | 23498/93371 [00:04<00:12, 5598.83 examples/s]
Map: 26%|█████▋ | 24344/93371 [00:04<00:12, 5610.94 examples/s]
Map: 27%|█████▊ | 24919/93371 [00:04<00:12, 5641.64 examples/s]
Map: 28%|██████ | 25750/93371 [00:04<00:12, 5602.21 examples/s]
Map: 28%|██████▎ | 26585/93371 [00:04<00:11, 5588.61 examples/s]
Map: 29%|██████▍ | 27424/93371 [00:04<00:11, 5588.54 examples/s]
Map: 30%|██████▌ | 27996/93371 [00:05<00:11, 5616.41 examples/s]
Map: 31%|██████▊ | 28837/93371 [00:05<00:11, 5611.31 examples/s]
Map: 32%|██████▉ | 29673/93371 [00:05<00:11, 5595.70 examples/s]
Map: 33%|███████▏ | 30509/93371 [00:05<00:11, 5585.21 examples/s]
Map: 34%|███████▍ | 31346/93371 [00:05<00:11, 5580.23 examples/s]
Map: 34%|███████▌ | 31930/93371 [00:05<00:10, 5637.87 examples/s]
Map: 35%|███████▋ | 32766/93371 [00:05<00:10, 5614.46 examples/s]
Map: 36%|███████▉ | 33601/93371 [00:06<00:10, 5597.14 examples/s]
Map: 37%|████████ | 34163/93371 [00:06<00:10, 5600.82 examples/s]
Map: 37%|████████▏ | 34728/93371 [00:06<00:10, 5609.31 examples/s]
Map: 38%|████████▍ | 35565/93371 [00:06<00:10, 5595.77 examples/s]
Map: 39%|████████▌ | 36402/93371 [00:06<00:10, 5586.59 examples/s]
Map: 40%|████████▋ | 36984/93371 [00:06<00:09, 5639.98 examples/s]
Map: 41%|████████▉ | 37823/93371 [00:06<00:09, 5620.22 examples/s]
Map: 41%|█████████ | 38667/93371 [00:06<00:09, 5621.42 examples/s]
Map: 42%|█████████▎ | 39501/93371 [00:07<00:09, 5599.67 examples/s]
Map: 43%|█████████▌ | 40335/93371 [00:07<00:09, 5583.81 examples/s]
Map: 44%|█████████▋ | 40897/93371 [00:07<00:09, 5590.69 examples/s]
Map: 45%|█████████▊ | 41747/93371 [00:07<00:09, 5611.52 examples/s]
Map: 46%|██████████ | 42587/93371 [00:07<00:09, 5605.72 examples/s]
Map: 47%|██████████▏ | 43427/93371 [00:07<00:08, 5602.28 examples/s]
Map: 47%|██████████▎ | 44000/93371 [00:07<00:08, 5595.00 examples/s]
Map: 48%|██████████▌ | 44578/93371 [00:07<00:08, 5639.20 examples/s]
Map: 49%|██████████▋ | 45412/93371 [00:08<00:08, 5608.32 examples/s]
Map: 49%|██████████▊ | 45995/93371 [00:08<00:08, 5661.11 examples/s]
Map: 50%|███████████ | 46827/93371 [00:08<00:08, 5618.13 examples/s]
Map: 51%|███████████▏ | 47668/93371 [00:08<00:08, 5612.61 examples/s]
Map: 52%|███████████▎ | 48231/93371 [00:08<00:08, 5612.52 examples/s]
Map: 52%|███████████▍ | 48799/93371 [00:08<00:07, 5626.36 examples/s]
Map: 53%|███████████▋ | 49637/93371 [00:08<00:07, 5608.06 examples/s]
Map: 54%|███████████▉ | 50473/93371 [00:09<00:07, 5593.76 examples/s]
Map: 55%|████████████ | 51035/93371 [00:09<00:07, 5597.01 examples/s]
Map: 55%|████████████▏ | 51602/93371 [00:09<00:07, 5615.08 examples/s]
Map: 56%|████████████▎ | 52452/93371 [00:09<00:07, 5629.61 examples/s]
Map: 57%|████████████▌ | 53288/93371 [00:09<00:07, 5599.14 examples/s]
Map: 58%|████████████▋ | 53867/93371 [00:09<00:07, 5642.07 examples/s]
Map: 59%|████████████▉ | 54702/93371 [00:09<00:06, 5612.82 examples/s]
Map: 59%|█████████████ | 55540/93371 [00:09<00:06, 5600.94 examples/s]
Map: 60%|█████████████▎ | 56378/93371 [00:10<00:06, 5592.10 examples/s]
Map: 61%|█████████████▍ | 56947/93371 [00:10<00:06, 5611.57 examples/s]
Map: 62%|█████████████▌ | 57786/93371 [00:10<00:06, 5600.58 examples/s]
Map: 63%|█████████████▊ | 58635/93371 [00:10<00:06, 5615.34 examples/s]
Map: 64%|██████████████ | 59475/93371 [00:10<00:06, 5607.23 examples/s]
Map: 65%|██████████████▏ | 60312/93371 [00:10<00:05, 5595.09 examples/s]
Map: 65%|██████████████▎ | 60880/93371 [00:10<00:05, 5612.44 examples/s]
Map: 66%|██████████████▌ | 61708/93371 [00:11<00:05, 5577.73 examples/s]
Map: 67%|██████████████▋ | 62535/93371 [00:11<00:05, 5553.40 examples/s]
Map: 68%|██████████████▉ | 63373/93371 [00:11<00:05, 5561.94 examples/s]
Map: 68%|███████████████ | 63949/93371 [00:11<00:05, 5604.68 examples/s]
Map: 69%|███████████████▎ | 64783/93371 [00:11<00:05, 5586.96 examples/s]
Map: 70%|███████████████▍ | 65627/93371 [00:11<00:04, 5597.47 examples/s]
Map: 71%|███████████████▋ | 66467/93371 [00:11<00:04, 5596.48 examples/s]
Map: 72%|███████████████▊ | 67028/93371 [00:11<00:04, 5595.81 examples/s]
Map: 72%|███████████████▉ | 67597/93371 [00:12<00:04, 5617.56 examples/s]
Map: 73%|████████████████▏ | 68445/93371 [00:12<00:04, 5626.06 examples/s]
Map: 74%|████████████████▎ | 69294/93371 [00:12<00:04, 5633.32 examples/s]
Map: 75%|████████████████▍ | 69864/93371 [00:12<00:04, 5646.46 examples/s]
Map: 75%|████████████████▌ | 70430/93371 [00:12<00:04, 5646.48 examples/s]
Map: 76%|████████████████▋ | 71000/93371 [00:12<00:03, 5617.03 examples/s]
Map: 77%|████████████████▊ | 71565/93371 [00:12<00:03, 5623.80 examples/s]
Map: 78%|█████████████████ | 72411/93371 [00:12<00:03, 5624.41 examples/s]
Map: 78%|█████████████████▏ | 72975/93371 [00:13<00:03, 5625.62 examples/s]
Map: 79%|█████████████████▍ | 73814/93371 [00:13<00:03, 5611.50 examples/s]
Map: 80%|█████████████████▌ | 74391/93371 [00:13<00:03, 5647.36 examples/s]
Map: 80%|█████████████████▋ | 74976/93371 [00:13<00:03, 5698.65 examples/s]
Map: 81%|█████████████████▊ | 75830/93371 [00:13<00:03, 5691.75 examples/s]
Map: 82%|██████████████████ | 76682/93371 [00:13<00:02, 5686.40 examples/s]
Map: 83%|██████████████████▎ | 77523/93371 [00:13<00:02, 5654.90 examples/s]
Map: 84%|██████████████████▍ | 78361/93371 [00:13<00:02, 5629.23 examples/s]
Map: 85%|██████████████████▌ | 78938/93371 [00:14<00:02, 5659.81 examples/s]
Map: 85%|██████████████████▊ | 79793/93371 [00:14<00:02, 5668.76 examples/s]
Map: 86%|██████████████████▉ | 80629/93371 [00:14<00:02, 5636.17 examples/s]
Map: 87%|███████████████████▏ | 81468/93371 [00:14<00:02, 5621.55 examples/s]
Map: 88%|███████████████████▍ | 82311/93371 [00:14<00:01, 5616.14 examples/s]
Map: 89%|███████████████████▌ | 82880/93371 [00:14<00:01, 5631.44 examples/s]
Map: 90%|███████████████████▋ | 83710/93371 [00:14<00:01, 5596.71 examples/s]
Map: 91%|███████████████████▉ | 84545/93371 [00:15<00:01, 5583.06 examples/s]
Map: 91%|████████████████████ | 85382/93371 [00:15<00:01, 5577.08 examples/s]
Map: 92%|████████████████████▏ | 85942/93371 [00:15<00:01, 5581.48 examples/s]
Map: 93%|████████████████████▍ | 86774/93371 [00:15<00:01, 5565.29 examples/s]
Map: 94%|████████████████████▋ | 87612/93371 [00:15<00:01, 5569.09 examples/s]
Map: 94%|████████████████████▊ | 88175/93371 [00:15<00:00, 5579.14 examples/s]
Map: 95%|████████████████████▉ | 88747/93371 [00:15<00:00, 5612.06 examples/s]
Map: 96%|█████████████████████ | 89589/93371 [00:15<00:00, 5609.38 examples/s]
Map: 97%|█████████████████████▎| 90420/93371 [00:16<00:00, 5583.87 examples/s]
Map: 97%|█████████████████████▍| 90990/93371 [00:16<00:00, 5611.54 examples/s]
Map: 98%|█████████████████████▋| 91833/93371 [00:16<00:00, 5607.88 examples/s]
Map: 99%|█████████████████████▊| 92668/93371 [00:16<00:00, 5590.64 examples/s]
Map: 100%|██████████████████████| 93371/93371 [00:16<00:00, 4862.94 examples/s]
Map: 100%|██████████████████████| 93371/93371 [00:16<00:00, 5539.75 examples/s] |
|
Filter: 0%| | 0/93371 [00:00<?, ? examples/s]
Filter: 22%|███▊ | 21000/93371 [00:00<00:00, 205805.39 examples/s]
Filter: 46%|███████▊ | 43000/93371 [00:00<00:00, 211800.80 examples/s]
Filter: 80%|█████████████▋ | 75000/93371 [00:00<00:00, 209387.16 examples/s]
Filter: 100%|█████████████████| 93371/93371 [00:00<00:00, 193330.62 examples/s] |
| After formatting: 93371 examples |
| Filtering examples > 4096 tokens... |
|
Filter: 0%| | 0/93371 [00:00<?, ? examples/s]
Filter: 1%|▏ | 1000/93371 [00:00<00:41, 2229.31 examples/s]
Filter: 2%|▍ | 2000/93371 [00:00<00:40, 2266.71 examples/s]
Filter: 3%|▋ | 3000/93371 [00:01<00:40, 2222.03 examples/s]
Filter: 4%|▊ | 4000/93371 [00:01<00:40, 2229.07 examples/s]
Filter: 5%|█ | 5000/93371 [00:02<00:40, 2163.07 examples/s]
Filter: 6%|█▎ | 6000/93371 [00:02<00:39, 2212.13 examples/s]
Filter: 7%|█▍ | 7000/93371 [00:03<00:39, 2199.05 examples/s]
Filter: 9%|█▋ | 8000/93371 [00:03<00:38, 2200.74 examples/s]
Filter: 10%|█▉ | 9000/93371 [00:04<00:38, 2197.08 examples/s]
Filter: 11%|██ | 10000/93371 [00:04<00:37, 2217.87 examples/s]
Filter: 12%|██▏ | 11000/93371 [00:04<00:37, 2212.86 examples/s]
Filter: 13%|██▍ | 12000/93371 [00:05<00:36, 2216.92 examples/s]
Filter: 14%|██▋ | 13000/93371 [00:05<00:35, 2244.74 examples/s]
Filter: 15%|██▊ | 14000/93371 [00:06<00:36, 2195.96 examples/s]
Filter: 16%|███ | 15000/93371 [00:06<00:35, 2200.37 examples/s]
Filter: 17%|███▎ | 16000/93371 [00:07<00:34, 2224.51 examples/s]
Filter: 18%|███▍ | 17000/93371 [00:07<00:34, 2224.43 examples/s]
Filter: 19%|███▋ | 18000/93371 [00:08<00:34, 2211.83 examples/s]
Filter: 20%|███▊ | 19000/93371 [00:08<00:33, 2208.86 examples/s]
Filter: 21%|████ | 20000/93371 [00:09<00:32, 2232.72 examples/s]
Filter: 22%|████▎ | 21000/93371 [00:09<00:32, 2241.12 examples/s]
Filter: 24%|████▍ | 22000/93371 [00:09<00:32, 2225.58 examples/s]
Filter: 25%|████▋ | 23000/93371 [00:10<00:31, 2206.03 examples/s]
Filter: 26%|████▉ | 24000/93371 [00:10<00:31, 2210.28 examples/s]
Filter: 27%|█████ | 25000/93371 [00:11<00:30, 2209.48 examples/s]
Filter: 28%|█████▎ | 26000/93371 [00:11<00:30, 2201.32 examples/s]
Filter: 29%|█████▍ | 27000/93371 [00:12<00:30, 2199.88 examples/s]
Filter: 30%|█████▋ | 28000/93371 [00:12<00:29, 2217.51 examples/s]
Filter: 31%|█████▉ | 29000/93371 [00:13<00:28, 2230.54 examples/s]
Filter: 32%|██████ | 30000/93371 [00:13<00:28, 2251.86 examples/s]
Filter: 33%|██████▎ | 31000/93371 [00:13<00:28, 2212.32 examples/s]
Filter: 34%|██████▌ | 32000/93371 [00:14<00:27, 2237.02 examples/s]
Filter: 35%|██████▋ | 33000/93371 [00:14<00:26, 2241.31 examples/s]
Filter: 36%|██████▉ | 34000/93371 [00:15<00:26, 2259.74 examples/s]
Filter: 37%|███████ | 35000/93371 [00:15<00:25, 2266.66 examples/s]
Filter: 39%|███████▎ | 36000/93371 [00:16<00:25, 2249.57 examples/s]
Filter: 40%|███████▌ | 37000/93371 [00:16<00:25, 2245.17 examples/s]
Filter: 41%|███████▋ | 38000/93371 [00:17<00:24, 2249.78 examples/s]
Filter: 42%|███████▉ | 39000/93371 [00:17<00:24, 2257.05 examples/s]
Filter: 43%|████████▏ | 40000/93371 [00:17<00:23, 2249.19 examples/s]
Filter: 44%|████████▎ | 41000/93371 [00:18<00:23, 2248.86 examples/s]
Filter: 45%|████████▌ | 42000/93371 [00:18<00:22, 2262.31 examples/s]
Filter: 46%|████████▊ | 43000/93371 [00:19<00:22, 2283.70 examples/s]
Filter: 47%|████████▉ | 44000/93371 [00:19<00:21, 2258.15 examples/s]
Filter: 48%|█████████▏ | 45000/93371 [00:20<00:21, 2263.08 examples/s]
Filter: 49%|█████████▎ | 46000/93371 [00:20<00:21, 2226.48 examples/s]
Filter: 50%|█████████▌ | 47000/93371 [00:21<00:20, 2221.45 examples/s]
Filter: 51%|█████████▊ | 48000/93371 [00:21<00:20, 2204.87 examples/s]
Filter: 52%|█████████▉ | 49000/93371 [00:22<00:20, 2217.12 examples/s]
Filter: 54%|██████████▏ | 50000/93371 [00:22<00:19, 2215.55 examples/s]
Filter: 55%|██████████▍ | 51000/93371 [00:22<00:19, 2206.69 examples/s]
Filter: 56%|██████████▌ | 52000/93371 [00:23<00:18, 2223.84 examples/s]
Filter: 57%|██████████▊ | 53000/93371 [00:23<00:18, 2229.82 examples/s]
Filter: 58%|██████████▉ | 54000/93371 [00:24<00:17, 2220.65 examples/s]
Filter: 59%|███████████▏ | 55000/93371 [00:24<00:17, 2208.25 examples/s]
Filter: 60%|███████████▍ | 56000/93371 [00:25<00:16, 2216.64 examples/s]
Filter: 61%|███████████▌ | 57000/93371 [00:25<00:16, 2240.25 examples/s]
Filter: 62%|███████████▊ | 58000/93371 [00:26<00:15, 2240.46 examples/s]
Filter: 63%|████████████ | 59000/93371 [00:26<00:15, 2252.08 examples/s]
Filter: 64%|████████████▏ | 60000/93371 [00:26<00:14, 2266.79 examples/s]
Filter: 65%|████████████▍ | 61000/93371 [00:27<00:14, 2263.50 examples/s]
Filter: 66%|████████████▌ | 62000/93371 [00:27<00:14, 2226.01 examples/s]
Filter: 67%|████████████▊ | 63000/93371 [00:28<00:13, 2182.99 examples/s]
Filter: 69%|█████████████ | 64000/93371 [00:28<00:13, 2207.24 examples/s]
Filter: 70%|█████████████▏ | 65000/93371 [00:29<00:12, 2231.38 examples/s]
Filter: 71%|█████████████▍ | 66000/93371 [00:29<00:12, 2233.03 examples/s]
Filter: 72%|█████████████▋ | 67000/93371 [00:30<00:11, 2241.43 examples/s]
Filter: 73%|█████████████▊ | 68000/93371 [00:30<00:11, 2242.74 examples/s]
Filter: 74%|██████████████ | 69000/93371 [00:30<00:10, 2224.46 examples/s]
Filter: 75%|██████████████▏ | 70000/93371 [00:31<00:10, 2244.22 examples/s]
Filter: 76%|██████████████▍ | 71000/93371 [00:31<00:09, 2253.08 examples/s]
Filter: 77%|██████████████▋ | 72000/93371 [00:32<00:09, 2233.75 examples/s]
Filter: 78%|██████████████▊ | 73000/93371 [00:32<00:09, 2233.98 examples/s]
Filter: 79%|███████████████ | 74000/93371 [00:33<00:08, 2229.34 examples/s]
Filter: 80%|███████████████▎ | 75000/93371 [00:33<00:08, 2226.02 examples/s]
Filter: 81%|███████████████▍ | 76000/93371 [00:34<00:07, 2232.53 examples/s]
Filter: 82%|███████████████▋ | 77000/93371 [00:34<00:07, 2251.64 examples/s]
Filter: 84%|███████████████▊ | 78000/93371 [00:34<00:06, 2236.61 examples/s]
Filter: 85%|████████████████ | 79000/93371 [00:35<00:06, 2229.20 examples/s]
Filter: 86%|████████████████▎ | 80000/93371 [00:35<00:05, 2234.08 examples/s]
Filter: 87%|████████████████▍ | 81000/93371 [00:36<00:05, 2247.25 examples/s]
Filter: 88%|████████████████▋ | 82000/93371 [00:36<00:05, 2233.77 examples/s]
Filter: 89%|████████████████▉ | 83000/93371 [00:37<00:04, 2214.45 examples/s]
Filter: 90%|█████████████████ | 84000/93371 [00:37<00:04, 2229.89 examples/s]
Filter: 91%|█████████████████▎ | 85000/93371 [00:38<00:03, 2253.46 examples/s]
Filter: 92%|█████████████████▌ | 86000/93371 [00:38<00:03, 2240.97 examples/s]
Filter: 93%|█████████████████▋ | 87000/93371 [00:39<00:02, 2218.54 examples/s]
Filter: 94%|█████████████████▉ | 88000/93371 [00:39<00:02, 2257.12 examples/s]
Filter: 95%|██████████████████ | 89000/93371 [00:39<00:01, 2269.55 examples/s]
Filter: 96%|██████████████████▎| 90000/93371 [00:40<00:01, 2263.69 examples/s]
Filter: 97%|██████████████████▌| 91000/93371 [00:40<00:01, 2267.60 examples/s]
Filter: 99%|██████████████████▋| 92000/93371 [00:41<00:00, 2268.92 examples/s]
Filter: 100%|██████████████████▉| 93000/93371 [00:41<00:00, 2261.93 examples/s]
Filter: 100%|███████████████████| 93371/93371 [00:41<00:00, 2256.12 examples/s]
Filter: 100%|███████████████████| 93371/93371 [00:41<00:00, 2230.32 examples/s] |
| Removed 0 examples exceeding 4096 tokens |
| Final training set: 93371 examples |
|
|
| Loading model: Qwen/Qwen3-1.7B |
|
Loading weights: 0%| | 0/311 [00:00<?, ?it/s]
Loading weights: 19%|████▍ | 60/311 [00:00<00:00, 599.17it/s]
Loading weights: 39%|████████▍ | 120/311 [00:00<00:00, 534.00it/s]
Loading weights: 56%|████████████▎ | 174/311 [00:00<00:00, 462.42it/s]
Loading weights: 72%|███████████████▊ | 224/311 [00:00<00:00, 473.31it/s]
Loading weights: 88%|███████████████████▎ | 273/311 [00:00<00:00, 449.22it/s]
Loading weights: 100%|██████████████████████| 311/311 [00:00<00:00, 470.87it/s] |
| W&B logging disabled |
| [RANK 0] Padding-free training is enabled, but the attention implementation is not set to a supported flash attention variant. Padding-free training flattens batches into a single sequence, and only the following implementations are known to reliably support this: flash_attention_2, flash_attention_3, kernels-community/flash-attn2, kernels-community/flash-attn3, kernels-community/vllm-flash-attn3. Using other implementations may lead to unexpected behavior. To ensure compatibility, set `attn_implementation` in the model configuration to one of these supported options or verify that your attention mechanism can handle flattened sequences. |
| [RANK 0] You are using packing, but the attention implementation is not set to a supported flash attention variant. Packing gathers multiple samples into a single sequence, and only the following implementations are known to reliably support this: flash_attention_2, flash_attention_3, kernels-community/flash-attn2, kernels-community/flash-attn3, kernels-community/vllm-flash-attn3. Using other implementations may lead to cross-contamination between samples. To avoid this, either disable packing by setting `packing=False`, or set `attn_implementation` in the model configuration to one of these supported options. |
|
Adding EOS to train dataset: 0%| | 0/93371 [00:00<?, ? examples/s]
Adding EOS to train dataset: 2%| | 1796/93371 [00:00<00:05, 17792.12 examples
Adding EOS to train dataset: 5%| | 4467/93371 [00:00<00:04, 17798.96 examples
Adding EOS to train dataset: 8%| | 7127/93371 [00:00<00:04, 17760.58 examples
Adding EOS to train dataset: 10%| | 8975/93371 [00:00<00:04, 17985.90 examples
Adding EOS to train dataset: 12%| | 11663/93371 [00:00<00:04, 17955.11 example
Adding EOS to train dataset: 15%|▏| 14327/93371 [00:00<00:04, 17882.15 example
Adding EOS to train dataset: 18%|▏| 17000/93371 [00:00<00:04, 17839.43 example
Adding EOS to train dataset: 20%|▏| 18832/93371 [00:01<00:04, 17956.45 example
Adding EOS to train dataset: 23%|▏| 21507/93371 [00:01<00:04, 17912.16 example
Adding EOS to train dataset: 26%|▎| 24124/93371 [00:01<00:03, 17757.07 example
Adding EOS to train dataset: 28%|▎| 25938/93371 [00:01<00:03, 17756.21 example
Adding EOS to train dataset: 31%|▎| 28513/93371 [00:01<00:03, 17551.91 example
Adding EOS to train dataset: 33%|▎| 31141/93371 [00:01<00:03, 17538.64 example
Adding EOS to train dataset: 36%|▎| 33733/93371 [00:01<00:03, 17453.83 example
Adding EOS to train dataset: 39%|▍| 36319/93371 [00:02<00:03, 17383.73 example
Adding EOS to train dataset: 42%|▍| 38831/93371 [00:02<00:03, 17184.91 example
Adding EOS to train dataset: 44%|▍| 41372/93371 [00:02<00:03, 17106.36 example
Adding EOS to train dataset: 47%|▍| 43969/93371 [00:02<00:02, 17165.98 example
Adding EOS to train dataset: 50%|▍| 46530/93371 [00:02<00:02, 17135.24 example
Adding EOS to train dataset: 53%|▌| 49048/93371 [00:02<00:02, 17024.81 example
Adding EOS to train dataset: 54%|▌| 50767/93371 [00:02<00:02, 17058.49 example
Adding EOS to train dataset: 57%|▌| 53299/93371 [00:03<00:02, 16996.92 example
Adding EOS to train dataset: 60%|▌| 55919/93371 [00:03<00:02, 17084.27 example
Adding EOS to train dataset: 62%|▌| 57644/93371 [00:03<00:02, 17120.65 example
Adding EOS to train dataset: 64%|▋| 60191/93371 [00:03<00:01, 17071.37 example
Adding EOS to train dataset: 66%|▋| 61932/93371 [00:03<00:01, 17147.63 example
Adding EOS to train dataset: 69%|▋| 64490/93371 [00:03<00:01, 17112.58 example
Adding EOS to train dataset: 72%|▋| 66885/93371 [00:03<00:01, 16734.16 example
Adding EOS to train dataset: 74%|▋| 69357/93371 [00:04<00:01, 16650.16 example
Adding EOS to train dataset: 77%|▊| 71887/93371 [00:04<00:01, 16715.60 example
Adding EOS to train dataset: 80%|▊| 74379/93371 [00:04<00:01, 16680.73 example
Adding EOS to train dataset: 82%|▊| 76881/93371 [00:04<00:00, 16674.76 example
Adding EOS to train dataset: 85%|▊| 79367/93371 [00:04<00:00, 16640.64 example
Adding EOS to train dataset: 88%|▉| 81876/93371 [00:04<00:00, 16642.56 example
Adding EOS to train dataset: 90%|▉| 84364/93371 [00:04<00:00, 16623.85 example
Adding EOS to train dataset: 93%|▉| 86878/93371 [00:05<00:00, 16607.41 example
Adding EOS to train dataset: 95%|▉| 88543/93371 [00:05<00:00, 16615.83 example
Adding EOS to train dataset: 97%|▉| 91000/93371 [00:05<00:00, 16498.78 example
Adding EOS to train dataset: 99%|▉| 92697/93371 [00:05<00:00, 16605.58 example
Adding EOS to train dataset: 100%|█| 93371/93371 [00:05<00:00, 16582.40 example |
|
Tokenizing train dataset: 0%| | 0/93371 [00:00<?, ? examples/s]
Tokenizing train dataset: 0%| | 219/93371 [00:00<00:42, 2172.98 examples/s]
Tokenizing train dataset: 0%| | 454/93371 [00:00<00:41, 2263.05 examples/s]
Tokenizing train dataset: 1%| | 722/93371 [00:00<00:46, 1984.09 examples/s]
Tokenizing train dataset: 1%| | 930/93371 [00:00<00:46, 2007.89 examples/s]
Tokenizing train dataset: 1%| | 1189/93371 [00:00<00:56, 1629.06 examples/s]
Tokenizing train dataset: 1%| | 1389/93371 [00:00<00:53, 1723.13 examples/s]
Tokenizing train dataset: 2%| | 1592/93371 [00:00<00:50, 1804.60 examples/s]
Tokenizing train dataset: 2%| | 1782/93371 [00:00<00:50, 1826.89 examples/s]
Tokenizing train dataset: 2%| | 1975/93371 [00:01<00:49, 1854.33 examples/s]
Tokenizing train dataset: 2%| | 2229/93371 [00:01<00:50, 1789.54 examples/s]
Tokenizing train dataset: 3%| | 2509/93371 [00:01<00:50, 1811.92 examples/s]
Tokenizing train dataset: 3%| | 2710/93371 [00:01<00:48, 1854.46 examples/s]
Tokenizing train dataset: 3%| | 2973/93371 [00:01<00:49, 1817.47 examples/s]
Tokenizing train dataset: 3%| | 3230/93371 [00:01<00:50, 1779.14 examples/s]
Tokenizing train dataset: 4%| | 3432/93371 [00:01<00:48, 1835.99 examples/s]
Tokenizing train dataset: 4%| | 3626/93371 [00:01<00:48, 1860.32 examples/s]
Tokenizing train dataset: 4%| | 3907/93371 [00:02<00:48, 1861.02 examples/s]
Tokenizing train dataset: 4%| | 4143/93371 [00:02<00:50, 1763.10 examples/s]
Tokenizing train dataset: 5%| | 4402/93371 [00:02<00:50, 1746.35 examples/s]
Tokenizing train dataset: 5%| | 4605/93371 [00:02<00:49, 1811.05 examples/s]
Tokenizing train dataset: 5%| | 4869/93371 [00:02<00:49, 1786.82 examples/s]
Tokenizing train dataset: 5%| | 5095/93371 [00:02<00:52, 1689.66 examples/s]
Tokenizing train dataset: 6%| | 5307/93371 [00:02<00:49, 1789.55 examples/s]
Tokenizing train dataset: 6%| | 5505/93371 [00:03<00:47, 1835.03 examples/s]
Tokenizing train dataset: 6%| | 5700/93371 [00:03<00:47, 1848.45 examples/s]
Tokenizing train dataset: 6%|▏ | 5900/93371 [00:03<00:46, 1887.18 examples/s]
Tokenizing train dataset: 7%|▏ | 6181/93371 [00:03<00:48, 1798.90 examples/s]
Tokenizing train dataset: 7%|▏ | 6366/93371 [00:03<00:48, 1808.43 examples/s]
Tokenizing train dataset: 7%|▏ | 6558/93371 [00:03<00:47, 1831.18 examples/s]
Tokenizing train dataset: 7%|▏ | 6744/93371 [00:03<00:47, 1838.89 examples/s]
Tokenizing train dataset: 7%|▏ | 6938/93371 [00:03<00:46, 1865.62 examples/s]
Tokenizing train dataset: 8%|▏ | 7199/93371 [00:03<00:47, 1797.53 examples/s]
Tokenizing train dataset: 8%|▏ | 7388/93371 [00:04<00:47, 1819.45 examples/s]
Tokenizing train dataset: 8%|▏ | 7581/93371 [00:04<00:46, 1844.32 examples/s]
Tokenizing train dataset: 8%|▏ | 7775/93371 [00:04<00:46, 1860.43 examples/s]
Tokenizing train dataset: 9%|▏ | 8023/93371 [00:04<00:47, 1778.86 examples/s]
Tokenizing train dataset: 9%|▏ | 8212/93371 [00:04<00:47, 1805.39 examples/s]
Tokenizing train dataset: 9%|▏ | 8399/93371 [00:04<00:46, 1822.56 examples/s]
Tokenizing train dataset: 9%|▏ | 8602/93371 [00:04<00:45, 1878.62 examples/s]
Tokenizing train dataset: 10%|▏ | 8879/93371 [00:04<00:45, 1863.66 examples/s]
Tokenizing train dataset: 10%|▏ | 9128/93371 [00:05<00:47, 1789.94 examples/s]
Tokenizing train dataset: 10%|▏ | 9332/93371 [00:05<00:45, 1850.96 examples/s]
Tokenizing train dataset: 10%|▏ | 9605/93371 [00:05<00:45, 1832.64 examples/s]
Tokenizing train dataset: 11%|▏ | 9809/93371 [00:05<00:44, 1883.40 examples/s]
Tokenizing train dataset: 11%| | 10000/93371 [00:05<00:45, 1831.34 examples/s]
Tokenizing train dataset: 11%| | 10186/93371 [00:05<00:45, 1834.42 examples/s]
Tokenizing train dataset: 11%| | 10394/93371 [00:05<00:43, 1899.48 examples/s]
Tokenizing train dataset: 11%| | 10677/93371 [00:05<00:43, 1892.82 examples/s]
Tokenizing train dataset: 12%| | 10879/93371 [00:05<00:42, 1923.17 examples/s]
Tokenizing train dataset: 12%| | 11121/93371 [00:06<00:45, 1809.76 examples/s]
Tokenizing train dataset: 12%| | 11312/93371 [00:06<00:44, 1829.45 examples/s]
Tokenizing train dataset: 12%| | 11507/93371 [00:06<00:44, 1856.05 examples/s]
Tokenizing train dataset: 13%|▏| 11711/93371 [00:06<00:42, 1905.83 examples/s]
Tokenizing train dataset: 13%|▏| 11913/93371 [00:06<00:42, 1933.66 examples/s]
Tokenizing train dataset: 13%|▏| 12201/93371 [00:06<00:44, 1838.61 examples/s]
Tokenizing train dataset: 13%|▏| 12410/93371 [00:06<00:42, 1898.43 examples/s]
Tokenizing train dataset: 14%|▏| 12699/93371 [00:06<00:42, 1906.32 examples/s]
Tokenizing train dataset: 14%|▏| 12910/93371 [00:07<00:41, 1947.48 examples/s]
Tokenizing train dataset: 14%|▏| 13184/93371 [00:07<00:43, 1839.50 examples/s]
Tokenizing train dataset: 14%|▏| 13453/93371 [00:07<00:44, 1812.50 examples/s]
Tokenizing train dataset: 15%|▏| 13639/93371 [00:07<00:43, 1818.91 examples/s]
Tokenizing train dataset: 15%|▏| 13831/93371 [00:07<00:43, 1835.43 examples/s]
Tokenizing train dataset: 15%|▏| 14084/93371 [00:07<00:45, 1730.35 examples/s]
Tokenizing train dataset: 15%|▏| 14284/93371 [00:07<00:44, 1793.25 examples/s]
Tokenizing train dataset: 16%|▏| 14484/93371 [00:07<00:42, 1845.86 examples/s]
Tokenizing train dataset: 16%|▏| 14692/93371 [00:07<00:41, 1904.05 examples/s]
Tokenizing train dataset: 16%|▏| 14894/93371 [00:08<00:40, 1932.72 examples/s]
Tokenizing train dataset: 16%|▏| 15139/93371 [00:08<00:43, 1818.03 examples/s]
Tokenizing train dataset: 16%|▏| 15346/93371 [00:08<00:41, 1875.97 examples/s]
Tokenizing train dataset: 17%|▏| 15554/93371 [00:08<00:40, 1928.45 examples/s]
Tokenizing train dataset: 17%|▏| 15843/93371 [00:08<00:40, 1923.82 examples/s]
Tokenizing train dataset: 17%|▏| 16093/93371 [00:08<00:42, 1825.09 examples/s]
Tokenizing train dataset: 17%|▏| 16293/93371 [00:08<00:41, 1865.67 examples/s]
Tokenizing train dataset: 18%|▏| 16499/93371 [00:08<00:40, 1915.48 examples/s]
Tokenizing train dataset: 18%|▏| 16776/93371 [00:09<00:40, 1887.22 examples/s]
Tokenizing train dataset: 18%|▏| 17024/93371 [00:09<00:42, 1805.55 examples/s]
Tokenizing train dataset: 18%|▏| 17209/93371 [00:09<00:41, 1814.25 examples/s]
Tokenizing train dataset: 19%|▏| 17409/93371 [00:09<00:40, 1858.07 examples/s]
Tokenizing train dataset: 19%|▏| 17691/93371 [00:09<00:40, 1858.06 examples/s]
Tokenizing train dataset: 19%|▏| 17879/93371 [00:09<00:40, 1859.85 examples/s]
Tokenizing train dataset: 19%|▏| 18130/93371 [00:09<00:41, 1792.50 examples/s]
Tokenizing train dataset: 20%|▏| 18319/93371 [00:09<00:41, 1812.26 examples/s]
Tokenizing train dataset: 20%|▏| 18508/93371 [00:10<00:40, 1830.64 examples/s]
Tokenizing train dataset: 20%|▏| 18720/93371 [00:10<00:39, 1903.26 examples/s]
Tokenizing train dataset: 20%|▏| 18999/93371 [00:10<00:39, 1879.71 examples/s]
Tokenizing train dataset: 21%|▏| 19267/93371 [00:10<00:40, 1843.33 examples/s]
Tokenizing train dataset: 21%|▏| 19463/93371 [00:10<00:39, 1869.33 examples/s]
Tokenizing train dataset: 21%|▏| 19746/93371 [00:10<00:39, 1870.20 examples/s]
Tokenizing train dataset: 21%|▏| 19941/93371 [00:10<00:38, 1888.40 examples/s]
Tokenizing train dataset: 22%|▏| 20196/93371 [00:10<00:40, 1813.74 examples/s]
Tokenizing train dataset: 22%|▏| 20388/93371 [00:11<00:39, 1837.33 examples/s]
Tokenizing train dataset: 22%|▏| 20584/93371 [00:11<00:39, 1862.01 examples/s]
Tokenizing train dataset: 22%|▏| 20790/93371 [00:11<00:38, 1909.33 examples/s]
Tokenizing train dataset: 22%|▏| 20987/93371 [00:11<00:37, 1915.61 examples/s]
Tokenizing train dataset: 23%|▏| 21221/93371 [00:11<00:40, 1780.12 examples/s]
Tokenizing train dataset: 23%|▏| 21422/93371 [00:11<00:39, 1839.02 examples/s]
Tokenizing train dataset: 23%|▏| 21618/93371 [00:11<00:38, 1870.17 examples/s]
Tokenizing train dataset: 23%|▏| 21817/93371 [00:11<00:37, 1900.01 examples/s]
Tokenizing train dataset: 24%|▏| 22048/93371 [00:11<00:40, 1761.35 examples/s]
Tokenizing train dataset: 24%|▏| 22332/93371 [00:12<00:39, 1802.07 examples/s]
Tokenizing train dataset: 24%|▏| 22597/93371 [00:12<00:39, 1785.47 examples/s]
Tokenizing train dataset: 24%|▏| 22796/93371 [00:12<00:39, 1790.45 examples/s]
Tokenizing train dataset: 25%|▏| 23000/93371 [00:12<00:39, 1759.53 examples/s]
Tokenizing train dataset: 25%|▏| 23190/93371 [00:12<00:39, 1793.46 examples/s]
Tokenizing train dataset: 25%|▎| 23378/93371 [00:12<00:38, 1815.15 examples/s]
Tokenizing train dataset: 25%|▎| 23564/93371 [00:12<00:38, 1825.89 examples/s]
Tokenizing train dataset: 25%|▎| 23757/93371 [00:12<00:37, 1848.61 examples/s]
Tokenizing train dataset: 26%|▎| 23965/93371 [00:13<00:36, 1906.25 examples/s]
Tokenizing train dataset: 26%|▎| 24194/93371 [00:13<00:39, 1745.34 examples/s]
Tokenizing train dataset: 26%|▎| 24392/93371 [00:13<00:38, 1800.41 examples/s]
Tokenizing train dataset: 26%|▎| 24589/93371 [00:13<00:37, 1841.32 examples/s]
Tokenizing train dataset: 27%|▎| 24879/93371 [00:13<00:36, 1872.90 examples/s]
Tokenizing train dataset: 27%|▎| 25110/93371 [00:13<00:38, 1753.68 examples/s]
Tokenizing train dataset: 27%|▎| 25305/93371 [00:13<00:37, 1799.10 examples/s]
Tokenizing train dataset: 27%|▎| 25493/93371 [00:13<00:37, 1815.72 examples/s]
Tokenizing train dataset: 28%|▎| 25754/93371 [00:14<00:37, 1782.55 examples/s]
Tokenizing train dataset: 28%|▎| 25957/93371 [00:14<00:36, 1841.55 examples/s]
Tokenizing train dataset: 28%|▎| 26215/93371 [00:14<00:37, 1796.19 examples/s]
Tokenizing train dataset: 28%|▎| 26398/93371 [00:14<00:37, 1804.32 examples/s]
Tokenizing train dataset: 28%|▎| 26596/93371 [00:14<00:36, 1847.20 examples/s]
Tokenizing train dataset: 29%|▎| 26789/93371 [00:14<00:35, 1866.37 examples/s]
Tokenizing train dataset: 29%|▎| 26984/93371 [00:14<00:35, 1888.13 examples/s]
Tokenizing train dataset: 29%|▎| 27236/93371 [00:14<00:36, 1801.69 examples/s]
Tokenizing train dataset: 29%|▎| 27426/93371 [00:14<00:36, 1823.41 examples/s]
Tokenizing train dataset: 30%|▎| 27619/93371 [00:15<00:35, 1849.62 examples/s]
Tokenizing train dataset: 30%|▎| 27812/93371 [00:15<00:35, 1869.27 examples/s]
Tokenizing train dataset: 30%|▎| 28105/93371 [00:15<00:35, 1833.69 examples/s]
Tokenizing train dataset: 30%|▎| 28299/93371 [00:15<00:35, 1856.19 examples/s]
Tokenizing train dataset: 31%|▎| 28496/93371 [00:15<00:34, 1884.77 examples/s]
Tokenizing train dataset: 31%|▎| 28693/93371 [00:15<00:33, 1905.34 examples/s]
Tokenizing train dataset: 31%|▎| 28974/93371 [00:15<00:34, 1885.77 examples/s]
Tokenizing train dataset: 31%|▎| 29223/93371 [00:15<00:35, 1803.55 examples/s]
Tokenizing train dataset: 32%|▎| 29446/93371 [00:16<01:03, 1004.74 examples/s]
Tokenizing train dataset: 32%|▎| 29650/93371 [00:16<00:54, 1164.81 examples/s]
Tokenizing train dataset: 32%|▎| 29848/93371 [00:16<00:48, 1312.25 examples/s]
Tokenizing train dataset: 32%|▎| 30095/93371 [00:16<00:45, 1403.90 examples/s]
Tokenizing train dataset: 32%|▎| 30286/93371 [00:16<00:41, 1509.39 examples/s]
Tokenizing train dataset: 33%|▎| 30474/93371 [00:16<00:39, 1592.06 examples/s]
Tokenizing train dataset: 33%|▎| 30740/93371 [00:17<00:38, 1646.88 examples/s]
Tokenizing train dataset: 33%|▎| 30937/93371 [00:17<00:36, 1721.73 examples/s]
Tokenizing train dataset: 33%|▎| 31193/93371 [00:17<00:36, 1680.99 examples/s]
Tokenizing train dataset: 34%|▎| 31391/93371 [00:17<00:35, 1749.51 examples/s]
Tokenizing train dataset: 34%|▎| 31601/93371 [00:17<00:33, 1836.41 examples/s]
Tokenizing train dataset: 34%|▎| 31797/93371 [00:17<00:33, 1865.72 examples/s]
Tokenizing train dataset: 34%|▎| 32000/93371 [00:17<00:33, 1811.40 examples/s]
Tokenizing train dataset: 34%|▎| 32202/93371 [00:17<00:32, 1860.77 examples/s]
Tokenizing train dataset: 35%|▎| 32401/93371 [00:17<00:32, 1895.22 examples/s]
Tokenizing train dataset: 35%|▎| 32601/93371 [00:18<00:31, 1922.61 examples/s]
Tokenizing train dataset: 35%|▎| 32798/93371 [00:18<00:31, 1926.82 examples/s]
Tokenizing train dataset: 35%|▎| 33035/93371 [00:18<00:33, 1791.52 examples/s]
Tokenizing train dataset: 36%|▎| 33237/93371 [00:18<00:32, 1849.34 examples/s]
Tokenizing train dataset: 36%|▎| 33431/93371 [00:18<00:32, 1871.44 examples/s]
Tokenizing train dataset: 36%|▎| 33627/93371 [00:18<00:31, 1892.03 examples/s]
Tokenizing train dataset: 36%|▎| 33820/93371 [00:18<00:31, 1900.89 examples/s]
Tokenizing train dataset: 37%|▎| 34104/93371 [00:18<00:31, 1863.24 examples/s]
Tokenizing train dataset: 37%|▎| 34302/93371 [00:18<00:31, 1892.39 examples/s]
Tokenizing train dataset: 37%|▎| 34501/93371 [00:19<00:30, 1913.66 examples/s]
Tokenizing train dataset: 37%|▎| 34786/93371 [00:19<00:30, 1906.22 examples/s]
Tokenizing train dataset: 37%|▎| 34984/93371 [00:19<00:30, 1920.35 examples/s]
Tokenizing train dataset: 38%|▍| 35240/93371 [00:19<00:31, 1836.20 examples/s]
Tokenizing train dataset: 38%|▍| 35519/93371 [00:19<00:31, 1842.29 examples/s]
Tokenizing train dataset: 38%|▍| 35810/93371 [00:19<00:30, 1872.44 examples/s]
Tokenizing train dataset: 39%|▍| 36000/93371 [00:19<00:32, 1791.15 examples/s]
Tokenizing train dataset: 39%|▍| 36200/93371 [00:19<00:31, 1839.86 examples/s]
Tokenizing train dataset: 39%|▍| 36411/93371 [00:20<00:29, 1905.86 examples/s]
Tokenizing train dataset: 39%|▍| 36606/93371 [00:20<00:29, 1913.91 examples/s]
Tokenizing train dataset: 40%|▍| 36882/93371 [00:20<00:29, 1884.24 examples/s]
Tokenizing train dataset: 40%|▍| 37116/93371 [00:20<00:31, 1770.77 examples/s]
Tokenizing train dataset: 40%|▍| 37305/93371 [00:20<00:31, 1798.87 examples/s]
Tokenizing train dataset: 40%|▍| 37520/93371 [00:20<00:29, 1888.92 examples/s]
Tokenizing train dataset: 40%|▍| 37809/93371 [00:20<00:29, 1893.01 examples/s]
Tokenizing train dataset: 41%|▍| 38046/93371 [00:20<00:31, 1781.38 examples/s]
Tokenizing train dataset: 41%|▍| 38253/93371 [00:21<00:29, 1850.33 examples/s]
Tokenizing train dataset: 41%|▍| 38544/93371 [00:21<00:29, 1869.42 examples/s]
Tokenizing train dataset: 41%|▍| 38746/93371 [00:21<00:28, 1904.72 examples/s]
Tokenizing train dataset: 42%|▍| 39000/93371 [00:21<00:30, 1789.23 examples/s]
Tokenizing train dataset: 42%|▍| 39190/93371 [00:21<00:29, 1807.73 examples/s]
Tokenizing train dataset: 42%|▍| 39383/93371 [00:21<00:29, 1836.15 examples/s]
Tokenizing train dataset: 42%|▍| 39576/93371 [00:21<00:28, 1859.19 examples/s]
Tokenizing train dataset: 43%|▍| 39768/93371 [00:21<00:28, 1871.69 examples/s]
Tokenizing train dataset: 43%|▍| 39975/93371 [00:22<00:27, 1926.79 examples/s]
Tokenizing train dataset: 43%|▍| 40226/93371 [00:22<00:29, 1825.24 examples/s]
Tokenizing train dataset: 43%|▍| 40422/93371 [00:22<00:28, 1852.12 examples/s]
Tokenizing train dataset: 44%|▍| 40618/93371 [00:22<00:28, 1878.26 examples/s]
Tokenizing train dataset: 44%|▍| 40813/93371 [00:22<00:27, 1896.60 examples/s]
Tokenizing train dataset: 44%|▍| 41087/93371 [00:22<00:29, 1774.60 examples/s]
Tokenizing train dataset: 44%|▍| 41280/93371 [00:22<00:28, 1809.06 examples/s]
Tokenizing train dataset: 44%|▍| 41484/93371 [00:22<00:27, 1861.42 examples/s]
Tokenizing train dataset: 45%|▍| 41695/93371 [00:22<00:26, 1925.20 examples/s]
Tokenizing train dataset: 45%|▍| 41995/93371 [00:23<00:26, 1946.70 examples/s]
Tokenizing train dataset: 45%|▍| 42246/93371 [00:23<00:27, 1849.10 examples/s]
Tokenizing train dataset: 45%|▍| 42457/93371 [00:23<00:26, 1908.83 examples/s]
Tokenizing train dataset: 46%|▍| 42671/93371 [00:23<00:25, 1967.32 examples/s]
Tokenizing train dataset: 46%|▍| 42951/93371 [00:23<00:26, 1928.06 examples/s]
Tokenizing train dataset: 46%|▍| 43212/93371 [00:23<00:27, 1857.13 examples/s]
Tokenizing train dataset: 46%|▍| 43413/93371 [00:23<00:26, 1889.58 examples/s]
Tokenizing train dataset: 47%|▍| 43702/93371 [00:24<00:26, 1895.35 examples/s]
Tokenizing train dataset: 47%|▍| 43977/93371 [00:24<00:26, 1872.76 examples/s]
Tokenizing train dataset: 47%|▍| 44220/93371 [00:24<00:27, 1785.31 examples/s]
Tokenizing train dataset: 48%|▍| 44425/93371 [00:24<00:26, 1836.57 examples/s]
Tokenizing train dataset: 48%|▍| 44624/93371 [00:24<00:26, 1873.99 examples/s]
Tokenizing train dataset: 48%|▍| 44831/93371 [00:24<00:25, 1924.53 examples/s]
Tokenizing train dataset: 48%|▍| 45094/93371 [00:24<00:26, 1828.21 examples/s]
Tokenizing train dataset: 49%|▍| 45290/93371 [00:24<00:25, 1858.58 examples/s]
Tokenizing train dataset: 49%|▍| 45571/93371 [00:25<00:25, 1861.44 examples/s]
Tokenizing train dataset: 49%|▍| 45845/93371 [00:25<00:25, 1842.10 examples/s]
Tokenizing train dataset: 49%|▍| 46096/93371 [00:25<00:26, 1766.74 examples/s]
Tokenizing train dataset: 50%|▍| 46295/93371 [00:25<00:26, 1809.86 examples/s]
Tokenizing train dataset: 50%|▍| 46561/93371 [00:25<00:26, 1795.59 examples/s]
Tokenizing train dataset: 50%|▌| 46765/93371 [00:25<00:25, 1850.96 examples/s]
Tokenizing train dataset: 50%|▌| 46961/93371 [00:25<00:24, 1874.82 examples/s]
Tokenizing train dataset: 51%|▌| 47197/93371 [00:25<00:26, 1760.71 examples/s]
Tokenizing train dataset: 51%|▌| 47381/93371 [00:26<00:25, 1777.19 examples/s]
Tokenizing train dataset: 51%|▌| 47576/93371 [00:26<00:25, 1809.11 examples/s]
Tokenizing train dataset: 51%|▌| 47778/93371 [00:26<00:24, 1863.62 examples/s]
Tokenizing train dataset: 51%|▌| 47975/93371 [00:26<00:24, 1891.20 examples/s]
Tokenizing train dataset: 52%|▌| 48241/93371 [00:26<00:24, 1843.78 examples/s]
Tokenizing train dataset: 52%|▌| 48524/93371 [00:26<00:24, 1852.19 examples/s]
Tokenizing train dataset: 52%|▌| 48718/93371 [00:26<00:23, 1870.41 examples/s]
Tokenizing train dataset: 52%|▌| 49000/93371 [00:26<00:24, 1805.61 examples/s]
Tokenizing train dataset: 53%|▌| 49183/93371 [00:27<00:24, 1809.28 examples/s]
Tokenizing train dataset: 53%|▌| 49370/93371 [00:27<00:24, 1824.03 examples/s]
Tokenizing train dataset: 53%|▌| 49556/93371 [00:27<00:23, 1831.58 examples/s]
Tokenizing train dataset: 53%|▌| 49755/93371 [00:27<00:23, 1863.81 examples/s]
Tokenizing train dataset: 54%|▌| 49959/93371 [00:27<00:22, 1909.59 examples/s]
Tokenizing train dataset: 54%|▌| 50200/93371 [00:27<00:24, 1791.99 examples/s]
Tokenizing train dataset: 54%|▌| 50384/93371 [00:27<00:23, 1800.76 examples/s]
Tokenizing train dataset: 54%|▌| 50567/93371 [00:27<00:23, 1807.85 examples/s]
Tokenizing train dataset: 54%|▌| 50855/93371 [00:27<00:23, 1839.63 examples/s]
Tokenizing train dataset: 55%|▌| 51110/93371 [00:28<00:23, 1783.74 examples/s]
Tokenizing train dataset: 55%|▌| 51318/93371 [00:28<00:22, 1855.54 examples/s]
Tokenizing train dataset: 55%|▌| 51506/93371 [00:28<00:22, 1851.72 examples/s]
Tokenizing train dataset: 55%|▌| 51704/93371 [00:28<00:22, 1880.87 examples/s]
Tokenizing train dataset: 56%|▌| 51899/93371 [00:28<00:21, 1899.21 examples/s]
Tokenizing train dataset: 56%|▌| 52142/93371 [00:28<00:23, 1785.08 examples/s]
Tokenizing train dataset: 56%|▌| 52346/93371 [00:28<00:22, 1847.94 examples/s]
Tokenizing train dataset: 56%|▌| 52630/93371 [00:28<00:21, 1860.23 examples/s]
Tokenizing train dataset: 57%|▌| 52818/93371 [00:28<00:21, 1864.51 examples/s]
Tokenizing train dataset: 57%|▌| 53096/93371 [00:29<00:22, 1800.85 examples/s]
Tokenizing train dataset: 57%|▌| 53353/93371 [00:29<00:22, 1767.29 examples/s]
Tokenizing train dataset: 57%|▌| 53554/93371 [00:29<00:21, 1820.77 examples/s]
Tokenizing train dataset: 58%|▌| 53755/93371 [00:29<00:21, 1865.67 examples/s]
Tokenizing train dataset: 58%|▌| 53953/93371 [00:29<00:20, 1894.52 examples/s]
Tokenizing train dataset: 58%|▌| 54190/93371 [00:29<00:22, 1776.29 examples/s]
Tokenizing train dataset: 58%|▌| 54460/93371 [00:29<00:21, 1781.75 examples/s]
Tokenizing train dataset: 59%|▌| 54658/93371 [00:29<00:21, 1829.12 examples/s]
Tokenizing train dataset: 59%|▌| 54853/93371 [00:30<00:20, 1857.04 examples/s]
Tokenizing train dataset: 59%|▌| 55096/93371 [00:30<00:21, 1766.99 examples/s]
Tokenizing train dataset: 59%|▌| 55298/93371 [00:30<00:20, 1830.38 examples/s]
Tokenizing train dataset: 59%|▌| 55492/93371 [00:30<00:20, 1858.03 examples/s]
Tokenizing train dataset: 60%|▌| 55682/93371 [00:30<00:20, 1866.98 examples/s]
Tokenizing train dataset: 60%|▌| 55978/93371 [00:30<00:19, 1895.92 examples/s]
Tokenizing train dataset: 60%|▌| 56228/93371 [00:30<00:20, 1806.66 examples/s]
Tokenizing train dataset: 60%|▌| 56434/93371 [00:30<00:19, 1865.14 examples/s]
Tokenizing train dataset: 61%|▌| 56641/93371 [00:31<00:19, 1913.37 examples/s]
Tokenizing train dataset: 61%|▌| 56838/93371 [00:31<00:19, 1920.98 examples/s]
Tokenizing train dataset: 61%|▌| 57076/93371 [00:31<00:20, 1775.16 examples/s]
Tokenizing train dataset: 61%|▌| 57265/93371 [00:31<00:20, 1802.91 examples/s]
Tokenizing train dataset: 62%|▌| 57476/93371 [00:31<00:19, 1881.24 examples/s]
Tokenizing train dataset: 62%|▌| 57668/93371 [00:31<00:18, 1887.43 examples/s]
Tokenizing train dataset: 62%|▌| 57953/93371 [00:31<00:18, 1887.50 examples/s]
Tokenizing train dataset: 62%|▌| 58208/93371 [00:31<00:19, 1817.34 examples/s]
Tokenizing train dataset: 63%|▋| 58396/93371 [00:32<00:19, 1829.84 examples/s]
Tokenizing train dataset: 63%|▋| 58599/93371 [00:32<00:18, 1880.11 examples/s]
Tokenizing train dataset: 63%|▋| 58795/93371 [00:32<00:18, 1898.70 examples/s]
Tokenizing train dataset: 63%|▋| 58989/93371 [00:32<00:18, 1905.33 examples/s]
Tokenizing train dataset: 63%|▋| 59244/93371 [00:32<00:18, 1819.19 examples/s]
Tokenizing train dataset: 64%|▋| 59451/93371 [00:32<00:18, 1878.00 examples/s]
Tokenizing train dataset: 64%|▋| 59733/93371 [00:32<00:17, 1873.54 examples/s]
Tokenizing train dataset: 64%|▋| 59942/93371 [00:32<00:17, 1921.59 examples/s]
Tokenizing train dataset: 64%|▋| 60210/93371 [00:32<00:17, 1871.52 examples/s]
Tokenizing train dataset: 65%|▋| 60415/93371 [00:33<00:17, 1913.16 examples/s]
Tokenizing train dataset: 65%|▋| 60700/93371 [00:33<00:17, 1900.03 examples/s]
Tokenizing train dataset: 65%|▋| 60990/93371 [00:33<00:17, 1904.14 examples/s]
Tokenizing train dataset: 66%|▋| 61224/93371 [00:33<00:18, 1784.14 examples/s]
Tokenizing train dataset: 66%|▋| 61420/93371 [00:33<00:17, 1821.53 examples/s]
Tokenizing train dataset: 66%|▋| 61611/93371 [00:33<00:17, 1836.92 examples/s]
Tokenizing train dataset: 66%|▋| 61875/93371 [00:33<00:17, 1805.40 examples/s]
Tokenizing train dataset: 67%|▋| 62120/93371 [00:34<00:17, 1741.78 examples/s]
Tokenizing train dataset: 67%|▋| 62316/93371 [00:34<00:17, 1788.95 examples/s]
Tokenizing train dataset: 67%|▋| 62510/93371 [00:34<00:16, 1822.22 examples/s]
Tokenizing train dataset: 67%|▋| 62695/93371 [00:34<00:16, 1828.93 examples/s]
Tokenizing train dataset: 67%|▋| 62970/93371 [00:34<00:16, 1816.28 examples/s]
Tokenizing train dataset: 68%|▋| 63226/93371 [00:34<00:17, 1769.11 examples/s]
Tokenizing train dataset: 68%|▋| 63428/93371 [00:34<00:16, 1827.30 examples/s]
Tokenizing train dataset: 68%|▋| 63616/93371 [00:34<00:16, 1836.91 examples/s]
Tokenizing train dataset: 68%|▋| 63805/93371 [00:34<00:15, 1850.24 examples/s]
Tokenizing train dataset: 69%|▋| 64000/93371 [00:35<00:16, 1798.12 examples/s]
Tokenizing train dataset: 69%|▋| 64189/93371 [00:35<00:16, 1819.54 examples/s]
Tokenizing train dataset: 69%|▋| 64393/93371 [00:35<00:15, 1878.02 examples/s]
Tokenizing train dataset: 69%|▋| 64593/93371 [00:35<00:15, 1912.13 examples/s]
Tokenizing train dataset: 69%|▋| 64788/93371 [00:35<00:14, 1919.25 examples/s]
Tokenizing train dataset: 70%|▋| 64989/93371 [00:35<00:14, 1943.66 examples/s]
Tokenizing train dataset: 70%|▋| 65241/93371 [00:35<00:15, 1832.75 examples/s]
Tokenizing train dataset: 70%|▋| 65437/93371 [00:35<00:14, 1865.26 examples/s]
Tokenizing train dataset: 70%|▋| 65626/93371 [00:35<00:14, 1865.48 examples/s]
Tokenizing train dataset: 71%|▋| 65915/93371 [00:36<00:14, 1883.32 examples/s]
Tokenizing train dataset: 71%|▋| 66194/93371 [00:36<00:14, 1833.35 examples/s]
Tokenizing train dataset: 71%|▋| 66390/93371 [00:36<00:14, 1859.94 examples/s]
Tokenizing train dataset: 71%|▋| 66585/93371 [00:36<00:14, 1880.98 examples/s]
Tokenizing train dataset: 72%|▋| 66777/93371 [00:36<00:14, 1886.95 examples/s]
Tokenizing train dataset: 72%|▋| 66986/93371 [00:36<00:13, 1939.83 examples/s]
Tokenizing train dataset: 72%|▋| 67247/93371 [00:36<00:14, 1862.12 examples/s]
Tokenizing train dataset: 72%|▋| 67444/93371 [00:36<00:13, 1887.82 examples/s]
Tokenizing train dataset: 73%|▋| 67725/93371 [00:37<00:13, 1879.08 examples/s]
Tokenizing train dataset: 73%|▋| 67918/93371 [00:37<00:13, 1887.74 examples/s]
Tokenizing train dataset: 73%|▋| 68203/93371 [00:37<00:13, 1843.60 examples/s]
Tokenizing train dataset: 73%|▋| 68390/93371 [00:37<00:13, 1848.87 examples/s]
Tokenizing train dataset: 73%|▋| 68578/93371 [00:37<00:13, 1856.05 examples/s]
Tokenizing train dataset: 74%|▋| 68855/93371 [00:37<00:13, 1851.15 examples/s]
Tokenizing train dataset: 74%|▋| 69108/93371 [00:37<00:13, 1792.85 examples/s]
Tokenizing train dataset: 74%|▋| 69295/93371 [00:37<00:13, 1808.17 examples/s]
Tokenizing train dataset: 74%|▋| 69499/93371 [00:37<00:12, 1866.20 examples/s]
Tokenizing train dataset: 75%|▋| 69691/93371 [00:38<00:12, 1874.35 examples/s]
Tokenizing train dataset: 75%|▋| 69885/93371 [00:38<00:12, 1889.70 examples/s]
Tokenizing train dataset: 75%|▊| 70139/93371 [00:38<00:12, 1813.69 examples/s]
Tokenizing train dataset: 75%|▊| 70334/93371 [00:38<00:12, 1839.72 examples/s]
Tokenizing train dataset: 76%|▊| 70541/93371 [00:38<00:12, 1901.42 examples/s]
Tokenizing train dataset: 76%|▊| 70739/93371 [00:38<00:11, 1919.94 examples/s]
Tokenizing train dataset: 76%|▊| 71000/93371 [00:38<00:12, 1827.31 examples/s]
Tokenizing train dataset: 76%|▊| 71202/93371 [00:38<00:11, 1864.25 examples/s]
Tokenizing train dataset: 77%|▊| 71468/93371 [00:39<00:11, 1830.65 examples/s]
Tokenizing train dataset: 77%|▊| 71656/93371 [00:39<00:11, 1835.12 examples/s]
Tokenizing train dataset: 77%|▊| 71856/93371 [00:39<00:11, 1874.32 examples/s]
Tokenizing train dataset: 77%|▊| 72115/93371 [00:39<00:11, 1817.19 examples/s]
Tokenizing train dataset: 78%|▊| 72392/93371 [00:39<00:11, 1824.50 examples/s]
Tokenizing train dataset: 78%|▊| 72583/93371 [00:39<00:11, 1840.39 examples/s]
Tokenizing train dataset: 78%|▊| 72769/93371 [00:39<00:11, 1842.56 examples/s]
Tokenizing train dataset: 78%|▊| 72983/93371 [00:39<00:10, 1918.34 examples/s]
Tokenizing train dataset: 78%|▊| 73236/93371 [00:40<00:11, 1829.78 examples/s]
Tokenizing train dataset: 79%|▊| 73431/93371 [00:40<00:10, 1859.53 examples/s]
Tokenizing train dataset: 79%|▊| 73710/93371 [00:40<00:10, 1854.40 examples/s]
Tokenizing train dataset: 79%|▊| 73994/93371 [00:40<00:10, 1864.66 examples/s]
Tokenizing train dataset: 80%|▊| 74239/93371 [00:40<00:10, 1784.78 examples/s]
Tokenizing train dataset: 80%|▊| 74436/93371 [00:40<00:10, 1826.27 examples/s]
Tokenizing train dataset: 80%|▊| 74704/93371 [00:40<00:10, 1811.93 examples/s]
Tokenizing train dataset: 80%|▊| 74901/93371 [00:40<00:10, 1846.98 examples/s]
Tokenizing train dataset: 80%|▊| 75140/93371 [00:41<00:10, 1752.21 examples/s]
Tokenizing train dataset: 81%|▊| 75337/93371 [00:41<00:10, 1802.63 examples/s]
Tokenizing train dataset: 81%|▊| 75538/93371 [00:41<00:09, 1854.86 examples/s]
Tokenizing train dataset: 81%|▊| 75738/93371 [00:41<00:09, 1892.39 examples/s]
Tokenizing train dataset: 81%|▊| 75935/93371 [00:41<00:09, 1912.16 examples/s]
Tokenizing train dataset: 82%|▊| 76197/93371 [00:41<00:09, 1824.36 examples/s]
Tokenizing train dataset: 82%|▊| 76386/93371 [00:41<00:09, 1836.92 examples/s]
Tokenizing train dataset: 82%|▊| 76594/93371 [00:41<00:08, 1900.36 examples/s]
Tokenizing train dataset: 82%|▊| 76792/93371 [00:41<00:08, 1917.94 examples/s]
Tokenizing train dataset: 82%|▊| 76988/93371 [00:42<00:08, 1924.56 examples/s]
Tokenizing train dataset: 83%|▊| 77218/93371 [00:42<00:09, 1774.34 examples/s]
Tokenizing train dataset: 83%|▊| 77418/93371 [00:42<00:08, 1831.82 examples/s]
Tokenizing train dataset: 83%|▊| 77614/93371 [00:42<00:08, 1862.37 examples/s]
Tokenizing train dataset: 83%|▊| 77809/93371 [00:42<00:08, 1883.04 examples/s]
Tokenizing train dataset: 84%|▊| 78053/93371 [00:42<00:08, 1782.33 examples/s]
Tokenizing train dataset: 84%|▊| 78258/93371 [00:42<00:08, 1847.18 examples/s]
Tokenizing train dataset: 84%|▊| 78527/93371 [00:42<00:08, 1827.01 examples/s]
Tokenizing train dataset: 84%|▊| 78793/93371 [00:43<00:08, 1803.34 examples/s]
Tokenizing train dataset: 85%|▊| 79000/93371 [00:43<00:08, 1770.57 examples/s]
Tokenizing train dataset: 85%|▊| 79194/93371 [00:43<00:07, 1808.03 examples/s]
Tokenizing train dataset: 85%|▊| 79385/93371 [00:43<00:07, 1831.06 examples/s]
Tokenizing train dataset: 85%|▊| 79579/93371 [00:43<00:07, 1858.91 examples/s]
Tokenizing train dataset: 85%|▊| 79783/93371 [00:43<00:07, 1908.78 examples/s]
Tokenizing train dataset: 86%|▊| 80033/93371 [00:43<00:07, 1807.47 examples/s]
Tokenizing train dataset: 86%|▊| 80231/93371 [00:43<00:07, 1846.44 examples/s]
Tokenizing train dataset: 86%|▊| 80519/93371 [00:43<00:06, 1869.93 examples/s]
Tokenizing train dataset: 86%|▊| 80730/93371 [00:44<00:06, 1927.49 examples/s]
Tokenizing train dataset: 87%|▊| 81000/93371 [00:44<00:06, 1825.44 examples/s]
Tokenizing train dataset: 87%|▊| 81187/93371 [00:44<00:06, 1833.31 examples/s]
Tokenizing train dataset: 87%|▊| 81391/93371 [00:44<00:06, 1885.82 examples/s]
Tokenizing train dataset: 87%|▊| 81678/93371 [00:44<00:06, 1888.36 examples/s]
Tokenizing train dataset: 88%|▉| 81869/93371 [00:44<00:06, 1887.76 examples/s]
Tokenizing train dataset: 88%|▉| 82092/93371 [00:44<00:06, 1733.31 examples/s]
Tokenizing train dataset: 88%|▉| 82282/93371 [00:44<00:06, 1771.51 examples/s]
Tokenizing train dataset: 88%|▉| 82491/93371 [00:45<00:05, 1852.61 examples/s]
Tokenizing train dataset: 89%|▉| 82766/93371 [00:45<00:05, 1836.46 examples/s]
Tokenizing train dataset: 89%|▉| 82956/93371 [00:45<00:05, 1847.72 examples/s]
Tokenizing train dataset: 89%|▉| 83203/93371 [00:45<00:05, 1774.40 examples/s]
Tokenizing train dataset: 89%|▉| 83406/93371 [00:45<00:05, 1835.54 examples/s]
Tokenizing train dataset: 90%|▉| 83601/93371 [00:45<00:05, 1860.16 examples/s]
Tokenizing train dataset: 90%|▉| 83898/93371 [00:45<00:04, 1894.94 examples/s]
Tokenizing train dataset: 90%|▉| 84093/93371 [00:45<00:05, 1809.86 examples/s]
Tokenizing train dataset: 90%|▉| 84294/93371 [00:46<00:04, 1855.58 examples/s]
Tokenizing train dataset: 90%|▉| 84497/93371 [00:46<00:04, 1901.11 examples/s]
Tokenizing train dataset: 91%|▉| 84696/93371 [00:46<00:04, 1923.65 examples/s]
Tokenizing train dataset: 91%|▉| 84901/93371 [00:46<00:04, 1957.52 examples/s]
Tokenizing train dataset: 91%|▉| 85099/93371 [00:46<00:04, 1862.03 examples/s]
Tokenizing train dataset: 91%|▉| 85294/93371 [00:46<00:04, 1875.85 examples/s]
Tokenizing train dataset: 92%|▉| 85485/93371 [00:46<00:04, 1876.87 examples/s]
Tokenizing train dataset: 92%|▉| 85687/93371 [00:46<00:04, 1916.35 examples/s]
Tokenizing train dataset: 92%|▉| 85960/93371 [00:46<00:03, 1874.29 examples/s]
Tokenizing train dataset: 92%|▉| 86215/93371 [00:47<00:03, 1811.10 examples/s]
Tokenizing train dataset: 93%|▉| 86410/93371 [00:47<00:03, 1843.92 examples/s]
Tokenizing train dataset: 93%|▉| 86687/93371 [00:47<00:03, 1842.63 examples/s]
Tokenizing train dataset: 93%|▉| 86969/93371 [00:47<00:03, 1852.09 examples/s]
Tokenizing train dataset: 93%|▉| 87232/93371 [00:47<00:03, 1816.22 examples/s]
Tokenizing train dataset: 94%|▉| 87435/93371 [00:47<00:03, 1864.39 examples/s]
Tokenizing train dataset: 94%|▉| 87636/93371 [00:47<00:03, 1899.85 examples/s]
Tokenizing train dataset: 94%|▉| 87834/93371 [00:47<00:02, 1910.88 examples/s]
Tokenizing train dataset: 94%|▉| 88097/93371 [00:48<00:02, 1852.07 examples/s]
Tokenizing train dataset: 95%|▉| 88291/93371 [00:48<00:02, 1873.11 examples/s]
Tokenizing train dataset: 95%|▉| 88501/93371 [00:48<00:02, 1932.38 examples/s]
Tokenizing train dataset: 95%|▉| 88699/93371 [00:48<00:02, 1941.88 examples/s]
Tokenizing train dataset: 95%|▉| 88897/93371 [00:48<00:02, 1944.54 examples/s]
Tokenizing train dataset: 95%|▉| 89155/93371 [00:48<00:02, 1856.74 examples/s]
Tokenizing train dataset: 96%|▉| 89360/93371 [00:48<00:02, 1903.72 examples/s]
Tokenizing train dataset: 96%|▉| 89554/93371 [00:48<00:01, 1909.78 examples/s]
Tokenizing train dataset: 96%|▉| 89843/93371 [00:48<00:01, 1909.46 examples/s]
Tokenizing train dataset: 96%|▉| 90090/93371 [00:49<00:01, 1804.54 examples/s]
Tokenizing train dataset: 97%|▉| 90293/93371 [00:49<00:01, 1857.03 examples/s]
Tokenizing train dataset: 97%|▉| 90490/93371 [00:49<00:01, 1873.79 examples/s]
Tokenizing train dataset: 97%|▉| 90789/93371 [00:49<00:01, 1909.45 examples/s]
Tokenizing train dataset: 97%|▉| 90995/93371 [00:49<00:01, 1943.31 examples/s]
Tokenizing train dataset: 98%|▉| 91246/93371 [00:49<00:01, 1846.46 examples/s]
Tokenizing train dataset: 98%|▉| 91532/93371 [00:49<00:00, 1861.47 examples/s]
Tokenizing train dataset: 98%|▉| 91730/93371 [00:49<00:00, 1889.58 examples/s]
Tokenizing train dataset: 98%|▉| 91934/93371 [00:50<00:00, 1924.87 examples/s]
Tokenizing train dataset: 99%|▉| 92195/93371 [00:50<00:00, 1844.64 examples/s]
Tokenizing train dataset: 99%|▉| 92482/93371 [00:50<00:00, 1861.50 examples/s]
Tokenizing train dataset: 99%|▉| 92670/93371 [00:50<00:00, 1865.05 examples/s]
Tokenizing train dataset: 99%|▉| 92872/93371 [00:50<00:00, 1901.05 examples/s]
Tokenizing train dataset: 100%|▉| 93126/93371 [00:50<00:00, 1825.93 examples/s]
Tokenizing train dataset: 100%|▉| 93326/93371 [00:50<00:00, 1865.08 examples/s]
Tokenizing train dataset: 100%|█| 93371/93371 [00:51<00:00, 1828.13 examples/s] |
|
Packing train dataset: 0%| | 0/93371 [00:00<?, ? examples/s]
Packing train dataset: 4%|▏ | 4000/93371 [00:00<00:02, 37405.06 examples/s]
Packing train dataset: 11%|▎ | 10000/93371 [00:00<00:02, 39289.62 examples/s]
Packing train dataset: 16%|▍ | 15000/93371 [00:00<00:01, 39690.82 examples/s]
Packing train dataset: 20%|▌ | 19000/93371 [00:00<00:01, 39707.60 examples/s]
Packing train dataset: 25%|▋ | 23000/93371 [00:00<00:01, 39629.53 examples/s]
Packing train dataset: 30%|▉ | 28000/93371 [00:00<00:01, 39952.27 examples/s]
Packing train dataset: 36%|█ | 34000/93371 [00:00<00:01, 40654.32 examples/s]
Packing train dataset: 43%|█▎ | 40000/93371 [00:00<00:01, 41095.57 examples/s]
Packing train dataset: 48%|█▍ | 45000/93371 [00:01<00:01, 41093.73 examples/s]
Packing train dataset: 54%|█▌ | 50000/93371 [00:01<00:01, 40742.90 examples/s]
Packing train dataset: 59%|█▊ | 55000/93371 [00:01<00:00, 40558.61 examples/s]
Packing train dataset: 65%|█▉ | 61000/93371 [00:01<00:00, 40905.16 examples/s]
Packing train dataset: 72%|██▏| 67000/93371 [00:01<00:00, 40462.20 examples/s]
Packing train dataset: 77%|██▎| 72000/93371 [00:01<00:00, 40637.19 examples/s]
Packing train dataset: 84%|██▌| 78000/93371 [00:01<00:00, 41055.18 examples/s]
Packing train dataset: 90%|██▋| 84000/93371 [00:02<00:00, 41558.99 examples/s]
Packing train dataset: 96%|██▉| 90000/93371 [00:02<00:00, 41944.79 examples/s]
Packing train dataset: 100%|███| 93371/93371 [00:02<00:00, 37889.09 examples/s] |
|
|
| Starting multilingual SFT training... |
| [transformers] The tokenizer has new PAD/BOS/EOS tokens that differ from the model config and generation config. The model config and generation config were aligned accordingly, being updated with the tokenizer's values. Updated tokens: {'bos_token_id': None, 'pad_token_id': 151643}. |
|
0%| | 0/1101 [00:00<?, ?it/s]
0%| | 1/1101 [00:12<3:50:13, 12.56s/it]
0%| | 2/1101 [00:24<3:42:52, 12.17s/it]
0%| | 3/1101 [00:36<3:42:42, 12.17s/it]
0%|▏ | 4/1101 [00:48<3:41:12, 12.10s/it]
0%|▏ | 5/1101 [01:00<3:41:00, 12.10s/it]
1%|▏ | 6/1101 [01:12<3:39:55, 12.05s/it]
1%|▏ | 7/1101 [01:24<3:40:36, 12.10s/it]
1%|▎ | 8/1101 [01:36<3:39:51, 12.07s/it]
1%|▎ | 9/1101 [01:48<3:39:00, 12.03s/it]
1%|▎ | 10/1101 [02:01<3:39:59, 12.10s/it]
{'loss': '2.287', 'grad_norm': '21.88', 'learning_rate': '1.029e-07', 'entropy': '0.9958', 'num_tokens': '6.544e+05', 'mean_token_accuracy': '0.6042', 'epoch': '0.02726'} |
|
1%|▎ | 10/1101 [02:01<3:39:59, 12.10s/it]
1%|▍ | 11/1101 [02:12<3:38:13, 12.01s/it]
1%|▍ | 12/1101 [02:24<3:35:46, 11.89s/it]
1%|▍ | 13/1101 [02:36<3:36:06, 11.92s/it]
1%|▍ | 14/1101 [02:48<3:37:06, 11.98s/it]
1%|▌ | 15/1101 [02:59<3:32:53, 11.76s/it]
1%|▌ | 16/1101 [03:11<3:33:25, 11.80s/it]
2%|▌ | 17/1101 [03:23<3:35:34, 11.93s/it]
2%|▌ | 18/1101 [03:35<3:35:14, 11.92s/it]
2%|▋ | 19/1101 [03:48<3:36:25, 12.00s/it]
2%|▋ | 20/1101 [04:00<3:36:19, 12.01s/it]
{'loss': '2.355', 'grad_norm': '28.12', 'learning_rate': '2.171e-07', 'entropy': '1.008', 'num_tokens': '1.302e+06', 'mean_token_accuracy': '0.601', 'epoch': '0.05451'} |
|
2%|▋ | 20/1101 [04:00<3:36:19, 12.01s/it]
2%|▋ | 21/1101 [04:12<3:37:52, 12.10s/it]
2%|▊ | 22/1101 [04:24<3:36:39, 12.05s/it]
2%|▊ | 23/1101 [04:36<3:35:58, 12.02s/it]
2%|▊ | 24/1101 [04:48<3:36:06, 12.04s/it]
2%|▊ | 25/1101 [05:00<3:35:32, 12.02s/it]
2%|▉ | 26/1101 [05:12<3:34:32, 11.97s/it]
2%|▉ | 27/1101 [05:24<3:34:27, 11.98s/it]
3%|▉ | 28/1101 [05:36<3:35:06, 12.03s/it]
3%|█ | 29/1101 [05:48<3:34:19, 12.00s/it]
3%|█ | 30/1101 [05:59<3:29:07, 11.72s/it]
{'loss': '2.393', 'grad_norm': '26.25', 'learning_rate': '3.314e-07', 'entropy': '1.019', 'num_tokens': '1.949e+06', 'mean_token_accuracy': '0.5983', 'epoch': '0.08177'} |
|
3%|█ | 30/1101 [05:59<3:29:07, 11.72s/it]
3%|█ | 31/1101 [06:11<3:30:05, 11.78s/it]
3%|█ | 32/1101 [06:23<3:30:50, 11.83s/it]
3%|█▏ | 33/1101 [06:35<3:31:37, 11.89s/it]
3%|█▏ | 34/1101 [06:47<3:31:54, 11.92s/it]
3%|█▏ | 35/1101 [06:59<3:32:48, 11.98s/it]
3%|█▏ | 36/1101 [07:11<3:32:21, 11.96s/it]
3%|█▎ | 37/1101 [07:23<3:31:56, 11.95s/it]
3%|█▎ | 38/1101 [07:35<3:31:32, 11.94s/it]
4%|█▎ | 39/1101 [07:46<3:29:02, 11.81s/it]
4%|█▍ | 40/1101 [07:58<3:28:06, 11.77s/it]
{'loss': '2.445', 'grad_norm': '37.25', 'learning_rate': '4.457e-07', 'entropy': '1.023', 'num_tokens': '2.598e+06', 'mean_token_accuracy': '0.5971', 'epoch': '0.109'} |
|
4%|█▍ | 40/1101 [07:58<3:28:06, 11.77s/it]
4%|█▍ | 41/1101 [08:10<3:28:29, 11.80s/it]
4%|█▍ | 42/1101 [08:21<3:27:37, 11.76s/it]
4%|█▍ | 43/1101 [08:33<3:28:07, 11.80s/it]
4%|█▌ | 44/1101 [08:45<3:29:40, 11.90s/it]
4%|█▌ | 45/1101 [08:57<3:29:33, 11.91s/it]
4%|█▌ | 46/1101 [09:09<3:29:33, 11.92s/it]
4%|█▌ | 47/1101 [09:21<3:30:56, 12.01s/it]
4%|█▋ | 48/1101 [09:33<3:28:32, 11.88s/it]
4%|█▋ | 49/1101 [09:44<3:25:36, 11.73s/it]
5%|█▋ | 50/1101 [09:56<3:26:20, 11.78s/it]
{'loss': '2.369', 'grad_norm': '23', 'learning_rate': '5.6e-07', 'entropy': '1.024', 'num_tokens': '3.244e+06', 'mean_token_accuracy': '0.5976', 'epoch': '0.1363'} |
|
5%|█▋ | 50/1101 [09:56<3:26:20, 11.78s/it]
5%|█▊ | 51/1101 [10:08<3:25:29, 11.74s/it]
5%|█▊ | 52/1101 [10:20<3:26:19, 11.80s/it]
5%|█▊ | 53/1101 [10:32<3:26:58, 11.85s/it]
5%|█▊ | 54/1101 [10:44<3:28:04, 11.92s/it]
5%|█▉ | 55/1101 [10:56<3:27:57, 11.93s/it]
5%|█▉ | 56/1101 [11:08<3:28:56, 12.00s/it]
5%|█▉ | 57/1101 [11:20<3:28:31, 11.98s/it]
5%|██ | 58/1101 [11:31<3:25:06, 11.80s/it]
5%|██ | 59/1101 [11:43<3:25:45, 11.85s/it]
5%|██ | 60/1101 [11:55<3:25:42, 11.86s/it]
{'loss': '2.419', 'grad_norm': '29', 'learning_rate': '6.743e-07', 'entropy': '1.049', 'num_tokens': '3.891e+06', 'mean_token_accuracy': '0.5952', 'epoch': '0.1635'} |
|
5%|██ | 60/1101 [11:55<3:25:42, 11.86s/it]
6%|██ | 61/1101 [12:07<3:26:52, 11.93s/it]
6%|██▏ | 62/1101 [12:19<3:26:43, 11.94s/it]
6%|██▏ | 63/1101 [12:31<3:25:30, 11.88s/it]
6%|██▏ | 64/1101 [12:43<3:25:11, 11.87s/it]
6%|██▏ | 65/1101 [12:55<3:26:50, 11.98s/it]
6%|██▎ | 66/1101 [13:07<3:26:12, 11.95s/it]
6%|██▎ | 67/1101 [13:19<3:27:30, 12.04s/it]
6%|██▎ | 68/1101 [13:31<3:26:42, 12.01s/it]
6%|██▍ | 69/1101 [13:43<3:25:28, 11.95s/it]
6%|██▍ | 70/1101 [13:55<3:26:48, 12.04s/it]
{'loss': '2.403', 'grad_norm': '21', 'learning_rate': '7.886e-07', 'entropy': '1.056', 'num_tokens': '4.542e+06', 'mean_token_accuracy': '0.5937', 'epoch': '0.1908'} |
|
6%|██▍ | 70/1101 [13:55<3:26:48, 12.04s/it]
6%|██▍ | 71/1101 [14:07<3:26:05, 12.01s/it]
7%|██▍ | 72/1101 [14:19<3:26:54, 12.06s/it]
7%|██▌ | 73/1101 [14:31<3:26:15, 12.04s/it]
7%|██▌ | 74/1101 [14:44<3:26:44, 12.08s/it]
7%|██▌ | 75/1101 [14:55<3:25:45, 12.03s/it]
7%|██▌ | 76/1101 [15:07<3:24:47, 11.99s/it]
7%|██▋ | 77/1101 [15:19<3:22:57, 11.89s/it]
7%|██▋ | 78/1101 [15:31<3:22:49, 11.90s/it]
7%|██▋ | 79/1101 [15:43<3:21:36, 11.84s/it]
7%|██▊ | 80/1101 [15:55<3:21:53, 11.86s/it]
{'loss': '2.316', 'grad_norm': '26.25', 'learning_rate': '9.029e-07', 'entropy': '1.044', 'num_tokens': '5.19e+06', 'mean_token_accuracy': '0.5982', 'epoch': '0.2181'} |
|
7%|██▊ | 80/1101 [15:55<3:21:53, 11.86s/it]
7%|██▊ | 81/1101 [16:06<3:21:14, 11.84s/it]
7%|██▊ | 82/1101 [16:18<3:18:26, 11.68s/it]
8%|██▊ | 83/1101 [16:30<3:19:23, 11.75s/it]
8%|██▉ | 84/1101 [16:42<3:21:27, 11.89s/it]
8%|██▉ | 85/1101 [16:54<3:21:51, 11.92s/it]
8%|██▉ | 86/1101 [17:06<3:23:14, 12.01s/it]
8%|███ | 87/1101 [17:17<3:19:36, 11.81s/it]
8%|███ | 88/1101 [17:30<3:21:46, 11.95s/it]
8%|███ | 89/1101 [17:42<3:21:33, 11.95s/it]
8%|███ | 90/1101 [17:53<3:21:23, 11.95s/it]
{'loss': '2.317', 'grad_norm': '21.75', 'learning_rate': '1.017e-06', 'entropy': '1.038', 'num_tokens': '5.836e+06', 'mean_token_accuracy': '0.5991', 'epoch': '0.2453'} |
|
8%|███ | 90/1101 [17:54<3:21:23, 11.95s/it]
8%|███▏ | 91/1101 [18:05<3:21:15, 11.96s/it]
8%|███▏ | 92/1101 [18:17<3:20:45, 11.94s/it]
8%|███▏ | 93/1101 [18:29<3:21:33, 12.00s/it]
9%|███▏ | 94/1101 [18:41<3:21:07, 11.98s/it]
9%|███▎ | 95/1101 [18:54<3:21:18, 12.01s/it]
9%|███▎ | 96/1101 [19:05<3:20:59, 12.00s/it]
9%|███▎ | 97/1101 [19:17<3:20:26, 11.98s/it]
9%|███▍ | 98/1101 [19:30<3:20:49, 12.01s/it]
9%|███▍ | 99/1101 [19:41<3:20:01, 11.98s/it]
9%|███▎ | 100/1101 [19:53<3:20:22, 12.01s/it]
{'loss': '2.211', 'grad_norm': '24.88', 'learning_rate': '1.131e-06', 'entropy': '1.034', 'num_tokens': '6.489e+06', 'mean_token_accuracy': '0.6071', 'epoch': '0.2726'} |
|
9%|███▎ | 100/1101 [19:54<3:20:22, 12.01s/it]
9%|███▍ | 101/1101 [20:05<3:19:44, 11.98s/it]
9%|███▍ | 102/1101 [20:18<3:20:17, 12.03s/it]
9%|███▍ | 103/1101 [20:29<3:17:03, 11.85s/it]
9%|███▍ | 104/1101 [20:41<3:18:00, 11.92s/it]
10%|███▌ | 105/1101 [20:53<3:17:51, 11.92s/it]
10%|███▌ | 106/1101 [21:05<3:18:11, 11.95s/it]
10%|███▌ | 107/1101 [21:17<3:18:07, 11.96s/it]
10%|███▋ | 108/1101 [21:29<3:18:00, 11.96s/it]
10%|███▋ | 109/1101 [21:41<3:18:23, 12.00s/it]
10%|███▋ | 110/1101 [21:53<3:16:38, 11.91s/it]
{'loss': '2.269', 'grad_norm': '21.25', 'learning_rate': '1.246e-06', 'entropy': '1.076', 'num_tokens': '7.138e+06', 'mean_token_accuracy': '0.6004', 'epoch': '0.2998'} |
|
10%|███▋ | 110/1101 [21:53<3:16:38, 11.91s/it]
10%|███▋ | 111/1101 [22:05<3:17:28, 11.97s/it]
10%|███▊ | 112/1101 [22:17<3:17:37, 11.99s/it]
10%|███▊ | 113/1101 [22:29<3:17:11, 11.97s/it]
10%|███▊ | 114/1101 [22:41<3:17:58, 12.03s/it]
10%|███▊ | 115/1101 [22:53<3:17:04, 11.99s/it]
11%|███▉ | 116/1101 [23:05<3:17:44, 12.04s/it]
11%|███▉ | 117/1101 [23:17<3:16:41, 11.99s/it]
11%|███▉ | 118/1101 [23:29<3:18:19, 12.11s/it]
11%|███▉ | 119/1101 [23:41<3:17:07, 12.04s/it]
11%|████ | 120/1101 [23:53<3:16:33, 12.02s/it]
{'loss': '2.256', 'grad_norm': '24.88', 'learning_rate': '1.36e-06', 'entropy': '1.094', 'num_tokens': '7.792e+06', 'mean_token_accuracy': '0.6002', 'epoch': '0.3271'} |
|
11%|████ | 120/1101 [23:53<3:16:33, 12.02s/it]
11%|████ | 121/1101 [24:05<3:17:09, 12.07s/it]
11%|████ | 122/1101 [24:17<3:16:00, 12.01s/it]
11%|████▏ | 123/1101 [24:29<3:16:49, 12.07s/it]
11%|████▏ | 124/1101 [24:41<3:16:23, 12.06s/it]
11%|████▏ | 125/1101 [24:53<3:15:01, 11.99s/it]
11%|████▏ | 126/1101 [25:05<3:12:24, 11.84s/it]
12%|████▎ | 127/1101 [25:16<3:11:33, 11.80s/it]
12%|████▎ | 128/1101 [25:29<3:13:28, 11.93s/it]
12%|████▎ | 129/1101 [25:41<3:13:03, 11.92s/it]
12%|████▎ | 130/1101 [25:53<3:14:03, 11.99s/it]
{'loss': '2.167', 'grad_norm': '15.38', 'learning_rate': '1.474e-06', 'entropy': '1.1', 'num_tokens': '8.439e+06', 'mean_token_accuracy': '0.6024', 'epoch': '0.3543'} |
|
12%|████▎ | 130/1101 [25:53<3:14:03, 11.99s/it]
12%|████▍ | 131/1101 [26:05<3:13:22, 11.96s/it]
12%|████▍ | 132/1101 [26:16<3:12:08, 11.90s/it]
12%|████▍ | 133/1101 [26:28<3:12:40, 11.94s/it]
12%|████▌ | 134/1101 [26:40<3:12:36, 11.95s/it]
12%|████▌ | 135/1101 [26:53<3:13:21, 12.01s/it]
12%|████▌ | 136/1101 [27:05<3:12:57, 12.00s/it]
12%|████▌ | 137/1101 [27:17<3:13:42, 12.06s/it]
13%|████▋ | 138/1101 [27:29<3:12:33, 12.00s/it]
13%|████▋ | 139/1101 [27:41<3:13:01, 12.04s/it]
13%|████▋ | 140/1101 [27:53<3:12:18, 12.01s/it]
{'loss': '2.099', 'grad_norm': '14.81', 'learning_rate': '1.589e-06', 'entropy': '1.108', 'num_tokens': '9.092e+06', 'mean_token_accuracy': '0.608', 'epoch': '0.3816'} |
|
13%|████▋ | 140/1101 [27:53<3:12:18, 12.01s/it]
13%|████▋ | 141/1101 [28:05<3:12:11, 12.01s/it]
13%|████▊ | 142/1101 [28:17<3:11:27, 11.98s/it]
13%|████▊ | 143/1101 [28:28<3:10:18, 11.92s/it]
13%|████▊ | 144/1101 [28:40<3:08:59, 11.85s/it]
13%|████▊ | 145/1101 [28:52<3:08:52, 11.85s/it]
13%|████▉ | 146/1101 [29:04<3:10:20, 11.96s/it]
13%|████▉ | 147/1101 [29:16<3:10:10, 11.96s/it]
13%|████▉ | 148/1101 [29:28<3:09:00, 11.90s/it]
14%|█████ | 149/1101 [29:40<3:08:52, 11.90s/it]
14%|█████ | 150/1101 [29:52<3:08:35, 11.90s/it]
{'loss': '2.158', 'grad_norm': '19.62', 'learning_rate': '1.703e-06', 'entropy': '1.169', 'num_tokens': '9.74e+06', 'mean_token_accuracy': '0.6006', 'epoch': '0.4089'} |
|
14%|█████ | 150/1101 [29:52<3:08:35, 11.90s/it]
14%|█████ | 151/1101 [30:03<3:07:50, 11.86s/it]
14%|█████ | 152/1101 [30:15<3:06:53, 11.82s/it]
14%|█████▏ | 153/1101 [30:27<3:08:24, 11.92s/it]
14%|█████▏ | 154/1101 [30:39<3:08:12, 11.92s/it]
14%|█████▏ | 155/1101 [30:51<3:09:33, 12.02s/it]
14%|█████▏ | 156/1101 [31:03<3:09:02, 12.00s/it]
14%|█████▎ | 157/1101 [31:15<3:08:46, 12.00s/it]
14%|█████▎ | 158/1101 [31:28<3:09:46, 12.07s/it]
14%|█████▎ | 159/1101 [31:39<3:06:53, 11.90s/it]
15%|█████▍ | 160/1101 [31:51<3:07:56, 11.98s/it]
{'loss': '2.015', 'grad_norm': '14', 'learning_rate': '1.817e-06', 'entropy': '1.158', 'num_tokens': '1.039e+07', 'mean_token_accuracy': '0.6119', 'epoch': '0.4361'} |
|
15%|█████▍ | 160/1101 [31:51<3:07:56, 11.98s/it]
15%|█████▍ | 161/1101 [32:03<3:07:15, 11.95s/it]
15%|█████▍ | 162/1101 [32:15<3:07:48, 12.00s/it]
15%|█████▍ | 163/1101 [32:27<3:07:50, 12.02s/it]
15%|█████▌ | 164/1101 [32:39<3:07:05, 11.98s/it]
15%|█████▌ | 165/1101 [32:51<3:07:45, 12.04s/it]
15%|█████▌ | 166/1101 [33:03<3:07:02, 12.00s/it]
15%|█████▌ | 167/1101 [33:16<3:07:45, 12.06s/it]
15%|█████▋ | 168/1101 [33:28<3:07:16, 12.04s/it]
15%|█████▋ | 169/1101 [33:39<3:06:16, 11.99s/it]
15%|█████▋ | 170/1101 [33:51<3:05:34, 11.96s/it]
{'loss': '1.938', 'grad_norm': '13.62', 'learning_rate': '1.931e-06', 'entropy': '1.167', 'num_tokens': '1.104e+07', 'mean_token_accuracy': '0.6176', 'epoch': '0.4634'} |
|
15%|█████▋ | 170/1101 [33:51<3:05:34, 11.96s/it]
16%|█████▋ | 171/1101 [34:03<3:05:24, 11.96s/it]
16%|█████▊ | 172/1101 [34:16<3:06:19, 12.03s/it]
16%|█████▊ | 173/1101 [34:27<3:05:33, 12.00s/it]
16%|█████▊ | 174/1101 [34:39<3:04:31, 11.94s/it]
16%|█████▉ | 175/1101 [34:51<3:04:26, 11.95s/it]
16%|█████▉ | 176/1101 [35:03<3:04:52, 11.99s/it]
16%|█████▉ | 177/1101 [35:15<3:04:21, 11.97s/it]
16%|█████▉ | 178/1101 [35:27<3:04:44, 12.01s/it]
16%|██████ | 179/1101 [35:39<3:04:01, 11.98s/it]
16%|██████ | 180/1101 [35:51<3:03:23, 11.95s/it]
{'loss': '1.929', 'grad_norm': '11.62', 'learning_rate': '2.046e-06', 'entropy': '1.214', 'num_tokens': '1.169e+07', 'mean_token_accuracy': '0.6164', 'epoch': '0.4906'} |
|
16%|██████ | 180/1101 [35:51<3:03:23, 11.95s/it]
16%|██████ | 181/1101 [36:03<3:04:01, 12.00s/it]
17%|██████ | 182/1101 [36:15<3:03:41, 11.99s/it]
17%|██████▏ | 183/1101 [36:26<3:00:11, 11.78s/it]
17%|██████▏ | 184/1101 [36:38<3:00:43, 11.82s/it]
17%|██████▏ | 185/1101 [36:50<3:01:38, 11.90s/it]
17%|██████▎ | 186/1101 [37:02<3:01:29, 11.90s/it]
17%|██████▎ | 187/1101 [37:14<2:59:25, 11.78s/it]
17%|██████▎ | 188/1101 [37:26<3:00:42, 11.88s/it]
17%|██████▎ | 189/1101 [37:38<3:00:39, 11.89s/it]
17%|██████▍ | 190/1101 [37:50<3:01:26, 11.95s/it]
{'loss': '1.826', 'grad_norm': '10', 'learning_rate': '2.16e-06', 'entropy': '1.233', 'num_tokens': '1.234e+07', 'mean_token_accuracy': '0.6231', 'epoch': '0.5179'} |
|
17%|██████▍ | 190/1101 [37:50<3:01:26, 11.95s/it]
17%|██████▍ | 191/1101 [38:02<3:01:09, 11.94s/it]
17%|██████▍ | 192/1101 [38:14<3:01:51, 12.00s/it]
18%|██████▍ | 193/1101 [38:26<3:01:16, 11.98s/it]
18%|██████▌ | 194/1101 [38:38<3:01:01, 11.98s/it]
18%|██████▌ | 195/1101 [38:50<3:01:44, 12.04s/it]
18%|██████▌ | 196/1101 [39:02<3:00:44, 11.98s/it]
18%|██████▌ | 197/1101 [39:14<2:58:49, 11.87s/it]
18%|██████▋ | 198/1101 [39:26<2:58:59, 11.89s/it]
18%|██████▋ | 199/1101 [39:38<2:59:50, 11.96s/it]
18%|██████▋ | 200/1101 [39:49<2:57:42, 11.83s/it]
{'loss': '1.735', 'grad_norm': '12', 'learning_rate': '2.274e-06', 'entropy': '1.245', 'num_tokens': '1.299e+07', 'mean_token_accuracy': '0.6286', 'epoch': '0.5451'} |
|
18%|██████▋ | 200/1101 [39:49<2:57:42, 11.83s/it]
18%|██████▊ | 201/1101 [40:01<2:57:45, 11.85s/it]
18%|██████▊ | 202/1101 [40:13<2:58:41, 11.93s/it]
18%|██████▊ | 203/1101 [40:25<2:58:05, 11.90s/it]
19%|██████▊ | 204/1101 [40:37<2:56:50, 11.83s/it]
19%|██████▉ | 205/1101 [40:48<2:55:02, 11.72s/it]
19%|██████▉ | 206/1101 [41:00<2:54:05, 11.67s/it]
19%|██████▉ | 207/1101 [41:12<2:54:47, 11.73s/it]
19%|██████▉ | 208/1101 [41:24<2:55:32, 11.79s/it]
19%|███████ | 209/1101 [41:35<2:54:29, 11.74s/it]
19%|███████ | 210/1101 [41:47<2:55:12, 11.80s/it]
{'loss': '1.667', 'grad_norm': '7.531', 'learning_rate': '2.389e-06', 'entropy': '1.27', 'num_tokens': '1.363e+07', 'mean_token_accuracy': '0.6359', 'epoch': '0.5724'} |
|
19%|███████ | 210/1101 [41:47<2:55:12, 11.80s/it]
19%|███████ | 211/1101 [41:59<2:54:43, 11.78s/it]
19%|███████ | 212/1101 [42:11<2:55:08, 11.82s/it]
19%|███████▏ | 213/1101 [42:23<2:56:13, 11.91s/it]
19%|███████▏ | 214/1101 [42:35<2:55:59, 11.90s/it]
20%|███████▏ | 215/1101 [42:47<2:56:43, 11.97s/it]
20%|███████▎ | 216/1101 [42:59<2:56:26, 11.96s/it]
20%|███████▎ | 217/1101 [43:11<2:56:23, 11.97s/it]
20%|███████▎ | 218/1101 [43:23<2:57:05, 12.03s/it]
20%|███████▎ | 219/1101 [43:35<2:56:35, 12.01s/it]
20%|███████▍ | 220/1101 [43:47<2:54:33, 11.89s/it]
{'loss': '1.617', 'grad_norm': '6', 'learning_rate': '2.503e-06', 'entropy': '1.296', 'num_tokens': '1.428e+07', 'mean_token_accuracy': '0.643', 'epoch': '0.5997'} |
|
20%|███████▍ | 220/1101 [43:47<2:54:33, 11.89s/it]
20%|███████▍ | 221/1101 [43:59<2:54:41, 11.91s/it]
20%|███████▍ | 222/1101 [44:11<2:55:21, 11.97s/it]
20%|███████▍ | 223/1101 [44:23<2:54:50, 11.95s/it]
20%|███████▌ | 224/1101 [44:34<2:54:26, 11.93s/it]
20%|███████▌ | 225/1101 [44:47<2:55:06, 11.99s/it]
21%|███████▌ | 226/1101 [44:59<2:54:50, 11.99s/it]
21%|███████▋ | 227/1101 [45:11<2:55:01, 12.01s/it]
21%|███████▋ | 228/1101 [45:23<2:54:41, 12.01s/it]
21%|███████▋ | 229/1101 [45:35<2:55:07, 12.05s/it]
21%|███████▋ | 230/1101 [45:47<2:54:06, 11.99s/it]
{'loss': '1.639', 'grad_norm': '5.281', 'learning_rate': '2.617e-06', 'entropy': '1.36', 'num_tokens': '1.494e+07', 'mean_token_accuracy': '0.6416', 'epoch': '0.6269'} |
|
21%|███████▋ | 230/1101 [45:47<2:54:06, 11.99s/it]
21%|███████▊ | 231/1101 [45:59<2:54:02, 12.00s/it]
21%|███████▊ | 232/1101 [46:11<2:54:27, 12.05s/it]
21%|███████▊ | 233/1101 [46:22<2:52:14, 11.91s/it]
21%|███████▊ | 234/1101 [46:35<2:53:19, 12.00s/it]
21%|███████▉ | 235/1101 [46:47<2:53:17, 12.01s/it]
21%|███████▉ | 236/1101 [46:59<2:53:26, 12.03s/it]
22%|███████▉ | 237/1101 [47:11<2:52:43, 11.99s/it]
22%|███████▉ | 238/1101 [47:23<2:52:13, 11.97s/it]
22%|████████ | 239/1101 [47:35<2:52:38, 12.02s/it]
22%|████████ | 240/1101 [47:46<2:51:42, 11.97s/it]
{'loss': '1.553', 'grad_norm': '5.188', 'learning_rate': '2.731e-06', 'entropy': '1.329', 'num_tokens': '1.559e+07', 'mean_token_accuracy': '0.6492', 'epoch': '0.6542'} |
|
22%|████████ | 240/1101 [47:47<2:51:42, 11.97s/it]
22%|████████ | 241/1101 [47:58<2:50:06, 11.87s/it]
22%|████████▏ | 242/1101 [48:10<2:50:12, 11.89s/it]
22%|████████▏ | 243/1101 [48:22<2:50:56, 11.95s/it]
22%|████████▏ | 244/1101 [48:34<2:50:42, 11.95s/it]
22%|████████▏ | 245/1101 [48:45<2:47:58, 11.77s/it]
22%|████████▎ | 246/1101 [48:58<2:49:05, 11.87s/it]
22%|████████▎ | 247/1101 [49:10<2:49:25, 11.90s/it]
23%|████████▎ | 248/1101 [49:22<2:49:48, 11.94s/it]
23%|████████▎ | 249/1101 [49:33<2:48:36, 11.87s/it]
23%|████████▍ | 250/1101 [49:45<2:49:31, 11.95s/it]
{'loss': '1.545', 'grad_norm': '5.531', 'learning_rate': '2.846e-06', 'entropy': '1.374', 'num_tokens': '1.624e+07', 'mean_token_accuracy': '0.6514', 'epoch': '0.6814'} |
|
23%|████████▍ | 250/1101 [49:45<2:49:31, 11.95s/it]
23%|████████▍ | 251/1101 [49:57<2:49:28, 11.96s/it]
23%|████████▍ | 252/1101 [50:10<2:50:01, 12.02s/it]
23%|████████▌ | 253/1101 [50:22<2:49:39, 12.00s/it]
23%|████████▌ | 254/1101 [50:33<2:48:49, 11.96s/it]
23%|████████▌ | 255/1101 [50:45<2:48:22, 11.94s/it]
23%|████████▌ | 256/1101 [50:57<2:48:04, 11.93s/it]
23%|████████▋ | 257/1101 [51:09<2:48:42, 11.99s/it]
23%|████████▋ | 258/1101 [51:21<2:47:59, 11.96s/it]
24%|████████▋ | 259/1101 [51:33<2:48:10, 11.98s/it]
24%|████████▋ | 260/1101 [51:45<2:46:38, 11.89s/it]
{'loss': '1.504', 'grad_norm': '6', 'learning_rate': '2.96e-06', 'entropy': '1.365', 'num_tokens': '1.689e+07', 'mean_token_accuracy': '0.6556', 'epoch': '0.7087'} |
|
24%|████████▋ | 260/1101 [51:45<2:46:38, 11.89s/it]
24%|████████▊ | 261/1101 [51:57<2:45:10, 11.80s/it]
24%|████████▊ | 262/1101 [52:09<2:46:46, 11.93s/it]
24%|████████▊ | 263/1101 [52:21<2:46:45, 11.94s/it]
24%|████████▊ | 264/1101 [52:33<2:47:35, 12.01s/it]
24%|████████▉ | 265/1101 [52:45<2:46:57, 11.98s/it]
24%|████████▉ | 266/1101 [52:57<2:47:35, 12.04s/it]
24%|████████▉ | 267/1101 [53:09<2:46:54, 12.01s/it]
24%|█████████ | 268/1101 [53:21<2:46:28, 11.99s/it]
24%|█████████ | 269/1101 [53:33<2:47:03, 12.05s/it]
25%|█████████ | 270/1101 [53:45<2:46:29, 12.02s/it]
{'loss': '1.474', 'grad_norm': '5.875', 'learning_rate': '3.074e-06', 'entropy': '1.354', 'num_tokens': '1.754e+07', 'mean_token_accuracy': '0.6621', 'epoch': '0.7359'} |
|
25%|█████████ | 270/1101 [53:45<2:46:29, 12.02s/it]
25%|█████████ | 271/1101 [53:57<2:46:36, 12.04s/it]
25%|█████████▏ | 272/1101 [54:09<2:45:51, 12.00s/it]
25%|█████████▏ | 273/1101 [54:21<2:46:24, 12.06s/it]
25%|█████████▏ | 274/1101 [54:33<2:45:39, 12.02s/it]
25%|█████████▏ | 275/1101 [54:44<2:42:42, 11.82s/it]
25%|█████████▎ | 276/1101 [54:57<2:43:59, 11.93s/it]
25%|█████████▎ | 277/1101 [55:09<2:44:02, 11.94s/it]
25%|█████████▎ | 278/1101 [55:21<2:44:44, 12.01s/it]
25%|█████████▍ | 279/1101 [55:33<2:44:24, 12.00s/it]
25%|█████████▍ | 280/1101 [55:45<2:44:43, 12.04s/it]
{'loss': '1.438', 'grad_norm': '6.438', 'learning_rate': '3.189e-06', 'entropy': '1.333', 'num_tokens': '1.82e+07', 'mean_token_accuracy': '0.6696', 'epoch': '0.7632'} |
|
25%|█████████▍ | 280/1101 [55:45<2:44:43, 12.04s/it]
26%|█████████▍ | 281/1101 [55:57<2:43:03, 11.93s/it]
26%|█████████▍ | 282/1101 [56:09<2:42:51, 11.93s/it]
26%|█████████▌ | 283/1101 [56:20<2:40:34, 11.78s/it]
26%|█████████▌ | 284/1101 [56:32<2:40:49, 11.81s/it]
26%|█████████▌ | 285/1101 [56:44<2:42:00, 11.91s/it]
26%|█████████▌ | 286/1101 [56:55<2:36:20, 11.51s/it]
26%|█████████▋ | 287/1101 [57:07<2:38:49, 11.71s/it]
26%|█████████▋ | 288/1101 [57:19<2:39:29, 11.77s/it]
26%|█████████▋ | 289/1101 [57:30<2:38:35, 11.72s/it]
26%|█████████▋ | 290/1101 [57:42<2:39:34, 11.81s/it]
{'loss': '1.487', 'grad_norm': '4.594', 'learning_rate': '3.303e-06', 'entropy': '1.39', 'num_tokens': '1.883e+07', 'mean_token_accuracy': '0.6686', 'epoch': '0.7905'} |
|
26%|█████████▋ | 290/1101 [57:42<2:39:34, 11.81s/it]
26%|█████████▊ | 291/1101 [57:54<2:40:15, 11.87s/it]
27%|█████████▊ | 292/1101 [58:06<2:41:21, 11.97s/it]
27%|█████████▊ | 293/1101 [58:18<2:40:55, 11.95s/it]
27%|█████████▉ | 294/1101 [58:30<2:39:31, 11.86s/it]
27%|█████████▉ | 295/1101 [58:42<2:39:35, 11.88s/it]
27%|█████████▉ | 296/1101 [58:54<2:38:19, 11.80s/it]
27%|█████████▉ | 297/1101 [59:06<2:38:45, 11.85s/it]
27%|██████████ | 298/1101 [59:17<2:38:36, 11.85s/it]
27%|██████████ | 299/1101 [59:29<2:39:10, 11.91s/it]
27%|██████████ | 300/1101 [59:41<2:39:14, 11.93s/it]
{'loss': '1.438', 'grad_norm': '1.992', 'learning_rate': '3.417e-06', 'entropy': '1.354', 'num_tokens': '1.948e+07', 'mean_token_accuracy': '0.6723', 'epoch': '0.8177'} |
|
27%|██████████ | 300/1101 [59:41<2:39:14, 11.93s/it]
27%|██████████ | 301/1101 [59:53<2:38:21, 11.88s/it]
27%|█████████▌ | 302/1101 [1:00:05<2:38:16, 11.89s/it]
28%|█████████▋ | 303/1101 [1:00:17<2:39:07, 11.96s/it]
28%|█████████▋ | 304/1101 [1:00:29<2:38:29, 11.93s/it]
28%|█████████▋ | 305/1101 [1:00:41<2:38:19, 11.93s/it]
28%|█████████▋ | 306/1101 [1:00:53<2:38:42, 11.98s/it]
28%|█████████▊ | 307/1101 [1:01:05<2:37:56, 11.94s/it]
28%|█████████▊ | 308/1101 [1:01:17<2:38:26, 11.99s/it]
28%|█████████▊ | 309/1101 [1:01:28<2:36:01, 11.82s/it]
28%|█████████▊ | 310/1101 [1:01:40<2:36:39, 11.88s/it]
{'loss': '1.424', 'grad_norm': '1.781', 'learning_rate': '3.531e-06', 'entropy': '1.35', 'num_tokens': '2.013e+07', 'mean_token_accuracy': '0.6767', 'epoch': '0.845'} |
|
28%|█████████▊ | 310/1101 [1:01:41<2:36:39, 11.88s/it]
28%|█████████▉ | 311/1101 [1:01:52<2:36:34, 11.89s/it]
28%|█████████▉ | 312/1101 [1:02:04<2:36:17, 11.88s/it]
28%|█████████▉ | 313/1101 [1:02:16<2:37:02, 11.96s/it]
29%|█████████▉ | 314/1101 [1:02:28<2:36:56, 11.97s/it]
29%|██████████ | 315/1101 [1:02:40<2:37:20, 12.01s/it]
29%|██████████ | 316/1101 [1:02:52<2:36:48, 11.99s/it]
29%|██████████ | 317/1101 [1:03:05<2:37:21, 12.04s/it]
29%|██████████ | 318/1101 [1:03:17<2:36:39, 12.00s/it]
29%|██████████▏ | 319/1101 [1:03:28<2:36:18, 11.99s/it]
29%|██████████▏ | 320/1101 [1:03:41<2:36:26, 12.02s/it]
{'loss': '1.361', 'grad_norm': '1.391', 'learning_rate': '3.646e-06', 'entropy': '1.303', 'num_tokens': '2.079e+07', 'mean_token_accuracy': '0.6824', 'epoch': '0.8722'} |
|
29%|██████████▏ | 320/1101 [1:03:41<2:36:26, 12.02s/it]
29%|██████████▏ | 321/1101 [1:03:52<2:35:52, 11.99s/it]
29%|██████████▏ | 322/1101 [1:04:05<2:36:12, 12.03s/it]
29%|██████████▎ | 323/1101 [1:04:17<2:35:40, 12.01s/it]
29%|██████████▎ | 324/1101 [1:04:29<2:36:20, 12.07s/it]
30%|██████████▎ | 325/1101 [1:04:41<2:35:34, 12.03s/it]
30%|██████████▎ | 326/1101 [1:04:53<2:36:05, 12.08s/it]
30%|██████████▍ | 327/1101 [1:05:05<2:34:20, 11.96s/it]
30%|██████████▍ | 328/1101 [1:05:16<2:33:50, 11.94s/it]
30%|██████████▍ | 329/1101 [1:05:29<2:34:30, 12.01s/it]
30%|██████████▍ | 330/1101 [1:05:41<2:34:05, 11.99s/it]
{'loss': '1.38', 'grad_norm': '1.383', 'learning_rate': '3.76e-06', 'entropy': '1.339', 'num_tokens': '2.144e+07', 'mean_token_accuracy': '0.683', 'epoch': '0.8995'} |
|
30%|██████████▍ | 330/1101 [1:05:41<2:34:05, 11.99s/it]
30%|██████████▌ | 331/1101 [1:05:52<2:32:36, 11.89s/it]
30%|██████████▌ | 332/1101 [1:06:04<2:32:36, 11.91s/it]
30%|██████████▌ | 333/1101 [1:06:16<2:33:25, 11.99s/it]
30%|██████████▌ | 334/1101 [1:06:28<2:32:58, 11.97s/it]
30%|██████████▋ | 335/1101 [1:06:40<2:32:37, 11.96s/it]
31%|██████████▋ | 336/1101 [1:06:52<2:32:52, 11.99s/it]
31%|██████████▋ | 337/1101 [1:07:04<2:32:36, 11.98s/it]
31%|██████████▋ | 338/1101 [1:07:16<2:32:59, 12.03s/it]
31%|██████████▊ | 339/1101 [1:07:28<2:32:26, 12.00s/it]
31%|██████████▊ | 340/1101 [1:07:40<2:32:43, 12.04s/it]
{'loss': '1.38', 'grad_norm': '1.281', 'learning_rate': '3.874e-06', 'entropy': '1.342', 'num_tokens': '2.209e+07', 'mean_token_accuracy': '0.6807', 'epoch': '0.9267'} |
|
31%|██████████▊ | 340/1101 [1:07:41<2:32:43, 12.04s/it]
31%|██████████▊ | 341/1101 [1:07:52<2:32:06, 12.01s/it]
31%|██████████▊ | 342/1101 [1:08:04<2:31:48, 12.00s/it]
31%|██████████▉ | 343/1101 [1:08:16<2:31:57, 12.03s/it]
31%|██████████▉ | 344/1101 [1:08:28<2:31:08, 11.98s/it]
31%|██████████▉ | 345/1101 [1:08:40<2:31:16, 12.01s/it]
31%|██████████▉ | 346/1101 [1:08:52<2:30:38, 11.97s/it]
32%|███████████ | 347/1101 [1:09:04<2:31:05, 12.02s/it]
32%|███████████ | 348/1101 [1:09:16<2:30:19, 11.98s/it]
32%|███████████ | 349/1101 [1:09:28<2:29:57, 11.96s/it]
32%|███████████▏ | 350/1101 [1:09:40<2:28:32, 11.87s/it]
{'loss': '1.342', 'grad_norm': '1.305', 'learning_rate': '3.989e-06', 'entropy': '1.312', 'num_tokens': '2.274e+07', 'mean_token_accuracy': '0.6888', 'epoch': '0.954'} |
|
32%|███████████▏ | 350/1101 [1:09:40<2:28:32, 11.87s/it]
32%|███████████▏ | 351/1101 [1:09:51<2:27:11, 11.78s/it]
32%|███████████▏ | 352/1101 [1:10:04<2:28:10, 11.87s/it]
32%|███████████▏ | 353/1101 [1:10:15<2:28:11, 11.89s/it]
32%|███████████▎ | 354/1101 [1:10:27<2:27:01, 11.81s/it]
32%|███████████▎ | 355/1101 [1:10:39<2:27:01, 11.83s/it]
32%|███████████▎ | 356/1101 [1:10:51<2:27:20, 11.87s/it]
32%|███████████▎ | 357/1101 [1:11:03<2:28:03, 11.94s/it]
33%|███████████▍ | 358/1101 [1:11:15<2:27:48, 11.94s/it]
33%|███████████▍ | 359/1101 [1:11:27<2:27:29, 11.93s/it]
33%|███████████▍ | 360/1101 [1:11:38<2:25:08, 11.75s/it]
{'loss': '1.344', 'grad_norm': '1.453', 'learning_rate': '4.103e-06', 'entropy': '1.32', 'num_tokens': '2.339e+07', 'mean_token_accuracy': '0.688', 'epoch': '0.9813'} |
|
33%|███████████▍ | 360/1101 [1:11:38<2:25:08, 11.75s/it]
33%|███████████▍ | 361/1101 [1:11:50<2:26:32, 11.88s/it]
33%|███████████▌ | 362/1101 [1:12:02<2:24:54, 11.77s/it]
33%|███████████▌ | 363/1101 [1:12:14<2:26:06, 11.88s/it]
33%|███████████▌ | 364/1101 [1:12:26<2:25:59, 11.89s/it]
33%|███████████▌ | 365/1101 [1:12:38<2:26:00, 11.90s/it]
33%|███████████▋ | 366/1101 [1:12:50<2:26:44, 11.98s/it]
33%|███████████▋ | 367/1101 [1:13:00<2:20:51, 11.51s/it] |
|
Writing model shards: 0%| | 0/1 [00:00<?, ?it/s][A |
|
Writing model shards: 100%|██████████████████████| 1/1 [00:12<00:00, 12.65s/it][A
Writing model shards: 100%|██████████████████████| 1/1 [00:12<00:00, 12.65s/it] |
|
33%|███████████▋ | 368/1101 [1:13:39<3:59:16, 19.59s/it]
34%|███████████▋ | 369/1101 [1:13:51<3:30:49, 17.28s/it]
34%|███████████▊ | 370/1101 [1:14:03<3:12:34, 15.81s/it]
{'loss': '1.347', 'grad_norm': '1.148', 'learning_rate': '4.217e-06', 'entropy': '1.327', 'num_tokens': '2.403e+07', 'mean_token_accuracy': '0.6872', 'epoch': '1.008'} |
|
34%|███████████▊ | 370/1101 [1:14:03<3:12:34, 15.81s/it]
34%|███████████▊ | 371/1101 [1:14:15<2:58:11, 14.65s/it]
34%|███████████▊ | 372/1101 [1:14:27<2:48:14, 13.85s/it]
34%|███████████▊ | 373/1101 [1:14:39<2:41:24, 13.30s/it]
34%|███████████▉ | 374/1101 [1:14:50<2:34:10, 12.72s/it]
34%|███████████▉ | 375/1101 [1:15:03<2:31:43, 12.54s/it]
34%|███████████▉ | 376/1101 [1:15:15<2:29:19, 12.36s/it]
34%|███████████▉ | 377/1101 [1:15:27<2:28:18, 12.29s/it]
34%|████████████ | 378/1101 [1:15:39<2:26:48, 12.18s/it]
34%|████████████ | 379/1101 [1:15:50<2:25:02, 12.05s/it]
35%|████████████ | 380/1101 [1:16:02<2:25:01, 12.07s/it]
{'loss': '1.347', 'grad_norm': '1.086', 'learning_rate': '4.331e-06', 'entropy': '1.33', 'num_tokens': '2.468e+07', 'mean_token_accuracy': '0.6895', 'epoch': '1.035'} |
|
35%|████████████ | 380/1101 [1:16:02<2:25:01, 12.07s/it]
35%|████████████ | 381/1101 [1:16:14<2:24:21, 12.03s/it]
35%|████████████▏ | 382/1101 [1:16:27<2:24:45, 12.08s/it]
35%|████████████▏ | 383/1101 [1:16:38<2:23:52, 12.02s/it]
35%|████████████▏ | 384/1101 [1:16:51<2:24:21, 12.08s/it]
35%|████████████▏ | 385/1101 [1:17:03<2:23:29, 12.02s/it]
35%|████████████▎ | 386/1101 [1:17:15<2:22:57, 12.00s/it]
35%|████████████▎ | 387/1101 [1:17:27<2:23:37, 12.07s/it]
35%|████████████▎ | 388/1101 [1:17:38<2:21:23, 11.90s/it]
35%|████████████▎ | 389/1101 [1:17:50<2:22:22, 12.00s/it]
35%|████████████▍ | 390/1101 [1:18:02<2:20:10, 11.83s/it]
{'loss': '1.309', 'grad_norm': '1.375', 'learning_rate': '4.446e-06', 'entropy': '1.289', 'num_tokens': '2.533e+07', 'mean_token_accuracy': '0.6927', 'epoch': '1.063'} |
|
35%|████████████▍ | 390/1101 [1:18:02<2:20:10, 11.83s/it]
36%|████████████▍ | 391/1101 [1:18:14<2:19:51, 11.82s/it]
36%|████████████▍ | 392/1101 [1:18:26<2:19:55, 11.84s/it]
36%|████████████▍ | 393/1101 [1:18:38<2:19:58, 11.86s/it]
36%|████████████▌ | 394/1101 [1:18:49<2:19:19, 11.82s/it]
36%|████████████▌ | 395/1101 [1:19:01<2:19:20, 11.84s/it]
36%|████████████▌ | 396/1101 [1:19:13<2:20:32, 11.96s/it]
36%|████████████▌ | 397/1101 [1:19:25<2:20:01, 11.93s/it]
36%|████████████▋ | 398/1101 [1:19:37<2:20:47, 12.02s/it]
36%|████████████▋ | 399/1101 [1:19:49<2:20:28, 12.01s/it]
36%|████████████▋ | 400/1101 [1:20:02<2:20:52, 12.06s/it]
{'loss': '1.342', 'grad_norm': '1.102', 'learning_rate': '4.56e-06', 'entropy': '1.326', 'num_tokens': '2.598e+07', 'mean_token_accuracy': '0.6882', 'epoch': '1.09'} |
|
36%|████████████▋ | 400/1101 [1:20:02<2:20:52, 12.06s/it]
36%|████████████▋ | 401/1101 [1:20:13<2:18:36, 11.88s/it]
37%|████████████▊ | 402/1101 [1:20:25<2:18:27, 11.89s/it]
37%|████████████▊ | 403/1101 [1:20:37<2:19:12, 11.97s/it]
37%|████████████▊ | 404/1101 [1:20:49<2:18:32, 11.93s/it]
37%|████████████▊ | 405/1101 [1:21:01<2:19:08, 11.99s/it]
37%|████████████▉ | 406/1101 [1:21:13<2:16:52, 11.82s/it]
37%|████████████▉ | 407/1101 [1:21:24<2:15:57, 11.75s/it]
37%|████████████▉ | 408/1101 [1:21:36<2:14:38, 11.66s/it]
37%|█████████████ | 409/1101 [1:21:47<2:15:18, 11.73s/it]
37%|█████████████ | 410/1101 [1:22:00<2:16:54, 11.89s/it]
{'loss': '1.323', 'grad_norm': '1.031', 'learning_rate': '4.674e-06', 'entropy': '1.308', 'num_tokens': '2.662e+07', 'mean_token_accuracy': '0.6947', 'epoch': '1.117'} |
|
37%|█████████████ | 410/1101 [1:22:00<2:16:54, 11.89s/it]
37%|█████████████ | 411/1101 [1:22:12<2:16:49, 11.90s/it]
37%|█████████████ | 412/1101 [1:22:24<2:17:57, 12.01s/it]
38%|█████████████▏ | 413/1101 [1:22:36<2:17:16, 11.97s/it]
38%|█████████████▏ | 414/1101 [1:22:48<2:17:43, 12.03s/it]
38%|█████████████▏ | 415/1101 [1:23:00<2:17:16, 12.01s/it]
38%|█████████████▏ | 416/1101 [1:23:12<2:16:45, 11.98s/it]
38%|█████████████▎ | 417/1101 [1:23:24<2:17:31, 12.06s/it]
38%|█████████████▎ | 418/1101 [1:23:36<2:16:43, 12.01s/it]
38%|█████████████▎ | 419/1101 [1:23:48<2:16:03, 11.97s/it]
38%|█████████████▎ | 420/1101 [1:24:00<2:15:37, 11.95s/it]
{'loss': '1.312', 'grad_norm': '1.062', 'learning_rate': '4.789e-06', 'entropy': '1.301', 'num_tokens': '2.727e+07', 'mean_token_accuracy': '0.6931', 'epoch': '1.144'} |
|
38%|█████████████▎ | 420/1101 [1:24:00<2:15:37, 11.95s/it]
38%|█████████████▍ | 421/1101 [1:24:12<2:16:20, 12.03s/it]
38%|█████████████▍ | 422/1101 [1:24:23<2:14:11, 11.86s/it]
38%|█████████████▍ | 423/1101 [1:24:35<2:12:15, 11.70s/it]
39%|█████████████▍ | 424/1101 [1:24:47<2:13:40, 11.85s/it]
39%|█████████████▌ | 425/1101 [1:24:59<2:13:46, 11.87s/it]
39%|█████████████▌ | 426/1101 [1:25:11<2:15:06, 12.01s/it]
39%|█████████████▌ | 427/1101 [1:25:23<2:14:41, 11.99s/it]
39%|█████████████▌ | 428/1101 [1:25:35<2:15:20, 12.07s/it]
39%|█████████████▋ | 429/1101 [1:25:47<2:14:39, 12.02s/it]
39%|█████████████▋ | 430/1101 [1:25:59<2:13:47, 11.96s/it]
{'loss': '1.318', 'grad_norm': '1.375', 'learning_rate': '4.903e-06', 'entropy': '1.303', 'num_tokens': '2.792e+07', 'mean_token_accuracy': '0.6949', 'epoch': '1.172'} |
|
39%|█████████████▋ | 430/1101 [1:25:59<2:13:47, 11.96s/it]
39%|█████████████▋ | 431/1101 [1:26:11<2:14:43, 12.06s/it]
39%|█████████████▋ | 432/1101 [1:26:23<2:14:04, 12.02s/it]
39%|█████████████▊ | 433/1101 [1:26:36<2:14:34, 12.09s/it]
39%|█████████████▊ | 434/1101 [1:26:48<2:13:41, 12.03s/it]
40%|█████████████▊ | 435/1101 [1:27:00<2:14:00, 12.07s/it]
40%|█████████████▊ | 436/1101 [1:27:12<2:13:13, 12.02s/it]
40%|█████████████▉ | 437/1101 [1:27:24<2:13:33, 12.07s/it]
40%|█████████████▉ | 438/1101 [1:27:35<2:12:08, 11.96s/it]
40%|█████████████▉ | 439/1101 [1:27:47<2:11:43, 11.94s/it]
40%|█████████████▉ | 440/1101 [1:28:00<2:12:26, 12.02s/it]
{'loss': '1.289', 'grad_norm': '1.234', 'learning_rate': '5.017e-06', 'entropy': '1.276', 'num_tokens': '2.857e+07', 'mean_token_accuracy': '0.6952', 'epoch': '1.199'} |
|
40%|█████████████▉ | 440/1101 [1:28:00<2:12:26, 12.02s/it]
40%|██████████████ | 441/1101 [1:28:11<2:11:49, 11.98s/it]
40%|██████████████ | 442/1101 [1:28:24<2:12:22, 12.05s/it]
40%|██████████████ | 443/1101 [1:28:36<2:11:34, 12.00s/it]
40%|██████████████ | 444/1101 [1:28:48<2:12:14, 12.08s/it]
40%|██████████████▏ | 445/1101 [1:28:59<2:09:45, 11.87s/it]
41%|██████████████▏ | 446/1101 [1:29:11<2:09:52, 11.90s/it]
41%|██████████████▏ | 447/1101 [1:29:23<2:10:50, 12.00s/it]
41%|██████████████▏ | 448/1101 [1:29:35<2:10:39, 12.00s/it]
41%|██████████████▎ | 449/1101 [1:29:48<2:11:06, 12.07s/it]
41%|██████████████▎ | 450/1101 [1:30:00<2:10:17, 12.01s/it]
{'loss': '1.25', 'grad_norm': '1.07', 'learning_rate': '5.131e-06', 'entropy': '1.242', 'num_tokens': '2.922e+07', 'mean_token_accuracy': '0.701', 'epoch': '1.226'} |
|
41%|██████████████▎ | 450/1101 [1:30:00<2:10:17, 12.01s/it]
41%|██████████████▎ | 451/1101 [1:30:12<2:10:35, 12.06s/it]
41%|██████████████▎ | 452/1101 [1:30:23<2:08:34, 11.89s/it]
41%|██████████████▍ | 453/1101 [1:30:35<2:08:30, 11.90s/it]
41%|██████████████▍ | 454/1101 [1:30:47<2:09:11, 11.98s/it]
41%|██████████████▍ | 455/1101 [1:30:59<2:07:12, 11.81s/it]
41%|██████████████▍ | 456/1101 [1:31:11<2:08:10, 11.92s/it]
42%|██████████████▌ | 457/1101 [1:31:23<2:08:00, 11.93s/it]
42%|██████████████▌ | 458/1101 [1:31:35<2:08:41, 12.01s/it]
42%|██████████████▌ | 459/1101 [1:31:47<2:08:30, 12.01s/it]
42%|██████████████▌ | 460/1101 [1:31:59<2:08:19, 12.01s/it]
{'loss': '1.269', 'grad_norm': '1.703', 'learning_rate': '5.246e-06', 'entropy': '1.257', 'num_tokens': '2.987e+07', 'mean_token_accuracy': '0.6976', 'epoch': '1.253'} |
|
42%|██████████████▌ | 460/1101 [1:31:59<2:08:19, 12.01s/it]
42%|██████████████▋ | 461/1101 [1:32:11<2:08:42, 12.07s/it]
42%|██████████████▋ | 462/1101 [1:32:23<2:07:54, 12.01s/it]
42%|██████████████▋ | 463/1101 [1:32:35<2:07:39, 12.01s/it]
42%|██████████████▊ | 464/1101 [1:32:47<2:06:29, 11.91s/it]
42%|██████████████▊ | 465/1101 [1:32:59<2:05:49, 11.87s/it]
42%|██████████████▊ | 466/1101 [1:33:10<2:04:29, 11.76s/it]
42%|██████████████▊ | 467/1101 [1:33:22<2:04:42, 11.80s/it]
43%|██████████████▉ | 468/1101 [1:33:34<2:05:56, 11.94s/it]
43%|██████████████▉ | 469/1101 [1:33:46<2:05:25, 11.91s/it]
43%|██████████████▉ | 470/1101 [1:33:58<2:06:13, 12.00s/it]
{'loss': '1.287', 'grad_norm': '1.344', 'learning_rate': '5.36e-06', 'entropy': '1.281', 'num_tokens': '3.052e+07', 'mean_token_accuracy': '0.6993', 'epoch': '1.281'} |
|
43%|██████████████▉ | 470/1101 [1:33:58<2:06:13, 12.00s/it]
43%|██████████████▉ | 471/1101 [1:34:10<2:06:01, 12.00s/it]
43%|███████████████ | 472/1101 [1:34:23<2:06:47, 12.10s/it]
43%|███████████████ | 473/1101 [1:34:35<2:06:13, 12.06s/it]
43%|███████████████ | 474/1101 [1:34:46<2:05:27, 12.01s/it]
43%|███████████████ | 475/1101 [1:34:58<2:05:02, 11.98s/it]
43%|███████████████▏ | 476/1101 [1:35:10<2:04:31, 11.95s/it]
43%|███████████████▏ | 477/1101 [1:35:22<2:04:55, 12.01s/it]
43%|███████████████▏ | 478/1101 [1:35:34<2:04:26, 11.98s/it]
44%|███████████████▏ | 479/1101 [1:35:47<2:04:55, 12.05s/it]
44%|███████████████▎ | 480/1101 [1:35:58<2:02:41, 11.85s/it]
{'loss': '1.268', 'grad_norm': '1.172', 'learning_rate': '5.474e-06', 'entropy': '1.258', 'num_tokens': '3.117e+07', 'mean_token_accuracy': '0.6998', 'epoch': '1.308'} |
|
44%|███████████████▎ | 480/1101 [1:35:58<2:02:41, 11.85s/it]
44%|███████████████▎ | 481/1101 [1:36:10<2:02:50, 11.89s/it]
44%|███████████████▎ | 482/1101 [1:36:21<2:00:34, 11.69s/it]
44%|███████████████▎ | 483/1101 [1:36:33<2:01:10, 11.76s/it]
44%|███████████████▍ | 484/1101 [1:36:45<2:02:15, 11.89s/it]
44%|███████████████▍ | 485/1101 [1:36:57<2:01:59, 11.88s/it]
44%|███████████████▍ | 486/1101 [1:37:09<2:02:48, 11.98s/it]
44%|███████████████▍ | 487/1101 [1:37:21<2:02:15, 11.95s/it]
44%|███████████████▌ | 488/1101 [1:37:33<2:02:38, 12.00s/it]
44%|███████████████▌ | 489/1101 [1:37:45<2:01:54, 11.95s/it]
45%|███████████████▌ | 490/1101 [1:37:57<2:01:16, 11.91s/it]
{'loss': '1.297', 'grad_norm': '1.094', 'learning_rate': '5.589e-06', 'entropy': '1.282', 'num_tokens': '3.182e+07', 'mean_token_accuracy': '0.6981', 'epoch': '1.335'} |
|
45%|███████████████▌ | 490/1101 [1:37:57<2:01:16, 11.91s/it]
45%|███████████████▌ | 491/1101 [1:38:09<2:02:00, 12.00s/it]
45%|███████████████▋ | 492/1101 [1:38:21<2:01:26, 11.96s/it]
45%|███████████████▋ | 493/1101 [1:38:33<2:01:46, 12.02s/it]
45%|███████████████▋ | 494/1101 [1:38:45<2:00:12, 11.88s/it]
45%|███████████████▋ | 495/1101 [1:38:57<2:00:46, 11.96s/it]
45%|███████████████▊ | 496/1101 [1:39:09<2:00:33, 11.96s/it]
45%|███████████████▊ | 497/1101 [1:39:21<2:00:01, 11.92s/it]
45%|███████████████▊ | 498/1101 [1:39:33<2:00:17, 11.97s/it]
45%|███████████████▊ | 499/1101 [1:39:45<1:59:40, 11.93s/it]
45%|███████████████▉ | 500/1101 [1:39:57<2:00:20, 12.01s/it]
{'loss': '1.31', 'grad_norm': '1.062', 'learning_rate': '5.703e-06', 'entropy': '1.296', 'num_tokens': '3.247e+07', 'mean_token_accuracy': '0.6965', 'epoch': '1.363'} |
|
45%|███████████████▉ | 500/1101 [1:39:57<2:00:20, 12.01s/it]
46%|███████████████▉ | 501/1101 [1:40:08<1:58:27, 11.85s/it]
46%|███████████████▉ | 502/1101 [1:40:20<1:57:52, 11.81s/it]
46%|███████████████▉ | 503/1101 [1:40:32<1:58:08, 11.85s/it]
46%|████████████████ | 504/1101 [1:40:43<1:56:49, 11.74s/it]
46%|████████████████ | 505/1101 [1:40:56<1:57:41, 11.85s/it]
46%|████████████████ | 506/1101 [1:41:07<1:57:09, 11.81s/it]
46%|████████████████ | 507/1101 [1:41:20<1:58:14, 11.94s/it]
46%|████████████████▏ | 508/1101 [1:41:31<1:58:07, 11.95s/it]
46%|████████████████▏ | 509/1101 [1:41:44<1:58:38, 12.02s/it]
46%|████████████████▏ | 510/1101 [1:41:56<1:58:13, 12.00s/it]
{'loss': '1.265', 'grad_norm': '1.047', 'learning_rate': '5.817e-06', 'entropy': '1.256', 'num_tokens': '3.312e+07', 'mean_token_accuracy': '0.702', 'epoch': '1.39'} |
|
46%|████████████████▏ | 510/1101 [1:41:56<1:58:13, 12.00s/it]
46%|████████████████▏ | 511/1101 [1:42:08<1:57:53, 11.99s/it]
47%|████████████████▎ | 512/1101 [1:42:20<1:57:29, 11.97s/it]
47%|████████████████▎ | 513/1101 [1:42:32<1:57:21, 11.97s/it]
47%|████████████████▎ | 514/1101 [1:42:44<1:57:51, 12.05s/it]
47%|████████████████▎ | 515/1101 [1:42:56<1:57:01, 11.98s/it]
47%|████████████████▍ | 516/1101 [1:43:08<1:57:17, 12.03s/it]
47%|████████████████▍ | 517/1101 [1:43:20<1:56:52, 12.01s/it]
47%|████████████████▍ | 518/1101 [1:43:32<1:57:08, 12.06s/it]
47%|████████████████▍ | 519/1101 [1:43:44<1:56:21, 11.99s/it]
47%|████████████████▌ | 520/1101 [1:43:56<1:55:53, 11.97s/it]
{'loss': '1.279', 'grad_norm': '1.219', 'learning_rate': '5.931e-06', 'entropy': '1.27', 'num_tokens': '3.377e+07', 'mean_token_accuracy': '0.7015', 'epoch': '1.417'} |
|
47%|████████████████▌ | 520/1101 [1:43:56<1:55:53, 11.97s/it]
47%|████████████████▌ | 521/1101 [1:44:08<1:56:04, 12.01s/it]
47%|████████████████▌ | 522/1101 [1:44:20<1:55:32, 11.97s/it]
48%|████████████████▋ | 523/1101 [1:44:32<1:55:28, 11.99s/it]
48%|████████████████▋ | 524/1101 [1:44:43<1:53:25, 11.79s/it]
48%|████████████████▋ | 525/1101 [1:44:55<1:54:18, 11.91s/it]
48%|████████████████▋ | 526/1101 [1:45:07<1:53:24, 11.83s/it]
48%|████████████████▊ | 527/1101 [1:45:18<1:52:24, 11.75s/it]
48%|████████████████▊ | 528/1101 [1:45:31<1:53:41, 11.90s/it]
48%|████████████████▊ | 529/1101 [1:45:42<1:53:00, 11.85s/it]
48%|████████████████▊ | 530/1101 [1:45:55<1:54:04, 11.99s/it]
{'loss': '1.272', 'grad_norm': '1.109', 'learning_rate': '6.046e-06', 'entropy': '1.265', 'num_tokens': '3.442e+07', 'mean_token_accuracy': '0.7005', 'epoch': '1.444'} |
|
48%|████████████████▊ | 530/1101 [1:45:55<1:54:04, 11.99s/it]
48%|████████████████▉ | 531/1101 [1:46:06<1:52:39, 11.86s/it]
48%|████████████████▉ | 532/1101 [1:46:18<1:53:29, 11.97s/it]
48%|████████████████▉ | 533/1101 [1:46:30<1:53:03, 11.94s/it]
49%|████████████████▉ | 534/1101 [1:46:42<1:52:52, 11.94s/it]
49%|█████████████████ | 535/1101 [1:46:54<1:53:31, 12.04s/it]
49%|█████████████████ | 536/1101 [1:47:06<1:52:54, 11.99s/it]
49%|█████████████████ | 537/1101 [1:47:19<1:53:22, 12.06s/it]
49%|█████████████████ | 538/1101 [1:47:31<1:52:46, 12.02s/it]
49%|█████████████████▏ | 539/1101 [1:47:43<1:53:24, 12.11s/it]
49%|█████████████████▏ | 540/1101 [1:47:55<1:52:36, 12.04s/it]
{'loss': '1.253', 'grad_norm': '1.281', 'learning_rate': '6.16e-06', 'entropy': '1.251', 'num_tokens': '3.507e+07', 'mean_token_accuracy': '0.7026', 'epoch': '1.472'} |
|
49%|█████████████████▏ | 540/1101 [1:47:55<1:52:36, 12.04s/it]
49%|█████████████████▏ | 541/1101 [1:48:06<1:50:51, 11.88s/it]
49%|█████████████████▏ | 542/1101 [1:48:18<1:51:38, 11.98s/it]
49%|█████████████████▎ | 543/1101 [1:48:30<1:51:13, 11.96s/it]
49%|█████████████████▎ | 544/1101 [1:48:43<1:51:44, 12.04s/it]
50%|█████████████████▎ | 545/1101 [1:48:55<1:51:25, 12.02s/it]
50%|█████████████████▎ | 546/1101 [1:49:07<1:51:35, 12.06s/it]
50%|█████████████████▍ | 547/1101 [1:49:19<1:51:03, 12.03s/it]
50%|█████████████████▍ | 548/1101 [1:49:31<1:51:18, 12.08s/it]
50%|█████████████████▍ | 549/1101 [1:49:43<1:50:29, 12.01s/it]
50%|█████████████████▍ | 550/1101 [1:49:55<1:50:04, 11.99s/it]
{'loss': '1.291', 'grad_norm': '1.07', 'learning_rate': '6.274e-06', 'entropy': '1.283', 'num_tokens': '3.572e+07', 'mean_token_accuracy': '0.7015', 'epoch': '1.499'} |
|
50%|█████████████████▍ | 550/1101 [1:49:55<1:50:04, 11.99s/it]
50%|█████████████████▌ | 551/1101 [1:50:07<1:49:43, 11.97s/it]
50%|█████████████████▌ | 552/1101 [1:50:19<1:49:32, 11.97s/it]
50%|█████████████████▌ | 553/1101 [1:50:31<1:49:47, 12.02s/it]
50%|█████████████████▌ | 554/1101 [1:50:42<1:48:34, 11.91s/it]
50%|█████████████████▋ | 555/1101 [1:50:54<1:49:01, 11.98s/it]
50%|█████████████████▋ | 556/1101 [1:51:06<1:48:40, 11.96s/it]
51%|█████████████████▋ | 557/1101 [1:51:18<1:48:04, 11.92s/it]
51%|█████████████████▋ | 558/1101 [1:51:30<1:48:25, 11.98s/it]
51%|█████████████████▊ | 559/1101 [1:51:42<1:47:58, 11.95s/it]
51%|█████████████████▊ | 560/1101 [1:51:54<1:48:27, 12.03s/it]
{'loss': '1.257', 'grad_norm': '1.117', 'learning_rate': '6.389e-06', 'entropy': '1.248', 'num_tokens': '3.638e+07', 'mean_token_accuracy': '0.7024', 'epoch': '1.526'} |
|
51%|█████████████████▊ | 560/1101 [1:51:54<1:48:27, 12.03s/it]
51%|█████████████████▊ | 561/1101 [1:52:06<1:46:23, 11.82s/it]
51%|█████████████████▊ | 562/1101 [1:52:18<1:47:13, 11.94s/it]
51%|█████████████████▉ | 563/1101 [1:52:30<1:46:48, 11.91s/it]
51%|█████████████████▉ | 564/1101 [1:52:42<1:46:43, 11.92s/it]
51%|█████████████████▉ | 565/1101 [1:52:54<1:47:24, 12.02s/it]
51%|█████████████████▉ | 566/1101 [1:53:06<1:47:01, 12.00s/it]
51%|██████████████████ | 567/1101 [1:53:18<1:47:24, 12.07s/it]
52%|██████████████████ | 568/1101 [1:53:30<1:46:50, 12.03s/it]
52%|██████████████████ | 569/1101 [1:53:42<1:47:06, 12.08s/it]
52%|██████████████████ | 570/1101 [1:53:54<1:46:34, 12.04s/it]
{'loss': '1.263', 'grad_norm': '1.086', 'learning_rate': '6.503e-06', 'entropy': '1.258', 'num_tokens': '3.703e+07', 'mean_token_accuracy': '0.7022', 'epoch': '1.553'} |
|
52%|██████████████████ | 570/1101 [1:53:54<1:46:34, 12.04s/it]
52%|██████████████████▏ | 571/1101 [1:54:06<1:46:07, 12.01s/it]
52%|██████████████████▏ | 572/1101 [1:54:18<1:46:32, 12.08s/it]
52%|██████████████████▏ | 573/1101 [1:54:30<1:45:55, 12.04s/it]
52%|██████████████████▏ | 574/1101 [1:54:42<1:45:40, 12.03s/it]
52%|██████████████████▎ | 575/1101 [1:54:54<1:45:17, 12.01s/it]
52%|██████████████████▎ | 576/1101 [1:55:07<1:45:37, 12.07s/it]
52%|██████████████████▎ | 577/1101 [1:55:18<1:44:51, 12.01s/it]
52%|██████████████████▎ | 578/1101 [1:55:30<1:43:12, 11.84s/it]
53%|██████████████████▍ | 579/1101 [1:55:42<1:43:48, 11.93s/it]
53%|██████████████████▍ | 580/1101 [1:55:54<1:43:32, 11.93s/it]
{'loss': '1.24', 'grad_norm': '1.07', 'learning_rate': '6.617e-06', 'entropy': '1.235', 'num_tokens': '3.768e+07', 'mean_token_accuracy': '0.7057', 'epoch': '1.581'} |
|
53%|██████████████████▍ | 580/1101 [1:55:54<1:43:32, 11.93s/it]
53%|██████████████████▍ | 581/1101 [1:56:06<1:44:04, 12.01s/it]
53%|██████████████████▌ | 582/1101 [1:56:18<1:43:41, 11.99s/it]
53%|██████████████████▌ | 583/1101 [1:56:30<1:43:52, 12.03s/it]
53%|██████████████████▌ | 584/1101 [1:56:42<1:43:22, 12.00s/it]
53%|██████████████████▌ | 585/1101 [1:56:54<1:43:34, 12.04s/it]
53%|██████████████████▋ | 586/1101 [1:57:06<1:42:57, 11.99s/it]
53%|██████████████████▋ | 587/1101 [1:57:18<1:42:31, 11.97s/it]
53%|██████████████████▋ | 588/1101 [1:57:30<1:41:40, 11.89s/it]
53%|██████████████████▋ | 589/1101 [1:57:41<1:40:09, 11.74s/it]
54%|██████████████████▊ | 590/1101 [1:57:53<1:41:03, 11.87s/it]
{'loss': '1.265', 'grad_norm': '1.031', 'learning_rate': '6.731e-06', 'entropy': '1.262', 'num_tokens': '3.833e+07', 'mean_token_accuracy': '0.7034', 'epoch': '1.608'} |
|
54%|██████████████████▊ | 590/1101 [1:57:53<1:41:03, 11.87s/it]
54%|██████████████████▊ | 591/1101 [1:58:05<1:39:37, 11.72s/it]
54%|██████████████████▊ | 592/1101 [1:58:17<1:40:26, 11.84s/it]
54%|██████████████████▊ | 593/1101 [1:58:29<1:40:28, 11.87s/it]
54%|██████████████████▉ | 594/1101 [1:58:41<1:40:12, 11.86s/it]
54%|██████████████████▉ | 595/1101 [1:58:53<1:40:10, 11.88s/it]
54%|██████████████████▉ | 596/1101 [1:59:04<1:40:01, 11.88s/it]
54%|██████████████████▉ | 597/1101 [1:59:17<1:40:26, 11.96s/it]
54%|███████████████████ | 598/1101 [1:59:28<1:40:04, 11.94s/it]
54%|███████████████████ | 599/1101 [1:59:41<1:40:23, 12.00s/it]
54%|███████████████████ | 600/1101 [1:59:53<1:39:59, 11.98s/it]
{'loss': '1.278', 'grad_norm': '1.477', 'learning_rate': '6.846e-06', 'entropy': '1.272', 'num_tokens': '3.898e+07', 'mean_token_accuracy': '0.7013', 'epoch': '1.635'} |
|
54%|███████████████████ | 600/1101 [1:59:53<1:39:59, 11.98s/it]
55%|███████████████████ | 601/1101 [2:00:05<1:39:54, 11.99s/it]
55%|███████████████████▏ | 602/1101 [2:00:17<1:40:14, 12.05s/it]
55%|███████████████████▏ | 603/1101 [2:00:29<1:39:47, 12.02s/it]
55%|███████████████████▏ | 604/1101 [2:00:41<1:39:57, 12.07s/it]
55%|███████████████████▏ | 605/1101 [2:00:53<1:39:26, 12.03s/it]
55%|███████████████████▎ | 606/1101 [2:01:05<1:39:47, 12.10s/it]
55%|███████████████████▎ | 607/1101 [2:01:17<1:38:28, 11.96s/it]
55%|███████████████████▎ | 608/1101 [2:01:28<1:36:46, 11.78s/it]
55%|███████████████████▎ | 609/1101 [2:01:40<1:36:07, 11.72s/it]
55%|███████████████████▍ | 610/1101 [2:01:52<1:36:25, 11.78s/it]
{'loss': '1.228', 'grad_norm': '1.055', 'learning_rate': '6.96e-06', 'entropy': '1.221', 'num_tokens': '3.962e+07', 'mean_token_accuracy': '0.709', 'epoch': '1.662'} |
|
55%|███████████████████▍ | 610/1101 [2:01:52<1:36:25, 11.78s/it]
55%|███████████████████▍ | 611/1101 [2:02:04<1:37:15, 11.91s/it]
56%|███████████████████▍ | 612/1101 [2:02:16<1:37:03, 11.91s/it]
56%|███████████████████▍ | 613/1101 [2:02:28<1:37:39, 12.01s/it]
56%|███████████████████▌ | 614/1101 [2:02:40<1:37:16, 11.98s/it]
56%|███████████████████▌ | 615/1101 [2:02:52<1:36:50, 11.96s/it]
56%|███████████████████▌ | 616/1101 [2:03:04<1:37:11, 12.02s/it]
56%|███████████████████▌ | 617/1101 [2:03:16<1:36:53, 12.01s/it]
56%|███████████████████▋ | 618/1101 [2:03:28<1:37:25, 12.10s/it]
56%|███████████████████▋ | 619/1101 [2:03:40<1:36:55, 12.06s/it]
56%|███████████████████▋ | 620/1101 [2:03:52<1:37:06, 12.11s/it]
{'loss': '1.269', 'grad_norm': '1.055', 'learning_rate': '7.074e-06', 'entropy': '1.264', 'num_tokens': '4.028e+07', 'mean_token_accuracy': '0.7042', 'epoch': '1.69'} |
|
56%|███████████████████▋ | 620/1101 [2:03:52<1:37:06, 12.11s/it]
56%|███████████████████▋ | 621/1101 [2:04:04<1:36:36, 12.08s/it]
56%|███████████████████▊ | 622/1101 [2:04:17<1:36:59, 12.15s/it]
57%|███████████████████▊ | 623/1101 [2:04:29<1:36:11, 12.07s/it]
57%|███████████████████▊ | 624/1101 [2:04:40<1:34:29, 11.89s/it]
57%|███████████████████▊ | 625/1101 [2:04:52<1:35:10, 12.00s/it]
57%|███████████████████▉ | 626/1101 [2:05:04<1:34:47, 11.97s/it]
57%|███████████████████▉ | 627/1101 [2:05:16<1:34:54, 12.01s/it]
57%|███████████████████▉ | 628/1101 [2:05:28<1:34:20, 11.97s/it]
57%|███████████████████▉ | 629/1101 [2:05:39<1:32:21, 11.74s/it]
57%|████████████████████ | 630/1101 [2:05:51<1:31:37, 11.67s/it]
{'loss': '1.258', 'grad_norm': '1.125', 'learning_rate': '7.189e-06', 'entropy': '1.252', 'num_tokens': '4.092e+07', 'mean_token_accuracy': '0.7046', 'epoch': '1.717'} |
|
57%|████████████████████ | 630/1101 [2:05:51<1:31:37, 11.67s/it]
57%|████████████████████ | 631/1101 [2:06:02<1:30:50, 11.60s/it]
57%|████████████████████ | 632/1101 [2:06:15<1:32:01, 11.77s/it]
57%|████████████████████ | 633/1101 [2:06:26<1:31:10, 11.69s/it]
58%|████████████████████▏ | 634/1101 [2:06:38<1:32:10, 11.84s/it]
58%|████████████████████▏ | 635/1101 [2:06:50<1:31:06, 11.73s/it]
58%|████████████████████▏ | 636/1101 [2:07:02<1:31:59, 11.87s/it]
58%|████████████████████▏ | 637/1101 [2:07:14<1:31:52, 11.88s/it]
58%|████████████████████▎ | 638/1101 [2:07:26<1:31:38, 11.88s/it]
58%|████████████████████▎ | 639/1101 [2:07:38<1:32:15, 11.98s/it]
58%|████████████████████▎ | 640/1101 [2:07:50<1:31:51, 11.95s/it]
{'loss': '1.247', 'grad_norm': '1.07', 'learning_rate': '7.303e-06', 'entropy': '1.247', 'num_tokens': '4.157e+07', 'mean_token_accuracy': '0.7051', 'epoch': '1.744'} |
|
58%|████████████████████▎ | 640/1101 [2:07:50<1:31:51, 11.95s/it]
58%|████████████████████▍ | 641/1101 [2:08:02<1:32:02, 12.01s/it]
58%|████████████████████▍ | 642/1101 [2:08:13<1:30:21, 11.81s/it]
58%|████████████████████▍ | 643/1101 [2:08:25<1:30:53, 11.91s/it]
58%|████████████████████▍ | 644/1101 [2:08:37<1:30:32, 11.89s/it]
59%|████████████████████▌ | 645/1101 [2:08:49<1:30:21, 11.89s/it]
59%|████████████████████▌ | 646/1101 [2:09:01<1:30:37, 11.95s/it]
59%|████████████████████▌ | 647/1101 [2:09:13<1:30:28, 11.96s/it]
59%|████████████████████▌ | 648/1101 [2:09:25<1:30:35, 12.00s/it]
59%|████████████████████▋ | 649/1101 [2:09:37<1:28:49, 11.79s/it]
59%|████████████████████▋ | 650/1101 [2:09:49<1:29:25, 11.90s/it]
{'loss': '1.261', 'grad_norm': '1.344', 'learning_rate': '7.417e-06', 'entropy': '1.254', 'num_tokens': '4.222e+07', 'mean_token_accuracy': '0.7057', 'epoch': '1.771'} |
|
59%|████████████████████▋ | 650/1101 [2:09:49<1:29:25, 11.90s/it]
59%|████████████████████▋ | 651/1101 [2:10:01<1:29:12, 11.89s/it]
59%|████████████████████▋ | 652/1101 [2:10:13<1:29:02, 11.90s/it]
59%|████████████████████▊ | 653/1101 [2:10:25<1:29:22, 11.97s/it]
59%|████████████████████▊ | 654/1101 [2:10:37<1:29:02, 11.95s/it]
59%|████████████████████▊ | 655/1101 [2:10:48<1:27:45, 11.81s/it]
60%|████████████████████▊ | 656/1101 [2:11:00<1:27:41, 11.82s/it]
60%|████████████████████▉ | 657/1101 [2:11:12<1:28:11, 11.92s/it]
60%|████████████████████▉ | 658/1101 [2:11:24<1:27:58, 11.91s/it]
60%|████████████████████▉ | 659/1101 [2:11:36<1:28:17, 11.99s/it]
60%|████████████████████▉ | 660/1101 [2:11:48<1:27:59, 11.97s/it]
{'loss': '1.247', 'grad_norm': '1.086', 'learning_rate': '7.531e-06', 'entropy': '1.246', 'num_tokens': '4.286e+07', 'mean_token_accuracy': '0.7052', 'epoch': '1.799'} |
|
60%|████████████████████▉ | 660/1101 [2:11:48<1:27:59, 11.97s/it]
60%|█████████████████████ | 661/1101 [2:12:00<1:27:40, 11.96s/it]
60%|█████████████████████ | 662/1101 [2:12:12<1:27:47, 12.00s/it]
60%|█████████████████████ | 663/1101 [2:12:24<1:27:20, 11.97s/it]
60%|█████████████████████ | 664/1101 [2:12:36<1:27:32, 12.02s/it]
60%|█████████████████████▏ | 665/1101 [2:12:48<1:27:10, 12.00s/it]
60%|█████████████████████▏ | 666/1101 [2:13:00<1:27:49, 12.11s/it]
61%|█████████████████████▏ | 667/1101 [2:13:12<1:27:20, 12.08s/it]
61%|█████████████████████▏ | 668/1101 [2:13:24<1:26:52, 12.04s/it]
61%|█████████████████████▎ | 669/1101 [2:13:36<1:26:06, 11.96s/it]
61%|█████████████████████▎ | 670/1101 [2:13:48<1:25:51, 11.95s/it]
{'loss': '1.219', 'grad_norm': '1.008', 'learning_rate': '7.646e-06', 'entropy': '1.217', 'num_tokens': '4.352e+07', 'mean_token_accuracy': '0.708', 'epoch': '1.826'} |
|
61%|█████████████████████▎ | 670/1101 [2:13:48<1:25:51, 11.95s/it]
61%|█████████████████████▎ | 671/1101 [2:14:00<1:25:15, 11.90s/it]
61%|█████████████████████▎ | 672/1101 [2:14:12<1:25:03, 11.90s/it]
61%|█████████████████████▍ | 673/1101 [2:14:24<1:25:29, 11.99s/it]
61%|█████████████████████▍ | 674/1101 [2:14:35<1:24:06, 11.82s/it]
61%|█████████████████████▍ | 675/1101 [2:14:47<1:24:16, 11.87s/it]
61%|█████████████████████▍ | 676/1101 [2:15:00<1:24:46, 11.97s/it]
61%|█████████████████████▌ | 677/1101 [2:15:11<1:23:37, 11.83s/it]
62%|█████████████████████▌ | 678/1101 [2:15:23<1:24:09, 11.94s/it]
62%|█████████████████████▌ | 679/1101 [2:15:35<1:23:54, 11.93s/it]
62%|█████████████████████▌ | 680/1101 [2:15:47<1:24:06, 11.99s/it]
{'loss': '1.229', 'grad_norm': '1.086', 'learning_rate': '7.76e-06', 'entropy': '1.225', 'num_tokens': '4.416e+07', 'mean_token_accuracy': '0.7096', 'epoch': '1.853'} |
|
62%|█████████████████████▌ | 680/1101 [2:15:47<1:24:06, 11.99s/it]
62%|█████████████████████▋ | 681/1101 [2:15:59<1:22:43, 11.82s/it]
62%|█████████████████████▋ | 682/1101 [2:16:11<1:22:47, 11.86s/it]
62%|█████████████████████▋ | 683/1101 [2:16:23<1:23:10, 11.94s/it]
62%|█████████████████████▋ | 684/1101 [2:16:35<1:22:58, 11.94s/it]
62%|█████████████████████▊ | 685/1101 [2:16:47<1:23:17, 12.01s/it]
62%|█████████████████████▊ | 686/1101 [2:16:59<1:22:53, 11.98s/it]
62%|█████████████████████▊ | 687/1101 [2:17:11<1:22:57, 12.02s/it]
62%|█████████████████████▊ | 688/1101 [2:17:23<1:22:32, 11.99s/it]
63%|█████████████████████▉ | 689/1101 [2:17:35<1:22:10, 11.97s/it]
63%|█████████████████████▉ | 690/1101 [2:17:47<1:22:35, 12.06s/it]
{'loss': '1.248', 'grad_norm': '1.047', 'learning_rate': '7.874e-06', 'entropy': '1.245', 'num_tokens': '4.481e+07', 'mean_token_accuracy': '0.7061', 'epoch': '1.88'} |
|
63%|█████████████████████▉ | 690/1101 [2:17:47<1:22:35, 12.06s/it]
63%|█████████████████████▉ | 691/1101 [2:17:59<1:22:10, 12.02s/it]
63%|█████████████████████▉ | 692/1101 [2:18:11<1:22:08, 12.05s/it]
63%|██████████████████████ | 693/1101 [2:18:23<1:21:39, 12.01s/it]
63%|██████████████████████ | 694/1101 [2:18:35<1:21:50, 12.07s/it]
63%|██████████████████████ | 695/1101 [2:18:47<1:21:34, 12.06s/it]
63%|██████████████████████▏ | 696/1101 [2:18:59<1:21:25, 12.06s/it]
63%|██████████████████████▏ | 697/1101 [2:19:11<1:20:37, 11.97s/it]
63%|██████████████████████▏ | 698/1101 [2:19:23<1:20:24, 11.97s/it]
63%|██████████████████████▏ | 699/1101 [2:19:35<1:20:36, 12.03s/it]
64%|██████████████████████▎ | 700/1101 [2:19:47<1:19:47, 11.94s/it]
{'loss': '1.215', 'grad_norm': '0.9297', 'learning_rate': '7.989e-06', 'entropy': '1.21', 'num_tokens': '4.546e+07', 'mean_token_accuracy': '0.7115', 'epoch': '1.908'} |
|
64%|██████████████████████▎ | 700/1101 [2:19:47<1:19:47, 11.94s/it]
64%|██████████████████████▎ | 701/1101 [2:19:59<1:20:04, 12.01s/it]
64%|██████████████████████▎ | 702/1101 [2:20:11<1:19:43, 11.99s/it]
64%|██████████████████████▎ | 703/1101 [2:20:23<1:19:47, 12.03s/it]
64%|██████████████████████▍ | 704/1101 [2:20:35<1:18:29, 11.86s/it]
64%|██████████████████████▍ | 705/1101 [2:20:47<1:18:27, 11.89s/it]
64%|██████████████████████▍ | 706/1101 [2:20:59<1:18:46, 11.96s/it]
64%|██████████████████████▍ | 707/1101 [2:21:10<1:17:33, 11.81s/it]
64%|██████████████████████▌ | 708/1101 [2:21:22<1:18:04, 11.92s/it]
64%|██████████████████████▌ | 709/1101 [2:21:34<1:17:47, 11.91s/it]
64%|██████████████████████▌ | 710/1101 [2:21:46<1:18:03, 11.98s/it]
{'loss': '1.217', 'grad_norm': '0.9922', 'learning_rate': '8.103e-06', 'entropy': '1.211', 'num_tokens': '4.611e+07', 'mean_token_accuracy': '0.7112', 'epoch': '1.935'} |
|
64%|██████████████████████▌ | 710/1101 [2:21:46<1:18:03, 11.98s/it]
65%|██████████████████████▌ | 711/1101 [2:21:58<1:17:38, 11.94s/it]
65%|██████████████████████▋ | 712/1101 [2:22:10<1:16:38, 11.82s/it]
65%|██████████████████████▋ | 713/1101 [2:22:22<1:16:59, 11.91s/it]
65%|██████████████████████▋ | 714/1101 [2:22:34<1:17:01, 11.94s/it]
65%|██████████████████████▋ | 715/1101 [2:22:46<1:17:22, 12.03s/it]
65%|██████████████████████▊ | 716/1101 [2:22:58<1:16:57, 11.99s/it]
65%|██████████████████████▊ | 717/1101 [2:23:10<1:16:58, 12.03s/it]
65%|██████████████████████▊ | 718/1101 [2:23:22<1:16:37, 12.00s/it]
65%|██████████████████████▊ | 719/1101 [2:23:34<1:16:16, 11.98s/it]
65%|██████████████████████▉ | 720/1101 [2:23:46<1:15:23, 11.87s/it]
{'loss': '1.259', 'grad_norm': '1.055', 'learning_rate': '8.217e-06', 'entropy': '1.25', 'num_tokens': '4.676e+07', 'mean_token_accuracy': '0.7066', 'epoch': '1.962'} |
|
65%|██████████████████████▉ | 720/1101 [2:23:46<1:15:23, 11.87s/it]
65%|██████████████████████▉ | 721/1101 [2:23:58<1:15:15, 11.88s/it]
66%|██████████████████████▉ | 722/1101 [2:24:10<1:15:35, 11.97s/it]
66%|██████████████████████▉ | 723/1101 [2:24:21<1:14:25, 11.81s/it]
66%|███████████████████████ | 724/1101 [2:24:33<1:14:42, 11.89s/it]
66%|███████████████████████ | 725/1101 [2:24:45<1:14:27, 11.88s/it]
66%|███████████████████████ | 726/1101 [2:24:57<1:14:22, 11.90s/it]
66%|███████████████████████ | 727/1101 [2:25:09<1:14:36, 11.97s/it]
66%|███████████████████████▏ | 728/1101 [2:25:21<1:14:26, 11.97s/it]
66%|███████████████████████▏ | 729/1101 [2:25:33<1:14:40, 12.04s/it]
66%|███████████████████████▏ | 730/1101 [2:25:45<1:13:33, 11.90s/it]
{'loss': '1.256', 'grad_norm': '1.055', 'learning_rate': '8.331e-06', 'entropy': '1.254', 'num_tokens': '4.741e+07', 'mean_token_accuracy': '0.7046', 'epoch': '1.989'} |
|
66%|███████████████████████▏ | 730/1101 [2:25:45<1:13:33, 11.90s/it]
66%|███████████████████████▏ | 731/1101 [2:25:57<1:13:52, 11.98s/it]
66%|███████████████████████▎ | 732/1101 [2:26:09<1:13:28, 11.95s/it]
67%|███████████████████████▎ | 733/1101 [2:26:21<1:13:49, 12.04s/it]
67%|███████████████████████▎ | 734/1101 [2:26:32<1:10:41, 11.56s/it] |
|
Writing model shards: 0%| | 0/1 [00:00<?, ?it/s][A |
|
Writing model shards: 100%|██████████████████████| 1/1 [00:08<00:00, 8.79s/it][A
Writing model shards: 100%|██████████████████████| 1/1 [00:08<00:00, 8.79s/it] |
|
67%|███████████████████████▎ | 735/1101 [2:27:05<1:50:10, 18.06s/it]
67%|███████████████████████▍ | 736/1101 [2:27:17<1:39:07, 16.29s/it]
67%|███████████████████████▍ | 737/1101 [2:27:29<1:30:50, 14.97s/it]
67%|███████████████████████▍ | 738/1101 [2:27:41<1:25:27, 14.13s/it]
67%|███████████████████████▍ | 739/1101 [2:27:53<1:21:08, 13.45s/it]
67%|███████████████████████▌ | 740/1101 [2:28:05<1:18:13, 13.00s/it]
{'loss': '1.223', 'grad_norm': '0.9922', 'learning_rate': '8.446e-06', 'entropy': '1.223', 'num_tokens': '4.806e+07', 'mean_token_accuracy': '0.7112', 'epoch': '2.016'} |
|
67%|███████████████████████▌ | 740/1101 [2:28:05<1:18:13, 13.00s/it]
67%|███████████████████████▌ | 741/1101 [2:28:17<1:16:27, 12.74s/it]
67%|███████████████████████▌ | 742/1101 [2:28:29<1:14:48, 12.50s/it]
67%|███████████████████████▌ | 743/1101 [2:28:41<1:14:01, 12.41s/it]
68%|███████████████████████▋ | 744/1101 [2:28:53<1:12:13, 12.14s/it]
68%|███████████████████████▋ | 745/1101 [2:29:05<1:12:03, 12.14s/it]
68%|███████████████████████▋ | 746/1101 [2:29:17<1:11:35, 12.10s/it]
68%|███████████████████████▋ | 747/1101 [2:29:29<1:11:02, 12.04s/it]
68%|███████████████████████▊ | 748/1101 [2:29:41<1:10:34, 12.00s/it]
68%|███████████████████████▊ | 749/1101 [2:29:53<1:10:08, 11.96s/it]
68%|███████████████████████▊ | 750/1101 [2:30:05<1:10:14, 12.01s/it]
{'loss': '1.205', 'grad_norm': '1.023', 'learning_rate': '8.56e-06', 'entropy': '1.204', 'num_tokens': '4.871e+07', 'mean_token_accuracy': '0.7127', 'epoch': '2.044'} |
|
68%|███████████████████████▊ | 750/1101 [2:30:05<1:10:14, 12.01s/it]
68%|███████████████████████▊ | 751/1101 [2:30:17<1:09:52, 11.98s/it]
68%|███████████████████████▉ | 752/1101 [2:30:29<1:10:02, 12.04s/it]
68%|███████████████████████▉ | 753/1101 [2:30:41<1:09:18, 11.95s/it]
68%|███████████████████████▉ | 754/1101 [2:30:52<1:09:06, 11.95s/it]
69%|████████████████████████ | 755/1101 [2:31:05<1:09:16, 12.01s/it]
69%|████████████████████████ | 756/1101 [2:31:17<1:08:59, 12.00s/it]
69%|████████████████████████ | 757/1101 [2:31:29<1:09:02, 12.04s/it]
69%|████████████████████████ | 758/1101 [2:31:41<1:08:36, 12.00s/it]
69%|████████████████████████▏ | 759/1101 [2:31:53<1:08:43, 12.06s/it]
69%|████████████████████████▏ | 760/1101 [2:32:05<1:08:19, 12.02s/it]
{'loss': '1.184', 'grad_norm': '1.031', 'learning_rate': '8.674e-06', 'entropy': '1.181', 'num_tokens': '4.936e+07', 'mean_token_accuracy': '0.7155', 'epoch': '2.071'} |
|
69%|████████████████████████▏ | 760/1101 [2:32:05<1:08:19, 12.02s/it]
69%|████████████████████████▏ | 761/1101 [2:32:16<1:07:27, 11.90s/it]
69%|████████████████████████▏ | 762/1101 [2:32:29<1:07:41, 11.98s/it]
69%|████████████████████████▎ | 763/1101 [2:32:39<1:04:58, 11.54s/it]
69%|████████████████████████▎ | 764/1101 [2:32:51<1:05:50, 11.72s/it]
69%|████████████████████████▎ | 765/1101 [2:33:03<1:05:57, 11.78s/it]
70%|████████████████████████▎ | 766/1101 [2:33:15<1:06:25, 11.90s/it]
70%|████████████████████████▍ | 767/1101 [2:33:27<1:06:17, 11.91s/it]
70%|████████████████████████▍ | 768/1101 [2:33:39<1:06:36, 12.00s/it]
70%|████████████████████████▍ | 769/1101 [2:33:51<1:06:15, 11.97s/it]
70%|████████████████████████▍ | 770/1101 [2:34:03<1:06:00, 11.97s/it]
{'loss': '1.237', 'grad_norm': '1.047', 'learning_rate': '8.789e-06', 'entropy': '1.231', 'num_tokens': '5.001e+07', 'mean_token_accuracy': '0.7089', 'epoch': '2.098'} |
|
70%|████████████████████████▍ | 770/1101 [2:34:03<1:06:00, 11.97s/it]
70%|████████████████████████▌ | 771/1101 [2:34:16<1:06:17, 12.05s/it]
70%|████████████████████████▌ | 772/1101 [2:34:28<1:05:54, 12.02s/it]
70%|████████████████████████▌ | 773/1101 [2:34:40<1:06:03, 12.08s/it]
70%|████████████████████████▌ | 774/1101 [2:34:52<1:05:37, 12.04s/it]
70%|████████████████████████▋ | 775/1101 [2:35:04<1:05:14, 12.01s/it]
70%|████████████████████████▋ | 776/1101 [2:35:16<1:04:50, 11.97s/it]
71%|████████████████████████▋ | 777/1101 [2:35:27<1:04:36, 11.97s/it]
71%|████████████████████████▋ | 778/1101 [2:35:40<1:04:37, 12.01s/it]
71%|████████████████████████▊ | 779/1101 [2:35:51<1:04:15, 11.97s/it]
71%|████████████████████████▊ | 780/1101 [2:36:04<1:04:21, 12.03s/it]
{'loss': '1.237', 'grad_norm': '1.094', 'learning_rate': '8.903e-06', 'entropy': '1.235', 'num_tokens': '5.066e+07', 'mean_token_accuracy': '0.7083', 'epoch': '2.125'} |
|
71%|████████████████████████▊ | 780/1101 [2:36:04<1:04:21, 12.03s/it]
71%|████████████████████████▊ | 781/1101 [2:36:16<1:03:58, 11.99s/it]
71%|████████████████████████▊ | 782/1101 [2:36:27<1:03:17, 11.90s/it]
71%|████████████████████████▉ | 783/1101 [2:36:39<1:03:04, 11.90s/it]
71%|████████████████████████▉ | 784/1101 [2:36:51<1:02:45, 11.88s/it]
71%|████████████████████████▉ | 785/1101 [2:37:03<1:02:57, 11.96s/it]
71%|████████████████████████▉ | 786/1101 [2:37:15<1:02:43, 11.95s/it]
71%|█████████████████████████ | 787/1101 [2:37:27<1:02:56, 12.03s/it]
72%|█████████████████████████ | 788/1101 [2:37:39<1:02:25, 11.97s/it]
72%|█████████████████████████ | 789/1101 [2:37:51<1:02:35, 12.04s/it]
72%|█████████████████████████ | 790/1101 [2:38:03<1:02:18, 12.02s/it]
{'loss': '1.217', 'grad_norm': '0.9766', 'learning_rate': '9.017e-06', 'entropy': '1.214', 'num_tokens': '5.131e+07', 'mean_token_accuracy': '0.7117', 'epoch': '2.153'} |
|
72%|█████████████████████████ | 790/1101 [2:38:03<1:02:18, 12.02s/it]
72%|█████████████████████████▏ | 791/1101 [2:38:15<1:01:56, 11.99s/it]
72%|█████████████████████████▏ | 792/1101 [2:38:27<1:02:01, 12.04s/it]
72%|█████████████████████████▏ | 793/1101 [2:38:39<1:01:35, 12.00s/it]nohup: ignoring input |
| nohup: ignoring input |
| usage: sft_multilingual_impl.py [-h] [--train_file TRAIN_FILE] |
| [--max_samples MAX_SAMPLES] |
| [--output_dir OUTPUT_DIR] |
| [--num_train_epochs NUM_TRAIN_EPOCHS] |
| [--learning_rate LEARNING_RATE] |
| [--per_device_train_batch_size PER_DEVICE_TRAIN_BATCH_SIZE] |
| [--gradient_accumulation_steps GRADIENT_ACCUMULATION_STEPS] |
| [--warmup_ratio WARMUP_RATIO] [--seed SEED] |
| sft_multilingual_impl.py: error: unrecognized arguments: --resume_from_checkpoint /scratch/checkpoints/multilingual_sft_v2_ci_bs1_ga16_lr2e5_ep3/checkpoint-734 |
| usage: sft_multilingual_impl.py [-h] [--train_file TRAIN_FILE] |
| [--max_samples MAX_SAMPLES] |
| [--output_dir OUTPUT_DIR] |
| [--num_train_epochs NUM_TRAIN_EPOCHS] |
| [--learning_rate LEARNING_RATE] |
| [--per_device_train_batch_size PER_DEVICE_TRAIN_BATCH_SIZE] |
| [--gradient_accumulation_steps GRADIENT_ACCUMULATION_STEPS] |
| [--warmup_ratio WARMUP_RATIO] [--seed SEED] |
| sft_multilingual_impl.py: error: unrecognized arguments: --resume_from_checkpoint /scratch/checkpoints/multilingual_sft_v2_ci_bs1_ga16_lr2e5_ep3/checkpoint-734 |
| nohup: ignoring input |
| usage: sft_multilingual_impl.py [-h] [--train_file TRAIN_FILE] |
| [--max_samples MAX_SAMPLES] |
| [--output_dir OUTPUT_DIR] |
| [--num_train_epochs NUM_TRAIN_EPOCHS] |
| [--learning_rate LEARNING_RATE] |
| [--per_device_train_batch_size PER_DEVICE_TRAIN_BATCH_SIZE] |
| [--gradient_accumulation_steps GRADIENT_ACCUMULATION_STEPS] |
| [--warmup_ratio WARMUP_RATIO] [--seed SEED] |
| sft_multilingual_impl.py: error: unrecognized arguments: --resume_from_checkpoint /scratch/checkpoints/multilingual_sft_v2_ci_bs1_ga16_lr2e5_ep3/checkpoint-734 |
| nohup: ignoring input |
| usage: sft_multilingual_impl.py [-h] [--train_file TRAIN_FILE] |
| [--max_samples MAX_SAMPLES] |
| [--output_dir OUTPUT_DIR] |
| [--num_train_epochs NUM_TRAIN_EPOCHS] |
| [--learning_rate LEARNING_RATE] |
| [--per_device_train_batch_size PER_DEVICE_TRAIN_BATCH_SIZE] |
| [--gradient_accumulation_steps GRADIENT_ACCUMULATION_STEPS] |
| [--warmup_ratio WARMUP_RATIO] [--seed SEED] |
| sft_multilingual_impl.py: error: unrecognized arguments: --resume_from_checkpoint /scratch/checkpoints/multilingual_sft_v2_ci_bs1_ga16_lr2e5_ep3/checkpoint-734 |
| nohup: ignoring input |
| Loading tokenizer from Qwen/Qwen3-1.7B |
| Loading multilingual SFT dataset from /scratch/nico/data/multilingual_v2_ci/sft_train.jsonl |
| Full dataset size: 93371 |
| Formatting examples with Qwen chat template... |
| After formatting: 93371 examples |
| Filtering examples > 4096 tokens... |
| Removed 0 examples exceeding 4096 tokens |
| Final training set: 93371 examples |
|
|
| Loading model: Qwen/Qwen3-1.7B |
| Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. |
|
Loading weights: 0%| | 0/311 [00:00<?, ?it/s]
Loading weights: 100%|█████████████████████| 311/311 [00:00<00:00, 6552.25it/s] |
| W&B logging disabled |
| [RANK 0] Padding-free training is enabled, but the attention implementation is not set to a supported flash attention variant. Padding-free training flattens batches into a single sequence, and only the following implementations are known to reliably support this: flash_attention_2, flash_attention_3, kernels-community/flash-attn2, kernels-community/flash-attn3, kernels-community/vllm-flash-attn3. Using other implementations may lead to unexpected behavior. To ensure compatibility, set `attn_implementation` in the model configuration to one of these supported options or verify that your attention mechanism can handle flattened sequences. |
| [RANK 0] You are using packing, but the attention implementation is not set to a supported flash attention variant. Packing gathers multiple samples into a single sequence, and only the following implementations are known to reliably support this: flash_attention_2, flash_attention_3, kernels-community/flash-attn2, kernels-community/flash-attn3, kernels-community/vllm-flash-attn3. Using other implementations may lead to cross-contamination between samples. To avoid this, either disable packing by setting `packing=False`, or set `attn_implementation` in the model configuration to one of these supported options. |
| |
| Starting multilingual SFT training... |
| [transformers] The tokenizer has new PAD/BOS/EOS tokens that differ from the model config and generation config. The model config and generation config were aligned accordingly, being updated with the tokenizer's values. Updated tokens: {'bos_token_id': None, 'pad_token_id': 151643}. |
| [transformers] There were missing keys in the checkpoint model loaded: ['lm_head.weight']. |
| nohup: ignoring input |
| Loading tokenizer from Qwen/Qwen3-1.7B |
| Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. |
| Loading multilingual SFT dataset from /scratch/nico/data/multilingual_v2_ci/sft_train.jsonl |
| Full dataset size: 93371 |
| Formatting examples with Qwen chat template... |
| nohup: ignoring input |
| After formatting: 93371 examples |
| Filtering examples > 4096 tokens... |
|
0%| | 0/1101 [00:00<?, ?it/s] Removed 0 examples exceeding 4096 tokens |
| Final training set: 93371 examples |
| |
| Loading model: Qwen/Qwen3-1.7B |
|
Loading weights: 0%| | 0/311 [00:00<?, ?it/s]
Loading weights: 100%|█████████████████████| 311/311 [00:00<00:00, 5992.22it/s] |
| W&B logging disabled |
| [RANK 0] Padding-free training is enabled, but the attention implementation is not set to a supported flash attention variant. Padding-free training flattens batches into a single sequence, and only the following implementations are known to reliably support this: flash_attention_2, flash_attention_3, kernels-community/flash-attn2, kernels-community/flash-attn3, kernels-community/vllm-flash-attn3. Using other implementations may lead to unexpected behavior. To ensure compatibility, set `attn_implementation` in the model configuration to one of these supported options or verify that your attention mechanism can handle flattened sequences. |
| [RANK 0] You are using packing, but the attention implementation is not set to a supported flash attention variant. Packing gathers multiple samples into a single sequence, and only the following implementations are known to reliably support this: flash_attention_2, flash_attention_3, kernels-community/flash-attn2, kernels-community/flash-attn3, kernels-community/vllm-flash-attn3. Using other implementations may lead to cross-contamination between samples. To avoid this, either disable packing by setting `packing=False`, or set `attn_implementation` in the model configuration to one of these supported options. |
| Loading tokenizer from Qwen/Qwen3-1.7B |
| |
| Starting multilingual SFT training... |
| [transformers] The tokenizer has new PAD/BOS/EOS tokens that differ from the model config and generation config. The model config and generation config were aligned accordingly, being updated with the tokenizer's values. Updated tokens: {'bos_token_id': None, 'pad_token_id': 151643}. |
| Loading multilingual SFT dataset from /scratch/nico/data/multilingual_v2_ci/sft_train.jsonl |
| [transformers] There were missing keys in the checkpoint model loaded: ['lm_head.weight']. |
| Full dataset size: 93371 |
| Formatting examples with Qwen chat template... |
| After formatting: 93371 examples |
| Filtering examples > 4096 tokens... |
| Removed 0 examples exceeding 4096 tokens |
| Final training set: 93371 examples |
| |
| Loading model: Qwen/Qwen3-1.7B |
|
67%|██████████████████████████ | 735/1101 [00:12<00:06, 59.30it/s]Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. |
|
Loading weights: 0%| | 0/311 [00:00<?, ?it/s]
Loading weights: 100%|█████████████████████| 311/311 [00:00<00:00, 5928.47it/s] |
| W&B logging disabled |
| [RANK 0] Padding-free training is enabled, but the attention implementation is not set to a supported flash attention variant. Padding-free training flattens batches into a single sequence, and only the following implementations are known to reliably support this: flash_attention_2, flash_attention_3, kernels-community/flash-attn2, kernels-community/flash-attn3, kernels-community/vllm-flash-attn3. Using other implementations may lead to unexpected behavior. To ensure compatibility, set `attn_implementation` in the model configuration to one of these supported options or verify that your attention mechanism can handle flattened sequences. |
| [RANK 0] You are using packing, but the attention implementation is not set to a supported flash attention variant. Packing gathers multiple samples into a single sequence, and only the following implementations are known to reliably support this: flash_attention_2, flash_attention_3, kernels-community/flash-attn2, kernels-community/flash-attn3, kernels-community/vllm-flash-attn3. Using other implementations may lead to cross-contamination between samples. To avoid this, either disable packing by setting `packing=False`, or set `attn_implementation` in the model configuration to one of these supported options. |
|
0%| | 0/1101 [00:00<?, ?it/s]Traceback (most recent call last): |
| File "/scratch/nico/training/sft_multilingual_impl.py", line 210, in <module> |
| main() |
| File "/scratch/nico/training/sft_multilingual_impl.py", line 150, in main |
| trainer.train(resume_from_checkpoint=args.resume_from_checkpoint) |
| File "/usr/local/lib/python3.12/dist-packages/transformers/trainer.py", line 1427, in train |
| return inner_training_loop( |
| ^^^^^^^^^^^^^^^^^^^^ |
| File "/usr/local/lib/python3.12/dist-packages/transformers/trainer.py", line 1509, in _inner_training_loop |
| self._run_epoch( |
| File "/usr/local/lib/python3.12/dist-packages/transformers/trainer.py", line 1737, in _run_epoch |
| tr_loss_step = self.training_step(model, inputs, num_items_in_batch) |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| File "/usr/local/lib/python3.12/dist-packages/trl/trainer/sft_trainer.py", line 1395, in training_step |
| return super().training_step(*args, **kwargs) |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| File "/usr/local/lib/python3.12/dist-packages/transformers/trainer.py", line 1909, in training_step |
| loss = self.compute_loss(model, inputs, num_items_in_batch=num_items_in_batch) |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| File "/usr/local/lib/python3.12/dist-packages/trl/trainer/sft_trainer.py", line 1283, in compute_loss |
| (loss, outputs) = super().compute_loss( |
| ^^^^^^^^^^^^^^^^^^^^^ |
| File "/usr/local/lib/python3.12/dist-packages/transformers/trainer.py", line 1981, in compute_loss |
| outputs = model(**inputs) |
| ^^^^^^^^^^^^^^^ |
| File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl |
| return self._call_impl(*args, **kwargs) |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1787, in _call_impl |
| return forward_call(*args, **kwargs) |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| File "/usr/local/lib/python3.12/dist-packages/accelerate/utils/operations.py", line 823, in forward |
| return model_forward(*args, **kwargs) |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| File "/usr/local/lib/python3.12/dist-packages/accelerate/utils/operations.py", line 811, in __call__ |
| return convert_to_fp32(self.model_forward(*args, **kwargs)) |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| File "/usr/local/lib/python3.12/dist-packages/torch/amp/autocast_mode.py", line 44, in decorate_autocast |
| return func(*args, **kwargs) |
| ^^^^^^^^^^^^^^^^^^^^^ |
| File "/usr/local/lib/python3.12/dist-packages/transformers/utils/generic.py", line 900, in wrapper |
| output = func(self, *args, **kwargs) |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| File "/usr/local/lib/python3.12/dist-packages/transformers/models/qwen3/modeling_qwen3.py", line 509, in forward |
| loss = self.loss_function(logits=logits, labels=labels, vocab_size=self.config.vocab_size, **kwargs) |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| File "/usr/local/lib/python3.12/dist-packages/transformers/loss/loss_utils.py", line 56, in ForCausalLMLoss |
| logits = logits.float() |
| ^^^^^^^^^^^^^^ |
| torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.31 GiB. GPU 0 has a total capacity of 39.49 GiB of which 1.39 GiB is free. Process 984 has 25.86 GiB memory in use. Including non-PyTorch memory, this process has 11.82 GiB memory in use. Process 1450 has 416.00 MiB memory in use. Of the allocated memory 11.30 GiB is allocated by PyTorch, and 29.39 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) |
|
0%| | 0/1101 [00:01<?, ?it/s] |
| Traceback (most recent call last): |
| File "/scratch/nico/training/sft_multilingual_impl.py", line 210, in <module> |
| main() |
| File "/scratch/nico/training/sft_multilingual_impl.py", line 142, in main |
| trainer = SFTTrainer( |
| ^^^^^^^^^^^ |
| File "/usr/local/lib/python3.12/dist-packages/trl/trainer/sft_trainer.py", line 990, in __init__ |
| super().__init__( |
| File "/usr/local/lib/python3.12/dist-packages/transformers/trainer.py", line 480, in __init__ |
| self._move_model_to_device(model, args.device) |
| File "/usr/local/lib/python3.12/dist-packages/transformers/trainer.py", line 4415, in _move_model_to_device |
| model = model.to(device) |
| ^^^^^^^^^^^^^^^^ |
| File "/usr/local/lib/python3.12/dist-packages/transformers/modeling_utils.py", line 3650, in to |
| return super().to(*args, **kwargs) |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1381, in to |
| return self._apply(convert) |
| ^^^^^^^^^^^^^^^^^^^^ |
| File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 933, in _apply |
| module._apply(fn) |
| File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 933, in _apply |
| module._apply(fn) |
| File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 933, in _apply |
| module._apply(fn) |
| [Previous line repeated 2 more times] |
| File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 964, in _apply |
| param_applied = fn(param) |
| ^^^^^^^^^ |
| File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1367, in convert |
| return t.to( |
| ^^^^^ |
| torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 24.00 MiB. GPU 0 has a total capacity of 39.49 GiB of which 4.38 MiB is free. Process 984 has 25.86 GiB memory in use. Process 1317 has 11.82 GiB memory in use. Including non-PyTorch memory, this process has 1.79 GiB memory in use. Of the allocated memory 1.38 GiB is allocated by PyTorch, and 5.93 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) |
|
67%|██████████████████████████ | 735/1101 [00:24<00:06, 59.30it/s]
67%|██████████████████████████ | 736/1101 [00:24<00:14, 24.52it/s]
67%|██████████████████████████ | 737/1101 [00:36<00:27, 13.44it/s]
67%|██████████████████████████▏ | 738/1101 [00:48<00:43, 8.27it/s]
67%|██████████████████████████▏ | 739/1101 [01:00<01:08, 5.32it/s]
67%|██████████████████████████▏ | 740/1101 [01:12<01:41, 3.56it/s]
{'loss': '1.226', 'grad_norm': '0.9922', 'learning_rate': '8.446e-06', 'entropy': '1.228', 'num_tokens': '3.926e+05', 'mean_token_accuracy': '0.7088', 'epoch': '2.016'} |
|
67%|██████████████████████████▏ | 740/1101 [01:12<01:41, 3.56it/s]
67%|██████████████████████████▏ | 741/1101 [01:25<02:28, 2.42it/s]
67%|██████████████████████████▎ | 742/1101 [01:37<03:33, 1.68it/s]
67%|██████████████████████████▎ | 743/1101 [01:48<05:02, 1.19it/s]
68%|██████████████████████████▎ | 744/1101 [02:00<06:59, 1.18s/it]
68%|██████████████████████████▍ | 745/1101 [02:12<09:37, 1.62s/it]
68%|██████████████████████████▍ | 746/1101 [02:24<13:07, 2.22s/it]
68%|██████████████████████████▍ | 747/1101 [02:36<17:20, 2.94s/it]
68%|██████████████████████████▍ | 748/1101 [02:48<22:18, 3.79s/it]
68%|██████████████████████████▌ | 749/1101 [03:00<27:57, 4.76s/it]
68%|██████████████████████████▌ | 750/1101 [03:12<33:59, 5.81s/it]
{'loss': '1.205', 'grad_norm': '1', 'learning_rate': '8.56e-06', 'entropy': '1.204', 'num_tokens': '1.044e+06', 'mean_token_accuracy': '0.7126', 'epoch': '2.044'} |
|
68%|██████████████████████████▌ | 750/1101 [03:12<33:59, 5.81s/it]
68%|██████████████████████████▌ | 751/1101 [03:24<40:16, 6.90s/it]
68%|██████████████████████████▋ | 752/1101 [03:36<45:58, 7.90s/it]
68%|██████████████████████████▋ | 753/1101 [03:48<51:04, 8.81s/it]
68%|██████████████████████████▋ | 754/1101 [04:00<55:18, 9.56s/it]
69%|██████████████████████████▋ | 755/1101 [04:12<59:00, 10.23s/it]
69%|█████████████████████████▍ | 756/1101 [04:24<1:01:32, 10.70s/it]
69%|█████████████████████████▍ | 757/1101 [04:36<1:03:17, 11.04s/it]
69%|█████████████████████████▍ | 758/1101 [04:48<1:04:52, 11.35s/it]
69%|█████████████████████████▌ | 759/1101 [05:00<1:05:40, 11.52s/it]
69%|█████████████████████████▌ | 760/1101 [05:12<1:06:34, 11.71s/it]
{'loss': '1.183', 'grad_norm': '1.031', 'learning_rate': '8.674e-06', 'entropy': '1.181', 'num_tokens': '1.697e+06', 'mean_token_accuracy': '0.7156', 'epoch': '2.071'} |
|
69%|█████████████████████████▌ | 760/1101 [05:12<1:06:34, 11.71s/it]
69%|█████████████████████████▌ | 761/1101 [05:24<1:06:16, 11.70s/it]
69%|█████████████████████████▌ | 762/1101 [05:36<1:06:49, 11.83s/it]
69%|█████████████████████████▋ | 763/1101 [05:46<1:04:25, 11.44s/it]
69%|█████████████████████████▋ | 764/1101 [05:58<1:05:04, 11.59s/it]
69%|█████████████████████████▋ | 765/1101 [06:11<1:05:50, 11.76s/it]
70%|█████████████████████████▋ | 766/1101 [06:22<1:05:49, 11.79s/it]
70%|█████████████████████████▊ | 767/1101 [06:35<1:06:15, 11.90s/it]
70%|█████████████████████████▊ | 768/1101 [06:47<1:06:07, 11.92s/it]
70%|█████████████████████████▊ | 769/1101 [06:59<1:06:25, 12.00s/it]
70%|█████████████████████████▉ | 770/1101 [07:11<1:06:09, 11.99s/it]
{'loss': '1.237', 'grad_norm': '1.062', 'learning_rate': '8.789e-06', 'entropy': '1.231', 'num_tokens': '2.342e+06', 'mean_token_accuracy': '0.7089', 'epoch': '2.098'} |
|
70%|█████████████████████████▉ | 770/1101 [07:11<1:06:09, 11.99s/it]
70%|█████████████████████████▉ | 771/1101 [07:23<1:06:26, 12.08s/it]
70%|█████████████████████████▉ | 772/1101 [07:35<1:06:00, 12.04s/it]
70%|█████████████████████████▉ | 773/1101 [07:47<1:05:42, 12.02s/it]
70%|██████████████████████████ | 774/1101 [07:59<1:05:51, 12.08s/it]
70%|██████████████████████████ | 775/1101 [08:11<1:04:57, 11.96s/it]
70%|██████████████████████████ | 776/1101 [08:23<1:05:05, 12.02s/it]
71%|██████████████████████████ | 777/1101 [08:35<1:04:48, 12.00s/it]
71%|██████████████████████████▏ | 778/1101 [08:47<1:04:46, 12.03s/it]
71%|██████████████████████████▏ | 779/1101 [08:59<1:04:22, 12.00s/it]
71%|██████████████████████████▏ | 780/1101 [09:11<1:04:02, 11.97s/it]
{'loss': '1.237', 'grad_norm': '1.094', 'learning_rate': '8.903e-06', 'entropy': '1.235', 'num_tokens': '2.995e+06', 'mean_token_accuracy': '0.7084', 'epoch': '2.125'} |
|
71%|██████████████████████████▏ | 780/1101 [09:11<1:04:02, 11.97s/it]
71%|██████████████████████████▏ | 781/1101 [09:23<1:04:10, 12.03s/it]
71%|██████████████████████████▎ | 782/1101 [09:34<1:03:00, 11.85s/it]
71%|██████████████████████████▎ | 783/1101 [09:47<1:03:16, 11.94s/it]
71%|██████████████████████████▎ | 784/1101 [09:58<1:02:55, 11.91s/it]
71%|██████████████████████████▍ | 785/1101 [10:11<1:03:07, 11.98s/it]
71%|██████████████████████████▍ | 786/1101 [10:23<1:02:52, 11.98s/it]
71%|██████████████████████████▍ | 787/1101 [10:35<1:02:36, 11.96s/it]
72%|██████████████████████████▍ | 788/1101 [10:47<1:02:32, 11.99s/it]
72%|██████████████████████████▌ | 789/1101 [10:58<1:02:13, 11.97s/it]
72%|██████████████████████████▌ | 790/1101 [11:11<1:02:31, 12.06s/it]
{'loss': '1.217', 'grad_norm': '0.9844', 'learning_rate': '9.017e-06', 'entropy': '1.214', 'num_tokens': '3.646e+06', 'mean_token_accuracy': '0.7117', 'epoch': '2.153'} |
|
72%|██████████████████████████▌ | 790/1101 [11:11<1:02:31, 12.06s/it]
72%|██████████████████████████▌ | 791/1101 [11:23<1:02:06, 12.02s/it]
72%|██████████████████████████▌ | 792/1101 [11:35<1:02:11, 12.07s/it]
72%|██████████████████████████▋ | 793/1101 [11:47<1:01:43, 12.02s/it]
72%|██████████████████████████▋ | 794/1101 [11:59<1:01:26, 12.01s/it]
72%|██████████████████████████▋ | 795/1101 [12:11<1:00:55, 11.95s/it]
72%|██████████████████████████▊ | 796/1101 [12:22<1:00:00, 11.81s/it]
72%|██████████████████████████▊ | 797/1101 [12:34<1:00:25, 11.92s/it]
72%|██████████████████████████▊ | 798/1101 [12:46<1:00:18, 11.94s/it]
73%|██████████████████████████▊ | 799/1101 [12:58<1:00:31, 12.02s/it]
73%|██████████████████████████▉ | 800/1101 [13:10<1:00:00, 11.96s/it]
{'loss': '1.229', 'grad_norm': '1.25', 'learning_rate': '9.131e-06', 'entropy': '1.23', 'num_tokens': '4.296e+06', 'mean_token_accuracy': '0.7098', 'epoch': '2.18'} |
|
73%|██████████████████████████▉ | 800/1101 [13:10<1:00:00, 11.96s/it]
73%|████████████████████████████▎ | 801/1101 [13:22<59:41, 11.94s/it]
73%|████████████████████████████▍ | 802/1101 [13:34<59:33, 11.95s/it]
73%|████████████████████████████▍ | 803/1101 [13:46<58:41, 11.82s/it]
73%|████████████████████████████▍ | 804/1101 [13:58<58:59, 11.92s/it]
73%|████████████████████████████▌ | 805/1101 [14:10<58:48, 11.92s/it]
73%|████████████████████████████▌ | 806/1101 [14:22<58:59, 12.00s/it]
73%|████████████████████████████▌ | 807/1101 [14:34<58:43, 11.99s/it]
73%|████████████████████████████▌ | 808/1101 [14:46<58:57, 12.07s/it]
73%|████████████████████████████▋ | 809/1101 [14:57<57:42, 11.86s/it]
74%|████████████████████████████▋ | 810/1101 [15:09<57:36, 11.88s/it]
{'loss': '1.21', 'grad_norm': '1.023', 'learning_rate': '9.246e-06', 'entropy': '1.211', 'num_tokens': '4.944e+06', 'mean_token_accuracy': '0.7118', 'epoch': '2.207'} |
|
74%|████████████████████████████▋ | 810/1101 [15:09<57:36, 11.88s/it]
74%|████████████████████████████▋ | 811/1101 [15:22<57:55, 11.98s/it]
74%|████████████████████████████▊ | 812/1101 [15:33<56:41, 11.77s/it]
74%|████████████████████████████▊ | 813/1101 [15:45<57:07, 11.90s/it]
74%|████████████████████████████▊ | 814/1101 [15:57<57:02, 11.93s/it]
74%|████████████████████████████▊ | 815/1101 [16:09<57:07, 11.99s/it]
74%|████████████████████████████▉ | 816/1101 [16:21<56:53, 11.98s/it]
74%|████████████████████████████▉ | 817/1101 [16:33<56:00, 11.83s/it]
74%|████████████████████████████▉ | 818/1101 [16:45<56:15, 11.93s/it]
74%|█████████████████████████████ | 819/1101 [16:57<56:06, 11.94s/it]
74%|█████████████████████████████ | 820/1101 [17:09<56:05, 11.98s/it]
{'loss': '1.154', 'grad_norm': '1.008', 'learning_rate': '9.36e-06', 'entropy': '1.152', 'num_tokens': '5.591e+06', 'mean_token_accuracy': '0.7216', 'epoch': '2.234'} |
|
74%|█████████████████████████████ | 820/1101 [17:09<56:05, 11.98s/it]
75%|█████████████████████████████ | 821/1101 [17:21<55:51, 11.97s/it]
75%|█████████████████████████████ | 822/1101 [17:33<55:58, 12.04s/it]
75%|█████████████████████████████▏ | 823/1101 [17:45<55:32, 11.99s/it]
75%|█████████████████████████████▏ | 824/1101 [17:57<55:14, 11.97s/it]
75%|█████████████████████████████▏ | 825/1101 [18:08<54:36, 11.87s/it]
75%|█████████████████████████████▎ | 826/1101 [18:20<54:33, 11.90s/it]
75%|█████████████████████████████▎ | 827/1101 [18:33<54:45, 11.99s/it]
75%|█████████████████████████████▎ | 828/1101 [18:45<54:26, 11.97s/it]
75%|█████████████████████████████▎ | 829/1101 [18:57<54:38, 12.05s/it]
75%|█████████████████████████████▍ | 830/1101 [19:09<54:13, 12.00s/it]
{'loss': '1.215', 'grad_norm': '1.055', 'learning_rate': '9.474e-06', 'entropy': '1.207', 'num_tokens': '6.243e+06', 'mean_token_accuracy': '0.714', 'epoch': '2.262'} |
|
75%|█████████████████████████████▍ | 830/1101 [19:09<54:13, 12.00s/it]
75%|█████████████████████████████▍ | 831/1101 [19:21<53:57, 11.99s/it]
76%|█████████████████████████████▍ | 832/1101 [19:33<53:58, 12.04s/it]
76%|█████████████████████████████▌ | 833/1101 [19:45<53:34, 12.00s/it]
76%|█████████████████████████████▌ | 834/1101 [19:56<52:53, 11.89s/it]
76%|█████████████████████████████▌ | 835/1101 [20:07<51:43, 11.67s/it]
76%|█████████████████████████████▌ | 836/1101 [20:20<52:14, 11.83s/it]
76%|█████████████████████████████▋ | 837/1101 [20:32<52:11, 11.86s/it]
76%|█████████████████████████████▋ | 838/1101 [20:44<52:03, 11.88s/it]
76%|█████████████████████████████▋ | 839/1101 [20:55<51:31, 11.80s/it]
76%|█████████████████████████████▊ | 840/1101 [21:07<50:50, 11.69s/it]
{'loss': '1.203', 'grad_norm': '0.9922', 'learning_rate': '9.589e-06', 'entropy': '1.205', 'num_tokens': '6.884e+06', 'mean_token_accuracy': '0.713', 'epoch': '2.289'} |
|
76%|█████████████████████████████▊ | 840/1101 [21:07<50:50, 11.69s/it]
76%|█████████████████████████████▊ | 841/1101 [21:19<51:24, 11.86s/it]
76%|█████████████████████████████▊ | 842/1101 [21:31<51:16, 11.88s/it]
77%|█████████████████████████████▊ | 843/1101 [21:43<51:09, 11.90s/it]
77%|█████████████████████████████▉ | 844/1101 [21:55<50:58, 11.90s/it]
77%|█████████████████████████████▉ | 845/1101 [22:07<51:05, 11.98s/it]
77%|█████████████████████████████▉ | 846/1101 [22:18<50:34, 11.90s/it]
77%|██████████████████████████████ | 847/1101 [22:30<50:29, 11.93s/it]
77%|██████████████████████████████ | 848/1101 [22:43<50:31, 11.98s/it]
77%|██████████████████████████████ | 849/1101 [22:54<49:30, 11.79s/it]
77%|██████████████████████████████ | 850/1101 [23:06<49:49, 11.91s/it]
{'loss': '1.198', 'grad_norm': '1.016', 'learning_rate': '9.703e-06', 'entropy': '1.198', 'num_tokens': '7.533e+06', 'mean_token_accuracy': '0.7126', 'epoch': '2.316'} |
|
77%|██████████████████████████████ | 850/1101 [23:06<49:49, 11.91s/it]
77%|██████████████████████████████▏ | 851/1101 [23:18<49:43, 11.94s/it]
77%|██████████████████████████████▏ | 852/1101 [23:30<49:52, 12.02s/it]
77%|██████████████████████████████▏ | 853/1101 [23:42<49:34, 11.99s/it]
78%|██████████████████████████████▎ | 854/1101 [23:54<49:17, 11.97s/it]
78%|██████████████████████████████▎ | 855/1101 [24:06<49:18, 12.03s/it]
78%|██████████████████████████████▎ | 856/1101 [24:18<49:02, 12.01s/it]
78%|██████████████████████████████▎ | 857/1101 [24:30<49:05, 12.07s/it]
78%|██████████████████████████████▍ | 858/1101 [24:42<48:43, 12.03s/it]
78%|██████████████████████████████▍ | 859/1101 [24:54<48:00, 11.90s/it]
78%|██████████████████████████████▍ | 860/1101 [25:06<47:33, 11.84s/it]
{'loss': '1.228', 'grad_norm': '1.086', 'learning_rate': '9.817e-06', 'entropy': '1.224', 'num_tokens': '8.183e+06', 'mean_token_accuracy': '0.71', 'epoch': '2.343'} |
|
78%|██████████████████████████████▍ | 860/1101 [25:06<47:33, 11.84s/it]
78%|██████████████████████████████▍ | 861/1101 [25:18<47:31, 11.88s/it]
78%|██████████████████████████████▌ | 862/1101 [25:30<47:36, 11.95s/it]
78%|██████████████████████████████▌ | 863/1101 [25:41<46:58, 11.84s/it]
78%|██████████████████████████████▌ | 864/1101 [25:53<46:30, 11.77s/it]
79%|██████████████████████████████▋ | 865/1101 [26:05<45:58, 11.69s/it]
79%|██████████████████████████████▋ | 866/1101 [26:17<46:21, 11.83s/it]
79%|██████████████████████████████▋ | 867/1101 [26:29<46:19, 11.88s/it]
79%|██████████████████████████████▋ | 868/1101 [26:41<46:12, 11.90s/it]
79%|██████████████████████████████▊ | 869/1101 [26:53<46:15, 11.96s/it]
79%|██████████████████████████████▊ | 870/1101 [27:05<45:59, 11.95s/it]
{'loss': '1.222', 'grad_norm': '1.102', 'learning_rate': '9.931e-06', 'entropy': '1.218', 'num_tokens': '8.83e+06', 'mean_token_accuracy': '0.7119', 'epoch': '2.371'} |
|
79%|██████████████████████████████▊ | 870/1101 [27:05<45:59, 11.95s/it]
79%|██████████████████████████████▊ | 871/1101 [27:17<46:03, 12.01s/it]
79%|██████████████████████████████▉ | 872/1101 [27:29<45:41, 11.97s/it]
79%|██████████████████████████████▉ | 873/1101 [27:40<44:37, 11.75s/it]
79%|██████████████████████████████▉ | 874/1101 [27:52<44:35, 11.78s/it]
79%|██████████████████████████████▉ | 875/1101 [28:04<44:32, 11.82s/it]
80%|███████████████████████████████ | 876/1101 [28:16<44:43, 11.93s/it]
80%|███████████████████████████████ | 877/1101 [28:27<43:51, 11.75s/it]
80%|███████████████████████████████ | 878/1101 [28:39<43:50, 11.80s/it]
80%|███████████████████████████████▏ | 879/1101 [28:51<43:47, 11.84s/it]
80%|███████████████████████████████▏ | 880/1101 [29:03<43:59, 11.94s/it]
{'loss': '1.194', 'grad_norm': '1.055', 'learning_rate': '1.005e-05', 'entropy': '1.196', 'num_tokens': '9.475e+06', 'mean_token_accuracy': '0.7176', 'epoch': '2.398'} |
|
80%|███████████████████████████████▏ | 880/1101 [29:03<43:59, 11.94s/it]
80%|███████████████████████████████▏ | 881/1101 [29:15<43:46, 11.94s/it]
80%|███████████████████████████████▏ | 882/1101 [29:27<43:53, 12.03s/it]
80%|███████████████████████████████▎ | 883/1101 [29:39<43:16, 11.91s/it]
80%|███████████████████████████████▎ | 884/1101 [29:51<43:05, 11.91s/it]
80%|███████████████████████████████▎ | 885/1101 [30:03<43:16, 12.02s/it]
80%|███████████████████████████████▍ | 886/1101 [30:15<42:55, 11.98s/it]
81%|███████████████████████████████▍ | 887/1101 [30:27<42:56, 12.04s/it]
81%|███████████████████████████████▍ | 888/1101 [30:39<42:36, 12.00s/it]
81%|███████████████████████████████▍ | 889/1101 [30:51<42:04, 11.91s/it]
81%|███████████████████████████████▌ | 890/1101 [31:03<41:59, 11.94s/it]
{'loss': '1.219', 'grad_norm': '1.047', 'learning_rate': '1.016e-05', 'entropy': '1.218', 'num_tokens': '1.013e+07', 'mean_token_accuracy': '0.7131', 'epoch': '2.425'} |
|
81%|███████████████████████████████▌ | 890/1101 [31:03<41:59, 11.94s/it]
81%|███████████████████████████████▌ | 891/1101 [31:15<41:47, 11.94s/it]
81%|███████████████████████████████▌ | 892/1101 [31:27<41:47, 12.00s/it]
81%|███████████████████████████████▋ | 893/1101 [31:38<41:05, 11.86s/it]
81%|███████████████████████████████▋ | 894/1101 [31:51<41:12, 11.94s/it]
81%|███████████████████████████████▋ | 895/1101 [32:03<41:03, 11.96s/it]
81%|███████████████████████████████▋ | 896/1101 [32:15<41:06, 12.03s/it]
81%|███████████████████████████████▊ | 897/1101 [32:27<40:50, 12.01s/it]
82%|███████████████████████████████▊ | 898/1101 [32:39<40:33, 11.99s/it]
82%|███████████████████████████████▊ | 899/1101 [32:50<40:00, 11.88s/it]
82%|███████████████████████████████▉ | 900/1101 [33:02<39:52, 11.90s/it]
{'loss': '1.247', 'grad_norm': '1.094', 'learning_rate': '1.027e-05', 'entropy': '1.244', 'num_tokens': '1.077e+07', 'mean_token_accuracy': '0.7092', 'epoch': '2.452'} |
|
82%|███████████████████████████████▉ | 900/1101 [33:02<39:52, 11.90s/it]
82%|███████████████████████████████▉ | 901/1101 [33:14<39:25, 11.83s/it]
82%|███████████████████████████████▉ | 902/1101 [33:25<38:20, 11.56s/it]
82%|███████████████████████████████▉ | 903/1101 [33:37<38:33, 11.69s/it]
82%|████████████████████████████████ | 904/1101 [33:49<38:38, 11.77s/it]
82%|████████████████████████████████ | 905/1101 [34:01<38:38, 11.83s/it]
82%|████████████████████████████████ | 906/1101 [34:13<38:44, 11.92s/it]
82%|████████████████████████████████▏ | 907/1101 [34:25<38:33, 11.92s/it]
82%|████████████████████████████████▏ | 908/1101 [34:37<38:34, 11.99s/it]
83%|████████████████████████████████▏ | 909/1101 [34:49<38:19, 11.97s/it]
83%|████████████████████████████████▏ | 910/1101 [35:01<38:15, 12.02s/it]
{'loss': '1.232', 'grad_norm': '1.008', 'learning_rate': '1.039e-05', 'entropy': '1.229', 'num_tokens': '1.142e+07', 'mean_token_accuracy': '0.7122', 'epoch': '2.48'} |
|
83%|████████████████████████████████▏ | 910/1101 [35:01<38:15, 12.02s/it]
83%|████████████████████████████████▎ | 911/1101 [35:13<37:30, 11.84s/it]
83%|████████████████████████████████▎ | 912/1101 [35:24<37:20, 11.86s/it]
83%|████████████████████████████████▎ | 913/1101 [35:37<37:26, 11.95s/it]
83%|████████████████████████████████▍ | 914/1101 [35:49<37:15, 11.95s/it]
83%|████████████████████████████████▍ | 915/1101 [36:00<36:46, 11.86s/it]
83%|████████████████████████████████▍ | 916/1101 [36:12<36:30, 11.84s/it]
83%|████████████████████████████████▍ | 917/1101 [36:24<36:39, 11.96s/it]
83%|████████████████████████████████▌ | 918/1101 [36:36<36:24, 11.94s/it]
83%|████████████████████████████████▌ | 919/1101 [36:48<36:25, 12.01s/it]
84%|████████████████████████████████▌ | 920/1101 [37:00<36:11, 12.00s/it]
{'loss': '1.182', 'grad_norm': '0.9844', 'learning_rate': '1.05e-05', 'entropy': '1.183', 'num_tokens': '1.207e+07', 'mean_token_accuracy': '0.7172', 'epoch': '2.507'} |
|
84%|████████████████████████████████▌ | 920/1101 [37:00<36:11, 12.00s/it]
84%|████████████████████████████████▌ | 921/1101 [37:12<35:52, 11.96s/it]
84%|████████████████████████████████▋ | 922/1101 [37:24<35:20, 11.85s/it]
84%|████████████████████████████████▋ | 923/1101 [37:36<35:10, 11.86s/it]
84%|████████████████████████████████▋ | 924/1101 [37:48<35:16, 11.96s/it]
84%|████████████████████████████████▊ | 925/1101 [38:00<35:01, 11.94s/it]
84%|████████████████████████████████▊ | 926/1101 [38:12<35:00, 12.00s/it]
84%|████████████████████████████████▊ | 927/1101 [38:24<34:43, 11.97s/it]
84%|████████████████████████████████▊ | 928/1101 [38:36<34:30, 11.97s/it]
84%|████████████████████████████████▉ | 929/1101 [38:48<34:31, 12.04s/it]
84%|████████████████████████████████▉ | 930/1101 [39:00<34:10, 11.99s/it]
{'loss': '1.217', 'grad_norm': '1.008', 'learning_rate': '1.062e-05', 'entropy': '1.212', 'num_tokens': '1.272e+07', 'mean_token_accuracy': '0.7126', 'epoch': '2.534'} |
|
84%|████████████████████████████████▉ | 930/1101 [39:00<34:10, 11.99s/it]
85%|████████████████████████████████▉ | 931/1101 [39:12<34:13, 12.08s/it]
85%|█████████████████████████████████ | 932/1101 [39:24<33:55, 12.04s/it]
85%|█████████████████████████████████ | 933/1101 [39:36<33:42, 12.04s/it]
85%|█████████████████████████████████ | 934/1101 [39:48<33:26, 12.01s/it]
85%|█████████████████████████████████ | 935/1101 [40:00<33:11, 12.00s/it]
85%|█████████████████████████████████▏ | 936/1101 [40:12<33:14, 12.09s/it]
85%|█████████████████████████████████▏ | 937/1101 [40:24<32:53, 12.04s/it]
85%|█████████████████████████████████▏ | 938/1101 [40:36<32:51, 12.09s/it]
85%|█████████████████████████████████▎ | 939/1101 [40:48<32:33, 12.06s/it]
85%|█████████████████████████████████▎ | 940/1101 [41:01<32:29, 12.11s/it]
{'loss': '1.184', 'grad_norm': '0.9258', 'learning_rate': '1.073e-05', 'entropy': '1.184', 'num_tokens': '1.337e+07', 'mean_token_accuracy': '0.7159', 'epoch': '2.561'} |
|
85%|█████████████████████████████████▎ | 940/1101 [41:01<32:29, 12.11s/it]
85%|█████████████████████████████████▎ | 941/1101 [41:13<32:13, 12.08s/it]
86%|█████████████████████████████████▎ | 942/1101 [41:24<31:51, 12.02s/it]
86%|█████████████████████████████████▍ | 943/1101 [41:36<31:18, 11.89s/it]
86%|█████████████████████████████████▍ | 944/1101 [41:48<31:04, 11.88s/it]
86%|█████████████████████████████████▍ | 945/1101 [42:00<31:09, 11.98s/it]
86%|█████████████████████████████████▌ | 946/1101 [42:12<30:54, 11.97s/it]
86%|█████████████████████████████████▌ | 947/1101 [42:24<30:50, 12.02s/it]
86%|█████████████████████████████████▌ | 948/1101 [42:36<30:34, 11.99s/it]
86%|█████████████████████████████████▌ | 949/1101 [42:48<30:19, 11.97s/it]
86%|█████████████████████████████████▋ | 950/1101 [43:00<30:17, 12.04s/it]
{'loss': '1.21', 'grad_norm': '1.016', 'learning_rate': '1.085e-05', 'entropy': '1.208', 'num_tokens': '1.402e+07', 'mean_token_accuracy': '0.7146', 'epoch': '2.589'} |
|
86%|█████████████████████████████████▋ | 950/1101 [43:00<30:17, 12.04s/it]
86%|█████████████████████████████████▋ | 951/1101 [43:12<29:50, 11.94s/it]
86%|█████████████████████████████████▋ | 952/1101 [43:24<29:41, 11.95s/it]
87%|█████████████████████████████████▊ | 953/1101 [43:36<29:29, 11.95s/it]
87%|█████████████████████████████████▊ | 954/1101 [43:48<29:28, 12.03s/it]
87%|█████████████████████████████████▊ | 955/1101 [44:00<29:11, 12.00s/it]
87%|█████████████████████████████████▊ | 956/1101 [44:12<29:08, 12.06s/it]
87%|█████████████████████████████████▉ | 957/1101 [44:24<28:53, 12.04s/it]
87%|█████████████████████████████████▉ | 958/1101 [44:36<28:34, 11.99s/it]
87%|█████████████████████████████████▉ | 959/1101 [44:48<28:30, 12.05s/it]
87%|██████████████████████████████████ | 960/1101 [45:00<27:50, 11.85s/it]
{'loss': '1.151', 'grad_norm': '0.9648', 'learning_rate': '1.096e-05', 'entropy': '1.153', 'num_tokens': '1.467e+07', 'mean_token_accuracy': '0.7211', 'epoch': '2.616'} |
|
87%|██████████████████████████████████ | 960/1101 [45:00<27:50, 11.85s/it]
87%|██████████████████████████████████ | 961/1101 [45:12<27:48, 11.92s/it]
87%|██████████████████████████████████ | 962/1101 [45:24<27:39, 11.94s/it]
87%|██████████████████████████████████ | 963/1101 [45:36<27:37, 12.01s/it]
88%|██████████████████████████████████▏ | 964/1101 [45:48<27:22, 11.99s/it]
88%|██████████████████████████████████▏ | 965/1101 [46:00<27:08, 11.98s/it]
88%|██████████████████████████████████▏ | 966/1101 [46:12<27:03, 12.03s/it]
88%|██████████████████████████████████▎ | 967/1101 [46:24<26:45, 11.98s/it]
88%|██████████████████████████████████▎ | 968/1101 [46:35<26:15, 11.85s/it]
88%|██████████████████████████████████▎ | 969/1101 [46:47<25:49, 11.74s/it]
88%|██████████████████████████████████▎ | 970/1101 [46:59<25:54, 11.86s/it]
{'loss': '1.217', 'grad_norm': '0.9922', 'learning_rate': '1.107e-05', 'entropy': '1.217', 'num_tokens': '1.532e+07', 'mean_token_accuracy': '0.7137', 'epoch': '2.643'} |
|
88%|██████████████████████████████████▎ | 970/1101 [46:59<25:54, 11.86s/it]
88%|██████████████████████████████████▍ | 971/1101 [47:11<25:44, 11.88s/it]
88%|██████████████████████████████████▍ | 972/1101 [47:23<25:33, 11.89s/it]
88%|██████████████████████████████████▍ | 973/1101 [47:35<25:32, 11.97s/it]
88%|██████████████████████████████████▌ | 974/1101 [47:46<24:59, 11.81s/it]
89%|██████████████████████████████████▌ | 975/1101 [47:59<25:04, 11.94s/it]
89%|██████████████████████████████████▌ | 976/1101 [48:10<24:47, 11.90s/it]
89%|██████████████████████████████████▌ | 977/1101 [48:22<24:26, 11.83s/it]
89%|██████████████████████████████████▋ | 978/1101 [48:34<24:18, 11.86s/it]
89%|██████████████████████████████████▋ | 979/1101 [48:46<24:07, 11.87s/it]
89%|██████████████████████████████████▋ | 980/1101 [48:58<24:07, 11.96s/it]
{'loss': '1.172', 'grad_norm': '0.9922', 'learning_rate': '1.119e-05', 'entropy': '1.178', 'num_tokens': '1.597e+07', 'mean_token_accuracy': '0.7182', 'epoch': '2.671'} |
|
89%|██████████████████████████████████▋ | 980/1101 [48:58<24:07, 11.96s/it]
89%|██████████████████████████████████▋ | 981/1101 [49:10<23:56, 11.97s/it]
89%|██████████████████████████████████▊ | 982/1101 [49:22<23:50, 12.02s/it]
89%|██████████████████████████████████▊ | 983/1101 [49:34<23:15, 11.82s/it]
89%|██████████████████████████████████▊ | 984/1101 [49:46<23:15, 11.93s/it]
89%|██████████████████████████████████▉ | 985/1101 [49:58<23:03, 11.93s/it]
90%|██████████████████████████████████▉ | 986/1101 [50:10<22:49, 11.91s/it]
90%|██████████████████████████████████▉ | 987/1101 [50:22<22:44, 11.97s/it]
90%|██████████████████████████████████▉ | 988/1101 [50:33<22:13, 11.80s/it]
90%|███████████████████████████████████ | 989/1101 [50:45<21:55, 11.75s/it]
90%|███████████████████████████████████ | 990/1101 [50:57<21:50, 11.80s/it]
{'loss': '1.177', 'grad_norm': '0.9688', 'learning_rate': '1.13e-05', 'entropy': '1.174', 'num_tokens': '1.662e+07', 'mean_token_accuracy': '0.7186', 'epoch': '2.698'} |
|
90%|███████████████████████████████████ | 990/1101 [50:57<21:50, 11.80s/it]
90%|███████████████████████████████████ | 991/1101 [51:09<21:49, 11.90s/it]
90%|███████████████████████████████████▏ | 992/1101 [51:21<21:38, 11.91s/it]
90%|███████████████████████████████████▏ | 993/1101 [51:33<21:34, 11.98s/it]
90%|███████████████████████████████████▏ | 994/1101 [51:45<21:17, 11.94s/it]
90%|███████████████████████████████████▏ | 995/1101 [51:57<21:09, 11.98s/it]
90%|███████████████████████████████████▎ | 996/1101 [52:09<21:01, 12.01s/it]
91%|███████████████████████████████████▎ | 997/1101 [52:21<20:45, 11.97s/it]
91%|███████████████████████████████████▎ | 998/1101 [52:33<20:39, 12.03s/it]
91%|███████████████████████████████████▍ | 999/1101 [52:45<20:24, 12.01s/it]
91%|██████████████████████████████████▌ | 1000/1101 [52:57<20:18, 12.06s/it]
{'loss': '1.16', 'grad_norm': '1.062', 'learning_rate': '1.142e-05', 'entropy': '1.159', 'num_tokens': '1.727e+07', 'mean_token_accuracy': '0.7205', 'epoch': '2.725'} |
|
91%|██████████████████████████████████▌ | 1000/1101 [52:57<20:18, 12.06s/it]
91%|██████████████████████████████████▌ | 1001/1101 [53:09<20:01, 12.01s/it]
91%|██████████████████████████████████▌ | 1002/1101 [53:21<19:45, 11.97s/it]
91%|██████████████████████████████████▌ | 1003/1101 [53:33<19:39, 12.03s/it]
91%|██████████████████████████████████▋ | 1004/1101 [53:45<19:24, 12.00s/it]
91%|██████████████████████████████████▋ | 1005/1101 [53:57<19:18, 12.07s/it]
91%|██████████████████████████████████▋ | 1006/1101 [54:09<19:00, 12.01s/it]
91%|██████████████████████████████████▊ | 1007/1101 [54:21<18:56, 12.09s/it]
92%|██████████████████████████████████▊ | 1008/1101 [54:33<18:40, 12.04s/it]
92%|██████████████████████████████████▊ | 1009/1101 [54:45<18:24, 12.00s/it]
92%|██████████████████████████████████▊ | 1010/1101 [54:57<18:16, 12.05s/it]
{'loss': '1.185', 'grad_norm': '1.031', 'learning_rate': '1.153e-05', 'entropy': '1.183', 'num_tokens': '1.792e+07', 'mean_token_accuracy': '0.7198', 'epoch': '2.752'} |
|
92%|██████████████████████████████████▊ | 1010/1101 [54:57<18:16, 12.05s/it]
92%|██████████████████████████████████▉ | 1011/1101 [55:09<18:02, 12.03s/it]
92%|██████████████████████████████████▉ | 1012/1101 [55:21<17:53, 12.06s/it]
92%|██████████████████████████████████▉ | 1013/1101 [55:33<17:40, 12.05s/it]
92%|██████████████████████████████████▉ | 1014/1101 [55:46<17:31, 12.09s/it]
92%|███████████████████████████████████ | 1015/1101 [55:58<17:14, 12.03s/it]
92%|███████████████████████████████████ | 1016/1101 [56:09<16:58, 11.98s/it]
92%|███████████████████████████████████ | 1017/1101 [56:21<16:39, 11.90s/it]
92%|███████████████████████████████████▏ | 1018/1101 [56:33<16:27, 11.90s/it]
93%|███████████████████████████████████▏ | 1019/1101 [56:45<16:23, 11.99s/it]
93%|███████████████████████████████████▏ | 1020/1101 [56:57<16:10, 11.98s/it]
{'loss': '1.2', 'grad_norm': '1.055', 'learning_rate': '1.165e-05', 'entropy': '1.2', 'num_tokens': '1.858e+07', 'mean_token_accuracy': '0.719', 'epoch': '2.78'} |
|
93%|███████████████████████████████████▏ | 1020/1101 [56:57<16:10, 11.98s/it]
93%|███████████████████████████████████▏ | 1021/1101 [57:09<16:03, 12.04s/it]
93%|███████████████████████████████████▎ | 1022/1101 [57:21<15:39, 11.89s/it]
93%|███████████████████████████████████▎ | 1023/1101 [57:33<15:28, 11.90s/it]
93%|███████████████████████████████████▎ | 1024/1101 [57:45<15:21, 11.97s/it]
93%|███████████████████████████████████▍ | 1025/1101 [57:57<15:04, 11.90s/it]
93%|███████████████████████████████████▍ | 1026/1101 [58:09<15:00, 12.00s/it]
93%|███████████████████████████████████▍ | 1027/1101 [58:21<14:47, 12.00s/it]
93%|███████████████████████████████████▍ | 1028/1101 [58:33<14:38, 12.03s/it]
93%|███████████████████████████████████▌ | 1029/1101 [58:45<14:15, 11.89s/it]
94%|███████████████████████████████████▌ | 1030/1101 [58:57<14:09, 11.96s/it]
{'loss': '1.169', 'grad_norm': '0.9609', 'learning_rate': '1.176e-05', 'entropy': '1.17', 'num_tokens': '1.923e+07', 'mean_token_accuracy': '0.7213', 'epoch': '2.807'} |
|
94%|███████████████████████████████████▌ | 1030/1101 [58:57<14:09, 11.96s/it]
94%|███████████████████████████████████▌ | 1031/1101 [59:09<13:56, 11.95s/it]
94%|███████████████████████████████████▌ | 1032/1101 [59:19<13:22, 11.62s/it]
94%|███████████████████████████████████▋ | 1033/1101 [59:32<13:20, 11.78s/it]
94%|███████████████████████████████████▋ | 1034/1101 [59:44<13:12, 11.83s/it]
94%|███████████████████████████████████▋ | 1035/1101 [59:56<13:06, 11.91s/it]
94%|█████████████████████████████████▊ | 1036/1101 [1:00:08<12:53, 11.91s/it]
94%|█████████████████████████████████▉ | 1037/1101 [1:00:20<12:46, 11.98s/it]
94%|█████████████████████████████████▉ | 1038/1101 [1:00:32<12:33, 11.96s/it]
94%|█████████████████████████████████▉ | 1039/1101 [1:00:43<12:19, 11.93s/it]
94%|██████████████████████████████████ | 1040/1101 [1:00:56<12:11, 12.00s/it]
{'loss': '1.187', 'grad_norm': '0.9883', 'learning_rate': '1.187e-05', 'entropy': '1.185', 'num_tokens': '1.987e+07', 'mean_token_accuracy': '0.7195', 'epoch': '2.834'} |
|
94%|██████████████████████████████████ | 1040/1101 [1:00:56<12:11, 12.00s/it]
95%|██████████████████████████████████ | 1041/1101 [1:01:08<11:59, 12.00s/it]
95%|██████████████████████████████████ | 1042/1101 [1:01:20<11:51, 12.06s/it]
95%|██████████████████████████████████ | 1043/1101 [1:01:32<11:37, 12.03s/it]
95%|██████████████████████████████████▏ | 1044/1101 [1:01:44<11:28, 12.08s/it]
95%|██████████████████████████████████▏ | 1045/1101 [1:01:56<11:15, 12.06s/it]
95%|██████████████████████████████████▏ | 1046/1101 [1:02:08<11:01, 12.03s/it]
95%|██████████████████████████████████▏ | 1047/1101 [1:02:20<10:49, 12.03s/it]
95%|██████████████████████████████████▎ | 1048/1101 [1:02:32<10:36, 12.00s/it]
95%|██████████████████████████████████▎ | 1049/1101 [1:02:44<10:27, 12.06s/it]
95%|██████████████████████████████████▎ | 1050/1101 [1:02:56<10:04, 11.86s/it]
{'loss': '1.164', 'grad_norm': '0.957', 'learning_rate': '1.199e-05', 'entropy': '1.166', 'num_tokens': '2.052e+07', 'mean_token_accuracy': '0.7222', 'epoch': '2.861'} |
|
95%|██████████████████████████████████▎ | 1050/1101 [1:02:56<10:04, 11.86s/it]
95%|██████████████████████████████████▎ | 1051/1101 [1:03:08<09:56, 11.93s/it]
96%|██████████████████████████████████▍ | 1052/1101 [1:03:20<09:44, 11.94s/it]
96%|██████████████████████████████████▍ | 1053/1101 [1:03:31<09:26, 11.81s/it]
96%|██████████████████████████████████▍ | 1054/1101 [1:03:43<09:20, 11.92s/it]
96%|██████████████████████████████████▍ | 1055/1101 [1:03:55<09:07, 11.89s/it]
96%|██████████████████████████████████▌ | 1056/1101 [1:04:07<08:58, 11.97s/it]
96%|██████████████████████████████████▌ | 1057/1101 [1:04:19<08:46, 11.96s/it]
96%|██████████████████████████████████▌ | 1058/1101 [1:04:31<08:36, 12.02s/it]
96%|██████████████████████████████████▋ | 1059/1101 [1:04:43<08:23, 11.98s/it]
96%|██████████████████████████████████▋ | 1060/1101 [1:04:55<08:10, 11.96s/it]
{'loss': '1.168', 'grad_norm': '1.234', 'learning_rate': '1.21e-05', 'entropy': '1.17', 'num_tokens': '2.118e+07', 'mean_token_accuracy': '0.72', 'epoch': '2.889'} |
|
96%|██████████████████████████████████▋ | 1060/1101 [1:04:55<08:10, 11.96s/it]
96%|██████████████████████████████████▋ | 1061/1101 [1:05:07<08:00, 12.02s/it]
96%|██████████████████████████████████▋ | 1062/1101 [1:05:19<07:47, 12.00s/it]
97%|██████████████████████████████████▊ | 1063/1101 [1:05:31<07:33, 11.92s/it]
97%|██████████████████████████████████▊ | 1064/1101 [1:05:43<07:21, 11.93s/it]
97%|██████████████████████████████████▊ | 1065/1101 [1:05:55<07:07, 11.87s/it]
97%|██████████████████████████████████▊ | 1066/1101 [1:06:07<06:55, 11.87s/it]
97%|██████████████████████████████████▉ | 1067/1101 [1:06:19<06:46, 11.96s/it]
97%|██████████████████████████████████▉ | 1068/1101 [1:06:31<06:34, 11.96s/it]
97%|██████████████████████████████████▉ | 1069/1101 [1:06:43<06:22, 11.96s/it]
97%|██████████████████████████████████▉ | 1070/1101 [1:06:55<06:12, 12.02s/it]
{'loss': '1.187', 'grad_norm': '1.164', 'learning_rate': '1.222e-05', 'entropy': '1.187', 'num_tokens': '2.183e+07', 'mean_token_accuracy': '0.7183', 'epoch': '2.916'} |
|
97%|██████████████████████████████████▉ | 1070/1101 [1:06:55<06:12, 12.02s/it]
97%|███████████████████████████████████ | 1071/1101 [1:07:07<05:59, 11.99s/it]
97%|███████████████████████████████████ | 1072/1101 [1:07:19<05:49, 12.06s/it]
97%|███████████████████████████████████ | 1073/1101 [1:07:31<05:36, 12.01s/it]
98%|███████████████████████████████████ | 1074/1101 [1:07:43<05:25, 12.06s/it]
98%|███████████████████████████████████▏| 1075/1101 [1:07:55<05:12, 12.03s/it]
98%|███████████████████████████████████▏| 1076/1101 [1:08:06<04:55, 11.81s/it]
98%|███████████████████████████████████▏| 1077/1101 [1:08:18<04:46, 11.92s/it]
98%|███████████████████████████████████▏| 1078/1101 [1:08:30<04:30, 11.75s/it]
98%|███████████████████████████████████▎| 1079/1101 [1:08:42<04:21, 11.89s/it]
98%|███████████████████████████████████▎| 1080/1101 [1:08:54<04:09, 11.90s/it]
{'loss': '1.194', 'grad_norm': '1.086', 'learning_rate': '1.233e-05', 'entropy': '1.193', 'num_tokens': '2.248e+07', 'mean_token_accuracy': '0.7175', 'epoch': '2.943'} |
|
98%|███████████████████████████████████▎| 1080/1101 [1:08:54<04:09, 11.90s/it]
98%|███████████████████████████████████▎| 1081/1101 [1:09:06<03:59, 11.98s/it]
98%|███████████████████████████████████▍| 1082/1101 [1:09:18<03:47, 11.97s/it]
98%|███████████████████████████████████▍| 1083/1101 [1:09:30<03:34, 11.89s/it]
98%|███████████████████████████████████▍| 1084/1101 [1:09:42<03:23, 11.97s/it]
99%|███████████████████████████████████▍| 1085/1101 [1:09:54<03:11, 11.96s/it]
99%|███████████████████████████████████▌| 1086/1101 [1:10:06<03:00, 12.03s/it]
99%|███████████████████████████████████▌| 1087/1101 [1:10:18<02:46, 11.87s/it]
99%|███████████████████████████████████▌| 1088/1101 [1:10:30<02:35, 11.94s/it]
99%|███████████████████████████████████▌| 1089/1101 [1:10:42<02:23, 11.95s/it]
99%|███████████████████████████████████▋| 1090/1101 [1:10:53<02:10, 11.88s/it]
{'loss': '1.188', 'grad_norm': '1.219', 'learning_rate': '1.245e-05', 'entropy': '1.186', 'num_tokens': '2.313e+07', 'mean_token_accuracy': '0.72', 'epoch': '2.97'} |
|
99%|███████████████████████████████████▋| 1090/1101 [1:10:53<02:10, 11.88s/it]
99%|███████████████████████████████████▋| 1091/1101 [1:11:05<01:59, 11.97s/it]
99%|███████████████████████████████████▋| 1092/1101 [1:11:17<01:47, 11.96s/it]
99%|███████████████████████████████████▋| 1093/1101 [1:11:29<01:35, 11.95s/it]
99%|███████████████████████████████████▊| 1094/1101 [1:11:41<01:23, 11.90s/it]
99%|███████████████████████████████████▊| 1095/1101 [1:11:53<01:11, 11.96s/it]
100%|███████████████████████████████████▊| 1096/1101 [1:12:05<00:59, 11.93s/it]
100%|███████████████████████████████████▊| 1097/1101 [1:12:17<00:47, 11.83s/it]
100%|███████████████████████████████████▉| 1098/1101 [1:12:29<00:35, 11.95s/it]
100%|███████████████████████████████████▉| 1099/1101 [1:12:41<00:23, 11.95s/it]
100%|███████████████████████████████████▉| 1100/1101 [1:12:53<00:12, 12.01s/it]
{'loss': '1.18', 'grad_norm': '1.117', 'learning_rate': '1.256e-05', 'entropy': '1.181', 'num_tokens': '2.378e+07', 'mean_token_accuracy': '0.7191', 'epoch': '2.998'} |
|
100%|███████████████████████████████████▉| 1100/1101 [1:12:53<00:12, 12.01s/it]
100%|████████████████████████████████████| 1101/1101 [1:13:03<00:00, 11.54s/it] |
|
Writing model shards: 0%| | 0/1 [00:00<?, ?it/s][A |
|
Writing model shards: 100%|██████████████████████| 1/1 [00:08<00:00, 8.98s/it][A
Writing model shards: 100%|██████████████████████| 1/1 [00:08<00:00, 8.98s/it] |
|
{'train_runtime': '4406', 'train_samples_per_second': '3.997', 'train_steps_per_second': '0.25', 'train_loss': '0.3994', 'entropy': '1.182', 'num_tokens': '2.383e+07', 'mean_token_accuracy': '0.7228', 'epoch': '3'} |
|
100%|████████████████████████████████████| 1101/1101 [1:13:25<00:00, 11.54s/it]
100%|████████████████████████████████████| 1101/1101 [1:13:25<00:00, 4.00s/it] |
|
|
| Saving model to /scratch/checkpoints/multilingual_sft_v2_ci_bs1_ga16_lr2e5_ep3 |
|
Writing model shards: 0%| | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|██████████████████████| 1/1 [00:10<00:00, 10.03s/it]
Writing model shards: 100%|██████████████████████| 1/1 [00:10<00:00, 10.03s/it] |
| Saved /scratch/checkpoints/multilingual_sft_v2_ci_bs1_ga16_lr2e5_ep3/generation_config.json |
| Embedded chat_template into tokenizer_config.json |
|
|
| Done! Checkpoint saved to: /scratch/checkpoints/multilingual_sft_v2_ci_bs1_ga16_lr2e5_ep3 |
| Next steps: |
| 1. Test generation from /scratch/checkpoints/multilingual_sft_v2_ci_bs1_ga16_lr2e5_ep3 |
| 2. Push to HF multilingual_model repo |
|
|
