model training desc:  做知识选择，使用QuALITY数据集，随机选择的知识和关键句训练
2023-12-16 23:07:37.734 | INFO     | __main__:init_components:108 - Initializing components...

Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:  50%|█████     | 1/2 [00:48<00:48, 48.58s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [01:02<00:00, 28.35s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [01:02<00:00, 31.38s/it]
You are using the default legacy behaviour of the <class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'>. If you see this, DO NOT PANIC! This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thouroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565
2023-12-16 23:08:41.148 | INFO     | __main__:init_components:155 - 

2023-12-16 23:08:41.148 | INFO     | __main__:init_components:156 - ********************
2023-12-16 23:08:41.148 | INFO     | __main__:init_components:157 - using llama2 model
2023-12-16 23:08:41.148 | INFO     | __main__:init_components:158 - ********************
2023-12-16 23:08:41.148 | INFO     | __main__:init_components:159 - 

memory footprint of model: 4.024436950683594 GB
trainable params: 319,815,680 || all params: 7,058,231,296 || trainable%: 4.531102291607305
2023-12-16 23:08:44.549 | INFO     | component.dataset:__init__:14 - Loading data: /data0/maqi/KGLQA-data/datasets/QuALITY/random_select/with_knowledge_without_select_instruction/train.jsonl
2023-12-16 23:08:44.647 | INFO     | component.dataset:__init__:19 - there are 2523 data in dataset
2023-12-16 23:08:44.773 | INFO     | __main__:main:231 - *** starting training ***

  0%|          | 0/630 [00:00<?, ?it/s]
  0%|          | 1/630 [01:28<15:29:52, 88.70s/it]
  0%|          | 2/630 [02:48<14:34:48, 83.58s/it]
  0%|          | 3/630 [04:10<14:23:46, 82.66s/it]
  1%|          | 4/630 [05:32<14:18:41, 82.30s/it]
  1%|          | 5/630 [06:53<14:15:21, 82.11s/it]
  1%|          | 6/630 [08:16<14:15:39, 82.28s/it]
  1%|          | 7/630 [09:38<14:12:59, 82.15s/it]
  1%|▏         | 8/630 [10:59<14:09:57, 81.99s/it]
  1%|▏         | 9/630 [12:21<14:08:12, 81.95s/it]
  2%|▏         | 10/630 [13:16<12:38:25, 73.40s/it]
  2%|▏         | 11/630 [13:52<10:41:19, 62.16s/it]
  2%|▏         | 12/630 [14:29<9:20:14, 54.39s/it] 
  2%|▏         | 13/630 [15:05<8:23:57, 49.01s/it]
  2%|▏         | 14/630 [15:42<7:43:09, 45.11s/it]
  2%|▏         | 15/630 [16:18<7:16:08, 42.55s/it]
  3%|▎         | 16/630 [16:55<6:57:10, 40.77s/it]
  3%|▎         | 17/630 [17:32<6:44:01, 39.55s/it]
  3%|▎         | 18/630 [18:08<6:34:23, 38.67s/it]
  3%|▎         | 19/630 [18:45<6:28:19, 38.13s/it]
  3%|▎         | 20/630 [19:20<6:17:18, 37.11s/it]
                                                  
{'loss': 0.5195, 'learning_rate': 3.1746031746031745e-05, 'global_step': 20, 'epoch': 0.1}

  3%|▎         | 20/630 [19:20<6:17:18, 37.11s/it]
  3%|▎         | 21/630 [19:57<6:15:54, 37.03s/it]
  3%|▎         | 22/630 [20:33<6:14:20, 36.94s/it]
  4%|▎         | 23/630 [21:10<6:12:23, 36.81s/it]
  4%|▍         | 24/630 [21:47<6:11:31, 36.78s/it]
  4%|▍         | 25/630 [22:23<6:10:36, 36.75s/it]
  4%|▍         | 26/630 [23:00<6:09:53, 36.74s/it]
  4%|▍         | 27/630 [23:37<6:09:29, 36.76s/it]
  4%|▍         | 28/630 [24:14<6:08:44, 36.75s/it]
  5%|▍         | 29/630 [24:50<6:06:48, 36.62s/it]
  5%|▍         | 30/630 [25:27<6:06:30, 36.65s/it]
  5%|▍         | 31/630 [26:03<6:06:41, 36.73s/it]
  5%|▌         | 32/630 [26:40<6:05:45, 36.70s/it]
  5%|▌         | 33/630 [27:17<6:05:11, 36.70s/it]
  5%|▌         | 34/630 [27:54<6:04:39, 36.71s/it]
  6%|▌         | 35/630 [28:30<6:03:47, 36.68s/it]
  6%|▌         | 36/630 [29:07<6:03:48, 36.75s/it]
  6%|▌         | 37/630 [29:44<6:02:54, 36.72s/it]
  6%|▌         | 38/630 [30:20<6:02:17, 36.72s/it]
  6%|▌         | 39/630 [30:57<6:01:33, 36.71s/it]
  6%|▋         | 40/630 [31:34<6:00:55, 36.70s/it]
                                                  
{'loss': 0.5055, 'learning_rate': 6.349206349206349e-05, 'global_step': 40, 'epoch': 0.19}

  6%|▋         | 40/630 [31:34<6:00:55, 36.70s/it]
  7%|▋         | 41/630 [32:10<6:00:05, 36.68s/it]
  7%|▋         | 42/630 [32:47<5:59:28, 36.68s/it]
  7%|▋         | 43/630 [33:24<5:58:37, 36.66s/it]
  7%|▋         | 44/630 [34:00<5:57:55, 36.65s/it]
  7%|▋         | 45/630 [34:37<5:57:30, 36.67s/it]
  7%|▋         | 46/630 [35:14<5:57:03, 36.68s/it]
  7%|▋         | 47/630 [35:50<5:56:33, 36.70s/it]
  8%|▊         | 48/630 [36:27<5:56:19, 36.74s/it]
  8%|▊         | 49/630 [37:04<5:55:23, 36.70s/it]
  8%|▊         | 50/630 [37:41<5:55:16, 36.75s/it]
  8%|▊         | 51/630 [38:17<5:54:16, 36.71s/it]
  8%|▊         | 52/630 [38:54<5:53:36, 36.71s/it]
  8%|▊         | 53/630 [39:31<5:52:58, 36.70s/it]
  9%|▊         | 54/630 [40:08<5:52:24, 36.71s/it]
  9%|▊         | 55/630 [40:44<5:51:43, 36.70s/it]
  9%|▉         | 56/630 [41:21<5:51:02, 36.69s/it]
  9%|▉         | 57/630 [41:58<5:50:25, 36.69s/it]
  9%|▉         | 58/630 [42:34<5:49:38, 36.67s/it]
  9%|▉         | 59/630 [43:11<5:49:06, 36.68s/it]
 10%|▉         | 60/630 [43:48<5:48:36, 36.70s/it]
                                                  
{'loss': 0.5483, 'learning_rate': 9.523809523809524e-05, 'global_step': 60, 'epoch': 0.29}

 10%|▉         | 60/630 [43:48<5:48:36, 36.70s/it]
 10%|▉         | 61/630 [44:24<5:48:05, 36.71s/it]
 10%|▉         | 62/630 [45:01<5:47:12, 36.68s/it]
 10%|█         | 63/630 [45:38<5:46:13, 36.64s/it]
 10%|█         | 64/630 [46:14<5:45:51, 36.66s/it]
 10%|█         | 65/630 [46:51<5:45:21, 36.68s/it]
 10%|█         | 66/630 [47:28<5:45:26, 36.75s/it]
 11%|█         | 67/630 [48:05<5:44:57, 36.76s/it]
 11%|█         | 68/630 [48:41<5:44:27, 36.78s/it]
 11%|█         | 69/630 [49:18<5:43:38, 36.75s/it]
 11%|█         | 70/630 [49:54<5:41:28, 36.59s/it]
 11%|█▏        | 71/630 [50:31<5:41:02, 36.61s/it]
 11%|█▏        | 72/630 [51:08<5:40:28, 36.61s/it]
 12%|█▏        | 73/630 [51:44<5:39:51, 36.61s/it]
 12%|█▏        | 74/630 [52:21<5:39:47, 36.67s/it]
 12%|█▏        | 75/630 [52:58<5:39:09, 36.67s/it]
 12%|█▏        | 76/630 [53:34<5:38:41, 36.68s/it]
 12%|█▏        | 77/630 [54:11<5:38:25, 36.72s/it]
 12%|█▏        | 78/630 [54:48<5:38:03, 36.75s/it]
 13%|█▎        | 79/630 [55:25<5:37:16, 36.73s/it]
 13%|█▎        | 80/630 [56:02<5:36:51, 36.75s/it]
                                                  
{'loss': 0.4659, 'learning_rate': 0.0001, 'global_step': 80, 'epoch': 0.38}

 13%|█▎        | 80/630 [56:02<5:36:51, 36.75s/it]
 13%|█▎        | 81/630 [56:38<5:35:51, 36.71s/it]
 13%|█▎        | 82/630 [57:15<5:35:14, 36.71s/it]
 13%|█▎        | 83/630 [57:52<5:34:37, 36.71s/it]
 13%|█▎        | 84/630 [58:28<5:34:32, 36.76s/it]
 13%|█▎        | 85/630 [59:05<5:33:41, 36.74s/it]
 14%|█▎        | 86/630 [59:42<5:33:31, 36.79s/it]
 14%|█▍        | 87/630 [1:00:19<5:32:41, 36.76s/it]
 14%|█▍        | 88/630 [1:00:55<5:31:40, 36.72s/it]
 14%|█▍        | 89/630 [1:01:32<5:31:14, 36.74s/it]
 14%|█▍        | 90/630 [1:02:09<5:30:50, 36.76s/it]
 14%|█▍        | 91/630 [1:02:46<5:30:03, 36.74s/it]
 15%|█▍        | 92/630 [1:03:22<5:29:23, 36.73s/it]
 15%|█▍        | 93/630 [1:03:59<5:28:42, 36.73s/it]
 15%|█▍        | 94/630 [1:04:36<5:28:16, 36.75s/it]
 15%|█▌        | 95/630 [1:05:12<5:27:19, 36.71s/it]
 15%|█▌        | 96/630 [1:05:49<5:26:29, 36.69s/it]
 15%|█▌        | 97/630 [1:06:26<5:26:04, 36.71s/it]
 16%|█▌        | 98/630 [1:07:32<6:43:26, 45.50s/it]
 16%|█▌        | 99/630 [1:08:52<8:15:43, 56.01s/it]
 16%|█▌        | 100/630 [1:10:14<9:23:01, 63.74s/it]
                                                     
{'loss': 0.5625, 'learning_rate': 0.0001, 'global_step': 100, 'epoch': 0.48}

 16%|█▌        | 100/630 [1:10:14<9:23:01, 63.74s/it]
 16%|█▌        | 101/630 [1:11:36<10:09:46, 69.16s/it]
 16%|█▌        | 102/630 [1:12:58<10:42:36, 73.02s/it]
 16%|█▋        | 103/630 [1:14:20<11:04:01, 75.60s/it]
 17%|█▋        | 104/630 [1:15:42<11:19:25, 77.50s/it]
 17%|█▋        | 105/630 [1:17:03<11:28:59, 78.74s/it]
 17%|█▋        | 106/630 [1:18:25<11:35:56, 79.69s/it]
 17%|█▋        | 107/630 [1:19:47<11:39:41, 80.27s/it]
 17%|█▋        | 108/630 [1:21:08<11:41:52, 80.68s/it]
 17%|█▋        | 109/630 [1:22:30<11:43:33, 81.02s/it]
 17%|█▋        | 110/630 [1:23:52<11:43:00, 81.12s/it]
 18%|█▊        | 111/630 [1:25:14<11:44:41, 81.47s/it]
 18%|█▊        | 112/630 [1:26:36<11:44:32, 81.61s/it]
 18%|█▊        | 113/630 [1:27:58<11:43:51, 81.69s/it]
 18%|█▊        | 114/630 [1:29:19<11:42:07, 81.64s/it]
 18%|█▊        | 115/630 [1:30:41<11:40:43, 81.64s/it]
 18%|█▊        | 116/630 [1:32:03<11:40:05, 81.72s/it]
 19%|█▊        | 117/630 [1:33:25<11:39:02, 81.76s/it]
 19%|█▊        | 118/630 [1:34:47<11:38:07, 81.81s/it]
 19%|█▉        | 119/630 [1:36:09<11:37:18, 81.88s/it]
 19%|█▉        | 120/630 [1:37:28<11:30:08, 81.19s/it]
                                                      
{'loss': 0.545, 'learning_rate': 0.0001, 'global_step': 120, 'epoch': 0.57}

 19%|█▉        | 120/630 [1:37:28<11:30:08, 81.19s/it]
 19%|█▉        | 121/630 [1:38:50<11:30:27, 81.39s/it]
 19%|█▉        | 122/630 [1:40:11<11:29:23, 81.42s/it]
 20%|█▉        | 123/630 [1:41:34<11:29:43, 81.62s/it]
 20%|█▉        | 124/630 [1:42:55<11:29:06, 81.71s/it]
 20%|█▉        | 125/630 [1:44:17<11:27:27, 81.68s/it]
 20%|██        | 126/630 [1:45:39<11:25:25, 81.60s/it]
 20%|██        | 127/630 [1:47:00<11:24:39, 81.67s/it]
 20%|██        | 128/630 [1:48:20<11:17:02, 80.92s/it]
 20%|██        | 129/630 [1:49:41<11:17:34, 81.15s/it]
 21%|██        | 130/630 [1:51:03<11:18:30, 81.42s/it]
 21%|██        | 131/630 [1:52:25<11:18:42, 81.61s/it]
 21%|██        | 132/630 [1:53:47<11:18:20, 81.73s/it]
 21%|██        | 133/630 [1:55:09<11:16:46, 81.70s/it]
 21%|██▏       | 134/630 [1:56:31<11:17:03, 81.90s/it]
 21%|██▏       | 135/630 [1:57:50<11:08:23, 81.02s/it]
 22%|██▏       | 136/630 [1:59:07<10:56:04, 79.68s/it]
 22%|██▏       | 137/630 [2:00:28<10:59:34, 80.27s/it]
 22%|██▏       | 138/630 [2:01:50<11:02:16, 80.76s/it]
 22%|██▏       | 139/630 [2:03:12<11:02:53, 81.01s/it]
 22%|██▏       | 140/630 [2:04:34<11:03:02, 81.19s/it]
                                                      
{'loss': 0.4962, 'learning_rate': 0.0001, 'global_step': 140, 'epoch': 0.67}

 22%|██▏       | 140/630 [2:04:34<11:03:02, 81.19s/it]
 22%|██▏       | 141/630 [2:05:56<11:03:47, 81.45s/it]
 23%|██▎       | 142/630 [2:07:18<11:03:36, 81.59s/it]
 23%|██▎       | 143/630 [2:08:39<11:02:59, 81.68s/it]
 23%|██▎       | 144/630 [2:10:01<11:02:11, 81.75s/it]
 23%|██▎       | 145/630 [2:11:23<11:01:10, 81.80s/it]
 23%|██▎       | 146/630 [2:12:45<10:59:37, 81.77s/it]
 23%|██▎       | 147/630 [2:14:07<10:58:24, 81.79s/it]
 23%|██▎       | 148/630 [2:15:29<10:57:18, 81.82s/it]
 24%|██▎       | 149/630 [2:16:51<10:56:10, 81.85s/it]
 24%|██▍       | 150/630 [2:18:13<10:54:54, 81.86s/it]
 24%|██▍       | 151/630 [2:19:34<10:53:13, 81.82s/it]
 24%|██▍       | 152/630 [2:20:56<10:52:38, 81.92s/it]
 24%|██▍       | 153/630 [2:22:18<10:50:37, 81.84s/it]
 24%|██▍       | 154/630 [2:23:40<10:49:02, 81.81s/it]
 25%|██▍       | 155/630 [2:25:03<10:51:47, 82.33s/it]
 25%|██▍       | 156/630 [2:26:27<10:53:18, 82.70s/it]
 25%|██▍       | 157/630 [2:27:49<10:49:56, 82.45s/it]
 25%|██▌       | 158/630 [2:29:11<10:47:46, 82.34s/it]
 25%|██▌       | 159/630 [2:30:33<10:45:18, 82.20s/it]
 25%|██▌       | 160/630 [2:31:54<10:42:42, 82.05s/it]
                                                      
{'loss': 0.5508, 'learning_rate': 0.0001, 'global_step': 160, 'epoch': 0.76}

 25%|██▌       | 160/630 [2:31:54<10:42:42, 82.05s/it]
 26%|██▌       | 161/630 [2:33:16<10:40:25, 81.93s/it]
 26%|██▌       | 162/630 [2:34:38<10:38:02, 81.80s/it]
 26%|██▌       | 163/630 [2:36:00<10:37:19, 81.88s/it]
 26%|██▌       | 164/630 [2:37:21<10:33:44, 81.60s/it]
 26%|██▌       | 165/630 [2:38:42<10:32:23, 81.60s/it]
 26%|██▋       | 166/630 [2:40:04<10:31:41, 81.68s/it]
 27%|██▋       | 167/630 [2:41:26<10:30:15, 81.68s/it]
 27%|██▋       | 168/630 [2:42:47<10:28:49, 81.67s/it]
 27%|██▋       | 169/630 [2:44:09<10:27:50, 81.71s/it]
 27%|██▋       | 170/630 [2:45:31<10:27:21, 81.83s/it]
 27%|██▋       | 171/630 [2:46:53<10:26:38, 81.91s/it]
 27%|██▋       | 172/630 [2:48:15<10:25:16, 81.91s/it]
 27%|██▋       | 173/630 [2:49:38<10:24:43, 82.02s/it]
 28%|██▊       | 174/630 [2:50:59<10:22:55, 81.96s/it]
 28%|██▊       | 175/630 [2:52:21<10:21:21, 81.94s/it]
 28%|██▊       | 176/630 [2:53:44<10:20:46, 82.04s/it]
 28%|██▊       | 177/630 [2:55:06<10:19:28, 82.05s/it]
 28%|██▊       | 178/630 [2:56:27<10:17:36, 81.98s/it]
 28%|██▊       | 179/630 [2:57:49<10:15:59, 81.95s/it]
 29%|██▊       | 180/630 [2:59:11<10:13:51, 81.85s/it]
                                                      
{'loss': 0.5314, 'learning_rate': 0.0001, 'global_step': 180, 'epoch': 0.86}

 29%|██▊       | 180/630 [2:59:11<10:13:51, 81.85s/it]
 29%|██▊       | 181/630 [3:00:33<10:12:41, 81.87s/it]
 29%|██▉       | 182/630 [3:01:54<10:10:10, 81.72s/it]
 29%|██▉       | 183/630 [3:03:16<10:08:40, 81.70s/it]
 29%|██▉       | 184/630 [3:04:38<10:08:04, 81.80s/it]
 29%|██▉       | 185/630 [3:05:58<10:03:33, 81.38s/it]
 30%|██▉       | 186/630 [3:07:20<10:03:04, 81.50s/it]
 30%|██▉       | 187/630 [3:08:42<10:02:12, 81.56s/it]
 30%|██▉       | 188/630 [3:10:03<10:00:57, 81.58s/it]
 30%|███       | 189/630 [3:11:25<10:00:18, 81.68s/it]
 30%|███       | 190/630 [3:12:47<9:58:54, 81.67s/it] 
 30%|███       | 191/630 [3:14:09<9:59:05, 81.88s/it]
 30%|███       | 192/630 [3:15:31<9:57:08, 81.80s/it]
 31%|███       | 193/630 [3:16:53<9:56:32, 81.91s/it]
 31%|███       | 194/630 [3:18:15<9:55:06, 81.90s/it]
 31%|███       | 195/630 [3:19:37<9:53:02, 81.80s/it]
 31%|███       | 196/630 [3:20:59<9:52:13, 81.88s/it]
 31%|███▏      | 197/630 [3:22:21<9:50:57, 81.89s/it]
 31%|███▏      | 198/630 [3:23:42<9:49:23, 81.86s/it]
 32%|███▏      | 199/630 [3:25:04<9:47:31, 81.79s/it]
 32%|███▏      | 200/630 [3:26:26<9:45:53, 81.75s/it]
                                                     
{'loss': 0.5251, 'learning_rate': 0.0001, 'global_step': 200, 'epoch': 0.95}

 32%|███▏      | 200/630 [3:26:26<9:45:53, 81.75s/it]
 32%|███▏      | 201/630 [3:27:47<9:44:42, 81.78s/it]
 32%|███▏      | 202/630 [3:29:10<9:44:02, 81.88s/it]
 32%|███▏      | 203/630 [3:30:31<9:42:42, 81.88s/it]
 32%|███▏      | 204/630 [3:31:53<9:40:47, 81.80s/it]
 33%|███▎      | 205/630 [3:33:15<9:40:30, 81.96s/it]
 33%|███▎      | 206/630 [3:34:37<9:38:53, 81.92s/it]
 33%|███▎      | 207/630 [3:35:59<9:37:21, 81.90s/it]
 33%|███▎      | 208/630 [3:37:21<9:35:55, 81.89s/it]
 33%|███▎      | 209/630 [3:38:43<9:34:01, 81.81s/it]
 33%|███▎      | 210/630 [3:40:04<9:32:44, 81.82s/it]
 33%|███▎      | 211/630 [3:41:21<9:21:02, 80.34s/it]
 34%|███▎      | 212/630 [3:42:43<9:22:53, 80.80s/it]
 34%|███▍      | 213/630 [3:44:05<9:23:41, 81.11s/it]
 34%|███▍      | 214/630 [3:45:27<9:23:30, 81.28s/it]
 34%|███▍      | 215/630 [3:46:49<9:23:21, 81.45s/it]
 34%|███▍      | 216/630 [3:48:09<9:19:44, 81.12s/it]
 34%|███▍      | 217/630 [3:49:31<9:19:51, 81.34s/it]
 35%|███▍      | 218/630 [3:50:53<9:20:33, 81.63s/it]
 35%|███▍      | 219/630 [3:52:15<9:19:09, 81.63s/it]
 35%|███▍      | 220/630 [3:53:37<9:18:21, 81.71s/it]
                                                     
{'loss': 0.347, 'learning_rate': 0.0001, 'global_step': 220, 'epoch': 1.05}

 35%|███▍      | 220/630 [3:53:37<9:18:21, 81.71s/it]
 35%|███▌      | 221/630 [3:54:58<9:17:11, 81.74s/it]
 35%|███▌      | 222/630 [3:56:20<9:16:33, 81.85s/it]
 35%|███▌      | 223/630 [3:57:42<9:14:36, 81.76s/it]
 36%|███▌      | 224/630 [3:59:04<9:13:20, 81.77s/it]
 36%|███▌      | 225/630 [4:00:26<9:11:49, 81.75s/it]
 36%|███▌      | 226/630 [4:01:48<9:11:14, 81.87s/it]
 36%|███▌      | 227/630 [4:03:10<9:10:51, 82.01s/it]
 36%|███▌      | 228/630 [4:04:32<9:09:14, 81.98s/it]
 36%|███▋      | 229/630 [4:05:54<9:07:39, 81.95s/it]
 37%|███▋      | 230/630 [4:07:16<9:06:14, 81.94s/it]
 37%|███▋      | 231/630 [4:08:38<9:04:45, 81.92s/it]
 37%|███▋      | 232/630 [4:09:59<9:03:23, 81.92s/it]
 37%|███▋      | 233/630 [4:11:21<9:01:36, 81.85s/it]
 37%|███▋      | 234/630 [4:12:43<9:00:02, 81.82s/it]
 37%|███▋      | 235/630 [4:14:05<8:58:49, 81.85s/it]
 37%|███▋      | 236/630 [4:15:27<8:57:08, 81.80s/it]
 38%|███▊      | 237/630 [4:16:49<8:56:24, 81.90s/it]
 38%|███▊      | 238/630 [4:18:11<8:54:58, 81.89s/it]
 38%|███▊      | 239/630 [4:19:33<8:54:27, 82.01s/it]
 38%|███▊      | 240/630 [4:20:55<8:52:26, 81.92s/it]
                                                     
{'loss': 0.2006, 'learning_rate': 0.0001, 'global_step': 240, 'epoch': 1.14}

 38%|███▊      | 240/630 [4:20:55<8:52:26, 81.92s/it]
 38%|███▊      | 241/630 [4:22:17<8:51:17, 81.95s/it]
 38%|███▊      | 242/630 [4:23:39<8:50:16, 82.00s/it]
 39%|███▊      | 243/630 [4:25:01<8:48:40, 81.97s/it]
 39%|███▊      | 244/630 [4:26:22<8:46:45, 81.88s/it]
 39%|███▉      | 245/630 [4:27:44<8:44:58, 81.81s/it]
 39%|███▉      | 246/630 [4:29:06<8:43:15, 81.76s/it]
 39%|███▉      | 247/630 [4:30:23<8:34:15, 80.56s/it]
 39%|███▉      | 248/630 [4:31:45<8:35:40, 81.00s/it]
 40%|███▉      | 249/630 [4:33:05<8:31:06, 80.49s/it]
 40%|███▉      | 250/630 [4:34:26<8:32:13, 80.88s/it]
 40%|███▉      | 251/630 [4:35:48<8:32:46, 81.18s/it]
 40%|████      | 252/630 [4:37:10<8:32:37, 81.37s/it]
 40%|████      | 253/630 [4:38:32<8:32:10, 81.51s/it]
 40%|████      | 254/630 [4:39:54<8:31:53, 81.68s/it]
 40%|████      | 255/630 [4:41:16<8:30:48, 81.73s/it]
 41%|████      | 256/630 [4:42:38<8:29:35, 81.75s/it]
 41%|████      | 257/630 [4:43:59<8:28:23, 81.78s/it]
 41%|████      | 258/630 [4:45:22<8:27:27, 81.85s/it]
 41%|████      | 259/630 [4:46:39<8:18:52, 80.68s/it]
 41%|████▏     | 260/630 [4:48:01<8:19:41, 81.03s/it]
                                                     
{'loss': 0.165, 'learning_rate': 0.0001, 'global_step': 260, 'epoch': 1.24}

 41%|████▏     | 260/630 [4:48:01<8:19:41, 81.03s/it]
 41%|████▏     | 261/630 [4:49:23<8:19:20, 81.19s/it]
 42%|████▏     | 262/630 [4:50:45<8:19:17, 81.41s/it]
 42%|████▏     | 263/630 [4:52:06<8:18:12, 81.45s/it]
 42%|████▏     | 264/630 [4:53:28<8:17:31, 81.56s/it]
 42%|████▏     | 265/630 [4:54:50<8:16:38, 81.64s/it]
 42%|████▏     | 266/630 [4:56:13<8:18:35, 82.19s/it]
 42%|████▏     | 267/630 [4:57:35<8:16:34, 82.08s/it]
 43%|████▎     | 268/630 [4:58:57<8:14:08, 81.90s/it]
 43%|████▎     | 269/630 [5:00:18<8:12:13, 81.81s/it]
 43%|████▎     | 270/630 [5:01:39<8:09:16, 81.54s/it]
 43%|████▎     | 271/630 [5:03:01<8:08:22, 81.62s/it]
 43%|████▎     | 272/630 [5:04:23<8:07:03, 81.63s/it]
 43%|████▎     | 273/630 [5:05:45<8:06:07, 81.70s/it]
 43%|████▎     | 274/630 [5:07:06<8:04:41, 81.69s/it]
 44%|████▎     | 275/630 [5:08:27<8:01:07, 81.32s/it]
 44%|████▍     | 276/630 [5:09:46<7:56:14, 80.72s/it]
 44%|████▍     | 277/630 [5:11:08<7:56:51, 81.05s/it]
 44%|████▍     | 278/630 [5:12:30<7:57:01, 81.31s/it]
 44%|████▍     | 279/630 [5:13:52<7:57:01, 81.54s/it]
 44%|████▍     | 280/630 [5:15:14<7:56:17, 81.65s/it]
                                                     
{'loss': 0.2124, 'learning_rate': 0.0001, 'global_step': 280, 'epoch': 1.33}

 44%|████▍     | 280/630 [5:15:14<7:56:17, 81.65s/it]
 45%|████▍     | 281/630 [5:16:36<7:55:59, 81.83s/it]
 45%|████▍     | 282/630 [5:17:58<7:55:22, 81.96s/it]
 45%|████▍     | 283/630 [5:19:20<7:53:35, 81.89s/it]
 45%|████▌     | 284/630 [5:20:42<7:51:51, 81.83s/it]
 45%|████▌     | 285/630 [5:22:02<7:47:33, 81.32s/it]
 45%|████▌     | 286/630 [5:23:23<7:46:44, 81.41s/it]
 46%|████▌     | 287/630 [5:24:45<7:44:49, 81.31s/it]
 46%|████▌     | 288/630 [5:26:06<7:44:19, 81.46s/it]
 46%|████▌     | 289/630 [5:27:28<7:43:58, 81.64s/it]
 46%|████▌     | 290/630 [5:28:51<7:43:42, 81.83s/it]
 46%|████▌     | 291/630 [5:30:12<7:41:24, 81.67s/it]
 46%|████▋     | 292/630 [5:31:31<7:36:26, 81.03s/it]
 47%|████▋     | 293/630 [5:32:53<7:36:25, 81.26s/it]
 47%|████▋     | 294/630 [5:34:15<7:35:59, 81.43s/it]
 47%|████▋     | 295/630 [5:35:37<7:34:57, 81.49s/it]
 47%|████▋     | 296/630 [5:36:58<7:33:30, 81.47s/it]
 47%|████▋     | 297/630 [5:38:17<7:27:50, 80.69s/it]
 47%|████▋     | 298/630 [5:39:39<7:28:35, 81.07s/it]
 47%|████▋     | 299/630 [5:41:01<7:28:52, 81.37s/it]
 48%|████▊     | 300/630 [5:42:23<7:28:02, 81.46s/it]
                                                     
{'loss': 0.2007, 'learning_rate': 0.0001, 'global_step': 300, 'epoch': 1.43}

 48%|████▊     | 300/630 [5:42:23<7:28:02, 81.46s/it]
 48%|████▊     | 301/630 [5:43:45<7:27:55, 81.69s/it]
 48%|████▊     | 302/630 [5:45:07<7:26:49, 81.74s/it]
 48%|████▊     | 303/630 [5:46:28<7:25:17, 81.70s/it]
 48%|████▊     | 304/630 [5:47:50<7:24:14, 81.76s/it]
 48%|████▊     | 305/630 [5:49:12<7:22:50, 81.76s/it]
 49%|████▊     | 306/630 [5:50:34<7:21:27, 81.75s/it]
 49%|████▊     | 307/630 [5:51:56<7:20:17, 81.79s/it]
 49%|████▉     | 308/630 [5:53:17<7:18:57, 81.79s/it]
 49%|████▉     | 309/630 [5:54:38<7:16:09, 81.52s/it]
 49%|████▉     | 310/630 [5:56:00<7:14:37, 81.49s/it]
 49%|████▉     | 311/630 [5:57:21<7:13:29, 81.53s/it]
 50%|████▉     | 312/630 [5:58:43<7:12:19, 81.57s/it]
 50%|████▉     | 313/630 [6:00:05<7:11:55, 81.75s/it]
 50%|████▉     | 314/630 [6:01:27<7:11:16, 81.89s/it]
 50%|█████     | 315/630 [6:02:49<7:09:32, 81.82s/it]
 50%|█████     | 316/630 [6:04:11<7:08:31, 81.88s/it]
 50%|█████     | 317/630 [6:05:33<7:06:45, 81.81s/it]
 50%|█████     | 318/630 [6:06:54<7:05:08, 81.76s/it]
 51%|█████     | 319/630 [6:08:16<7:03:36, 81.73s/it]
 51%|█████     | 320/630 [6:09:38<7:01:58, 81.67s/it]
                                                     
{'loss': 0.1981, 'learning_rate': 0.0001, 'global_step': 320, 'epoch': 1.52}

 51%|█████     | 320/630 [6:09:38<7:01:58, 81.67s/it]
 51%|█████     | 321/630 [6:10:59<7:00:51, 81.72s/it]
 51%|█████     | 322/630 [6:12:21<6:59:21, 81.69s/it]
 51%|█████▏    | 323/630 [6:13:43<6:58:36, 81.81s/it]
 51%|█████▏    | 324/630 [6:15:05<6:57:26, 81.85s/it]
 52%|█████▏    | 325/630 [6:16:27<6:55:46, 81.79s/it]
 52%|█████▏    | 326/630 [6:17:49<6:54:38, 81.84s/it]
 52%|█████▏    | 327/630 [6:19:11<6:53:37, 81.90s/it]
 52%|█████▏    | 328/630 [6:20:33<6:52:13, 81.90s/it]
 52%|█████▏    | 329/630 [6:21:54<6:50:23, 81.81s/it]
 52%|█████▏    | 330/630 [6:23:16<6:48:46, 81.76s/it]
 53%|█████▎    | 331/630 [6:24:38<6:47:23, 81.75s/it]
 53%|█████▎    | 332/630 [6:25:59<6:45:55, 81.73s/it]
 53%|█████▎    | 333/630 [6:27:21<6:44:47, 81.78s/it]
 53%|█████▎    | 334/630 [6:28:43<6:43:11, 81.73s/it]
 53%|█████▎    | 335/630 [6:30:05<6:42:05, 81.78s/it]
 53%|█████▎    | 336/630 [6:31:27<6:41:31, 81.94s/it]
 53%|█████▎    | 337/630 [6:32:49<6:39:46, 81.87s/it]
 54%|█████▎    | 338/630 [6:34:11<6:38:28, 81.88s/it]
 54%|█████▍    | 339/630 [6:35:32<6:36:52, 81.83s/it]
 54%|█████▍    | 340/630 [6:36:54<6:34:47, 81.68s/it]
                                                     
{'loss': 0.2031, 'learning_rate': 0.0001, 'global_step': 340, 'epoch': 1.62}

 54%|█████▍    | 340/630 [6:36:54<6:34:47, 81.68s/it]
 54%|█████▍    | 341/630 [6:38:15<6:33:27, 81.69s/it]
 54%|█████▍    | 342/630 [6:39:37<6:32:42, 81.81s/it]
 54%|█████▍    | 343/630 [6:41:00<6:32:04, 81.97s/it]
 55%|█████▍    | 344/630 [6:42:22<6:30:47, 81.98s/it]
 55%|█████▍    | 345/630 [6:43:42<6:26:53, 81.45s/it]
 55%|█████▍    | 346/630 [6:45:04<6:26:26, 81.64s/it]
 55%|█████▌    | 347/630 [6:46:26<6:25:24, 81.71s/it]
 55%|█████▌    | 348/630 [6:47:48<6:24:39, 81.84s/it]
 55%|█████▌    | 349/630 [6:49:10<6:23:17, 81.84s/it]
 56%|█████▌    | 350/630 [6:50:32<6:21:58, 81.85s/it]
 56%|█████▌    | 351/630 [6:51:54<6:20:38, 81.86s/it]
 56%|█████▌    | 352/630 [6:53:16<6:19:57, 82.01s/it]
 56%|█████▌    | 353/630 [6:54:38<6:18:02, 81.89s/it]
 56%|█████▌    | 354/630 [6:55:59<6:16:28, 81.84s/it]
 56%|█████▋    | 355/630 [6:57:21<6:15:09, 81.85s/it]
 57%|█████▋    | 356/630 [6:58:43<6:13:49, 81.86s/it]
 57%|█████▋    | 357/630 [7:00:05<6:12:12, 81.81s/it]
 57%|█████▋    | 358/630 [7:01:27<6:11:10, 81.88s/it]
 57%|█████▋    | 359/630 [7:02:49<6:10:06, 81.94s/it]
 57%|█████▋    | 360/630 [7:04:11<6:08:23, 81.87s/it]
                                                     
{'loss': 0.1392, 'learning_rate': 0.0001, 'global_step': 360, 'epoch': 1.71}

 57%|█████▋    | 360/630 [7:04:11<6:08:23, 81.87s/it]
 57%|█████▋    | 361/630 [7:05:32<6:06:45, 81.80s/it]
 57%|█████▋    | 362/630 [7:06:54<6:05:23, 81.80s/it]
 58%|█████▊    | 363/630 [7:08:16<6:04:07, 81.82s/it]
 58%|█████▊    | 364/630 [7:09:38<6:02:54, 81.86s/it]
 58%|█████▊    | 365/630 [7:11:00<6:01:48, 81.92s/it]
 58%|█████▊    | 366/630 [7:12:22<6:00:41, 81.98s/it]
 58%|█████▊    | 367/630 [7:13:44<5:59:12, 81.95s/it]
 58%|█████▊    | 368/630 [7:15:06<5:57:41, 81.92s/it]
 59%|█████▊    | 369/630 [7:16:28<5:56:16, 81.90s/it]
 59%|█████▊    | 370/630 [7:17:49<5:54:34, 81.83s/it]
 59%|█████▉    | 371/630 [7:19:11<5:53:00, 81.78s/it]
 59%|█████▉    | 372/630 [7:20:33<5:51:44, 81.80s/it]
 59%|█████▉    | 373/630 [7:21:55<5:50:29, 81.83s/it]
 59%|█████▉    | 374/630 [7:23:17<5:49:10, 81.84s/it]
 60%|█████▉    | 375/630 [7:24:31<5:38:05, 79.55s/it]
 60%|█████▉    | 376/630 [7:25:53<5:39:45, 80.26s/it]
 60%|█████▉    | 377/630 [7:27:12<5:37:05, 79.94s/it]
 60%|██████    | 378/630 [7:28:34<5:38:08, 80.51s/it]
 60%|██████    | 379/630 [7:29:56<5:38:56, 81.02s/it]
 60%|██████    | 380/630 [7:31:18<5:38:18, 81.19s/it]
                                                     
{'loss': 0.204, 'learning_rate': 0.0001, 'global_step': 380, 'epoch': 1.81}

 60%|██████    | 380/630 [7:31:18<5:38:18, 81.19s/it]
 60%|██████    | 381/630 [7:32:40<5:38:01, 81.45s/it]
 61%|██████    | 382/630 [7:34:02<5:37:44, 81.71s/it]
 61%|██████    | 383/630 [7:35:24<5:36:36, 81.77s/it]
 61%|██████    | 384/630 [7:36:46<5:35:27, 81.82s/it]
 61%|██████    | 385/630 [7:38:08<5:34:45, 81.98s/it]
 61%|██████▏   | 386/630 [7:39:30<5:33:19, 81.96s/it]
 61%|██████▏   | 387/630 [7:40:52<5:31:52, 81.95s/it]
 62%|██████▏   | 388/630 [7:42:14<5:30:28, 81.94s/it]
 62%|██████▏   | 389/630 [7:43:36<5:29:04, 81.93s/it]
 62%|██████▏   | 390/630 [7:44:54<5:23:01, 80.76s/it]
 62%|██████▏   | 391/630 [7:46:16<5:23:12, 81.14s/it]
 62%|██████▏   | 392/630 [7:47:37<5:21:51, 81.14s/it]
 62%|██████▏   | 393/630 [7:48:59<5:21:04, 81.29s/it]
 63%|██████▎   | 394/630 [7:50:19<5:18:31, 80.98s/it]
 63%|██████▎   | 395/630 [7:51:41<5:18:13, 81.25s/it]
 63%|██████▎   | 396/630 [7:53:02<5:17:20, 81.37s/it]
 63%|██████▎   | 397/630 [7:54:24<5:16:31, 81.51s/it]
 63%|██████▎   | 398/630 [7:55:46<5:15:19, 81.55s/it]
 63%|██████▎   | 399/630 [7:57:08<5:14:20, 81.65s/it]
 63%|██████▎   | 400/630 [7:58:30<5:13:11, 81.70s/it]
                                                     
{'loss': 0.1626, 'learning_rate': 0.0001, 'global_step': 400, 'epoch': 1.9}

 63%|██████▎   | 400/630 [7:58:30<5:13:11, 81.70s/it]
 64%|██████▎   | 401/630 [7:59:51<5:11:58, 81.74s/it]
 64%|██████▍   | 402/630 [8:01:14<5:10:56, 81.83s/it]
 64%|██████▍   | 403/630 [8:02:35<5:09:31, 81.81s/it]
 64%|██████▍   | 404/630 [8:03:57<5:07:46, 81.71s/it]
 64%|██████▍   | 405/630 [8:05:18<5:06:18, 81.68s/it]
 64%|██████▍   | 406/630 [8:06:40<5:05:04, 81.71s/it]
 65%|██████▍   | 407/630 [8:07:57<4:58:42, 80.37s/it]
 65%|██████▍   | 408/630 [8:09:19<4:59:03, 80.83s/it]
 65%|██████▍   | 409/630 [8:10:41<4:58:51, 81.14s/it]
 65%|██████▌   | 410/630 [8:12:03<4:58:13, 81.33s/it]
 65%|██████▌   | 411/630 [8:13:24<4:56:51, 81.33s/it]
 65%|██████▌   | 412/630 [8:14:46<4:56:14, 81.54s/it]
 66%|██████▌   | 413/630 [8:16:09<4:55:43, 81.77s/it]
 66%|██████▌   | 414/630 [8:17:30<4:54:21, 81.77s/it]
 66%|██████▌   | 415/630 [8:18:52<4:53:06, 81.80s/it]
 66%|██████▌   | 416/630 [8:20:14<4:52:10, 81.92s/it]
 66%|██████▌   | 417/630 [8:21:37<4:51:12, 82.03s/it]
 66%|██████▋   | 418/630 [8:22:58<4:49:25, 81.91s/it]
 67%|██████▋   | 419/630 [8:24:20<4:48:00, 81.90s/it]
 67%|██████▋   | 420/630 [8:25:42<4:46:26, 81.84s/it]
                                                     
{'loss': 0.146, 'learning_rate': 0.0001, 'global_step': 420, 'epoch': 2.0}

 67%|██████▋   | 420/630 [8:25:42<4:46:26, 81.84s/it]
 67%|██████▋   | 421/630 [8:26:59<4:40:28, 80.52s/it]
 67%|██████▋   | 422/630 [8:28:21<4:40:41, 80.97s/it]
 67%|██████▋   | 423/630 [8:29:43<4:40:24, 81.28s/it]
 67%|██████▋   | 424/630 [8:31:05<4:39:34, 81.43s/it]
 67%|██████▋   | 425/630 [8:32:27<4:38:49, 81.61s/it]
 68%|██████▊   | 426/630 [8:33:49<4:37:52, 81.73s/it]
 68%|██████▊   | 427/630 [8:35:11<4:36:34, 81.75s/it]
 68%|██████▊   | 428/630 [8:36:33<4:35:29, 81.83s/it]
 68%|██████▊   | 429/630 [8:37:55<4:34:05, 81.82s/it]
 68%|██████▊   | 430/630 [8:39:17<4:32:42, 81.81s/it]
 68%|██████▊   | 431/630 [8:40:38<4:31:22, 81.82s/it]
 69%|██████▊   | 432/630 [8:42:00<4:30:00, 81.82s/it]
 69%|██████▊   | 433/630 [8:43:22<4:28:39, 81.82s/it]
 69%|██████▉   | 434/630 [8:44:44<4:27:40, 81.94s/it]
 69%|██████▉   | 435/630 [8:46:07<4:26:34, 82.02s/it]
 69%|██████▉   | 436/630 [8:47:28<4:25:00, 81.96s/it]
 69%|██████▉   | 437/630 [8:48:50<4:23:14, 81.84s/it]
 70%|██████▉   | 438/630 [8:50:11<4:21:36, 81.75s/it]
 70%|██████▉   | 439/630 [8:51:33<4:20:04, 81.70s/it]
 70%|██████▉   | 440/630 [8:52:55<4:18:33, 81.65s/it]
                                                     
{'loss': 0.0404, 'learning_rate': 0.0001, 'global_step': 440, 'epoch': 2.09}

 70%|██████▉   | 440/630 [8:52:55<4:18:33, 81.65s/it]
 70%|███████   | 441/630 [8:54:15<4:15:47, 81.20s/it]
 70%|███████   | 442/630 [8:55:36<4:14:57, 81.37s/it]
 70%|███████   | 443/630 [8:56:58<4:13:57, 81.48s/it]
 70%|███████   | 444/630 [8:58:20<4:12:36, 81.49s/it]
 71%|███████   | 445/630 [8:59:41<4:10:46, 81.33s/it]
 71%|███████   | 446/630 [9:01:02<4:09:37, 81.40s/it]
 71%|███████   | 447/630 [9:02:24<4:08:20, 81.42s/it]
 71%|███████   | 448/630 [9:03:45<4:07:05, 81.46s/it]
 71%|███████▏  | 449/630 [9:05:07<4:05:57, 81.54s/it]
 71%|███████▏  | 450/630 [9:06:28<4:04:32, 81.51s/it]
 72%|███████▏  | 451/630 [9:07:49<4:02:25, 81.26s/it]
 72%|███████▏  | 452/630 [9:09:11<4:01:28, 81.40s/it]
 72%|███████▏  | 453/630 [9:10:33<4:00:24, 81.50s/it]
 72%|███████▏  | 454/630 [9:11:54<3:59:00, 81.48s/it]
 72%|███████▏  | 455/630 [9:13:16<3:57:44, 81.51s/it]
 72%|███████▏  | 456/630 [9:14:37<3:56:00, 81.38s/it]
 73%|███████▎  | 457/630 [9:15:58<3:55:00, 81.50s/it]
 73%|███████▎  | 458/630 [9:17:21<3:54:14, 81.71s/it]
 73%|███████▎  | 459/630 [9:18:42<3:52:54, 81.72s/it]
 73%|███████▎  | 460/630 [9:20:04<3:51:25, 81.68s/it]
                                                     
{'loss': 0.0304, 'learning_rate': 0.0001, 'global_step': 460, 'epoch': 2.19}

 73%|███████▎  | 460/630 [9:20:04<3:51:25, 81.68s/it]
 73%|███████▎  | 461/630 [9:21:26<3:50:00, 81.66s/it]
 73%|███████▎  | 462/630 [9:22:47<3:48:33, 81.63s/it]
 73%|███████▎  | 463/630 [9:24:09<3:47:08, 81.61s/it]
 74%|███████▎  | 464/630 [9:25:30<3:45:46, 81.61s/it]
 74%|███████▍  | 465/630 [9:26:52<3:44:42, 81.71s/it]
 74%|███████▍  | 466/630 [9:28:14<3:43:21, 81.72s/it]
 74%|███████▍  | 467/630 [9:29:36<3:42:12, 81.79s/it]
 74%|███████▍  | 468/630 [9:30:57<3:40:21, 81.62s/it]
 74%|███████▍  | 469/630 [9:32:19<3:39:05, 81.65s/it]
 75%|███████▍  | 470/630 [9:33:41<3:37:47, 81.67s/it]
 75%|███████▍  | 471/630 [9:35:02<3:36:35, 81.73s/it]
 75%|███████▍  | 472/630 [9:36:24<3:34:59, 81.64s/it]
 75%|███████▌  | 473/630 [9:37:45<3:33:10, 81.47s/it]
 75%|███████▌  | 474/630 [9:39:07<3:32:01, 81.55s/it]
 75%|███████▌  | 475/630 [9:40:28<3:30:50, 81.62s/it]
 76%|███████▌  | 476/630 [9:41:50<3:29:33, 81.65s/it]
 76%|███████▌  | 477/630 [9:43:12<3:28:33, 81.79s/it]
 76%|███████▌  | 478/630 [9:44:34<3:27:07, 81.76s/it]
 76%|███████▌  | 479/630 [9:45:56<3:25:39, 81.72s/it]
 76%|███████▌  | 480/630 [9:47:15<3:22:28, 80.99s/it]
                                                     
{'loss': 0.0166, 'learning_rate': 0.0001, 'global_step': 480, 'epoch': 2.28}

 76%|███████▌  | 480/630 [9:47:15<3:22:28, 80.99s/it]
 76%|███████▋  | 481/630 [9:48:36<3:21:27, 81.13s/it]
 77%|███████▋  | 482/630 [9:49:58<3:20:18, 81.21s/it]
 77%|███████▋  | 483/630 [9:51:19<3:19:19, 81.36s/it]
 77%|███████▋  | 484/630 [9:52:41<3:18:21, 81.52s/it]
 77%|███████▋  | 485/630 [9:54:03<3:17:05, 81.56s/it]
 77%|███████▋  | 486/630 [9:55:24<3:15:38, 81.52s/it]
 77%|███████▋  | 487/630 [9:56:46<3:14:14, 81.50s/it]
 77%|███████▋  | 488/630 [9:58:07<3:12:50, 81.48s/it]
 78%|███████▊  | 489/630 [9:59:29<3:11:25, 81.46s/it]
 78%|███████▊  | 490/630 [10:00:50<3:09:45, 81.33s/it]
 78%|███████▊  | 491/630 [10:02:11<3:08:37, 81.42s/it]
 78%|███████▊  | 492/630 [10:03:31<3:06:00, 80.87s/it]
 78%|███████▊  | 493/630 [10:04:53<3:05:21, 81.18s/it]
 78%|███████▊  | 494/630 [10:06:15<3:04:28, 81.38s/it]
 79%|███████▊  | 495/630 [10:07:37<3:03:27, 81.54s/it]
 79%|███████▊  | 496/630 [10:08:59<3:02:22, 81.66s/it]
 79%|███████▉  | 497/630 [10:10:20<3:01:11, 81.74s/it]
 79%|███████▉  | 498/630 [10:11:42<2:59:46, 81.72s/it]
 79%|███████▉  | 499/630 [10:13:04<2:58:25, 81.72s/it]
 79%|███████▉  | 500/630 [10:14:25<2:56:56, 81.66s/it]
                                                      
{'loss': 0.0366, 'learning_rate': 0.0001, 'global_step': 500, 'epoch': 2.38}

 79%|███████▉  | 500/630 [10:14:25<2:56:56, 81.66s/it]
 80%|███████▉  | 501/630 [10:15:47<2:55:31, 81.64s/it]
 80%|███████▉  | 502/630 [10:17:09<2:54:05, 81.60s/it]
 80%|███████▉  | 503/630 [10:18:30<2:52:40, 81.58s/it]
 80%|████████  | 504/630 [10:19:52<2:51:20, 81.59s/it]
 80%|████████  | 505/630 [10:21:13<2:50:06, 81.65s/it]
 80%|████████  | 506/630 [10:22:35<2:48:50, 81.70s/it]
 80%|████████  | 507/630 [10:23:57<2:47:38, 81.77s/it]
 81%|████████  | 508/630 [10:25:19<2:46:06, 81.69s/it]
 81%|████████  | 509/630 [10:26:41<2:45:01, 81.83s/it]
 81%|████████  | 510/630 [10:28:03<2:43:37, 81.81s/it]
 81%|████████  | 511/630 [10:29:24<2:42:05, 81.72s/it]
 81%|████████▏ | 512/630 [10:30:46<2:40:38, 81.68s/it]
 81%|████████▏ | 513/630 [10:32:07<2:39:11, 81.64s/it]
 82%|████████▏ | 514/630 [10:33:29<2:37:49, 81.63s/it]
 82%|████████▏ | 515/630 [10:34:50<2:36:23, 81.60s/it]
 82%|████████▏ | 516/630 [10:36:12<2:35:00, 81.59s/it]
 82%|████████▏ | 517/630 [10:37:34<2:33:43, 81.63s/it]
 82%|████████▏ | 518/630 [10:38:55<2:32:26, 81.66s/it]
 82%|████████▏ | 519/630 [10:40:17<2:30:59, 81.62s/it]
 83%|████████▎ | 520/630 [10:41:38<2:29:08, 81.35s/it]
                                                      
{'loss': 0.018, 'learning_rate': 0.0001, 'global_step': 520, 'epoch': 2.47}

 83%|████████▎ | 520/630 [10:41:38<2:29:08, 81.35s/it]
 83%|████████▎ | 521/630 [10:42:59<2:28:00, 81.47s/it]
 83%|████████▎ | 522/630 [10:44:21<2:26:54, 81.61s/it]
 83%|████████▎ | 523/630 [10:45:43<2:25:34, 81.63s/it]
 83%|████████▎ | 524/630 [10:47:05<2:24:08, 81.59s/it]
 83%|████████▎ | 525/630 [10:48:26<2:22:46, 81.58s/it]
 83%|████████▎ | 526/630 [10:49:48<2:21:22, 81.56s/it]
 84%|████████▎ | 527/630 [10:51:09<2:19:59, 81.55s/it]
 84%|████████▍ | 528/630 [10:52:30<2:18:23, 81.40s/it]
 84%|████████▍ | 529/630 [10:53:52<2:17:16, 81.55s/it]
 84%|████████▍ | 530/630 [10:55:14<2:15:53, 81.53s/it]
 84%|████████▍ | 531/630 [10:56:35<2:14:33, 81.55s/it]
 84%|████████▍ | 532/630 [10:57:57<2:13:09, 81.52s/it]
 85%|████████▍ | 533/630 [10:59:18<2:11:45, 81.50s/it]
 85%|████████▍ | 534/630 [11:00:40<2:10:41, 81.68s/it]
 85%|████████▍ | 535/630 [11:02:02<2:09:35, 81.85s/it]
 85%|████████▌ | 536/630 [11:03:18<2:05:08, 79.88s/it]
 85%|████████▌ | 537/630 [11:04:40<2:04:51, 80.56s/it]
 85%|████████▌ | 538/630 [11:06:01<2:04:01, 80.88s/it]
 86%|████████▌ | 539/630 [11:07:24<2:03:12, 81.24s/it]
 86%|████████▌ | 540/630 [11:08:46<2:02:17, 81.53s/it]
                                                      
{'loss': 0.0186, 'learning_rate': 0.0001, 'global_step': 540, 'epoch': 2.57}

 86%|████████▌ | 540/630 [11:08:46<2:02:17, 81.53s/it]
 86%|████████▌ | 541/630 [11:10:08<2:01:09, 81.68s/it]
 86%|████████▌ | 542/630 [11:11:30<1:59:54, 81.76s/it]
 86%|████████▌ | 543/630 [11:12:52<1:58:38, 81.83s/it]
 86%|████████▋ | 544/630 [11:14:14<1:57:25, 81.93s/it]
 87%|████████▋ | 545/630 [11:15:36<1:56:07, 81.98s/it]
 87%|████████▋ | 546/630 [11:16:58<1:54:41, 81.92s/it]
 87%|████████▋ | 547/630 [11:18:19<1:53:13, 81.85s/it]
 87%|████████▋ | 548/630 [11:19:41<1:51:49, 81.82s/it]
 87%|████████▋ | 549/630 [11:21:02<1:50:07, 81.57s/it]
 87%|████████▋ | 550/630 [11:22:23<1:48:36, 81.46s/it]
 87%|████████▋ | 551/630 [11:23:45<1:47:22, 81.55s/it]
 88%|████████▊ | 552/630 [11:25:07<1:46:09, 81.65s/it]
 88%|████████▊ | 553/630 [11:26:23<1:42:39, 79.99s/it]
 88%|████████▊ | 554/630 [11:27:45<1:42:02, 80.56s/it]
 88%|████████▊ | 555/630 [11:29:07<1:41:06, 80.89s/it]
 88%|████████▊ | 556/630 [11:30:25<1:38:54, 80.20s/it]
 88%|████████▊ | 557/630 [11:31:47<1:38:06, 80.64s/it]
 89%|████████▊ | 558/630 [11:33:09<1:37:14, 81.03s/it]
 89%|████████▊ | 559/630 [11:34:31<1:36:06, 81.23s/it]
 89%|████████▉ | 560/630 [11:35:52<1:34:59, 81.42s/it]
                                                      
{'loss': 0.0308, 'learning_rate': 0.0001, 'global_step': 560, 'epoch': 2.66}

 89%|████████▉ | 560/630 [11:35:52<1:34:59, 81.42s/it]
 89%|████████▉ | 561/630 [11:37:14<1:33:44, 81.51s/it]
 89%|████████▉ | 562/630 [11:38:36<1:32:26, 81.56s/it]
 89%|████████▉ | 563/630 [11:39:58<1:31:18, 81.77s/it]
 90%|████████▉ | 564/630 [11:41:19<1:29:41, 81.54s/it]
 90%|████████▉ | 565/630 [11:42:41<1:28:19, 81.53s/it]
 90%|████████▉ | 566/630 [11:44:02<1:26:57, 81.52s/it]
 90%|█████████ | 567/630 [11:45:24<1:25:48, 81.72s/it]
 90%|█████████ | 568/630 [11:46:46<1:24:22, 81.66s/it]
 90%|█████████ | 569/630 [11:48:08<1:23:10, 81.81s/it]
 90%|█████████ | 570/630 [11:49:30<1:21:47, 81.79s/it]
 91%|█████████ | 571/630 [11:50:51<1:20:24, 81.76s/it]
 91%|█████████ | 572/630 [11:52:13<1:19:06, 81.84s/it]
 91%|█████████ | 573/630 [11:53:35<1:17:45, 81.84s/it]
 91%|█████████ | 574/630 [11:54:57<1:16:18, 81.76s/it]
 91%|█████████▏| 575/630 [11:56:18<1:14:51, 81.66s/it]
 91%|█████████▏| 576/630 [11:57:40<1:13:31, 81.69s/it]
 92%|█████████▏| 577/630 [11:59:02<1:12:15, 81.79s/it]
 92%|█████████▏| 578/630 [12:00:22<1:10:25, 81.26s/it]
 92%|█████████▏| 579/630 [12:01:44<1:09:11, 81.40s/it]
 92%|█████████▏| 580/630 [12:03:05<1:07:51, 81.43s/it]
                                                      
{'loss': 0.0536, 'learning_rate': 0.0001, 'global_step': 580, 'epoch': 2.76}

 92%|█████████▏| 580/630 [12:03:05<1:07:51, 81.43s/it]
 92%|█████████▏| 581/630 [12:04:27<1:06:33, 81.50s/it]
 92%|█████████▏| 582/630 [12:05:49<1:05:18, 81.64s/it]
 93%|█████████▎| 583/630 [12:07:11<1:03:58, 81.66s/it]
 93%|█████████▎| 584/630 [12:08:33<1:02:40, 81.74s/it]
 93%|█████████▎| 585/630 [12:09:55<1:01:21, 81.82s/it]
 93%|█████████▎| 586/630 [12:11:16<59:58, 81.79s/it]  
 93%|█████████▎| 587/630 [12:12:38<58:36, 81.77s/it]
 93%|█████████▎| 588/630 [12:14:00<57:11, 81.71s/it]
 93%|█████████▎| 589/630 [12:15:21<55:50, 81.73s/it]
 94%|█████████▎| 590/630 [12:16:43<54:31, 81.80s/it]
 94%|█████████▍| 591/630 [12:18:05<53:08, 81.76s/it]
 94%|█████████▍| 592/630 [12:19:27<51:46, 81.76s/it]
 94%|█████████▍| 593/630 [12:20:49<50:26, 81.79s/it]
 94%|█████████▍| 594/630 [12:22:11<49:05, 81.83s/it]
 94%|█████████▍| 595/630 [12:23:33<47:46, 81.91s/it]
 95%|█████████▍| 596/630 [12:24:55<46:25, 81.92s/it]
 95%|█████████▍| 597/630 [12:26:16<45:01, 81.87s/it]
 95%|█████████▍| 598/630 [12:27:38<43:40, 81.88s/it]
 95%|█████████▌| 599/630 [12:29:00<42:21, 81.97s/it]
 95%|█████████▌| 600/630 [12:30:22<40:58, 81.97s/it]
                                                    
{'loss': 0.0563, 'learning_rate': 0.0001, 'global_step': 600, 'epoch': 2.85}

 95%|█████████▌| 600/630 [12:30:22<40:58, 81.97s/it]
 95%|█████████▌| 601/630 [12:31:44<39:33, 81.84s/it]
 96%|█████████▌| 602/630 [12:33:06<38:11, 81.83s/it]
 96%|█████████▌| 603/630 [12:34:26<36:32, 81.21s/it]
 96%|█████████▌| 604/630 [12:35:47<35:16, 81.39s/it]
 96%|█████████▌| 605/630 [12:37:09<33:57, 81.50s/it]
 96%|█████████▌| 606/630 [12:38:31<32:38, 81.59s/it]
 96%|█████████▋| 607/630 [12:39:52<31:16, 81.57s/it]
 97%|█████████▋| 608/630 [12:41:14<29:56, 81.64s/it]
 97%|█████████▋| 609/630 [12:42:36<28:35, 81.68s/it]
 97%|█████████▋| 610/630 [12:43:58<27:14, 81.75s/it]
 97%|█████████▋| 611/630 [12:45:20<25:53, 81.75s/it]
 97%|█████████▋| 612/630 [12:46:41<24:31, 81.76s/it]
 97%|█████████▋| 613/630 [12:48:03<23:10, 81.77s/it]
 97%|█████████▋| 614/630 [12:49:25<21:48, 81.78s/it]
 98%|█████████▊| 615/630 [12:50:47<20:25, 81.72s/it]
 98%|█████████▊| 616/630 [12:52:08<19:04, 81.74s/it]
 98%|█████████▊| 617/630 [12:53:30<17:42, 81.71s/it]
 98%|█████████▊| 618/630 [12:54:52<16:21, 81.81s/it]
 98%|█████████▊| 619/630 [12:56:14<14:59, 81.77s/it]
 98%|█████████▊| 620/630 [12:57:35<13:36, 81.68s/it]
                                                    
{'loss': 0.0242, 'learning_rate': 0.0001, 'global_step': 620, 'epoch': 2.95}

 98%|█████████▊| 620/630 [12:57:35<13:36, 81.68s/it]
 99%|█████████▊| 621/630 [12:58:57<12:16, 81.80s/it]
 99%|█████████▊| 622/630 [13:00:19<10:53, 81.71s/it]
 99%|█████████▉| 623/630 [13:01:41<09:31, 81.71s/it]
 99%|█████████▉| 624/630 [13:03:02<08:09, 81.66s/it]
 99%|█████████▉| 625/630 [13:04:21<06:44, 80.87s/it]
 99%|█████████▉| 626/630 [13:05:43<05:24, 81.09s/it]
100%|█████████▉| 627/630 [13:07:04<04:03, 81.23s/it]
100%|█████████▉| 628/630 [13:08:26<02:42, 81.33s/it]
100%|█████████▉| 629/630 [13:09:48<01:21, 81.48s/it]
100%|██████████| 630/630 [13:11:09<00:00, 81.50s/it]
                                                    
{'train_runtime': 47469.8541, 'train_samples_per_second': 0.159, 'train_steps_per_second': 0.013, 'train_loss': 0.24650317042592973, 'epoch': 3.0}

100%|██████████| 630/630 [13:11:09<00:00, 81.50s/it]
100%|██████████| 630/630 [13:11:09<00:00, 75.35s/it]
***** train metrics *****
  epoch                    =         3.0
  train_loss               =      0.2465
  train_runtime            = 13:11:09.85
  train_samples_per_second =       0.159
  train_steps_per_second   =       0.013