| ๅ ่ฝฝ่ฎญ็ป้: /tmp/home/wzh/file/train_data.csv | |
| ๅ ่ฝฝ้ช่ฏ้: /tmp/home/wzh/file/val_data.csv | |
| ๆญฃๅจ่ฎก็ฎ็ฑปๅซๆ้... | |
| ่ฎก็ฎๅบ็็ฑปๅซๆ้: [0.58033253 3.61206429] | |
| ๆญฃๅจไธ่ฝฝ/ๅ ่ฝฝๆจกๅ: roberta-base ... | |
| Map: 0%| | 0/40000 [00:00<?, ? examples/s] Map: 2%|โ | 1000/40000 [00:00<00:09, 4176.70 examples/s] Map: 5%|โ | 2000/40000 [00:00<00:09, 4220.60 examples/s] Map: 8%|โ | 3000/40000 [00:00<00:08, 4261.20 examples/s] Map: 10%|โ | 4000/40000 [00:01<00:11, 3037.93 examples/s] Map: 12%|โโ | 5000/40000 [00:01<00:10, 3398.31 examples/s] Map: 15%|โโ | 6000/40000 [00:01<00:09, 3576.90 examples/s] Map: 18%|โโ | 7000/40000 [00:01<00:08, 3772.83 examples/s] Map: 20%|โโ | 8000/40000 [00:02<00:08, 3972.11 examples/s] Map: 22%|โโโ | 9000/40000 [00:02<00:07, 4082.89 examples/s] Map: 25%|โโโ | 10000/40000 [00:02<00:07, 4142.79 examples/s] Map: 28%|โโโ | 11000/40000 [00:02<00:06, 4148.22 examples/s] Map: 30%|โโโ | 12000/40000 [00:03<00:06, 4157.05 examples/s] Map: 32%|โโโโ | 13000/40000 [00:03<00:06, 4189.95 examples/s] Map: 35%|โโโโ | 14000/40000 [00:03<00:06, 4262.87 examples/s] Map: 38%|โโโโ | 15000/40000 [00:03<00:05, 4302.67 examples/s] Map: 40%|โโโโ | 16000/40000 [00:03<00:05, 4228.55 examples/s] Map: 42%|โโโโโ | 17000/40000 [00:04<00:05, 4296.47 examples/s] Map: 45%|โโโโโ | 18000/40000 [00:04<00:05, 4311.63 examples/s] Map: 48%|โโโโโ | 19000/40000 [00:04<00:04, 4300.71 examples/s] Map: 50%|โโโโโ | 20000/40000 [00:04<00:04, 4364.00 examples/s] Map: 52%|โโโโโโ | 21000/40000 [00:05<00:04, 4197.62 examples/s] Map: 55%|โโโโโโ | 22000/40000 [00:05<00:04, 4228.29 examples/s] Map: 57%|โโโโโโ | 23000/40000 [00:05<00:03, 4266.51 examples/s] Map: 60%|โโโโโโ | 24000/40000 [00:05<00:03, 4299.29 examples/s] Map: 62%|โโโโโโโ | 25000/40000 [00:06<00:03, 4308.04 examples/s] Map: 65%|โโโโโโโ | 26000/40000 [00:06<00:03, 4345.02 examples/s] Map: 68%|โโโโโโโ | 27000/40000 [00:06<00:03, 4317.96 examples/s] Map: 70%|โโโโโโโ | 28000/40000 [00:06<00:02, 4319.04 examples/s] Map: 72%|โโโโโโโโ | 29000/40000 [00:07<00:02, 4343.22 examples/s] Map: 75%|โโโโโโโโ | 30000/40000 [00:07<00:02, 4344.63 examples/s] Map: 78%|โโโโโโโโ | 31000/40000 [00:07<00:02, 4362.97 examples/s] Map: 80%|โโโโโโโโ | 32000/40000 [00:07<00:01, 4397.59 examples/s] Map: 82%|โโโโโโโโโ | 33000/40000 [00:07<00:01, 4392.02 examples/s] Map: 85%|โโโโโโโโโ | 34000/40000 [00:08<00:01, 3290.20 examples/s] Map: 88%|โโโโโโโโโ | 35000/40000 [00:08<00:01, 3566.64 examples/s] Map: 90%|โโโโโโโโโ | 36000/40000 [00:08<00:01, 3749.19 examples/s] Map: 92%|โโโโโโโโโโ| 37000/40000 [00:09<00:00, 3928.32 examples/s] Map: 95%|โโโโโโโโโโ| 38000/40000 [00:09<00:00, 4065.60 examples/s] Map: 98%|โโโโโโโโโโ| 39000/40000 [00:09<00:00, 4125.08 examples/s] Map: 100%|โโโโโโโโโโ| 40000/40000 [00:09<00:00, 4199.39 examples/s] Map: 100%|โโโโโโโโโโ| 40000/40000 [00:09<00:00, 4089.96 examples/s] | |
| Map: 0%| | 0/5000 [00:00<?, ? examples/s] Map: 20%|โโ | 1000/5000 [00:00<00:00, 4687.37 examples/s] Map: 40%|โโโโ | 2000/5000 [00:00<00:00, 4537.19 examples/s] Map: 60%|โโโโโโ | 3000/5000 [00:00<00:00, 4452.60 examples/s] Map: 80%|โโโโโโโโ | 4000/5000 [00:00<00:00, 4445.04 examples/s] Map: 100%|โโโโโโโโโโ| 5000/5000 [00:01<00:00, 4391.97 examples/s] Map: 100%|โโโโโโโโโโ| 5000/5000 [00:01<00:00, 4418.94 examples/s] | |
| /tmp/home/wzh/final/train.py:145: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `CustomTrainer.__init__`. Use `processing_class` instead. | |
| trainer = CustomTrainer( | |
| Some weights of RobertaForSequenceClassification were not initialized from the model checkpoint at roberta-base and are newly initialized: ['classifier.dense.bias', 'classifier.dense.weight', 'classifier.out_proj.bias', 'classifier.out_proj.weight'] | |
| You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference. | |
| [I 2025-11-26 12:05:51,188] A new study created in memory with name: no-name-0b166e0d-c37b-4d46-a204-3b08905480a6 | |
| Some weights of RobertaForSequenceClassification were not initialized from the model checkpoint at roberta-base and are newly initialized: ['classifier.dense.bias', 'classifier.dense.weight', 'classifier.out_proj.bias', 'classifier.out_proj.weight'] | |
| You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference. | |
| ================================================== | |
| ๐ ๅผๅง่ชๅจ่ถ ๅๆฐๆ็ดข (็ฎๆ : ๆๅคงๅ Macro-F1)... | |
| ================================================== | |
| 0%| | 0/1565 [00:00<?, ?it/s]7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360481 [0] NCCL INFO cudaDriverVersion 12020 | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360481 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360481 [0] NCCL INFO Bootstrap: Using eth0:10.0.0.13<0> | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360481 [0] NCCL INFO NCCL version 2.27.5+cuda12.9 | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360878 [2] NCCL INFO NET/Plugin: Plugin name set by env to libnccl-net.so | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360878 [2] NCCL INFO NET/Plugin: Failed to find ncclNetPlugin_v10 symbol. | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360878 [2] NCCL INFO NET/Plugin: Failed to find ncclNetPlugin_v9 symbol. | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360878 [2] NCCL INFO NET/Plugin: Failed to find ncclNetPlugin_v8 symbol. | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360878 [2] NCCL INFO NET/Plugin: Failed to find ncclNetPlugin_v7 symbol. | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360878 [2] NCCL INFO NET/Plugin: Loaded net plugin NCCL RDMA Plugin v6 (v6) | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360878 [2] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v10 symbol. | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360878 [2] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v9 symbol. | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360878 [2] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v8 symbol. | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360878 [2] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v7 symbol. | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360878 [2] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v6 symbol. | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360878 [2] NCCL INFO Successfully loaded external plugin libnccl-net.so | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360878 [2] NCCL INFO Plugin Path : /opt/nccl-rdma-sharp-plugins/lib/libnccl-net.so | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360878 [2] NCCL INFO P2P plugin IBext | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360878 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360878 [2] NCCL INFO NET/IB : Using [0]mlx5_ib0:1/IB [RO]; OOB eth0:10.0.0.13<0> | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360878 [2] NCCL INFO Initialized NET plugin NCCL RDMA Plugin v6 | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360878 [2] NCCL INFO Assigned NET plugin NCCL RDMA Plugin v6 to comm | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360878 [2] NCCL INFO Using network NCCL RDMA Plugin v6 | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360877 [1] NCCL INFO Assigned NET plugin NCCL RDMA Plugin v6 to comm | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360877 [1] NCCL INFO Using network NCCL RDMA Plugin v6 | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360879 [3] NCCL INFO Assigned NET plugin NCCL RDMA Plugin v6 to comm | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360879 [3] NCCL INFO Using network NCCL RDMA Plugin v6 | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360876 [0] NCCL INFO Assigned NET plugin NCCL RDMA Plugin v6 to comm | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360876 [0] NCCL INFO Using network NCCL RDMA Plugin v6 | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360879 [3] NCCL INFO ncclCommInitAll comm 0x12465150 rank 3 nranks 4 cudaDev 3 nvmlDev 7 busId 800000 commId 0xa5c975f20549b316 - Init START | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360876 [0] NCCL INFO ncclCommInitAll comm 0x12132bb0 rank 0 nranks 4 cudaDev 0 nvmlDev 4 busId 500000 commId 0xa5c975f20549b316 - Init START | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360878 [2] NCCL INFO ncclCommInitAll comm 0x12353800 rank 2 nranks 4 cudaDev 2 nvmlDev 6 busId 700000 commId 0xa5c975f20549b316 - Init START | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360877 [1] NCCL INFO ncclCommInitAll comm 0x12242130 rank 1 nranks 4 cudaDev 1 nvmlDev 5 busId 600000 commId 0xa5c975f20549b316 - Init START | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360879 [3] NCCL INFO RAS client listening socket at 127.0.0.1<28028> | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360876 [0] NCCL INFO Bootstrap timings total 0.000982 (create 0.000036, send 0.000108, recv 0.000413, ring 0.000085, delay 0.000000) | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360878 [2] NCCL INFO Bootstrap timings total 0.000995 (create 0.000028, send 0.000098, recv 0.000241, ring 0.000073, delay 0.000000) | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360879 [3] NCCL INFO Bootstrap timings total 0.001021 (create 0.000043, send 0.000119, recv 0.000335, ring 0.000103, delay 0.000001) | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360877 [1] NCCL INFO Bootstrap timings total 0.000960 (create 0.000032, send 0.000097, recv 0.000472, ring 0.000064, delay 0.000000) | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360877 [1] NCCL INFO Setting affinity for GPU 5 to 20-39 | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360877 [1] NCCL INFO NVLS multicast support is not available on dev 1 (NVLS_NCHANNELS 0) | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360878 [2] NCCL INFO Setting affinity for GPU 6 to 20-39 | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360878 [2] NCCL INFO NVLS multicast support is not available on dev 2 (NVLS_NCHANNELS 0) | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360879 [3] NCCL INFO Setting affinity for GPU 7 to 20-39 | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360879 [3] NCCL INFO NVLS multicast support is not available on dev 3 (NVLS_NCHANNELS 0) | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360876 [0] NCCL INFO Setting affinity for GPU 4 to 20-39 | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360876 [0] NCCL INFO NVLS multicast support is not available on dev 0 (NVLS_NCHANNELS 0) | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360878 [2] NCCL INFO comm 0x12353800 rank 2 nRanks 4 nNodes 1 localRanks 4 localRank 2 MNNVL 0 | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360876 [0] NCCL INFO comm 0x12132bb0 rank 0 nRanks 4 nNodes 1 localRanks 4 localRank 0 MNNVL 0 | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360877 [1] NCCL INFO comm 0x12242130 rank 1 nRanks 4 nNodes 1 localRanks 4 localRank 1 MNNVL 0 | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360878 [2] NCCL INFO Trees [0] 3/-1/-1->2->1 [1] 1/-1/-1->2->3 [2] 1/-1/-1->2->3 [3] 3/-1/-1->2->0 [4] 3/-1/-1->2->1 [5] 1/-1/-1->2->3 [6] 1/-1/-1->2->3 [7] 3/-1/-1->2->0 | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360879 [3] NCCL INFO comm 0x12465150 rank 3 nRanks 4 nNodes 1 localRanks 4 localRank 3 MNNVL 0 | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360877 [1] NCCL INFO Trees [0] 2/-1/-1->1->0 [1] -1/-1/-1->1->2 [2] -1/-1/-1->1->2 [3] -1/-1/-1->1->3 [4] 2/-1/-1->1->0 [5] -1/-1/-1->1->2 [6] -1/-1/-1->1->2 [7] -1/-1/-1->1->3 | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360877 [1] NCCL INFO P2P Chunksize set to 524288 | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360878 [2] NCCL INFO P2P Chunksize set to 524288 | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360879 [3] NCCL INFO Trees [0] -1/-1/-1->3->2 [1] 2/-1/-1->3->0 [2] 2/-1/-1->3->0 [3] 1/-1/-1->3->2 [4] -1/-1/-1->3->2 [5] 2/-1/-1->3->0 [6] 2/-1/-1->3->0 [7] 1/-1/-1->3->2 | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360879 [3] NCCL INFO P2P Chunksize set to 524288 | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360876 [0] NCCL INFO Channel 00/08 : 0 1 2 3 | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360876 [0] NCCL INFO Channel 01/08 : 0 3 2 1 | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360876 [0] NCCL INFO Channel 02/08 : 0 3 1 2 | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360876 [0] NCCL INFO Channel 03/08 : 0 2 1 3 | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360876 [0] NCCL INFO Channel 04/08 : 0 1 2 3 | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360876 [0] NCCL INFO Channel 05/08 : 0 3 2 1 | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360876 [0] NCCL INFO Channel 06/08 : 0 3 1 2 | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360876 [0] NCCL INFO Channel 07/08 : 0 2 1 3 | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360876 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 3/-1/-1->0->-1 [2] 3/-1/-1->0->-1 [3] 2/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 3/-1/-1->0->-1 [6] 3/-1/-1->0->-1 [7] 2/-1/-1->0->-1 | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360876 [0] NCCL INFO P2P Chunksize set to 524288 | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360877 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360888 [1] NCCL INFO [Proxy Service] Device 1 CPU core 22 | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360889 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 23 | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360878 [2] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360890 [2] NCCL INFO [Proxy Service] Device 2 CPU core 28 | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360891 [2] NCCL INFO [Proxy Service UDS] Device 2 CPU core 30 | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360879 [3] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360892 [3] NCCL INFO [Proxy Service] Device 3 CPU core 20 | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360893 [3] NCCL INFO [Proxy Service UDS] Device 3 CPU core 27 | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360876 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360876 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 1 | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360894 [0] NCCL INFO [Proxy Service] Device 0 CPU core 29 | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360895 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 37 | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360877 [1] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 512 | 512 | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360877 [1] NCCL INFO 8 coll channels, 8 collnet channels, 0 nvls channels, 8 p2p channels, 2 p2p channels per peer | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360878 [2] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 512 | 512 | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360878 [2] NCCL INFO 8 coll channels, 8 collnet channels, 0 nvls channels, 8 p2p channels, 2 p2p channels per peer | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360879 [3] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 512 | 512 | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360879 [3] NCCL INFO 8 coll channels, 8 collnet channels, 0 nvls channels, 8 p2p channels, 2 p2p channels per peer | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360876 [0] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 512 | 512 | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360876 [0] NCCL INFO 8 coll channels, 8 collnet channels, 0 nvls channels, 8 p2p channels, 2 p2p channels per peer | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360876 [0] NCCL INFO CC Off, workFifoBytes 1048576 | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360877 [1] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin. | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360877 [1] NCCL INFO TUNER/Plugin: Failed to find ncclTunerPlugin_v4 symbol. | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360877 [1] NCCL INFO TUNER/Plugin: Failed to find ncclTunerPlugin_v3 symbol. | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360877 [1] NCCL INFO TUNER/Plugin: Failed to find ncclTunerPlugin_v2 symbol, using internal tuner instead. | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360877 [1] NCCL INFO ncclCommInitAll comm 0x12242130 rank 1 nranks 4 cudaDev 1 nvmlDev 5 busId 600000 commId 0xa5c975f20549b316 - Init COMPLETE | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360877 [1] NCCL INFO Init timings - ncclCommInitAll: rank 1 nranks 4 total 0.53 (kernels 0.46, alloc 0.02, bootstrap 0.00, allgathers 0.00, topo 0.02, graphs 0.01, connections 0.02, rest 0.00) | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360879 [3] NCCL INFO ncclCommInitAll comm 0x12465150 rank 3 nranks 4 cudaDev 3 nvmlDev 7 busId 800000 commId 0xa5c975f20549b316 - Init COMPLETE | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360879 [3] NCCL INFO Init timings - ncclCommInitAll: rank 3 nranks 4 total 0.53 (kernels 0.46, alloc 0.02, bootstrap 0.00, allgathers 0.00, topo 0.02, graphs 0.01, connections 0.02, rest 0.01) | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360878 [2] NCCL INFO ncclCommInitAll comm 0x12353800 rank 2 nranks 4 cudaDev 2 nvmlDev 6 busId 700000 commId 0xa5c975f20549b316 - Init COMPLETE | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360878 [2] NCCL INFO Init timings - ncclCommInitAll: rank 2 nranks 4 total 0.53 (kernels 0.46, alloc 0.02, bootstrap 0.00, allgathers 0.00, topo 0.02, graphs 0.01, connections 0.02, rest 0.00) | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360876 [0] NCCL INFO ncclCommInitAll comm 0x12132bb0 rank 0 nranks 4 cudaDev 0 nvmlDev 4 busId 500000 commId 0xa5c975f20549b316 - Init COMPLETE | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360876 [0] NCCL INFO Init timings - ncclCommInitAll: rank 0 nranks 4 total 0.53 (kernels 0.46, alloc 0.02, bootstrap 0.00, allgathers 0.00, topo 0.02, graphs 0.01, connections 0.02, rest 0.00) | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360896 [3] NCCL INFO Channel 00/0 : 3[7] -> 0[4] via P2P/direct pointer | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360898 [1] NCCL INFO Channel 00/0 : 1[5] -> 2[6] via P2P/direct pointer | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360896 [3] NCCL INFO Channel 03/0 : 3[7] -> 0[4] via P2P/direct pointer | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360898 [1] NCCL INFO Channel 02/0 : 1[5] -> 2[6] via P2P/direct pointer | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360896 [3] NCCL INFO Channel 04/0 : 3[7] -> 0[4] via P2P/direct pointer | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360897 [2] NCCL INFO Channel 00/0 : 2[6] -> 3[7] via P2P/direct pointer | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360899 [0] NCCL INFO Channel 00/0 : 0[4] -> 1[5] via P2P/direct pointer | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360898 [1] NCCL INFO Channel 04/0 : 1[5] -> 2[6] via P2P/direct pointer | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360896 [3] NCCL INFO Channel 07/0 : 3[7] -> 0[4] via P2P/direct pointer | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360897 [2] NCCL INFO Channel 04/0 : 2[6] -> 3[7] via P2P/direct pointer | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360899 [0] NCCL INFO Channel 04/0 : 0[4] -> 1[5] via P2P/direct pointer | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360898 [1] NCCL INFO Channel 06/0 : 1[5] -> 2[6] via P2P/direct pointer | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360896 [3] NCCL INFO Channel 02/0 : 3[7] -> 1[5] via P2P/direct pointer | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360899 [0] NCCL INFO Channel 03/0 : 0[4] -> 2[6] via P2P/direct pointer | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360897 [2] NCCL INFO Channel 02/0 : 2[6] -> 0[4] via P2P/direct pointer | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360898 [1] NCCL INFO Channel 03/0 : 1[5] -> 3[7] via P2P/direct pointer | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360896 [3] NCCL INFO Channel 06/0 : 3[7] -> 1[5] via P2P/direct pointer | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360899 [0] NCCL INFO Channel 07/0 : 0[4] -> 2[6] via P2P/direct pointer | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360897 [2] NCCL INFO Channel 06/0 : 2[6] -> 0[4] via P2P/direct pointer | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360898 [1] NCCL INFO Channel 07/0 : 1[5] -> 3[7] via P2P/direct pointer | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360899 [0] NCCL INFO Channel 01/0 : 0[4] -> 3[7] via P2P/direct pointer | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360897 [2] NCCL INFO Channel 01/0 : 2[6] -> 1[5] via P2P/direct pointer | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360899 [0] NCCL INFO Channel 02/0 : 0[4] -> 3[7] via P2P/direct pointer | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360897 [2] NCCL INFO Channel 03/0 : 2[6] -> 1[5] via P2P/direct pointer | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360899 [0] NCCL INFO Channel 05/0 : 0[4] -> 3[7] via P2P/direct pointer | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360897 [2] NCCL INFO Channel 05/0 : 2[6] -> 1[5] via P2P/direct pointer | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360896 [3] NCCL INFO Channel 01/0 : 3[7] -> 2[6] via P2P/direct pointer | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360898 [1] NCCL INFO Channel 01/0 : 1[5] -> 0[4] via P2P/direct pointer | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360899 [0] NCCL INFO Channel 06/0 : 0[4] -> 3[7] via P2P/direct pointer | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360897 [2] NCCL INFO Channel 07/0 : 2[6] -> 1[5] via P2P/direct pointer | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360896 [3] NCCL INFO Channel 05/0 : 3[7] -> 2[6] via P2P/direct pointer | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360898 [1] NCCL INFO Channel 05/0 : 1[5] -> 0[4] via P2P/direct pointer | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360899 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1 | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360897 [2] NCCL INFO Connected all rings, use ring PXN 0 GDR 1 | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360898 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 1 | |
| 7a05a34eb459462580c1cf15dccd48e000002L:2360481:2360896 [3] NCCL INFO Connected all rings, use ring PXN 0 GDR 1 | |
| 0%| | 1/1565 [00:03<1:25:15, 3.27s/it] 0%| | 2/1565 [00:04<49:45, 1.91s/it] 0%| | 3/1565 [00:05<38:26, 1.48s/it] 0%| | 4/1565 [00:06<33:10, 1.28s/it] 0%| | 5/1565 [00:07<30:12, 1.16s/it] 0%| | 6/1565 [00:08<28:24, 1.09s/it] 0%| | 7/1565 [00:09<27:17, 1.05s/it] 1%| | 8/1565 [00:10<26:33, 1.02s/it] 1%| | 9/1565 [00:10<26:02, 1.00s/it] 1%| | 10/1565 [00:11<25:41, 1.01it/s] 1%| | 11/1565 [00:12<25:27, 1.02it/s] 1%| | 12/1565 [00:13<25:16, 1.02it/s] 1%| | 13/1565 [00:14<25:09, 1.03it/s] 1%| | 14/1565 [00:15<25:02, 1.03it/s] 1%| | 15/1565 [00:16<25:00, 1.03it/s] 1%| | 16/1565 [00:17<24:56, 1.03it/s] 1%| | 17/1565 [00:18<24:54, 1.04it/s] 1%| | 18/1565 [00:19<24:52, 1.04it/s] 1%| | 19/1565 [00:20<24:52, 1.04it/s] 1%|โ | 20/1565 [00:21<24:52, 1.04it/s] 1%|โ | 21/1565 [00:22<24:48, 1.04it/s] 1%|โ | 22/1565 [00:23<24:48, 1.04it/s] 1%|โ | 23/1565 [00:24<24:48, 1.04it/s] 2%|โ | 24/1565 [00:25<24:47, 1.04it/s] 2%|โ | 25/1565 [00:26<24:45, 1.04it/s] 2%|โ | 26/1565 [00:27<24:46, 1.04it/s] 2%|โ | 27/1565 [00:28<24:45, 1.04it/s] 2%|โ | 28/1565 [00:29<24:46, 1.03it/s] 2%|โ | 29/1565 [00:30<24:44, 1.03it/s] 2%|โ | 30/1565 [00:31<24:45, 1.03it/s] 2%|โ | 31/1565 [00:32<24:46, 1.03it/s] 2%|โ | 32/1565 [00:33<24:45, 1.03it/s] 2%|โ | 33/1565 [00:34<24:43, 1.03it/s] 2%|โ | 34/1565 [00:35<24:44, 1.03it/s] 2%|โ | 35/1565 [00:36<24:46, 1.03it/s] 2%|โ | 36/1565 [00:37<24:47, 1.03it/s] 2%|โ | 37/1565 [00:38<24:46, 1.03it/s] 2%|โ | 38/1565 [00:39<24:45, 1.03it/s] 2%|โ | 39/1565 [00:39<24:45, 1.03it/s] 3%|โ | 40/1565 [00:40<24:41, 1.03it/s] 3%|โ | 41/1565 [00:41<24:40, 1.03it/s] 3%|โ | 42/1565 [00:42<24:40, 1.03it/s] 3%|โ | 43/1565 [00:43<24:40, 1.03it/s] 3%|โ | 44/1565 [00:44<24:38, 1.03it/s] 3%|โ | 45/1565 [00:45<24:34, 1.03it/s] 3%|โ | 46/1565 [00:46<24:37, 1.03it/s] 3%|โ | 47/1565 [00:48<26:28, 1.05s/it] 3%|โ | 48/1565 [00:48<25:52, 1.02s/it] 3%|โ | 49/1565 [00:49<25:28, 1.01s/it] 3%|โ | 50/1565 [00:50<25:14, 1.00it/s] 3%|โ | 51/1565 [00:51<25:00, 1.01it/s] 3%|โ | 52/1565 [00:52<24:51, 1.01it/s] 3%|โ | 53/1565 [00:53<24:43, 1.02it/s] 3%|โ | 54/1565 [00:54<24:38, 1.02it/s] 4%|โ | 55/1565 [00:55<24:35, 1.02it/s] 4%|โ | 56/1565 [00:56<24:32, 1.02it/s] 4%|โ | 57/1565 [00:57<24:29, 1.03it/s] 4%|โ | 58/1565 [00:58<24:28, 1.03it/s] 4%|โ | 59/1565 [00:59<24:27, 1.03it/s] 4%|โ | 60/1565 [01:00<24:27, 1.03it/s] 4%|โ | 61/1565 [01:01<24:28, 1.02it/s] 4%|โ | 62/1565 [01:02<24:26, 1.02it/s] 4%|โ | 63/1565 [01:03<24:25, 1.02it/s] 4%|โ | 64/1565 [01:04<24:24, 1.03it/s] 4%|โ | 65/1565 [01:05<24:23, 1.02it/s] 4%|โ | 66/1565 [01:06<24:22, 1.02it/s] 4%|โ | 67/1565 [01:07<24:20, 1.03it/s] 4%|โ | 68/1565 [01:08<24:20, 1.02it/s] 4%|โ | 69/1565 [01:09<24:22, 1.02it/s] 4%|โ | 70/1565 [01:10<24:20, 1.02it/s] 5%|โ | 71/1565 [01:11<24:17, 1.02it/s] 5%|โ | 72/1565 [01:12<24:15, 1.03it/s] 5%|โ | 73/1565 [01:13<24:14, 1.03it/s] 5%|โ | 74/1565 [01:14<24:15, 1.02it/s] 5%|โ | 75/1565 [01:15<24:11, 1.03it/s] 5%|โ | 76/1565 [01:16<24:10, 1.03it/s] 5%|โ | 77/1565 [01:17<24:09, 1.03it/s] 5%|โ | 78/1565 [01:18<24:09, 1.03it/s] 5%|โ | 79/1565 [01:19<24:08, 1.03it/s] 5%|โ | 80/1565 [01:20<24:07, 1.03it/s] 5%|โ | 81/1565 [01:21<24:06, 1.03it/s] 5%|โ | 82/1565 [01:22<24:05, 1.03it/s] 5%|โ | 83/1565 [01:23<24:06, 1.02it/s] 5%|โ | 84/1565 [01:24<24:06, 1.02it/s] 5%|โ | 85/1565 [01:25<24:05, 1.02it/s] 5%|โ | 86/1565 [01:26<24:03, 1.02it/s] 6%|โ | 87/1565 [01:27<24:06, 1.02it/s] 6%|โ | 88/1565 [01:27<24:05, 1.02it/s] 6%|โ | 89/1565 [01:28<24:03, 1.02it/s] 6%|โ | 90/1565 [01:29<24:01, 1.02it/s] 6%|โ | 91/1565 [01:30<23:58, 1.02it/s] 6%|โ | 92/1565 [01:31<23:58, 1.02it/s] 6%|โ | 93/1565 [01:32<23:55, 1.03it/s] 6%|โ | 94/1565 [01:34<25:41, 1.05s/it] 6%|โ | 95/1565 [01:35<25:07, 1.03s/it] 6%|โ | 96/1565 [01:36<24:45, 1.01s/it] 6%|โ | 97/1565 [01:37<24:29, 1.00s/it] 6%|โ | 98/1565 [01:37<24:18, 1.01it/s] 6%|โ | 99/1565 [01:38<24:10, 1.01it/s] 6%|โ | 100/1565 [01:39<24:03, 1.02it/s] 6%|โ | 101/1565 [01:40<24:00, 1.02it/s] 7%|โ | 102/1565 [01:41<24:00, 1.02it/s] 7%|โ | 103/1565 [01:42<23:55, 1.02it/s] 7%|โ | 104/1565 [01:43<23:51, 1.02it/s] 7%|โ | 105/1565 [01:44<23:48, 1.02it/s] 7%|โ | 106/1565 [01:45<23:47, 1.02it/s] 7%|โ | 107/1565 [01:46<23:48, 1.02it/s] 7%|โ | 108/1565 [01:47<23:45, 1.02it/s] 7%|โ | 109/1565 [01:48<23:44, 1.02it/s] 7%|โ | 110/1565 [01:49<23:42, 1.02it/s] 7%|โ | 111/1565 [01:50<23:38, 1.02it/s] 7%|โ | 112/1565 [01:51<23:39, 1.02it/s] 7%|โ | 113/1565 [01:52<23:36, 1.02it/s] 7%|โ | 114/1565 [01:53<23:36, 1.02it/s] 7%|โ | 115/1565 [01:54<23:34, 1.03it/s] 7%|โ | 116/1565 [01:55<23:33, 1.03it/s] 7%|โ | 117/1565 [01:56<23:33, 1.02it/s] 8%|โ | 118/1565 [01:57<23:32, 1.02it/s] 8%|โ | 119/1565 [01:58<23:31, 1.02it/s] 8%|โ | 120/1565 [01:59<23:31, 1.02it/s] 8%|โ | 121/1565 [02:00<23:30, 1.02it/s] 8%|โ | 122/1565 [02:01<23:27, 1.02it/s] 8%|โ | 123/1565 [02:02<23:26, 1.02it/s] 8%|โ | 124/1565 [02:03<23:25, 1.03it/s] 8%|โ | 125/1565 [02:04<23:25, 1.02it/s] 8%|โ | 126/1565 [02:05<23:24, 1.02it/s] 8%|โ | 127/1565 [02:06<23:28, 1.02it/s] 8%|โ | 128/1565 [02:07<23:26, 1.02it/s] 8%|โ | 129/1565 [02:08<23:24, 1.02it/s] 8%|โ | 130/1565 [02:09<23:23, 1.02it/s] 8%|โ | 131/1565 [02:10<23:19, 1.02it/s] 8%|โ | 132/1565 [02:11<23:17, 1.03it/s] 8%|โ | 133/1565 [02:12<23:18, 1.02it/s] 9%|โ | 134/1565 [02:13<23:16, 1.02it/s] 9%|โ | 135/1565 [02:14<23:14, 1.03it/s] 9%|โ | 136/1565 [02:15<23:12, 1.03it/s] 9%|โ | 137/1565 [02:16<23:13, 1.02it/s] 9%|โ | 138/1565 [02:17<23:13, 1.02it/s] 9%|โ | 139/1565 [02:18<23:10, 1.03it/s] 9%|โ | 140/1565 [02:19<23:07, 1.03it/s] 9%|โ | 141/1565 [02:20<24:43, 1.04s/it] 9%|โ | 142/1565 [02:21<24:13, 1.02s/it] 9%|โ | 143/1565 [02:22<23:52, 1.01s/it] 9%|โ | 144/1565 [02:23<23:38, 1.00it/s] 9%|โ | 145/1565 [02:24<23:29, 1.01it/s] 9%|โ | 146/1565 [02:25<23:21, 1.01it/s] 9%|โ | 147/1565 [02:26<23:14, 1.02it/s] 9%|โ | 148/1565 [02:27<23:10, 1.02it/s] 10%|โ | 149/1565 [02:28<23:08, 1.02it/s] 10%|โ | 150/1565 [02:28<23:09, 1.02it/s] 10%|โ | 151/1565 [02:29<23:10, 1.02it/s] 10%|โ | 152/1565 [02:30<23:07, 1.02it/s] 10%|โ | 153/1565 [02:31<23:04, 1.02it/s] 10%|โ | 154/1565 [02:32<23:02, 1.02it/s] 10%|โ | 155/1565 [02:33<23:02, 1.02it/s] 10%|โ | 156/1565 [02:34<22:59, 1.02it/s] 10%|โ | 157/1565 [02:35<22:58, 1.02it/s] 10%|โ | 158/1565 [02:36<22:54, 1.02it/s] 10%|โ | 159/1565 [02:37<22:53, 1.02it/s] 10%|โ | 160/1565 [02:38<22:51, 1.02it/s] 10%|โ | 161/1565 [02:39<22:48, 1.03it/s] 10%|โ | 162/1565 [02:40<22:51, 1.02it/s] 10%|โ | 163/1565 [02:41<22:51, 1.02it/s] 10%|โ | 164/1565 [02:42<22:49, 1.02it/s] 11%|โ | 165/1565 [02:43<22:50, 1.02it/s] 11%|โ | 166/1565 [02:44<22:50, 1.02it/s] 11%|โ | 167/1565 [02:45<22:50, 1.02it/s] 11%|โ | 168/1565 [02:46<22:47, 1.02it/s] 11%|โ | 169/1565 [02:47<22:48, 1.02it/s] 11%|โ | 170/1565 [02:48<22:47, 1.02it/s] 11%|โ | 171/1565 [02:49<22:46, 1.02it/s] 11%|โ | 172/1565 [02:50<22:44, 1.02it/s] 11%|โ | 173/1565 [02:51<22:43, 1.02it/s] 11%|โ | 174/1565 [02:52<22:41, 1.02it/s] 11%|โ | 175/1565 [02:53<22:39, 1.02it/s] 11%|โ | 176/1565 [02:54<22:37, 1.02it/s] 11%|โโ | 177/1565 [02:55<22:38, 1.02it/s] 11%|โโ | 178/1565 [02:56<22:36, 1.02it/s] 11%|โโ | 179/1565 [02:57<22:35, 1.02it/s] 12%|โโ | 180/1565 [02:58<22:35, 1.02it/s] 12%|โโ | 181/1565 [02:59<22:33, 1.02it/s] 12%|โโ | 182/1565 [03:00<22:30, 1.02it/s] 12%|โโ | 183/1565 [03:01<22:32, 1.02it/s] 12%|โโ | 184/1565 [03:02<22:31, 1.02it/s] 12%|โโ | 185/1565 [03:03<22:27, 1.02it/s] 12%|โโ | 186/1565 [03:04<22:25, 1.02it/s] 12%|โโ | 187/1565 [03:05<22:24, 1.03it/s] 12%|โโ | 188/1565 [03:06<23:55, 1.04s/it] 12%|โโ | 189/1565 [03:07<23:25, 1.02s/it] 12%|โโ | 190/1565 [03:08<23:08, 1.01s/it] 12%|โโ | 191/1565 [03:09<22:54, 1.00s/it] 12%|โโ | 192/1565 [03:10<22:44, 1.01it/s] 12%|โโ | 193/1565 [03:11<22:33, 1.01it/s] 12%|โโ | 194/1565 [03:12<22:28, 1.02it/s] 12%|โโ | 195/1565 [03:13<22:25, 1.02it/s] 13%|โโ | 196/1565 [03:14<22:22, 1.02it/s] 13%|โโ | 197/1565 [03:15<22:20, 1.02it/s] 13%|โโ | 198/1565 [03:16<22:16, 1.02it/s] 13%|โโ | 199/1565 [03:17<22:13, 1.02it/s] 13%|โโ | 200/1565 [03:18<22:11, 1.03it/s] 13%|โโ | 201/1565 [03:19<22:11, 1.02it/s] 13%|โโ | 202/1565 [03:20<22:11, 1.02it/s] 13%|โโ | 203/1565 [03:21<22:11, 1.02it/s] 13%|โโ | 204/1565 [03:22<22:08, 1.02it/s] 13%|โโ | 205/1565 [03:22<22:05, 1.03it/s] 13%|โโ | 206/1565 [03:23<22:05, 1.03it/s] 13%|โโ | 207/1565 [03:24<22:07, 1.02it/s] 13%|โโ | 208/1565 [03:25<22:06, 1.02it/s] 13%|โโ | 209/1565 [03:26<22:05, 1.02it/s] 13%|โโ | 210/1565 [03:27<22:03, 1.02it/s] 13%|โโ | 211/1565 [03:28<22:01, 1.02it/s] 14%|โโ | 212/1565 [03:29<21:59, 1.03it/s] 14%|โโ | 213/1565 [03:30<22:01, 1.02it/s] 14%|โโ | 214/1565 [03:31<22:00, 1.02it/s] 14%|โโ | 215/1565 [03:32<21:57, 1.02it/s] 14%|โโ | 216/1565 [03:33<21:58, 1.02it/s] 14%|โโ | 217/1565 [03:34<21:56, 1.02it/s] 14%|โโ | 218/1565 [03:35<21:58, 1.02it/s] 14%|โโ | 219/1565 [03:36<21:55, 1.02it/s] 14%|โโ | 220/1565 [03:37<21:53, 1.02it/s] 14%|โโ | 221/1565 [03:38<21:52, 1.02it/s] 14%|โโ | 222/1565 [03:39<21:50, 1.02it/s] 14%|โโ | 223/1565 [03:40<21:51, 1.02it/s] 14%|โโ | 224/1565 [03:41<21:50, 1.02it/s] 14%|โโ | 225/1565 [03:42<21:50, 1.02it/s] 14%|โโ | 226/1565 [03:43<21:47, 1.02it/s] 15%|โโ | 227/1565 [03:44<21:46, 1.02it/s] 15%|โโ | 228/1565 [03:45<21:46, 1.02it/s] 15%|โโ | 229/1565 [03:46<21:45, 1.02it/s] 15%|โโ | 230/1565 [03:47<21:43, 1.02it/s] 15%|โโ | 231/1565 [03:48<21:40, 1.03it/s] 15%|โโ | 232/1565 [03:49<21:40, 1.02it/s] 15%|โโ | 233/1565 [03:50<21:41, 1.02it/s] 15%|โโ | 234/1565 [03:51<21:43, 1.02it/s] 15%|โโ | 235/1565 [03:52<23:13, 1.05s/it] 15%|โโ | 236/1565 [03:53<22:45, 1.03s/it] 15%|โโ | 237/1565 [03:54<22:24, 1.01s/it] 15%|โโ | 238/1565 [03:55<22:08, 1.00s/it] 15%|โโ | 239/1565 [03:56<21:56, 1.01it/s] 15%|โโ | 240/1565 [03:57<21:49, 1.01it/s] 15%|โโ | 241/1565 [03:58<21:45, 1.01it/s] 15%|โโ | 242/1565 [03:59<21:42, 1.02it/s] 16%|โโ | 243/1565 [04:00<21:39, 1.02it/s] 16%|โโ | 244/1565 [04:01<21:35, 1.02it/s] 16%|โโ | 245/1565 [04:02<21:32, 1.02it/s] 16%|โโ | 246/1565 [04:03<21:31, 1.02it/s] 16%|โโ | 247/1565 [04:04<21:30, 1.02it/s] 16%|โโ | 248/1565 [04:05<21:27, 1.02it/s] 16%|โโ | 249/1565 [04:06<21:27, 1.02it/s] 16%|โโ | 250/1565 [04:07<21:25, 1.02it/s] 16%|โโ | 251/1565 [04:08<21:24, 1.02it/s] 16%|โโ | 252/1565 [04:09<21:21, 1.02it/s] 16%|โโ | 253/1565 [04:10<21:23, 1.02it/s] 16%|โโ | 254/1565 [04:11<21:20, 1.02it/s] 16%|โโ | 255/1565 [04:12<21:20, 1.02it/s] 16%|โโ | 256/1565 [04:13<21:20, 1.02it/s] 16%|โโ | 257/1565 [04:14<21:21, 1.02it/s] 16%|โโ | 258/1565 [04:15<21:19, 1.02it/s] 17%|โโ | 259/1565 [04:15<21:18, 1.02it/s] 17%|โโ | 260/1565 [04:16<21:14, 1.02it/s] 17%|โโ | 261/1565 [04:17<21:13, 1.02it/s] 17%|โโ | 262/1565 [04:18<21:12, 1.02it/s] 17%|โโ | 263/1565 [04:19<21:14, 1.02it/s] 17%|โโ | 264/1565 [04:20<21:10, 1.02it/s] 17%|โโ | 265/1565 [04:21<21:07, 1.03it/s] 17%|โโ | 266/1565 [04:22<21:07, 1.03it/s] 17%|โโ | 267/1565 [04:23<21:07, 1.02it/s] 17%|โโ | 268/1565 [04:24<21:07, 1.02it/s] 17%|โโ | 269/1565 [04:25<21:05, 1.02it/s] 17%|โโ | 270/1565 [04:26<21:02, 1.03it/s] 17%|โโ | 271/1565 [04:27<21:01, 1.03it/s] 17%|โโ | 272/1565 [04:28<21:00, 1.03it/s] 17%|โโ | 273/1565 [04:29<20:59, 1.03it/s] 18%|โโ | 274/1565 [04:30<20:59, 1.03it/s] 18%|โโ | 275/1565 [04:31<20:57, 1.03it/s] 18%|โโ | 276/1565 [04:32<20:56, 1.03it/s] 18%|โโ | 277/1565 [04:33<20:55, 1.03it/s] 18%|โโ | 278/1565 [04:34<20:54, 1.03it/s] 18%|โโ | 279/1565 [04:35<20:55, 1.02it/s] 18%|โโ | 280/1565 [04:36<20:53, 1.02it/s] 18%|โโ | 281/1565 [04:37<20:52, 1.03it/s] 18%|โโ | 282/1565 [04:38<22:26, 1.05s/it] 18%|โโ | 283/1565 [04:39<21:57, 1.03s/it] 18%|โโ | 284/1565 [04:40<21:36, 1.01s/it] 18%|โโ | 285/1565 [04:41<21:21, 1.00s/it] 18%|โโ | 286/1565 [04:42<21:11, 1.01it/s] 18%|โโ | 287/1565 [04:43<21:07, 1.01it/s] 18%|โโ | 288/1565 [04:44<21:00, 1.01it/s] 18%|โโ | 289/1565 [04:45<20:54, 1.02it/s] 19%|โโ | 290/1565 [04:46<20:51, 1.02it/s] 19%|โโ | 291/1565 [04:47<20:50, 1.02it/s] 19%|โโ | 292/1565 [04:48<20:48, 1.02it/s] 19%|โโ | 293/1565 [04:49<20:48, 1.02it/s] 19%|โโ | 294/1565 [04:50<20:45, 1.02it/s] 19%|โโ | 295/1565 [04:51<20:42, 1.02it/s] 19%|โโ | 296/1565 [04:52<20:39, 1.02it/s] 19%|โโ | 297/1565 [04:53<20:38, 1.02it/s] 19%|โโ | 298/1565 [04:54<20:39, 1.02it/s] 19%|โโ | 299/1565 [04:55<20:37, 1.02it/s] 19%|โโ | 300/1565 [04:56<20:34, 1.02it/s] 19%|โโ | 301/1565 [04:57<20:31, 1.03it/s] 19%|โโ | 302/1565 [04:58<20:31, 1.03it/s] 19%|โโ | 303/1565 [04:59<20:34, 1.02it/s] 19%|โโ | 304/1565 [05:00<20:35, 1.02it/s] 19%|โโ | 305/1565 [05:01<20:34, 1.02it/s] 20%|โโ | 306/1565 [05:02<20:31, 1.02it/s] 20%|โโ | 307/1565 [05:03<20:30, 1.02it/s] 20%|โโ | 308/1565 [05:04<20:28, 1.02it/s] 20%|โโ | 309/1565 [05:05<20:27, 1.02it/s] 20%|โโ | 310/1565 [05:06<20:27, 1.02it/s] 20%|โโ | 311/1565 [05:07<20:27, 1.02it/s] 20%|โโ | 312/1565 [05:07<20:11, 1.03it/s] 20%|โโ | 313/1565 [05:08<17:11, 1.21it/s] 20%|โโ | 314/1565 [05:09<18:39, 1.12it/s] 20%|โโ | 315/1565 [05:10<19:08, 1.09it/s] 20%|โโ | 316/1565 [05:11<19:31, 1.07it/s] 20%|โโ | 317/1565 [05:12<19:46, 1.05it/s] 20%|โโ | 318/1565 [05:13<19:52, 1.05it/s] 20%|โโ | 319/1565 [05:14<19:57, 1.04it/s] 20%|โโ | 320/1565 [05:15<20:02, 1.03it/s] 21%|โโ | 321/1565 [05:16<20:04, 1.03it/s] 21%|โโ | 322/1565 [05:17<20:06, 1.03it/s] 21%|โโ | 323/1565 [05:18<20:07, 1.03it/s] 21%|โโ | 324/1565 [05:19<20:07, 1.03it/s] 21%|โโ | 325/1565 [05:20<20:09, 1.03it/s] 21%|โโ | 326/1565 [05:21<20:10, 1.02it/s] 21%|โโ | 327/1565 [05:22<20:08, 1.02it/s] 21%|โโ | 328/1565 [05:23<20:06, 1.03it/s] 21%|โโ | 329/1565 [05:24<21:30, 1.04s/it] 21%|โโ | 330/1565 [05:25<21:04, 1.02s/it] 21%|โโ | 331/1565 [05:26<20:44, 1.01s/it] 21%|โโ | 332/1565 [05:27<20:32, 1.00it/s] 21%|โโโ | 333/1565 [05:28<20:23, 1.01it/s] 21%|โโโ | 334/1565 [05:29<20:18, 1.01it/s] 21%|โโโ | 335/1565 [05:30<20:11, 1.01it/s] 21%|โโโ | 336/1565 [05:31<20:07, 1.02it/s] 22%|โโโ | 337/1565 [05:32<20:04, 1.02it/s] 22%|โโโ | 338/1565 [05:33<20:03, 1.02it/s] 22%|โโโ | 339/1565 [05:34<20:01, 1.02it/s] 22%|โโโ | 340/1565 [05:35<19:58, 1.02it/s] 22%|โโโ | 341/1565 [05:36<19:55, 1.02it/s] 22%|โโโ | 342/1565 [05:37<19:54, 1.02it/s] 22%|โโโ | 343/1565 [05:38<19:55, 1.02it/s] 22%|โโโ | 344/1565 [05:39<19:56, 1.02it/s] 22%|โโโ | 345/1565 [05:40<19:54, 1.02it/s] 22%|โโโ | 346/1565 [05:41<19:53, 1.02it/s] 22%|โโโ | 347/1565 [05:41<19:50, 1.02it/s] 22%|โโโ | 348/1565 [05:42<19:48, 1.02it/s] 22%|โโโ | 349/1565 [05:43<19:47, 1.02it/s] 22%|โโโ | 350/1565 [05:44<19:46, 1.02it/s] 22%|โโโ | 351/1565 [05:45<19:49, 1.02it/s] 22%|โโโ | 352/1565 [05:46<19:47, 1.02it/s] 23%|โโโ | 353/1565 [05:47<19:43, 1.02it/s] 23%|โโโ | 354/1565 [05:48<19:41, 1.02it/s] 23%|โโโ | 355/1565 [05:49<19:41, 1.02it/s] 23%|โโโ | 356/1565 [05:50<19:41, 1.02it/s] 23%|โโโ | 357/1565 [05:51<19:40, 1.02it/s] 23%|โโโ | 358/1565 [05:52<19:39, 1.02it/s] 23%|โโโ | 359/1565 [05:53<19:36, 1.03it/s] 23%|โโโ | 360/1565 [05:54<19:36, 1.02it/s] 23%|โโโ | 361/1565 [05:55<19:36, 1.02it/s] 23%|โโโ | 362/1565 [05:56<19:34, 1.02it/s] 23%|โโโ | 363/1565 [05:57<19:32, 1.02it/s] 23%|โโโ | 364/1565 [05:58<19:32, 1.02it/s] 23%|โโโ | 365/1565 [05:59<19:30, 1.02it/s] 23%|โโโ | 366/1565 [06:00<19:29, 1.03it/s] 23%|โโโ | 367/1565 [06:01<19:31, 1.02it/s] 24%|โโโ | 368/1565 [06:02<19:28, 1.02it/s] 24%|โโโ | 369/1565 [06:03<19:27, 1.02it/s] 24%|โโโ | 370/1565 [06:04<19:26, 1.02it/s] 24%|โโโ | 371/1565 [06:05<19:25, 1.02it/s] 24%|โโโ | 372/1565 [06:06<19:26, 1.02it/s] 24%|โโโ | 373/1565 [06:07<19:23, 1.02it/s] 24%|โโโ | 374/1565 [06:08<19:22, 1.02it/s] 24%|โโโ | 375/1565 [06:09<19:19, 1.03it/s] 24%|โโโ | 376/1565 [06:10<20:39, 1.04s/it] 24%|โโโ | 377/1565 [06:11<20:14, 1.02s/it] 24%|โโโ | 378/1565 [06:12<19:56, 1.01s/it] 24%|โโโ | 379/1565 [06:13<19:46, 1.00s/it] 24%|โโโ | 380/1565 [06:14<19:38, 1.01it/s] 24%|โโโ | 381/1565 [06:15<19:31, 1.01it/s] 24%|โโโ | 382/1565 [06:16<19:24, 1.02it/s] 24%|โโโ | 383/1565 [06:17<19:19, 1.02it/s] 25%|โโโ | 384/1565 [06:18<19:17, 1.02it/s] 25%|โโโ | 385/1565 [06:19<19:17, 1.02it/s] 25%|โโโ | 386/1565 [06:20<19:16, 1.02it/s] 25%|โโโ | 387/1565 [06:21<19:11, 1.02it/s] 25%|โโโ | 388/1565 [06:22<19:09, 1.02it/s] 25%|โโโ | 389/1565 [06:23<19:07, 1.03it/s] 25%|โโโ | 390/1565 [06:24<19:06, 1.02it/s] 25%|โโโ | 391/1565 [06:25<19:06, 1.02it/s] 25%|โโโ | 392/1565 [06:26<19:06, 1.02it/s] 25%|โโโ | 393/1565 [06:27<19:05, 1.02it/s] 25%|โโโ | 394/1565 [06:28<19:06, 1.02it/s] 25%|โโโ | 395/1565 [06:29<19:03, 1.02it/s] 25%|โโโ | 396/1565 [06:30<19:00, 1.02it/s] 25%|โโโ | 397/1565 [06:31<19:00, 1.02it/s] 25%|โโโ | 398/1565 [06:32<18:58, 1.03it/s] 25%|โโโ | 399/1565 [06:32<18:57, 1.03it/s] 26%|โโโ | 400/1565 [06:33<18:56, 1.03it/s] 26%|โโโ | 401/1565 [06:34<18:54, 1.03it/s] 26%|โโโ | 402/1565 [06:35<18:54, 1.03it/s] 26%|โโโ | 403/1565 [06:36<18:52, 1.03it/s] 26%|โโโ | 404/1565 [06:37<18:55, 1.02it/s] 26%|โโโ | 405/1565 [06:38<18:56, 1.02it/s] 26%|โโโ | 406/1565 [06:39<18:54, 1.02it/s] 26%|โโโ | 407/1565 [06:40<18:53, 1.02it/s] 26%|โโโ | 408/1565 [06:41<18:54, 1.02it/s] 26%|โโโ | 409/1565 [06:42<18:51, 1.02it/s] 26%|โโโ | 410/1565 [06:43<18:52, 1.02it/s] 26%|โโโ | 411/1565 [06:44<18:51, 1.02it/s] 26%|โโโ | 412/1565 [06:45<18:49, 1.02it/s] 26%|โโโ | 413/1565 [06:46<18:50, 1.02it/s] 26%|โโโ | 414/1565 [06:47<18:48, 1.02it/s] 27%|โโโ | 415/1565 [06:48<18:45, 1.02it/s] 27%|โโโ | 416/1565 [06:49<18:44, 1.02it/s] 27%|โโโ | 417/1565 [06:50<18:43, 1.02it/s] 27%|โโโ | 418/1565 [06:51<18:43, 1.02it/s] 27%|โโโ | 419/1565 [06:52<18:41, 1.02it/s] 27%|โโโ | 420/1565 [06:53<18:38, 1.02it/s] 27%|โโโ | 421/1565 [06:54<18:36, 1.02it/s] 27%|โโโ | 422/1565 [06:55<18:36, 1.02it/s] 27%|โโโ | 423/1565 [06:56<19:59, 1.05s/it] 27%|โโโ | 424/1565 [06:57<19:33, 1.03s/it] 27%|โโโ | 425/1565 [06:58<19:15, 1.01s/it] 27%|โโโ | 426/1565 [06:59<19:02, 1.00s/it] 27%|โโโ | 427/1565 [07:00<18:52, 1.01it/s] 27%|โโโ | 428/1565 [07:01<18:45, 1.01it/s] 27%|โโโ | 429/1565 [07:02<18:39, 1.01it/s] 27%|โโโ | 430/1565 [07:03<18:36, 1.02it/s] 28%|โโโ | 431/1565 [07:04<18:34, 1.02it/s] 28%|โโโ | 432/1565 [07:05<18:32, 1.02it/s] 28%|โโโ | 433/1565 [07:06<18:29, 1.02it/s] 28%|โโโ | 434/1565 [07:07<18:27, 1.02it/s] 28%|โโโ | 435/1565 [07:08<18:26, 1.02it/s] 28%|โโโ | 436/1565 [07:09<18:24, 1.02it/s] 28%|โโโ | 437/1565 [07:10<18:24, 1.02it/s] 28%|โโโ | 438/1565 [07:11<18:22, 1.02it/s] 28%|โโโ | 439/1565 [07:12<18:19, 1.02it/s] 28%|โโโ | 440/1565 [07:13<18:17, 1.03it/s] 28%|โโโ | 441/1565 [07:14<18:17, 1.02it/s] 28%|โโโ | 442/1565 [07:15<18:16, 1.02it/s] 28%|โโโ | 443/1565 [07:16<18:15, 1.02it/s] 28%|โโโ | 444/1565 [07:17<18:14, 1.02it/s] 28%|โโโ | 445/1565 [07:18<18:14, 1.02it/s] 28%|โโโ | 446/1565 [07:19<18:15, 1.02it/s] 29%|โโโ | 447/1565 [07:20<18:15, 1.02it/s] 29%|โโโ | 448/1565 [07:21<18:14, 1.02it/s] 29%|โโโ | 449/1565 [07:22<18:10, 1.02it/s] 29%|โโโ | 450/1565 [07:23<18:08, 1.02it/s] 29%|โโโ | 451/1565 [07:24<18:05, 1.03it/s] 29%|โโโ | 452/1565 [07:25<18:05, 1.03it/s] 29%|โโโ | 453/1565 [07:26<18:02, 1.03it/s] 29%|โโโ | 454/1565 [07:26<18:01, 1.03it/s] 29%|โโโ | 455/1565 [07:27<18:01, 1.03it/s] 29%|โโโ | 456/1565 [07:28<17:59, 1.03it/s] 29%|โโโ | 457/1565 [07:29<17:59, 1.03it/s] 29%|โโโ | 458/1565 [07:30<17:59, 1.03it/s] 29%|โโโ | 459/1565 [07:31<17:57, 1.03it/s] 29%|โโโ | 460/1565 [07:32<17:56, 1.03it/s] 29%|โโโ | 461/1565 [07:33<17:54, 1.03it/s] 30%|โโโ | 462/1565 [07:34<17:51, 1.03it/s] 30%|โโโ | 463/1565 [07:35<17:51, 1.03it/s] 30%|โโโ | 464/1565 [07:36<17:50, 1.03it/s] 30%|โโโ | 465/1565 [07:37<17:50, 1.03it/s] 30%|โโโ | 466/1565 [07:38<17:50, 1.03it/s] 30%|โโโ | 467/1565 [07:39<17:51, 1.02it/s] 30%|โโโ | 468/1565 [07:40<17:49, 1.03it/s] 30%|โโโ | 469/1565 [07:41<17:46, 1.03it/s] 30%|โโโ | 470/1565 [07:42<19:00, 1.04s/it] 30%|โโโ | 471/1565 [07:43<18:36, 1.02s/it] 30%|โโโ | 472/1565 [07:44<18:19, 1.01s/it] 30%|โโโ | 473/1565 [07:45<18:06, 1.01it/s] 30%|โโโ | 474/1565 [07:46<17:55, 1.01it/s] 30%|โโโ | 475/1565 [07:47<17:49, 1.02it/s] 30%|โโโ | 476/1565 [07:48<17:46, 1.02it/s] 30%|โโโ | 477/1565 [07:49<17:43, 1.02it/s] 31%|โโโ | 478/1565 [07:50<17:40, 1.03it/s] 31%|โโโ | 479/1565 [07:51<17:37, 1.03it/s] 31%|โโโ | 480/1565 [07:52<17:35, 1.03it/s] 31%|โโโ | 481/1565 [07:53<17:33, 1.03it/s] 31%|โโโ | 482/1565 [07:54<17:32, 1.03it/s] 31%|โโโ | 483/1565 [07:55<17:32, 1.03it/s] 31%|โโโ | 484/1565 [07:56<17:30, 1.03it/s] 31%|โโโ | 485/1565 [07:57<17:29, 1.03it/s] 31%|โโโ | 486/1565 [07:58<17:29, 1.03it/s] 31%|โโโ | 487/1565 [07:59<17:29, 1.03it/s] 31%|โโโ | 488/1565 [08:00<17:29, 1.03it/s] 31%|โโโ | 489/1565 [08:01<17:27, 1.03it/s] 31%|โโโโ | 490/1565 [08:02<17:26, 1.03it/s] 31%|โโโโ | 491/1565 [08:03<17:26, 1.03it/s] 31%|โโโโ | 492/1565 [08:04<17:24, 1.03it/s] 32%|โโโโ | 493/1565 [08:05<17:22, 1.03it/s] 32%|โโโโ | 494/1565 [08:06<17:21, 1.03it/s] 32%|โโโโ | 495/1565 [08:07<17:22, 1.03it/s] 32%|โโโโ | 496/1565 [08:08<17:22, 1.03it/s] 32%|โโโโ | 497/1565 [08:09<17:21, 1.03it/s] 32%|โโโโ | 498/1565 [08:10<17:19, 1.03it/s] 32%|โโโโ | 499/1565 [08:11<17:17, 1.03it/s] 32%|โโโโ | 500/1565 [08:11<17:16, 1.03it/s] 32%|โโโโ | 501/1565 [08:12<17:15, 1.03it/s] 32%|โโโโ | 502/1565 [08:13<17:16, 1.03it/s] 32%|โโโโ | 503/1565 [08:14<17:13, 1.03it/s] 32%|โโโโ | 504/1565 [08:15<17:13, 1.03it/s] 32%|โโโโ | 505/1565 [08:16<17:13, 1.03it/s] 32%|โโโโ | 506/1565 [08:17<17:12, 1.03it/s] 32%|โโโโ | 507/1565 [08:18<17:11, 1.03it/s] 32%|โโโโ | 508/1565 [08:19<17:09, 1.03it/s] 33%|โโโโ | 509/1565 [08:20<17:07, 1.03it/s] 33%|โโโโ | 510/1565 [08:21<17:06, 1.03it/s] 33%|โโโโ | 511/1565 [08:22<17:05, 1.03it/s] 33%|โโโโ | 512/1565 [08:23<17:06, 1.03it/s] 33%|โโโโ | 513/1565 [08:24<17:03, 1.03it/s] 33%|โโโโ | 514/1565 [08:25<17:01, 1.03it/s] 33%|โโโโ | 515/1565 [08:26<16:59, 1.03it/s] 33%|โโโโ | 516/1565 [08:27<16:58, 1.03it/s] 33%|โโโโ | 517/1565 [08:28<18:11, 1.04s/it] 33%|โโโโ | 518/1565 [08:29<17:49, 1.02s/it] 33%|โโโโ | 519/1565 [08:30<17:31, 1.01s/it] 33%|โโโโ | 520/1565 [08:31<17:20, 1.00it/s] 33%|โโโโ | 521/1565 [08:32<17:10, 1.01it/s] 33%|โโโโ | 522/1565 [08:33<17:04, 1.02it/s] 33%|โโโโ | 523/1565 [08:34<17:00, 1.02it/s] 33%|โโโโ | 524/1565 [08:35<16:56, 1.02it/s] 34%|โโโโ | 525/1565 [08:36<16:55, 1.02it/s] 34%|โโโโ | 526/1565 [08:37<16:54, 1.02it/s] 34%|โโโโ | 527/1565 [08:38<16:54, 1.02it/s] 34%|โโโโ | 528/1565 [08:39<16:51, 1.03it/s] 34%|โโโโ | 529/1565 [08:40<16:49, 1.03it/s] 34%|โโโโ | 530/1565 [08:41<16:48, 1.03it/s] 34%|โโโโ | 531/1565 [08:42<16:46, 1.03it/s] 34%|โโโโ | 532/1565 [08:43<16:45, 1.03it/s] 34%|โโโโ | 533/1565 [08:44<16:43, 1.03it/s] 34%|โโโโ | 534/1565 [08:45<16:41, 1.03it/s] 34%|โโโโ | 535/1565 [08:46<16:40, 1.03it/s] 34%|โโโโ | 536/1565 [08:47<16:39, 1.03it/s] 34%|โโโโ | 537/1565 [08:48<16:37, 1.03it/s] 34%|โโโโ | 538/1565 [08:49<16:35, 1.03it/s] 34%|โโโโ | 539/1565 [08:50<16:34, 1.03it/s] 35%|โโโโ | 540/1565 [08:51<16:35, 1.03it/s] 35%|โโโโ | 541/1565 [08:52<16:33, 1.03it/s] 35%|โโโโ | 542/1565 [08:53<16:31, 1.03it/s] 35%|โโโโ | 543/1565 [08:54<16:29, 1.03it/s] 35%|โโโโ | 544/1565 [08:54<16:29, 1.03it/s] 35%|โโโโ | 545/1565 [08:55<16:30, 1.03it/s] 35%|โโโโ | 546/1565 [08:56<16:31, 1.03it/s] 35%|โโโโ | 547/1565 [08:57<16:28, 1.03it/s] 35%|โโโโ | 548/1565 [08:58<16:27, 1.03it/s] 35%|โโโโ | 549/1565 [08:59<16:27, 1.03it/s] 35%|โโโโ | 550/1565 [09:00<16:28, 1.03it/s] 35%|โโโโ | 551/1565 [09:01<16:29, 1.02it/s] 35%|โโโโ | 552/1565 [09:02<16:29, 1.02it/s] 35%|โโโโ | 553/1565 [09:03<16:43, 1.01it/s] 35%|โโโโ | 554/1565 [09:04<16:38, 1.01it/s] 35%|โโโโ | 555/1565 [09:05<16:32, 1.02it/s] 36%|โโโโ | 556/1565 [09:06<16:27, 1.02it/s] 36%|โโโโ | 557/1565 [09:07<16:24, 1.02it/s] 36%|โโโโ | 558/1565 [09:08<16:22, 1.02it/s] 36%|โโโโ | 559/1565 [09:09<16:20, 1.03it/s] 36%|โโโโ | 560/1565 [09:10<16:20, 1.03it/s] 36%|โโโโ | 561/1565 [09:11<16:21, 1.02it/s] 36%|โโโโ | 562/1565 [09:12<16:22, 1.02it/s] 36%|โโโโ | 563/1565 [09:13<16:20, 1.02it/s] 36%|โโโโ | 564/1565 [09:14<17:49, 1.07s/it] 36%|โโโโ | 565/1565 [09:15<17:21, 1.04s/it] 36%|โโโโ | 566/1565 [09:16<17:00, 1.02s/it] 36%|โโโโ | 567/1565 [09:17<16:44, 1.01s/it] 36%|โโโโ | 568/1565 [09:18<16:34, 1.00it/s] 36%|โโโโ | 569/1565 [09:19<16:30, 1.01it/s] 36%|โโโโ | 570/1565 [09:20<16:24, 1.01it/s] 36%|โโโโ | 571/1565 [09:21<16:20, 1.01it/s] 37%|โโโโ | 572/1565 [09:22<16:16, 1.02it/s] 37%|โโโโ | 573/1565 [09:23<16:10, 1.02it/s] 37%|โโโโ | 574/1565 [09:24<16:06, 1.03it/s] 37%|โโโโ | 575/1565 [09:25<16:04, 1.03it/s] 37%|โโโโ | 576/1565 [09:26<16:04, 1.02it/s] 37%|โโโโ | 577/1565 [09:27<16:19, 1.01it/s] 37%|โโโโ | 578/1565 [09:28<16:13, 1.01it/s] 37%|โโโโ | 579/1565 [09:29<16:11, 1.02it/s] 37%|โโโโ | 580/1565 [09:30<16:06, 1.02it/s] 37%|โโโโ | 581/1565 [09:31<16:02, 1.02it/s] 37%|โโโโ | 582/1565 [09:32<15:59, 1.02it/s] 37%|โโโโ | 583/1565 [09:33<15:59, 1.02it/s] 37%|โโโโ | 584/1565 [09:34<15:56, 1.03it/s] 37%|โโโโ | 585/1565 [09:35<15:54, 1.03it/s] 37%|โโโโ | 586/1565 [09:36<15:51, 1.03it/s] 38%|โโโโ | 587/1565 [09:37<15:50, 1.03it/s] 38%|โโโโ | 588/1565 [09:38<15:50, 1.03it/s] 38%|โโโโ | 589/1565 [09:39<15:51, 1.03it/s] 38%|โโโโ | 590/1565 [09:40<15:52, 1.02it/s] 38%|โโโโ | 591/1565 [09:41<15:49, 1.03it/s] 38%|โโโโ | 592/1565 [09:42<15:47, 1.03it/s] 38%|โโโโ | 593/1565 [09:43<15:45, 1.03it/s] 38%|โโโโ | 594/1565 [09:44<15:43, 1.03it/s] 38%|โโโโ | 595/1565 [09:45<15:42, 1.03it/s] 38%|โโโโ | 596/1565 [09:46<15:42, 1.03it/s] 38%|โโโโ | 597/1565 [09:47<15:43, 1.03it/s] 38%|โโโโ | 598/1565 [09:48<15:41, 1.03it/s] 38%|โโโโ | 599/1565 [09:48<15:39, 1.03it/s] 38%|โโโโ | 600/1565 [09:49<15:37, 1.03it/s] 38%|โโโโ | 601/1565 [09:50<15:37, 1.03it/s] 38%|โโโโ | 602/1565 [09:51<15:36, 1.03it/s] 39%|โโโโ | 603/1565 [09:52<15:35, 1.03it/s] 39%|โโโโ | 604/1565 [09:53<15:33, 1.03it/s] 39%|โโโโ | 605/1565 [09:54<15:31, 1.03it/s] 39%|โโโโ | 606/1565 [09:55<15:31, 1.03it/s] 39%|โโโโ | 607/1565 [09:56<15:30, 1.03it/s] 39%|โโโโ | 608/1565 [09:57<15:28, 1.03it/s] 39%|โโโโ | 609/1565 [09:58<15:27, 1.03it/s] 39%|โโโโ | 610/1565 [09:59<15:25, 1.03it/s] 39%|โโโโ | 611/1565 [10:00<16:38, 1.05s/it] 39%|โโโโ | 612/1565 [10:01<16:14, 1.02s/it] 39%|โโโโ | 613/1565 [10:02<15:59, 1.01s/it] 39%|โโโโ | 614/1565 [10:03<15:47, 1.00it/s] 39%|โโโโ | 615/1565 [10:04<15:40, 1.01it/s] 39%|โโโโ | 616/1565 [10:05<15:35, 1.01it/s] 39%|โโโโ | 617/1565 [10:06<15:31, 1.02it/s] 39%|โโโโ | 618/1565 [10:07<15:28, 1.02it/s] 40%|โโโโ | 619/1565 [10:08<15:24, 1.02it/s] 40%|โโโโ | 620/1565 [10:09<15:22, 1.02it/s] 40%|โโโโ | 621/1565 [10:10<15:20, 1.03it/s] 40%|โโโโ | 622/1565 [10:11<15:20, 1.02it/s] 40%|โโโโ | 623/1565 [10:12<15:20, 1.02it/s] 40%|โโโโ | 624/1565 [10:13<15:18, 1.02it/s] 40%|โโโโ | 625/1565 [10:14<15:06, 1.04it/s] 40%|โโโโ | 626/1565 [10:14<12:51, 1.22it/s] 40%|โโโโ | 627/1565 [10:16<13:57, 1.12it/s] 40%|โโโโ | 628/1565 [10:17<14:19, 1.09it/s] 40%|โโโโ | 629/1565 [10:17<14:32, 1.07it/s] 40%|โโโโ | 630/1565 [10:18<14:41, 1.06it/s] 40%|โโโโ | 631/1565 [10:19<14:49, 1.05it/s] 40%|โโโโ | 632/1565 [10:20<14:52, 1.05it/s] 40%|โโโโ | 633/1565 [10:21<14:55, 1.04it/s] 41%|โโโโ | 634/1565 [10:22<14:57, 1.04it/s] 41%|โโโโ | 635/1565 [10:23<14:59, 1.03it/s] 41%|โโโโ | 636/1565 [10:24<14:58, 1.03it/s] 41%|โโโโ | 637/1565 [10:25<14:57, 1.03it/s] 41%|โโโโ | 638/1565 [10:26<14:57, 1.03it/s] 41%|โโโโ | 639/1565 [10:27<14:56, 1.03it/s] 41%|โโโโ | 640/1565 [10:28<14:54, 1.03it/s] 41%|โโโโ | 641/1565 [10:29<14:54, 1.03it/s] 41%|โโโโ | 642/1565 [10:30<14:55, 1.03it/s] 41%|โโโโ | 643/1565 [10:31<14:55, 1.03it/s] 41%|โโโโ | 644/1565 [10:32<14:56, 1.03it/s] 41%|โโโโ | 645/1565 [10:33<14:54, 1.03it/s] 41%|โโโโโ | 646/1565 [10:34<14:52, 1.03it/s] 41%|โโโโโ | 647/1565 [10:35<14:50, 1.03it/s] 41%|โโโโโ | 648/1565 [10:36<14:49, 1.03it/s] 41%|โโโโโ | 649/1565 [10:37<14:49, 1.03it/s] 42%|โโโโโ | 650/1565 [10:38<14:48, 1.03it/s] 42%|โโโโโ | 651/1565 [10:39<14:48, 1.03it/s] 42%|โโโโโ | 652/1565 [10:40<14:46, 1.03it/s] 42%|โโโโโ | 653/1565 [10:41<14:46, 1.03it/s] 42%|โโโโโ | 654/1565 [10:42<14:44, 1.03it/s] 42%|โโโโโ | 655/1565 [10:43<14:43, 1.03it/s] 42%|โโโโโ | 656/1565 [10:44<14:41, 1.03it/s] 42%|โโโโโ | 657/1565 [10:45<14:43, 1.03it/s] 42%|โโโโโ | 658/1565 [10:46<15:46, 1.04s/it] 42%|โโโโโ | 659/1565 [10:47<15:27, 1.02s/it] 42%|โโโโโ | 660/1565 [10:48<15:12, 1.01s/it] 42%|โโโโโ | 661/1565 [10:49<15:03, 1.00it/s] 42%|โโโโโ | 662/1565 [10:50<14:55, 1.01it/s] 42%|โโโโโ | 663/1565 [10:51<14:49, 1.01it/s] 42%|โโโโโ | 664/1565 [10:52<14:44, 1.02it/s] 42%|โโโโโ | 665/1565 [10:53<14:40, 1.02it/s] 43%|โโโโโ | 666/1565 [10:54<14:38, 1.02it/s] 43%|โโโโโ | 667/1565 [10:55<14:34, 1.03it/s] 43%|โโโโโ | 668/1565 [10:56<14:33, 1.03it/s] 43%|โโโโโ | 669/1565 [10:57<14:32, 1.03it/s] 43%|โโโโโ | 670/1565 [10:58<14:31, 1.03it/s] 43%|โโโโโ | 671/1565 [10:59<14:31, 1.03it/s] 43%|โโโโโ | 672/1565 [10:59<14:29, 1.03it/s] 43%|โโโโโ | 673/1565 [11:00<14:28, 1.03it/s] 43%|โโโโโ | 674/1565 [11:01<14:28, 1.03it/s] 43%|โโโโโ | 675/1565 [11:02<14:26, 1.03it/s] 43%|โโโโโ | 676/1565 [11:03<14:24, 1.03it/s] 43%|โโโโโ | 677/1565 [11:04<14:23, 1.03it/s] 43%|โโโโโ | 678/1565 [11:05<14:23, 1.03it/s] 43%|โโโโโ | 679/1565 [11:06<14:21, 1.03it/s] 43%|โโโโโ | 680/1565 [11:07<14:20, 1.03it/s] 44%|โโโโโ | 681/1565 [11:08<14:20, 1.03it/s] 44%|โโโโโ | 682/1565 [11:09<14:19, 1.03it/s] 44%|โโโโโ | 683/1565 [11:10<14:18, 1.03it/s] 44%|โโโโโ | 684/1565 [11:11<14:19, 1.03it/s] 44%|โโโโโ | 685/1565 [11:12<14:18, 1.03it/s] 44%|โโโโโ | 686/1565 [11:13<14:16, 1.03it/s] 44%|โโโโโ | 687/1565 [11:14<14:15, 1.03it/s] 44%|โโโโโ | 688/1565 [11:15<14:14, 1.03it/s] 44%|โโโโโ | 689/1565 [11:16<14:14, 1.03it/s] 44%|โโโโโ | 690/1565 [11:17<14:13, 1.03it/s] 44%|โโโโโ | 691/1565 [11:18<14:10, 1.03it/s] 44%|โโโโโ | 692/1565 [11:19<14:07, 1.03it/s] 44%|โโโโโ | 693/1565 [11:20<14:06, 1.03it/s] 44%|โโโโโ | 694/1565 [11:21<14:05, 1.03it/s] 44%|โโโโโ | 695/1565 [11:22<14:03, 1.03it/s] 44%|โโโโโ | 696/1565 [11:23<14:01, 1.03it/s] 45%|โโโโโ | 697/1565 [11:24<14:00, 1.03it/s] 45%|โโโโโ | 698/1565 [11:25<14:01, 1.03it/s] 45%|โโโโโ | 699/1565 [11:26<14:00, 1.03it/s] 45%|โโโโโ | 700/1565 [11:27<13:59, 1.03it/s] 45%|โโโโโ | 701/1565 [11:28<13:58, 1.03it/s] 45%|โโโโโ | 702/1565 [11:29<13:58, 1.03it/s] 45%|โโโโโ | 703/1565 [11:30<13:56, 1.03it/s] 45%|โโโโโ | 704/1565 [11:31<13:55, 1.03it/s] 45%|โโโโโ | 705/1565 [11:32<14:54, 1.04s/it] 45%|โโโโโ | 706/1565 [11:33<14:34, 1.02s/it] 45%|โโโโโ | 707/1565 [11:34<14:21, 1.00s/it] 45%|โโโโโ | 708/1565 [11:35<14:13, 1.00it/s] 45%|โโโโโ | 709/1565 [11:36<14:07, 1.01it/s] 45%|โโโโโ | 710/1565 [11:37<14:01, 1.02it/s] 45%|โโโโโ | 711/1565 [11:38<13:56, 1.02it/s] 45%|โโโโโ | 712/1565 [11:39<13:52, 1.02it/s] 46%|โโโโโ | 713/1565 [11:40<13:50, 1.03it/s] 46%|โโโโโ | 714/1565 [11:41<13:49, 1.03it/s] 46%|โโโโโ | 715/1565 [11:42<13:47, 1.03it/s] 46%|โโโโโ | 716/1565 [11:42<13:46, 1.03it/s] 46%|โโโโโ | 717/1565 [11:43<13:45, 1.03it/s] 46%|โโโโโ | 718/1565 [11:44<13:46, 1.02it/s] 46%|โโโโโ | 719/1565 [11:45<13:45, 1.02it/s] 46%|โโโโโ | 720/1565 [11:46<13:42, 1.03it/s] 46%|โโโโโ | 721/1565 [11:47<13:39, 1.03it/s] 46%|โโโโโ | 722/1565 [11:48<13:39, 1.03it/s] 46%|โโโโโ | 723/1565 [11:49<13:37, 1.03it/s] 46%|โโโโโ | 724/1565 [11:50<13:36, 1.03it/s] 46%|โโโโโ | 725/1565 [11:51<13:34, 1.03it/s] 46%|โโโโโ | 726/1565 [11:52<13:35, 1.03it/s] 46%|โโโโโ | 727/1565 [11:53<13:33, 1.03it/s] 47%|โโโโโ | 728/1565 [11:54<13:32, 1.03it/s] 47%|โโโโโ | 729/1565 [11:55<13:30, 1.03it/s] 47%|โโโโโ | 730/1565 [11:56<13:30, 1.03it/s] 47%|โโโโโ | 731/1565 [11:57<13:28, 1.03it/s] 47%|โโโโโ | 732/1565 [11:58<13:27, 1.03it/s] 47%|โโโโโ | 733/1565 [11:59<13:28, 1.03it/s] 47%|โโโโโ | 734/1565 [12:00<13:28, 1.03it/s] 47%|โโโโโ | 735/1565 [12:01<13:27, 1.03it/s] 47%|โโโโโ | 736/1565 [12:02<13:26, 1.03it/s] 47%|โโโโโ | 737/1565 [12:03<13:26, 1.03it/s] 47%|โโโโโ | 738/1565 [12:04<13:26, 1.03it/s] 47%|โโโโโ | 739/1565 [12:05<13:25, 1.03it/s] 47%|โโโโโ | 740/1565 [12:06<13:23, 1.03it/s] 47%|โโโโโ | 741/1565 [12:07<13:22, 1.03it/s] 47%|โโโโโ | 742/1565 [12:08<13:20, 1.03it/s] 47%|โโโโโ | 743/1565 [12:09<13:19, 1.03it/s] 48%|โโโโโ | 744/1565 [12:10<13:17, 1.03it/s] 48%|โโโโโ | 745/1565 [12:11<13:15, 1.03it/s] 48%|โโโโโ | 746/1565 [12:12<13:14, 1.03it/s] 48%|โโโโโ | 747/1565 [12:13<13:14, 1.03it/s] 48%|โโโโโ | 748/1565 [12:14<13:14, 1.03it/s] 48%|โโโโโ | 749/1565 [12:15<13:14, 1.03it/s] 48%|โโโโโ | 750/1565 [12:16<13:14, 1.03it/s] 48%|โโโโโ | 751/1565 [12:17<13:13, 1.03it/s] 48%|โโโโโ | 752/1565 [12:18<14:07, 1.04s/it] 48%|โโโโโ | 753/1565 [12:19<13:49, 1.02s/it] 48%|โโโโโ | 754/1565 [12:20<13:37, 1.01s/it] 48%|โโโโโ | 755/1565 [12:21<13:27, 1.00it/s] 48%|โโโโโ | 756/1565 [12:22<13:21, 1.01it/s] 48%|โโโโโ | 757/1565 [12:23<13:15, 1.02it/s] 48%|โโโโโ | 758/1565 [12:24<13:12, 1.02it/s] 48%|โโโโโ | 759/1565 [12:25<13:09, 1.02it/s] 49%|โโโโโ | 760/1565 [12:26<13:06, 1.02it/s] 49%|โโโโโ | 761/1565 [12:26<13:03, 1.03it/s] 49%|โโโโโ | 762/1565 [12:27<13:02, 1.03it/s] 49%|โโโโโ | 763/1565 [12:28<13:00, 1.03it/s] 49%|โโโโโ | 764/1565 [12:29<12:58, 1.03it/s] 49%|โโโโโ | 765/1565 [12:30<12:57, 1.03it/s] 49%|โโโโโ | 766/1565 [12:31<12:57, 1.03it/s] 49%|โโโโโ | 767/1565 [12:32<12:56, 1.03it/s] 49%|โโโโโ | 768/1565 [12:33<12:55, 1.03it/s] 49%|โโโโโ | 769/1565 [12:34<12:54, 1.03it/s] 49%|โโโโโ | 770/1565 [12:35<12:54, 1.03it/s] 49%|โโโโโ | 771/1565 [12:36<12:53, 1.03it/s] 49%|โโโโโ | 772/1565 [12:37<12:51, 1.03it/s] 49%|โโโโโ | 773/1565 [12:38<12:49, 1.03it/s] 49%|โโโโโ | 774/1565 [12:39<12:48, 1.03it/s] 50%|โโโโโ | 775/1565 [12:40<12:48, 1.03it/s] 50%|โโโโโ | 776/1565 [12:41<12:47, 1.03it/s] 50%|โโโโโ | 777/1565 [12:42<12:48, 1.03it/s] 50%|โโโโโ | 778/1565 [12:43<12:46, 1.03it/s] 50%|โโโโโ | 779/1565 [12:44<12:44, 1.03it/s] 50%|โโโโโ | 780/1565 [12:45<12:42, 1.03it/s] 50%|โโโโโ | 781/1565 [12:46<12:41, 1.03it/s] 50%|โโโโโ | 782/1565 [12:47<12:40, 1.03it/s] 50%|โโโโโ | 783/1565 [12:48<12:39, 1.03it/s] 50%|โโโโโ | 784/1565 [12:49<12:37, 1.03it/s] 50%|โโโโโ | 785/1565 [12:50<12:37, 1.03it/s] 50%|โโโโโ | 786/1565 [12:51<12:38, 1.03it/s] 50%|โโโโโ | 787/1565 [12:52<12:37, 1.03it/s] 50%|โโโโโ | 788/1565 [12:53<12:37, 1.03it/s] 50%|โโโโโ | 789/1565 [12:54<12:35, 1.03it/s] 50%|โโโโโ | 790/1565 [12:55<12:33, 1.03it/s] 51%|โโโโโ | 791/1565 [12:56<12:32, 1.03it/s] 51%|โโโโโ | 792/1565 [12:57<12:32, 1.03it/s] 51%|โโโโโ | 793/1565 [12:58<12:31, 1.03it/s] 51%|โโโโโ | 794/1565 [12:59<12:30, 1.03it/s] 51%|โโโโโ | 795/1565 [13:00<12:28, 1.03it/s] 51%|โโโโโ | 796/1565 [13:01<12:28, 1.03it/s] 51%|โโโโโ | 797/1565 [13:02<12:25, 1.03it/s] 51%|โโโโโ | 798/1565 [13:02<12:24, 1.03it/s] 51%|โโโโโ | 799/1565 [13:04<13:18, 1.04s/it] 51%|โโโโโ | 800/1565 [13:05<13:02, 1.02s/it] 51%|โโโโโ | 801/1565 [13:06<12:50, 1.01s/it] 51%|โโโโโ | 802/1565 [13:07<12:41, 1.00it/s] 51%|โโโโโโ | 803/1565 [13:08<12:32, 1.01it/s] 51%|โโโโโโ | 804/1565 [13:09<12:26, 1.02it/s] 51%|โโโโโโ | 805/1565 [13:10<12:23, 1.02it/s] 52%|โโโโโโ | 806/1565 [13:10<12:21, 1.02it/s] 52%|โโโโโโ | 807/1565 [13:11<12:19, 1.02it/s] 52%|โโโโโโ | 808/1565 [13:12<12:18, 1.03it/s] 52%|โโโโโโ | 809/1565 [13:13<12:17, 1.02it/s] 52%|โโโโโโ | 810/1565 [13:14<12:16, 1.03it/s] 52%|โโโโโโ | 811/1565 [13:15<12:15, 1.03it/s] 52%|โโโโโโ | 812/1565 [13:16<12:14, 1.03it/s] 52%|โโโโโโ | 813/1565 [13:17<12:11, 1.03it/s] 52%|โโโโโโ | 814/1565 [13:18<12:09, 1.03it/s] 52%|โโโโโโ | 815/1565 [13:19<12:07, 1.03it/s] 52%|โโโโโโ | 816/1565 [13:20<12:05, 1.03it/s] 52%|โโโโโโ | 817/1565 [13:21<12:06, 1.03it/s] 52%|โโโโโโ | 818/1565 [13:22<12:06, 1.03it/s] 52%|โโโโโโ | 819/1565 [13:23<12:05, 1.03it/s] 52%|โโโโโโ | 820/1565 [13:24<12:05, 1.03it/s] 52%|โโโโโโ | 821/1565 [13:25<12:05, 1.03it/s] 53%|โโโโโโ | 822/1565 [13:26<12:02, 1.03it/s] 53%|โโโโโโ | 823/1565 [13:27<12:02, 1.03it/s] 53%|โโโโโโ | 824/1565 [13:28<12:00, 1.03it/s] 53%|โโโโโโ | 825/1565 [13:29<11:59, 1.03it/s] 53%|โโโโโโ | 826/1565 [13:30<11:59, 1.03it/s] 53%|โโโโโโ | 827/1565 [13:31<11:58, 1.03it/s] 53%|โโโโโโ | 828/1565 [13:32<11:57, 1.03it/s] 53%|โโโโโโ | 829/1565 [13:33<11:56, 1.03it/s] 53%|โโโโโโ | 830/1565 [13:34<11:55, 1.03it/s] 53%|โโโโโโ | 831/1565 [13:35<11:55, 1.03it/s] 53%|โโโโโโ | 832/1565 [13:36<11:54, 1.03it/s] 53%|โโโโโโ | 833/1565 [13:37<11:52, 1.03it/s] 53%|โโโโโโ | 834/1565 [13:38<11:53, 1.02it/s] 53%|โโโโโโ | 835/1565 [13:39<11:51, 1.03it/s] 53%|โโโโโโ | 836/1565 [13:40<11:50, 1.03it/s] 53%|โโโโโโ | 837/1565 [13:41<11:48, 1.03it/s] 54%|โโโโโโ | 838/1565 [13:42<11:47, 1.03it/s] 54%|โโโโโโ | 839/1565 [13:43<11:46, 1.03it/s] 54%|โโโโโโ | 840/1565 [13:44<11:46, 1.03it/s] 54%|โโโโโโ | 841/1565 [13:45<11:45, 1.03it/s] 54%|โโโโโโ | 842/1565 [13:46<11:44, 1.03it/s] 54%|โโโโโโ | 843/1565 [13:47<11:43, 1.03it/s] 54%|โโโโโโ | 844/1565 [13:47<11:41, 1.03it/s] 54%|โโโโโโ | 845/1565 [13:48<11:40, 1.03it/s] 54%|โโโโโโ | 846/1565 [13:50<12:28, 1.04s/it] 54%|โโโโโโ | 847/1565 [13:51<12:13, 1.02s/it] 54%|โโโโโโ | 848/1565 [13:52<12:02, 1.01s/it] 54%|โโโโโโ | 849/1565 [13:53<11:54, 1.00it/s] 54%|โโโโโโ | 850/1565 [13:54<11:47, 1.01it/s] 54%|โโโโโโ | 851/1565 [13:55<11:43, 1.01it/s] 54%|โโโโโโ | 852/1565 [13:55<11:39, 1.02it/s] 55%|โโโโโโ | 853/1565 [13:56<11:36, 1.02it/s] 55%|โโโโโโ | 854/1565 [13:57<11:33, 1.02it/s] 55%|โโโโโโ | 855/1565 [13:58<11:32, 1.03it/s] 55%|โโโโโโ | 856/1565 [13:59<11:31, 1.03it/s] 55%|โโโโโโ | 857/1565 [14:00<11:30, 1.03it/s] 55%|โโโโโโ | 858/1565 [14:01<11:29, 1.03it/s] 55%|โโโโโโ | 859/1565 [14:02<11:27, 1.03it/s] 55%|โโโโโโ | 860/1565 [14:03<11:26, 1.03it/s] 55%|โโโโโโ | 861/1565 [14:04<11:25, 1.03it/s] 55%|โโโโโโ | 862/1565 [14:05<11:24, 1.03it/s] 55%|โโโโโโ | 863/1565 [14:06<11:23, 1.03it/s] 55%|โโโโโโ | 864/1565 [14:07<11:22, 1.03it/s] 55%|โโโโโโ | 865/1565 [14:08<11:22, 1.03it/s] 55%|โโโโโโ | 866/1565 [14:09<11:21, 1.03it/s] 55%|โโโโโโ | 867/1565 [14:10<11:20, 1.03it/s] 55%|โโโโโโ | 868/1565 [14:11<11:18, 1.03it/s] 56%|โโโโโโ | 869/1565 [14:12<11:17, 1.03it/s] 56%|โโโโโโ | 870/1565 [14:13<11:16, 1.03it/s] 56%|โโโโโโ | 871/1565 [14:14<11:14, 1.03it/s] 56%|โโโโโโ | 872/1565 [14:15<11:14, 1.03it/s] 56%|โโโโโโ | 873/1565 [14:16<11:13, 1.03it/s] 56%|โโโโโโ | 874/1565 [14:17<11:13, 1.03it/s] 56%|โโโโโโ | 875/1565 [14:18<11:12, 1.03it/s] 56%|โโโโโโ | 876/1565 [14:19<11:11, 1.03it/s] 56%|โโโโโโ | 877/1565 [14:20<11:10, 1.03it/s] 56%|โโโโโโ | 878/1565 [14:21<11:08, 1.03it/s] 56%|โโโโโโ | 879/1565 [14:22<11:08, 1.03it/s] 56%|โโโโโโ | 880/1565 [14:23<11:07, 1.03it/s] 56%|โโโโโโ | 881/1565 [14:24<11:06, 1.03it/s] 56%|โโโโโโ | 882/1565 [14:25<11:05, 1.03it/s] 56%|โโโโโโ | 883/1565 [14:26<11:04, 1.03it/s] 56%|โโโโโโ | 884/1565 [14:27<11:03, 1.03it/s] 57%|โโโโโโ | 885/1565 [14:28<11:01, 1.03it/s] 57%|โโโโโโ | 886/1565 [14:29<10:59, 1.03it/s] 57%|โโโโโโ | 887/1565 [14:30<10:57, 1.03it/s] 57%|โโโโโโ | 888/1565 [14:31<10:56, 1.03it/s] 57%|โโโโโโ | 889/1565 [14:31<10:56, 1.03it/s] 57%|โโโโโโ | 890/1565 [14:32<10:56, 1.03it/s] 57%|โโโโโโ | 891/1565 [14:33<10:56, 1.03it/s] 57%|โโโโโโ | 892/1565 [14:34<10:56, 1.03it/s] 57%|โโโโโโ | 893/1565 [14:36<11:41, 1.04s/it] 57%|โโโโโโ | 894/1565 [14:37<11:27, 1.02s/it] 57%|โโโโโโ | 895/1565 [14:38<11:14, 1.01s/it] 57%|โโโโโโ | 896/1565 [14:39<11:06, 1.00it/s] 57%|โโโโโโ | 897/1565 [14:40<11:01, 1.01it/s] 57%|โโโโโโ | 898/1565 [14:40<10:56, 1.02it/s] 57%|โโโโโโ | 899/1565 [14:41<10:53, 1.02it/s] 58%|โโโโโโ | 900/1565 [14:42<10:50, 1.02it/s] 58%|โโโโโโ | 901/1565 [14:43<10:47, 1.02it/s] 58%|โโโโโโ | 902/1565 [14:44<10:45, 1.03it/s] 58%|โโโโโโ | 903/1565 [14:45<10:43, 1.03it/s] 58%|โโโโโโ | 904/1565 [14:46<10:42, 1.03it/s] 58%|โโโโโโ | 905/1565 [14:47<10:39, 1.03it/s] 58%|โโโโโโ | 906/1565 [14:48<10:38, 1.03it/s] 58%|โโโโโโ | 907/1565 [14:49<10:37, 1.03it/s] 58%|โโโโโโ | 908/1565 [14:50<10:37, 1.03it/s] 58%|โโโโโโ | 909/1565 [14:51<10:37, 1.03it/s] 58%|โโโโโโ | 910/1565 [14:52<10:36, 1.03it/s] 58%|โโโโโโ | 911/1565 [14:53<10:35, 1.03it/s] 58%|โโโโโโ | 912/1565 [14:54<10:34, 1.03it/s] 58%|โโโโโโ | 913/1565 [14:55<10:32, 1.03it/s] 58%|โโโโโโ | 914/1565 [14:56<10:31, 1.03it/s] 58%|โโโโโโ | 915/1565 [14:57<10:31, 1.03it/s] 59%|โโโโโโ | 916/1565 [14:58<10:30, 1.03it/s] 59%|โโโโโโ | 917/1565 [14:59<10:29, 1.03it/s] 59%|โโโโโโ | 918/1565 [15:00<10:27, 1.03it/s] 59%|โโโโโโ | 919/1565 [15:01<10:27, 1.03it/s] 59%|โโโโโโ | 920/1565 [15:02<10:25, 1.03it/s] 59%|โโโโโโ | 921/1565 [15:03<10:24, 1.03it/s] 59%|โโโโโโ | 922/1565 [15:04<10:24, 1.03it/s] 59%|โโโโโโ | 923/1565 [15:05<10:23, 1.03it/s] 59%|โโโโโโ | 924/1565 [15:06<10:23, 1.03it/s] 59%|โโโโโโ | 925/1565 [15:07<10:23, 1.03it/s] 59%|โโโโโโ | 926/1565 [15:08<10:21, 1.03it/s] 59%|โโโโโโ | 927/1565 [15:09<10:21, 1.03it/s] 59%|โโโโโโ | 928/1565 [15:10<10:20, 1.03it/s] 59%|โโโโโโ | 929/1565 [15:11<10:20, 1.02it/s] 59%|โโโโโโ | 930/1565 [15:12<10:18, 1.03it/s] 59%|โโโโโโ | 931/1565 [15:13<10:16, 1.03it/s] 60%|โโโโโโ | 932/1565 [15:14<10:16, 1.03it/s] 60%|โโโโโโ | 933/1565 [15:15<10:15, 1.03it/s] 60%|โโโโโโ | 934/1565 [15:15<10:15, 1.03it/s] 60%|โโโโโโ | 935/1565 [15:16<10:13, 1.03it/s] 60%|โโโโโโ | 936/1565 [15:17<10:12, 1.03it/s] 60%|โโโโโโ | 937/1565 [15:18<10:11, 1.03it/s] 60%|โโโโโโ | 938/1565 [15:19<10:04, 1.04it/s] 60%|โโโโโโ | 939/1565 [15:20<08:33, 1.22it/s] 60%|โโโโโโ | 940/1565 [15:21<10:00, 1.04it/s] 60%|โโโโโโ | 941/1565 [15:22<10:02, 1.04it/s] 60%|โโโโโโ | 942/1565 [15:23<10:03, 1.03it/s] 60%|โโโโโโ | 943/1565 [15:24<10:02, 1.03it/s] 60%|โโโโโโ | 944/1565 [15:25<10:02, 1.03it/s] 60%|โโโโโโ | 945/1565 [15:26<10:01, 1.03it/s] 60%|โโโโโโ | 946/1565 [15:27<10:01, 1.03it/s] 61%|โโโโโโ | 947/1565 [15:28<09:59, 1.03it/s] 61%|โโโโโโ | 948/1565 [15:29<09:59, 1.03it/s] 61%|โโโโโโ | 949/1565 [15:30<09:58, 1.03it/s] 61%|โโโโโโ | 950/1565 [15:31<09:57, 1.03it/s] 61%|โโโโโโ | 951/1565 [15:32<09:57, 1.03it/s] 61%|โโโโโโ | 952/1565 [15:33<09:56, 1.03it/s] 61%|โโโโโโ | 953/1565 [15:34<09:55, 1.03it/s] 61%|โโโโโโ | 954/1565 [15:35<09:54, 1.03it/s] 61%|โโโโโโ | 955/1565 [15:36<09:53, 1.03it/s] 61%|โโโโโโ | 956/1565 [15:37<09:52, 1.03it/s] 61%|โโโโโโ | 957/1565 [15:38<09:51, 1.03it/s] 61%|โโโโโโ | 958/1565 [15:39<09:50, 1.03it/s] 61%|โโโโโโโ | 959/1565 [15:40<09:49, 1.03it/s] 61%|โโโโโโโ | 960/1565 [15:41<09:48, 1.03it/s] 61%|โโโโโโโ | 961/1565 [15:42<09:48, 1.03it/s] 61%|โโโโโโโ | 962/1565 [15:43<09:46, 1.03it/s] 62%|โโโโโโโ | 963/1565 [15:43<09:45, 1.03it/s] 62%|โโโโโโโ | 964/1565 [15:44<09:44, 1.03it/s] 62%|โโโโโโโ | 965/1565 [15:45<09:42, 1.03it/s] 62%|โโโโโโโ | 966/1565 [15:46<09:40, 1.03it/s] 62%|โโโโโโโ | 967/1565 [15:47<09:38, 1.03it/s] 62%|โโโโโโโ | 968/1565 [15:48<09:37, 1.03it/s] 62%|โโโโโโโ | 969/1565 [15:49<09:36, 1.03it/s] 62%|โโโโโโโ | 970/1565 [15:50<09:36, 1.03it/s] 62%|โโโโโโโ | 971/1565 [15:51<09:36, 1.03it/s] 62%|โโโโโโโ | 972/1565 [15:52<09:34, 1.03it/s] 62%|โโโโโโโ | 973/1565 [15:53<09:34, 1.03it/s] 62%|โโโโโโโ | 974/1565 [15:54<09:34, 1.03it/s] 62%|โโโโโโโ | 975/1565 [15:55<09:32, 1.03it/s] 62%|โโโโโโโ | 976/1565 [15:56<09:30, 1.03it/s] 62%|โโโโโโโ | 977/1565 [15:57<09:30, 1.03it/s] 62%|โโโโโโโ | 978/1565 [15:58<09:30, 1.03it/s] 63%|โโโโโโโ | 979/1565 [15:59<09:29, 1.03it/s] 63%|โโโโโโโ | 980/1565 [16:00<09:27, 1.03it/s] 63%|โโโโโโโ | 981/1565 [16:01<09:26, 1.03it/s] 63%|โโโโโโโ | 982/1565 [16:02<09:25, 1.03it/s] 63%|โโโโโโโ | 983/1565 [16:03<09:25, 1.03it/s] 63%|โโโโโโโ | 984/1565 [16:04<09:24, 1.03it/s] 63%|โโโโโโโ | 985/1565 [16:05<09:23, 1.03it/s] 63%|โโโโโโโ | 986/1565 [16:06<09:22, 1.03it/s] 63%|โโโโโโโ | 987/1565 [16:07<10:01, 1.04s/it] 63%|โโโโโโโ | 988/1565 [16:08<09:48, 1.02s/it] 63%|โโโโโโโ | 989/1565 [16:09<09:40, 1.01s/it] 63%|โโโโโโโ | 990/1565 [16:10<09:34, 1.00it/s] 63%|โโโโโโโ | 991/1565 [16:11<09:29, 1.01it/s] 63%|โโโโโโโ | 992/1565 [16:12<09:24, 1.01it/s] 63%|โโโโโโโ | 993/1565 [16:13<09:20, 1.02it/s] 64%|โโโโโโโ | 994/1565 [16:14<09:18, 1.02it/s] 64%|โโโโโโโ | 995/1565 [16:15<09:16, 1.02it/s] 64%|โโโโโโโ | 996/1565 [16:16<09:15, 1.02it/s] 64%|โโโโโโโ | 997/1565 [16:17<09:13, 1.03it/s] 64%|โโโโโโโ | 998/1565 [16:18<09:11, 1.03it/s] 64%|โโโโโโโ | 999/1565 [16:19<09:11, 1.03it/s] 64%|โโโโโโโ | 1000/1565 [16:20<09:10, 1.03it/s] 64%|โโโโโโโ | 1000/1565 [16:20<09:10, 1.03it/s]{'loss': 0.5945, 'grad_norm': 5.812021732330322, 'learning_rate': 1.9007625957379082e-06, 'epoch': 3.19} | |
| 0%| | 0/79 [00:00<?, ?it/s][A | |
| 3%|โ | 2/79 [00:00<00:07, 10.44it/s][A | |
| 5%|โ | 4/79 [00:00<00:11, 6.57it/s][A | |
| 6%|โ | 5/79 [00:00<00:12, 6.12it/s][A | |
| 8%|โ | 6/79 [00:00<00:12, 5.83it/s][A | |
| 9%|โ | 7/79 [00:01<00:12, 5.65it/s][A | |
| 10%|โ | 8/79 [00:01<00:12, 5.53it/s][A | |
| 11%|โโ | 9/79 [00:01<00:12, 5.44it/s][A | |
| 13%|โโ | 10/79 [00:01<00:12, 5.37it/s][A | |
| 14%|โโ | 11/79 [00:01<00:12, 5.33it/s][A | |
| 15%|โโ | 12/79 [00:02<00:12, 5.29it/s][A | |
| 16%|โโ | 13/79 [00:02<00:12, 5.28it/s][A | |
| 18%|โโ | 14/79 [00:02<00:12, 5.26it/s][A | |
| 19%|โโ | 15/79 [00:02<00:12, 5.25it/s][A | |
| 20%|โโ | 16/79 [00:02<00:12, 5.24it/s][A | |
| 22%|โโโ | 17/79 [00:03<00:11, 5.24it/s][A | |
| 23%|โโโ | 18/79 [00:03<00:11, 5.22it/s][A | |
| 24%|โโโ | 19/79 [00:03<00:11, 5.23it/s][A | |
| 25%|โโโ | 20/79 [00:03<00:11, 5.22it/s][A | |
| 27%|โโโ | 21/79 [00:03<00:11, 5.20it/s][A | |
| 28%|โโโ | 22/79 [00:04<00:10, 5.22it/s][A | |
| 29%|โโโ | 23/79 [00:04<00:10, 5.24it/s][A | |
| 30%|โโโ | 24/79 [00:04<00:10, 5.22it/s][A | |
| 32%|โโโโ | 25/79 [00:04<00:10, 5.22it/s][A | |
| 33%|โโโโ | 26/79 [00:04<00:10, 5.21it/s][A | |
| 34%|โโโโ | 27/79 [00:04<00:10, 5.20it/s][A | |
| 35%|โโโโ | 28/79 [00:05<00:09, 5.19it/s][A | |
| 37%|โโโโ | 29/79 [00:05<00:09, 5.19it/s][A | |
| 38%|โโโโ | 30/79 [00:05<00:09, 5.22it/s][A | |
| 39%|โโโโ | 31/79 [00:05<00:09, 5.24it/s][A | |
| 41%|โโโโ | 32/79 [00:05<00:08, 5.24it/s][A | |
| 42%|โโโโโ | 33/79 [00:06<00:08, 5.24it/s][A | |
| 43%|โโโโโ | 34/79 [00:06<00:08, 5.25it/s][A | |
| 44%|โโโโโ | 35/79 [00:06<00:08, 5.25it/s][A | |
| 46%|โโโโโ | 36/79 [00:06<00:08, 5.24it/s][A | |
| 47%|โโโโโ | 37/79 [00:06<00:07, 5.25it/s][A | |
| 48%|โโโโโ | 38/79 [00:07<00:07, 5.24it/s][A | |
| 49%|โโโโโ | 39/79 [00:07<00:07, 5.23it/s][A | |
| 51%|โโโโโ | 40/79 [00:07<00:10, 3.85it/s][A | |
| 52%|โโโโโโ | 41/79 [00:07<00:09, 4.18it/s][A | |
| 53%|โโโโโโ | 42/79 [00:08<00:08, 4.45it/s][A | |
| 54%|โโโโโโ | 43/79 [00:08<00:07, 4.67it/s][A | |
| 56%|โโโโโโ | 44/79 [00:08<00:07, 4.83it/s][A | |
| 57%|โโโโโโ | 45/79 [00:08<00:06, 4.94it/s][A | |
| 58%|โโโโโโ | 46/79 [00:08<00:06, 5.02it/s][A | |
| 59%|โโโโโโ | 47/79 [00:09<00:06, 5.09it/s][A | |
| 61%|โโโโโโ | 48/79 [00:09<00:06, 5.13it/s][A | |
| 62%|โโโโโโโ | 49/79 [00:09<00:05, 5.18it/s][A | |
| 63%|โโโโโโโ | 50/79 [00:09<00:05, 5.20it/s][A | |
| 65%|โโโโโโโ | 51/79 [00:09<00:05, 5.19it/s][A | |
| 66%|โโโโโโโ | 52/79 [00:09<00:05, 5.19it/s][A | |
| 67%|โโโโโโโ | 53/79 [00:10<00:05, 5.18it/s][A | |
| 68%|โโโโโโโ | 54/79 [00:10<00:04, 5.19it/s][A | |
| 70%|โโโโโโโ | 55/79 [00:10<00:04, 5.18it/s][A | |
| 71%|โโโโโโโ | 56/79 [00:10<00:04, 5.21it/s][A | |
| 72%|โโโโโโโโ | 57/79 [00:10<00:04, 5.22it/s][A | |
| 73%|โโโโโโโโ | 58/79 [00:11<00:04, 5.23it/s][A | |
| 75%|โโโโโโโโ | 59/79 [00:11<00:03, 5.23it/s][A | |
| 76%|โโโโโโโโ | 60/79 [00:11<00:03, 5.23it/s][A | |
| 77%|โโโโโโโโ | 61/79 [00:11<00:03, 5.24it/s][A | |
| 78%|โโโโโโโโ | 62/79 [00:11<00:03, 5.25it/s][A | |
| 80%|โโโโโโโโ | 63/79 [00:12<00:03, 5.25it/s][A | |
| 81%|โโโโโโโโ | 64/79 [00:12<00:02, 5.25it/s][A | |
| 82%|โโโโโโโโโ | 65/79 [00:12<00:02, 5.22it/s][A | |
| 84%|โโโโโโโโโ | 66/79 [00:12<00:02, 5.22it/s][A | |
| 85%|โโโโโโโโโ | 67/79 [00:12<00:02, 5.21it/s][A | |
| 86%|โโโโโโโโโ | 68/79 [00:13<00:02, 5.20it/s][A | |
| 87%|โโโโโโโโโ | 69/79 [00:13<00:01, 5.20it/s][A | |
| 89%|โโโโโโโโโ | 70/79 [00:13<00:01, 5.20it/s][A | |
| 90%|โโโโโโโโโ | 71/79 [00:13<00:01, 5.20it/s][A | |
| 91%|โโโโโโโโโ | 72/79 [00:13<00:01, 5.20it/s][A | |
| 92%|โโโโโโโโโโ| 73/79 [00:14<00:01, 5.21it/s][A | |
| 94%|โโโโโโโโโโ| 74/79 [00:14<00:00, 5.22it/s][A | |
| 95%|โโโโโโโโโโ| 75/79 [00:14<00:00, 5.23it/s][A | |
| 96%|โโโโโโโโโโ| 76/79 [00:14<00:00, 5.23it/s][A | |
| 97%|โโโโโโโโโโ| 77/79 [00:14<00:00, 5.24it/s][A | |
| 99%|โโโโโโโโโโ| 78/79 [00:14<00:00, 5.50it/s][A[W 2025-11-26 12:22:27,446] Trial 0 failed with parameters: {'learning_rate': 5.2556421595933325e-06, 'num_train_epochs': 5, 'seed': 12, 'per_device_train_batch_size': 32} because of the following error: TypeError("'float' object does not support item assignment"). | |
| Traceback (most recent call last): | |
| File "/tmp/conda/wzh/lib/python3.11/site-packages/optuna/study/_optimize.py", line 205, in _run_trial | |
| value_or_values = func(trial) | |
| ^^^^^^^^^^^ | |
| File "/tmp/conda/wzh/lib/python3.11/site-packages/transformers/integrations/integration_utils.py", line 277, in _objective | |
| trainer.train(resume_from_checkpoint=checkpoint, trial=trial) | |
| File "/tmp/conda/wzh/lib/python3.11/site-packages/transformers/trainer.py", line 2325, in train | |
| return inner_training_loop( | |
| ^^^^^^^^^^^^^^^^^^^^ | |
| File "/tmp/conda/wzh/lib/python3.11/site-packages/transformers/trainer.py", line 2756, in _inner_training_loop | |
| self._maybe_log_save_evaluate( | |
| File "/tmp/conda/wzh/lib/python3.11/site-packages/transformers/trainer.py", line 3221, in _maybe_log_save_evaluate | |
| metrics = self._evaluate(trial, ignore_keys_for_eval) | |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| File "/tmp/conda/wzh/lib/python3.11/site-packages/transformers/trainer.py", line 3170, in _evaluate | |
| metrics = self.evaluate(ignore_keys=ignore_keys_for_eval) | |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| File "/tmp/conda/wzh/lib/python3.11/site-packages/transformers/trainer.py", line 4489, in evaluate | |
| output = eval_loop( | |
| ^^^^^^^^^^ | |
| File "/tmp/conda/wzh/lib/python3.11/site-packages/transformers/trainer.py", line 4792, in evaluation_loop | |
| metrics[f"{metric_key_prefix}_loss"] = all_losses.mean().item() | |
| ~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| TypeError: 'float' object does not support item assignment | |
| [W 2025-11-26 12:22:27,448] Trial 0 failed with value None. | |
| Traceback (most recent call last): | |
| File "/tmp/home/wzh/final/train.py", line 160, in <module> | |
| best_run = trainer.hyperparameter_search( | |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| File "/tmp/conda/wzh/lib/python3.11/site-packages/transformers/trainer.py", line 3767, in hyperparameter_search | |
| best_run = backend_obj.run(self, n_trials, direction, **kwargs) | |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| File "/tmp/conda/wzh/lib/python3.11/site-packages/transformers/hyperparameter_search.py", line 72, in run | |
| return run_hp_search_optuna(trainer, n_trials, direction, **kwargs) | |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| File "/tmp/conda/wzh/lib/python3.11/site-packages/transformers/integrations/integration_utils.py", line 295, in run_hp_search_optuna | |
| study.optimize(_objective, n_trials=n_trials, timeout=timeout, n_jobs=n_jobs, gc_after_trial=gc_after_trial) | |
| File "/tmp/conda/wzh/lib/python3.11/site-packages/optuna/study/study.py", line 490, in optimize | |
| _optimize( | |
| File "/tmp/conda/wzh/lib/python3.11/site-packages/optuna/study/_optimize.py", line 67, in _optimize | |
| _optimize_sequential( | |
| File "/tmp/conda/wzh/lib/python3.11/site-packages/optuna/study/_optimize.py", line 164, in _optimize_sequential | |
| frozen_trial_id = _run_trial(study, func, catch) | |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| File "/tmp/conda/wzh/lib/python3.11/site-packages/optuna/study/_optimize.py", line 262, in _run_trial | |
| raise func_err | |
| File "/tmp/conda/wzh/lib/python3.11/site-packages/optuna/study/_optimize.py", line 205, in _run_trial | |
| value_or_values = func(trial) | |
| ^^^^^^^^^^^ | |
| File "/tmp/conda/wzh/lib/python3.11/site-packages/transformers/integrations/integration_utils.py", line 277, in _objective | |
| trainer.train(resume_from_checkpoint=checkpoint, trial=trial) | |
| File "/tmp/conda/wzh/lib/python3.11/site-packages/transformers/trainer.py", line 2325, in train | |
| return inner_training_loop( | |
| ^^^^^^^^^^^^^^^^^^^^ | |
| File "/tmp/conda/wzh/lib/python3.11/site-packages/transformers/trainer.py", line 2756, in _inner_training_loop | |
| self._maybe_log_save_evaluate( | |
| File "/tmp/conda/wzh/lib/python3.11/site-packages/transformers/trainer.py", line 3221, in _maybe_log_save_evaluate | |
| metrics = self._evaluate(trial, ignore_keys_for_eval) | |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| File "/tmp/conda/wzh/lib/python3.11/site-packages/transformers/trainer.py", line 3170, in _evaluate | |
| metrics = self.evaluate(ignore_keys=ignore_keys_for_eval) | |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| File "/tmp/conda/wzh/lib/python3.11/site-packages/transformers/trainer.py", line 4489, in evaluate | |
| output = eval_loop( | |
| ^^^^^^^^^^ | |
| File "/tmp/conda/wzh/lib/python3.11/site-packages/transformers/trainer.py", line 4792, in evaluation_loop | |
| metrics[f"{metric_key_prefix}_loss"] = all_losses.mean().item() | |
| ~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| TypeError: 'float' object does not support item assignment | |
| 64%|โโโโโโโ | 1000/1565 [16:35<09:22, 1.00it/s] | |
| [A |