File size: 26,356 Bytes
8717f59 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 | 2026-07-24:00:35:38 INFO [_cli.run:388] Selected Tasks: ['humaneval', 'mbpp']
2026-07-24:00:35:40 INFO [evaluator:214] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2026-07-24:00:35:40 INFO [evaluator:239] Initializing local-completions model, with arguments: {'model': 'student', 'base_url': 'http://127.0.0.1:8399/v1/completions', 'tokenizer': 'outputs/qwen35_pruned_keep25', 'num_concurrent': 64, 'tokenized_requests': False, 'max_retries': 3}
2026-07-24:00:35:40 INFO [models.openai_completions:42] Remote tokenizer not supported. Using huggingface tokenizer backend.
2026-07-24:00:35:40 INFO [models.api_models:179] Using max length 2048 - 1
2026-07-24:00:35:40 INFO [models.api_models:200] Using tokenizer huggingface
2026-07-24:00:35:48 INFO [evaluator_utils:446] Selected tasks:
2026-07-24:00:35:48 INFO [evaluator_utils:480] Task: humaneval (humaneval/humaneval.yaml)
2026-07-24:00:35:48 INFO [evaluator_utils:480] Task: mbpp (mbpp/mbpp.yaml)
2026-07-24:00:35:48 INFO [evaluator:314] humaneval: Using gen_kwargs: {'until': ['\nclass', '\ndef', '\n#', '\nif', '\nprint'], 'max_gen_toks': 1024, 'do_sample': False}
2026-07-24:00:35:48 INFO [evaluator:314] mbpp: Using gen_kwargs: {'until': ['[DONE]'], 'do_sample': False}
2026-07-24:00:35:48 INFO [api.task:312] Building contexts for humaneval on rank 0...
0%| | 0/164 [00:00<?, ?it/s]
87%|████████▋ | 143/164 [00:00<00:00, 1428.35it/s]
100%|██████████| 164/164 [00:00<00:00, 1462.23it/s]
2026-07-24:00:35:48 INFO [api.task:312] Building contexts for mbpp on rank 0...
0%| | 0/500 [00:00<?, ?it/s]
3%|▎ | 17/500 [00:00<00:03, 160.81it/s]
7%|▋ | 34/500 [00:00<00:02, 161.91it/s]
10%|█ | 51/500 [00:00<00:02, 162.43it/s]
14%|█▎ | 68/500 [00:00<00:02, 162.82it/s]
17%|█▋ | 85/500 [00:00<00:02, 163.22it/s]
20%|██ | 102/500 [00:00<00:02, 163.52it/s]
24%|██▍ | 119/500 [00:00<00:02, 163.70it/s]
27%|██▋ | 136/500 [00:00<00:02, 163.71it/s]
31%|███ | 153/500 [00:00<00:02, 163.69it/s]
34%|███▍ | 170/500 [00:01<00:02, 163.72it/s]
37%|███▋ | 187/500 [00:01<00:01, 163.77it/s]
41%|████ | 204/500 [00:01<00:01, 163.98it/s]
44%|████▍ | 221/500 [00:01<00:01, 164.08it/s]
48%|████▊ | 238/500 [00:01<00:01, 164.17it/s]
51%|█████ | 255/500 [00:01<00:01, 164.21it/s]
54%|█████▍ | 272/500 [00:01<00:01, 164.34it/s]
58%|█████▊ | 289/500 [00:01<00:01, 164.33it/s]
61%|██████ | 306/500 [00:01<00:01, 164.27it/s]
65%|██████▍ | 323/500 [00:01<00:01, 164.43it/s]
68%|██████▊ | 340/500 [00:02<00:00, 164.47it/s]
71%|███████▏ | 357/500 [00:02<00:00, 164.55it/s]
75%|███████▍ | 374/500 [00:02<00:00, 164.72it/s]
78%|███████▊ | 391/500 [00:02<00:00, 164.66it/s]
82%|████████▏ | 408/500 [00:02<00:00, 164.76it/s]
85%|████████▌ | 425/500 [00:02<00:00, 164.85it/s]
88%|████████▊ | 442/500 [00:02<00:00, 165.03it/s]
92%|█████████▏| 459/500 [00:02<00:00, 164.93it/s]
95%|█████████▌| 476/500 [00:02<00:00, 164.76it/s]
99%|█████████▊| 493/500 [00:03<00:00, 164.86it/s]
100%|██████████| 500/500 [00:03<00:00, 164.18it/s]
2026-07-24:00:35:51 INFO [evaluator:585] Running generate_until requests
2026-07-24:00:35:51 INFO [models.api_models:747] Tokenized requests are disabled. Context + generation length is not checked.
Requesting API: 0%| | 0/164 [00:00<?, ?it/s]
Requesting API: 1%| | 1/164 [00:03<08:19, 3.07s/it]
Requesting API: 1%| | 2/164 [00:04<04:54, 1.82s/it]
Requesting API: 2%|▏ | 3/164 [00:11<12:25, 4.63s/it]
Requesting API: 2%|▏ | 4/164 [01:23<1:23:07, 31.17s/it]
Requesting API: 3%|▎ | 5/164 [01:35<1:03:57, 24.14s/it]
Requesting API: 4%|▎ | 6/164 [01:35<42:07, 15.99s/it]
Requesting API: 7%|▋ | 12/164 [01:35<10:23, 4.10s/it]
Requesting API: 17%|█▋ | 28/164 [01:36<02:27, 1.08s/it]
Requesting API: 23%|██▎ | 38/164 [01:36<01:23, 1.51it/s]
Requesting API: 29%|██▉ | 48/164 [01:36<00:49, 2.33it/s]
Requesting API: 37%|███▋ | 61/164 [01:36<00:27, 3.81it/s]
Requesting API: 40%|████ | 66/164 [01:38<00:28, 3.43it/s]
Requesting API: 40%|████ | 66/164 [01:53<00:28, 3.43it/s]
Requesting API: 42%|████▏ | 69/164 [02:59<06:07, 3.87s/it]
Requesting API: 43%|████▎ | 70/164 [03:00<05:48, 3.71s/it]
Requesting API: 45%|████▍ | 73/164 [03:11<05:35, 3.68s/it]
Requesting API: 48%|████▊ | 78/164 [03:11<03:33, 2.48s/it]
Requesting API: 58%|█████▊ | 95/164 [03:11<01:06, 1.03it/s]
Requesting API: 63%|██████▎ | 104/164 [03:12<00:39, 1.50it/s]
Requesting API: 68%|██████▊ | 112/164 [03:12<00:25, 2.08it/s]
Requesting API: 76%|███████▌ | 125/164 [03:12<00:11, 3.42it/s]
Requesting API: 80%|████████ | 132/164 [03:22<00:17, 1.81it/s]
Requesting API: 80%|████████ | 132/164 [03:33<00:17, 1.81it/s]
Requesting API: 81%|████████ | 133/164 [04:34<01:58, 3.83s/it]
Requesting API: 82%|████████▏ | 134/164 [04:35<01:49, 3.66s/it]
Requesting API: 84%|████████▍ | 138/164 [04:46<01:28, 3.40s/it]
Requesting API: 91%|█████████▏| 150/164 [04:46<00:22, 1.62s/it]
Requesting API: 100%|██████████| 164/164 [04:46<00:00, 1.75s/it]
Requesting API: 0%| | 0/500 [00:00<?, ?it/s]
Requesting API: 0%| | 1/500 [00:07<1:02:58, 7.57s/it]
Requesting API: 0%| | 2/500 [00:07<27:47, 3.35s/it]
Requesting API: 1%| | 3/500 [00:08<16:08, 1.95s/it]
Requesting API: 1%| | 5/500 [00:08<07:36, 1.08it/s]
Requesting API: 1%| | 6/500 [00:08<06:01, 1.37it/s]
Requesting API: 2%|▏ | 8/500 [00:09<03:46, 2.17it/s]
Requesting API: 2%|▏ | 9/500 [00:09<03:12, 2.55it/s]
Requesting API: 2%|▏ | 12/500 [00:09<02:28, 3.29it/s]
Requesting API: 3%|▎ | 16/500 [00:10<01:28, 5.45it/s]
Requesting API: 4%|▎ | 18/500 [00:10<01:36, 5.01it/s]
Requesting API: 4%|▍ | 21/500 [00:10<01:15, 6.33it/s]
Requesting API: 5%|▍ | 23/500 [00:11<01:37, 4.91it/s]
Requesting API: 6%|▌ | 31/500 [00:11<00:45, 10.35it/s]
Requesting API: 7%|▋ | 33/500 [00:12<00:49, 9.46it/s]
Requesting API: 7%|▋ | 36/500 [00:12<00:48, 9.66it/s]
Requesting API: 8%|▊ | 38/500 [00:12<00:51, 8.89it/s]
Requesting API: 8%|▊ | 40/500 [00:13<01:26, 5.35it/s]
Requesting API: 8%|▊ | 41/500 [00:13<01:22, 5.54it/s]
Requesting API: 9%|▊ | 43/500 [00:13<01:10, 6.50it/s]
Requesting API: 9%|▉ | 45/500 [00:13<01:01, 7.37it/s]
Requesting API: 10%|▉ | 48/500 [00:14<01:06, 6.83it/s]
Requesting API: 10%|▉ | 49/500 [00:14<01:16, 5.93it/s]
Requesting API: 10%|█ | 51/500 [00:14<01:01, 7.31it/s]
Requesting API: 10%|█ | 52/500 [00:15<01:03, 7.02it/s]
Requesting API: 11%|█ | 53/500 [00:15<01:14, 6.00it/s]
Requesting API: 11%|█ | 56/500 [00:15<01:03, 6.98it/s]
Requesting API: 11%|█▏ | 57/500 [00:16<01:25, 5.16it/s]
Requesting API: 12%|█▏ | 58/500 [00:16<01:49, 4.04it/s]
Requesting API: 12%|█▏ | 61/500 [00:16<01:10, 6.24it/s]
Requesting API: 12%|█▏ | 62/500 [00:16<01:11, 6.11it/s]
Requesting API: 13%|█▎ | 64/500 [00:17<01:05, 6.65it/s]
Requesting API: 13%|█▎ | 67/500 [00:17<00:48, 8.91it/s]
Requesting API: 14%|█▍ | 69/500 [00:17<01:07, 6.35it/s]
Requesting API: 15%|█▍ | 73/500 [00:18<00:45, 9.31it/s]
Requesting API: 15%|█▌ | 75/500 [00:18<00:43, 9.80it/s]
Requesting API: 15%|█▌ | 77/500 [00:18<00:47, 8.84it/s]
Requesting API: 16%|█▌ | 79/500 [00:18<00:51, 8.21it/s]
Requesting API: 16%|█▌ | 80/500 [00:19<01:06, 6.31it/s]
Requesting API: 16%|█▌ | 81/500 [00:19<01:14, 5.65it/s]
Requesting API: 17%|█▋ | 84/500 [00:19<00:51, 8.15it/s]
Requesting API: 17%|█▋ | 86/500 [00:19<00:45, 9.03it/s]
Requesting API: 18%|█▊ | 88/500 [00:20<01:11, 5.80it/s]
Requesting API: 18%|█▊ | 89/500 [00:20<01:09, 5.88it/s]
Requesting API: 18%|█▊ | 92/500 [00:20<00:49, 8.28it/s]
Requesting API: 19%|█▉ | 94/500 [00:20<00:45, 8.94it/s]
Requesting API: 19%|█▉ | 96/500 [00:21<00:49, 8.22it/s]
Requesting API: 19%|█▉ | 97/500 [00:21<00:58, 6.92it/s]
Requesting API: 20%|█▉ | 98/500 [00:21<01:11, 5.63it/s]
Requesting API: 20%|█▉ | 99/500 [00:21<01:10, 5.69it/s]
Requesting API: 20%|██ | 100/500 [00:22<01:10, 5.65it/s]
Requesting API: 20%|██ | 101/500 [00:22<01:28, 4.51it/s]
Requesting API: 20%|██ | 102/500 [00:22<01:21, 4.88it/s]
Requesting API: 21%|██ | 106/500 [00:23<01:02, 6.29it/s]
Requesting API: 22%|██▏ | 108/500 [00:23<00:58, 6.72it/s]
Requesting API: 22%|██▏ | 110/500 [00:23<01:14, 5.20it/s]
Requesting API: 22%|██▏ | 112/500 [00:24<01:02, 6.19it/s]
Requesting API: 23%|██▎ | 116/500 [00:24<00:59, 6.46it/s]
Requesting API: 23%|██▎ | 117/500 [00:24<01:04, 5.97it/s]
Requesting API: 24%|██▍ | 119/500 [00:25<00:59, 6.36it/s]
Requesting API: 24%|██▍ | 120/500 [00:25<01:11, 5.28it/s]
Requesting API: 24%|██▍ | 122/500 [00:25<00:57, 6.52it/s]
Requesting API: 25%|██▌ | 127/500 [00:26<00:40, 9.31it/s]
Requesting API: 26%|██▌ | 130/500 [00:26<00:38, 9.61it/s]
Requesting API: 27%|██▋ | 133/500 [00:26<00:35, 10.22it/s]
Requesting API: 27%|██▋ | 136/500 [00:26<00:35, 10.28it/s]
Requesting API: 28%|██▊ | 138/500 [00:27<00:39, 9.23it/s]
Requesting API: 28%|██▊ | 141/500 [00:27<00:37, 9.57it/s]
Requesting API: 29%|██▉ | 144/500 [00:28<01:04, 5.50it/s]
Requesting API: 29%|██▉ | 146/500 [00:28<01:07, 5.27it/s]
Requesting API: 29%|██▉ | 147/500 [00:29<01:06, 5.28it/s]
Requesting API: 30%|███ | 151/500 [00:29<00:57, 6.03it/s]
Requesting API: 31%|███ | 153/500 [00:29<00:50, 6.85it/s]
Requesting API: 31%|███ | 156/500 [00:30<00:46, 7.46it/s]
Requesting API: 31%|███▏ | 157/500 [00:30<00:51, 6.72it/s]
Requesting API: 32%|███▏ | 158/500 [00:30<00:51, 6.58it/s]
Requesting API: 32%|███▏ | 159/500 [00:30<00:53, 6.43it/s]
Requesting API: 32%|███▏ | 160/500 [00:31<01:06, 5.13it/s]
Requesting API: 32%|███▏ | 161/500 [00:31<01:10, 4.82it/s]
Requesting API: 32%|███▏ | 162/500 [00:31<01:06, 5.10it/s]
Requesting API: 33%|███▎ | 167/500 [00:31<00:34, 9.69it/s]
Requesting API: 34%|███▎ | 168/500 [00:32<00:40, 8.13it/s]
Requesting API: 34%|███▍ | 170/500 [00:32<00:42, 7.70it/s]
Requesting API: 34%|███▍ | 172/500 [00:32<00:38, 8.45it/s]
Requesting API: 35%|███▍ | 173/500 [00:32<00:46, 7.04it/s]
Requesting API: 35%|███▍ | 174/500 [00:32<00:52, 6.18it/s]
Requesting API: 35%|███▌ | 175/500 [00:33<00:52, 6.15it/s]
Requesting API: 35%|███▌ | 176/500 [00:33<00:53, 6.09it/s]
Requesting API: 36%|███▌ | 178/500 [00:33<00:52, 6.13it/s]
Requesting API: 36%|███▌ | 180/500 [00:33<00:46, 6.93it/s]
Requesting API: 37%|███▋ | 184/500 [00:34<00:53, 5.85it/s]
Requesting API: 37%|███▋ | 187/500 [00:34<00:41, 7.58it/s]
Requesting API: 38%|███▊ | 189/500 [00:34<00:36, 8.44it/s]
Requesting API: 38%|███▊ | 191/500 [00:35<00:59, 5.15it/s]
Requesting API: 38%|███▊ | 192/500 [00:36<01:10, 4.36it/s]
Requesting API: 39%|███▉ | 196/500 [00:36<00:43, 7.06it/s]
Requesting API: 40%|███▉ | 198/500 [00:36<00:49, 6.08it/s]
Requesting API: 40%|████ | 200/500 [00:37<00:42, 7.01it/s]
Requesting API: 41%|████ | 203/500 [00:37<00:32, 9.03it/s]
Requesting API: 41%|████ | 205/500 [00:37<00:33, 8.90it/s]
Requesting API: 41%|████▏ | 207/500 [00:37<00:35, 8.26it/s]
Requesting API: 42%|████▏ | 208/500 [00:37<00:37, 7.86it/s]
Requesting API: 42%|████▏ | 211/500 [00:38<00:30, 9.38it/s]
Requesting API: 43%|████▎ | 213/500 [00:38<00:31, 9.16it/s]
Requesting API: 43%|████▎ | 214/500 [00:38<00:39, 7.19it/s]
Requesting API: 43%|████▎ | 215/500 [00:38<00:44, 6.34it/s]
Requesting API: 43%|████▎ | 217/500 [00:39<00:45, 6.24it/s]
Requesting API: 44%|████▍ | 219/500 [00:39<00:44, 6.30it/s]
Requesting API: 44%|████▍ | 220/500 [00:39<00:48, 5.78it/s]
Requesting API: 44%|████▍ | 221/500 [00:39<00:53, 5.24it/s]
Requesting API: 44%|████▍ | 222/500 [00:40<00:50, 5.46it/s]
Requesting API: 45%|████▌ | 225/500 [00:40<00:45, 6.11it/s]
Requesting API: 45%|████▌ | 227/500 [00:40<00:44, 6.14it/s]
Requesting API: 46%|████▌ | 228/500 [00:41<00:52, 5.17it/s]
Requesting API: 46%|████▌ | 229/500 [00:41<00:50, 5.32it/s]
Requesting API: 47%|████▋ | 233/500 [00:41<00:32, 8.31it/s]
Requesting API: 47%|████▋ | 234/500 [00:41<00:40, 6.60it/s]
Requesting API: 47%|████▋ | 237/500 [00:42<00:31, 8.36it/s]
Requesting API: 48%|████▊ | 239/500 [00:42<00:28, 9.17it/s]
Requesting API: 48%|████▊ | 240/500 [00:42<00:46, 5.60it/s]
Requesting API: 48%|████▊ | 241/500 [00:43<00:45, 5.73it/s]
Requesting API: 49%|████▊ | 243/500 [00:43<00:40, 6.38it/s]
Requesting API: 49%|████▉ | 245/500 [00:43<00:36, 7.07it/s]
Requesting API: 50%|█████ | 251/500 [00:44<00:29, 8.46it/s]
Requesting API: 50%|█████ | 252/500 [00:44<00:37, 6.69it/s]
Requesting API: 51%|█████ | 254/500 [00:45<00:48, 5.03it/s]
Requesting API: 51%|█████ | 256/500 [00:45<00:41, 5.89it/s]
Requesting API: 52%|█████▏ | 258/500 [00:46<00:54, 4.44it/s]
Requesting API: 52%|█████▏ | 261/500 [00:46<00:39, 6.09it/s]
Requesting API: 53%|█████▎ | 264/500 [00:46<00:29, 7.93it/s]
Requesting API: 53%|█████▎ | 266/500 [00:46<00:28, 8.07it/s]
Requesting API: 54%|█████▍ | 269/500 [00:47<00:31, 7.42it/s]
Requesting API: 54%|█████▍ | 270/500 [00:47<00:36, 6.38it/s]
Requesting API: 54%|█████▍ | 271/500 [00:47<00:42, 5.40it/s]
Requesting API: 55%|█████▍ | 273/500 [00:47<00:34, 6.62it/s]
Requesting API: 55%|█████▌ | 277/500 [00:48<00:38, 5.82it/s]
Requesting API: 56%|█████▌ | 279/500 [00:48<00:33, 6.61it/s]
Requesting API: 56%|█████▌ | 281/500 [00:49<00:29, 7.53it/s]
Requesting API: 56%|█████▋ | 282/500 [00:49<00:43, 5.01it/s]
Requesting API: 57%|█████▋ | 285/500 [00:50<00:42, 5.09it/s]
Requesting API: 58%|█████▊ | 289/500 [00:50<00:32, 6.44it/s]
Requesting API: 58%|█████▊ | 291/500 [00:50<00:30, 6.88it/s]
Requesting API: 58%|█████▊ | 292/500 [00:51<00:35, 5.89it/s]
Requesting API: 59%|█████▊ | 293/500 [00:51<00:34, 5.95it/s]
Requesting API: 61%|██████ | 303/500 [00:51<00:13, 14.93it/s]
Requesting API: 61%|██████ | 305/500 [00:52<00:20, 9.54it/s]
Requesting API: 61%|██████▏ | 307/500 [00:52<00:27, 7.10it/s]
Requesting API: 62%|██████▏ | 308/500 [00:52<00:27, 6.88it/s]
Requesting API: 62%|██████▏ | 309/500 [00:53<00:32, 5.87it/s]
Requesting API: 62%|██████▏ | 311/500 [00:53<00:27, 7.00it/s]
Requesting API: 63%|██████▎ | 314/500 [00:53<00:29, 6.34it/s]
Requesting API: 63%|██████▎ | 315/500 [00:54<00:29, 6.30it/s]
Requesting API: 63%|██████▎ | 316/500 [00:54<00:32, 5.71it/s]
Requesting API: 63%|██████▎ | 317/500 [00:54<00:34, 5.25it/s]
Requesting API: 64%|██████▎ | 318/500 [00:54<00:33, 5.47it/s]
Requesting API: 64%|██████▍ | 319/500 [00:54<00:36, 4.99it/s]
Requesting API: 64%|██████▍ | 320/500 [00:55<00:34, 5.28it/s]
Requesting API: 64%|██████▍ | 322/500 [00:55<00:34, 5.13it/s]
Requesting API: 65%|██████▍ | 323/500 [00:55<00:33, 5.30it/s]
Requesting API: 65%|██████▌ | 327/500 [00:55<00:20, 8.52it/s]
Requesting API: 66%|██████▌ | 328/500 [00:56<00:25, 6.81it/s]
Requesting API: 66%|██████▌ | 329/500 [00:56<00:25, 6.63it/s]
Requesting API: 66%|██████▌ | 330/500 [00:56<00:26, 6.52it/s]
Requesting API: 66%|██████▋ | 332/500 [00:57<00:33, 5.01it/s]
Requesting API: 67%|██████▋ | 334/500 [00:57<00:26, 6.19it/s]
Requesting API: 67%|██████▋ | 336/500 [00:57<00:21, 7.49it/s]
Requesting API: 67%|██████▋ | 337/500 [00:57<00:29, 5.50it/s]
Requesting API: 68%|██████▊ | 338/500 [00:57<00:28, 5.61it/s]
Requesting API: 68%|██████▊ | 341/500 [00:58<00:25, 6.34it/s]
Requesting API: 68%|██████▊ | 342/500 [00:58<00:29, 5.34it/s]
Requesting API: 69%|██████▉ | 345/500 [00:59<00:32, 4.81it/s]
Requesting API: 70%|██████▉ | 348/500 [00:59<00:30, 4.95it/s]
Requesting API: 70%|███████ | 350/500 [01:00<00:30, 4.93it/s]
Requesting API: 71%|███████ | 355/500 [01:00<00:17, 8.24it/s]
Requesting API: 71%|███████▏ | 357/500 [01:00<00:19, 7.46it/s]
Requesting API: 72%|███████▏ | 358/500 [01:01<00:20, 6.85it/s]
Requesting API: 72%|███████▏ | 359/500 [01:01<00:22, 6.32it/s]
Requesting API: 72%|███████▏ | 360/500 [01:01<00:22, 6.25it/s]
Requesting API: 72%|███████▏ | 362/500 [01:01<00:18, 7.61it/s]
Requesting API: 73%|███████▎ | 364/500 [01:01<00:15, 8.79it/s]
Requesting API: 73%|███████▎ | 366/500 [01:02<00:15, 8.88it/s]
Requesting API: 73%|███████▎ | 367/500 [01:02<00:17, 7.54it/s]
Requesting API: 74%|███████▎ | 368/500 [01:02<00:20, 6.60it/s]
Requesting API: 74%|███████▍ | 370/500 [01:02<00:16, 8.07it/s]
Requesting API: 74%|███████▍ | 371/500 [01:03<00:22, 5.62it/s]
Requesting API: 74%|███████▍ | 372/500 [01:03<00:24, 5.14it/s]
Requesting API: 75%|███████▍ | 373/500 [01:03<00:26, 4.78it/s]
Requesting API: 75%|███████▍ | 374/500 [01:03<00:24, 5.11it/s]
Requesting API: 75%|███████▌ | 376/500 [01:04<00:31, 3.96it/s]
Requesting API: 76%|███████▋ | 382/500 [01:04<00:15, 7.49it/s]
Requesting API: 77%|███████▋ | 384/500 [01:05<00:20, 5.53it/s]
Requesting API: 77%|███████▋ | 386/500 [01:05<00:20, 5.58it/s]
Requesting API: 77%|███████▋ | 387/500 [01:06<00:24, 4.64it/s]
Requesting API: 78%|███████▊ | 390/500 [01:06<00:17, 6.44it/s]
Requesting API: 78%|███████▊ | 392/500 [01:06<00:15, 7.00it/s]
Requesting API: 79%|███████▊ | 393/500 [01:06<00:15, 6.89it/s]
Requesting API: 79%|███████▉ | 395/500 [01:07<00:14, 7.46it/s]
Requesting API: 79%|███████▉ | 396/500 [01:07<00:14, 7.24it/s]
Requesting API: 79%|███████▉ | 397/500 [01:07<00:16, 6.37it/s]
Requesting API: 80%|███████▉ | 398/500 [01:07<00:16, 6.37it/s]
Requesting API: 80%|████████ | 400/500 [01:07<00:12, 7.84it/s]
Requesting API: 80%|████████ | 401/500 [01:08<00:18, 5.47it/s]
Requesting API: 80%|████████ | 402/500 [01:08<00:17, 5.67it/s]
Requesting API: 81%|████████ | 403/500 [01:08<00:25, 3.87it/s]
Requesting API: 81%|████████ | 406/500 [01:08<00:14, 6.44it/s]
Requesting API: 82%|████████▏ | 408/500 [01:09<00:13, 7.04it/s]
Requesting API: 82%|████████▏ | 409/500 [01:09<00:19, 4.60it/s]
Requesting API: 82%|████████▏ | 410/500 [01:09<00:19, 4.73it/s]
Requesting API: 83%|████████▎ | 413/500 [01:10<00:11, 7.41it/s]
Requesting API: 83%|████████▎ | 416/500 [01:10<00:13, 6.12it/s]
Requesting API: 83%|████████▎ | 417/500 [01:10<00:15, 5.37it/s]
Requesting API: 84%|████████▍ | 422/500 [01:11<00:11, 6.59it/s]
Requesting API: 85%|████████▍ | 423/500 [01:11<00:13, 5.75it/s]
Requesting API: 85%|████████▍ | 424/500 [01:12<00:14, 5.38it/s]
Requesting API: 85%|████████▌ | 425/500 [01:12<00:14, 5.06it/s]
Requesting API: 85%|████████▌ | 426/500 [01:12<00:14, 5.27it/s]
Requesting API: 86%|████████▌ | 428/500 [01:13<00:17, 4.14it/s]
Requesting API: 86%|████████▌ | 431/500 [01:13<00:11, 6.20it/s]
Requesting API: 86%|████████▋ | 432/500 [01:13<00:12, 5.38it/s]
Requesting API: 87%|████████▋ | 433/500 [01:13<00:12, 5.54it/s]
Requesting API: 87%|████████▋ | 436/500 [01:14<00:07, 8.27it/s]
Requesting API: 88%|████████▊ | 438/500 [01:14<00:10, 6.09it/s]
Requesting API: 88%|████████▊ | 440/500 [01:14<00:09, 6.58it/s]
Requesting API: 89%|████████▉ | 444/500 [01:14<00:05, 9.74it/s]
Requesting API: 89%|████████▉ | 447/500 [01:15<00:04, 11.26it/s]
Requesting API: 90%|████████▉ | 449/500 [01:15<00:04, 11.16it/s]
Requesting API: 90%|█████████ | 452/500 [01:15<00:04, 9.93it/s]
Requesting API: 91%|█████████ | 454/500 [01:16<00:05, 8.28it/s]
Requesting API: 91%|█████████▏| 457/500 [01:16<00:04, 9.98it/s]
Requesting API: 92%|█████████▏| 459/500 [01:16<00:04, 10.21it/s]
Requesting API: 92%|█████████▏| 461/500 [01:16<00:03, 10.37it/s]
Requesting API: 93%|█████████▎| 465/500 [01:16<00:02, 13.40it/s]
Requesting API: 93%|█████████▎| 467/500 [01:17<00:03, 10.02it/s]
Requesting API: 94%|█████████▍| 469/500 [01:17<00:03, 9.19it/s]
Requesting API: 94%|█████████▍| 471/500 [01:17<00:03, 9.60it/s]
Requesting API: 95%|█████████▍| 473/500 [01:18<00:04, 5.71it/s]
Requesting API: 95%|█████████▌| 475/500 [01:18<00:03, 6.60it/s]
Requesting API: 95%|█████████▌| 477/500 [01:19<00:04, 4.71it/s]
Requesting API: 96%|█████████▌| 478/500 [01:19<00:05, 3.94it/s]
Requesting API: 96%|█████████▌| 479/500 [01:20<00:08, 2.50it/s]
Requesting API: 96%|█████████▌| 481/500 [01:21<00:06, 2.83it/s]
Requesting API: 96%|█████████▋| 482/500 [01:21<00:06, 2.81it/s]
Requesting API: 97%|█████████▋| 483/500 [01:23<00:10, 1.66it/s]
Requesting API: 97%|█████████▋| 484/500 [01:26<00:21, 1.34s/it]
Requesting API: 97%|█████████▋| 485/500 [01:27<00:18, 1.23s/it]
Requesting API: 97%|█████████▋| 487/500 [01:30<00:16, 1.27s/it]
Requesting API: 98%|█████████▊| 488/500 [01:30<00:13, 1.14s/it]
Requesting API: 98%|█████████▊| 489/500 [01:31<00:10, 1.08it/s]
Requesting API: 98%|█████████▊| 490/500 [01:32<00:10, 1.01s/it]
Requesting API: 98%|█████████▊| 491/500 [01:33<00:08, 1.02it/s]
Requesting API: 98%|█████████▊| 492/500 [01:33<00:06, 1.24it/s]
Requesting API: 99%|█████████▉| 494/500 [01:34<00:03, 1.88it/s]
Requesting API: 99%|█████████▉| 497/500 [01:34<00:01, 2.56it/s]
Requesting API: 100%|█████████▉| 499/500 [01:36<00:00, 2.20it/s]
Requesting API: 100%|██████████| 500/500 [01:36<00:00, 5.20it/s]
2026-07-24:00:44:58 INFO [loggers.evaluation_tracker:247] Saving results aggregated
2026-07-24:00:44:58 INFO [loggers.evaluation_tracker:119] Saving per-task samples to outputs/evals/protocolF/keep25_unhealed/student/*.jsonl
local-completions ({'model': 'student', 'base_url': 'http://127.0.0.1:8399/v1/completions', 'tokenizer': 'outputs/qwen35_pruned_keep25', 'num_concurrent': 64, 'tokenized_requests': False, 'max_retries': 3}), gen_kwargs: ({}), limit: None, num_fewshot: None, batch_size: 1
| Tasks |Version| Filter |n-shot| Metric | |Value| |Stderr|
|---------|------:|-----------|-----:|---------|---|----:|---|-----:|
|humaneval| 1|create_test| 0|pass@1 |↑ |0.000|± |0.0000|
|mbpp | 1|none | 3|pass_at_1|↑ |0.004|± |0.0028|
|