File size: 29,322 Bytes
8717f59 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 | 2026-07-24:04:18:24 INFO [_cli.run:388] Selected Tasks: ['humaneval', 'mbpp']
2026-07-24:04:18:25 INFO [evaluator:214] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2026-07-24:04:18:25 INFO [evaluator:239] Initializing local-completions model, with arguments: {'model': 'student', 'base_url': 'http://127.0.0.1:8399/v1/completions', 'tokenizer': 'outputs/healed_dl/keep25_step0200', 'num_concurrent': 64, 'tokenized_requests': False, 'max_retries': 3}
2026-07-24:04:18:25 INFO [models.openai_completions:42] Remote tokenizer not supported. Using huggingface tokenizer backend.
2026-07-24:04:18:25 INFO [models.api_models:179] Using max length 2048 - 1
2026-07-24:04:18:25 INFO [models.api_models:200] Using tokenizer huggingface
2026-07-24:04:18:33 INFO [evaluator_utils:446] Selected tasks:
2026-07-24:04:18:33 INFO [evaluator_utils:480] Task: humaneval (humaneval/humaneval.yaml)
2026-07-24:04:18:33 INFO [evaluator_utils:480] Task: mbpp (mbpp/mbpp.yaml)
2026-07-24:04:18:33 INFO [evaluator:314] humaneval: Using gen_kwargs: {'until': ['\nclass', '\ndef', '\n#', '\nif', '\nprint'], 'max_gen_toks': 1024, 'do_sample': False}
2026-07-24:04:18:33 INFO [evaluator:314] mbpp: Using gen_kwargs: {'until': ['[DONE]'], 'do_sample': False}
2026-07-24:04:18:33 INFO [api.task:312] Building contexts for humaneval on rank 0...
0%| | 0/164 [00:00<?, ?it/s]
88%|████████▊ | 144/164 [00:00<00:00, 1436.85it/s]
100%|██████████| 164/164 [00:00<00:00, 1469.01it/s]
2026-07-24:04:18:33 INFO [api.task:312] Building contexts for mbpp on rank 0...
0%| | 0/500 [00:00<?, ?it/s]
3%|▎ | 17/500 [00:00<00:03, 160.25it/s]
7%|▋ | 34/500 [00:00<00:02, 161.11it/s]
10%|█ | 51/500 [00:00<00:02, 161.48it/s]
14%|█▎ | 68/500 [00:00<00:02, 161.60it/s]
17%|█▋ | 85/500 [00:00<00:02, 162.08it/s]
20%|██ | 102/500 [00:00<00:02, 162.49it/s]
24%|██▍ | 119/500 [00:00<00:02, 162.69it/s]
27%|██▋ | 136/500 [00:00<00:02, 162.94it/s]
31%|███ | 153/500 [00:00<00:02, 163.12it/s]
34%|███▍ | 170/500 [00:01<00:02, 163.30it/s]
37%|███▋ | 187/500 [00:01<00:01, 163.59it/s]
41%|████ | 204/500 [00:01<00:01, 163.72it/s]
44%|████▍ | 221/500 [00:01<00:01, 163.70it/s]
48%|████▊ | 238/500 [00:01<00:01, 163.62it/s]
51%|█████ | 255/500 [00:01<00:01, 163.50it/s]
54%|█████▍ | 272/500 [00:01<00:01, 163.71it/s]
58%|█████▊ | 289/500 [00:01<00:01, 163.82it/s]
61%|██████ | 306/500 [00:01<00:01, 163.78it/s]
65%|██████▍ | 323/500 [00:01<00:01, 163.82it/s]
68%|██████▊ | 340/500 [00:02<00:00, 163.90it/s]
71%|███████▏ | 357/500 [00:02<00:00, 163.92it/s]
75%|███████▍ | 374/500 [00:02<00:00, 164.05it/s]
78%|███████▊ | 391/500 [00:02<00:00, 163.92it/s]
82%|████████▏ | 408/500 [00:02<00:00, 163.97it/s]
85%|████████▌ | 425/500 [00:02<00:00, 164.00it/s]
88%|████████▊ | 442/500 [00:02<00:00, 164.18it/s]
92%|█████████▏| 459/500 [00:02<00:00, 164.20it/s]
95%|█████████▌| 476/500 [00:02<00:00, 164.16it/s]
99%|█████████▊| 493/500 [00:03<00:00, 164.31it/s]
100%|██████████| 500/500 [00:03<00:00, 163.52it/s]
2026-07-24:04:18:36 INFO [evaluator:585] Running generate_until requests
2026-07-24:04:18:36 INFO [models.api_models:747] Tokenized requests are disabled. Context + generation length is not checked.
Requesting API: 0%| | 0/164 [00:00<?, ?it/s]
Requesting API: 1%| | 1/164 [00:03<08:22, 3.08s/it]
Requesting API: 1%| | 2/164 [00:12<17:41, 6.55s/it]
Requesting API: 2%|▏ | 3/164 [01:35<1:52:08, 41.79s/it]
Requesting API: 2%|▏ | 4/164 [01:35<1:07:38, 25.37s/it]
Requesting API: 6%|▌ | 10/164 [01:36<14:53, 5.80s/it]
Requesting API: 16%|█▌ | 26/164 [01:36<03:24, 1.49s/it]
Requesting API: 23%|██▎ | 37/164 [01:36<01:49, 1.16it/s]
Requesting API: 29%|██▉ | 48/164 [01:36<01:03, 1.83it/s]
Requesting API: 37%|███▋ | 61/164 [01:36<00:34, 2.95it/s]
Requesting API: 41%|████ | 67/164 [01:40<00:37, 2.56it/s]
Requesting API: 41%|████ | 67/164 [01:53<00:37, 2.56it/s]
Requesting API: 43%|████▎ | 70/164 [03:10<06:30, 4.15s/it]
Requesting API: 43%|████▎ | 71/164 [03:10<06:04, 3.92s/it]
Requesting API: 47%|████▋ | 77/164 [03:10<03:49, 2.64s/it]
Requesting API: 57%|█████▋ | 93/164 [03:10<01:23, 1.18s/it]
Requesting API: 63%|██████▎ | 103/164 [03:11<00:48, 1.26it/s]
Requesting API: 69%|██████▉ | 113/164 [03:11<00:27, 1.84it/s]
Requesting API: 76%|███████▌ | 125/164 [03:11<00:13, 2.82it/s]
Requesting API: 80%|████████ | 132/164 [03:14<00:12, 2.61it/s]
Requesting API: 80%|████████ | 132/164 [03:26<00:12, 2.61it/s]
Requesting API: 82%|████████▏ | 134/164 [04:44<02:04, 4.16s/it]
Requesting API: 82%|████████▏ | 135/164 [04:44<01:54, 3.93s/it]
Requesting API: 91%|█████████ | 149/164 [04:44<00:27, 1.84s/it]
Requesting API: 100%|██████████| 164/164 [04:44<00:00, 1.74s/it]
Requesting API: 0%| | 0/500 [00:00<?, ?it/s]
Requesting API: 0%| | 1/500 [00:06<50:43, 6.10s/it]
Requesting API: 0%| | 2/500 [00:08<31:09, 3.75s/it]
Requesting API: 1%| | 4/500 [00:08<12:03, 1.46s/it]
Requesting API: 1%| | 6/500 [00:08<06:40, 1.23it/s]
Requesting API: 1%|▏ | 7/500 [00:08<05:20, 1.54it/s]
Requesting API: 2%|▏ | 8/500 [00:09<04:28, 1.83it/s]
Requesting API: 2%|▏ | 10/500 [00:09<03:08, 2.60it/s]
Requesting API: 2%|▏ | 12/500 [00:09<02:29, 3.27it/s]
Requesting API: 3%|▎ | 14/500 [00:10<02:35, 3.12it/s]
Requesting API: 4%|▎ | 18/500 [00:10<01:28, 5.46it/s]
Requesting API: 4%|▍ | 20/500 [00:10<01:15, 6.36it/s]
Requesting API: 5%|▍ | 23/500 [00:11<01:04, 7.37it/s]
Requesting API: 5%|▌ | 25/500 [00:11<01:04, 7.34it/s]
Requesting API: 6%|▌ | 28/500 [00:11<00:57, 8.17it/s]
Requesting API: 6%|▌ | 30/500 [00:11<00:59, 7.89it/s]
Requesting API: 7%|▋ | 34/500 [00:12<00:48, 9.60it/s]
Requesting API: 7%|▋ | 36/500 [00:12<00:51, 8.93it/s]
Requesting API: 7%|▋ | 37/500 [00:12<01:03, 7.26it/s]
Requesting API: 8%|▊ | 39/500 [00:13<01:18, 5.90it/s]
Requesting API: 8%|▊ | 41/500 [00:13<01:12, 6.35it/s]
Requesting API: 9%|▉ | 44/500 [00:13<01:04, 7.09it/s]
Requesting API: 9%|▉ | 45/500 [00:14<01:23, 5.46it/s]
Requesting API: 9%|▉ | 46/500 [00:14<01:21, 5.58it/s]
Requesting API: 9%|▉ | 47/500 [00:14<01:29, 5.07it/s]
Requesting API: 10%|▉ | 49/500 [00:14<01:10, 6.42it/s]
Requesting API: 11%|█ | 54/500 [00:15<01:02, 7.18it/s]
Requesting API: 11%|█ | 55/500 [00:15<01:16, 5.84it/s]
Requesting API: 11%|█▏ | 57/500 [00:16<01:11, 6.24it/s]
Requesting API: 12%|█▏ | 58/500 [00:16<01:18, 5.67it/s]
Requesting API: 12%|█▏ | 60/500 [00:16<01:11, 6.14it/s]
Requesting API: 12%|█▏ | 61/500 [00:16<01:19, 5.55it/s]
Requesting API: 12%|█▏ | 62/500 [00:17<01:27, 5.03it/s]
Requesting API: 13%|█▎ | 64/500 [00:17<01:08, 6.34it/s]
Requesting API: 13%|█▎ | 66/500 [00:17<01:10, 6.16it/s]
Requesting API: 13%|█▎ | 67/500 [00:17<01:10, 6.16it/s]
Requesting API: 14%|█▎ | 68/500 [00:18<01:28, 4.90it/s]
Requesting API: 14%|█▍ | 71/500 [00:18<00:57, 7.51it/s]
Requesting API: 15%|█▌ | 75/500 [00:18<00:49, 8.65it/s]
Requesting API: 15%|█▌ | 77/500 [00:19<00:50, 8.44it/s]
Requesting API: 16%|█▌ | 78/500 [00:19<00:58, 7.19it/s]
Requesting API: 16%|█▌ | 79/500 [00:19<01:05, 6.39it/s]
Requesting API: 16%|█▌ | 80/500 [00:19<01:07, 6.24it/s]
Requesting API: 16%|█▋ | 82/500 [00:19<00:55, 7.49it/s]
Requesting API: 17%|█▋ | 83/500 [00:20<00:58, 7.16it/s]
Requesting API: 17%|█▋ | 86/500 [00:20<00:57, 7.23it/s]
Requesting API: 17%|█▋ | 87/500 [00:20<01:09, 5.98it/s]
Requesting API: 18%|█▊ | 88/500 [00:20<01:05, 6.25it/s]
Requesting API: 18%|█▊ | 89/500 [00:21<01:05, 6.24it/s]
Requesting API: 18%|█▊ | 90/500 [00:21<01:41, 4.04it/s]
Requesting API: 19%|█▊ | 93/500 [00:22<01:31, 4.44it/s]
Requesting API: 19%|█▉ | 96/500 [00:22<01:19, 5.07it/s]
Requesting API: 19%|█▉ | 97/500 [00:22<01:15, 5.37it/s]
Requesting API: 20%|█▉ | 98/500 [00:23<01:21, 4.96it/s]
Requesting API: 20%|█▉ | 99/500 [00:23<01:41, 3.94it/s]
Requesting API: 21%|██ | 104/500 [00:23<01:02, 6.34it/s]
Requesting API: 21%|██ | 105/500 [00:24<01:08, 5.80it/s]
Requesting API: 21%|██▏ | 107/500 [00:24<01:12, 5.42it/s]
Requesting API: 22%|██▏ | 109/500 [00:24<01:06, 5.91it/s]
Requesting API: 22%|██▏ | 112/500 [00:25<00:52, 7.34it/s]
Requesting API: 23%|██▎ | 114/500 [00:25<00:58, 6.60it/s]
Requesting API: 23%|██▎ | 116/500 [00:25<00:55, 6.86it/s]
Requesting API: 24%|██▍ | 119/500 [00:26<00:50, 7.52it/s]
Requesting API: 24%|██▍ | 120/500 [00:26<01:16, 4.98it/s]
Requesting API: 24%|██▍ | 121/500 [00:27<01:25, 4.41it/s]
Requesting API: 25%|██▍ | 123/500 [00:27<01:08, 5.51it/s]
Requesting API: 25%|██▌ | 125/500 [00:27<00:56, 6.63it/s]
Requesting API: 25%|██▌ | 126/500 [00:27<00:57, 6.55it/s]
Requesting API: 25%|██▌ | 127/500 [00:28<01:18, 4.76it/s]
Requesting API: 26%|██▌ | 129/500 [00:28<00:59, 6.20it/s]
Requesting API: 26%|██▌ | 130/500 [00:28<01:13, 5.02it/s]
Requesting API: 26%|██▌ | 131/500 [00:28<01:10, 5.25it/s]
Requesting API: 27%|██▋ | 133/500 [00:29<02:00, 3.04it/s]
Requesting API: 27%|██▋ | 136/500 [00:29<01:14, 4.86it/s]
Requesting API: 28%|██▊ | 139/500 [00:30<01:25, 4.22it/s]
Requesting API: 28%|██▊ | 142/500 [00:30<01:02, 5.77it/s]
Requesting API: 29%|██▊ | 143/500 [00:31<01:08, 5.22it/s]
Requesting API: 29%|██▉ | 144/500 [00:31<01:14, 4.77it/s]
Requesting API: 29%|██▉ | 145/500 [00:31<01:17, 4.58it/s]
Requesting API: 29%|██▉ | 146/500 [00:32<01:34, 3.75it/s]
Requesting API: 30%|██▉ | 148/500 [00:32<01:10, 5.02it/s]
Requesting API: 30%|██▉ | 149/500 [00:32<01:19, 4.42it/s]
Requesting API: 30%|███ | 150/500 [00:32<01:13, 4.74it/s]
Requesting API: 31%|███ | 153/500 [00:33<01:14, 4.67it/s]
Requesting API: 31%|███ | 154/500 [00:34<01:28, 3.89it/s]
Requesting API: 31%|███ | 155/500 [00:34<01:51, 3.11it/s]
Requesting API: 31%|███ | 156/500 [00:35<02:00, 2.86it/s]
Requesting API: 32%|███▏ | 158/500 [00:35<01:24, 4.07it/s]
Requesting API: 32%|███▏ | 159/500 [00:35<01:24, 4.04it/s]
Requesting API: 32%|███▏ | 162/500 [00:35<00:57, 5.89it/s]
Requesting API: 33%|███▎ | 163/500 [00:36<01:51, 3.03it/s]
Requesting API: 33%|███▎ | 164/500 [00:37<01:52, 3.00it/s]
Requesting API: 33%|███▎ | 166/500 [00:37<01:49, 3.05it/s]
Requesting API: 33%|███▎ | 167/500 [00:37<01:38, 3.36it/s]
Requesting API: 34%|███▍ | 169/500 [00:38<01:18, 4.24it/s]
Requesting API: 34%|███▍ | 171/500 [00:38<01:23, 3.92it/s]
Requesting API: 34%|███▍ | 172/500 [00:38<01:18, 4.16it/s]
Requesting API: 35%|███▍ | 173/500 [00:39<01:13, 4.45it/s]
Requesting API: 35%|███▌ | 176/500 [00:39<01:11, 4.51it/s]
Requesting API: 36%|███▌ | 178/500 [00:40<01:23, 3.84it/s]
Requesting API: 36%|███▌ | 180/500 [00:41<01:36, 3.31it/s]
Requesting API: 36%|███▌ | 181/500 [00:41<01:55, 2.77it/s]
Requesting API: 37%|███▋ | 183/500 [00:42<01:47, 2.94it/s]
Requesting API: 37%|███▋ | 184/500 [00:43<02:05, 2.53it/s]
Requesting API: 37%|███▋ | 187/500 [00:43<01:16, 4.11it/s]
Requesting API: 38%|███▊ | 188/500 [00:43<01:11, 4.38it/s]
Requesting API: 38%|███▊ | 189/500 [00:43<01:10, 4.40it/s]
Requesting API: 38%|███▊ | 190/500 [00:44<01:28, 3.51it/s]
Requesting API: 39%|███▊ | 193/500 [00:44<00:53, 5.71it/s]
Requesting API: 39%|███▉ | 194/500 [00:44<00:56, 5.42it/s]
Requesting API: 39%|███▉ | 195/500 [00:44<01:05, 4.66it/s]
Requesting API: 39%|███▉ | 196/500 [00:45<01:01, 4.95it/s]
Requesting API: 39%|███▉ | 197/500 [00:45<01:05, 4.63it/s]
Requesting API: 40%|███▉ | 199/500 [00:45<01:08, 4.41it/s]
Requesting API: 40%|████ | 200/500 [00:45<01:05, 4.59it/s]
Requesting API: 40%|████ | 202/500 [00:46<01:12, 4.10it/s]
Requesting API: 41%|████ | 203/500 [00:46<01:08, 4.32it/s]
Requesting API: 41%|████ | 205/500 [00:47<01:09, 4.24it/s]
Requesting API: 41%|████▏ | 207/500 [00:47<01:06, 4.42it/s]
Requesting API: 42%|████▏ | 211/500 [00:47<00:39, 7.38it/s]
Requesting API: 42%|████▏ | 212/500 [00:48<01:00, 4.76it/s]
Requesting API: 43%|████▎ | 214/500 [00:49<01:15, 3.77it/s]
Requesting API: 43%|████▎ | 215/500 [00:49<01:11, 3.99it/s]
Requesting API: 43%|████▎ | 216/500 [00:49<01:06, 4.29it/s]
Requesting API: 44%|████▍ | 220/500 [00:50<00:50, 5.53it/s]
Requesting API: 44%|████▍ | 221/500 [00:50<00:49, 5.63it/s]
Requesting API: 45%|████▍ | 224/500 [00:50<00:38, 7.18it/s]
Requesting API: 45%|████▌ | 225/500 [00:51<01:05, 4.21it/s]
Requesting API: 45%|████▌ | 226/500 [00:51<01:11, 3.82it/s]
Requesting API: 46%|████▌ | 228/500 [00:51<00:55, 4.93it/s]
Requesting API: 46%|████▌ | 229/500 [00:51<00:52, 5.16it/s]
Requesting API: 46%|████▌ | 230/500 [00:52<00:59, 4.52it/s]
Requesting API: 46%|████▌ | 231/500 [00:52<01:01, 4.37it/s]
Requesting API: 47%|████▋ | 233/500 [00:52<00:44, 5.96it/s]
Requesting API: 47%|████▋ | 235/500 [00:53<00:59, 4.49it/s]
Requesting API: 48%|████▊ | 238/500 [00:53<00:39, 6.59it/s]
Requesting API: 48%|████▊ | 240/500 [00:53<00:43, 5.93it/s]
Requesting API: 48%|████▊ | 241/500 [00:54<00:57, 4.51it/s]
Requesting API: 48%|████▊ | 242/500 [00:54<00:55, 4.66it/s]
Requesting API: 49%|████▉ | 244/500 [00:54<00:42, 6.01it/s]
Requesting API: 49%|████▉ | 247/500 [00:55<00:35, 7.15it/s]
Requesting API: 50%|████▉ | 249/500 [00:55<00:33, 7.60it/s]
Requesting API: 50%|█████ | 251/500 [00:55<00:45, 5.52it/s]
Requesting API: 51%|█████ | 255/500 [00:56<00:29, 8.36it/s]
Requesting API: 51%|█████▏ | 257/500 [00:56<00:45, 5.39it/s]
Requesting API: 52%|█████▏ | 258/500 [00:57<00:54, 4.47it/s]
Requesting API: 52%|█████▏ | 259/500 [00:57<00:59, 4.08it/s]
Requesting API: 52%|█████▏ | 261/500 [00:57<00:46, 5.14it/s]
Requesting API: 52%|█████▏ | 262/500 [00:58<00:51, 4.58it/s]
Requesting API: 53%|█████▎ | 264/500 [00:58<00:39, 5.90it/s]
Requesting API: 53%|█████▎ | 265/500 [00:58<00:54, 4.28it/s]
Requesting API: 54%|█████▎ | 268/500 [00:58<00:35, 6.50it/s]
Requesting API: 54%|█████▍ | 269/500 [00:59<00:55, 4.16it/s]
Requesting API: 54%|█████▍ | 271/500 [00:59<00:43, 5.26it/s]
Requesting API: 55%|█████▍ | 273/500 [01:00<00:37, 6.02it/s]
Requesting API: 55%|█████▌ | 275/500 [01:00<00:37, 6.04it/s]
Requesting API: 55%|█████▌ | 276/500 [01:00<00:49, 4.49it/s]
Requesting API: 56%|█████▌ | 281/500 [01:01<00:32, 6.65it/s]
Requesting API: 56%|█████▋ | 282/500 [01:01<00:35, 6.18it/s]
Requesting API: 57%|█████▋ | 285/500 [01:02<00:42, 5.10it/s]
Requesting API: 57%|█████▋ | 286/500 [01:02<00:46, 4.58it/s]
Requesting API: 58%|█████▊ | 288/500 [01:02<00:43, 4.90it/s]
Requesting API: 58%|█████▊ | 289/500 [01:03<00:41, 5.10it/s]
Requesting API: 58%|█████▊ | 291/500 [01:03<00:36, 5.80it/s]
Requesting API: 58%|█████▊ | 292/500 [01:03<00:35, 5.90it/s]
Requesting API: 59%|█████▉ | 294/500 [01:04<00:42, 4.89it/s]
Requesting API: 59%|█████▉ | 297/500 [01:04<00:28, 7.01it/s]
Requesting API: 60%|█████▉ | 298/500 [01:04<00:36, 5.46it/s]
Requesting API: 60%|█████▉ | 299/500 [01:04<00:35, 5.60it/s]
Requesting API: 60%|██████ | 301/500 [01:05<00:31, 6.26it/s]
Requesting API: 61%|██████ | 303/500 [01:05<00:31, 6.28it/s]
Requesting API: 61%|██████ | 304/500 [01:05<00:33, 5.83it/s]
Requesting API: 61%|██████▏ | 307/500 [01:06<00:30, 6.41it/s]
Requesting API: 62%|██████▏ | 308/500 [01:06<00:35, 5.43it/s]
Requesting API: 62%|██████▏ | 310/500 [01:06<00:28, 6.73it/s]
Requesting API: 62%|██████▏ | 311/500 [01:06<00:34, 5.53it/s]
Requesting API: 62%|██████▏ | 312/500 [01:07<00:36, 5.09it/s]
Requesting API: 63%|██████▎ | 313/500 [01:07<00:50, 3.72it/s]
Requesting API: 63%|██████▎ | 315/500 [01:07<00:36, 5.06it/s]
Requesting API: 63%|██████▎ | 316/500 [01:08<00:41, 4.47it/s]
Requesting API: 63%|██████▎ | 317/500 [01:08<00:49, 3.71it/s]
Requesting API: 64%|██████▍ | 319/500 [01:08<00:35, 5.06it/s]
Requesting API: 64%|██████▍ | 320/500 [01:08<00:34, 5.29it/s]
Requesting API: 64%|██████▍ | 321/500 [01:09<00:50, 3.56it/s]
Requesting API: 64%|██████▍ | 322/500 [01:09<00:45, 3.91it/s]
Requesting API: 65%|██████▍ | 323/500 [01:10<00:57, 3.10it/s]
Requesting API: 65%|██████▍ | 324/500 [01:10<00:50, 3.52it/s]
Requesting API: 65%|██████▌ | 325/500 [01:10<00:48, 3.63it/s]
Requesting API: 65%|██████▌ | 326/500 [01:10<00:42, 4.14it/s]
Requesting API: 66%|██████▌ | 328/500 [01:10<00:32, 5.29it/s]
Requesting API: 66%|██████▌ | 329/500 [01:11<00:31, 5.52it/s]
Requesting API: 66%|██████▌ | 330/500 [01:11<00:29, 5.71it/s]
Requesting API: 66%|██████▌ | 331/500 [01:11<00:28, 5.86it/s]
Requesting API: 66%|██████▋ | 332/500 [01:11<00:35, 4.75it/s]
Requesting API: 67%|██████▋ | 333/500 [01:11<00:37, 4.51it/s]
Requesting API: 67%|██████▋ | 334/500 [01:12<00:33, 4.91it/s]
Requesting API: 67%|██████▋ | 336/500 [01:12<00:27, 5.94it/s]
Requesting API: 67%|██████▋ | 337/500 [01:12<00:33, 4.90it/s]
Requesting API: 68%|██████▊ | 338/500 [01:13<00:41, 3.87it/s]
Requesting API: 68%|██████▊ | 342/500 [01:13<00:31, 5.05it/s]
Requesting API: 69%|██████▊ | 343/500 [01:14<00:35, 4.46it/s]
Requesting API: 69%|██████▉ | 344/500 [01:14<00:33, 4.61it/s]
Requesting API: 69%|██████▉ | 346/500 [01:14<00:34, 4.47it/s]
Requesting API: 70%|██████▉ | 349/500 [01:15<00:31, 4.79it/s]
Requesting API: 70%|███████ | 351/500 [01:15<00:32, 4.53it/s]
Requesting API: 71%|███████ | 354/500 [01:16<00:33, 4.30it/s]
Requesting API: 71%|███████ | 356/500 [01:16<00:31, 4.62it/s]
Requesting API: 71%|███████▏ | 357/500 [01:17<00:41, 3.43it/s]
Requesting API: 72%|███████▏ | 358/500 [01:17<00:38, 3.68it/s]
Requesting API: 72%|███████▏ | 359/500 [01:18<00:42, 3.33it/s]
Requesting API: 72%|███████▏ | 360/500 [01:18<00:38, 3.65it/s]
Requesting API: 72%|███████▏ | 361/500 [01:18<00:37, 3.73it/s]
Requesting API: 72%|███████▏ | 362/500 [01:19<00:42, 3.25it/s]
Requesting API: 73%|███████▎ | 364/500 [01:19<00:44, 3.08it/s]
Requesting API: 73%|███████▎ | 367/500 [01:19<00:26, 5.03it/s]
Requesting API: 74%|███████▍ | 369/500 [01:20<00:30, 4.32it/s]
Requesting API: 74%|███████▍ | 371/500 [01:20<00:27, 4.68it/s]
Requesting API: 74%|███████▍ | 372/500 [01:21<00:26, 4.91it/s]
Requesting API: 75%|███████▌ | 375/500 [01:21<00:24, 5.17it/s]
Requesting API: 75%|███████▌ | 376/500 [01:21<00:23, 5.20it/s]
Requesting API: 76%|███████▌ | 378/500 [01:22<00:28, 4.22it/s]
Requesting API: 76%|███████▌ | 379/500 [01:22<00:27, 4.41it/s]
Requesting API: 76%|███████▌ | 380/500 [01:23<00:38, 3.08it/s]
Requesting API: 76%|███████▌ | 381/500 [01:23<00:34, 3.42it/s]
Requesting API: 76%|███████▋ | 382/500 [01:23<00:33, 3.54it/s]
Requesting API: 77%|███████▋ | 385/500 [01:23<00:18, 6.12it/s]
Requesting API: 77%|███████▋ | 386/500 [01:24<00:27, 4.20it/s]
Requesting API: 77%|███████▋ | 387/500 [01:24<00:31, 3.54it/s]
Requesting API: 78%|███████▊ | 388/500 [01:25<00:38, 2.93it/s]
Requesting API: 78%|███████▊ | 389/500 [01:25<00:33, 3.32it/s]
Requesting API: 78%|███████▊ | 390/500 [01:25<00:31, 3.46it/s]
Requesting API: 78%|███████▊ | 392/500 [01:26<00:23, 4.58it/s]
Requesting API: 79%|███████▉ | 394/500 [01:26<00:18, 5.64it/s]
Requesting API: 79%|███████▉ | 395/500 [01:26<00:27, 3.85it/s]
Requesting API: 80%|███████▉ | 398/500 [01:27<00:16, 6.04it/s]
Requesting API: 80%|████████ | 400/500 [01:27<00:13, 7.23it/s]
Requesting API: 80%|████████ | 401/500 [01:27<00:22, 4.49it/s]
Requesting API: 81%|████████ | 405/500 [01:27<00:12, 7.52it/s]
Requesting API: 81%|████████▏ | 407/500 [01:28<00:16, 5.63it/s]
Requesting API: 82%|████████▏ | 408/500 [01:28<00:16, 5.59it/s]
Requesting API: 82%|████████▏ | 411/500 [01:28<00:12, 7.36it/s]
Requesting API: 82%|████████▏ | 412/500 [01:29<00:17, 4.97it/s]
Requesting API: 83%|████████▎ | 413/500 [01:29<00:19, 4.39it/s]
Requesting API: 83%|████████▎ | 416/500 [01:30<00:12, 6.50it/s]
Requesting API: 83%|████████▎ | 417/500 [01:30<00:12, 6.46it/s]
Requesting API: 84%|████████▍ | 419/500 [01:30<00:11, 7.14it/s]
Requesting API: 84%|████████▍ | 421/500 [01:31<00:18, 4.25it/s]
Requesting API: 85%|████████▍ | 424/500 [01:31<00:12, 6.08it/s]
Requesting API: 85%|████████▌ | 425/500 [01:31<00:14, 5.02it/s]
Requesting API: 85%|████████▌ | 426/500 [01:32<00:14, 5.22it/s]
Requesting API: 86%|████████▌ | 429/500 [01:32<00:10, 6.98it/s]
Requesting API: 86%|████████▌ | 430/500 [01:32<00:10, 6.86it/s]
Requesting API: 86%|████████▌ | 431/500 [01:32<00:13, 5.21it/s]
Requesting API: 86%|████████▋ | 432/500 [01:33<00:13, 4.88it/s]
Requesting API: 87%|████████▋ | 433/500 [01:33<00:17, 3.93it/s]
Requesting API: 87%|████████▋ | 435/500 [01:33<00:16, 3.96it/s]
Requesting API: 87%|████████▋ | 437/500 [01:34<00:12, 5.16it/s]
Requesting API: 88%|████████▊ | 439/500 [01:34<00:15, 4.06it/s]
Requesting API: 89%|████████▊ | 443/500 [01:35<00:08, 6.79it/s]
Requesting API: 89%|████████▉ | 445/500 [01:35<00:07, 6.89it/s]
Requesting API: 90%|████████▉ | 448/500 [01:35<00:07, 7.28it/s]
Requesting API: 90%|█████████ | 451/500 [01:35<00:05, 8.20it/s]
Requesting API: 90%|█████████ | 452/500 [01:36<00:06, 7.65it/s]
Requesting API: 91%|█████████ | 453/500 [01:36<00:06, 7.14it/s]
Requesting API: 91%|█████████ | 455/500 [01:36<00:08, 5.44it/s]
Requesting API: 92%|█████████▏| 458/500 [01:37<00:07, 5.83it/s]
Requesting API: 92%|█████████▏| 460/500 [01:37<00:05, 6.71it/s]
Requesting API: 92%|█████████▏| 461/500 [01:37<00:06, 6.45it/s]
Requesting API: 92%|█████████▏| 462/500 [01:37<00:06, 6.24it/s]
Requesting API: 93%|█████████▎| 463/500 [01:38<00:06, 5.40it/s]
Requesting API: 93%|█████████▎| 464/500 [01:38<00:06, 5.41it/s]
Requesting API: 93%|█████████▎| 465/500 [01:38<00:06, 5.37it/s]
Requesting API: 93%|█████████▎| 466/500 [01:38<00:06, 5.39it/s]
Requesting API: 93%|█████████▎| 467/500 [01:39<00:06, 4.77it/s]
Requesting API: 94%|█████████▎| 468/500 [01:39<00:11, 2.80it/s]
Requesting API: 94%|█████████▍| 469/500 [01:41<00:24, 1.29it/s]
Requesting API: 94%|█████████▍| 470/500 [01:42<00:21, 1.40it/s]
Requesting API: 94%|█████████▍| 472/500 [01:42<00:13, 2.11it/s]
Requesting API: 95%|█████████▍| 473/500 [01:43<00:15, 1.71it/s]
Requesting API: 95%|█████████▍| 474/500 [01:43<00:13, 1.90it/s]
Requesting API: 95%|█████████▌| 475/500 [01:44<00:12, 1.96it/s]
Requesting API: 95%|█████████▌| 477/500 [01:44<00:09, 2.34it/s]
Requesting API: 96%|█████████▌| 478/500 [01:46<00:16, 1.35it/s]
Requesting API: 96%|█████████▌| 479/500 [01:48<00:20, 1.03it/s]
Requesting API: 96%|█████████▌| 480/500 [01:49<00:18, 1.07it/s]
Requesting API: 96%|█████████▌| 481/500 [01:49<00:15, 1.25it/s]
Requesting API: 96%|█████████▋| 482/500 [01:52<00:26, 1.45s/it]
Requesting API: 97%|█████████▋| 484/500 [01:52<00:13, 1.18it/s]
Requesting API: 97%|█████████▋| 486/500 [01:53<00:08, 1.56it/s]
Requesting API: 98%|█████████▊| 488/500 [01:53<00:05, 2.09it/s]
Requesting API: 98%|█████████▊| 490/500 [01:54<00:04, 2.36it/s]
Requesting API: 99%|█████████▊| 493/500 [01:55<00:02, 2.99it/s]
Requesting API: 99%|█████████▉| 495/500 [01:55<00:01, 3.44it/s]
Requesting API: 99%|█████████▉| 496/500 [01:56<00:01, 3.00it/s]
Requesting API: 99%|█████████▉| 497/500 [01:56<00:01, 2.95it/s]
Requesting API: 100%|█████████▉| 499/500 [01:56<00:00, 4.00it/s]
Requesting API: 100%|██████████| 500/500 [01:56<00:00, 4.29it/s]
2026-07-24:04:28:02 INFO [loggers.evaluation_tracker:247] Saving results aggregated
2026-07-24:04:28:02 INFO [loggers.evaluation_tracker:119] Saving per-task samples to outputs/evals/protocolF/keep25_step0200/student/*.jsonl
local-completions ({'model': 'student', 'base_url': 'http://127.0.0.1:8399/v1/completions', 'tokenizer': 'outputs/healed_dl/keep25_step0200', 'num_concurrent': 64, 'tokenized_requests': False, 'max_retries': 3}), gen_kwargs: ({}), limit: None, num_fewshot: None, batch_size: 1
| Tasks |Version| Filter |n-shot| Metric | |Value | |Stderr|
|---------|------:|-----------|-----:|---------|---|-----:|---|-----:|
|humaneval| 1|create_test| 0|pass@1 |↑ |0.0122|± |0.0086|
|mbpp | 1|none | 3|pass_at_1|↑ |0.0600|± |0.0106|
|