File size: 33,043 Bytes
8717f59 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 | 2026-07-24:12:04:18 INFO [_cli.run:388] Selected Tasks: ['humaneval', 'mbpp']
2026-07-24:12:04:20 INFO [evaluator:214] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2026-07-24:12:04:20 INFO [evaluator:239] Initializing local-completions model, with arguments: {'model': 'student', 'base_url': 'http://127.0.0.1:8399/v1/completions', 'tokenizer': 'outputs/healed_dl/keep50_step0200', 'num_concurrent': 64, 'tokenized_requests': False, 'max_retries': 3}
2026-07-24:12:04:20 INFO [models.openai_completions:42] Remote tokenizer not supported. Using huggingface tokenizer backend.
2026-07-24:12:04:20 INFO [models.api_models:179] Using max length 2048 - 1
2026-07-24:12:04:20 INFO [models.api_models:200] Using tokenizer huggingface
2026-07-24:12:04:27 INFO [evaluator_utils:446] Selected tasks:
2026-07-24:12:04:27 INFO [evaluator_utils:480] Task: humaneval (humaneval/humaneval.yaml)
2026-07-24:12:04:27 INFO [evaluator_utils:480] Task: mbpp (mbpp/mbpp.yaml)
2026-07-24:12:04:27 INFO [evaluator:314] humaneval: Using gen_kwargs: {'until': ['\nclass', '\ndef', '\n#', '\nif', '\nprint'], 'max_gen_toks': 1024, 'do_sample': False}
2026-07-24:12:04:27 INFO [evaluator:314] mbpp: Using gen_kwargs: {'until': ['[DONE]'], 'do_sample': False}
2026-07-24:12:04:27 INFO [api.task:312] Building contexts for humaneval on rank 0...
0%| | 0/164 [00:00<?, ?it/s]
88%|████████▊ | 144/164 [00:00<00:00, 1433.62it/s]
100%|██████████| 164/164 [00:00<00:00, 1465.32it/s]
2026-07-24:12:04:27 INFO [api.task:312] Building contexts for mbpp on rank 0...
0%| | 0/500 [00:00<?, ?it/s]
3%|▎ | 17/500 [00:00<00:03, 160.71it/s]
7%|▋ | 34/500 [00:00<00:02, 161.88it/s]
10%|█ | 51/500 [00:00<00:02, 162.18it/s]
14%|█▎ | 68/500 [00:00<00:02, 162.44it/s]
17%|█▋ | 85/500 [00:00<00:02, 162.45it/s]
20%|██ | 102/500 [00:00<00:02, 162.83it/s]
24%|██▍ | 119/500 [00:00<00:02, 163.08it/s]
27%|██▋ | 136/500 [00:00<00:02, 163.08it/s]
31%|███ | 153/500 [00:00<00:02, 163.16it/s]
34%|███▍ | 170/500 [00:01<00:02, 163.34it/s]
37%|███▋ | 187/500 [00:01<00:01, 163.46it/s]
41%|████ | 204/500 [00:01<00:01, 163.56it/s]
44%|████▍ | 221/500 [00:01<00:01, 163.69it/s]
48%|████▊ | 238/500 [00:01<00:01, 163.43it/s]
51%|█████ | 255/500 [00:01<00:01, 163.41it/s]
54%|█████▍ | 272/500 [00:01<00:01, 163.41it/s]
58%|█████▊ | 289/500 [00:01<00:01, 163.52it/s]
61%|██████ | 306/500 [00:01<00:01, 163.58it/s]
65%|██████▍ | 323/500 [00:01<00:01, 163.74it/s]
68%|██████▊ | 340/500 [00:02<00:00, 163.81it/s]
71%|███████▏ | 357/500 [00:02<00:00, 163.83it/s]
75%|███████▍ | 374/500 [00:02<00:00, 164.02it/s]
78%|███████▊ | 391/500 [00:02<00:00, 164.06it/s]
82%|████████▏ | 408/500 [00:02<00:00, 163.75it/s]
85%|████████▌ | 425/500 [00:02<00:00, 163.77it/s]
88%|████████▊ | 442/500 [00:02<00:00, 163.88it/s]
92%|█████████▏| 459/500 [00:02<00:00, 163.93it/s]
95%|█████████▌| 476/500 [00:02<00:00, 163.93it/s]
99%|█████████▊| 493/500 [00:03<00:00, 164.07it/s]
100%|██████████| 500/500 [00:03<00:00, 163.49it/s]
2026-07-24:12:04:30 INFO [evaluator:585] Running generate_until requests
2026-07-24:12:04:30 INFO [models.api_models:747] Tokenized requests are disabled. Context + generation length is not checked.
Requesting API: 0%| | 0/164 [00:00<?, ?it/s]
Requesting API: 1%| | 1/164 [00:02<06:48, 2.51s/it]
Requesting API: 1%| | 2/164 [00:02<03:23, 1.26s/it]
Requesting API: 4%|▎ | 6/164 [00:03<01:02, 2.51it/s]
Requesting API: 4%|▍ | 7/164 [00:03<00:55, 2.83it/s]
Requesting API: 5%|▌ | 9/164 [00:03<00:37, 4.11it/s]
Requesting API: 6%|▌ | 10/164 [00:04<00:41, 3.69it/s]
Requesting API: 7%|▋ | 11/164 [00:04<00:38, 3.95it/s]
Requesting API: 7%|▋ | 12/164 [00:04<00:42, 3.57it/s]
Requesting API: 8%|▊ | 13/164 [00:06<01:54, 1.32it/s]
Requesting API: 9%|▊ | 14/164 [00:07<01:30, 1.65it/s]
Requesting API: 10%|▉ | 16/164 [00:07<01:02, 2.37it/s]
Requesting API: 10%|█ | 17/164 [00:07<00:54, 2.72it/s]
Requesting API: 11%|█ | 18/164 [00:07<00:44, 3.25it/s]
Requesting API: 12%|█▏ | 19/164 [00:08<01:13, 1.98it/s]
Requesting API: 12%|█▏ | 20/164 [00:08<01:00, 2.38it/s]
Requesting API: 13%|█▎ | 21/164 [00:09<00:48, 2.96it/s]
Requesting API: 13%|█▎ | 22/164 [00:09<00:48, 2.91it/s]
Requesting API: 15%|█▍ | 24/164 [00:11<01:37, 1.44it/s]
Requesting API: 15%|█▌ | 25/164 [00:12<01:47, 1.29it/s]
Requesting API: 16%|█▌ | 26/164 [00:13<01:52, 1.22it/s]
Requesting API: 16%|█▋ | 27/164 [00:16<03:02, 1.33s/it]
Requesting API: 18%|█▊ | 29/164 [00:17<02:00, 1.12it/s]
Requesting API: 18%|█▊ | 30/164 [00:18<02:23, 1.07s/it]
Requesting API: 19%|█▉ | 31/164 [00:19<02:07, 1.04it/s]
Requesting API: 20%|█▉ | 32/164 [00:19<01:39, 1.32it/s]
Requesting API: 20%|██ | 33/164 [00:20<01:46, 1.23it/s]
Requesting API: 21%|██ | 34/164 [00:20<01:23, 1.56it/s]
Requesting API: 21%|██▏ | 35/164 [00:22<02:00, 1.07it/s]
Requesting API: 22%|██▏ | 36/164 [00:22<01:37, 1.31it/s]
Requesting API: 23%|██▎ | 37/164 [00:23<01:46, 1.19it/s]
Requesting API: 23%|██▎ | 38/164 [00:24<01:45, 1.20it/s]
Requesting API: 24%|██▍ | 39/164 [00:25<02:02, 1.02it/s]
Requesting API: 24%|██▍ | 40/164 [00:26<01:48, 1.14it/s]
Requesting API: 25%|██▌ | 41/164 [00:28<02:18, 1.13s/it]
Requesting API: 26%|██▌ | 42/164 [00:29<02:10, 1.07s/it]
Requesting API: 27%|██▋ | 44/164 [00:31<02:11, 1.09s/it]
Requesting API: 29%|██▊ | 47/164 [00:32<01:19, 1.48it/s]
Requesting API: 30%|██▉ | 49/164 [00:32<00:56, 2.03it/s]
Requesting API: 30%|███ | 50/164 [00:32<00:55, 2.04it/s]
Requesting API: 31%|███ | 51/164 [00:34<01:12, 1.55it/s]
Requesting API: 32%|███▏ | 53/164 [00:35<01:12, 1.52it/s]
Requesting API: 33%|███▎ | 54/164 [00:37<01:36, 1.14it/s]
Requesting API: 34%|███▎ | 55/164 [00:41<03:05, 1.71s/it]
Requesting API: 35%|███▍ | 57/164 [00:45<03:03, 1.72s/it]
Requesting API: 35%|███▌ | 58/164 [00:46<02:47, 1.58s/it]
Requesting API: 36%|███▌ | 59/164 [00:51<04:27, 2.55s/it]
Requesting API: 37%|███▋ | 60/164 [00:56<05:19, 3.07s/it]
Requesting API: 37%|███▋ | 61/164 [00:59<05:15, 3.07s/it]
Requesting API: 38%|███▊ | 62/164 [01:44<24:49, 14.60s/it]
Requesting API: 43%|████▎ | 71/164 [01:44<05:00, 3.24s/it]
Requesting API: 48%|████▊ | 79/164 [01:44<02:18, 1.63s/it]
Requesting API: 52%|█████▏ | 85/164 [01:44<01:24, 1.08s/it]
Requesting API: 57%|█████▋ | 93/164 [01:45<00:46, 1.52it/s]
Requesting API: 61%|██████ | 100/164 [01:45<00:28, 2.24it/s]
Requesting API: 63%|██████▎ | 104/164 [01:45<00:23, 2.56it/s]
Requesting API: 65%|██████▌ | 107/164 [01:46<00:18, 3.03it/s]
Requesting API: 67%|██████▋ | 110/164 [01:46<00:14, 3.64it/s]
Requesting API: 69%|██████▉ | 113/164 [01:46<00:11, 4.42it/s]
Requesting API: 72%|███████▏ | 118/164 [01:46<00:07, 6.26it/s]
Requesting API: 74%|███████▍ | 121/164 [01:47<00:06, 6.58it/s]
Requesting API: 75%|███████▌ | 123/164 [01:47<00:06, 6.66it/s]
Requesting API: 76%|███████▌ | 125/164 [01:47<00:05, 6.75it/s]
Requesting API: 77%|███████▋ | 127/164 [01:49<00:11, 3.28it/s]
Requesting API: 78%|███████▊ | 128/164 [01:49<00:10, 3.48it/s]
Requesting API: 79%|███████▊ | 129/164 [01:49<00:10, 3.49it/s]
Requesting API: 80%|███████▉ | 131/164 [01:49<00:07, 4.47it/s]
Requesting API: 80%|████████ | 132/164 [01:50<00:06, 4.63it/s]
Requesting API: 81%|████████ | 133/164 [01:50<00:07, 4.03it/s]
Requesting API: 82%|████████▏ | 134/164 [01:51<00:10, 2.94it/s]
Requesting API: 82%|████████▏ | 135/164 [01:51<00:10, 2.69it/s]
Requesting API: 83%|████████▎ | 136/164 [01:53<00:19, 1.46it/s]
Requesting API: 84%|████████▎ | 137/164 [01:53<00:16, 1.67it/s]
Requesting API: 84%|████████▍ | 138/164 [01:54<00:16, 1.62it/s]
Requesting API: 85%|████████▍ | 139/164 [01:57<00:36, 1.48s/it]
Requesting API: 85%|████████▌ | 140/164 [02:01<00:50, 2.09s/it]
Requesting API: 86%|████████▌ | 141/164 [02:03<00:47, 2.05s/it]
Requesting API: 87%|████████▋ | 142/164 [02:08<01:03, 2.91s/it]
Requesting API: 87%|████████▋ | 143/164 [02:09<00:48, 2.29s/it]
Requesting API: 88%|████████▊ | 144/164 [02:11<00:48, 2.40s/it]
Requesting API: 88%|████████▊ | 145/164 [02:12<00:35, 1.85s/it]
Requesting API: 89%|████████▉ | 146/164 [02:16<00:44, 2.45s/it]
Requesting API: 90%|████████▉ | 147/164 [02:20<00:50, 2.95s/it]
Requesting API: 90%|█████████ | 148/164 [02:21<00:37, 2.34s/it]
Requesting API: 91%|█████████ | 149/164 [02:41<01:55, 7.72s/it]
Requesting API: 91%|█████████▏| 150/164 [03:18<03:52, 16.59s/it]
Requesting API: 95%|█████████▍| 155/164 [03:19<00:47, 5.31s/it]
Requesting API: 97%|█████████▋| 159/164 [03:19<00:15, 3.00s/it]
Requesting API: 100%|██████████| 164/164 [03:19<00:00, 1.22s/it]
Requesting API: 0%| | 0/500 [00:00<?, ?it/s]
Requesting API: 0%| | 1/500 [00:08<1:12:44, 8.75s/it]
Requesting API: 1%| | 3/500 [00:08<19:21, 2.34s/it]
Requesting API: 1%| | 6/500 [00:09<08:56, 1.09s/it]
Requesting API: 1%|▏ | 7/500 [00:10<08:00, 1.03it/s]
Requesting API: 2%|▏ | 8/500 [00:10<06:24, 1.28it/s]
Requesting API: 2%|▏ | 9/500 [00:10<05:09, 1.59it/s]
Requesting API: 3%|▎ | 13/500 [00:11<02:49, 2.87it/s]
Requesting API: 3%|▎ | 14/500 [00:11<02:39, 3.05it/s]
Requesting API: 3%|▎ | 16/500 [00:11<02:10, 3.71it/s]
Requesting API: 4%|▎ | 18/500 [00:12<02:21, 3.41it/s]
Requesting API: 4%|▍ | 20/500 [00:12<01:50, 4.34it/s]
Requesting API: 4%|▍ | 21/500 [00:13<01:48, 4.42it/s]
Requesting API: 5%|▍ | 23/500 [00:13<01:28, 5.42it/s]
Requesting API: 5%|▌ | 26/500 [00:13<01:23, 5.68it/s]
Requesting API: 6%|▌ | 28/500 [00:14<01:21, 5.80it/s]
Requesting API: 6%|▌ | 30/500 [00:14<01:07, 6.94it/s]
Requesting API: 6%|▋ | 32/500 [00:14<01:07, 6.95it/s]
Requesting API: 7%|▋ | 35/500 [00:15<01:43, 4.51it/s]
Requesting API: 7%|▋ | 37/500 [00:15<01:27, 5.32it/s]
Requesting API: 8%|▊ | 39/500 [00:15<01:15, 6.07it/s]
Requesting API: 8%|▊ | 40/500 [00:16<01:19, 5.81it/s]
Requesting API: 9%|▉ | 44/500 [00:16<00:56, 8.11it/s]
Requesting API: 9%|▉ | 46/500 [00:16<01:12, 6.25it/s]
Requesting API: 9%|▉ | 47/500 [00:17<01:49, 4.16it/s]
Requesting API: 10%|▉ | 49/500 [00:17<01:28, 5.11it/s]
Requesting API: 10%|█ | 50/500 [00:18<01:29, 5.02it/s]
Requesting API: 11%|█ | 54/500 [00:18<00:54, 8.16it/s]
Requesting API: 11%|█ | 56/500 [00:18<00:52, 8.49it/s]
Requesting API: 12%|█▏ | 58/500 [00:18<00:57, 7.68it/s]
Requesting API: 12%|█▏ | 60/500 [00:19<01:21, 5.38it/s]
Requesting API: 12%|█▏ | 62/500 [00:19<01:21, 5.35it/s]
Requesting API: 13%|█▎ | 63/500 [00:20<01:29, 4.90it/s]
Requesting API: 13%|█▎ | 66/500 [00:20<01:08, 6.30it/s]
Requesting API: 14%|█▎ | 68/500 [00:21<01:47, 4.01it/s]
Requesting API: 14%|█▍ | 72/500 [00:21<01:21, 5.25it/s]
Requesting API: 15%|█▍ | 74/500 [00:22<01:15, 5.62it/s]
Requesting API: 15%|█▌ | 75/500 [00:22<01:28, 4.82it/s]
Requesting API: 15%|█▌ | 77/500 [00:22<01:14, 5.67it/s]
Requesting API: 16%|█▌ | 78/500 [00:23<01:28, 4.78it/s]
Requesting API: 16%|█▌ | 79/500 [00:23<01:27, 4.79it/s]
Requesting API: 16%|█▌ | 81/500 [00:23<01:18, 5.32it/s]
Requesting API: 17%|█▋ | 83/500 [00:23<01:05, 6.33it/s]
Requesting API: 18%|█▊ | 88/500 [00:24<00:50, 8.13it/s]
Requesting API: 18%|█▊ | 90/500 [00:24<01:08, 6.00it/s]
Requesting API: 18%|█▊ | 92/500 [00:25<01:10, 5.82it/s]
Requesting API: 19%|█▊ | 93/500 [00:25<01:40, 4.03it/s]
Requesting API: 19%|█▉ | 95/500 [00:26<01:34, 4.28it/s]
Requesting API: 19%|█▉ | 96/500 [00:26<01:31, 4.41it/s]
Requesting API: 19%|█▉ | 97/500 [00:27<02:02, 3.30it/s]
Requesting API: 20%|██ | 101/500 [00:27<01:06, 6.01it/s]
Requesting API: 20%|██ | 102/500 [00:27<01:08, 5.79it/s]
Requesting API: 21%|██ | 103/500 [00:28<01:35, 4.14it/s]
Requesting API: 21%|██ | 104/500 [00:28<01:31, 4.32it/s]
Requesting API: 21%|██ | 106/500 [00:28<01:19, 4.96it/s]
Requesting API: 22%|██▏ | 108/500 [00:28<01:05, 5.98it/s]
Requesting API: 22%|██▏ | 110/500 [00:28<00:55, 6.97it/s]
Requesting API: 22%|██▏ | 112/500 [00:29<00:55, 7.00it/s]
Requesting API: 23%|██▎ | 113/500 [00:29<01:05, 5.91it/s]
Requesting API: 23%|██▎ | 114/500 [00:30<01:54, 3.37it/s]
Requesting API: 23%|██▎ | 116/500 [00:30<01:39, 3.86it/s]
Requesting API: 24%|██▍ | 120/500 [00:31<01:17, 4.88it/s]
Requesting API: 24%|██▍ | 122/500 [00:31<01:12, 5.23it/s]
Requesting API: 25%|██▍ | 123/500 [00:31<01:23, 4.51it/s]
Requesting API: 25%|██▌ | 125/500 [00:32<01:13, 5.07it/s]
Requesting API: 25%|██▌ | 126/500 [00:32<01:19, 4.69it/s]
Requesting API: 25%|██▌ | 127/500 [00:33<01:48, 3.43it/s]
Requesting API: 26%|██▌ | 128/500 [00:33<01:40, 3.71it/s]
Requesting API: 26%|██▌ | 129/500 [00:33<01:41, 3.64it/s]
Requesting API: 26%|██▌ | 130/500 [00:33<01:35, 3.89it/s]
Requesting API: 27%|██▋ | 134/500 [00:34<01:09, 5.29it/s]
Requesting API: 27%|██▋ | 135/500 [00:34<01:20, 4.54it/s]
Requesting API: 27%|██▋ | 136/500 [00:34<01:18, 4.65it/s]
Requesting API: 28%|██▊ | 138/500 [00:35<01:28, 4.11it/s]
Requesting API: 28%|██▊ | 140/500 [00:35<01:09, 5.17it/s]
Requesting API: 28%|██▊ | 141/500 [00:35<01:09, 5.16it/s]
Requesting API: 28%|██▊ | 142/500 [00:36<01:25, 4.18it/s]
Requesting API: 29%|██▊ | 143/500 [00:36<01:21, 4.38it/s]
Requesting API: 29%|██▉ | 144/500 [00:36<01:27, 4.08it/s]
Requesting API: 29%|██▉ | 145/500 [00:37<01:31, 3.89it/s]
Requesting API: 29%|██▉ | 146/500 [00:37<01:35, 3.69it/s]
Requesting API: 30%|██▉ | 149/500 [00:37<00:56, 6.25it/s]
Requesting API: 30%|███ | 152/500 [00:38<01:06, 5.24it/s]
Requesting API: 31%|███ | 153/500 [00:38<01:06, 5.23it/s]
Requesting API: 31%|███ | 155/500 [00:38<01:01, 5.61it/s]
Requesting API: 31%|███▏ | 157/500 [00:39<01:01, 5.57it/s]
Requesting API: 32%|███▏ | 158/500 [00:39<01:14, 4.62it/s]
Requesting API: 32%|███▏ | 159/500 [00:39<01:19, 4.30it/s]
Requesting API: 32%|███▏ | 161/500 [00:40<01:02, 5.43it/s]
Requesting API: 32%|███▏ | 162/500 [00:40<01:36, 3.52it/s]
Requesting API: 33%|███▎ | 164/500 [00:41<01:40, 3.33it/s]
Requesting API: 33%|███▎ | 165/500 [00:42<02:08, 2.60it/s]
Requesting API: 34%|███▎ | 168/500 [00:42<01:48, 3.07it/s]
Requesting API: 34%|███▍ | 169/500 [00:43<01:39, 3.31it/s]
Requesting API: 34%|███▍ | 170/500 [00:43<01:33, 3.54it/s]
Requesting API: 35%|███▍ | 174/500 [00:44<01:19, 4.12it/s]
Requesting API: 36%|███▌ | 180/500 [00:44<00:41, 7.71it/s]
Requesting API: 36%|███▋ | 182/500 [00:45<01:04, 4.96it/s]
Requesting API: 37%|███▋ | 186/500 [00:46<01:03, 4.94it/s]
Requesting API: 38%|███▊ | 188/500 [00:46<01:02, 4.98it/s]
Requesting API: 38%|███▊ | 189/500 [00:46<01:06, 4.70it/s]
Requesting API: 38%|███▊ | 190/500 [00:46<01:05, 4.76it/s]
Requesting API: 38%|███▊ | 192/500 [00:47<00:58, 5.24it/s]
Requesting API: 39%|███▊ | 193/500 [00:47<01:25, 3.59it/s]
Requesting API: 39%|███▉ | 195/500 [00:48<01:25, 3.58it/s]
Requesting API: 39%|███▉ | 197/500 [00:48<01:20, 3.78it/s]
Requesting API: 40%|███▉ | 199/500 [00:49<01:30, 3.32it/s]
Requesting API: 40%|████ | 202/500 [00:49<01:01, 4.86it/s]
Requesting API: 41%|████ | 205/500 [00:50<00:54, 5.41it/s]
Requesting API: 41%|████ | 206/500 [00:50<01:00, 4.87it/s]
Requesting API: 41%|████▏ | 207/500 [00:50<00:56, 5.19it/s]
Requesting API: 42%|████▏ | 209/500 [00:51<00:56, 5.19it/s]
Requesting API: 42%|████▏ | 210/500 [00:52<01:48, 2.67it/s]
Requesting API: 43%|████▎ | 213/500 [00:52<01:07, 4.26it/s]
Requesting API: 43%|████▎ | 215/500 [00:52<00:54, 5.18it/s]
Requesting API: 43%|████▎ | 216/500 [00:52<00:58, 4.86it/s]
Requesting API: 44%|████▎ | 218/500 [00:53<01:11, 3.94it/s]
Requesting API: 44%|████▍ | 221/500 [00:54<01:10, 3.98it/s]
Requesting API: 45%|████▍ | 223/500 [00:54<00:57, 4.81it/s]
Requesting API: 45%|████▌ | 225/500 [00:55<01:09, 3.95it/s]
Requesting API: 46%|████▌ | 228/500 [00:55<01:05, 4.17it/s]
Requesting API: 46%|████▌ | 230/500 [00:56<01:01, 4.40it/s]
Requesting API: 46%|████▋ | 232/500 [00:56<00:54, 4.90it/s]
Requesting API: 47%|████▋ | 233/500 [00:57<01:09, 3.84it/s]
Requesting API: 47%|████▋ | 235/500 [00:57<01:03, 4.18it/s]
Requesting API: 47%|████▋ | 237/500 [00:57<00:50, 5.16it/s]
Requesting API: 48%|████▊ | 239/500 [00:58<00:49, 5.27it/s]
Requesting API: 48%|████▊ | 240/500 [00:58<00:49, 5.26it/s]
Requesting API: 48%|████▊ | 242/500 [00:58<00:40, 6.36it/s]
Requesting API: 49%|████▊ | 243/500 [00:58<00:42, 6.10it/s]
Requesting API: 49%|████▉ | 245/500 [00:58<00:38, 6.64it/s]
Requesting API: 49%|████▉ | 246/500 [00:59<00:40, 6.31it/s]
Requesting API: 49%|████▉ | 247/500 [00:59<00:46, 5.48it/s]
Requesting API: 50%|████▉ | 249/500 [00:59<00:37, 6.76it/s]
Requesting API: 50%|█████ | 250/500 [01:00<00:52, 4.72it/s]
Requesting API: 50%|█████ | 251/500 [01:00<01:08, 3.64it/s]
Requesting API: 51%|█████ | 253/500 [01:00<00:50, 4.92it/s]
Requesting API: 51%|█████ | 254/500 [01:01<00:58, 4.19it/s]
Requesting API: 51%|█████ | 255/500 [01:01<00:55, 4.40it/s]
Requesting API: 51%|█████▏ | 257/500 [01:02<01:15, 3.21it/s]
Requesting API: 52%|█████▏ | 262/500 [01:02<00:52, 4.55it/s]
Requesting API: 53%|█████▎ | 263/500 [01:03<00:51, 4.62it/s]
Requesting API: 53%|█████▎ | 264/500 [01:03<00:50, 4.70it/s]
Requesting API: 53%|█████▎ | 266/500 [01:03<00:54, 4.28it/s]
Requesting API: 54%|█████▎ | 268/500 [01:04<00:51, 4.53it/s]
Requesting API: 54%|█████▍ | 269/500 [01:04<00:53, 4.29it/s]
Requesting API: 54%|█████▍ | 270/500 [01:04<00:51, 4.47it/s]
Requesting API: 54%|█████▍ | 271/500 [01:04<00:44, 5.11it/s]
Requesting API: 55%|█████▍ | 273/500 [01:05<00:35, 6.44it/s]
Requesting API: 55%|█████▍ | 274/500 [01:05<01:14, 3.04it/s]
Requesting API: 55%|█████▌ | 276/500 [01:06<00:49, 4.53it/s]
Requesting API: 56%|█████▌ | 279/500 [01:06<00:38, 5.74it/s]
Requesting API: 56%|█████▌ | 280/500 [01:06<00:41, 5.31it/s]
Requesting API: 56%|█████▌ | 281/500 [01:06<00:41, 5.31it/s]
Requesting API: 56%|█████▋ | 282/500 [01:07<00:41, 5.29it/s]
Requesting API: 57%|█████▋ | 284/500 [01:07<00:32, 6.56it/s]
Requesting API: 57%|█████▋ | 285/500 [01:07<00:34, 6.24it/s]
Requesting API: 57%|█████▋ | 286/500 [01:07<00:48, 4.39it/s]
Requesting API: 57%|█████▋ | 287/500 [01:08<00:51, 4.12it/s]
Requesting API: 58%|█████▊ | 288/500 [01:08<00:48, 4.37it/s]
Requesting API: 58%|█████▊ | 290/500 [01:08<00:40, 5.20it/s]
Requesting API: 58%|█████▊ | 292/500 [01:09<00:39, 5.24it/s]
Requesting API: 59%|█████▊ | 293/500 [01:09<00:46, 4.45it/s]
Requesting API: 59%|█████▉ | 294/500 [01:09<00:44, 4.62it/s]
Requesting API: 59%|█████▉ | 295/500 [01:09<00:48, 4.26it/s]
Requesting API: 59%|█████▉ | 297/500 [01:10<00:35, 5.66it/s]
Requesting API: 60%|█████▉ | 299/500 [01:10<00:35, 5.67it/s]
Requesting API: 60%|██████ | 301/500 [01:10<00:31, 6.29it/s]
Requesting API: 60%|██████ | 302/500 [01:11<00:50, 3.90it/s]
Requesting API: 61%|██████ | 303/500 [01:11<00:47, 4.11it/s]
Requesting API: 61%|██████ | 305/500 [01:12<00:49, 3.95it/s]
Requesting API: 61%|██████▏ | 307/500 [01:12<00:38, 5.04it/s]
Requesting API: 62%|██████▏ | 308/500 [01:12<00:52, 3.65it/s]
Requesting API: 62%|██████▏ | 309/500 [01:13<00:57, 3.35it/s]
Requesting API: 62%|██████▏ | 312/500 [01:13<00:34, 5.43it/s]
Requesting API: 63%|██████▎ | 313/500 [01:14<00:53, 3.48it/s]
Requesting API: 63%|██████▎ | 314/500 [01:14<00:49, 3.72it/s]
Requesting API: 63%|██████▎ | 317/500 [01:14<00:31, 5.87it/s]
Requesting API: 64%|██████▍ | 319/500 [01:15<00:38, 4.70it/s]
Requesting API: 64%|██████▍ | 320/500 [01:15<00:37, 4.80it/s]
Requesting API: 64%|██████▍ | 321/500 [01:15<00:47, 3.80it/s]
Requesting API: 65%|██████▍ | 324/500 [01:15<00:30, 5.86it/s]
Requesting API: 65%|██████▌ | 325/500 [01:16<00:30, 5.75it/s]
Requesting API: 65%|██████▌ | 326/500 [01:16<00:33, 5.19it/s]
Requesting API: 65%|██████▌ | 327/500 [01:16<00:33, 5.22it/s]
Requesting API: 66%|██████▌ | 328/500 [01:16<00:40, 4.20it/s]
Requesting API: 66%|██████▌ | 329/500 [01:17<00:38, 4.44it/s]
Requesting API: 66%|██████▌ | 330/500 [01:17<00:41, 4.12it/s]
Requesting API: 66%|██████▌ | 331/500 [01:17<00:38, 4.38it/s]
Requesting API: 67%|██████▋ | 334/500 [01:17<00:25, 6.41it/s]
Requesting API: 67%|██████▋ | 336/500 [01:18<00:36, 4.52it/s]
Requesting API: 68%|██████▊ | 338/500 [01:18<00:29, 5.54it/s]
Requesting API: 68%|██████▊ | 339/500 [01:19<00:43, 3.67it/s]
Requesting API: 68%|██████▊ | 342/500 [01:19<00:32, 4.82it/s]
Requesting API: 69%|██████▊ | 343/500 [01:20<00:41, 3.75it/s]
Requesting API: 69%|██████▉ | 345/500 [01:20<00:39, 3.96it/s]
Requesting API: 70%|██████▉ | 348/500 [01:21<00:39, 3.80it/s]
Requesting API: 71%|███████ | 353/500 [01:21<00:22, 6.62it/s]
Requesting API: 71%|███████ | 355/500 [01:22<00:26, 5.51it/s]
Requesting API: 71%|███████▏ | 357/500 [01:23<00:30, 4.74it/s]
Requesting API: 72%|███████▏ | 359/500 [01:23<00:32, 4.33it/s]
Requesting API: 72%|███████▏ | 361/500 [01:23<00:27, 5.11it/s]
Requesting API: 72%|███████▏ | 362/500 [01:24<00:37, 3.69it/s]
Requesting API: 73%|███████▎ | 366/500 [01:24<00:22, 6.07it/s]
Requesting API: 74%|███████▎ | 368/500 [01:24<00:20, 6.45it/s]
Requesting API: 74%|███████▍ | 369/500 [01:25<00:22, 5.87it/s]
Requesting API: 74%|███████▍ | 371/500 [01:25<00:23, 5.40it/s]
Requesting API: 74%|███████▍ | 372/500 [01:25<00:25, 5.06it/s]
Requesting API: 75%|███████▍ | 373/500 [01:26<00:27, 4.63it/s]
Requesting API: 75%|███████▌ | 376/500 [01:26<00:19, 6.28it/s]
Requesting API: 75%|███████▌ | 377/500 [01:27<00:29, 4.16it/s]
Requesting API: 76%|███████▌ | 378/500 [01:27<00:28, 4.32it/s]
Requesting API: 76%|███████▌ | 381/500 [01:27<00:19, 5.98it/s]
Requesting API: 76%|███████▋ | 382/500 [01:28<00:29, 4.02it/s]
Requesting API: 77%|███████▋ | 386/500 [01:28<00:21, 5.32it/s]
Requesting API: 77%|███████▋ | 387/500 [01:28<00:21, 5.31it/s]
Requesting API: 78%|███████▊ | 390/500 [01:29<00:20, 5.41it/s]
Requesting API: 78%|███████▊ | 391/500 [01:29<00:21, 4.97it/s]
Requesting API: 79%|███████▊ | 393/500 [01:29<00:17, 5.96it/s]
Requesting API: 79%|███████▉ | 397/500 [01:30<00:19, 5.27it/s]
Requesting API: 80%|███████▉ | 398/500 [01:31<00:22, 4.59it/s]
Requesting API: 80%|████████ | 400/500 [01:31<00:21, 4.76it/s]
Requesting API: 80%|████████ | 401/500 [01:31<00:20, 4.84it/s]
Requesting API: 80%|████████ | 402/500 [01:32<00:21, 4.50it/s]
Requesting API: 81%|████████ | 403/500 [01:32<00:20, 4.66it/s]
Requesting API: 81%|████████ | 404/500 [01:32<00:22, 4.31it/s]
Requesting API: 81%|████████ | 406/500 [01:32<00:16, 5.69it/s]
Requesting API: 81%|████████▏ | 407/500 [01:33<00:19, 4.65it/s]
Requesting API: 82%|████████▏ | 409/500 [01:33<00:15, 5.97it/s]
Requesting API: 82%|████████▏ | 411/500 [01:34<00:24, 3.67it/s]
Requesting API: 83%|████████▎ | 414/500 [01:34<00:18, 4.76it/s]
Requesting API: 83%|████████▎ | 417/500 [01:34<00:13, 6.19it/s]
Requesting API: 84%|████████▍ | 419/500 [01:35<00:14, 5.74it/s]
Requesting API: 84%|████████▍ | 422/500 [01:35<00:10, 7.50it/s]
Requesting API: 85%|████████▍ | 423/500 [01:35<00:10, 7.06it/s]
Requesting API: 85%|████████▌ | 425/500 [01:35<00:11, 6.78it/s]
Requesting API: 86%|████████▌ | 428/500 [01:36<00:08, 8.76it/s]
Requesting API: 86%|████████▌ | 430/500 [01:36<00:13, 5.03it/s]
Requesting API: 86%|████████▌ | 431/500 [01:37<00:14, 4.69it/s]
Requesting API: 87%|████████▋ | 434/500 [01:37<00:09, 6.60it/s]
Requesting API: 87%|████████▋ | 436/500 [01:37<00:10, 5.96it/s]
Requesting API: 87%|████████▋ | 437/500 [01:38<00:10, 5.84it/s]
Requesting API: 88%|████████▊ | 439/500 [01:38<00:10, 5.66it/s]
Requesting API: 88%|████████▊ | 442/500 [01:38<00:07, 7.65it/s]
Requesting API: 89%|████████▉ | 446/500 [01:38<00:05, 9.53it/s]
Requesting API: 90%|████████▉ | 448/500 [01:39<00:08, 6.25it/s]
Requesting API: 90%|█████████ | 452/500 [01:39<00:05, 8.74it/s]
Requesting API: 91%|█████████ | 454/500 [01:40<00:06, 7.62it/s]
Requesting API: 91%|█████████ | 456/500 [01:40<00:05, 8.17it/s]
Requesting API: 92%|█████████▏| 458/500 [01:40<00:05, 7.20it/s]
Requesting API: 92%|█████████▏| 459/500 [01:40<00:06, 6.82it/s]
Requesting API: 92%|█████████▏| 461/500 [01:41<00:06, 6.29it/s]
Requesting API: 93%|█████████▎| 463/500 [01:41<00:07, 4.77it/s]
Requesting API: 93%|█████████▎| 464/500 [01:42<00:08, 4.18it/s]
Requesting API: 93%|█████████▎| 465/500 [01:42<00:09, 3.50it/s]
Requesting API: 93%|█████████▎| 466/500 [01:43<00:12, 2.72it/s]
Requesting API: 93%|█████████▎| 467/500 [01:44<00:19, 1.68it/s]
Requesting API: 94%|█████████▎| 468/500 [01:44<00:16, 1.95it/s]
Requesting API: 94%|█████████▍| 469/500 [01:45<00:13, 2.35it/s]
Requesting API: 94%|█████████▍| 470/500 [01:45<00:12, 2.42it/s]
Requesting API: 94%|█████████▍| 471/500 [01:45<00:10, 2.66it/s]
Requesting API: 94%|█████████▍| 472/500 [01:46<00:12, 2.19it/s]
Requesting API: 95%|█████████▍| 473/500 [01:46<00:10, 2.46it/s]
Requesting API: 95%|█████████▍| 474/500 [01:47<00:12, 2.08it/s]
Requesting API: 95%|█████████▌| 476/500 [01:48<00:13, 1.72it/s]
Requesting API: 95%|█████████▌| 477/500 [01:49<00:13, 1.73it/s]
Requesting API: 96%|█████████▌| 479/500 [01:50<00:11, 1.88it/s]
Requesting API: 96%|█████████▌| 480/500 [01:51<00:12, 1.56it/s]
Requesting API: 96%|█████████▌| 481/500 [01:53<00:17, 1.09it/s]
Requesting API: 96%|█████████▋| 482/500 [01:54<00:17, 1.03it/s]
Requesting API: 97%|█████████▋| 483/500 [01:54<00:14, 1.17it/s]
Requesting API: 97%|█████████▋| 484/500 [01:55<00:12, 1.29it/s]
Requesting API: 97%|█████████▋| 485/500 [01:55<00:11, 1.36it/s]
Requesting API: 97%|█████████▋| 487/500 [01:57<00:08, 1.53it/s]
Requesting API: 98%|█████████▊| 489/500 [01:58<00:07, 1.38it/s]
Requesting API: 98%|█████████▊| 490/500 [01:59<00:07, 1.27it/s]
Requesting API: 98%|█████████▊| 492/500 [02:00<00:04, 1.60it/s]
Requesting API: 99%|█████████▊| 493/500 [02:00<00:03, 1.89it/s]
Requesting API: 99%|█████████▉| 494/500 [02:00<00:02, 2.13it/s]
Requesting API: 99%|█████████▉| 496/500 [02:01<00:01, 3.15it/s]
Requesting API: 100%|█████████▉| 498/500 [02:01<00:00, 4.25it/s]
Requesting API: 100%|█████████▉| 499/500 [02:01<00:00, 3.34it/s]
Requesting API: 100%|██████████| 500/500 [02:01<00:00, 4.10it/s]
2026-07-24:12:12:43 INFO [loggers.evaluation_tracker:247] Saving results aggregated
2026-07-24:12:12:43 INFO [loggers.evaluation_tracker:119] Saving per-task samples to outputs/evals/protocolF/keep50_step0200/student/*.jsonl
local-completions ({'model': 'student', 'base_url': 'http://127.0.0.1:8399/v1/completions', 'tokenizer': 'outputs/healed_dl/keep50_step0200', 'num_concurrent': 64, 'tokenized_requests': False, 'max_retries': 3}), gen_kwargs: ({}), limit: None, num_fewshot: None, batch_size: 1
| Tasks |Version| Filter |n-shot| Metric | |Value | |Stderr|
|---------|------:|-----------|-----:|---------|---|-----:|---|-----:|
|humaneval| 1|create_test| 0|pass@1 |↑ |0.2439|± |0.0336|
|mbpp | 1|none | 3|pass_at_1|↑ |0.3440|± |0.0213|
|