File size: 16,921 Bytes
a82fb9e | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 | 2026-08-08:18:17:39 INFO [_cli.run:388] Selected Tasks: ['humaneval', 'mbpp']
2026-08-08:18:17:40 INFO [evaluator:214] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2026-08-08:18:17:40 INFO [evaluator:239] Initializing local-completions model, with arguments: {'model': 'student', 'base_url': 'http://127.0.0.1:8451/v1/completions', 'tokenizer': 'outputs/release/healed/release-offpolicy-general-keep25/step0150', 'num_concurrent': 48, 'tokenized_requests': False, 'max_retries': 3}
2026-08-08:18:17:40 INFO [models.openai_completions:42] Remote tokenizer not supported. Using huggingface tokenizer backend.
2026-08-08:18:17:40 INFO [models.api_models:179] Using max length 2048 - 1
2026-08-08:18:17:40 INFO [models.api_models:200] Using tokenizer huggingface
2026-08-08:18:17:47 INFO [evaluator_utils:446] Selected tasks:
2026-08-08:18:17:47 INFO [evaluator_utils:480] Task: humaneval (humaneval/humaneval.yaml)
2026-08-08:18:17:47 INFO [evaluator_utils:480] Task: mbpp (mbpp/mbpp.yaml)
2026-08-08:18:17:47 INFO [evaluator:314] humaneval: Using gen_kwargs: {'until': ['\nclass', '\ndef', '\n#', '\nif', '\nprint'], 'max_gen_toks': 1024, 'do_sample': False}
2026-08-08:18:17:47 INFO [evaluator:314] mbpp: Using gen_kwargs: {'until': ['[DONE]'], 'do_sample': False}
2026-08-08:18:17:47 INFO [api.task:312] Building contexts for humaneval on rank 0...
0%| | 0/164 [00:00<?, ?it/s]
90%|████████▉ | 147/164 [00:00<00:00, 1463.06it/s]
100%|██████████| 164/164 [00:00<00:00, 1489.47it/s]
2026-08-08:18:17:47 INFO [api.task:312] Building contexts for mbpp on rank 0...
0%| | 0/500 [00:00<?, ?it/s]
3%|▎ | 17/500 [00:00<00:03, 160.29it/s]
7%|▋ | 34/500 [00:00<00:02, 163.07it/s]
10%|█ | 51/500 [00:00<00:02, 164.14it/s]
14%|█▎ | 68/500 [00:00<00:02, 164.09it/s]
17%|█▋ | 85/500 [00:00<00:02, 164.59it/s]
20%|██ | 102/500 [00:00<00:02, 165.02it/s]
24%|██▍ | 119/500 [00:00<00:02, 165.36it/s]
27%|██▋ | 136/500 [00:00<00:02, 165.64it/s]
31%|███ | 153/500 [00:00<00:02, 165.90it/s]
34%|███▍ | 170/500 [00:01<00:01, 166.11it/s]
37%|███▋ | 187/500 [00:01<00:01, 166.24it/s]
41%|████ | 204/500 [00:01<00:01, 166.36it/s]
44%|████▍ | 221/500 [00:01<00:01, 166.44it/s]
48%|████▊ | 238/500 [00:01<00:01, 166.22it/s]
51%|█████ | 255/500 [00:01<00:01, 166.40it/s]
54%|█████▍ | 272/500 [00:01<00:01, 166.64it/s]
58%|█████▊ | 289/500 [00:01<00:01, 166.79it/s]
61%|██████ | 306/500 [00:01<00:01, 166.87it/s]
65%|██████▍ | 323/500 [00:01<00:01, 167.11it/s]
68%|██████▊ | 340/500 [00:02<00:00, 167.10it/s]
71%|███████▏ | 357/500 [00:02<00:00, 167.04it/s]
75%|███████▍ | 374/500 [00:02<00:00, 167.09it/s]
78%|███████▊ | 391/500 [00:02<00:00, 167.07it/s]
82%|████████▏ | 408/500 [00:02<00:00, 167.03it/s]
85%|████████▌ | 425/500 [00:02<00:00, 166.95it/s]
88%|████████▊ | 442/500 [00:02<00:00, 167.14it/s]
92%|█████████▏| 459/500 [00:02<00:00, 167.08it/s]
95%|█████████▌| 476/500 [00:02<00:00, 166.94it/s]
99%|█████████▊| 493/500 [00:02<00:00, 167.01it/s]
100%|██████████| 500/500 [00:03<00:00, 166.30it/s]
2026-08-08:18:17:50 INFO [evaluator:585] Running generate_until requests
2026-08-08:18:17:50 INFO [models.api_models:747] Tokenized requests are disabled. Context + generation length is not checked.
Requesting API: 0%| | 0/164 [00:00<?, ?it/s]
Requesting API: 1%| | 1/164 [00:00<01:36, 1.69it/s]
Requesting API: 2%|▏ | 3/164 [00:00<00:35, 4.56it/s]
Requesting API: 2%|▏ | 4/164 [00:00<00:30, 5.17it/s]
Requesting API: 5%|▍ | 8/164 [00:01<00:13, 11.87it/s]
Requesting API: 6%|▌ | 10/164 [00:01<00:18, 8.23it/s]
Requesting API: 7%|▋ | 12/164 [00:01<00:19, 7.66it/s]
Requesting API: 9%|▊ | 14/164 [00:01<00:15, 9.45it/s]
Requesting API: 10%|█ | 17/164 [00:01<00:11, 12.94it/s]
Requesting API: 12%|█▏ | 19/164 [00:02<00:21, 6.74it/s]
Requesting API: 13%|█▎ | 21/164 [00:02<00:22, 6.29it/s]
Requesting API: 14%|█▍ | 23/164 [00:03<00:25, 5.52it/s]
Requesting API: 15%|█▍ | 24/164 [00:03<00:24, 5.73it/s]
Requesting API: 15%|█▌ | 25/164 [00:03<00:25, 5.40it/s]
Requesting API: 16%|█▌ | 26/164 [00:03<00:22, 6.00it/s]
Requesting API: 16%|█▋ | 27/164 [00:04<00:24, 5.62it/s]
Requesting API: 17%|█▋ | 28/164 [00:04<00:22, 6.18it/s]
Requesting API: 18%|█▊ | 29/164 [00:04<00:24, 5.57it/s]
Requesting API: 18%|█▊ | 30/164 [00:04<00:26, 5.08it/s]
Requesting API: 19%|█▉ | 31/164 [00:05<00:35, 3.70it/s]
Requesting API: 20%|██ | 33/164 [00:05<00:26, 4.90it/s]
Requesting API: 21%|██ | 34/164 [00:06<00:42, 3.06it/s]
Requesting API: 21%|██▏ | 35/164 [00:06<00:38, 3.32it/s]
Requesting API: 22%|██▏ | 36/164 [00:09<02:02, 1.05it/s]
Requesting API: 23%|██▎ | 37/164 [00:13<03:51, 1.83s/it]
Requesting API: 23%|██▎ | 38/164 [00:13<02:50, 1.36s/it]
Requesting API: 24%|██▍ | 39/164 [00:15<03:32, 1.70s/it]
Requesting API: 37%|███▋ | 61/164 [00:16<00:17, 5.74it/s]
Requesting API: 41%|████ | 67/164 [00:16<00:13, 7.10it/s]
Requesting API: 44%|████▍ | 72/164 [00:16<00:11, 8.30it/s]
Requesting API: 46%|████▋ | 76/164 [00:16<00:09, 9.53it/s]
Requesting API: 49%|████▉ | 80/164 [00:16<00:07, 11.46it/s]
Requesting API: 51%|█████ | 84/164 [00:17<00:05, 13.42it/s]
Requesting API: 54%|█████▎ | 88/164 [00:17<00:04, 16.21it/s]
Requesting API: 56%|█████▌ | 92/164 [00:17<00:03, 18.60it/s]
Requesting API: 59%|█████▊ | 96/164 [00:17<00:05, 13.01it/s]
Requesting API: 60%|██████ | 99/164 [00:18<00:07, 8.36it/s]
Requesting API: 62%|██████▏ | 101/164 [00:18<00:08, 7.53it/s]
Requesting API: 63%|██████▎ | 103/164 [00:19<00:10, 5.79it/s]
Requesting API: 64%|██████▍ | 105/164 [00:20<00:14, 4.19it/s]
Requesting API: 65%|██████▍ | 106/164 [00:20<00:13, 4.18it/s]
Requesting API: 65%|██████▌ | 107/164 [00:20<00:12, 4.63it/s]
Requesting API: 66%|██████▌ | 108/164 [00:21<00:11, 4.75it/s]
Requesting API: 66%|██████▋ | 109/164 [00:21<00:13, 4.08it/s]
Requesting API: 68%|██████▊ | 111/164 [00:21<00:10, 4.84it/s]
Requesting API: 68%|██████▊ | 112/164 [00:22<00:11, 4.42it/s]
Requesting API: 70%|██████▉ | 114/164 [00:22<00:11, 4.25it/s]
Requesting API: 70%|███████ | 115/164 [00:24<00:25, 1.89it/s]
Requesting API: 71%|███████ | 116/164 [00:24<00:23, 2.06it/s]
Requesting API: 71%|███████▏ | 117/164 [00:28<01:04, 1.37s/it]
Requesting API: 72%|███████▏ | 118/164 [00:30<01:14, 1.61s/it]
Requesting API: 80%|████████ | 132/164 [00:31<00:08, 3.59it/s]
Requesting API: 82%|████████▏ | 134/164 [00:31<00:07, 3.84it/s]
Requesting API: 84%|████████▍ | 138/164 [00:31<00:05, 5.03it/s]
Requesting API: 86%|████████▌ | 141/164 [00:31<00:03, 6.21it/s]
Requesting API: 88%|████████▊ | 144/164 [00:31<00:02, 7.74it/s]
Requesting API: 90%|████████▉ | 147/164 [00:32<00:01, 9.46it/s]
Requesting API: 91%|█████████▏| 150/164 [00:32<00:01, 8.05it/s]
Requesting API: 93%|█████████▎| 152/164 [00:33<00:02, 5.70it/s]
Requesting API: 94%|█████████▍| 154/164 [00:33<00:01, 5.65it/s]
Requesting API: 95%|█████████▌| 156/164 [00:35<00:02, 3.16it/s]
Requesting API: 96%|█████████▌| 157/164 [00:35<00:02, 3.30it/s]
Requesting API: 96%|█████████▋| 158/164 [00:35<00:01, 3.37it/s]
Requesting API: 97%|█████████▋| 159/164 [00:35<00:01, 3.25it/s]
Requesting API: 98%|█████████▊| 160/164 [00:36<00:01, 2.77it/s]
Requesting API: 99%|█████████▉| 162/164 [00:37<00:00, 3.22it/s]
Requesting API: 99%|█████████▉| 163/164 [00:38<00:00, 1.76it/s]
Requesting API: 100%|██████████| 164/164 [00:38<00:00, 1.98it/s]
Requesting API: 100%|██████████| 164/164 [00:38<00:00, 4.23it/s]
Requesting API: 0%| | 0/500 [00:00<?, ?it/s]
Requesting API: 0%| | 1/500 [00:00<03:38, 2.29it/s]
Requesting API: 0%| | 2/500 [00:00<03:23, 2.45it/s]
Requesting API: 1%| | 4/500 [00:00<01:31, 5.43it/s]
Requesting API: 3%|▎ | 13/500 [00:01<00:24, 19.77it/s]
Requesting API: 4%|▍ | 19/500 [00:01<00:17, 27.81it/s]
Requesting API: 5%|▌ | 26/500 [00:01<00:13, 35.95it/s]
Requesting API: 6%|▌ | 31/500 [00:01<00:13, 35.94it/s]
Requesting API: 7%|▋ | 36/500 [00:01<00:12, 38.53it/s]
Requesting API: 9%|▊ | 43/500 [00:01<00:10, 43.98it/s]
Requesting API: 10%|▉ | 49/500 [00:01<00:10, 44.34it/s]
Requesting API: 11%|█ | 55/500 [00:01<00:09, 46.61it/s]
Requesting API: 12%|█▏ | 60/500 [00:02<00:10, 43.40it/s]
Requesting API: 14%|█▎ | 68/500 [00:02<00:08, 51.17it/s]
Requesting API: 15%|█▍ | 74/500 [00:02<00:08, 52.00it/s]
Requesting API: 16%|█▌ | 80/500 [00:02<00:08, 52.43it/s]
Requesting API: 17%|█▋ | 86/500 [00:02<00:08, 51.24it/s]
Requesting API: 18%|█▊ | 92/500 [00:02<00:07, 51.18it/s]
Requesting API: 20%|█▉ | 98/500 [00:02<00:08, 46.11it/s]
Requesting API: 21%|██ | 104/500 [00:02<00:08, 46.39it/s]
Requesting API: 22%|██▏ | 109/500 [00:03<00:08, 46.12it/s]
Requesting API: 23%|██▎ | 115/500 [00:03<00:08, 44.01it/s]
Requesting API: 24%|██▍ | 120/500 [00:03<00:08, 42.72it/s]
Requesting API: 25%|██▌ | 127/500 [00:03<00:08, 45.20it/s]
Requesting API: 26%|██▋ | 132/500 [00:03<00:08, 44.82it/s]
Requesting API: 27%|██▋ | 137/500 [00:03<00:08, 42.28it/s]
Requesting API: 28%|██▊ | 142/500 [00:03<00:08, 40.22it/s]
Requesting API: 30%|██▉ | 148/500 [00:04<00:09, 38.17it/s]
Requesting API: 31%|███▏ | 157/500 [00:04<00:07, 48.69it/s]
Requesting API: 33%|███▎ | 163/500 [00:04<00:07, 46.55it/s]
Requesting API: 34%|███▎ | 168/500 [00:04<00:08, 39.60it/s]
Requesting API: 35%|███▍ | 173/500 [00:04<00:07, 41.88it/s]
Requesting API: 36%|███▌ | 178/500 [00:04<00:08, 38.50it/s]
Requesting API: 37%|███▋ | 183/500 [00:04<00:08, 36.00it/s]
Requesting API: 38%|███▊ | 188/500 [00:04<00:08, 38.07it/s]
Requesting API: 38%|███▊ | 192/500 [00:05<00:08, 35.70it/s]
Requesting API: 40%|████ | 201/500 [00:05<00:06, 46.52it/s]
Requesting API: 41%|████▏ | 207/500 [00:05<00:06, 48.75it/s]
Requesting API: 43%|████▎ | 213/500 [00:05<00:05, 49.66it/s]
Requesting API: 44%|████▍ | 219/500 [00:05<00:06, 43.84it/s]
Requesting API: 45%|████▍ | 224/500 [00:05<00:07, 34.91it/s]
Requesting API: 46%|████▌ | 229/500 [00:05<00:07, 37.65it/s]
Requesting API: 48%|████▊ | 238/500 [00:06<00:05, 49.16it/s]
Requesting API: 49%|████▉ | 244/500 [00:06<00:05, 49.84it/s]
Requesting API: 50%|█████ | 250/500 [00:06<00:05, 47.25it/s]
Requesting API: 51%|█████ | 256/500 [00:06<00:05, 47.74it/s]
Requesting API: 52%|█████▏ | 261/500 [00:06<00:05, 43.75it/s]
Requesting API: 54%|█████▍ | 270/500 [00:06<00:04, 53.82it/s]
Requesting API: 55%|█████▌ | 276/500 [00:06<00:04, 45.86it/s]
Requesting API: 57%|█████▋ | 283/500 [00:07<00:05, 42.97it/s]
Requesting API: 58%|█████▊ | 289/500 [00:07<00:04, 46.01it/s]
Requesting API: 59%|█████▉ | 295/500 [00:07<00:04, 48.39it/s]
Requesting API: 60%|██████ | 301/500 [00:07<00:04, 45.24it/s]
Requesting API: 61%|██████▏ | 307/500 [00:07<00:04, 47.78it/s]
Requesting API: 63%|██████▎ | 313/500 [00:07<00:03, 49.72it/s]
Requesting API: 64%|██████▍ | 319/500 [00:07<00:03, 49.11it/s]
Requesting API: 65%|██████▌ | 325/500 [00:08<00:04, 36.52it/s]
Requesting API: 66%|██████▋ | 332/500 [00:08<00:03, 42.79it/s]
Requesting API: 68%|██████▊ | 341/500 [00:08<00:03, 52.41it/s]
Requesting API: 70%|██████▉ | 348/500 [00:08<00:02, 54.96it/s]
Requesting API: 71%|███████ | 355/500 [00:08<00:02, 49.13it/s]
Requesting API: 72%|███████▏ | 362/500 [00:08<00:02, 51.46it/s]
Requesting API: 74%|███████▎ | 368/500 [00:08<00:02, 52.70it/s]
Requesting API: 75%|███████▍ | 374/500 [00:08<00:02, 51.76it/s]
Requesting API: 76%|███████▌ | 380/500 [00:09<00:02, 49.56it/s]
Requesting API: 77%|███████▋ | 386/500 [00:09<00:02, 46.54it/s]
Requesting API: 78%|███████▊ | 391/500 [00:09<00:02, 46.86it/s]
Requesting API: 79%|███████▉ | 396/500 [00:09<00:02, 46.99it/s]
Requesting API: 80%|████████ | 401/500 [00:09<00:02, 40.99it/s]
Requesting API: 81%|████████ | 406/500 [00:09<00:02, 38.76it/s]
Requesting API: 82%|████████▏ | 411/500 [00:09<00:02, 36.03it/s]
Requesting API: 83%|████████▎ | 416/500 [00:09<00:02, 38.89it/s]
Requesting API: 84%|████████▍ | 421/500 [00:10<00:02, 35.02it/s]
Requesting API: 85%|████████▌ | 425/500 [00:10<00:02, 35.87it/s]
Requesting API: 87%|████████▋ | 433/500 [00:10<00:01, 45.69it/s]
Requesting API: 88%|████████▊ | 438/500 [00:10<00:01, 44.82it/s]
Requesting API: 89%|████████▊ | 443/500 [00:10<00:01, 43.83it/s]
Requesting API: 90%|████████▉ | 449/500 [00:10<00:01, 47.41it/s]
Requesting API: 91%|█████████ | 455/500 [00:10<00:00, 48.56it/s]
Requesting API: 92%|█████████▏| 460/500 [00:10<00:00, 47.17it/s]
Requesting API: 94%|█████████▎| 468/500 [00:11<00:00, 54.00it/s]
Requesting API: 95%|█████████▍| 474/500 [00:11<00:00, 39.53it/s]
Requesting API: 96%|█████████▌| 479/500 [00:11<00:00, 24.57it/s]
Requesting API: 97%|█████████▋| 483/500 [00:11<00:00, 22.38it/s]
Requesting API: 97%|█████████▋| 486/500 [00:12<00:00, 16.73it/s]
Requesting API: 98%|█████████▊| 489/500 [00:12<00:00, 16.82it/s]
Requesting API: 98%|█████████▊| 492/500 [00:12<00:00, 16.91it/s]
Requesting API: 99%|█████████▉| 495/500 [00:13<00:00, 11.51it/s]
Requesting API: 99%|█████████▉| 497/500 [00:13<00:00, 7.98it/s]
Requesting API: 100%|█████████▉| 499/500 [00:13<00:00, 8.07it/s]
Requesting API: 100%|██████████| 500/500 [00:14<00:00, 35.33it/s]
2026-08-08:18:21:03 INFO [loggers.evaluation_tracker:247] Saving results aggregated
2026-08-08:18:21:03 INFO [loggers.evaluation_tracker:119] Saving per-task samples to outputs/evals/release/general-keep25-step0150/student/*.jsonl
local-completions ({'model': 'student', 'base_url': 'http://127.0.0.1:8451/v1/completions', 'tokenizer': 'outputs/release/healed/release-offpolicy-general-keep25/step0150', 'num_concurrent': 48, 'tokenized_requests': False, 'max_retries': 3}), gen_kwargs: ({}), limit: None, num_fewshot: None, batch_size: 1
| Tasks |Version| Filter |n-shot| Metric | |Value | |Stderr|
|---------|------:|-----------|-----:|---------|---|-----:|---|-----:|
|humaneval| 1|create_test| 0|pass@1 |↑ |0.0183|± |0.0105|
|mbpp | 1|none | 3|pass_at_1|↑ |0.1120|± |0.0141|
|