File size: 14,274 Bytes
a82fb9e | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 | 2026-08-08:22:22:17 INFO [_cli.run:388] Selected Tasks: ['humaneval', 'mbpp']
2026-08-08:22:22:19 INFO [evaluator:214] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2026-08-08:22:22:19 INFO [evaluator:239] Initializing local-completions model, with arguments: {'model': 'student', 'base_url': 'http://127.0.0.1:8460/v1/completions', 'tokenizer': 'outputs/release/healed/release-offpolicy-math-keep75/step0150', 'num_concurrent': 48, 'tokenized_requests': False, 'max_retries': 3}
2026-08-08:22:22:19 INFO [models.openai_completions:42] Remote tokenizer not supported. Using huggingface tokenizer backend.
2026-08-08:22:22:19 INFO [models.api_models:179] Using max length 2048 - 1
2026-08-08:22:22:19 INFO [models.api_models:200] Using tokenizer huggingface
2026-08-08:22:22:26 INFO [evaluator_utils:446] Selected tasks:
2026-08-08:22:22:26 INFO [evaluator_utils:480] Task: humaneval (humaneval/humaneval.yaml)
2026-08-08:22:22:26 INFO [evaluator_utils:480] Task: mbpp (mbpp/mbpp.yaml)
2026-08-08:22:22:26 INFO [evaluator:314] humaneval: Using gen_kwargs: {'until': ['\nclass', '\ndef', '\n#', '\nif', '\nprint'], 'max_gen_toks': 1024, 'do_sample': False}
2026-08-08:22:22:26 INFO [evaluator:314] mbpp: Using gen_kwargs: {'until': ['[DONE]'], 'do_sample': False}
2026-08-08:22:22:26 INFO [api.task:312] Building contexts for humaneval on rank 0...
0%| | 0/164 [00:00<?, ?it/s]
88%|████████▊ | 145/164 [00:00<00:00, 1444.89it/s]
100%|██████████| 164/164 [00:00<00:00, 1476.50it/s]
2026-08-08:22:22:26 INFO [api.task:312] Building contexts for mbpp on rank 0...
0%| | 0/500 [00:00<?, ?it/s]
3%|▎ | 17/500 [00:00<00:02, 162.52it/s]
7%|▋ | 34/500 [00:00<00:02, 163.57it/s]
10%|█ | 51/500 [00:00<00:02, 163.84it/s]
14%|█▎ | 68/500 [00:00<00:02, 163.63it/s]
17%|█▋ | 85/500 [00:00<00:02, 163.54it/s]
20%|██ | 102/500 [00:00<00:02, 163.70it/s]
24%|██▍ | 119/500 [00:00<00:02, 163.63it/s]
27%|██▋ | 136/500 [00:00<00:02, 164.21it/s]
31%|███ | 153/500 [00:00<00:02, 164.55it/s]
34%|███▍ | 170/500 [00:01<00:02, 164.83it/s]
37%|███▋ | 187/500 [00:01<00:01, 164.74it/s]
41%|████ | 204/500 [00:01<00:01, 165.03it/s]
44%|████▍ | 221/500 [00:01<00:01, 165.10it/s]
48%|████▊ | 238/500 [00:01<00:01, 164.55it/s]
51%|█████ | 255/500 [00:01<00:01, 164.65it/s]
54%|█████▍ | 272/500 [00:01<00:01, 164.82it/s]
58%|█████▊ | 289/500 [00:01<00:01, 164.92it/s]
61%|██████ | 306/500 [00:01<00:01, 164.88it/s]
65%|██████▍ | 323/500 [00:01<00:01, 165.02it/s]
68%|██████▊ | 340/500 [00:02<00:00, 165.10it/s]
71%|███████▏ | 357/500 [00:02<00:00, 165.14it/s]
75%|███████▍ | 374/500 [00:02<00:00, 165.25it/s]
78%|███████▊ | 391/500 [00:02<00:00, 165.28it/s]
82%|████████▏ | 408/500 [00:02<00:00, 165.39it/s]
85%|████████▌ | 425/500 [00:02<00:00, 165.71it/s]
88%|████████▊ | 442/500 [00:02<00:00, 166.16it/s]
92%|█████████▏| 459/500 [00:02<00:00, 165.99it/s]
95%|█████████▌| 476/500 [00:02<00:00, 165.78it/s]
99%|█████████▊| 493/500 [00:02<00:00, 165.64it/s]
100%|██████████| 500/500 [00:03<00:00, 164.91it/s]
2026-08-08:22:22:29 INFO [evaluator:585] Running generate_until requests
2026-08-08:22:22:29 INFO [models.api_models:747] Tokenized requests are disabled. Context + generation length is not checked.
Requesting API: 0%| | 0/164 [00:00<?, ?it/s]
Requesting API: 1%| | 1/164 [00:00<02:39, 1.02it/s]
Requesting API: 6%|▌ | 10/164 [00:01<00:15, 10.22it/s]
Requesting API: 8%|▊ | 13/164 [00:01<00:13, 11.53it/s]
Requesting API: 9%|▉ | 15/164 [00:01<00:12, 12.36it/s]
Requesting API: 11%|█ | 18/164 [00:01<00:10, 14.46it/s]
Requesting API: 13%|█▎ | 22/164 [00:01<00:07, 17.82it/s]
Requesting API: 15%|█▌ | 25/164 [00:01<00:07, 19.18it/s]
Requesting API: 17%|█▋ | 28/164 [00:02<00:06, 20.93it/s]
Requesting API: 19%|█▉ | 31/164 [00:02<00:06, 20.30it/s]
Requesting API: 22%|██▏ | 36/164 [00:02<00:05, 24.13it/s]
Requesting API: 24%|██▍ | 39/164 [00:02<00:06, 20.50it/s]
Requesting API: 26%|██▌ | 43/164 [00:02<00:05, 23.89it/s]
Requesting API: 30%|██▉ | 49/164 [00:02<00:03, 29.63it/s]
Requesting API: 34%|███▎ | 55/164 [00:02<00:03, 32.73it/s]
Requesting API: 38%|███▊ | 62/164 [00:03<00:02, 38.54it/s]
Requesting API: 41%|████ | 67/164 [00:03<00:02, 39.48it/s]
Requesting API: 44%|████▍ | 72/164 [00:03<00:02, 34.72it/s]
Requesting API: 47%|████▋ | 77/164 [00:03<00:02, 37.01it/s]
Requesting API: 51%|█████ | 83/164 [00:03<00:02, 37.01it/s]
Requesting API: 55%|█████▍ | 90/164 [00:03<00:01, 43.94it/s]
Requesting API: 59%|█████▊ | 96/164 [00:03<00:01, 43.57it/s]
Requesting API: 62%|██████▏ | 101/164 [00:04<00:01, 38.40it/s]
Requesting API: 65%|██████▌ | 107/164 [00:04<00:01, 42.24it/s]
Requesting API: 69%|██████▉ | 113/164 [00:04<00:01, 43.81it/s]
Requesting API: 76%|███████▌ | 124/164 [00:04<00:00, 58.62it/s]
Requesting API: 80%|███████▉ | 131/164 [00:04<00:00, 61.05it/s]
Requesting API: 85%|████████▍ | 139/164 [00:04<00:00, 63.96it/s]
Requesting API: 89%|████████▉ | 146/164 [00:04<00:00, 45.53it/s]
Requesting API: 93%|█████████▎| 152/164 [00:05<00:00, 29.40it/s]
Requesting API: 96%|█████████▌| 157/164 [00:05<00:00, 30.88it/s]
Requesting API: 99%|█████████▉| 162/164 [00:06<00:00, 17.88it/s]
Requesting API: 100%|██████████| 164/164 [00:15<00:00, 10.58it/s]
Requesting API: 0%| | 0/500 [00:00<?, ?it/s]
Requesting API: 0%| | 1/500 [00:01<09:28, 1.14s/it]
Requesting API: 1%| | 4/500 [00:01<02:01, 4.07it/s]
Requesting API: 2%|▏ | 8/500 [00:01<00:56, 8.65it/s]
Requesting API: 4%|▎ | 18/500 [00:01<00:23, 20.83it/s]
Requesting API: 4%|▍ | 22/500 [00:01<00:20, 22.95it/s]
Requesting API: 5%|▌ | 27/500 [00:01<00:21, 22.22it/s]
Requesting API: 6%|▌ | 31/500 [00:02<00:21, 21.86it/s]
Requesting API: 8%|▊ | 38/500 [00:02<00:16, 28.14it/s]
Requesting API: 9%|▊ | 43/500 [00:02<00:14, 30.76it/s]
Requesting API: 9%|▉ | 47/500 [00:02<00:15, 28.50it/s]
Requesting API: 11%|█ | 54/500 [00:02<00:12, 35.97it/s]
Requesting API: 12%|█▏ | 59/500 [00:02<00:11, 38.55it/s]
Requesting API: 13%|█▎ | 64/500 [00:02<00:11, 38.17it/s]
Requesting API: 14%|█▍ | 71/500 [00:03<00:09, 45.19it/s]
Requesting API: 15%|█▌ | 76/500 [00:03<00:14, 29.85it/s]
Requesting API: 16%|█▌ | 80/500 [00:03<00:13, 31.20it/s]
Requesting API: 17%|█▋ | 84/500 [00:03<00:13, 31.64it/s]
Requesting API: 18%|█▊ | 88/500 [00:03<00:13, 30.24it/s]
Requesting API: 18%|█▊ | 92/500 [00:03<00:13, 29.65it/s]
Requesting API: 20%|█▉ | 98/500 [00:03<00:12, 33.22it/s]
Requesting API: 21%|██▏ | 107/500 [00:04<00:09, 43.23it/s]
Requesting API: 22%|██▏ | 112/500 [00:04<00:08, 43.49it/s]
Requesting API: 23%|██▎ | 117/500 [00:04<00:09, 40.92it/s]
Requesting API: 24%|██▍ | 122/500 [00:04<00:14, 25.72it/s]
Requesting API: 25%|██▌ | 126/500 [00:04<00:13, 26.94it/s]
Requesting API: 26%|██▋ | 132/500 [00:04<00:11, 32.29it/s]
Requesting API: 27%|██▋ | 136/500 [00:05<00:10, 33.43it/s]
Requesting API: 28%|██▊ | 141/500 [00:05<00:09, 36.93it/s]
Requesting API: 29%|██▉ | 146/500 [00:05<00:09, 35.49it/s]
Requesting API: 30%|███ | 150/500 [00:05<00:14, 24.77it/s]
Requesting API: 32%|███▏ | 159/500 [00:05<00:10, 33.90it/s]
Requesting API: 33%|███▎ | 164/500 [00:06<00:10, 31.40it/s]
Requesting API: 34%|███▎ | 168/500 [00:06<00:10, 30.67it/s]
Requesting API: 35%|███▍ | 173/500 [00:06<00:09, 33.72it/s]
Requesting API: 36%|███▌ | 178/500 [00:06<00:08, 37.30it/s]
Requesting API: 37%|███▋ | 183/500 [00:06<00:08, 36.04it/s]
Requesting API: 37%|███▋ | 187/500 [00:06<00:09, 34.36it/s]
Requesting API: 38%|███▊ | 191/500 [00:06<00:10, 29.10it/s]
Requesting API: 39%|███▉ | 195/500 [00:06<00:09, 30.57it/s]
Requesting API: 40%|███▉ | 199/500 [00:07<00:10, 29.96it/s]
Requesting API: 41%|████ | 203/500 [00:07<00:09, 29.71it/s]
Requesting API: 41%|████▏ | 207/500 [00:07<00:09, 30.69it/s]
Requesting API: 43%|████▎ | 214/500 [00:07<00:07, 39.23it/s]
Requesting API: 44%|████▍ | 219/500 [00:07<00:07, 37.54it/s]
Requesting API: 45%|████▌ | 225/500 [00:07<00:06, 41.36it/s]
Requesting API: 46%|████▌ | 230/500 [00:08<00:09, 29.13it/s]
Requesting API: 47%|████▋ | 236/500 [00:08<00:07, 33.84it/s]
Requesting API: 48%|████▊ | 240/500 [00:08<00:08, 32.41it/s]
Requesting API: 49%|████▉ | 244/500 [00:08<00:07, 33.17it/s]
Requesting API: 50%|████▉ | 249/500 [00:08<00:08, 30.67it/s]
Requesting API: 51%|█████ | 254/500 [00:08<00:07, 34.66it/s]
Requesting API: 52%|█████▏ | 258/500 [00:08<00:07, 32.54it/s]
Requesting API: 52%|█████▏ | 262/500 [00:08<00:07, 31.93it/s]
Requesting API: 54%|█████▎ | 268/500 [00:09<00:06, 37.34it/s]
Requesting API: 54%|█████▍ | 272/500 [00:09<00:06, 34.10it/s]
Requesting API: 55%|█████▌ | 276/500 [00:09<00:07, 31.52it/s]
Requesting API: 56%|█████▌ | 280/500 [00:09<00:06, 31.64it/s]
Requesting API: 57%|█████▋ | 284/500 [00:09<00:06, 32.42it/s]
Requesting API: 58%|█████▊ | 288/500 [00:09<00:07, 29.29it/s]
Requesting API: 59%|█████▉ | 295/500 [00:09<00:05, 37.92it/s]
Requesting API: 60%|██████ | 300/500 [00:09<00:04, 40.82it/s]
Requesting API: 61%|██████ | 305/500 [00:10<00:04, 42.35it/s]
Requesting API: 62%|██████▏ | 312/500 [00:10<00:03, 47.43it/s]
Requesting API: 63%|██████▎ | 317/500 [00:10<00:04, 37.86it/s]
Requesting API: 65%|██████▍ | 323/500 [00:10<00:04, 41.71it/s]
Requesting API: 66%|██████▌ | 328/500 [00:10<00:04, 38.55it/s]
Requesting API: 67%|██████▋ | 333/500 [00:10<00:04, 38.38it/s]
Requesting API: 68%|██████▊ | 338/500 [00:10<00:04, 38.23it/s]
Requesting API: 69%|██████▊ | 343/500 [00:11<00:03, 40.29it/s]
Requesting API: 70%|██████▉ | 348/500 [00:11<00:04, 37.85it/s]
Requesting API: 71%|███████ | 355/500 [00:11<00:03, 44.87it/s]
Requesting API: 72%|███████▏ | 360/500 [00:11<00:03, 43.75it/s]
Requesting API: 73%|███████▎ | 366/500 [00:11<00:03, 40.52it/s]
Requesting API: 74%|███████▍ | 372/500 [00:11<00:03, 40.31it/s]
Requesting API: 75%|███████▌ | 377/500 [00:11<00:03, 39.43it/s]
Requesting API: 77%|███████▋ | 384/500 [00:12<00:02, 43.70it/s]
Requesting API: 78%|███████▊ | 390/500 [00:12<00:02, 46.35it/s]
Requesting API: 79%|███████▉ | 395/500 [00:12<00:02, 46.86it/s]
Requesting API: 80%|████████ | 400/500 [00:12<00:02, 46.69it/s]
Requesting API: 81%|████████▏ | 407/500 [00:12<00:01, 51.19it/s]
Requesting API: 83%|████████▎ | 414/500 [00:12<00:01, 51.13it/s]
Requesting API: 84%|████████▍ | 421/500 [00:12<00:01, 48.89it/s]
Requesting API: 85%|████████▌ | 426/500 [00:12<00:01, 39.13it/s]
Requesting API: 87%|████████▋ | 436/500 [00:13<00:01, 51.11it/s]
Requesting API: 88%|████████▊ | 442/500 [00:13<00:01, 48.57it/s]
Requesting API: 90%|█████████ | 452/500 [00:13<00:00, 59.31it/s]
Requesting API: 92%|█████████▏| 459/500 [00:13<00:00, 59.08it/s]
Requesting API: 94%|█████████▎| 468/500 [00:13<00:00, 66.24it/s]
Requesting API: 95%|█████████▌| 475/500 [00:13<00:00, 59.32it/s]
Requesting API: 97%|█████████▋| 483/500 [00:13<00:00, 56.83it/s]
Requesting API: 98%|█████████▊| 490/500 [00:13<00:00, 53.66it/s]
Requesting API: 100%|█████████▉| 498/500 [00:14<00:00, 46.72it/s]
Requesting API: 100%|██████████| 500/500 [00:16<00:00, 30.65it/s]
2026-08-08:22:26:28 INFO [loggers.evaluation_tracker:247] Saving results aggregated
2026-08-08:22:26:28 INFO [loggers.evaluation_tracker:119] Saving per-task samples to outputs/evals/release/math-keep75-step0150/student/*.jsonl
local-completions ({'model': 'student', 'base_url': 'http://127.0.0.1:8460/v1/completions', 'tokenizer': 'outputs/release/healed/release-offpolicy-math-keep75/step0150', 'num_concurrent': 48, 'tokenized_requests': False, 'max_retries': 3}), gen_kwargs: ({}), limit: None, num_fewshot: None, batch_size: 1
| Tasks |Version| Filter |n-shot| Metric | |Value | |Stderr|
|---------|------:|-----------|-----:|---------|---|-----:|---|-----:|
|humaneval| 1|create_test| 0|pass@1 |↑ |0.2744|± |0.0349|
|mbpp | 1|none | 3|pass_at_1|↑ |0.2420|± |0.0192|
|