File size: 17,333 Bytes
a82fb9e | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 | 2026-08-09:15:45:18 INFO [_cli.run:388] Selected Tasks: ['humaneval', 'mbpp']
2026-08-09:15:45:20 INFO [evaluator:214] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2026-08-09:15:45:20 INFO [evaluator:239] Initializing local-completions model, with arguments: {'model': 'student', 'base_url': 'http://127.0.0.1:8471/v1/completions', 'tokenizer': 'outputs/release/healed/release-offpolicy-math-keep25/step0050', 'num_concurrent': 48, 'tokenized_requests': False, 'max_retries': 3}
2026-08-09:15:45:20 INFO [models.openai_completions:42] Remote tokenizer not supported. Using huggingface tokenizer backend.
2026-08-09:15:45:20 INFO [models.api_models:179] Using max length 2048 - 1
2026-08-09:15:45:20 INFO [models.api_models:200] Using tokenizer huggingface
2026-08-09:15:45:26 INFO [evaluator_utils:446] Selected tasks:
2026-08-09:15:45:26 INFO [evaluator_utils:480] Task: humaneval (humaneval/humaneval.yaml)
2026-08-09:15:45:26 INFO [evaluator_utils:480] Task: mbpp (mbpp/mbpp.yaml)
2026-08-09:15:45:26 INFO [evaluator:314] humaneval: Using gen_kwargs: {'until': ['\nclass', '\ndef', '\n#', '\nif', '\nprint'], 'max_gen_toks': 1024, 'do_sample': False}
2026-08-09:15:45:26 INFO [evaluator:314] mbpp: Using gen_kwargs: {'until': ['[DONE]'], 'do_sample': False}
2026-08-09:15:45:26 INFO [api.task:312] Building contexts for humaneval on rank 0...
0%| | 0/164 [00:00<?, ?it/s]
100%|██████████| 164/164 [00:00<00:00, 1704.21it/s]
2026-08-09:15:45:26 INFO [api.task:312] Building contexts for mbpp on rank 0...
0%| | 0/500 [00:00<?, ?it/s]
3%|▎ | 17/500 [00:00<00:03, 160.45it/s]
7%|▋ | 34/500 [00:00<00:02, 162.21it/s]
10%|█ | 51/500 [00:00<00:02, 162.22it/s]
14%|█▎ | 68/500 [00:00<00:02, 162.35it/s]
17%|█▋ | 85/500 [00:00<00:02, 162.93it/s]
20%|██ | 102/500 [00:00<00:02, 163.32it/s]
24%|██▍ | 119/500 [00:00<00:02, 163.25it/s]
27%|██▋ | 136/500 [00:00<00:02, 161.67it/s]
31%|███ | 153/500 [00:00<00:02, 155.18it/s]
34%|███▍ | 170/500 [00:01<00:02, 158.38it/s]
37%|███▋ | 187/500 [00:01<00:01, 160.72it/s]
41%|████ | 204/500 [00:01<00:01, 162.33it/s]
44%|████▍ | 221/500 [00:01<00:01, 163.50it/s]
48%|████▊ | 238/500 [00:01<00:01, 164.34it/s]
51%|█████ | 255/500 [00:01<00:01, 164.89it/s]
54%|█████▍ | 272/500 [00:01<00:01, 165.06it/s]
58%|█████▊ | 289/500 [00:01<00:01, 165.53it/s]
61%|██████ | 306/500 [00:01<00:01, 165.75it/s]
65%|██████▍ | 323/500 [00:01<00:01, 165.61it/s]
68%|██████▊ | 340/500 [00:02<00:00, 165.58it/s]
71%|███████▏ | 357/500 [00:02<00:00, 165.51it/s]
75%|███████▍ | 374/500 [00:02<00:00, 165.58it/s]
78%|███████▊ | 391/500 [00:02<00:00, 165.60it/s]
82%|████████▏ | 408/500 [00:02<00:00, 165.83it/s]
85%|████████▌ | 425/500 [00:02<00:00, 165.98it/s]
88%|████████▊ | 442/500 [00:02<00:00, 166.25it/s]
92%|█████████▏| 459/500 [00:02<00:00, 166.50it/s]
95%|█████████▌| 476/500 [00:02<00:00, 166.49it/s]
99%|█████████▊| 493/500 [00:03<00:00, 166.29it/s]
100%|██████████| 500/500 [00:03<00:00, 164.12it/s]
2026-08-09:15:45:29 INFO [evaluator:585] Running generate_until requests
2026-08-09:15:45:29 INFO [models.api_models:747] Tokenized requests are disabled. Context + generation length is not checked.
Requesting API: 0%| | 0/164 [00:00<?, ?it/s]
Requesting API: 1%| | 1/164 [00:00<01:21, 2.01it/s]
Requesting API: 3%|▎ | 5/164 [00:00<00:16, 9.73it/s]
Requesting API: 9%|▉ | 15/164 [00:00<00:05, 28.65it/s]
Requesting API: 14%|█▍ | 23/164 [00:00<00:03, 38.71it/s]
Requesting API: 20%|█▉ | 32/164 [00:00<00:02, 50.02it/s]
Requesting API: 24%|██▍ | 39/164 [00:01<00:02, 49.19it/s]
Requesting API: 27%|██▋ | 45/164 [00:01<00:05, 20.60it/s]
Requesting API: 30%|███ | 50/164 [00:02<00:05, 20.64it/s]
Requesting API: 33%|███▎ | 54/164 [00:02<00:05, 20.20it/s]
Requesting API: 35%|███▌ | 58/164 [00:02<00:04, 22.64it/s]
Requesting API: 38%|███▊ | 62/164 [00:02<00:04, 21.36it/s]
Requesting API: 40%|███▉ | 65/164 [00:02<00:04, 20.29it/s]
Requesting API: 41%|████▏ | 68/164 [00:03<00:05, 18.03it/s]
Requesting API: 44%|████▍ | 72/164 [00:03<00:04, 21.52it/s]
Requesting API: 46%|████▌ | 75/164 [00:03<00:03, 22.88it/s]
Requesting API: 48%|████▊ | 78/164 [00:03<00:04, 19.10it/s]
Requesting API: 51%|█████ | 83/164 [00:03<00:03, 22.69it/s]
Requesting API: 52%|█████▏ | 86/164 [00:03<00:04, 18.15it/s]
Requesting API: 54%|█████▍ | 89/164 [00:04<00:03, 18.90it/s]
Requesting API: 57%|█████▋ | 93/164 [00:04<00:03, 22.60it/s]
Requesting API: 59%|█████▊ | 96/164 [00:04<00:03, 19.62it/s]
Requesting API: 60%|██████ | 99/164 [00:04<00:03, 19.24it/s]
Requesting API: 62%|██████▏ | 102/164 [00:04<00:03, 16.28it/s]
Requesting API: 63%|██████▎ | 104/164 [00:04<00:03, 16.93it/s]
Requesting API: 68%|██████▊ | 112/164 [00:05<00:01, 26.72it/s]
Requesting API: 70%|███████ | 115/164 [00:05<00:01, 25.85it/s]
Requesting API: 74%|███████▍ | 121/164 [00:05<00:01, 32.39it/s]
Requesting API: 76%|███████▌ | 125/164 [00:05<00:01, 24.63it/s]
Requesting API: 78%|███████▊ | 128/164 [00:05<00:01, 19.02it/s]
Requesting API: 80%|████████ | 132/164 [00:06<00:01, 18.72it/s]
Requesting API: 83%|████████▎ | 136/164 [00:06<00:01, 21.88it/s]
Requesting API: 85%|████████▍ | 139/164 [00:06<00:01, 19.49it/s]
Requesting API: 87%|████████▋ | 142/164 [00:06<00:01, 15.92it/s]
Requesting API: 88%|████████▊ | 144/164 [00:07<00:02, 7.95it/s]
Requesting API: 89%|████████▉ | 146/164 [00:08<00:02, 6.10it/s]
Requesting API: 90%|█████████ | 148/164 [00:09<00:05, 2.90it/s]
Requesting API: 91%|█████████ | 149/164 [00:11<00:07, 2.12it/s]
Requesting API: 91%|█████████▏| 150/164 [00:11<00:06, 2.32it/s]
Requesting API: 92%|█████████▏| 151/164 [00:12<00:08, 1.62it/s]
Requesting API: 93%|█████████▎| 152/164 [00:12<00:06, 1.83it/s]
Requesting API: 93%|█████████▎| 153/164 [00:15<00:10, 1.09it/s]
Requesting API: 98%|█████████▊| 161/164 [00:15<00:00, 3.89it/s]
Requesting API: 100%|██████████| 164/164 [00:18<00:00, 1.85it/s]
Requesting API: 100%|██████████| 164/164 [00:18<00:00, 8.63it/s]
Requesting API: 0%| | 0/500 [00:00<?, ?it/s]
Requesting API: 0%| | 1/500 [00:00<07:27, 1.12it/s]
Requesting API: 1%| | 3/500 [00:01<02:20, 3.53it/s]
Requesting API: 1%|▏ | 7/500 [00:01<00:58, 8.41it/s]
Requesting API: 2%|▏ | 11/500 [00:01<00:37, 13.05it/s]
Requesting API: 3%|▎ | 15/500 [00:01<00:27, 17.39it/s]
Requesting API: 4%|▎ | 18/500 [00:01<00:29, 16.28it/s]
Requesting API: 5%|▍ | 23/500 [00:01<00:26, 18.13it/s]
Requesting API: 5%|▌ | 26/500 [00:02<00:27, 17.02it/s]
Requesting API: 6%|▌ | 28/500 [00:02<00:31, 15.01it/s]
Requesting API: 6%|▌ | 30/500 [00:02<00:30, 15.31it/s]
Requesting API: 6%|▋ | 32/500 [00:02<00:29, 15.79it/s]
Requesting API: 7%|▋ | 34/500 [00:02<00:28, 16.08it/s]
Requesting API: 7%|▋ | 37/500 [00:02<00:27, 17.14it/s]
Requesting API: 8%|▊ | 39/500 [00:02<00:26, 17.59it/s]
Requesting API: 8%|▊ | 41/500 [00:03<00:38, 11.81it/s]
Requesting API: 9%|▉ | 44/500 [00:03<00:34, 13.17it/s]
Requesting API: 9%|▉ | 46/500 [00:03<00:34, 13.08it/s]
Requesting API: 10%|▉ | 49/500 [00:03<00:31, 14.22it/s]
Requesting API: 10%|█ | 52/500 [00:03<00:29, 15.14it/s]
Requesting API: 11%|█ | 55/500 [00:04<00:24, 17.94it/s]
Requesting API: 11%|█▏ | 57/500 [00:04<00:25, 17.59it/s]
Requesting API: 12%|█▏ | 61/500 [00:04<00:29, 14.97it/s]
Requesting API: 13%|█▎ | 65/500 [00:04<00:27, 15.80it/s]
Requesting API: 13%|█▎ | 67/500 [00:04<00:27, 15.51it/s]
Requesting API: 14%|█▍ | 70/500 [00:04<00:24, 17.79it/s]
Requesting API: 17%|█▋ | 85/500 [00:05<00:10, 41.46it/s]
Requesting API: 18%|█▊ | 90/500 [00:05<00:09, 41.16it/s]
Requesting API: 19%|█▉ | 95/500 [00:05<00:10, 36.94it/s]
Requesting API: 20%|█▉ | 99/500 [00:05<00:13, 30.39it/s]
Requesting API: 21%|██ | 103/500 [00:05<00:13, 29.47it/s]
Requesting API: 22%|██▏ | 110/500 [00:05<00:11, 34.72it/s]
Requesting API: 23%|██▎ | 115/500 [00:06<00:12, 31.14it/s]
Requesting API: 25%|██▍ | 123/500 [00:06<00:09, 40.17it/s]
Requesting API: 26%|██▌ | 128/500 [00:06<00:10, 36.34it/s]
Requesting API: 27%|██▋ | 133/500 [00:06<00:12, 29.60it/s]
Requesting API: 28%|██▊ | 138/500 [00:06<00:12, 28.55it/s]
Requesting API: 29%|██▉ | 145/500 [00:06<00:10, 34.32it/s]
Requesting API: 30%|██▉ | 149/500 [00:07<00:11, 31.56it/s]
Requesting API: 31%|███ | 153/500 [00:07<00:12, 26.88it/s]
Requesting API: 31%|███▏ | 157/500 [00:07<00:12, 27.77it/s]
Requesting API: 32%|███▏ | 161/500 [00:07<00:11, 29.05it/s]
Requesting API: 33%|███▎ | 165/500 [00:07<00:10, 31.17it/s]
Requesting API: 34%|███▍ | 169/500 [00:07<00:14, 23.08it/s]
Requesting API: 35%|███▍ | 173/500 [00:08<00:12, 25.86it/s]
Requesting API: 35%|███▌ | 177/500 [00:08<00:12, 25.57it/s]
Requesting API: 36%|███▌ | 180/500 [00:08<00:14, 21.45it/s]
Requesting API: 37%|███▋ | 183/500 [00:08<00:15, 20.68it/s]
Requesting API: 38%|███▊ | 188/500 [00:08<00:12, 24.40it/s]
Requesting API: 38%|███▊ | 191/500 [00:08<00:12, 23.84it/s]
Requesting API: 39%|███▉ | 197/500 [00:09<00:10, 29.28it/s]
Requesting API: 40%|████ | 202/500 [00:09<00:09, 33.08it/s]
Requesting API: 41%|████ | 206/500 [00:09<00:09, 31.54it/s]
Requesting API: 42%|████▏ | 212/500 [00:09<00:07, 37.63it/s]
Requesting API: 43%|████▎ | 216/500 [00:09<00:09, 30.46it/s]
Requesting API: 44%|████▍ | 220/500 [00:09<00:09, 29.98it/s]
Requesting API: 45%|████▍ | 224/500 [00:09<00:09, 27.87it/s]
Requesting API: 46%|████▌ | 231/500 [00:10<00:07, 34.33it/s]
Requesting API: 47%|████▋ | 235/500 [00:10<00:07, 33.19it/s]
Requesting API: 48%|████▊ | 239/500 [00:10<00:10, 25.19it/s]
Requesting API: 49%|████▉ | 246/500 [00:10<00:08, 30.88it/s]
Requesting API: 50%|█████ | 250/500 [00:10<00:08, 31.21it/s]
Requesting API: 51%|█████ | 254/500 [00:10<00:07, 32.38it/s]
Requesting API: 52%|█████▏ | 258/500 [00:10<00:07, 31.42it/s]
Requesting API: 52%|█████▏ | 262/500 [00:11<00:08, 26.93it/s]
Requesting API: 53%|█████▎ | 265/500 [00:11<00:09, 25.79it/s]
Requesting API: 54%|█████▎ | 268/500 [00:11<00:09, 25.01it/s]
Requesting API: 54%|█████▍ | 271/500 [00:11<00:08, 25.84it/s]
Requesting API: 55%|█████▌ | 276/500 [00:11<00:07, 29.79it/s]
Requesting API: 56%|█████▌ | 280/500 [00:11<00:08, 26.23it/s]
Requesting API: 57%|█████▋ | 285/500 [00:11<00:06, 31.29it/s]
Requesting API: 58%|█████▊ | 289/500 [00:12<00:06, 30.20it/s]
Requesting API: 59%|█████▊ | 293/500 [00:12<00:07, 27.57it/s]
Requesting API: 60%|██████ | 301/500 [00:12<00:05, 35.52it/s]
Requesting API: 61%|██████ | 305/500 [00:12<00:06, 32.26it/s]
Requesting API: 62%|██████▏ | 309/500 [00:12<00:06, 30.69it/s]
Requesting API: 63%|██████▎ | 313/500 [00:12<00:06, 28.75it/s]
Requesting API: 63%|██████▎ | 316/500 [00:12<00:06, 28.74it/s]
Requesting API: 64%|██████▍ | 319/500 [00:13<00:06, 28.44it/s]
Requesting API: 65%|██████▌ | 326/500 [00:13<00:05, 31.71it/s]
Requesting API: 66%|██████▋ | 332/500 [00:13<00:04, 37.70it/s]
Requesting API: 67%|██████▋ | 336/500 [00:13<00:05, 30.70it/s]
Requesting API: 68%|██████▊ | 340/500 [00:13<00:05, 30.64it/s]
Requesting API: 69%|██████▉ | 345/500 [00:13<00:04, 33.29it/s]
Requesting API: 71%|███████ | 353/500 [00:14<00:03, 37.43it/s]
Requesting API: 72%|███████▏ | 358/500 [00:14<00:03, 39.65it/s]
Requesting API: 73%|███████▎ | 363/500 [00:14<00:04, 32.97it/s]
Requesting API: 73%|███████▎ | 367/500 [00:14<00:05, 25.24it/s]
Requesting API: 74%|███████▍ | 372/500 [00:14<00:04, 28.75it/s]
Requesting API: 76%|███████▌ | 378/500 [00:14<00:03, 33.64it/s]
Requesting API: 77%|███████▋ | 384/500 [00:15<00:03, 36.58it/s]
Requesting API: 78%|███████▊ | 389/500 [00:15<00:03, 30.01it/s]
Requesting API: 79%|███████▉ | 397/500 [00:15<00:02, 39.29it/s]
Requesting API: 81%|████████ | 405/500 [00:15<00:02, 42.28it/s]
Requesting API: 82%|████████▏ | 410/500 [00:15<00:02, 36.03it/s]
Requesting API: 83%|████████▎ | 415/500 [00:15<00:02, 34.43it/s]
Requesting API: 84%|████████▍ | 419/500 [00:16<00:02, 32.52it/s]
Requesting API: 85%|████████▍ | 423/500 [00:16<00:02, 32.73it/s]
Requesting API: 86%|████████▌ | 428/500 [00:16<00:02, 35.64it/s]
Requesting API: 86%|████████▋ | 432/500 [00:16<00:01, 36.00it/s]
Requesting API: 88%|████████▊ | 438/500 [00:16<00:01, 41.50it/s]
Requesting API: 89%|████████▊ | 443/500 [00:16<00:01, 33.91it/s]
Requesting API: 89%|████████▉ | 447/500 [00:16<00:01, 34.59it/s]
Requesting API: 91%|█████████ | 453/500 [00:16<00:01, 36.55it/s]
Requesting API: 91%|█████████▏| 457/500 [00:17<00:01, 37.32it/s]
Requesting API: 92%|█████████▏| 461/500 [00:17<00:01, 32.93it/s]
Requesting API: 93%|█████████▎| 465/500 [00:17<00:01, 28.57it/s]
Requesting API: 94%|█████████▍| 469/500 [00:17<00:01, 23.54it/s]
Requesting API: 94%|█████████▍| 472/500 [00:17<00:01, 24.70it/s]
Requesting API: 95%|█████████▌| 476/500 [00:17<00:01, 23.59it/s]
Requesting API: 96%|█████████▌| 479/500 [00:18<00:01, 13.29it/s]
Requesting API: 96%|█████████▌| 481/500 [00:18<00:01, 12.73it/s]
Requesting API: 97%|█████████▋| 483/500 [00:18<00:01, 13.64it/s]
Requesting API: 97%|█████████▋| 485/500 [00:19<00:01, 11.39it/s]
Requesting API: 97%|█████████▋| 487/500 [00:19<00:01, 11.57it/s]
Requesting API: 98%|█████████▊| 489/500 [00:19<00:00, 11.10it/s]
Requesting API: 98%|█████████▊| 491/500 [00:19<00:01, 7.45it/s]
Requesting API: 99%|█████████▊| 493/500 [00:19<00:00, 8.98it/s]
Requesting API: 99%|█████████▉| 495/500 [00:20<00:00, 10.11it/s]
Requesting API: 99%|█████████▉| 497/500 [00:20<00:00, 10.11it/s]
Requesting API: 100%|██████████| 500/500 [00:20<00:00, 24.50it/s]
2026-08-09:15:51:34 INFO [loggers.evaluation_tracker:247] Saving results aggregated
2026-08-09:15:51:34 INFO [loggers.evaluation_tracker:119] Saving per-task samples to outputs/evals/release/math-keep25-step0050/student/*.jsonl
local-completions ({'model': 'student', 'base_url': 'http://127.0.0.1:8471/v1/completions', 'tokenizer': 'outputs/release/healed/release-offpolicy-math-keep25/step0050', 'num_concurrent': 48, 'tokenized_requests': False, 'max_retries': 3}), gen_kwargs: ({}), limit: None, num_fewshot: None, batch_size: 1
| Tasks |Version| Filter |n-shot| Metric | |Value| |Stderr|
|---------|------:|-----------|-----:|---------|---|----:|---|-----:|
|humaneval| 1|create_test| 0|pass@1 |↑ |0.000|± |0.0000|
|mbpp | 1|none | 3|pass_at_1|↑ |0.054|± |0.0101|
|