File size: 16,852 Bytes
a82fb9e | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 | 2026-08-08:22:10:01 INFO [_cli.run:388] Selected Tasks: ['humaneval', 'mbpp']
2026-08-08:22:10:02 INFO [evaluator:214] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2026-08-08:22:10:02 INFO [evaluator:239] Initializing local-completions model, with arguments: {'model': 'student', 'base_url': 'http://127.0.0.1:8460/v1/completions', 'tokenizer': 'outputs/release/healed/release-offpolicy-math-keep25/step0150', 'num_concurrent': 48, 'tokenized_requests': False, 'max_retries': 3}
2026-08-08:22:10:02 INFO [models.openai_completions:42] Remote tokenizer not supported. Using huggingface tokenizer backend.
2026-08-08:22:10:02 INFO [models.api_models:179] Using max length 2048 - 1
2026-08-08:22:10:02 INFO [models.api_models:200] Using tokenizer huggingface
2026-08-08:22:10:16 INFO [evaluator_utils:446] Selected tasks:
2026-08-08:22:10:16 INFO [evaluator_utils:480] Task: humaneval (humaneval/humaneval.yaml)
2026-08-08:22:10:16 INFO [evaluator_utils:480] Task: mbpp (mbpp/mbpp.yaml)
2026-08-08:22:10:16 INFO [evaluator:314] humaneval: Using gen_kwargs: {'until': ['\nclass', '\ndef', '\n#', '\nif', '\nprint'], 'max_gen_toks': 1024, 'do_sample': False}
2026-08-08:22:10:16 INFO [evaluator:314] mbpp: Using gen_kwargs: {'until': ['[DONE]'], 'do_sample': False}
2026-08-08:22:10:16 INFO [api.task:312] Building contexts for humaneval on rank 0...
0%| | 0/164 [00:00<?, ?it/s]
84%|████████▎ | 137/164 [00:00<00:00, 1365.76it/s]
100%|██████████| 164/164 [00:00<00:00, 1404.75it/s]
2026-08-08:22:10:16 INFO [api.task:312] Building contexts for mbpp on rank 0...
0%| | 0/500 [00:00<?, ?it/s]
3%|▎ | 16/500 [00:00<00:03, 153.27it/s]
6%|▋ | 32/500 [00:00<00:03, 154.42it/s]
10%|▉ | 48/500 [00:00<00:02, 155.19it/s]
13%|█▎ | 64/500 [00:00<00:02, 155.46it/s]
16%|█▌ | 80/500 [00:00<00:02, 154.05it/s]
19%|█▉ | 96/500 [00:00<00:02, 154.63it/s]
22%|██▏ | 112/500 [00:00<00:02, 155.24it/s]
26%|██▌ | 128/500 [00:00<00:02, 155.20it/s]
29%|██▉ | 144/500 [00:00<00:02, 155.06it/s]
32%|███▏ | 160/500 [00:01<00:02, 154.80it/s]
35%|███▌ | 176/500 [00:01<00:02, 154.94it/s]
38%|███▊ | 192/500 [00:01<00:01, 155.40it/s]
42%|████▏ | 208/500 [00:01<00:01, 155.74it/s]
45%|████▍ | 224/500 [00:01<00:01, 155.57it/s]
48%|████▊ | 240/500 [00:01<00:01, 155.79it/s]
51%|█████ | 256/500 [00:01<00:01, 155.50it/s]
54%|█████▍ | 272/500 [00:01<00:01, 155.85it/s]
58%|█████▊ | 288/500 [00:01<00:01, 156.05it/s]
61%|██████ | 304/500 [00:01<00:01, 156.87it/s]
64%|██████▍ | 320/500 [00:02<00:01, 157.07it/s]
67%|██████▋ | 336/500 [00:02<00:01, 157.46it/s]
70%|███████ | 352/500 [00:02<00:00, 157.72it/s]
74%|███████▎ | 368/500 [00:02<00:00, 156.92it/s]
77%|███████▋ | 384/500 [00:02<00:00, 157.08it/s]
80%|████████ | 400/500 [00:02<00:00, 157.03it/s]
83%|████████▎ | 416/500 [00:02<00:00, 157.13it/s]
87%|████████▋ | 433/500 [00:02<00:00, 158.18it/s]
90%|████████▉ | 449/500 [00:02<00:00, 158.71it/s]
93%|█████████▎| 465/500 [00:02<00:00, 158.69it/s]
96%|█████████▌| 481/500 [00:03<00:00, 158.96it/s]
99%|█████████▉| 497/500 [00:03<00:00, 158.84it/s]
100%|██████████| 500/500 [00:03<00:00, 156.57it/s]
2026-08-08:22:10:20 INFO [evaluator:585] Running generate_until requests
2026-08-08:22:10:20 INFO [models.api_models:747] Tokenized requests are disabled. Context + generation length is not checked.
Requesting API: 0%| | 0/164 [00:00<?, ?it/s]
Requesting API: 1%| | 1/164 [00:00<01:26, 1.89it/s]
Requesting API: 4%|▎ | 6/164 [00:00<00:13, 11.47it/s]
Requesting API: 9%|▉ | 15/164 [00:00<00:05, 27.75it/s]
Requesting API: 13%|█▎ | 21/164 [00:00<00:04, 32.27it/s]
Requesting API: 18%|█▊ | 29/164 [00:01<00:03, 39.91it/s]
Requesting API: 21%|██▏ | 35/164 [00:01<00:03, 41.06it/s]
Requesting API: 24%|██▍ | 40/164 [00:01<00:06, 19.58it/s]
Requesting API: 27%|██▋ | 44/164 [00:02<00:08, 14.91it/s]
Requesting API: 29%|██▊ | 47/164 [00:02<00:08, 13.34it/s]
Requesting API: 30%|███ | 50/164 [00:02<00:08, 13.12it/s]
Requesting API: 32%|███▏ | 52/164 [00:02<00:08, 13.82it/s]
Requesting API: 34%|███▎ | 55/164 [00:03<00:10, 10.58it/s]
Requesting API: 35%|███▍ | 57/164 [00:03<00:10, 10.66it/s]
Requesting API: 38%|███▊ | 62/164 [00:03<00:06, 15.54it/s]
Requesting API: 40%|███▉ | 65/164 [00:03<00:06, 16.04it/s]
Requesting API: 41%|████▏ | 68/164 [00:04<00:05, 16.45it/s]
Requesting API: 43%|████▎ | 70/164 [00:04<00:05, 16.79it/s]
Requesting API: 45%|████▌ | 74/164 [00:04<00:04, 19.78it/s]
Requesting API: 47%|████▋ | 77/164 [00:04<00:04, 20.29it/s]
Requesting API: 49%|████▉ | 81/164 [00:04<00:03, 24.03it/s]
Requesting API: 52%|█████▏ | 86/164 [00:04<00:03, 21.98it/s]
Requesting API: 54%|█████▍ | 89/164 [00:05<00:05, 14.63it/s]
Requesting API: 57%|█████▋ | 93/164 [00:05<00:03, 17.99it/s]
Requesting API: 59%|█████▊ | 96/164 [00:05<00:03, 17.49it/s]
Requesting API: 60%|██████ | 99/164 [00:05<00:05, 12.96it/s]
Requesting API: 62%|██████▏ | 101/164 [00:06<00:04, 13.95it/s]
Requesting API: 65%|██████▍ | 106/164 [00:06<00:03, 16.46it/s]
Requesting API: 68%|██████▊ | 111/164 [00:06<00:02, 21.49it/s]
Requesting API: 70%|██████▉ | 114/164 [00:06<00:02, 22.45it/s]
Requesting API: 71%|███████▏ | 117/164 [00:06<00:02, 19.49it/s]
Requesting API: 74%|███████▍ | 122/164 [00:06<00:01, 22.36it/s]
Requesting API: 76%|███████▌ | 125/164 [00:07<00:02, 19.14it/s]
Requesting API: 79%|███████▊ | 129/164 [00:07<00:01, 21.37it/s]
Requesting API: 80%|████████ | 132/164 [00:08<00:03, 9.09it/s]
Requesting API: 82%|████████▏ | 134/164 [00:08<00:03, 8.65it/s]
Requesting API: 83%|████████▎ | 136/164 [00:08<00:03, 9.19it/s]
Requesting API: 84%|████████▍ | 138/164 [00:08<00:03, 8.09it/s]
Requesting API: 85%|████████▌ | 140/164 [00:12<00:12, 1.88it/s]
Requesting API: 86%|████████▌ | 141/164 [00:15<00:20, 1.12it/s]
Requesting API: 95%|█████████▌| 156/164 [00:15<00:01, 4.68it/s]
Requesting API: 98%|█████████▊| 161/164 [00:16<00:00, 4.46it/s]
Requesting API: 100%|██████████| 164/164 [00:19<00:00, 8.60it/s]
Requesting API: 0%| | 0/500 [00:00<?, ?it/s]
Requesting API: 0%| | 1/500 [00:00<07:15, 1.15it/s]
Requesting API: 1%| | 3/500 [00:00<02:13, 3.72it/s]
Requesting API: 1%|▏ | 7/500 [00:01<00:51, 9.49it/s]
Requesting API: 2%|▏ | 10/500 [00:01<00:44, 11.05it/s]
Requesting API: 3%|▎ | 13/500 [00:01<00:35, 13.76it/s]
Requesting API: 4%|▍ | 19/500 [00:01<00:21, 22.57it/s]
Requesting API: 5%|▍ | 23/500 [00:01<00:23, 20.69it/s]
Requesting API: 6%|▌ | 28/500 [00:01<00:19, 23.91it/s]
Requesting API: 6%|▋ | 32/500 [00:02<00:19, 24.62it/s]
Requesting API: 7%|▋ | 35/500 [00:02<00:18, 25.52it/s]
Requesting API: 8%|▊ | 38/500 [00:02<00:18, 24.79it/s]
Requesting API: 8%|▊ | 41/500 [00:02<00:19, 23.53it/s]
Requesting API: 9%|▉ | 44/500 [00:02<00:23, 19.34it/s]
Requesting API: 10%|▉ | 48/500 [00:02<00:20, 22.42it/s]
Requesting API: 10%|█ | 51/500 [00:02<00:21, 21.33it/s]
Requesting API: 11%|█ | 54/500 [00:03<00:20, 21.96it/s]
Requesting API: 12%|█▏ | 58/500 [00:03<00:19, 22.18it/s]
Requesting API: 12%|█▏ | 62/500 [00:03<00:17, 24.59it/s]
Requesting API: 13%|█▎ | 65/500 [00:03<00:21, 20.26it/s]
Requesting API: 14%|█▎ | 68/500 [00:03<00:21, 19.84it/s]
Requesting API: 14%|█▍ | 71/500 [00:03<00:20, 20.44it/s]
Requesting API: 15%|█▍ | 74/500 [00:04<00:19, 21.69it/s]
Requesting API: 15%|█▌ | 77/500 [00:04<00:19, 22.07it/s]
Requesting API: 16%|█▌ | 81/500 [00:04<00:18, 22.19it/s]
Requesting API: 17%|█▋ | 84/500 [00:04<00:19, 21.37it/s]
Requesting API: 18%|█▊ | 90/500 [00:04<00:14, 28.40it/s]
Requesting API: 19%|█▉ | 95/500 [00:04<00:12, 31.91it/s]
Requesting API: 20%|█▉ | 99/500 [00:04<00:12, 32.76it/s]
Requesting API: 21%|██ | 103/500 [00:04<00:12, 32.13it/s]
Requesting API: 21%|██▏ | 107/500 [00:05<00:11, 33.06it/s]
Requesting API: 24%|██▎ | 118/500 [00:05<00:10, 36.32it/s]
Requesting API: 24%|██▍ | 122/500 [00:05<00:11, 34.31it/s]
Requesting API: 25%|██▌ | 127/500 [00:05<00:10, 36.37it/s]
Requesting API: 26%|██▌ | 131/500 [00:05<00:10, 35.14it/s]
Requesting API: 27%|██▋ | 136/500 [00:05<00:09, 38.06it/s]
Requesting API: 28%|██▊ | 142/500 [00:06<00:09, 37.70it/s]
Requesting API: 29%|██▉ | 147/500 [00:06<00:08, 39.49it/s]
Requesting API: 30%|███ | 152/500 [00:06<00:09, 35.13it/s]
Requesting API: 31%|███ | 156/500 [00:06<00:09, 35.56it/s]
Requesting API: 32%|███▏ | 161/500 [00:06<00:09, 35.00it/s]
Requesting API: 33%|███▎ | 165/500 [00:06<00:10, 33.10it/s]
Requesting API: 34%|███▍ | 171/500 [00:06<00:08, 38.76it/s]
Requesting API: 35%|███▌ | 176/500 [00:07<00:09, 34.33it/s]
Requesting API: 36%|███▌ | 180/500 [00:07<00:09, 34.01it/s]
Requesting API: 37%|███▋ | 185/500 [00:07<00:08, 35.27it/s]
Requesting API: 38%|███▊ | 189/500 [00:07<00:10, 30.25it/s]
Requesting API: 39%|███▉ | 194/500 [00:07<00:10, 28.91it/s]
Requesting API: 40%|███▉ | 198/500 [00:07<00:10, 30.00it/s]
Requesting API: 40%|████ | 202/500 [00:07<00:10, 28.59it/s]
Requesting API: 41%|████ | 205/500 [00:08<00:11, 24.63it/s]
Requesting API: 42%|████▏ | 209/500 [00:08<00:10, 27.39it/s]
Requesting API: 43%|████▎ | 215/500 [00:08<00:09, 28.76it/s]
Requesting API: 45%|████▍ | 223/500 [00:08<00:07, 37.08it/s]
Requesting API: 45%|████▌ | 227/500 [00:08<00:07, 34.45it/s]
Requesting API: 46%|████▌ | 231/500 [00:08<00:08, 32.95it/s]
Requesting API: 47%|████▋ | 236/500 [00:08<00:08, 29.86it/s]
Requesting API: 48%|████▊ | 241/500 [00:09<00:08, 32.30it/s]
Requesting API: 49%|████▉ | 245/500 [00:09<00:07, 33.37it/s]
Requesting API: 51%|█████ | 253/500 [00:09<00:05, 43.73it/s]
Requesting API: 52%|█████▏ | 258/500 [00:09<00:06, 37.75it/s]
Requesting API: 53%|█████▎ | 263/500 [00:09<00:06, 34.09it/s]
Requesting API: 54%|█████▎ | 268/500 [00:09<00:06, 34.45it/s]
Requesting API: 55%|█████▍ | 274/500 [00:09<00:05, 39.22it/s]
Requesting API: 56%|█████▌ | 279/500 [00:10<00:07, 29.56it/s]
Requesting API: 57%|█████▋ | 283/500 [00:10<00:06, 31.03it/s]
Requesting API: 57%|█████▋ | 287/500 [00:10<00:07, 30.43it/s]
Requesting API: 59%|█████▊ | 293/500 [00:10<00:05, 36.09it/s]
Requesting API: 60%|█████▉ | 299/500 [00:10<00:04, 40.59it/s]
Requesting API: 61%|██████ | 304/500 [00:10<00:04, 40.51it/s]
Requesting API: 62%|██████▏ | 309/500 [00:10<00:04, 39.10it/s]
Requesting API: 63%|██████▎ | 314/500 [00:11<00:04, 39.41it/s]
Requesting API: 64%|██████▍ | 319/500 [00:11<00:05, 31.82it/s]
Requesting API: 65%|██████▍ | 323/500 [00:11<00:05, 32.98it/s]
Requesting API: 65%|██████▌ | 327/500 [00:11<00:05, 34.01it/s]
Requesting API: 66%|██████▌ | 331/500 [00:11<00:04, 34.77it/s]
Requesting API: 67%|██████▋ | 335/500 [00:11<00:04, 33.84it/s]
Requesting API: 68%|██████▊ | 342/500 [00:11<00:03, 42.15it/s]
Requesting API: 69%|██████▉ | 347/500 [00:11<00:03, 41.79it/s]
Requesting API: 70%|███████ | 352/500 [00:12<00:03, 38.75it/s]
Requesting API: 71%|███████▏ | 357/500 [00:12<00:03, 37.73it/s]
Requesting API: 73%|███████▎ | 365/500 [00:12<00:02, 46.79it/s]
Requesting API: 74%|███████▍ | 370/500 [00:12<00:03, 42.13it/s]
Requesting API: 75%|███████▌ | 375/500 [00:12<00:03, 35.54it/s]
Requesting API: 76%|███████▌ | 380/500 [00:12<00:03, 36.89it/s]
Requesting API: 77%|███████▋ | 384/500 [00:12<00:03, 36.84it/s]
Requesting API: 78%|███████▊ | 388/500 [00:13<00:03, 36.83it/s]
Requesting API: 79%|███████▉ | 394/500 [00:13<00:02, 40.02it/s]
Requesting API: 80%|███████▉ | 399/500 [00:13<00:03, 29.62it/s]
Requesting API: 81%|████████ | 405/500 [00:13<00:02, 32.27it/s]
Requesting API: 82%|████████▏ | 409/500 [00:13<00:02, 30.54it/s]
Requesting API: 83%|████████▎ | 413/500 [00:13<00:02, 29.37it/s]
Requesting API: 83%|████████▎ | 417/500 [00:14<00:03, 23.77it/s]
Requesting API: 84%|████████▍ | 420/500 [00:14<00:03, 24.64it/s]
Requesting API: 85%|████████▍ | 424/500 [00:14<00:03, 25.20it/s]
Requesting API: 86%|████████▌ | 429/500 [00:14<00:02, 25.82it/s]
Requesting API: 87%|████████▋ | 436/500 [00:14<00:01, 33.89it/s]
Requesting API: 88%|████████▊ | 442/500 [00:14<00:01, 36.70it/s]
Requesting API: 89%|████████▉ | 446/500 [00:15<00:01, 32.39it/s]
Requesting API: 90%|█████████ | 450/500 [00:15<00:01, 33.54it/s]
Requesting API: 91%|█████████ | 455/500 [00:15<00:01, 35.77it/s]
Requesting API: 92%|█████████▏| 459/500 [00:15<00:01, 34.06it/s]
Requesting API: 93%|█████████▎| 463/500 [00:15<00:01, 31.68it/s]
Requesting API: 93%|█████████▎| 467/500 [00:15<00:01, 26.98it/s]
Requesting API: 94%|█████████▍| 470/500 [00:15<00:01, 27.50it/s]
Requesting API: 95%|█████████▌| 475/500 [00:16<00:00, 27.06it/s]
Requesting API: 96%|█████████▌| 478/500 [00:16<00:00, 23.35it/s]
Requesting API: 96%|█████████▌| 481/500 [00:16<00:01, 15.80it/s]
Requesting API: 97%|█████████▋| 483/500 [00:16<00:01, 15.09it/s]
Requesting API: 97%|█████████▋| 486/500 [00:16<00:00, 16.07it/s]
Requesting API: 98%|█████████▊| 491/500 [00:17<00:00, 21.70it/s]
Requesting API: 99%|█████████▉| 494/500 [00:17<00:00, 14.89it/s]
Requesting API: 99%|█████████▉| 497/500 [00:18<00:00, 7.07it/s]
Requesting API: 100%|█████████▉| 499/500 [00:18<00:00, 7.63it/s]
Requesting API: 100%|██████████| 500/500 [00:18<00:00, 26.60it/s]
2026-08-08:22:15:09 INFO [loggers.evaluation_tracker:247] Saving results aggregated
2026-08-08:22:15:09 INFO [loggers.evaluation_tracker:119] Saving per-task samples to outputs/evals/release/math-keep25-step0150/student/*.jsonl
local-completions ({'model': 'student', 'base_url': 'http://127.0.0.1:8460/v1/completions', 'tokenizer': 'outputs/release/healed/release-offpolicy-math-keep25/step0150', 'num_concurrent': 48, 'tokenized_requests': False, 'max_retries': 3}), gen_kwargs: ({}), limit: None, num_fewshot: None, batch_size: 1
| Tasks |Version| Filter |n-shot| Metric | |Value| |Stderr|
|---------|------:|-----------|-----:|---------|---|----:|---|-----:|
|humaneval| 1|create_test| 0|pass@1 |↑ |0.000|± |0.0000|
|mbpp | 1|none | 3|pass_at_1|↑ |0.068|± |0.0113|
|