File size: 17,313 Bytes
a82fb9e | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 | 2026-08-09:15:45:01 INFO [_cli.run:388] Selected Tasks: ['humaneval', 'mbpp']
2026-08-09:15:45:02 INFO [evaluator:214] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2026-08-09:15:45:02 INFO [evaluator:239] Initializing local-completions model, with arguments: {'model': 'student', 'base_url': 'http://127.0.0.1:8472/v1/completions', 'tokenizer': 'outputs/release/healed/release-offpolicy-math-keep25/step0100', 'num_concurrent': 48, 'tokenized_requests': False, 'max_retries': 3}
2026-08-09:15:45:02 INFO [models.openai_completions:42] Remote tokenizer not supported. Using huggingface tokenizer backend.
2026-08-09:15:45:02 INFO [models.api_models:179] Using max length 2048 - 1
2026-08-09:15:45:02 INFO [models.api_models:200] Using tokenizer huggingface
2026-08-09:15:45:08 INFO [evaluator_utils:446] Selected tasks:
2026-08-09:15:45:08 INFO [evaluator_utils:480] Task: humaneval (humaneval/humaneval.yaml)
2026-08-09:15:45:08 INFO [evaluator_utils:480] Task: mbpp (mbpp/mbpp.yaml)
2026-08-09:15:45:08 INFO [evaluator:314] humaneval: Using gen_kwargs: {'until': ['\nclass', '\ndef', '\n#', '\nif', '\nprint'], 'max_gen_toks': 1024, 'do_sample': False}
2026-08-09:15:45:08 INFO [evaluator:314] mbpp: Using gen_kwargs: {'until': ['[DONE]'], 'do_sample': False}
2026-08-09:15:45:08 INFO [api.task:312] Building contexts for humaneval on rank 0...
0%| | 0/164 [00:00<?, ?it/s]
87%|████████▋ | 142/164 [00:00<00:00, 1415.61it/s]
100%|██████████| 164/164 [00:00<00:00, 1451.90it/s]
2026-08-09:15:45:08 INFO [api.task:312] Building contexts for mbpp on rank 0...
0%| | 0/500 [00:00<?, ?it/s]
3%|▎ | 16/500 [00:00<00:03, 153.65it/s]
6%|▋ | 32/500 [00:00<00:02, 157.28it/s]
10%|▉ | 49/500 [00:00<00:02, 158.70it/s]
13%|█▎ | 66/500 [00:00<00:02, 160.60it/s]
17%|█▋ | 83/500 [00:00<00:02, 162.02it/s]
20%|██ | 100/500 [00:00<00:02, 162.84it/s]
23%|██▎ | 117/500 [00:00<00:02, 163.39it/s]
27%|██▋ | 134/500 [00:00<00:02, 163.79it/s]
30%|███ | 151/500 [00:00<00:02, 164.07it/s]
34%|███▎ | 168/500 [00:01<00:02, 164.46it/s]
37%|███▋ | 185/500 [00:01<00:01, 164.68it/s]
40%|████ | 202/500 [00:01<00:01, 164.88it/s]
44%|████▍ | 219/500 [00:01<00:01, 156.60it/s]
47%|████▋ | 235/500 [00:01<00:01, 157.44it/s]
50%|█████ | 252/500 [00:01<00:01, 159.42it/s]
54%|█████▍ | 269/500 [00:01<00:01, 160.98it/s]
57%|█████▋ | 286/500 [00:01<00:01, 162.12it/s]
61%|██████ | 303/500 [00:01<00:01, 163.06it/s]
64%|██████▍ | 320/500 [00:01<00:01, 163.76it/s]
67%|██████▋ | 337/500 [00:02<00:00, 164.16it/s]
71%|███████ | 354/500 [00:02<00:00, 164.65it/s]
74%|███████▍ | 371/500 [00:02<00:00, 164.25it/s]
78%|███████▊ | 388/500 [00:02<00:00, 163.33it/s]
81%|████████ | 405/500 [00:02<00:00, 162.37it/s]
84%|████████▍ | 422/500 [00:02<00:00, 159.63it/s]
88%|████████▊ | 438/500 [00:02<00:00, 154.30it/s]
91%|█████████ | 455/500 [00:02<00:00, 156.69it/s]
94%|█████████▍| 471/500 [00:02<00:00, 151.89it/s]
97%|█████████▋| 487/500 [00:03<00:00, 151.84it/s]
100%|██████████| 500/500 [00:03<00:00, 160.15it/s]
2026-08-09:15:45:12 INFO [evaluator:585] Running generate_until requests
2026-08-09:15:45:12 INFO [models.api_models:747] Tokenized requests are disabled. Context + generation length is not checked.
Requesting API: 0%| | 0/164 [00:00<?, ?it/s]
Requesting API: 1%| | 1/164 [00:00<01:23, 1.95it/s]
Requesting API: 3%|▎ | 5/164 [00:00<00:16, 9.57it/s]
Requesting API: 9%|▊ | 14/164 [00:00<00:05, 26.03it/s]
Requesting API: 16%|█▌ | 26/164 [00:00<00:03, 44.16it/s]
Requesting API: 20%|██ | 33/164 [00:01<00:02, 49.64it/s]
Requesting API: 24%|██▍ | 40/164 [00:01<00:02, 46.26it/s]
Requesting API: 28%|██▊ | 46/164 [00:02<00:06, 16.95it/s]
Requesting API: 30%|███ | 50/164 [00:02<00:06, 17.69it/s]
Requesting API: 33%|███▎ | 54/164 [00:02<00:06, 17.86it/s]
Requesting API: 35%|███▍ | 57/164 [00:02<00:06, 17.09it/s]
Requesting API: 37%|███▋ | 60/164 [00:02<00:05, 18.03it/s]
Requesting API: 39%|███▉ | 64/164 [00:03<00:05, 17.58it/s]
Requesting API: 41%|████ | 67/164 [00:03<00:05, 17.32it/s]
Requesting API: 43%|████▎ | 71/164 [00:03<00:04, 19.64it/s]
Requesting API: 45%|████▌ | 74/164 [00:03<00:05, 17.10it/s]
Requesting API: 46%|████▋ | 76/164 [00:03<00:05, 17.16it/s]
Requesting API: 48%|████▊ | 78/164 [00:03<00:04, 17.70it/s]
Requesting API: 49%|████▉ | 81/164 [00:03<00:04, 20.27it/s]
Requesting API: 52%|█████▏ | 85/164 [00:04<00:04, 18.39it/s]
Requesting API: 54%|█████▎ | 88/164 [00:04<00:03, 20.60it/s]
Requesting API: 55%|█████▌ | 91/164 [00:04<00:03, 19.66it/s]
Requesting API: 57%|█████▋ | 94/164 [00:04<00:03, 20.65it/s]
Requesting API: 60%|█████▉ | 98/164 [00:04<00:03, 19.07it/s]
Requesting API: 62%|██████▏ | 101/164 [00:05<00:03, 17.33it/s]
Requesting API: 63%|██████▎ | 104/164 [00:05<00:03, 19.19it/s]
Requesting API: 65%|██████▌ | 107/164 [00:05<00:02, 19.43it/s]
Requesting API: 68%|██████▊ | 112/164 [00:05<00:02, 23.95it/s]
Requesting API: 71%|███████ | 116/164 [00:05<00:01, 27.21it/s]
Requesting API: 73%|███████▎ | 119/164 [00:05<00:01, 25.85it/s]
Requesting API: 74%|███████▍ | 122/164 [00:05<00:01, 23.44it/s]
Requesting API: 76%|███████▌ | 125/164 [00:06<00:02, 18.17it/s]
Requesting API: 78%|███████▊ | 128/164 [00:06<00:02, 17.48it/s]
Requesting API: 79%|███████▉ | 130/164 [00:06<00:03, 9.31it/s]
Requesting API: 80%|████████ | 132/164 [00:07<00:03, 9.76it/s]
Requesting API: 82%|████████▏ | 134/164 [00:07<00:05, 5.57it/s]
Requesting API: 83%|████████▎ | 136/164 [00:08<00:05, 5.57it/s]
Requesting API: 85%|████████▌ | 140/164 [00:08<00:02, 8.71it/s]
Requesting API: 87%|████████▋ | 142/164 [00:08<00:03, 6.78it/s]
Requesting API: 88%|████████▊ | 144/164 [00:09<00:03, 5.85it/s]
Requesting API: 89%|████████▉ | 146/164 [00:15<00:16, 1.11it/s]
Requesting API: 96%|█████████▋| 158/164 [00:15<00:01, 3.53it/s]
Requesting API: 99%|█████████▉| 162/164 [00:16<00:00, 3.60it/s]
Requesting API: 100%|██████████| 164/164 [00:18<00:00, 8.66it/s]
Requesting API: 0%| | 0/500 [00:00<?, ?it/s]
Requesting API: 0%| | 1/500 [00:00<05:25, 1.53it/s]
Requesting API: 0%| | 2/500 [00:00<03:21, 2.47it/s]
Requesting API: 1%|▏ | 7/500 [00:01<00:48, 10.07it/s]
Requesting API: 2%|▏ | 9/500 [00:01<00:44, 11.06it/s]
Requesting API: 3%|▎ | 13/500 [00:01<00:29, 16.70it/s]
Requesting API: 3%|▎ | 16/500 [00:01<00:28, 17.13it/s]
Requesting API: 4%|▍ | 20/500 [00:01<00:30, 15.52it/s]
Requesting API: 5%|▍ | 24/500 [00:01<00:24, 19.57it/s]
Requesting API: 5%|▌ | 27/500 [00:02<00:30, 15.54it/s]
Requesting API: 6%|▌ | 30/500 [00:02<00:34, 13.65it/s]
Requesting API: 7%|▋ | 33/500 [00:02<00:30, 15.47it/s]
Requesting API: 7%|▋ | 35/500 [00:02<00:32, 14.47it/s]
Requesting API: 8%|▊ | 38/500 [00:02<00:29, 15.46it/s]
Requesting API: 8%|▊ | 40/500 [00:03<00:29, 15.46it/s]
Requesting API: 9%|▊ | 43/500 [00:03<00:27, 16.63it/s]
Requesting API: 9%|▉ | 46/500 [00:03<00:23, 19.10it/s]
Requesting API: 10%|▉ | 49/500 [00:03<00:27, 16.11it/s]
Requesting API: 10%|█ | 51/500 [00:03<00:27, 16.38it/s]
Requesting API: 11%|█ | 54/500 [00:03<00:24, 18.31it/s]
Requesting API: 11%|█ | 56/500 [00:03<00:29, 14.84it/s]
Requesting API: 12%|█▏ | 61/500 [00:04<00:21, 20.83it/s]
Requesting API: 13%|█▎ | 64/500 [00:04<00:27, 16.10it/s]
Requesting API: 13%|█▎ | 66/500 [00:04<00:26, 16.23it/s]
Requesting API: 14%|█▍ | 69/500 [00:04<00:23, 18.16it/s]
Requesting API: 15%|█▌ | 75/500 [00:04<00:15, 26.64it/s]
Requesting API: 18%|█▊ | 89/500 [00:04<00:08, 50.81it/s]
Requesting API: 19%|█▉ | 97/500 [00:05<00:08, 50.05it/s]
Requesting API: 21%|██ | 103/500 [00:05<00:09, 40.52it/s]
Requesting API: 22%|██▏ | 108/500 [00:05<00:11, 34.80it/s]
Requesting API: 22%|██▏ | 112/500 [00:05<00:11, 33.22it/s]
Requesting API: 23%|██▎ | 117/500 [00:05<00:11, 34.21it/s]
Requesting API: 24%|██▍ | 121/500 [00:05<00:11, 33.36it/s]
Requesting API: 26%|██▌ | 129/500 [00:05<00:08, 43.35it/s]
Requesting API: 27%|██▋ | 134/500 [00:06<00:09, 37.94it/s]
Requesting API: 28%|██▊ | 139/500 [00:06<00:11, 30.80it/s]
Requesting API: 29%|██▊ | 143/500 [00:06<00:12, 28.67it/s]
Requesting API: 29%|██▉ | 147/500 [00:06<00:11, 30.46it/s]
Requesting API: 30%|███ | 151/500 [00:06<00:14, 23.47it/s]
Requesting API: 31%|███▏ | 157/500 [00:07<00:11, 29.52it/s]
Requesting API: 32%|███▏ | 161/500 [00:07<00:11, 28.37it/s]
Requesting API: 33%|███▎ | 165/500 [00:07<00:12, 27.31it/s]
Requesting API: 34%|███▍ | 170/500 [00:07<00:10, 30.79it/s]
Requesting API: 35%|███▍ | 174/500 [00:07<00:11, 29.31it/s]
Requesting API: 36%|███▌ | 178/500 [00:07<00:10, 29.71it/s]
Requesting API: 36%|███▋ | 182/500 [00:07<00:11, 27.89it/s]
Requesting API: 37%|███▋ | 185/500 [00:08<00:12, 25.43it/s]
Requesting API: 38%|███▊ | 188/500 [00:08<00:12, 25.63it/s]
Requesting API: 39%|███▊ | 193/500 [00:08<00:10, 29.21it/s]
Requesting API: 39%|███▉ | 197/500 [00:08<00:10, 28.79it/s]
Requesting API: 41%|████ | 203/500 [00:08<00:09, 32.76it/s]
Requesting API: 41%|████▏ | 207/500 [00:08<00:09, 30.77it/s]
Requesting API: 42%|████▏ | 211/500 [00:08<00:09, 32.01it/s]
Requesting API: 44%|████▍ | 221/500 [00:09<00:06, 45.91it/s]
Requesting API: 45%|████▌ | 226/500 [00:09<00:07, 35.09it/s]
Requesting API: 46%|████▌ | 230/500 [00:09<00:07, 35.47it/s]
Requesting API: 47%|████▋ | 235/500 [00:09<00:06, 37.96it/s]
Requesting API: 48%|████▊ | 240/500 [00:09<00:06, 38.74it/s]
Requesting API: 49%|████▉ | 245/500 [00:09<00:06, 38.99it/s]
Requesting API: 50%|█████ | 251/500 [00:09<00:06, 40.75it/s]
Requesting API: 51%|█████ | 256/500 [00:09<00:06, 39.72it/s]
Requesting API: 52%|█████▏ | 261/500 [00:10<00:07, 33.36it/s]
Requesting API: 53%|█████▎ | 265/500 [00:10<00:07, 31.47it/s]
Requesting API: 54%|█████▍ | 270/500 [00:10<00:06, 33.86it/s]
Requesting API: 55%|█████▍ | 274/500 [00:10<00:07, 30.51it/s]
Requesting API: 56%|█████▌ | 279/500 [00:10<00:07, 29.57it/s]
Requesting API: 57%|█████▋ | 283/500 [00:10<00:07, 28.68it/s]
Requesting API: 58%|█████▊ | 289/500 [00:11<00:07, 29.74it/s]
Requesting API: 59%|█████▊ | 293/500 [00:11<00:07, 27.92it/s]
Requesting API: 60%|█████▉ | 298/500 [00:11<00:06, 29.48it/s]
Requesting API: 61%|██████ | 304/500 [00:11<00:05, 35.13it/s]
Requesting API: 62%|██████▏ | 308/500 [00:11<00:05, 32.54it/s]
Requesting API: 62%|██████▏ | 312/500 [00:11<00:06, 27.60it/s]
Requesting API: 63%|██████▎ | 315/500 [00:12<00:07, 24.70it/s]
Requesting API: 64%|██████▍ | 320/500 [00:12<00:06, 29.42it/s]
Requesting API: 65%|██████▌ | 325/500 [00:12<00:06, 28.80it/s]
Requesting API: 66%|██████▌ | 329/500 [00:12<00:06, 27.34it/s]
Requesting API: 67%|██████▋ | 334/500 [00:12<00:05, 29.86it/s]
Requesting API: 68%|██████▊ | 338/500 [00:12<00:05, 31.57it/s]
Requesting API: 69%|██████▊ | 343/500 [00:12<00:04, 34.18it/s]
Requesting API: 70%|███████ | 352/500 [00:13<00:03, 39.89it/s]
Requesting API: 71%|███████ | 356/500 [00:13<00:03, 38.37it/s]
Requesting API: 72%|███████▏ | 360/500 [00:13<00:04, 30.72it/s]
Requesting API: 73%|███████▎ | 364/500 [00:13<00:04, 28.99it/s]
Requesting API: 74%|███████▎ | 368/500 [00:13<00:04, 28.52it/s]
Requesting API: 74%|███████▍ | 372/500 [00:13<00:04, 29.72it/s]
Requesting API: 75%|███████▌ | 376/500 [00:13<00:03, 31.77it/s]
Requesting API: 76%|███████▋ | 382/500 [00:14<00:03, 38.02it/s]
Requesting API: 77%|███████▋ | 387/500 [00:14<00:02, 38.66it/s]
Requesting API: 78%|███████▊ | 391/500 [00:14<00:03, 32.69it/s]
Requesting API: 79%|███████▉ | 395/500 [00:14<00:03, 33.10it/s]
Requesting API: 80%|███████▉ | 399/500 [00:14<00:02, 34.15it/s]
Requesting API: 81%|████████ | 403/500 [00:14<00:02, 35.15it/s]
Requesting API: 81%|████████▏ | 407/500 [00:14<00:03, 27.69it/s]
Requesting API: 82%|████████▏ | 411/500 [00:15<00:03, 23.89it/s]
Requesting API: 83%|████████▎ | 414/500 [00:15<00:03, 24.09it/s]
Requesting API: 83%|████████▎ | 417/500 [00:15<00:03, 22.98it/s]
Requesting API: 84%|████████▍ | 421/500 [00:15<00:03, 24.02it/s]
Requesting API: 85%|████████▌ | 425/500 [00:15<00:02, 26.26it/s]
Requesting API: 86%|████████▌ | 430/500 [00:15<00:02, 28.36it/s]
Requesting API: 87%|████████▋ | 433/500 [00:16<00:02, 23.66it/s]
Requesting API: 87%|████████▋ | 436/500 [00:16<00:02, 24.03it/s]
Requesting API: 88%|████████▊ | 441/500 [00:16<00:02, 25.34it/s]
Requesting API: 89%|████████▉ | 444/500 [00:16<00:02, 25.43it/s]
Requesting API: 89%|████████▉ | 447/500 [00:16<00:02, 21.20it/s]
Requesting API: 91%|█████████ | 454/500 [00:16<00:01, 30.71it/s]
Requesting API: 92%|█████████▏| 459/500 [00:16<00:01, 29.92it/s]
Requesting API: 93%|█████████▎| 463/500 [00:17<00:01, 30.05it/s]
Requesting API: 94%|█████████▎| 468/500 [00:17<00:01, 29.63it/s]
Requesting API: 94%|█████████▍| 472/500 [00:17<00:01, 27.52it/s]
Requesting API: 95%|█████████▌| 475/500 [00:17<00:00, 25.02it/s]
Requesting API: 96%|█████████▌| 478/500 [00:17<00:01, 20.98it/s]
Requesting API: 96%|█████████▋| 482/500 [00:17<00:00, 22.85it/s]
Requesting API: 97%|█████████▋| 485/500 [00:18<00:00, 18.88it/s]
Requesting API: 98%|█████████▊| 489/500 [00:18<00:00, 21.18it/s]
Requesting API: 98%|█████████▊| 492/500 [00:18<00:00, 14.78it/s]
Requesting API: 99%|█████████▉| 494/500 [00:18<00:00, 13.34it/s]
Requesting API: 99%|█████████▉| 496/500 [00:19<00:00, 7.76it/s]
Requesting API: 100%|█████████▉| 498/500 [00:19<00:00, 7.26it/s]
Requesting API: 100%|██████████| 500/500 [00:20<00:00, 6.36it/s]
Requesting API: 100%|██████████| 500/500 [00:20<00:00, 24.62it/s]
2026-08-09:15:50:45 INFO [loggers.evaluation_tracker:247] Saving results aggregated
2026-08-09:15:50:45 INFO [loggers.evaluation_tracker:119] Saving per-task samples to outputs/evals/release/math-keep25-step0100/student/*.jsonl
local-completions ({'model': 'student', 'base_url': 'http://127.0.0.1:8472/v1/completions', 'tokenizer': 'outputs/release/healed/release-offpolicy-math-keep25/step0100', 'num_concurrent': 48, 'tokenized_requests': False, 'max_retries': 3}), gen_kwargs: ({}), limit: None, num_fewshot: None, batch_size: 1
| Tasks |Version| Filter |n-shot| Metric | |Value| |Stderr|
|---------|------:|-----------|-----:|---------|---|----:|---|-----:|
|humaneval| 1|create_test| 0|pass@1 |↑ |0.000|± |0.0000|
|mbpp | 1|none | 3|pass_at_1|↑ |0.062|± |0.0108|
|