File size: 14,051 Bytes
a82fb9e | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 | 2026-08-08:22:08:27 INFO [_cli.run:388] Selected Tasks: ['humaneval', 'mbpp']
2026-08-08:22:08:28 INFO [evaluator:214] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2026-08-08:22:08:28 INFO [evaluator:239] Initializing local-completions model, with arguments: {'model': 'student', 'base_url': 'http://127.0.0.1:8461/v1/completions', 'tokenizer': 'outputs/release/healed/release-offpolicy-math-keep50/step0150', 'num_concurrent': 48, 'tokenized_requests': False, 'max_retries': 3}
2026-08-08:22:08:28 INFO [models.openai_completions:42] Remote tokenizer not supported. Using huggingface tokenizer backend.
2026-08-08:22:08:28 INFO [models.api_models:179] Using max length 2048 - 1
2026-08-08:22:08:28 INFO [models.api_models:200] Using tokenizer huggingface
2026-08-08:22:08:41 INFO [evaluator_utils:446] Selected tasks:
2026-08-08:22:08:41 INFO [evaluator_utils:480] Task: humaneval (humaneval/humaneval.yaml)
2026-08-08:22:08:41 INFO [evaluator_utils:480] Task: mbpp (mbpp/mbpp.yaml)
2026-08-08:22:08:41 INFO [evaluator:314] humaneval: Using gen_kwargs: {'until': ['\nclass', '\ndef', '\n#', '\nif', '\nprint'], 'max_gen_toks': 1024, 'do_sample': False}
2026-08-08:22:08:41 INFO [evaluator:314] mbpp: Using gen_kwargs: {'until': ['[DONE]'], 'do_sample': False}
2026-08-08:22:08:41 INFO [api.task:312] Building contexts for humaneval on rank 0...
0%| | 0/164 [00:00<?, ?it/s]
82%|████████▏ | 134/164 [00:00<00:00, 1337.47it/s]
100%|██████████| 164/164 [00:00<00:00, 1391.86it/s]
2026-08-08:22:08:41 INFO [api.task:312] Building contexts for mbpp on rank 0...
0%| | 0/500 [00:00<?, ?it/s]
3%|▎ | 15/500 [00:00<00:03, 144.07it/s]
6%|▌ | 30/500 [00:00<00:03, 146.74it/s]
9%|▉ | 46/500 [00:00<00:03, 148.96it/s]
12%|█▏ | 61/500 [00:00<00:02, 149.28it/s]
15%|█▌ | 77/500 [00:00<00:02, 149.71it/s]
19%|█▊ | 93/500 [00:00<00:02, 149.64it/s]
22%|██▏ | 109/500 [00:00<00:02, 150.14it/s]
25%|██▌ | 125/500 [00:00<00:02, 150.51it/s]
28%|██▊ | 141/500 [00:00<00:02, 150.98it/s]
31%|███▏ | 157/500 [00:01<00:02, 151.60it/s]
35%|███▍ | 173/500 [00:01<00:02, 151.58it/s]
38%|███▊ | 189/500 [00:01<00:02, 150.79it/s]
41%|████ | 205/500 [00:01<00:01, 150.61it/s]
44%|████▍ | 221/500 [00:01<00:01, 150.63it/s]
47%|████▋ | 237/500 [00:01<00:01, 152.02it/s]
51%|█████ | 253/500 [00:01<00:01, 151.92it/s]
54%|█████▍ | 269/500 [00:01<00:01, 152.62it/s]
57%|█████▋ | 285/500 [00:01<00:01, 152.93it/s]
60%|██████ | 301/500 [00:01<00:01, 152.21it/s]
63%|██████▎ | 317/500 [00:02<00:01, 152.12it/s]
67%|██████▋ | 333/500 [00:02<00:01, 151.84it/s]
70%|██████▉ | 349/500 [00:02<00:00, 152.76it/s]
73%|███████▎ | 365/500 [00:02<00:00, 153.54it/s]
76%|███████▌ | 381/500 [00:02<00:00, 152.49it/s]
79%|███████▉ | 397/500 [00:02<00:00, 151.06it/s]
83%|████████▎ | 413/500 [00:02<00:00, 151.07it/s]
86%|████████▌ | 429/500 [00:02<00:00, 150.52it/s]
89%|████████▉ | 445/500 [00:02<00:00, 152.29it/s]
92%|█████████▏| 461/500 [00:03<00:00, 151.39it/s]
95%|█████████▌| 477/500 [00:03<00:00, 152.15it/s]
99%|█████████▊| 493/500 [00:03<00:00, 151.93it/s]
100%|██████████| 500/500 [00:03<00:00, 151.33it/s]
2026-08-08:22:08:44 INFO [evaluator:585] Running generate_until requests
2026-08-08:22:08:44 INFO [models.api_models:747] Tokenized requests are disabled. Context + generation length is not checked.
Requesting API: 0%| | 0/164 [00:00<?, ?it/s]
Requesting API: 1%| | 1/164 [00:00<02:24, 1.13it/s]
Requesting API: 2%|▏ | 3/164 [00:01<00:46, 3.44it/s]
Requesting API: 2%|▏ | 4/164 [00:01<00:41, 3.90it/s]
Requesting API: 3%|▎ | 5/164 [00:01<00:35, 4.52it/s]
Requesting API: 7%|▋ | 11/164 [00:01<00:11, 13.61it/s]
Requesting API: 9%|▊ | 14/164 [00:01<00:09, 15.37it/s]
Requesting API: 10%|█ | 17/164 [00:01<00:10, 14.59it/s]
Requesting API: 12%|█▏ | 19/164 [00:02<00:10, 14.29it/s]
Requesting API: 13%|█▎ | 22/164 [00:02<00:08, 16.46it/s]
Requesting API: 16%|█▌ | 26/164 [00:02<00:06, 20.87it/s]
Requesting API: 18%|█▊ | 29/164 [00:02<00:06, 22.35it/s]
Requesting API: 20%|██ | 33/164 [00:02<00:05, 24.68it/s]
Requesting API: 22%|██▏ | 36/164 [00:02<00:05, 23.81it/s]
Requesting API: 26%|██▌ | 42/164 [00:02<00:03, 31.57it/s]
Requesting API: 28%|██▊ | 46/164 [00:02<00:04, 26.94it/s]
Requesting API: 31%|███ | 51/164 [00:03<00:03, 29.88it/s]
Requesting API: 34%|███▎ | 55/164 [00:03<00:03, 31.28it/s]
Requesting API: 36%|███▌ | 59/164 [00:03<00:04, 24.14it/s]
Requesting API: 38%|███▊ | 63/164 [00:03<00:03, 27.00it/s]
Requesting API: 42%|████▏ | 69/164 [00:03<00:02, 33.72it/s]
Requesting API: 46%|████▋ | 76/164 [00:03<00:02, 39.61it/s]
Requesting API: 50%|█████ | 82/164 [00:03<00:01, 44.15it/s]
Requesting API: 53%|█████▎ | 87/164 [00:04<00:02, 37.51it/s]
Requesting API: 56%|█████▌ | 92/164 [00:04<00:02, 33.74it/s]
Requesting API: 60%|█████▉ | 98/164 [00:04<00:01, 35.97it/s]
Requesting API: 66%|██████▋ | 109/164 [00:04<00:01, 50.57it/s]
Requesting API: 70%|███████ | 115/164 [00:04<00:00, 50.20it/s]
Requesting API: 74%|███████▍ | 121/164 [00:04<00:00, 48.56it/s]
Requesting API: 79%|███████▊ | 129/164 [00:04<00:00, 55.69it/s]
Requesting API: 82%|████████▏ | 135/164 [00:05<00:00, 56.46it/s]
Requesting API: 86%|████████▌ | 141/164 [00:05<00:00, 53.32it/s]
Requesting API: 90%|████████▉ | 147/164 [00:05<00:00, 45.36it/s]
Requesting API: 93%|█████████▎| 152/164 [00:05<00:00, 38.86it/s]
Requesting API: 96%|█████████▌| 157/164 [00:06<00:00, 18.86it/s]
Requesting API: 98%|█████████▊| 161/164 [00:07<00:00, 7.09it/s]
Requesting API: 100%|██████████| 164/164 [00:18<00:00, 1.27it/s]
Requesting API: 100%|██████████| 164/164 [00:18<00:00, 9.09it/s]
Requesting API: 0%| | 0/500 [00:00<?, ?it/s]
Requesting API: 0%| | 1/500 [00:00<07:55, 1.05it/s]
Requesting API: 1%| | 3/500 [00:01<02:21, 3.52it/s]
Requesting API: 1%| | 5/500 [00:01<01:29, 5.52it/s]
Requesting API: 2%|▏ | 12/500 [00:01<00:30, 15.78it/s]
Requesting API: 4%|▍ | 20/500 [00:01<00:18, 25.41it/s]
Requesting API: 5%|▌ | 26/500 [00:01<00:14, 31.68it/s]
Requesting API: 6%|▌ | 31/500 [00:01<00:15, 30.70it/s]
Requesting API: 7%|▋ | 35/500 [00:01<00:15, 30.12it/s]
Requesting API: 8%|▊ | 39/500 [00:02<00:14, 31.37it/s]
Requesting API: 9%|▉ | 47/500 [00:02<00:11, 40.58it/s]
Requesting API: 10%|█ | 52/500 [00:02<00:12, 34.93it/s]
Requesting API: 12%|█▏ | 59/500 [00:02<00:11, 38.92it/s]
Requesting API: 13%|█▎ | 64/500 [00:02<00:12, 36.03it/s]
Requesting API: 14%|█▍ | 70/500 [00:02<00:11, 37.33it/s]
Requesting API: 15%|█▌ | 75/500 [00:02<00:12, 34.04it/s]
Requesting API: 16%|█▌ | 80/500 [00:03<00:11, 35.66it/s]
Requesting API: 17%|█▋ | 86/500 [00:03<00:10, 39.81it/s]
Requesting API: 18%|█▊ | 91/500 [00:03<00:10, 39.09it/s]
Requesting API: 20%|█▉ | 98/500 [00:03<00:09, 41.38it/s]
Requesting API: 21%|██ | 105/500 [00:03<00:09, 40.45it/s]
Requesting API: 22%|██▏ | 110/500 [00:03<00:10, 37.23it/s]
Requesting API: 23%|██▎ | 116/500 [00:03<00:09, 40.31it/s]
Requesting API: 24%|██▍ | 121/500 [00:04<00:09, 40.27it/s]
Requesting API: 25%|██▌ | 126/500 [00:04<00:10, 36.38it/s]
Requesting API: 26%|██▌ | 130/500 [00:04<00:10, 35.72it/s]
Requesting API: 27%|██▋ | 135/500 [00:04<00:10, 33.60it/s]
Requesting API: 28%|██▊ | 139/500 [00:04<00:11, 32.34it/s]
Requesting API: 29%|██▊ | 143/500 [00:04<00:11, 30.23it/s]
Requesting API: 30%|███ | 150/500 [00:05<00:10, 33.29it/s]
Requesting API: 31%|███ | 154/500 [00:05<00:10, 34.51it/s]
Requesting API: 32%|███▏ | 158/500 [00:05<00:10, 33.69it/s]
Requesting API: 34%|███▍ | 169/500 [00:05<00:06, 48.07it/s]
Requesting API: 35%|███▍ | 174/500 [00:05<00:09, 36.21it/s]
Requesting API: 36%|███▌ | 179/500 [00:05<00:09, 35.66it/s]
Requesting API: 37%|███▋ | 183/500 [00:05<00:09, 33.87it/s]
Requesting API: 38%|███▊ | 189/500 [00:06<00:09, 34.02it/s]
Requesting API: 39%|███▊ | 193/500 [00:06<00:08, 34.22it/s]
Requesting API: 40%|███▉ | 199/500 [00:06<00:08, 36.20it/s]
Requesting API: 41%|████ | 203/500 [00:06<00:08, 36.22it/s]
Requesting API: 42%|████▏ | 208/500 [00:06<00:08, 36.35it/s]
Requesting API: 42%|████▏ | 212/500 [00:06<00:08, 34.55it/s]
Requesting API: 43%|████▎ | 217/500 [00:06<00:08, 32.70it/s]
Requesting API: 45%|████▌ | 225/500 [00:07<00:06, 42.86it/s]
Requesting API: 46%|████▌ | 231/500 [00:07<00:06, 44.44it/s]
Requesting API: 47%|████▋ | 236/500 [00:07<00:05, 44.17it/s]
Requesting API: 49%|████▊ | 243/500 [00:07<00:05, 44.09it/s]
Requesting API: 50%|████▉ | 248/500 [00:07<00:05, 44.69it/s]
Requesting API: 51%|█████ | 253/500 [00:07<00:05, 43.69it/s]
Requesting API: 52%|█████▏ | 258/500 [00:07<00:06, 38.29it/s]
Requesting API: 53%|█████▎ | 263/500 [00:07<00:05, 40.16it/s]
Requesting API: 54%|█████▍ | 269/500 [00:08<00:05, 41.35it/s]
Requesting API: 55%|█████▍ | 274/500 [00:08<00:05, 38.60it/s]
Requesting API: 56%|█████▌ | 280/500 [00:08<00:05, 42.64it/s]
Requesting API: 57%|█████▋ | 285/500 [00:08<00:04, 44.16it/s]
Requesting API: 58%|█████▊ | 290/500 [00:08<00:05, 40.74it/s]
Requesting API: 59%|█████▉ | 295/500 [00:08<00:05, 40.46it/s]
Requesting API: 60%|██████ | 301/500 [00:08<00:04, 43.30it/s]
Requesting API: 62%|██████▏ | 309/500 [00:08<00:03, 50.29it/s]
Requesting API: 63%|██████▎ | 315/500 [00:09<00:04, 46.12it/s]
Requesting API: 64%|██████▍ | 321/500 [00:09<00:04, 40.90it/s]
Requesting API: 65%|██████▌ | 326/500 [00:09<00:04, 40.70it/s]
Requesting API: 67%|██████▋ | 334/500 [00:09<00:04, 41.42it/s]
Requesting API: 68%|██████▊ | 341/500 [00:09<00:03, 46.79it/s]
Requesting API: 69%|██████▉ | 346/500 [00:09<00:03, 43.64it/s]
Requesting API: 71%|███████ | 353/500 [00:09<00:03, 46.38it/s]
Requesting API: 72%|███████▏ | 360/500 [00:10<00:02, 50.00it/s]
Requesting API: 74%|███████▍ | 369/500 [00:10<00:02, 54.63it/s]
Requesting API: 76%|███████▌ | 378/500 [00:10<00:01, 62.68it/s]
Requesting API: 77%|███████▋ | 385/500 [00:10<00:02, 48.71it/s]
Requesting API: 78%|███████▊ | 391/500 [00:10<00:02, 50.13it/s]
Requesting API: 80%|███████▉ | 399/500 [00:10<00:01, 54.98it/s]
Requesting API: 82%|████████▏ | 408/500 [00:10<00:01, 60.57it/s]
Requesting API: 83%|████████▎ | 416/500 [00:11<00:01, 60.48it/s]
Requesting API: 85%|████████▍ | 423/500 [00:11<00:01, 55.32it/s]
Requesting API: 86%|████████▌ | 430/500 [00:11<00:01, 53.28it/s]
Requesting API: 88%|████████▊ | 441/500 [00:11<00:00, 66.39it/s]
Requesting API: 90%|████████▉ | 449/500 [00:11<00:00, 66.25it/s]
Requesting API: 91%|█████████▏| 457/500 [00:11<00:00, 68.19it/s]
Requesting API: 93%|█████████▎| 465/500 [00:11<00:00, 65.58it/s]
Requesting API: 94%|█████████▍| 472/500 [00:11<00:00, 59.48it/s]
Requesting API: 97%|█████████▋| 483/500 [00:12<00:00, 67.30it/s]
Requesting API: 98%|█████████▊| 490/500 [00:12<00:00, 61.75it/s]
Requesting API: 99%|█████████▉| 497/500 [00:12<00:00, 35.28it/s]
Requesting API: 100%|██████████| 500/500 [00:14<00:00, 34.19it/s]
2026-08-08:22:14:13 INFO [loggers.evaluation_tracker:247] Saving results aggregated
2026-08-08:22:14:13 INFO [loggers.evaluation_tracker:119] Saving per-task samples to outputs/evals/release/math-keep50-step0150/student/*.jsonl
local-completions ({'model': 'student', 'base_url': 'http://127.0.0.1:8461/v1/completions', 'tokenizer': 'outputs/release/healed/release-offpolicy-math-keep50/step0150', 'num_concurrent': 48, 'tokenized_requests': False, 'max_retries': 3}), gen_kwargs: ({}), limit: None, num_fewshot: None, batch_size: 1
| Tasks |Version| Filter |n-shot| Metric | |Value| |Stderr|
|---------|------:|-----------|-----:|---------|---|----:|---|-----:|
|humaneval| 1|create_test| 0|pass@1 |↑ |0.189|± |0.0307|
|mbpp | 1|none | 3|pass_at_1|↑ |0.150|± |0.0160|
|