File size: 14,078 Bytes
a050468 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 | 2026-08-10:13:56:22 INFO [_cli.run:388] Selected Tasks: ['humaneval', 'mbpp']
2026-08-10:13:56:23 INFO [evaluator:214] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2026-08-10:13:56:23 INFO [evaluator:239] Initializing local-completions model, with arguments: {'model': 'student', 'base_url': 'http://127.0.0.1:8492/v1/completions', 'tokenizer': 'outputs/release/healed/release-offpolicy-math-keep50/step0150', 'num_concurrent': 48, 'tokenized_requests': False, 'max_retries': 3}
2026-08-10:13:56:23 INFO [models.openai_completions:42] Remote tokenizer not supported. Using huggingface tokenizer backend.
2026-08-10:13:56:23 INFO [models.api_models:179] Using max length 2048 - 1
2026-08-10:13:56:23 INFO [models.api_models:200] Using tokenizer huggingface
2026-08-10:13:56:30 INFO [evaluator_utils:446] Selected tasks:
2026-08-10:13:56:30 INFO [evaluator_utils:480] Task: humaneval (humaneval/humaneval.yaml)
2026-08-10:13:56:30 INFO [evaluator_utils:480] Task: mbpp (mbpp/mbpp.yaml)
2026-08-10:13:56:30 INFO [evaluator:314] humaneval: Using gen_kwargs: {'until': ['\nclass', '\ndef', '\n#', '\nif', '\nprint'], 'max_gen_toks': 1024, 'do_sample': False}
2026-08-10:13:56:30 INFO [evaluator:314] mbpp: Using gen_kwargs: {'until': ['[DONE]'], 'do_sample': False}
2026-08-10:13:56:30 INFO [api.task:312] Building contexts for humaneval on rank 0...
0%| | 0/164 [00:00<?, ?it/s]
86%|████████▌ | 141/164 [00:00<00:00, 1409.34it/s]
100%|██████████| 164/164 [00:00<00:00, 1449.99it/s]
2026-08-10:13:56:30 INFO [api.task:312] Building contexts for mbpp on rank 0...
0%| | 0/500 [00:00<?, ?it/s]
3%|▎ | 17/500 [00:00<00:02, 161.84it/s]
7%|▋ | 34/500 [00:00<00:02, 162.55it/s]
10%|█ | 51/500 [00:00<00:02, 163.07it/s]
14%|█▎ | 68/500 [00:00<00:02, 162.81it/s]
17%|█▋ | 85/500 [00:00<00:02, 162.55it/s]
20%|██ | 102/500 [00:00<00:02, 162.89it/s]
24%|██▍ | 119/500 [00:00<00:02, 163.46it/s]
27%|██▋ | 136/500 [00:00<00:02, 163.87it/s]
31%|███ | 153/500 [00:00<00:02, 164.08it/s]
34%|███▍ | 170/500 [00:01<00:02, 164.23it/s]
37%|███▋ | 187/500 [00:01<00:01, 164.55it/s]
41%|████ | 204/500 [00:01<00:01, 164.80it/s]
44%|████▍ | 221/500 [00:01<00:01, 164.98it/s]
48%|████▊ | 238/500 [00:01<00:01, 164.92it/s]
51%|█████ | 255/500 [00:01<00:01, 164.63it/s]
54%|█████▍ | 272/500 [00:01<00:01, 164.47it/s]
58%|█████▊ | 289/500 [00:01<00:01, 164.57it/s]
61%|██████ | 306/500 [00:01<00:01, 164.86it/s]
65%|██████▍ | 323/500 [00:01<00:01, 165.04it/s]
68%|██████▊ | 340/500 [00:02<00:00, 165.17it/s]
71%|███████▏ | 357/500 [00:02<00:00, 165.36it/s]
75%|███████▍ | 374/500 [00:02<00:00, 165.47it/s]
78%|███████▊ | 391/500 [00:02<00:00, 165.47it/s]
82%|████████▏ | 408/500 [00:02<00:00, 165.50it/s]
85%|████████▌ | 425/500 [00:02<00:00, 165.34it/s]
88%|████████▊ | 442/500 [00:02<00:00, 165.53it/s]
92%|█████████▏| 459/500 [00:02<00:00, 165.51it/s]
95%|█████████▌| 476/500 [00:02<00:00, 165.50it/s]
99%|█████████▊| 493/500 [00:02<00:00, 165.60it/s]
100%|██████████| 500/500 [00:03<00:00, 164.68it/s]
2026-08-10:13:56:33 INFO [evaluator:585] Running generate_until requests
2026-08-10:13:56:33 INFO [models.api_models:747] Tokenized requests are disabled. Context + generation length is not checked.
Requesting API: 0%| | 0/164 [00:00<?, ?it/s]
Requesting API: 1%| | 1/164 [00:00<02:22, 1.15it/s]
Requesting API: 2%|▏ | 3/164 [00:01<00:45, 3.57it/s]
Requesting API: 2%|▏ | 4/164 [00:01<00:39, 4.04it/s]
Requesting API: 3%|▎ | 5/164 [00:01<00:34, 4.65it/s]
Requesting API: 6%|▌ | 10/164 [00:01<00:12, 12.59it/s]
Requesting API: 8%|▊ | 13/164 [00:01<00:09, 16.09it/s]
Requesting API: 10%|▉ | 16/164 [00:01<00:10, 14.19it/s]
Requesting API: 11%|█ | 18/164 [00:01<00:10, 13.50it/s]
Requesting API: 13%|█▎ | 22/164 [00:02<00:08, 16.27it/s]
Requesting API: 16%|█▌ | 26/164 [00:02<00:06, 20.12it/s]
Requesting API: 18%|█▊ | 29/164 [00:02<00:06, 21.52it/s]
Requesting API: 20%|█▉ | 32/164 [00:02<00:05, 23.21it/s]
Requesting API: 21%|██▏ | 35/164 [00:02<00:05, 21.88it/s]
Requesting API: 25%|██▌ | 41/164 [00:02<00:04, 29.95it/s]
Requesting API: 27%|██▋ | 45/164 [00:02<00:04, 25.31it/s]
Requesting API: 31%|███ | 51/164 [00:03<00:03, 29.55it/s]
Requesting API: 34%|███▎ | 55/164 [00:03<00:03, 30.79it/s]
Requesting API: 36%|███▌ | 59/164 [00:03<00:04, 23.94it/s]
Requesting API: 38%|███▊ | 63/164 [00:03<00:03, 26.50it/s]
Requesting API: 42%|████▏ | 69/164 [00:03<00:02, 32.49it/s]
Requesting API: 46%|████▋ | 76/164 [00:03<00:02, 38.04it/s]
Requesting API: 50%|█████ | 82/164 [00:03<00:01, 42.62it/s]
Requesting API: 53%|█████▎ | 87/164 [00:04<00:02, 36.60it/s]
Requesting API: 56%|█████▌ | 92/164 [00:04<00:02, 32.88it/s]
Requesting API: 59%|█████▉ | 97/164 [00:04<00:01, 36.26it/s]
Requesting API: 63%|██████▎ | 104/164 [00:04<00:01, 43.20it/s]
Requesting API: 68%|██████▊ | 111/164 [00:04<00:01, 49.10it/s]
Requesting API: 71%|███████▏ | 117/164 [00:04<00:00, 47.72it/s]
Requesting API: 76%|███████▌ | 125/164 [00:04<00:00, 53.76it/s]
Requesting API: 81%|████████ | 133/164 [00:05<00:00, 58.61it/s]
Requesting API: 85%|████████▌ | 140/164 [00:05<00:00, 53.95it/s]
Requesting API: 89%|████████▉ | 146/164 [00:05<00:00, 46.37it/s]
Requesting API: 92%|█████████▏| 151/164 [00:05<00:00, 39.04it/s]
Requesting API: 95%|█████████▌| 156/164 [00:06<00:00, 20.58it/s]
Requesting API: 98%|█████████▊| 160/164 [00:06<00:00, 13.15it/s]
Requesting API: 99%|█████████▉| 163/164 [00:15<00:00, 1.66it/s]
Requesting API: 100%|██████████| 164/164 [00:18<00:00, 9.08it/s]
Requesting API: 0%| | 0/500 [00:00<?, ?it/s]
Requesting API: 0%| | 1/500 [00:00<08:15, 1.01it/s]
Requesting API: 1%| | 3/500 [00:01<02:27, 3.38it/s]
Requesting API: 1%| | 5/500 [00:01<01:32, 5.33it/s]
Requesting API: 2%|▏ | 12/500 [00:01<00:31, 15.29it/s]
Requesting API: 4%|▍ | 20/500 [00:01<00:19, 24.86it/s]
Requesting API: 5%|▌ | 26/500 [00:01<00:15, 30.88it/s]
Requesting API: 6%|▌ | 31/500 [00:01<00:15, 29.91it/s]
Requesting API: 7%|▋ | 35/500 [00:01<00:15, 29.34it/s]
Requesting API: 8%|▊ | 39/500 [00:02<00:15, 30.52it/s]
Requesting API: 9%|▉ | 47/500 [00:02<00:11, 39.49it/s]
Requesting API: 10%|█ | 52/500 [00:02<00:13, 34.15it/s]
Requesting API: 12%|█▏ | 59/500 [00:02<00:11, 38.26it/s]
Requesting API: 13%|█▎ | 64/500 [00:02<00:12, 35.36it/s]
Requesting API: 14%|█▍ | 69/500 [00:02<00:11, 38.39it/s]
Requesting API: 15%|█▌ | 75/500 [00:03<00:12, 33.00it/s]
Requesting API: 16%|█▌ | 80/500 [00:03<00:12, 34.62it/s]
Requesting API: 17%|█▋ | 86/500 [00:03<00:10, 38.59it/s]
Requesting API: 18%|█▊ | 91/500 [00:03<00:10, 38.12it/s]
Requesting API: 19%|█▉ | 97/500 [00:03<00:10, 38.78it/s]
Requesting API: 21%|██ | 104/500 [00:03<00:10, 38.52it/s]
Requesting API: 22%|██▏ | 108/500 [00:03<00:10, 36.45it/s]
Requesting API: 23%|██▎ | 115/500 [00:04<00:09, 42.63it/s]
Requesting API: 24%|██▍ | 120/500 [00:04<00:09, 39.27it/s]
Requesting API: 25%|██▌ | 125/500 [00:04<00:11, 33.77it/s]
Requesting API: 26%|██▌ | 129/500 [00:04<00:10, 34.29it/s]
Requesting API: 27%|██▋ | 134/500 [00:04<00:13, 27.55it/s]
Requesting API: 28%|██▊ | 140/500 [00:04<00:10, 32.97it/s]
Requesting API: 29%|██▉ | 145/500 [00:05<00:10, 34.46it/s]
Requesting API: 30%|███ | 150/500 [00:05<00:10, 34.48it/s]
Requesting API: 31%|███ | 156/500 [00:05<00:09, 37.36it/s]
Requesting API: 32%|███▏ | 160/500 [00:05<00:09, 34.85it/s]
Requesting API: 34%|███▎ | 168/500 [00:05<00:08, 39.40it/s]
Requesting API: 34%|███▍ | 172/500 [00:05<00:09, 34.85it/s]
Requesting API: 35%|███▌ | 176/500 [00:05<00:09, 32.62it/s]
Requesting API: 36%|███▌ | 180/500 [00:06<00:10, 31.20it/s]
Requesting API: 37%|███▋ | 187/500 [00:06<00:08, 36.27it/s]
Requesting API: 38%|███▊ | 191/500 [00:06<00:08, 36.50it/s]
Requesting API: 39%|███▉ | 195/500 [00:06<00:08, 34.75it/s]
Requesting API: 40%|███▉ | 199/500 [00:06<00:08, 35.53it/s]
Requesting API: 41%|████ | 204/500 [00:06<00:07, 38.26it/s]
Requesting API: 42%|████▏ | 208/500 [00:06<00:09, 32.24it/s]
Requesting API: 43%|████▎ | 213/500 [00:06<00:08, 34.67it/s]
Requesting API: 44%|████▎ | 218/500 [00:07<00:08, 34.81it/s]
Requesting API: 45%|████▍ | 223/500 [00:07<00:08, 33.98it/s]
Requesting API: 46%|████▌ | 229/500 [00:07<00:07, 35.44it/s]
Requesting API: 47%|████▋ | 235/500 [00:07<00:06, 39.46it/s]
Requesting API: 48%|████▊ | 240/500 [00:07<00:06, 37.29it/s]
Requesting API: 49%|████▉ | 247/500 [00:07<00:06, 42.05it/s]
Requesting API: 50%|█████ | 252/500 [00:07<00:06, 37.96it/s]
Requesting API: 51%|█████▏ | 257/500 [00:08<00:06, 39.86it/s]
Requesting API: 52%|█████▏ | 262/500 [00:08<00:06, 37.53it/s]
Requesting API: 53%|█████▎ | 267/500 [00:08<00:06, 38.13it/s]
Requesting API: 55%|█████▍ | 274/500 [00:08<00:05, 43.19it/s]
Requesting API: 56%|█████▌ | 279/500 [00:08<00:05, 38.20it/s]
Requesting API: 57%|█████▋ | 285/500 [00:08<00:05, 42.11it/s]
Requesting API: 58%|█████▊ | 290/500 [00:08<00:05, 35.92it/s]
Requesting API: 59%|█████▉ | 294/500 [00:09<00:05, 34.65it/s]
Requesting API: 60%|██████ | 300/500 [00:09<00:04, 40.29it/s]
Requesting API: 61%|██████ | 306/500 [00:09<00:04, 45.04it/s]
Requesting API: 62%|██████▏ | 311/500 [00:09<00:04, 42.68it/s]
Requesting API: 63%|██████▎ | 317/500 [00:09<00:04, 44.39it/s]
Requesting API: 65%|██████▍ | 323/500 [00:09<00:03, 46.27it/s]
Requesting API: 66%|██████▌ | 328/500 [00:09<00:04, 36.91it/s]
Requesting API: 67%|██████▋ | 333/500 [00:09<00:04, 36.87it/s]
Requesting API: 68%|██████▊ | 340/500 [00:10<00:03, 43.09it/s]
Requesting API: 69%|██████▉ | 345/500 [00:10<00:03, 39.84it/s]
Requesting API: 70%|███████ | 351/500 [00:10<00:03, 42.56it/s]
Requesting API: 72%|███████▏ | 359/500 [00:10<00:03, 45.42it/s]
Requesting API: 74%|███████▎ | 368/500 [00:10<00:02, 52.94it/s]
Requesting API: 76%|███████▌ | 378/500 [00:10<00:02, 58.90it/s]
Requesting API: 77%|███████▋ | 384/500 [00:11<00:02, 42.61it/s]
Requesting API: 78%|███████▊ | 391/500 [00:11<00:02, 46.94it/s]
Requesting API: 81%|████████ | 403/500 [00:11<00:01, 61.32it/s]
Requesting API: 82%|████████▏ | 410/500 [00:11<00:01, 63.27it/s]
Requesting API: 83%|████████▎ | 417/500 [00:11<00:01, 50.19it/s]
Requesting API: 85%|████████▌ | 427/500 [00:11<00:01, 56.67it/s]
Requesting API: 87%|████████▋ | 434/500 [00:11<00:01, 58.69it/s]
Requesting API: 88%|████████▊ | 441/500 [00:12<00:01, 52.61it/s]
Requesting API: 90%|█████████ | 451/500 [00:12<00:00, 63.32it/s]
Requesting API: 92%|█████████▏| 458/500 [00:12<00:00, 58.90it/s]
Requesting API: 93%|█████████▎| 465/500 [00:12<00:00, 53.06it/s]
Requesting API: 95%|█████████▌| 476/500 [00:12<00:00, 65.15it/s]
Requesting API: 97%|█████████▋| 484/500 [00:12<00:00, 61.45it/s]
Requesting API: 98%|█████████▊| 491/500 [00:12<00:00, 51.11it/s]
Requesting API: 99%|█████████▉| 497/500 [00:13<00:00, 16.97it/s]
Requesting API: 100%|██████████| 500/500 [00:15<00:00, 32.78it/s]
2026-08-10:14:03:06 INFO [loggers.evaluation_tracker:247] Saving results aggregated
2026-08-10:14:03:06 INFO [loggers.evaluation_tracker:119] Saving per-task samples to outputs/evals/redo_policy/offpolicy_step0150/student/*.jsonl
local-completions ({'model': 'student', 'base_url': 'http://127.0.0.1:8492/v1/completions', 'tokenizer': 'outputs/release/healed/release-offpolicy-math-keep50/step0150', 'num_concurrent': 48, 'tokenized_requests': False, 'max_retries': 3}), gen_kwargs: ({}), limit: None, num_fewshot: None, batch_size: 1
| Tasks |Version| Filter |n-shot| Metric | |Value| |Stderr|
|---------|------:|-----------|-----:|---------|---|----:|---|-----:|
|humaneval| 1|create_test| 0|pass@1 |↑ |0.189|± |0.0307|
|mbpp | 1|none | 3|pass_at_1|↑ |0.154|± |0.0162|
|