File size: 14,179 Bytes
a050468 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 | 2026-08-10:13:56:17 INFO [_cli.run:388] Selected Tasks: ['humaneval', 'mbpp']
2026-08-10:13:56:18 INFO [evaluator:214] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2026-08-10:13:56:18 INFO [evaluator:239] Initializing local-completions model, with arguments: {'model': 'student', 'base_url': 'http://127.0.0.1:8491/v1/completions', 'tokenizer': 'outputs/release/healed/release-offpolicy-math-keep50/step0100', 'num_concurrent': 48, 'tokenized_requests': False, 'max_retries': 3}
2026-08-10:13:56:18 INFO [models.openai_completions:42] Remote tokenizer not supported. Using huggingface tokenizer backend.
2026-08-10:13:56:18 INFO [models.api_models:179] Using max length 2048 - 1
2026-08-10:13:56:18 INFO [models.api_models:200] Using tokenizer huggingface
2026-08-10:13:56:25 INFO [evaluator_utils:446] Selected tasks:
2026-08-10:13:56:25 INFO [evaluator_utils:480] Task: humaneval (humaneval/humaneval.yaml)
2026-08-10:13:56:25 INFO [evaluator_utils:480] Task: mbpp (mbpp/mbpp.yaml)
2026-08-10:13:56:25 INFO [evaluator:314] humaneval: Using gen_kwargs: {'until': ['\nclass', '\ndef', '\n#', '\nif', '\nprint'], 'max_gen_toks': 1024, 'do_sample': False}
2026-08-10:13:56:25 INFO [evaluator:314] mbpp: Using gen_kwargs: {'until': ['[DONE]'], 'do_sample': False}
2026-08-10:13:56:25 INFO [api.task:312] Building contexts for humaneval on rank 0...
0%| | 0/164 [00:00<?, ?it/s]
88%|████████▊ | 145/164 [00:00<00:00, 1447.69it/s]
100%|██████████| 164/164 [00:00<00:00, 1475.66it/s]
2026-08-10:13:56:25 INFO [api.task:312] Building contexts for mbpp on rank 0...
0%| | 0/500 [00:00<?, ?it/s]
3%|▎ | 17/500 [00:00<00:02, 161.67it/s]
7%|▋ | 34/500 [00:00<00:02, 162.65it/s]
10%|█ | 51/500 [00:00<00:02, 163.28it/s]
14%|█▎ | 68/500 [00:00<00:02, 163.51it/s]
17%|█▋ | 85/500 [00:00<00:02, 163.69it/s]
20%|██ | 102/500 [00:00<00:02, 158.29it/s]
24%|██▎ | 118/500 [00:00<00:02, 155.61it/s]
27%|██▋ | 135/500 [00:00<00:02, 158.53it/s]
30%|███ | 152/500 [00:00<00:02, 160.59it/s]
34%|███▍ | 169/500 [00:01<00:02, 162.06it/s]
37%|███▋ | 186/500 [00:01<00:01, 163.10it/s]
41%|████ | 203/500 [00:01<00:01, 163.95it/s]
44%|████▍ | 220/500 [00:01<00:01, 164.33it/s]
47%|████▋ | 237/500 [00:01<00:01, 164.69it/s]
51%|█████ | 254/500 [00:01<00:01, 165.06it/s]
54%|█████▍ | 271/500 [00:01<00:01, 165.21it/s]
58%|█████▊ | 288/500 [00:01<00:01, 165.46it/s]
61%|██████ | 305/500 [00:01<00:01, 165.54it/s]
64%|██████▍ | 322/500 [00:01<00:01, 165.74it/s]
68%|██████▊ | 339/500 [00:02<00:00, 165.88it/s]
71%|███████ | 356/500 [00:02<00:00, 165.80it/s]
75%|███████▍ | 373/500 [00:02<00:00, 165.56it/s]
78%|███████▊ | 390/500 [00:02<00:00, 165.39it/s]
81%|████████▏ | 407/500 [00:02<00:00, 165.49it/s]
85%|████████▍ | 424/500 [00:02<00:00, 165.63it/s]
88%|████████▊ | 441/500 [00:02<00:00, 165.67it/s]
92%|█████████▏| 458/500 [00:02<00:00, 165.79it/s]
95%|█████████▌| 475/500 [00:02<00:00, 165.85it/s]
98%|█████████▊| 492/500 [00:02<00:00, 166.03it/s]
100%|██████████| 500/500 [00:03<00:00, 164.11it/s]
2026-08-10:13:56:28 INFO [evaluator:585] Running generate_until requests
2026-08-10:13:56:28 INFO [models.api_models:747] Tokenized requests are disabled. Context + generation length is not checked.
Requesting API: 0%| | 0/164 [00:00<?, ?it/s]
Requesting API: 1%| | 1/164 [00:00<02:16, 1.19it/s]
Requesting API: 1%| | 2/164 [00:00<01:10, 2.30it/s]
Requesting API: 2%|▏ | 3/164 [00:01<00:52, 3.06it/s]
Requesting API: 2%|▏ | 4/164 [00:01<00:42, 3.79it/s]
Requesting API: 5%|▌ | 9/164 [00:01<00:13, 11.77it/s]
Requesting API: 8%|▊ | 13/164 [00:01<00:09, 16.15it/s]
Requesting API: 10%|▉ | 16/164 [00:01<00:09, 16.23it/s]
Requesting API: 12%|█▏ | 20/164 [00:01<00:07, 18.32it/s]
Requesting API: 14%|█▍ | 23/164 [00:02<00:08, 17.35it/s]
Requesting API: 16%|█▋ | 27/164 [00:02<00:06, 20.52it/s]
Requesting API: 19%|█▉ | 31/164 [00:02<00:05, 24.50it/s]
Requesting API: 21%|██ | 34/164 [00:02<00:05, 25.54it/s]
Requesting API: 24%|██▍ | 39/164 [00:02<00:04, 29.95it/s]
Requesting API: 27%|██▋ | 44/164 [00:02<00:03, 30.43it/s]
Requesting API: 29%|██▉ | 48/164 [00:02<00:04, 28.83it/s]
Requesting API: 32%|███▏ | 52/164 [00:03<00:03, 30.41it/s]
Requesting API: 34%|███▍ | 56/164 [00:03<00:03, 29.10it/s]
Requesting API: 37%|███▋ | 61/164 [00:03<00:03, 32.67it/s]
Requesting API: 40%|███▉ | 65/164 [00:03<00:03, 29.31it/s]
Requesting API: 43%|████▎ | 70/164 [00:03<00:03, 30.32it/s]
Requesting API: 46%|████▋ | 76/164 [00:03<00:02, 35.80it/s]
Requesting API: 49%|████▉ | 80/164 [00:03<00:02, 32.43it/s]
Requesting API: 52%|█████▏ | 86/164 [00:04<00:02, 36.22it/s]
Requesting API: 56%|█████▌ | 92/164 [00:04<00:01, 41.02it/s]
Requesting API: 59%|█████▉ | 97/164 [00:04<00:01, 43.19it/s]
Requesting API: 62%|██████▏ | 102/164 [00:04<00:01, 43.19it/s]
Requesting API: 65%|██████▌ | 107/164 [00:04<00:01, 37.40it/s]
Requesting API: 71%|███████▏ | 117/164 [00:04<00:00, 49.70it/s]
Requesting API: 75%|███████▌ | 123/164 [00:04<00:00, 51.70it/s]
Requesting API: 80%|████████ | 132/164 [00:04<00:00, 57.01it/s]
Requesting API: 84%|████████▍ | 138/164 [00:05<00:00, 53.68it/s]
Requesting API: 88%|████████▊ | 144/164 [00:05<00:00, 44.07it/s]
Requesting API: 91%|█████████ | 149/164 [00:05<00:00, 34.40it/s]
Requesting API: 93%|█████████▎| 153/164 [00:05<00:00, 31.18it/s]
Requesting API: 96%|█████████▌| 157/164 [00:06<00:00, 12.79it/s]
Requesting API: 98%|█████████▊| 160/164 [00:08<00:00, 5.89it/s]
Requesting API: 99%|█████████▉| 162/164 [00:15<00:01, 1.35it/s]
Requesting API: 100%|██████████| 164/164 [00:18<00:00, 1.14it/s]
Requesting API: 100%|██████████| 164/164 [00:18<00:00, 8.90it/s]
Requesting API: 0%| | 0/500 [00:00<?, ?it/s]
Requesting API: 0%| | 1/500 [00:00<04:13, 1.96it/s]
Requesting API: 0%| | 2/500 [00:00<03:52, 2.14it/s]
Requesting API: 1%| | 4/500 [00:01<02:01, 4.09it/s]
Requesting API: 3%|▎ | 13/500 [00:01<00:28, 17.08it/s]
Requesting API: 4%|▎ | 18/500 [00:01<00:22, 21.08it/s]
Requesting API: 5%|▍ | 23/500 [00:01<00:18, 25.59it/s]
Requesting API: 6%|▌ | 28/500 [00:01<00:16, 28.74it/s]
Requesting API: 7%|▋ | 35/500 [00:01<00:12, 36.18it/s]
Requesting API: 8%|▊ | 42/500 [00:01<00:12, 37.97it/s]
Requesting API: 9%|▉ | 47/500 [00:02<00:11, 38.64it/s]
Requesting API: 11%|█ | 53/500 [00:02<00:11, 37.33it/s]
Requesting API: 12%|█▏ | 58/500 [00:02<00:12, 36.72it/s]
Requesting API: 13%|█▎ | 64/500 [00:02<00:11, 37.65it/s]
Requesting API: 15%|█▍ | 73/500 [00:02<00:09, 45.44it/s]
Requesting API: 16%|█▌ | 78/500 [00:02<00:10, 39.53it/s]
Requesting API: 17%|█▋ | 83/500 [00:03<00:10, 38.67it/s]
Requesting API: 18%|█▊ | 88/500 [00:03<00:10, 40.54it/s]
Requesting API: 19%|█▊ | 93/500 [00:03<00:09, 42.38it/s]
Requesting API: 20%|█▉ | 98/500 [00:03<00:09, 41.34it/s]
Requesting API: 21%|██ | 103/500 [00:03<00:09, 43.20it/s]
Requesting API: 22%|██▏ | 108/500 [00:03<00:10, 37.79it/s]
Requesting API: 22%|██▏ | 112/500 [00:03<00:10, 37.64it/s]
Requesting API: 23%|██▎ | 117/500 [00:03<00:09, 39.35it/s]
Requesting API: 24%|██▍ | 122/500 [00:04<00:10, 36.05it/s]
Requesting API: 25%|██▌ | 126/500 [00:04<00:10, 36.77it/s]
Requesting API: 26%|██▌ | 130/500 [00:04<00:11, 33.38it/s]
Requesting API: 27%|██▋ | 134/500 [00:04<00:10, 33.42it/s]
Requesting API: 28%|██▊ | 138/500 [00:04<00:11, 30.59it/s]
Requesting API: 28%|██▊ | 142/500 [00:04<00:11, 30.54it/s]
Requesting API: 29%|██▉ | 146/500 [00:04<00:11, 31.15it/s]
Requesting API: 30%|███ | 150/500 [00:04<00:10, 33.21it/s]
Requesting API: 31%|███▏ | 157/500 [00:05<00:08, 42.55it/s]
Requesting API: 33%|███▎ | 164/500 [00:05<00:06, 48.16it/s]
Requesting API: 34%|███▍ | 169/500 [00:05<00:07, 45.72it/s]
Requesting API: 35%|███▍ | 174/500 [00:05<00:08, 38.32it/s]
Requesting API: 36%|███▌ | 179/500 [00:05<00:09, 34.19it/s]
Requesting API: 37%|███▋ | 186/500 [00:05<00:07, 41.25it/s]
Requesting API: 38%|███▊ | 191/500 [00:05<00:08, 38.27it/s]
Requesting API: 39%|███▉ | 196/500 [00:06<00:08, 34.18it/s]
Requesting API: 40%|████ | 202/500 [00:06<00:07, 39.33it/s]
Requesting API: 41%|████▏ | 207/500 [00:06<00:08, 33.05it/s]
Requesting API: 43%|████▎ | 216/500 [00:06<00:07, 37.19it/s]
Requesting API: 45%|████▍ | 224/500 [00:06<00:06, 42.00it/s]
Requesting API: 46%|████▌ | 230/500 [00:06<00:06, 41.07it/s]
Requesting API: 47%|████▋ | 236/500 [00:07<00:06, 40.57it/s]
Requesting API: 49%|████▊ | 243/500 [00:07<00:05, 46.72it/s]
Requesting API: 50%|████▉ | 248/500 [00:07<00:05, 45.85it/s]
Requesting API: 51%|█████ | 253/500 [00:07<00:05, 44.50it/s]
Requesting API: 52%|█████▏ | 258/500 [00:07<00:06, 38.74it/s]
Requesting API: 53%|█████▎ | 267/500 [00:07<00:04, 49.02it/s]
Requesting API: 55%|█████▍ | 273/500 [00:07<00:06, 37.17it/s]
Requesting API: 56%|█████▌ | 279/500 [00:08<00:05, 39.15it/s]
Requesting API: 57%|█████▋ | 285/500 [00:08<00:05, 41.78it/s]
Requesting API: 58%|█████▊ | 290/500 [00:08<00:04, 42.25it/s]
Requesting API: 59%|█████▉ | 295/500 [00:08<00:04, 42.93it/s]
Requesting API: 60%|██████ | 300/500 [00:08<00:04, 43.61it/s]
Requesting API: 61%|██████ | 306/500 [00:08<00:04, 47.13it/s]
Requesting API: 63%|██████▎ | 313/500 [00:08<00:03, 50.98it/s]
Requesting API: 64%|██████▍ | 319/500 [00:08<00:03, 50.76it/s]
Requesting API: 65%|██████▌ | 325/500 [00:09<00:04, 40.52it/s]
Requesting API: 66%|██████▌ | 330/500 [00:09<00:05, 33.94it/s]
Requesting API: 67%|██████▋ | 336/500 [00:09<00:04, 38.21it/s]
Requesting API: 68%|██████▊ | 341/500 [00:09<00:04, 39.40it/s]
Requesting API: 69%|██████▉ | 346/500 [00:09<00:03, 40.85it/s]
Requesting API: 71%|███████ | 354/500 [00:09<00:03, 48.58it/s]
Requesting API: 72%|███████▏ | 362/500 [00:09<00:02, 48.98it/s]
Requesting API: 74%|███████▍ | 369/500 [00:10<00:02, 50.41it/s]
Requesting API: 76%|███████▌ | 379/500 [00:10<00:02, 54.69it/s]
Requesting API: 77%|███████▋ | 385/500 [00:10<00:02, 43.54it/s]
Requesting API: 79%|███████▊ | 393/500 [00:10<00:02, 50.73it/s]
Requesting API: 80%|████████ | 401/500 [00:10<00:01, 55.36it/s]
Requesting API: 81%|████████▏ | 407/500 [00:10<00:01, 54.74it/s]
Requesting API: 83%|████████▎ | 413/500 [00:10<00:01, 50.49it/s]
Requesting API: 84%|████████▍ | 422/500 [00:10<00:01, 58.09it/s]
Requesting API: 86%|████████▌ | 429/500 [00:11<00:01, 56.59it/s]
Requesting API: 87%|████████▋ | 435/500 [00:11<00:01, 51.23it/s]
Requesting API: 88%|████████▊ | 441/500 [00:11<00:01, 52.39it/s]
Requesting API: 90%|█████████ | 451/500 [00:11<00:00, 63.61it/s]
Requesting API: 92%|█████████▏| 458/500 [00:11<00:00, 59.41it/s]
Requesting API: 93%|█████████▎| 465/500 [00:11<00:00, 58.87it/s]
Requesting API: 94%|█████████▍| 472/500 [00:11<00:00, 59.08it/s]
Requesting API: 96%|█████████▌| 479/500 [00:11<00:00, 61.67it/s]
Requesting API: 98%|█████████▊| 488/500 [00:12<00:00, 66.82it/s]
Requesting API: 99%|█████████▉| 495/500 [00:12<00:00, 27.23it/s]
Requesting API: 100%|██████████| 500/500 [00:15<00:00, 33.31it/s]
2026-08-10:14:02:46 INFO [loggers.evaluation_tracker:247] Saving results aggregated
2026-08-10:14:02:46 INFO [loggers.evaluation_tracker:119] Saving per-task samples to outputs/evals/redo_policy/offpolicy_step0100/student/*.jsonl
local-completions ({'model': 'student', 'base_url': 'http://127.0.0.1:8491/v1/completions', 'tokenizer': 'outputs/release/healed/release-offpolicy-math-keep50/step0100', 'num_concurrent': 48, 'tokenized_requests': False, 'max_retries': 3}), gen_kwargs: ({}), limit: None, num_fewshot: None, batch_size: 1
| Tasks |Version| Filter |n-shot| Metric | |Value | |Stderr|
|---------|------:|-----------|-----:|---------|---|-----:|---|-----:|
|humaneval| 1|create_test| 0|pass@1 |↑ |0.1951|± |0.0310|
|mbpp | 1|none | 3|pass_at_1|↑ |0.1420|± |0.0156|
|