File size: 14,770 Bytes
a82fb9e | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 | 2026-08-08:22:21:07 INFO [_cli.run:388] Selected Tasks: ['humaneval', 'mbpp']
2026-08-08:22:21:08 INFO [evaluator:214] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2026-08-08:22:21:08 INFO [evaluator:239] Initializing local-completions model, with arguments: {'model': 'student', 'base_url': 'http://127.0.0.1:8461/v1/completions', 'tokenizer': 'outputs/release/healed/release-offpolicy-general-keep75/step0150', 'num_concurrent': 48, 'tokenized_requests': False, 'max_retries': 3}
2026-08-08:22:21:08 INFO [models.openai_completions:42] Remote tokenizer not supported. Using huggingface tokenizer backend.
2026-08-08:22:21:08 INFO [models.api_models:179] Using max length 2048 - 1
2026-08-08:22:21:08 INFO [models.api_models:200] Using tokenizer huggingface
2026-08-08:22:21:16 INFO [evaluator_utils:446] Selected tasks:
2026-08-08:22:21:16 INFO [evaluator_utils:480] Task: humaneval (humaneval/humaneval.yaml)
2026-08-08:22:21:16 INFO [evaluator_utils:480] Task: mbpp (mbpp/mbpp.yaml)
2026-08-08:22:21:16 INFO [evaluator:314] humaneval: Using gen_kwargs: {'until': ['\nclass', '\ndef', '\n#', '\nif', '\nprint'], 'max_gen_toks': 1024, 'do_sample': False}
2026-08-08:22:21:16 INFO [evaluator:314] mbpp: Using gen_kwargs: {'until': ['[DONE]'], 'do_sample': False}
2026-08-08:22:21:16 INFO [api.task:312] Building contexts for humaneval on rank 0...
0%| | 0/164 [00:00<?, ?it/s]
98%|█████████▊| 161/164 [00:00<00:00, 1600.42it/s]
100%|██████████| 164/164 [00:00<00:00, 1601.33it/s]
2026-08-08:22:21:16 INFO [api.task:312] Building contexts for mbpp on rank 0...
0%| | 0/500 [00:00<?, ?it/s]
3%|▎ | 17/500 [00:00<00:02, 161.73it/s]
7%|▋ | 34/500 [00:00<00:02, 162.63it/s]
10%|█ | 51/500 [00:00<00:02, 163.29it/s]
14%|█▎ | 68/500 [00:00<00:02, 163.58it/s]
17%|█▋ | 85/500 [00:00<00:02, 163.88it/s]
20%|██ | 102/500 [00:00<00:02, 164.32it/s]
24%|██▍ | 119/500 [00:00<00:02, 164.57it/s]
27%|██▋ | 136/500 [00:00<00:02, 164.72it/s]
31%|███ | 153/500 [00:00<00:02, 164.85it/s]
34%|███▍ | 170/500 [00:01<00:02, 164.77it/s]
37%|███▋ | 187/500 [00:01<00:01, 165.19it/s]
41%|████ | 204/500 [00:01<00:01, 165.59it/s]
44%|████▍ | 221/500 [00:01<00:01, 165.71it/s]
48%|████▊ | 238/500 [00:01<00:01, 164.79it/s]
51%|█████ | 255/500 [00:01<00:01, 165.24it/s]
54%|█████▍ | 272/500 [00:01<00:01, 165.65it/s]
58%|█████▊ | 289/500 [00:01<00:01, 165.83it/s]
61%|██████ | 306/500 [00:01<00:01, 165.96it/s]
65%|██████▍ | 323/500 [00:01<00:01, 166.20it/s]
68%|██████▊ | 340/500 [00:02<00:00, 166.28it/s]
71%|███████▏ | 357/500 [00:02<00:00, 166.30it/s]
75%|███████▍ | 374/500 [00:02<00:00, 166.37it/s]
78%|███████▊ | 391/500 [00:02<00:00, 166.29it/s]
82%|████████▏ | 408/500 [00:02<00:00, 166.31it/s]
85%|████████▌ | 425/500 [00:02<00:00, 166.27it/s]
88%|████████▊ | 442/500 [00:02<00:00, 166.52it/s]
92%|█████████▏| 459/500 [00:02<00:00, 166.63it/s]
95%|█████████▌| 476/500 [00:02<00:00, 166.53it/s]
99%|█████████▊| 493/500 [00:02<00:00, 166.63it/s]
100%|██████████| 500/500 [00:03<00:00, 165.56it/s]
2026-08-08:22:21:19 INFO [evaluator:585] Running generate_until requests
2026-08-08:22:21:19 INFO [models.api_models:747] Tokenized requests are disabled. Context + generation length is not checked.
Requesting API: 0%| | 0/164 [00:00<?, ?it/s]
Requesting API: 1%| | 1/164 [00:00<02:37, 1.04it/s]
Requesting API: 4%|▎ | 6/164 [00:01<00:23, 6.64it/s]
Requesting API: 5%|▍ | 8/164 [00:01<00:21, 7.23it/s]
Requesting API: 6%|▌ | 10/164 [00:01<00:17, 8.61it/s]
Requesting API: 7%|▋ | 12/164 [00:01<00:20, 7.43it/s]
Requesting API: 10%|█ | 17/164 [00:01<00:10, 13.48it/s]
Requesting API: 12%|█▏ | 20/164 [00:02<00:10, 14.03it/s]
Requesting API: 15%|█▌ | 25/164 [00:02<00:06, 20.14it/s]
Requesting API: 17%|█▋ | 28/164 [00:02<00:06, 21.90it/s]
Requesting API: 19%|█▉ | 31/164 [00:02<00:05, 23.43it/s]
Requesting API: 21%|██ | 34/164 [00:02<00:06, 21.66it/s]
Requesting API: 23%|██▎ | 37/164 [00:02<00:05, 23.50it/s]
Requesting API: 24%|██▍ | 40/164 [00:02<00:05, 24.05it/s]
Requesting API: 28%|██▊ | 46/164 [00:02<00:03, 32.32it/s]
Requesting API: 30%|███ | 50/164 [00:03<00:04, 28.46it/s]
Requesting API: 34%|███▍ | 56/164 [00:03<00:03, 33.62it/s]
Requesting API: 37%|███▋ | 60/164 [00:03<00:03, 28.40it/s]
Requesting API: 39%|███▉ | 64/164 [00:03<00:03, 28.10it/s]
Requesting API: 41%|████▏ | 68/164 [00:03<00:03, 28.42it/s]
Requesting API: 45%|████▌ | 74/164 [00:03<00:02, 34.14it/s]
Requesting API: 49%|████▉ | 81/164 [00:03<00:01, 41.84it/s]
Requesting API: 52%|█████▏ | 86/164 [00:04<00:01, 41.80it/s]
Requesting API: 55%|█████▌ | 91/164 [00:04<00:01, 39.36it/s]
Requesting API: 59%|█████▉ | 97/164 [00:04<00:01, 39.88it/s]
Requesting API: 62%|██████▏ | 102/164 [00:04<00:01, 41.64it/s]
Requesting API: 66%|██████▌ | 108/164 [00:04<00:01, 44.82it/s]
Requesting API: 70%|███████ | 115/164 [00:04<00:00, 49.27it/s]
Requesting API: 75%|███████▌ | 123/164 [00:04<00:00, 55.13it/s]
Requesting API: 82%|████████▏ | 134/164 [00:04<00:00, 67.24it/s]
Requesting API: 86%|████████▌ | 141/164 [00:05<00:00, 54.04it/s]
Requesting API: 90%|█████████ | 148/164 [00:05<00:00, 56.30it/s]
Requesting API: 94%|█████████▍| 154/164 [00:05<00:00, 48.05it/s]
Requesting API: 98%|█████████▊| 160/164 [00:05<00:00, 26.79it/s]
Requesting API: 100%|██████████| 164/164 [00:15<00:00, 10.45it/s]
Requesting API: 0%| | 0/500 [00:00<?, ?it/s]
Requesting API: 0%| | 1/500 [00:01<09:58, 1.20s/it]
Requesting API: 1%| | 4/500 [00:01<02:07, 3.89it/s]
Requesting API: 2%|▏ | 9/500 [00:01<00:50, 9.76it/s]
Requesting API: 3%|▎ | 16/500 [00:01<00:29, 16.59it/s]
Requesting API: 4%|▍ | 20/500 [00:01<00:23, 20.04it/s]
Requesting API: 5%|▍ | 24/500 [00:01<00:22, 21.42it/s]
Requesting API: 6%|▌ | 28/500 [00:02<00:20, 23.58it/s]
Requesting API: 6%|▋ | 32/500 [00:02<00:17, 26.94it/s]
Requesting API: 7%|▋ | 36/500 [00:02<00:16, 28.72it/s]
Requesting API: 8%|▊ | 40/500 [00:02<00:15, 30.34it/s]
Requesting API: 9%|▉ | 46/500 [00:02<00:12, 36.71it/s]
Requesting API: 10%|█ | 51/500 [00:02<00:16, 28.01it/s]
Requesting API: 11%|█ | 55/500 [00:02<00:15, 29.31it/s]
Requesting API: 12%|█▏ | 59/500 [00:02<00:14, 29.81it/s]
Requesting API: 13%|█▎ | 64/500 [00:03<00:14, 29.32it/s]
Requesting API: 14%|█▎ | 68/500 [00:03<00:15, 27.66it/s]
Requesting API: 14%|█▍ | 71/500 [00:03<00:15, 27.55it/s]
Requesting API: 15%|█▌ | 77/500 [00:03<00:13, 31.84it/s]
Requesting API: 17%|█▋ | 83/500 [00:03<00:10, 37.94it/s]
Requesting API: 18%|█▊ | 88/500 [00:03<00:12, 34.17it/s]
Requesting API: 18%|█▊ | 92/500 [00:04<00:15, 25.98it/s]
Requesting API: 20%|█▉ | 98/500 [00:04<00:13, 30.81it/s]
Requesting API: 21%|██ | 103/500 [00:04<00:11, 33.98it/s]
Requesting API: 22%|██▏ | 108/500 [00:04<00:10, 37.33it/s]
Requesting API: 23%|██▎ | 114/500 [00:04<00:09, 38.68it/s]
Requesting API: 24%|██▍ | 119/500 [00:04<00:14, 26.61it/s]
Requesting API: 25%|██▍ | 123/500 [00:05<00:13, 27.22it/s]
Requesting API: 25%|██▌ | 127/500 [00:05<00:12, 29.05it/s]
Requesting API: 26%|██▋ | 132/500 [00:05<00:11, 31.36it/s]
Requesting API: 27%|██▋ | 136/500 [00:05<00:12, 28.93it/s]
Requesting API: 28%|██▊ | 141/500 [00:05<00:12, 27.98it/s]
Requesting API: 29%|██▉ | 144/500 [00:05<00:13, 26.48it/s]
Requesting API: 29%|██▉ | 147/500 [00:06<00:15, 22.13it/s]
Requesting API: 30%|███ | 152/500 [00:06<00:12, 27.59it/s]
Requesting API: 31%|███▏ | 157/500 [00:06<00:10, 31.68it/s]
Requesting API: 32%|███▏ | 161/500 [00:06<00:11, 29.07it/s]
Requesting API: 33%|███▎ | 165/500 [00:06<00:10, 31.42it/s]
Requesting API: 34%|███▍ | 169/500 [00:06<00:11, 27.66it/s]
Requesting API: 35%|███▍ | 173/500 [00:06<00:11, 28.95it/s]
Requesting API: 35%|███▌ | 177/500 [00:06<00:11, 28.86it/s]
Requesting API: 36%|███▌ | 181/500 [00:07<00:10, 30.54it/s]
Requesting API: 37%|███▋ | 185/500 [00:07<00:10, 30.26it/s]
Requesting API: 38%|███▊ | 191/500 [00:07<00:10, 30.80it/s]
Requesting API: 39%|███▉ | 196/500 [00:07<00:10, 28.92it/s]
Requesting API: 40%|████ | 201/500 [00:07<00:10, 29.85it/s]
Requesting API: 41%|████ | 205/500 [00:07<00:11, 26.24it/s]
Requesting API: 42%|████▏ | 211/500 [00:08<00:08, 32.14it/s]
Requesting API: 43%|████▎ | 215/500 [00:08<00:08, 33.83it/s]
Requesting API: 44%|████▍ | 220/500 [00:08<00:07, 37.39it/s]
Requesting API: 45%|████▌ | 225/500 [00:08<00:08, 32.66it/s]
Requesting API: 46%|████▌ | 229/500 [00:08<00:08, 30.43it/s]
Requesting API: 47%|████▋ | 235/500 [00:08<00:08, 32.52it/s]
Requesting API: 48%|████▊ | 239/500 [00:08<00:08, 29.49it/s]
Requesting API: 49%|████▊ | 243/500 [00:09<00:08, 31.55it/s]
Requesting API: 50%|████▉ | 248/500 [00:09<00:07, 32.98it/s]
Requesting API: 50%|█████ | 252/500 [00:09<00:07, 33.00it/s]
Requesting API: 52%|█████▏ | 258/500 [00:09<00:06, 36.31it/s]
Requesting API: 52%|█████▏ | 262/500 [00:09<00:07, 29.84it/s]
Requesting API: 54%|█████▎ | 268/500 [00:09<00:06, 35.37it/s]
Requesting API: 55%|█████▍ | 273/500 [00:09<00:06, 35.95it/s]
Requesting API: 56%|█████▌ | 278/500 [00:10<00:05, 37.44it/s]
Requesting API: 56%|█████▋ | 282/500 [00:10<00:07, 28.54it/s]
Requesting API: 57%|█████▋ | 286/500 [00:10<00:08, 24.33it/s]
Requesting API: 59%|█████▉ | 295/500 [00:10<00:06, 34.04it/s]
Requesting API: 61%|██████ | 303/500 [00:10<00:04, 42.08it/s]
Requesting API: 62%|██████▏ | 308/500 [00:10<00:05, 38.39it/s]
Requesting API: 63%|██████▎ | 313/500 [00:11<00:04, 38.03it/s]
Requesting API: 64%|██████▍ | 319/500 [00:11<00:04, 39.39it/s]
Requesting API: 65%|██████▌ | 325/500 [00:11<00:04, 42.90it/s]
Requesting API: 66%|██████▌ | 330/500 [00:11<00:03, 43.46it/s]
Requesting API: 67%|██████▋ | 335/500 [00:11<00:04, 34.12it/s]
Requesting API: 68%|██████▊ | 339/500 [00:11<00:05, 31.38it/s]
Requesting API: 69%|██████▊ | 343/500 [00:11<00:04, 32.58it/s]
Requesting API: 69%|██████▉ | 347/500 [00:12<00:04, 32.00it/s]
Requesting API: 71%|███████ | 355/500 [00:12<00:03, 41.09it/s]
Requesting API: 72%|███████▏ | 360/500 [00:12<00:03, 36.58it/s]
Requesting API: 74%|███████▎ | 368/500 [00:12<00:03, 43.16it/s]
Requesting API: 75%|███████▍ | 373/500 [00:12<00:03, 37.17it/s]
Requesting API: 76%|███████▌ | 379/500 [00:12<00:03, 39.39it/s]
Requesting API: 77%|███████▋ | 384/500 [00:12<00:02, 41.37it/s]
Requesting API: 78%|███████▊ | 389/500 [00:13<00:02, 38.00it/s]
Requesting API: 79%|███████▉ | 394/500 [00:13<00:02, 40.09it/s]
Requesting API: 80%|████████ | 401/500 [00:13<00:02, 46.42it/s]
Requesting API: 82%|████████▏ | 409/500 [00:13<00:01, 52.57it/s]
Requesting API: 83%|████████▎ | 415/500 [00:13<00:01, 50.44it/s]
Requesting API: 84%|████████▍ | 421/500 [00:13<00:01, 44.49it/s]
Requesting API: 85%|████████▌ | 426/500 [00:13<00:01, 38.11it/s]
Requesting API: 86%|████████▋ | 432/500 [00:13<00:01, 42.83it/s]
Requesting API: 88%|████████▊ | 438/500 [00:14<00:01, 45.82it/s]
Requesting API: 90%|████████▉ | 448/500 [00:14<00:00, 52.91it/s]
Requesting API: 91%|█████████ | 454/500 [00:14<00:00, 53.38it/s]
Requesting API: 93%|█████████▎| 464/500 [00:14<00:00, 64.54it/s]
Requesting API: 94%|█████████▍| 471/500 [00:14<00:00, 61.87it/s]
Requesting API: 96%|█████████▌| 478/500 [00:14<00:00, 61.50it/s]
Requesting API: 97%|█████████▋| 485/500 [00:14<00:00, 58.01it/s]
Requesting API: 98%|█████████▊| 491/500 [00:14<00:00, 57.19it/s]
Requesting API: 99%|█████████▉| 497/500 [00:15<00:00, 47.17it/s]
Requesting API: 100%|██████████| 500/500 [00:16<00:00, 30.80it/s]
2026-08-08:22:26:03 INFO [loggers.evaluation_tracker:247] Saving results aggregated
2026-08-08:22:26:03 INFO [loggers.evaluation_tracker:119] Saving per-task samples to outputs/evals/release/general-keep75-step0150/student/*.jsonl
local-completions ({'model': 'student', 'base_url': 'http://127.0.0.1:8461/v1/completions', 'tokenizer': 'outputs/release/healed/release-offpolicy-general-keep75/step0150', 'num_concurrent': 48, 'tokenized_requests': False, 'max_retries': 3}), gen_kwargs: ({}), limit: None, num_fewshot: None, batch_size: 1
| Tasks |Version| Filter |n-shot| Metric | |Value | |Stderr|
|---------|------:|-----------|-----:|---------|---|-----:|---|-----:|
|humaneval| 1|create_test| 0|pass@1 |↑ |0.3476|± |0.0373|
|mbpp | 1|none | 3|pass_at_1|↑ |0.2820|± |0.0201|
|