2026-08-08:18:17:39 INFO     [_cli.run:388] Selected Tasks: ['humaneval', 'mbpp']
2026-08-08:18:17:40 INFO     [evaluator:214] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2026-08-08:18:17:40 INFO     [evaluator:239] Initializing local-completions model, with arguments: {'model': 'student', 'base_url': 'http://127.0.0.1:8451/v1/completions', 'tokenizer': 'outputs/release/healed/release-offpolicy-general-keep25/step0150', 'num_concurrent': 48, 'tokenized_requests': False, 'max_retries': 3}
2026-08-08:18:17:40 INFO     [models.openai_completions:42] Remote tokenizer not supported. Using huggingface tokenizer backend.
2026-08-08:18:17:40 INFO     [models.api_models:179] Using max length 2048 - 1
2026-08-08:18:17:40 INFO     [models.api_models:200] Using tokenizer huggingface
2026-08-08:18:17:47 INFO     [evaluator_utils:446] Selected tasks:
2026-08-08:18:17:47 INFO     [evaluator_utils:480] Task: humaneval (humaneval/humaneval.yaml)
2026-08-08:18:17:47 INFO     [evaluator_utils:480] Task: mbpp (mbpp/mbpp.yaml)
2026-08-08:18:17:47 INFO     [evaluator:314] humaneval: Using gen_kwargs: {'until': ['\nclass', '\ndef', '\n#', '\nif', '\nprint'], 'max_gen_toks': 1024, 'do_sample': False}
2026-08-08:18:17:47 INFO     [evaluator:314] mbpp: Using gen_kwargs: {'until': ['[DONE]'], 'do_sample': False}
2026-08-08:18:17:47 INFO     [api.task:312] Building contexts for humaneval on rank 0...

  0%|          | 0/164 [00:00<?, ?it/s]
 90%|████████▉ | 147/164 [00:00<00:00, 1463.06it/s]
100%|██████████| 164/164 [00:00<00:00, 1489.47it/s]
2026-08-08:18:17:47 INFO     [api.task:312] Building contexts for mbpp on rank 0...

  0%|          | 0/500 [00:00<?, ?it/s]
  3%|▎         | 17/500 [00:00<00:03, 160.29it/s]
  7%|▋         | 34/500 [00:00<00:02, 163.07it/s]
 10%|█         | 51/500 [00:00<00:02, 164.14it/s]
 14%|█▎        | 68/500 [00:00<00:02, 164.09it/s]
 17%|█▋        | 85/500 [00:00<00:02, 164.59it/s]
 20%|██        | 102/500 [00:00<00:02, 165.02it/s]
 24%|██▍       | 119/500 [00:00<00:02, 165.36it/s]
 27%|██▋       | 136/500 [00:00<00:02, 165.64it/s]
 31%|███       | 153/500 [00:00<00:02, 165.90it/s]
 34%|███▍      | 170/500 [00:01<00:01, 166.11it/s]
 37%|███▋      | 187/500 [00:01<00:01, 166.24it/s]
 41%|████      | 204/500 [00:01<00:01, 166.36it/s]
 44%|████▍     | 221/500 [00:01<00:01, 166.44it/s]
 48%|████▊     | 238/500 [00:01<00:01, 166.22it/s]
 51%|█████     | 255/500 [00:01<00:01, 166.40it/s]
 54%|█████▍    | 272/500 [00:01<00:01, 166.64it/s]
 58%|█████▊    | 289/500 [00:01<00:01, 166.79it/s]
 61%|██████    | 306/500 [00:01<00:01, 166.87it/s]
 65%|██████▍   | 323/500 [00:01<00:01, 167.11it/s]
 68%|██████▊   | 340/500 [00:02<00:00, 167.10it/s]
 71%|███████▏  | 357/500 [00:02<00:00, 167.04it/s]
 75%|███████▍  | 374/500 [00:02<00:00, 167.09it/s]
 78%|███████▊  | 391/500 [00:02<00:00, 167.07it/s]
 82%|████████▏ | 408/500 [00:02<00:00, 167.03it/s]
 85%|████████▌ | 425/500 [00:02<00:00, 166.95it/s]
 88%|████████▊ | 442/500 [00:02<00:00, 167.14it/s]
 92%|█████████▏| 459/500 [00:02<00:00, 167.08it/s]
 95%|█████████▌| 476/500 [00:02<00:00, 166.94it/s]
 99%|█████████▊| 493/500 [00:02<00:00, 167.01it/s]
100%|██████████| 500/500 [00:03<00:00, 166.30it/s]
2026-08-08:18:17:50 INFO     [evaluator:585] Running generate_until requests
2026-08-08:18:17:50 INFO     [models.api_models:747] Tokenized requests are disabled. Context + generation length is not checked.

Requesting API:   0%|          | 0/164 [00:00<?, ?it/s]
Requesting API:   1%|          | 1/164 [00:00<01:36,  1.69it/s]
Requesting API:   2%|▏         | 3/164 [00:00<00:35,  4.56it/s]
Requesting API:   2%|▏         | 4/164 [00:00<00:30,  5.17it/s]
Requesting API:   5%|▍         | 8/164 [00:01<00:13, 11.87it/s]
Requesting API:   6%|▌         | 10/164 [00:01<00:18,  8.23it/s]
Requesting API:   7%|▋         | 12/164 [00:01<00:19,  7.66it/s]
Requesting API:   9%|▊         | 14/164 [00:01<00:15,  9.45it/s]
Requesting API:  10%|█         | 17/164 [00:01<00:11, 12.94it/s]
Requesting API:  12%|█▏        | 19/164 [00:02<00:21,  6.74it/s]
Requesting API:  13%|█▎        | 21/164 [00:02<00:22,  6.29it/s]
Requesting API:  14%|█▍        | 23/164 [00:03<00:25,  5.52it/s]
Requesting API:  15%|█▍        | 24/164 [00:03<00:24,  5.73it/s]
Requesting API:  15%|█▌        | 25/164 [00:03<00:25,  5.40it/s]
Requesting API:  16%|█▌        | 26/164 [00:03<00:22,  6.00it/s]
Requesting API:  16%|█▋        | 27/164 [00:04<00:24,  5.62it/s]
Requesting API:  17%|█▋        | 28/164 [00:04<00:22,  6.18it/s]
Requesting API:  18%|█▊        | 29/164 [00:04<00:24,  5.57it/s]
Requesting API:  18%|█▊        | 30/164 [00:04<00:26,  5.08it/s]
Requesting API:  19%|█▉        | 31/164 [00:05<00:35,  3.70it/s]
Requesting API:  20%|██        | 33/164 [00:05<00:26,  4.90it/s]
Requesting API:  21%|██        | 34/164 [00:06<00:42,  3.06it/s]
Requesting API:  21%|██▏       | 35/164 [00:06<00:38,  3.32it/s]
Requesting API:  22%|██▏       | 36/164 [00:09<02:02,  1.05it/s]
Requesting API:  23%|██▎       | 37/164 [00:13<03:51,  1.83s/it]
Requesting API:  23%|██▎       | 38/164 [00:13<02:50,  1.36s/it]
Requesting API:  24%|██▍       | 39/164 [00:15<03:32,  1.70s/it]
Requesting API:  37%|███▋      | 61/164 [00:16<00:17,  5.74it/s]
Requesting API:  41%|████      | 67/164 [00:16<00:13,  7.10it/s]
Requesting API:  44%|████▍     | 72/164 [00:16<00:11,  8.30it/s]
Requesting API:  46%|████▋     | 76/164 [00:16<00:09,  9.53it/s]
Requesting API:  49%|████▉     | 80/164 [00:16<00:07, 11.46it/s]
Requesting API:  51%|█████     | 84/164 [00:17<00:05, 13.42it/s]
Requesting API:  54%|█████▎    | 88/164 [00:17<00:04, 16.21it/s]
Requesting API:  56%|█████▌    | 92/164 [00:17<00:03, 18.60it/s]
Requesting API:  59%|█████▊    | 96/164 [00:17<00:05, 13.01it/s]
Requesting API:  60%|██████    | 99/164 [00:18<00:07,  8.36it/s]
Requesting API:  62%|██████▏   | 101/164 [00:18<00:08,  7.53it/s]
Requesting API:  63%|██████▎   | 103/164 [00:19<00:10,  5.79it/s]
Requesting API:  64%|██████▍   | 105/164 [00:20<00:14,  4.19it/s]
Requesting API:  65%|██████▍   | 106/164 [00:20<00:13,  4.18it/s]
Requesting API:  65%|██████▌   | 107/164 [00:20<00:12,  4.63it/s]
Requesting API:  66%|██████▌   | 108/164 [00:21<00:11,  4.75it/s]
Requesting API:  66%|██████▋   | 109/164 [00:21<00:13,  4.08it/s]
Requesting API:  68%|██████▊   | 111/164 [00:21<00:10,  4.84it/s]
Requesting API:  68%|██████▊   | 112/164 [00:22<00:11,  4.42it/s]
Requesting API:  70%|██████▉   | 114/164 [00:22<00:11,  4.25it/s]
Requesting API:  70%|███████   | 115/164 [00:24<00:25,  1.89it/s]
Requesting API:  71%|███████   | 116/164 [00:24<00:23,  2.06it/s]
Requesting API:  71%|███████▏  | 117/164 [00:28<01:04,  1.37s/it]
Requesting API:  72%|███████▏  | 118/164 [00:30<01:14,  1.61s/it]
Requesting API:  80%|████████  | 132/164 [00:31<00:08,  3.59it/s]
Requesting API:  82%|████████▏ | 134/164 [00:31<00:07,  3.84it/s]
Requesting API:  84%|████████▍ | 138/164 [00:31<00:05,  5.03it/s]
Requesting API:  86%|████████▌ | 141/164 [00:31<00:03,  6.21it/s]
Requesting API:  88%|████████▊ | 144/164 [00:31<00:02,  7.74it/s]
Requesting API:  90%|████████▉ | 147/164 [00:32<00:01,  9.46it/s]
Requesting API:  91%|█████████▏| 150/164 [00:32<00:01,  8.05it/s]
Requesting API:  93%|█████████▎| 152/164 [00:33<00:02,  5.70it/s]
Requesting API:  94%|█████████▍| 154/164 [00:33<00:01,  5.65it/s]
Requesting API:  95%|█████████▌| 156/164 [00:35<00:02,  3.16it/s]
Requesting API:  96%|█████████▌| 157/164 [00:35<00:02,  3.30it/s]
Requesting API:  96%|█████████▋| 158/164 [00:35<00:01,  3.37it/s]
Requesting API:  97%|█████████▋| 159/164 [00:35<00:01,  3.25it/s]
Requesting API:  98%|█████████▊| 160/164 [00:36<00:01,  2.77it/s]
Requesting API:  99%|█████████▉| 162/164 [00:37<00:00,  3.22it/s]
Requesting API:  99%|█████████▉| 163/164 [00:38<00:00,  1.76it/s]
Requesting API: 100%|██████████| 164/164 [00:38<00:00,  1.98it/s]
Requesting API: 100%|██████████| 164/164 [00:38<00:00,  4.23it/s]

Requesting API:   0%|          | 0/500 [00:00<?, ?it/s]
Requesting API:   0%|          | 1/500 [00:00<03:38,  2.29it/s]
Requesting API:   0%|          | 2/500 [00:00<03:23,  2.45it/s]
Requesting API:   1%|          | 4/500 [00:00<01:31,  5.43it/s]
Requesting API:   3%|▎         | 13/500 [00:01<00:24, 19.77it/s]
Requesting API:   4%|▍         | 19/500 [00:01<00:17, 27.81it/s]
Requesting API:   5%|▌         | 26/500 [00:01<00:13, 35.95it/s]
Requesting API:   6%|▌         | 31/500 [00:01<00:13, 35.94it/s]
Requesting API:   7%|▋         | 36/500 [00:01<00:12, 38.53it/s]
Requesting API:   9%|▊         | 43/500 [00:01<00:10, 43.98it/s]
Requesting API:  10%|▉         | 49/500 [00:01<00:10, 44.34it/s]
Requesting API:  11%|█         | 55/500 [00:01<00:09, 46.61it/s]
Requesting API:  12%|█▏        | 60/500 [00:02<00:10, 43.40it/s]
Requesting API:  14%|█▎        | 68/500 [00:02<00:08, 51.17it/s]
Requesting API:  15%|█▍        | 74/500 [00:02<00:08, 52.00it/s]
Requesting API:  16%|█▌        | 80/500 [00:02<00:08, 52.43it/s]
Requesting API:  17%|█▋        | 86/500 [00:02<00:08, 51.24it/s]
Requesting API:  18%|█▊        | 92/500 [00:02<00:07, 51.18it/s]
Requesting API:  20%|█▉        | 98/500 [00:02<00:08, 46.11it/s]
Requesting API:  21%|██        | 104/500 [00:02<00:08, 46.39it/s]
Requesting API:  22%|██▏       | 109/500 [00:03<00:08, 46.12it/s]
Requesting API:  23%|██▎       | 115/500 [00:03<00:08, 44.01it/s]
Requesting API:  24%|██▍       | 120/500 [00:03<00:08, 42.72it/s]
Requesting API:  25%|██▌       | 127/500 [00:03<00:08, 45.20it/s]
Requesting API:  26%|██▋       | 132/500 [00:03<00:08, 44.82it/s]
Requesting API:  27%|██▋       | 137/500 [00:03<00:08, 42.28it/s]
Requesting API:  28%|██▊       | 142/500 [00:03<00:08, 40.22it/s]
Requesting API:  30%|██▉       | 148/500 [00:04<00:09, 38.17it/s]
Requesting API:  31%|███▏      | 157/500 [00:04<00:07, 48.69it/s]
Requesting API:  33%|███▎      | 163/500 [00:04<00:07, 46.55it/s]
Requesting API:  34%|███▎      | 168/500 [00:04<00:08, 39.60it/s]
Requesting API:  35%|███▍      | 173/500 [00:04<00:07, 41.88it/s]
Requesting API:  36%|███▌      | 178/500 [00:04<00:08, 38.50it/s]
Requesting API:  37%|███▋      | 183/500 [00:04<00:08, 36.00it/s]
Requesting API:  38%|███▊      | 188/500 [00:04<00:08, 38.07it/s]
Requesting API:  38%|███▊      | 192/500 [00:05<00:08, 35.70it/s]
Requesting API:  40%|████      | 201/500 [00:05<00:06, 46.52it/s]
Requesting API:  41%|████▏     | 207/500 [00:05<00:06, 48.75it/s]
Requesting API:  43%|████▎     | 213/500 [00:05<00:05, 49.66it/s]
Requesting API:  44%|████▍     | 219/500 [00:05<00:06, 43.84it/s]
Requesting API:  45%|████▍     | 224/500 [00:05<00:07, 34.91it/s]
Requesting API:  46%|████▌     | 229/500 [00:05<00:07, 37.65it/s]
Requesting API:  48%|████▊     | 238/500 [00:06<00:05, 49.16it/s]
Requesting API:  49%|████▉     | 244/500 [00:06<00:05, 49.84it/s]
Requesting API:  50%|█████     | 250/500 [00:06<00:05, 47.25it/s]
Requesting API:  51%|█████     | 256/500 [00:06<00:05, 47.74it/s]
Requesting API:  52%|█████▏    | 261/500 [00:06<00:05, 43.75it/s]
Requesting API:  54%|█████▍    | 270/500 [00:06<00:04, 53.82it/s]
Requesting API:  55%|█████▌    | 276/500 [00:06<00:04, 45.86it/s]
Requesting API:  57%|█████▋    | 283/500 [00:07<00:05, 42.97it/s]
Requesting API:  58%|█████▊    | 289/500 [00:07<00:04, 46.01it/s]
Requesting API:  59%|█████▉    | 295/500 [00:07<00:04, 48.39it/s]
Requesting API:  60%|██████    | 301/500 [00:07<00:04, 45.24it/s]
Requesting API:  61%|██████▏   | 307/500 [00:07<00:04, 47.78it/s]
Requesting API:  63%|██████▎   | 313/500 [00:07<00:03, 49.72it/s]
Requesting API:  64%|██████▍   | 319/500 [00:07<00:03, 49.11it/s]
Requesting API:  65%|██████▌   | 325/500 [00:08<00:04, 36.52it/s]
Requesting API:  66%|██████▋   | 332/500 [00:08<00:03, 42.79it/s]
Requesting API:  68%|██████▊   | 341/500 [00:08<00:03, 52.41it/s]
Requesting API:  70%|██████▉   | 348/500 [00:08<00:02, 54.96it/s]
Requesting API:  71%|███████   | 355/500 [00:08<00:02, 49.13it/s]
Requesting API:  72%|███████▏  | 362/500 [00:08<00:02, 51.46it/s]
Requesting API:  74%|███████▎  | 368/500 [00:08<00:02, 52.70it/s]
Requesting API:  75%|███████▍  | 374/500 [00:08<00:02, 51.76it/s]
Requesting API:  76%|███████▌  | 380/500 [00:09<00:02, 49.56it/s]
Requesting API:  77%|███████▋  | 386/500 [00:09<00:02, 46.54it/s]
Requesting API:  78%|███████▊  | 391/500 [00:09<00:02, 46.86it/s]
Requesting API:  79%|███████▉  | 396/500 [00:09<00:02, 46.99it/s]
Requesting API:  80%|████████  | 401/500 [00:09<00:02, 40.99it/s]
Requesting API:  81%|████████  | 406/500 [00:09<00:02, 38.76it/s]
Requesting API:  82%|████████▏ | 411/500 [00:09<00:02, 36.03it/s]
Requesting API:  83%|████████▎ | 416/500 [00:09<00:02, 38.89it/s]
Requesting API:  84%|████████▍ | 421/500 [00:10<00:02, 35.02it/s]
Requesting API:  85%|████████▌ | 425/500 [00:10<00:02, 35.87it/s]
Requesting API:  87%|████████▋ | 433/500 [00:10<00:01, 45.69it/s]
Requesting API:  88%|████████▊ | 438/500 [00:10<00:01, 44.82it/s]
Requesting API:  89%|████████▊ | 443/500 [00:10<00:01, 43.83it/s]
Requesting API:  90%|████████▉ | 449/500 [00:10<00:01, 47.41it/s]
Requesting API:  91%|█████████ | 455/500 [00:10<00:00, 48.56it/s]
Requesting API:  92%|█████████▏| 460/500 [00:10<00:00, 47.17it/s]
Requesting API:  94%|█████████▎| 468/500 [00:11<00:00, 54.00it/s]
Requesting API:  95%|█████████▍| 474/500 [00:11<00:00, 39.53it/s]
Requesting API:  96%|█████████▌| 479/500 [00:11<00:00, 24.57it/s]
Requesting API:  97%|█████████▋| 483/500 [00:11<00:00, 22.38it/s]
Requesting API:  97%|█████████▋| 486/500 [00:12<00:00, 16.73it/s]
Requesting API:  98%|█████████▊| 489/500 [00:12<00:00, 16.82it/s]
Requesting API:  98%|█████████▊| 492/500 [00:12<00:00, 16.91it/s]
Requesting API:  99%|█████████▉| 495/500 [00:13<00:00, 11.51it/s]
Requesting API:  99%|█████████▉| 497/500 [00:13<00:00,  7.98it/s]
Requesting API: 100%|█████████▉| 499/500 [00:13<00:00,  8.07it/s]
Requesting API: 100%|██████████| 500/500 [00:14<00:00, 35.33it/s]
2026-08-08:18:21:03 INFO     [loggers.evaluation_tracker:247] Saving results aggregated
2026-08-08:18:21:03 INFO     [loggers.evaluation_tracker:119] Saving per-task samples to outputs/evals/release/general-keep25-step0150/student/*.jsonl
local-completions ({'model': 'student', 'base_url': 'http://127.0.0.1:8451/v1/completions', 'tokenizer': 'outputs/release/healed/release-offpolicy-general-keep25/step0150', 'num_concurrent': 48, 'tokenized_requests': False, 'max_retries': 3}), gen_kwargs: ({}), limit: None, num_fewshot: None, batch_size: 1
|  Tasks  |Version|  Filter   |n-shot| Metric  |   |Value |   |Stderr|
|---------|------:|-----------|-----:|---------|---|-----:|---|-----:|
|humaneval|      1|create_test|     0|pass@1   |↑  |0.0183|±  |0.0105|
|mbpp     |      1|none       |     3|pass_at_1|↑  |0.1120|±  |0.0141|