File size: 14,141 Bytes
a82fb9e | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 | 2026-08-08:22:08:16 INFO [_cli.run:388] Selected Tasks: ['humaneval', 'mbpp']
2026-08-08:22:08:17 INFO [evaluator:214] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2026-08-08:22:08:17 INFO [evaluator:239] Initializing local-completions model, with arguments: {'model': 'student', 'base_url': 'http://127.0.0.1:8462/v1/completions', 'tokenizer': 'outputs/release/healed/release-offpolicy-general-keep50/step0150', 'num_concurrent': 48, 'tokenized_requests': False, 'max_retries': 3}
2026-08-08:22:08:17 INFO [models.openai_completions:42] Remote tokenizer not supported. Using huggingface tokenizer backend.
2026-08-08:22:08:17 INFO [models.api_models:179] Using max length 2048 - 1
2026-08-08:22:08:17 INFO [models.api_models:200] Using tokenizer huggingface
2026-08-08:22:08:24 INFO [evaluator_utils:446] Selected tasks:
2026-08-08:22:08:24 INFO [evaluator_utils:480] Task: humaneval (humaneval/humaneval.yaml)
2026-08-08:22:08:24 INFO [evaluator_utils:480] Task: mbpp (mbpp/mbpp.yaml)
2026-08-08:22:08:24 INFO [evaluator:314] humaneval: Using gen_kwargs: {'until': ['\nclass', '\ndef', '\n#', '\nif', '\nprint'], 'max_gen_toks': 1024, 'do_sample': False}
2026-08-08:22:08:24 INFO [evaluator:314] mbpp: Using gen_kwargs: {'until': ['[DONE]'], 'do_sample': False}
2026-08-08:22:08:24 INFO [api.task:312] Building contexts for humaneval on rank 0...
0%| | 0/164 [00:00<?, ?it/s]
83%|████████▎ | 136/164 [00:00<00:00, 1358.33it/s]
100%|██████████| 164/164 [00:00<00:00, 1404.05it/s]
2026-08-08:22:08:25 INFO [api.task:312] Building contexts for mbpp on rank 0...
0%| | 0/500 [00:00<?, ?it/s]
3%|▎ | 16/500 [00:00<00:03, 155.06it/s]
6%|▋ | 32/500 [00:00<00:02, 157.19it/s]
10%|▉ | 48/500 [00:00<00:02, 156.62it/s]
13%|█▎ | 65/500 [00:00<00:02, 158.79it/s]
16%|█▋ | 82/500 [00:00<00:02, 160.78it/s]
20%|█▉ | 99/500 [00:00<00:02, 162.02it/s]
23%|██▎ | 116/500 [00:00<00:02, 162.80it/s]
27%|██▋ | 133/500 [00:00<00:02, 162.20it/s]
30%|███ | 150/500 [00:00<00:02, 161.60it/s]
33%|███▎ | 167/500 [00:01<00:02, 160.89it/s]
37%|███▋ | 184/500 [00:01<00:01, 160.67it/s]
40%|████ | 201/500 [00:01<00:01, 160.90it/s]
44%|████▎ | 218/500 [00:01<00:01, 161.01it/s]
47%|████▋ | 235/500 [00:01<00:01, 160.63it/s]
50%|█████ | 252/500 [00:01<00:01, 160.83it/s]
54%|█████▍ | 269/500 [00:01<00:01, 161.06it/s]
57%|█████▋ | 286/500 [00:01<00:01, 160.86it/s]
61%|██████ | 303/500 [00:01<00:01, 160.99it/s]
64%|██████▍ | 320/500 [00:01<00:01, 161.26it/s]
67%|██████▋ | 337/500 [00:02<00:01, 161.16it/s]
71%|███████ | 354/500 [00:02<00:00, 161.44it/s]
74%|███████▍ | 371/500 [00:02<00:00, 161.45it/s]
78%|███████▊ | 388/500 [00:02<00:00, 160.66it/s]
81%|████████ | 405/500 [00:02<00:00, 160.34it/s]
84%|████████▍ | 422/500 [00:02<00:00, 160.97it/s]
88%|████████▊ | 439/500 [00:02<00:00, 160.99it/s]
91%|█████████ | 456/500 [00:02<00:00, 161.19it/s]
95%|█████████▍| 473/500 [00:02<00:00, 161.02it/s]
98%|█████████▊| 490/500 [00:03<00:00, 160.07it/s]
100%|██████████| 500/500 [00:03<00:00, 160.81it/s]
2026-08-08:22:08:28 INFO [evaluator:585] Running generate_until requests
2026-08-08:22:08:28 INFO [models.api_models:747] Tokenized requests are disabled. Context + generation length is not checked.
Requesting API: 0%| | 0/164 [00:00<?, ?it/s]
Requesting API: 1%| | 1/164 [00:00<02:27, 1.11it/s]
Requesting API: 4%|▍ | 7/164 [00:01<00:17, 9.12it/s]
Requesting API: 7%|▋ | 11/164 [00:01<00:13, 10.94it/s]
Requesting API: 9%|▊ | 14/164 [00:01<00:12, 12.31it/s]
Requesting API: 11%|█ | 18/164 [00:01<00:08, 16.66it/s]
Requesting API: 13%|█▎ | 22/164 [00:01<00:06, 20.37it/s]
Requesting API: 16%|█▋ | 27/164 [00:01<00:05, 24.22it/s]
Requesting API: 19%|█▉ | 31/164 [00:02<00:05, 22.98it/s]
Requesting API: 23%|██▎ | 37/164 [00:02<00:04, 28.45it/s]
Requesting API: 26%|██▌ | 42/164 [00:02<00:03, 30.95it/s]
Requesting API: 28%|██▊ | 46/164 [00:02<00:03, 30.13it/s]
Requesting API: 34%|███▍ | 56/164 [00:02<00:02, 42.68it/s]
Requesting API: 37%|███▋ | 61/164 [00:02<00:02, 37.65it/s]
Requesting API: 40%|████ | 66/164 [00:02<00:02, 38.08it/s]
Requesting API: 43%|████▎ | 71/164 [00:03<00:02, 37.63it/s]
Requesting API: 46%|████▋ | 76/164 [00:03<00:02, 37.97it/s]
Requesting API: 51%|█████ | 83/164 [00:03<00:01, 42.59it/s]
Requesting API: 56%|█████▌ | 92/164 [00:03<00:01, 50.53it/s]
Requesting API: 60%|██████ | 99/164 [00:03<00:01, 47.97it/s]
Requesting API: 63%|██████▎ | 104/164 [00:03<00:01, 46.58it/s]
Requesting API: 67%|██████▋ | 110/164 [00:03<00:01, 49.44it/s]
Requesting API: 73%|███████▎ | 120/164 [00:03<00:00, 59.44it/s]
Requesting API: 77%|███████▋ | 127/164 [00:04<00:00, 61.03it/s]
Requesting API: 83%|████████▎ | 136/164 [00:04<00:00, 68.27it/s]
Requesting API: 87%|████████▋ | 143/164 [00:04<00:00, 53.96it/s]
Requesting API: 91%|█████████ | 149/164 [00:04<00:00, 48.88it/s]
Requesting API: 95%|█████████▍| 155/164 [00:04<00:00, 44.67it/s]
Requesting API: 98%|█████████▊| 160/164 [00:05<00:00, 21.10it/s]
Requesting API: 98%|█████████▊| 160/164 [00:16<00:00, 21.10it/s]
Requesting API: 100%|██████████| 164/164 [00:18<00:00, 1.38it/s]
Requesting API: 100%|██████████| 164/164 [00:18<00:00, 9.10it/s]
Requesting API: 0%| | 0/500 [00:00<?, ?it/s]
Requesting API: 0%| | 1/500 [00:00<08:13, 1.01it/s]
Requesting API: 1%| | 3/500 [00:01<02:26, 3.38it/s]
Requesting API: 2%|▏ | 12/500 [00:01<00:30, 16.24it/s]
Requesting API: 4%|▎ | 18/500 [00:01<00:22, 21.53it/s]
Requesting API: 5%|▍ | 23/500 [00:01<00:23, 20.35it/s]
Requesting API: 6%|▌ | 28/500 [00:01<00:18, 25.20it/s]
Requesting API: 6%|▋ | 32/500 [00:01<00:16, 27.97it/s]
Requesting API: 8%|▊ | 38/500 [00:01<00:14, 31.91it/s]
Requesting API: 9%|▉ | 44/500 [00:02<00:12, 36.65it/s]
Requesting API: 10%|▉ | 49/500 [00:02<00:13, 34.18it/s]
Requesting API: 11%|█ | 53/500 [00:02<00:14, 29.84it/s]
Requesting API: 12%|█▏ | 59/500 [00:02<00:13, 33.25it/s]
Requesting API: 13%|█▎ | 64/500 [00:02<00:11, 36.57it/s]
Requesting API: 14%|█▎ | 68/500 [00:02<00:11, 37.15it/s]
Requesting API: 15%|█▌ | 76/500 [00:03<00:10, 38.99it/s]
Requesting API: 16%|█▌ | 81/500 [00:03<00:10, 39.85it/s]
Requesting API: 17%|█▋ | 86/500 [00:03<00:10, 39.88it/s]
Requesting API: 18%|█▊ | 91/500 [00:03<00:09, 40.96it/s]
Requesting API: 19%|█▉ | 96/500 [00:03<00:09, 41.03it/s]
Requesting API: 20%|██ | 101/500 [00:03<00:09, 40.42it/s]
Requesting API: 21%|██ | 106/500 [00:03<00:09, 41.60it/s]
Requesting API: 22%|██▏ | 111/500 [00:03<00:09, 41.78it/s]
Requesting API: 23%|██▎ | 116/500 [00:03<00:09, 39.10it/s]
Requesting API: 24%|██▍ | 120/500 [00:04<00:11, 32.11it/s]
Requesting API: 25%|██▌ | 126/500 [00:04<00:10, 36.67it/s]
Requesting API: 26%|██▌ | 131/500 [00:04<00:09, 39.72it/s]
Requesting API: 27%|██▋ | 136/500 [00:04<00:10, 34.34it/s]
Requesting API: 28%|██▊ | 141/500 [00:04<00:10, 34.63it/s]
Requesting API: 29%|██▉ | 145/500 [00:04<00:12, 29.46it/s]
Requesting API: 30%|███ | 150/500 [00:05<00:10, 32.05it/s]
Requesting API: 31%|███ | 155/500 [00:05<00:10, 34.14it/s]
Requesting API: 32%|███▏ | 162/500 [00:05<00:08, 41.05it/s]
Requesting API: 33%|███▎ | 167/500 [00:05<00:08, 37.92it/s]
Requesting API: 34%|███▍ | 171/500 [00:05<00:08, 37.33it/s]
Requesting API: 35%|███▌ | 176/500 [00:05<00:08, 40.07it/s]
Requesting API: 36%|███▌ | 181/500 [00:05<00:10, 31.58it/s]
Requesting API: 37%|███▋ | 185/500 [00:06<00:10, 29.27it/s]
Requesting API: 39%|███▊ | 193/500 [00:06<00:09, 33.53it/s]
Requesting API: 39%|███▉ | 197/500 [00:06<00:08, 33.80it/s]
Requesting API: 40%|████ | 201/500 [00:06<00:08, 34.27it/s]
Requesting API: 41%|████ | 205/500 [00:06<00:08, 33.45it/s]
Requesting API: 42%|████▏ | 210/500 [00:06<00:08, 34.48it/s]
Requesting API: 43%|████▎ | 214/500 [00:06<00:08, 34.40it/s]
Requesting API: 44%|████▎ | 218/500 [00:06<00:08, 34.73it/s]
Requesting API: 45%|████▍ | 224/500 [00:07<00:07, 38.92it/s]
Requesting API: 46%|████▌ | 228/500 [00:07<00:07, 36.66it/s]
Requesting API: 46%|████▋ | 232/500 [00:07<00:07, 36.03it/s]
Requesting API: 47%|████▋ | 236/500 [00:07<00:07, 34.23it/s]
Requesting API: 48%|████▊ | 242/500 [00:07<00:06, 39.04it/s]
Requesting API: 49%|████▉ | 246/500 [00:07<00:06, 37.94it/s]
Requesting API: 50%|█████ | 252/500 [00:07<00:05, 41.58it/s]
Requesting API: 51%|█████▏ | 257/500 [00:08<00:07, 33.62it/s]
Requesting API: 53%|█████▎ | 263/500 [00:08<00:06, 38.31it/s]
Requesting API: 54%|█████▎ | 268/500 [00:08<00:05, 39.72it/s]
Requesting API: 55%|█████▍ | 273/500 [00:08<00:06, 35.40it/s]
Requesting API: 55%|█████▌ | 277/500 [00:08<00:06, 35.13it/s]
Requesting API: 57%|█████▋ | 283/500 [00:08<00:05, 38.12it/s]
Requesting API: 57%|█████▋ | 287/500 [00:08<00:06, 35.04it/s]
Requesting API: 58%|█████▊ | 291/500 [00:08<00:06, 32.38it/s]
Requesting API: 59%|█████▉ | 296/500 [00:09<00:06, 32.83it/s]
Requesting API: 60%|██████ | 301/500 [00:09<00:05, 36.31it/s]
Requesting API: 61%|██████ | 306/500 [00:09<00:05, 38.57it/s]
Requesting API: 63%|██████▎ | 314/500 [00:09<00:04, 43.89it/s]
Requesting API: 64%|██████▍ | 319/500 [00:09<00:04, 44.32it/s]
Requesting API: 65%|██████▍ | 324/500 [00:09<00:03, 44.44it/s]
Requesting API: 66%|██████▌ | 329/500 [00:09<00:03, 44.84it/s]
Requesting API: 67%|██████▋ | 334/500 [00:10<00:04, 33.46it/s]
Requesting API: 68%|██████▊ | 339/500 [00:10<00:04, 36.04it/s]
Requesting API: 69%|██████▉ | 347/500 [00:10<00:03, 44.68it/s]
Requesting API: 71%|███████ | 354/500 [00:10<00:02, 50.21it/s]
Requesting API: 72%|███████▏ | 360/500 [00:10<00:03, 44.08it/s]
Requesting API: 73%|███████▎ | 365/500 [00:10<00:03, 42.80it/s]
Requesting API: 74%|███████▍ | 370/500 [00:10<00:03, 42.01it/s]
Requesting API: 75%|███████▌ | 376/500 [00:10<00:02, 44.83it/s]
Requesting API: 76%|███████▌ | 381/500 [00:11<00:02, 43.44it/s]
Requesting API: 78%|███████▊ | 389/500 [00:11<00:02, 49.22it/s]
Requesting API: 79%|███████▉ | 394/500 [00:11<00:02, 48.02it/s]
Requesting API: 80%|███████▉ | 399/500 [00:11<00:02, 43.58it/s]
Requesting API: 81%|████████ | 406/500 [00:11<00:02, 46.55it/s]
Requesting API: 83%|████████▎ | 417/500 [00:11<00:01, 59.51it/s]
Requesting API: 85%|████████▍ | 424/500 [00:11<00:01, 45.41it/s]
Requesting API: 86%|████████▌ | 431/500 [00:12<00:01, 47.80it/s]
Requesting API: 88%|████████▊ | 438/500 [00:12<00:01, 51.33it/s]
Requesting API: 89%|████████▉ | 444/500 [00:12<00:01, 49.94it/s]
Requesting API: 91%|█████████ | 453/500 [00:12<00:00, 57.26it/s]
Requesting API: 92%|█████████▏| 460/500 [00:12<00:00, 54.71it/s]
Requesting API: 93%|█████████▎| 466/500 [00:12<00:00, 53.68it/s]
Requesting API: 95%|█████████▌| 477/500 [00:12<00:00, 65.74it/s]
Requesting API: 97%|█████████▋| 484/500 [00:12<00:00, 64.44it/s]
Requesting API: 98%|█████████▊| 491/500 [00:13<00:00, 34.93it/s]
Requesting API: 99%|█████████▉| 497/500 [00:14<00:00, 19.90it/s]
Requesting API: 100%|██████████| 500/500 [00:15<00:00, 31.52it/s]
2026-08-08:22:13:56 INFO [loggers.evaluation_tracker:247] Saving results aggregated
2026-08-08:22:13:56 INFO [loggers.evaluation_tracker:119] Saving per-task samples to outputs/evals/release/general-keep50-step0150/student/*.jsonl
local-completions ({'model': 'student', 'base_url': 'http://127.0.0.1:8462/v1/completions', 'tokenizer': 'outputs/release/healed/release-offpolicy-general-keep50/step0150', 'num_concurrent': 48, 'tokenized_requests': False, 'max_retries': 3}), gen_kwargs: ({}), limit: None, num_fewshot: None, batch_size: 1
| Tasks |Version| Filter |n-shot| Metric | |Value | |Stderr|
|---------|------:|-----------|-----:|---------|---|-----:|---|-----:|
|humaneval| 1|create_test| 0|pass@1 |↑ |0.2683|± |0.0347|
|mbpp | 1|none | 3|pass_at_1|↑ |0.2400|± |0.0191|
|