| 2026-07-23:22:12:13 WARNING [config.evaluate_config:287] --limit SHOULD ONLY BE USED FOR TESTING. REAL METRICS SHOULD NOT BE COMPUTED USING LIMIT. |
| 2026-07-23:22:12:20 INFO [_cli.run:388] Selected Tasks: ['mmlu_pro', 'gpqa_diamond_cot_zeroshot', 'minerva_math500', 'ifeval', 'gsm8k_cot_zeroshot'] |
| 2026-07-23:22:12:22 INFO [evaluator:214] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234 |
| 2026-07-23:22:12:22 WARNING [evaluator:226] generation_kwargs: {'max_gen_toks': 1280} specified through cli, these settings will update set parameters in yaml tasks. Ensure 'do_sample=True' for non-greedy decoding! |
| 2026-07-23:22:12:22 INFO [evaluator:239] Initializing local-chat-completions model, with arguments: {'model': 'student', 'base_url': 'http://127.0.0.1:8399/v1/chat/completions', 'num_concurrent': 32, 'tokenized_requests': False, 'max_retries': 3} |
| 2026-07-23:22:12:22 INFO [models.api_models:179] Using max length 2048 - 1 |
| 2026-07-23:22:12:22 INFO [models.api_models:200] Using tokenizer None |
|
Map: 0%| | 0/198 [00:00<?, ? examples/s]
Map: 100%|██████████| 198/198 [00:00<00:00, 1857.26 examples/s]
Map: 100%|██████████| 198/198 [00:00<00:00, 1545.93 examples/s] |
| 2026-07-23:22:12:36 INFO [evaluator_utils:446] Selected tasks: |
| 2026-07-23:22:12:36 INFO [evaluator_utils:462] Group: mmlu_pro |
| 2026-07-23:22:12:36 INFO [evaluator_utils:470] Task: mmlu_pro_biology (mmlu_pro/mmlu_pro_biology.yaml) |
| 2026-07-23:22:12:36 INFO [evaluator_utils:470] Task: mmlu_pro_business (mmlu_pro/mmlu_pro_business.yaml) |
| 2026-07-23:22:12:36 INFO [evaluator_utils:470] Task: mmlu_pro_chemistry (mmlu_pro/mmlu_pro_chemistry.yaml) |
| 2026-07-23:22:12:36 INFO [evaluator_utils:470] Task: mmlu_pro_computer_science (mmlu_pro/mmlu_pro_computer_science.yaml) |
| 2026-07-23:22:12:36 INFO [evaluator_utils:470] Task: mmlu_pro_economics (mmlu_pro/mmlu_pro_economics.yaml) |
| 2026-07-23:22:12:36 INFO [evaluator_utils:470] Task: mmlu_pro_engineering (mmlu_pro/mmlu_pro_engineering.yaml) |
| 2026-07-23:22:12:36 INFO [evaluator_utils:470] Task: mmlu_pro_health (mmlu_pro/mmlu_pro_health.yaml) |
| 2026-07-23:22:12:36 INFO [evaluator_utils:470] Task: mmlu_pro_history (mmlu_pro/mmlu_pro_history.yaml) |
| 2026-07-23:22:12:36 INFO [evaluator_utils:470] Task: mmlu_pro_law (mmlu_pro/mmlu_pro_law.yaml) |
| 2026-07-23:22:12:36 INFO [evaluator_utils:470] Task: mmlu_pro_math (mmlu_pro/mmlu_pro_math.yaml) |
| 2026-07-23:22:12:36 INFO [evaluator_utils:470] Task: mmlu_pro_other (mmlu_pro/mmlu_pro_other.yaml) |
| 2026-07-23:22:12:36 INFO [evaluator_utils:470] Task: mmlu_pro_philosophy (mmlu_pro/mmlu_pro_philosophy.yaml) |
| 2026-07-23:22:12:36 INFO [evaluator_utils:470] Task: mmlu_pro_physics (mmlu_pro/mmlu_pro_physics.yaml) |
| 2026-07-23:22:12:36 INFO [evaluator_utils:470] Task: mmlu_pro_psychology (mmlu_pro/mmlu_pro_psychology.yaml) |
| 2026-07-23:22:12:36 INFO [evaluator_utils:480] Task: gpqa_diamond_cot_zeroshot (gpqa/cot_zeroshot/gpqa_diamond_cot_zeroshot.yaml) |
| 2026-07-23:22:12:36 INFO [evaluator_utils:480] Task: gsm8k_cot_zeroshot (gsm8k/gsm8k-cot-zeroshot.yaml) |
| 2026-07-23:22:12:36 INFO [evaluator_utils:480] Task: ifeval (ifeval/ifeval.yaml) |
| 2026-07-23:22:12:36 INFO [evaluator_utils:480] Task: minerva_math500 (minerva_math/minerva_math500.yaml) |
| 2026-07-23:22:12:36 INFO [evaluator:314] mmlu_pro_biology: Using gen_kwargs: {'until': ['Question:'], 'max_gen_toks': 1280, 'do_sample': False, 'temperature': 0.0} |
| 2026-07-23:22:12:36 INFO [evaluator:314] mmlu_pro_business: Using gen_kwargs: {'until': ['Question:'], 'max_gen_toks': 1280, 'do_sample': False, 'temperature': 0.0} |
| 2026-07-23:22:12:36 INFO [evaluator:314] mmlu_pro_chemistry: Using gen_kwargs: {'until': ['Question:'], 'max_gen_toks': 1280, 'do_sample': False, 'temperature': 0.0} |
| 2026-07-23:22:12:36 INFO [evaluator:314] mmlu_pro_computer_science: Using gen_kwargs: {'until': ['Question:'], 'max_gen_toks': 1280, 'do_sample': False, 'temperature': 0.0} |
| 2026-07-23:22:12:36 INFO [evaluator:314] mmlu_pro_economics: Using gen_kwargs: {'until': ['Question:'], 'max_gen_toks': 1280, 'do_sample': False, 'temperature': 0.0} |
| 2026-07-23:22:12:36 INFO [evaluator:314] mmlu_pro_engineering: Using gen_kwargs: {'until': ['Question:'], 'max_gen_toks': 1280, 'do_sample': False, 'temperature': 0.0} |
| 2026-07-23:22:12:36 INFO [evaluator:314] mmlu_pro_health: Using gen_kwargs: {'until': ['Question:'], 'max_gen_toks': 1280, 'do_sample': False, 'temperature': 0.0} |
| 2026-07-23:22:12:36 INFO [evaluator:314] mmlu_pro_history: Using gen_kwargs: {'until': ['Question:'], 'max_gen_toks': 1280, 'do_sample': False, 'temperature': 0.0} |
| 2026-07-23:22:12:36 INFO [evaluator:314] mmlu_pro_law: Using gen_kwargs: {'until': ['Question:'], 'max_gen_toks': 1280, 'do_sample': False, 'temperature': 0.0} |
| 2026-07-23:22:12:36 INFO [evaluator:314] mmlu_pro_math: Using gen_kwargs: {'until': ['Question:'], 'max_gen_toks': 1280, 'do_sample': False, 'temperature': 0.0} |
| 2026-07-23:22:12:36 INFO [evaluator:314] mmlu_pro_other: Using gen_kwargs: {'until': ['Question:'], 'max_gen_toks': 1280, 'do_sample': False, 'temperature': 0.0} |
| 2026-07-23:22:12:36 INFO [evaluator:314] mmlu_pro_philosophy: Using gen_kwargs: {'until': ['Question:'], 'max_gen_toks': 1280, 'do_sample': False, 'temperature': 0.0} |
| 2026-07-23:22:12:36 INFO [evaluator:314] mmlu_pro_physics: Using gen_kwargs: {'until': ['Question:'], 'max_gen_toks': 1280, 'do_sample': False, 'temperature': 0.0} |
| 2026-07-23:22:12:36 INFO [evaluator:314] mmlu_pro_psychology: Using gen_kwargs: {'until': ['Question:'], 'max_gen_toks': 1280, 'do_sample': False, 'temperature': 0.0} |
| 2026-07-23:22:12:36 INFO [evaluator:314] gpqa_diamond_cot_zeroshot: Using gen_kwargs: {'until': ['</s>'], 'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 1280} |
| 2026-07-23:22:12:36 INFO [evaluator:314] minerva_math500: Using gen_kwargs: {'until': ['Problem:'], 'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 1280} |
| 2026-07-23:22:12:36 INFO [evaluator:314] ifeval: Using gen_kwargs: {'until': [], 'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 1280} |
| 2026-07-23:22:12:36 INFO [evaluator:314] gsm8k_cot_zeroshot: Using gen_kwargs: {'until': ['Q:', '</s>', '<|im_end|>'], 'do_sample': False, 'max_gen_toks': 1280} |
| 2026-07-23:22:12:36 INFO [api.task:312] Building contexts for mmlu_pro_biology on rank 0... |
|
0%| | 0/10 [00:00<?, ?it/s]
100%|██████████| 10/10 [00:00<00:00, 2830.55it/s] |
| 2026-07-23:22:12:36 INFO [api.task:312] Building contexts for mmlu_pro_business on rank 0... |
|
0%| | 0/10 [00:00<?, ?it/s]
100%|██████████| 10/10 [00:00<00:00, 3265.57it/s] |
| 2026-07-23:22:12:36 INFO [api.task:312] Building contexts for mmlu_pro_chemistry on rank 0... |
|
0%| | 0/10 [00:00<?, ?it/s]
100%|██████████| 10/10 [00:00<00:00, 3160.74it/s] |
| 2026-07-23:22:12:36 INFO [api.task:312] Building contexts for mmlu_pro_computer_science on rank 0... |
|
0%| | 0/10 [00:00<?, ?it/s]
100%|██████████| 10/10 [00:00<00:00, 3103.67it/s] |
| 2026-07-23:22:12:36 INFO [api.task:312] Building contexts for mmlu_pro_economics on rank 0... |
|
0%| | 0/10 [00:00<?, ?it/s]
100%|██████████| 10/10 [00:00<00:00, 3159.08it/s] |
| 2026-07-23:22:12:36 INFO [api.task:312] Building contexts for mmlu_pro_engineering on rank 0... |
|
0%| | 0/10 [00:00<?, ?it/s]
100%|██████████| 10/10 [00:00<00:00, 3100.00it/s] |
| 2026-07-23:22:12:36 INFO [api.task:312] Building contexts for mmlu_pro_health on rank 0... |
|
0%| | 0/10 [00:00<?, ?it/s]
100%|██████████| 10/10 [00:00<00:00, 3231.36it/s] |
| 2026-07-23:22:12:36 INFO [api.task:312] Building contexts for mmlu_pro_history on rank 0... |
|
0%| | 0/10 [00:00<?, ?it/s]
100%|██████████| 10/10 [00:00<00:00, 2916.56it/s] |
| 2026-07-23:22:12:36 INFO [api.task:312] Building contexts for mmlu_pro_law on rank 0... |
|
0%| | 0/10 [00:00<?, ?it/s]
100%|██████████| 10/10 [00:00<00:00, 3116.59it/s] |
| 2026-07-23:22:12:36 INFO [api.task:312] Building contexts for mmlu_pro_math on rank 0... |
|
0%| | 0/10 [00:00<?, ?it/s]
100%|██████████| 10/10 [00:00<00:00, 3102.53it/s] |
| 2026-07-23:22:12:36 INFO [api.task:312] Building contexts for mmlu_pro_other on rank 0... |
|
0%| | 0/10 [00:00<?, ?it/s]
100%|██████████| 10/10 [00:00<00:00, 3180.15it/s] |
| 2026-07-23:22:12:36 INFO [api.task:312] Building contexts for mmlu_pro_philosophy on rank 0... |
|
0%| | 0/10 [00:00<?, ?it/s]
100%|██████████| 10/10 [00:00<00:00, 3247.62it/s] |
| 2026-07-23:22:12:36 INFO [api.task:312] Building contexts for mmlu_pro_physics on rank 0... |
|
0%| | 0/10 [00:00<?, ?it/s]
100%|██████████| 10/10 [00:00<00:00, 3249.89it/s] |
| 2026-07-23:22:12:36 INFO [api.task:312] Building contexts for mmlu_pro_psychology on rank 0... |
|
0%| | 0/10 [00:00<?, ?it/s]
100%|██████████| 10/10 [00:00<00:00, 3171.50it/s] |
| 2026-07-23:22:12:36 INFO [api.task:312] Building contexts for gpqa_diamond_cot_zeroshot on rank 0... |
|
0%| | 0/10 [00:00<?, ?it/s]
100%|██████████| 10/10 [00:00<00:00, 1545.95it/s] |
| 2026-07-23:22:12:36 INFO [api.task:312] Building contexts for minerva_math500 on rank 0... |
|
0%| | 0/10 [00:00<?, ?it/s]
100%|██████████| 10/10 [00:00<00:00, 397.69it/s] |
| 2026-07-23:22:12:36 INFO [api.task:312] Building contexts for ifeval on rank 0... |
|
0%| | 0/10 [00:00<?, ?it/s]
100%|██████████| 10/10 [00:00<00:00, 52363.35it/s] |
| 2026-07-23:22:12:36 INFO [api.task:312] Building contexts for gsm8k_cot_zeroshot on rank 0... |
|
0%| | 0/10 [00:00<?, ?it/s]
100%|██████████| 10/10 [00:00<00:00, 1822.50it/s] |
| 2026-07-23:22:12:37 INFO [evaluator:585] Running generate_until requests |
| 2026-07-23:22:12:37 INFO [models.api_models:747] Tokenized requests are disabled. Context + generation length is not checked. |
|
Requesting API: 0%| | 0/140 [00:00<?, ?it/s]
Requesting API: 1%| | 1/140 [00:03<09:04, 3.92s/it]
Requesting API: 3%|▎ | 4/140 [00:04<02:01, 1.11it/s]
Requesting API: 4%|▎ | 5/140 [00:04<01:38, 1.38it/s]
Requesting API: 4%|▍ | 6/140 [00:05<01:31, 1.47it/s]
Requesting API: 5%|▌ | 7/140 [00:05<01:15, 1.76it/s]
Requesting API: 6%|▋ | 9/140 [00:05<00:49, 2.67it/s]
Requesting API: 7%|▋ | 10/140 [00:06<00:54, 2.37it/s]
Requesting API: 9%|▊ | 12/140 [00:06<00:39, 3.26it/s]
Requesting API: 10%|█ | 14/140 [00:07<00:30, 4.07it/s]
Requesting API: 11%|█ | 15/140 [00:07<00:31, 3.93it/s]
Requesting API: 12%|█▏ | 17/140 [00:07<00:26, 4.72it/s]
Requesting API: 13%|█▎ | 18/140 [00:08<00:34, 3.49it/s]
Requesting API: 14%|█▎ | 19/140 [00:08<00:34, 3.49it/s]
Requesting API: 16%|█▌ | 22/140 [00:08<00:22, 5.24it/s]
Requesting API: 16%|█▋ | 23/140 [00:09<00:24, 4.80it/s]
Requesting API: 17%|█▋ | 24/140 [00:09<00:26, 4.44it/s]
Requesting API: 19%|█▉ | 27/140 [00:09<00:23, 4.82it/s]
Requesting API: 20%|██ | 28/140 [00:10<00:30, 3.65it/s]
Requesting API: 22%|██▏ | 31/140 [00:11<00:25, 4.23it/s]
Requesting API: 24%|██▎ | 33/140 [00:11<00:22, 4.80it/s]
Requesting API: 25%|██▌ | 35/140 [00:11<00:19, 5.29it/s]
Requesting API: 27%|██▋ | 38/140 [00:11<00:15, 6.56it/s]
Requesting API: 29%|██▉ | 41/140 [00:12<00:13, 7.61it/s]
Requesting API: 30%|███ | 42/140 [00:12<00:15, 6.51it/s]
Requesting API: 31%|███ | 43/140 [00:12<00:17, 5.68it/s]
Requesting API: 31%|███▏ | 44/140 [00:13<00:18, 5.08it/s]
Requesting API: 32%|███▏ | 45/140 [00:13<00:20, 4.64it/s]
Requesting API: 33%|███▎ | 46/140 [00:14<00:34, 2.72it/s]
Requesting API: 34%|███▎ | 47/140 [00:14<00:32, 2.89it/s]
Requesting API: 35%|███▌ | 49/140 [00:15<00:34, 2.61it/s]
Requesting API: 37%|███▋ | 52/140 [00:15<00:21, 4.05it/s]
Requesting API: 38%|███▊ | 53/140 [00:15<00:22, 3.93it/s]
Requesting API: 40%|████ | 56/140 [00:16<00:15, 5.42it/s]
Requesting API: 41%|████ | 57/140 [00:16<00:16, 4.95it/s]
Requesting API: 41%|████▏ | 58/140 [00:17<00:22, 3.62it/s]
Requesting API: 42%|████▏ | 59/140 [00:17<00:27, 2.92it/s]
Requesting API: 43%|████▎ | 60/140 [00:19<00:54, 1.48it/s]
Requesting API: 44%|████▎ | 61/140 [00:19<00:45, 1.73it/s]
Requesting API: 45%|████▌ | 63/140 [00:20<00:35, 2.19it/s]
Requesting API: 46%|████▋ | 65/140 [00:20<00:25, 2.94it/s]
Requesting API: 47%|████▋ | 66/140 [00:21<00:39, 1.89it/s]
Requesting API: 50%|█████ | 70/140 [00:21<00:18, 3.75it/s]
Requesting API: 51%|█████▏ | 72/140 [00:22<00:18, 3.65it/s]
Requesting API: 54%|█████▎ | 75/140 [00:22<00:13, 4.83it/s]
Requesting API: 54%|█████▍ | 76/140 [00:23<00:14, 4.55it/s]
Requesting API: 56%|█████▋ | 79/140 [00:24<00:17, 3.52it/s]
Requesting API: 57%|█████▋ | 80/140 [00:24<00:16, 3.60it/s]
Requesting API: 59%|█████▊ | 82/140 [00:25<00:16, 3.57it/s]
Requesting API: 59%|█████▉ | 83/140 [00:25<00:14, 3.93it/s]
Requesting API: 61%|██████ | 85/140 [00:25<00:11, 4.65it/s]
Requesting API: 62%|██████▏ | 87/140 [00:25<00:11, 4.69it/s]
Requesting API: 63%|██████▎ | 88/140 [00:26<00:13, 3.95it/s]
Requesting API: 64%|██████▍ | 90/140 [00:26<00:10, 4.68it/s]
Requesting API: 65%|██████▌ | 91/140 [00:27<00:12, 3.91it/s]
Requesting API: 66%|██████▋ | 93/140 [00:27<00:09, 4.95it/s]
Requesting API: 69%|██████▊ | 96/140 [00:27<00:07, 5.94it/s]
Requesting API: 69%|██████▉ | 97/140 [00:27<00:08, 5.29it/s]
Requesting API: 70%|███████ | 98/140 [00:28<00:11, 3.77it/s]
Requesting API: 71%|███████ | 99/140 [00:28<00:10, 3.88it/s]
Requesting API: 76%|███████▌ | 106/140 [00:29<00:04, 7.23it/s]
Requesting API: 76%|███████▋ | 107/140 [00:29<00:05, 6.37it/s]
Requesting API: 77%|███████▋ | 108/140 [00:29<00:05, 5.68it/s]
Requesting API: 78%|███████▊ | 109/140 [00:30<00:07, 4.06it/s]
Requesting API: 79%|███████▉ | 111/140 [00:30<00:05, 4.88it/s]
Requesting API: 81%|████████ | 113/140 [00:30<00:04, 5.94it/s]
Requesting API: 83%|████████▎ | 116/140 [00:31<00:03, 7.99it/s]
Requesting API: 85%|████████▌ | 119/140 [00:31<00:02, 9.76it/s]
Requesting API: 86%|████████▋ | 121/140 [00:31<00:01, 9.90it/s]
Requesting API: 88%|████████▊ | 123/140 [00:31<00:01, 10.12it/s]
Requesting API: 90%|█████████ | 126/140 [00:31<00:01, 10.35it/s]
Requesting API: 91%|█████████▏| 128/140 [00:32<00:01, 8.38it/s]
Requesting API: 92%|█████████▏| 129/140 [00:32<00:02, 5.34it/s]
Requesting API: 93%|█████████▎| 130/140 [00:33<00:02, 4.56it/s]
Requesting API: 94%|█████████▎| 131/140 [00:33<00:02, 4.30it/s]
Requesting API: 94%|█████████▍| 132/140 [00:33<00:01, 4.13it/s]
Requesting API: 95%|█████████▌| 133/140 [00:34<00:01, 4.01it/s]
Requesting API: 96%|█████████▋| 135/140 [00:34<00:01, 3.86it/s]
Requesting API: 97%|█████████▋| 136/140 [00:35<00:01, 2.05it/s]
Requesting API: 98%|█████████▊| 137/140 [00:37<00:01, 1.53it/s]
Requesting API: 99%|█████████▊| 138/140 [00:37<00:01, 1.73it/s]
Requesting API: 99%|█████████▉| 139/140 [00:40<00:01, 1.29s/it]
Requesting API: 100%|██████████| 140/140 [00:47<00:00, 2.95s/it]
Requesting API: 100%|██████████| 140/140 [00:47<00:00, 2.93it/s] |
|
Requesting API: 0%| | 0/10 [00:00<?, ?it/s]
Requesting API: 10%|█ | 1/10 [00:01<00:14, 1.66s/it]
Requesting API: 20%|██ | 2/10 [00:02<00:09, 1.21s/it]
Requesting API: 30%|███ | 3/10 [00:14<00:41, 5.89s/it]
Requesting API: 40%|████ | 4/10 [00:16<00:26, 4.48s/it]
Requesting API: 50%|█████ | 5/10 [00:22<00:24, 4.93s/it]
Requesting API: 60%|██████ | 6/10 [00:24<00:16, 4.18s/it]
Requesting API: 70%|███████ | 7/10 [01:54<01:36, 32.29s/it]
Requesting API: 100%|██████████| 10/10 [01:54<00:00, 11.49s/it] |
|
Requesting API: 0%| | 0/10 [00:00<?, ?it/s]
Requesting API: 10%|█ | 1/10 [00:00<00:02, 3.72it/s]
Requesting API: 20%|██ | 2/10 [00:00<00:03, 2.29it/s]
Requesting API: 40%|████ | 4/10 [00:01<00:01, 4.01it/s]
Requesting API: 60%|██████ | 6/10 [00:01<00:00, 5.10it/s]
Requesting API: 100%|██████████| 10/10 [00:01<00:00, 6.85it/s] |
|
Requesting API: 0%| | 0/10 [00:00<?, ?it/s]
Requesting API: 10%|█ | 1/10 [00:00<00:07, 1.16it/s]
Requesting API: 20%|██ | 2/10 [00:03<00:16, 2.10s/it]
Requesting API: 30%|███ | 3/10 [00:05<00:12, 1.72s/it]
Requesting API: 40%|████ | 4/10 [00:10<00:18, 3.10s/it]
Requesting API: 50%|█████ | 5/10 [00:20<00:28, 5.77s/it]
Requesting API: 60%|██████ | 6/10 [00:32<00:31, 7.91s/it]
Requesting API: 70%|███████ | 7/10 [01:55<01:36, 32.21s/it]
Requesting API: 100%|██████████| 10/10 [01:55<00:00, 11.51s/it] |
|
Requesting API: 0%| | 0/10 [00:00<?, ?it/s]
Requesting API: 10%|█ | 1/10 [00:00<00:04, 2.10it/s]
Requesting API: 40%|████ | 4/10 [00:01<00:01, 3.52it/s]
Requesting API: 60%|██████ | 6/10 [00:11<00:09, 2.49s/it]
Requesting API: 70%|███████ | 7/10 [00:12<00:05, 1.96s/it]
Requesting API: 80%|████████ | 8/10 [00:21<00:07, 3.92s/it]
Requesting API: 90%|█████████ | 9/10 [01:02<00:13, 13.71s/it]
Requesting API: 100%|██████████| 10/10 [01:53<00:00, 23.92s/it]
Requesting API: 100%|██████████| 10/10 [01:53<00:00, 11.34s/it] |
| 2026-07-23:22:19:12 INFO [loggers.evaluation_tracker:247] Saving results aggregated |
| 2026-07-23:22:19:12 INFO [loggers.evaluation_tracker:119] Saving per-task samples to outputs/evals/protocolF_gate3/student/*.jsonl |
| local-chat-completions ({'model': 'student', 'base_url': 'http://127.0.0.1:8399/v1/chat/completions', 'num_concurrent': 32, 'tokenized_requests': False, 'max_retries': 3}), gen_kwargs: ({'max_gen_toks': 1280}), limit: 10.0, num_fewshot: None, batch_size: 1 |
| | Tasks |Version| Filter |n-shot| Metric | |Value | |Stderr| |
| |-------------------------|------:|----------------|-----:|-----------------------|---|-----:|---|------| |
| |mmlu_pro | 2.0|custom-extract | 5|exact_match |↑ |0.0571|± |0.0190| |
| | - biology | 3.1|custom-extract | 5|exact_match |↑ |0.0000|± | 0| |
| | - business | 3.1|custom-extract | 5|exact_match |↑ |0.3000|± |0.1528| |
| | - chemistry | 3.1|custom-extract | 5|exact_match |↑ |0.0000|± | 0| |
| | - computer_science | 3.1|custom-extract | 5|exact_match |↑ |0.1000|± |0.1000| |
| | - economics | 3.1|custom-extract | 5|exact_match |↑ |0.0000|± | 0| |
| | - engineering | 3.1|custom-extract | 5|exact_match |↑ |0.0000|± | 0| |
| | - health | 3.1|custom-extract | 5|exact_match |↑ |0.0000|± | 0| |
| | - history | 3.1|custom-extract | 5|exact_match |↑ |0.1000|± |0.1000| |
| | - law | 3.1|custom-extract | 5|exact_match |↑ |0.1000|± |0.1000| |
| | - math | 3.1|custom-extract | 5|exact_match |↑ |0.0000|± | 0| |
| | - other | 3.1|custom-extract | 5|exact_match |↑ |0.2000|± |0.1333| |
| | - philosophy | 3.1|custom-extract | 5|exact_match |↑ |0.0000|± | 0| |
| | - physics | 3.1|custom-extract | 5|exact_match |↑ |0.0000|± | 0| |
| | - psychology | 3.1|custom-extract | 5|exact_match |↑ |0.0000|± | 0| |
| |gpqa_diamond_cot_zeroshot| 2.2|flexible-extract| 0|exact_match |↑ |0.3000|± |0.1528| |
| | | |strict-match | 0|exact_match |↑ |0.0000|± | 0| |
| |gsm8k_cot_zeroshot | 3.0|flexible-extract| 0|exact_match |↑ |0.1000|± |0.1000| |
| | | |strict-match | 0|exact_match |↑ |0.0000|± | 0| |
| |ifeval | 4.0|none | 0|inst_level_loose_acc |↑ |0.4444|± | N/A| |
| | | |none | 0|inst_level_strict_acc |↑ |0.4444|± | N/A| |
| | | |none | 0|prompt_level_loose_acc |↑ |0.3000|± |0.1528| |
| | | |none | 0|prompt_level_strict_acc|↑ |0.3000|± |0.1528| |
| |minerva_math500 | 3.0|none | 4|exact_match |↑ |0.0000|± | 0| |
| | | |none | 4|math_verify |↑ |0.0000|± | 0| |
|
|
| | Groups |Version| Filter |n-shot| Metric | |Value | |Stderr| |
| |--------|------:|--------------|-----:|-----------|---|-----:|---|-----:| |
| |mmlu_pro| 2|custom-extract| 5|exact_match|↑ |0.0571|± | 0.019| |
|
|
|
|
