2026-07-23:22:12:13 WARNING  [config.evaluate_config:287] --limit SHOULD ONLY BE USED FOR TESTING. REAL METRICS SHOULD NOT BE COMPUTED USING LIMIT.
2026-07-23:22:12:20 INFO     [_cli.run:388] Selected Tasks: ['mmlu_pro', 'gpqa_diamond_cot_zeroshot', 'minerva_math500', 'ifeval', 'gsm8k_cot_zeroshot']
2026-07-23:22:12:22 INFO     [evaluator:214] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2026-07-23:22:12:22 WARNING  [evaluator:226] generation_kwargs: {'max_gen_toks': 1280} specified through cli, these settings will update set parameters in yaml tasks. Ensure 'do_sample=True' for non-greedy decoding!
2026-07-23:22:12:22 INFO     [evaluator:239] Initializing local-chat-completions model, with arguments: {'model': 'student', 'base_url': 'http://127.0.0.1:8399/v1/chat/completions', 'num_concurrent': 32, 'tokenized_requests': False, 'max_retries': 3}
2026-07-23:22:12:22 INFO     [models.api_models:179] Using max length 2048 - 1
2026-07-23:22:12:22 INFO     [models.api_models:200] Using tokenizer None

Map:   0%|          | 0/198 [00:00<?, ? examples/s]
Map: 100%|██████████| 198/198 [00:00<00:00, 1857.26 examples/s]
Map: 100%|██████████| 198/198 [00:00<00:00, 1545.93 examples/s]
2026-07-23:22:12:36 INFO     [evaluator_utils:446] Selected tasks:
2026-07-23:22:12:36 INFO     [evaluator_utils:462] Group: mmlu_pro
2026-07-23:22:12:36 INFO     [evaluator_utils:470]   Task: mmlu_pro_biology (mmlu_pro/mmlu_pro_biology.yaml)
2026-07-23:22:12:36 INFO     [evaluator_utils:470]   Task: mmlu_pro_business (mmlu_pro/mmlu_pro_business.yaml)
2026-07-23:22:12:36 INFO     [evaluator_utils:470]   Task: mmlu_pro_chemistry (mmlu_pro/mmlu_pro_chemistry.yaml)
2026-07-23:22:12:36 INFO     [evaluator_utils:470]   Task: mmlu_pro_computer_science (mmlu_pro/mmlu_pro_computer_science.yaml)
2026-07-23:22:12:36 INFO     [evaluator_utils:470]   Task: mmlu_pro_economics (mmlu_pro/mmlu_pro_economics.yaml)
2026-07-23:22:12:36 INFO     [evaluator_utils:470]   Task: mmlu_pro_engineering (mmlu_pro/mmlu_pro_engineering.yaml)
2026-07-23:22:12:36 INFO     [evaluator_utils:470]   Task: mmlu_pro_health (mmlu_pro/mmlu_pro_health.yaml)
2026-07-23:22:12:36 INFO     [evaluator_utils:470]   Task: mmlu_pro_history (mmlu_pro/mmlu_pro_history.yaml)
2026-07-23:22:12:36 INFO     [evaluator_utils:470]   Task: mmlu_pro_law (mmlu_pro/mmlu_pro_law.yaml)
2026-07-23:22:12:36 INFO     [evaluator_utils:470]   Task: mmlu_pro_math (mmlu_pro/mmlu_pro_math.yaml)
2026-07-23:22:12:36 INFO     [evaluator_utils:470]   Task: mmlu_pro_other (mmlu_pro/mmlu_pro_other.yaml)
2026-07-23:22:12:36 INFO     [evaluator_utils:470]   Task: mmlu_pro_philosophy (mmlu_pro/mmlu_pro_philosophy.yaml)
2026-07-23:22:12:36 INFO     [evaluator_utils:470]   Task: mmlu_pro_physics (mmlu_pro/mmlu_pro_physics.yaml)
2026-07-23:22:12:36 INFO     [evaluator_utils:470]   Task: mmlu_pro_psychology (mmlu_pro/mmlu_pro_psychology.yaml)
2026-07-23:22:12:36 INFO     [evaluator_utils:480] Task: gpqa_diamond_cot_zeroshot (gpqa/cot_zeroshot/gpqa_diamond_cot_zeroshot.yaml)
2026-07-23:22:12:36 INFO     [evaluator_utils:480] Task: gsm8k_cot_zeroshot (gsm8k/gsm8k-cot-zeroshot.yaml)
2026-07-23:22:12:36 INFO     [evaluator_utils:480] Task: ifeval (ifeval/ifeval.yaml)
2026-07-23:22:12:36 INFO     [evaluator_utils:480] Task: minerva_math500 (minerva_math/minerva_math500.yaml)
2026-07-23:22:12:36 INFO     [evaluator:314] mmlu_pro_biology: Using gen_kwargs: {'until': ['Question:'], 'max_gen_toks': 1280, 'do_sample': False, 'temperature': 0.0}
2026-07-23:22:12:36 INFO     [evaluator:314] mmlu_pro_business: Using gen_kwargs: {'until': ['Question:'], 'max_gen_toks': 1280, 'do_sample': False, 'temperature': 0.0}
2026-07-23:22:12:36 INFO     [evaluator:314] mmlu_pro_chemistry: Using gen_kwargs: {'until': ['Question:'], 'max_gen_toks': 1280, 'do_sample': False, 'temperature': 0.0}
2026-07-23:22:12:36 INFO     [evaluator:314] mmlu_pro_computer_science: Using gen_kwargs: {'until': ['Question:'], 'max_gen_toks': 1280, 'do_sample': False, 'temperature': 0.0}
2026-07-23:22:12:36 INFO     [evaluator:314] mmlu_pro_economics: Using gen_kwargs: {'until': ['Question:'], 'max_gen_toks': 1280, 'do_sample': False, 'temperature': 0.0}
2026-07-23:22:12:36 INFO     [evaluator:314] mmlu_pro_engineering: Using gen_kwargs: {'until': ['Question:'], 'max_gen_toks': 1280, 'do_sample': False, 'temperature': 0.0}
2026-07-23:22:12:36 INFO     [evaluator:314] mmlu_pro_health: Using gen_kwargs: {'until': ['Question:'], 'max_gen_toks': 1280, 'do_sample': False, 'temperature': 0.0}
2026-07-23:22:12:36 INFO     [evaluator:314] mmlu_pro_history: Using gen_kwargs: {'until': ['Question:'], 'max_gen_toks': 1280, 'do_sample': False, 'temperature': 0.0}
2026-07-23:22:12:36 INFO     [evaluator:314] mmlu_pro_law: Using gen_kwargs: {'until': ['Question:'], 'max_gen_toks': 1280, 'do_sample': False, 'temperature': 0.0}
2026-07-23:22:12:36 INFO     [evaluator:314] mmlu_pro_math: Using gen_kwargs: {'until': ['Question:'], 'max_gen_toks': 1280, 'do_sample': False, 'temperature': 0.0}
2026-07-23:22:12:36 INFO     [evaluator:314] mmlu_pro_other: Using gen_kwargs: {'until': ['Question:'], 'max_gen_toks': 1280, 'do_sample': False, 'temperature': 0.0}
2026-07-23:22:12:36 INFO     [evaluator:314] mmlu_pro_philosophy: Using gen_kwargs: {'until': ['Question:'], 'max_gen_toks': 1280, 'do_sample': False, 'temperature': 0.0}
2026-07-23:22:12:36 INFO     [evaluator:314] mmlu_pro_physics: Using gen_kwargs: {'until': ['Question:'], 'max_gen_toks': 1280, 'do_sample': False, 'temperature': 0.0}
2026-07-23:22:12:36 INFO     [evaluator:314] mmlu_pro_psychology: Using gen_kwargs: {'until': ['Question:'], 'max_gen_toks': 1280, 'do_sample': False, 'temperature': 0.0}
2026-07-23:22:12:36 INFO     [evaluator:314] gpqa_diamond_cot_zeroshot: Using gen_kwargs: {'until': ['</s>'], 'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 1280}
2026-07-23:22:12:36 INFO     [evaluator:314] minerva_math500: Using gen_kwargs: {'until': ['Problem:'], 'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 1280}
2026-07-23:22:12:36 INFO     [evaluator:314] ifeval: Using gen_kwargs: {'until': [], 'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 1280}
2026-07-23:22:12:36 INFO     [evaluator:314] gsm8k_cot_zeroshot: Using gen_kwargs: {'until': ['Q:', '</s>', '<|im_end|>'], 'do_sample': False, 'max_gen_toks': 1280}
2026-07-23:22:12:36 INFO     [api.task:312] Building contexts for mmlu_pro_biology on rank 0...

  0%|          | 0/10 [00:00<?, ?it/s]
100%|██████████| 10/10 [00:00<00:00, 2830.55it/s]
2026-07-23:22:12:36 INFO     [api.task:312] Building contexts for mmlu_pro_business on rank 0...

  0%|          | 0/10 [00:00<?, ?it/s]
100%|██████████| 10/10 [00:00<00:00, 3265.57it/s]
2026-07-23:22:12:36 INFO     [api.task:312] Building contexts for mmlu_pro_chemistry on rank 0...

  0%|          | 0/10 [00:00<?, ?it/s]
100%|██████████| 10/10 [00:00<00:00, 3160.74it/s]
2026-07-23:22:12:36 INFO     [api.task:312] Building contexts for mmlu_pro_computer_science on rank 0...

  0%|          | 0/10 [00:00<?, ?it/s]
100%|██████████| 10/10 [00:00<00:00, 3103.67it/s]
2026-07-23:22:12:36 INFO     [api.task:312] Building contexts for mmlu_pro_economics on rank 0...

  0%|          | 0/10 [00:00<?, ?it/s]
100%|██████████| 10/10 [00:00<00:00, 3159.08it/s]
2026-07-23:22:12:36 INFO     [api.task:312] Building contexts for mmlu_pro_engineering on rank 0...

  0%|          | 0/10 [00:00<?, ?it/s]
100%|██████████| 10/10 [00:00<00:00, 3100.00it/s]
2026-07-23:22:12:36 INFO     [api.task:312] Building contexts for mmlu_pro_health on rank 0...

  0%|          | 0/10 [00:00<?, ?it/s]
100%|██████████| 10/10 [00:00<00:00, 3231.36it/s]
2026-07-23:22:12:36 INFO     [api.task:312] Building contexts for mmlu_pro_history on rank 0...

  0%|          | 0/10 [00:00<?, ?it/s]
100%|██████████| 10/10 [00:00<00:00, 2916.56it/s]
2026-07-23:22:12:36 INFO     [api.task:312] Building contexts for mmlu_pro_law on rank 0...

  0%|          | 0/10 [00:00<?, ?it/s]
100%|██████████| 10/10 [00:00<00:00, 3116.59it/s]
2026-07-23:22:12:36 INFO     [api.task:312] Building contexts for mmlu_pro_math on rank 0...

  0%|          | 0/10 [00:00<?, ?it/s]
100%|██████████| 10/10 [00:00<00:00, 3102.53it/s]
2026-07-23:22:12:36 INFO     [api.task:312] Building contexts for mmlu_pro_other on rank 0...

  0%|          | 0/10 [00:00<?, ?it/s]
100%|██████████| 10/10 [00:00<00:00, 3180.15it/s]
2026-07-23:22:12:36 INFO     [api.task:312] Building contexts for mmlu_pro_philosophy on rank 0...

  0%|          | 0/10 [00:00<?, ?it/s]
100%|██████████| 10/10 [00:00<00:00, 3247.62it/s]
2026-07-23:22:12:36 INFO     [api.task:312] Building contexts for mmlu_pro_physics on rank 0...

  0%|          | 0/10 [00:00<?, ?it/s]
100%|██████████| 10/10 [00:00<00:00, 3249.89it/s]
2026-07-23:22:12:36 INFO     [api.task:312] Building contexts for mmlu_pro_psychology on rank 0...

  0%|          | 0/10 [00:00<?, ?it/s]
100%|██████████| 10/10 [00:00<00:00, 3171.50it/s]
2026-07-23:22:12:36 INFO     [api.task:312] Building contexts for gpqa_diamond_cot_zeroshot on rank 0...

  0%|          | 0/10 [00:00<?, ?it/s]
100%|██████████| 10/10 [00:00<00:00, 1545.95it/s]
2026-07-23:22:12:36 INFO     [api.task:312] Building contexts for minerva_math500 on rank 0...

  0%|          | 0/10 [00:00<?, ?it/s]
100%|██████████| 10/10 [00:00<00:00, 397.69it/s]
2026-07-23:22:12:36 INFO     [api.task:312] Building contexts for ifeval on rank 0...

  0%|          | 0/10 [00:00<?, ?it/s]
100%|██████████| 10/10 [00:00<00:00, 52363.35it/s]
2026-07-23:22:12:36 INFO     [api.task:312] Building contexts for gsm8k_cot_zeroshot on rank 0...

  0%|          | 0/10 [00:00<?, ?it/s]
100%|██████████| 10/10 [00:00<00:00, 1822.50it/s]
2026-07-23:22:12:37 INFO     [evaluator:585] Running generate_until requests
2026-07-23:22:12:37 INFO     [models.api_models:747] Tokenized requests are disabled. Context + generation length is not checked.

Requesting API:   0%|          | 0/140 [00:00<?, ?it/s]
Requesting API:   1%|          | 1/140 [00:03<09:04,  3.92s/it]
Requesting API:   3%|▎         | 4/140 [00:04<02:01,  1.11it/s]
Requesting API:   4%|▎         | 5/140 [00:04<01:38,  1.38it/s]
Requesting API:   4%|▍         | 6/140 [00:05<01:31,  1.47it/s]
Requesting API:   5%|▌         | 7/140 [00:05<01:15,  1.76it/s]
Requesting API:   6%|▋         | 9/140 [00:05<00:49,  2.67it/s]
Requesting API:   7%|▋         | 10/140 [00:06<00:54,  2.37it/s]
Requesting API:   9%|▊         | 12/140 [00:06<00:39,  3.26it/s]
Requesting API:  10%|█         | 14/140 [00:07<00:30,  4.07it/s]
Requesting API:  11%|█         | 15/140 [00:07<00:31,  3.93it/s]
Requesting API:  12%|█▏        | 17/140 [00:07<00:26,  4.72it/s]
Requesting API:  13%|█▎        | 18/140 [00:08<00:34,  3.49it/s]
Requesting API:  14%|█▎        | 19/140 [00:08<00:34,  3.49it/s]
Requesting API:  16%|█▌        | 22/140 [00:08<00:22,  5.24it/s]
Requesting API:  16%|█▋        | 23/140 [00:09<00:24,  4.80it/s]
Requesting API:  17%|█▋        | 24/140 [00:09<00:26,  4.44it/s]
Requesting API:  19%|█▉        | 27/140 [00:09<00:23,  4.82it/s]
Requesting API:  20%|██        | 28/140 [00:10<00:30,  3.65it/s]
Requesting API:  22%|██▏       | 31/140 [00:11<00:25,  4.23it/s]
Requesting API:  24%|██▎       | 33/140 [00:11<00:22,  4.80it/s]
Requesting API:  25%|██▌       | 35/140 [00:11<00:19,  5.29it/s]
Requesting API:  27%|██▋       | 38/140 [00:11<00:15,  6.56it/s]
Requesting API:  29%|██▉       | 41/140 [00:12<00:13,  7.61it/s]
Requesting API:  30%|███       | 42/140 [00:12<00:15,  6.51it/s]
Requesting API:  31%|███       | 43/140 [00:12<00:17,  5.68it/s]
Requesting API:  31%|███▏      | 44/140 [00:13<00:18,  5.08it/s]
Requesting API:  32%|███▏      | 45/140 [00:13<00:20,  4.64it/s]
Requesting API:  33%|███▎      | 46/140 [00:14<00:34,  2.72it/s]
Requesting API:  34%|███▎      | 47/140 [00:14<00:32,  2.89it/s]
Requesting API:  35%|███▌      | 49/140 [00:15<00:34,  2.61it/s]
Requesting API:  37%|███▋      | 52/140 [00:15<00:21,  4.05it/s]
Requesting API:  38%|███▊      | 53/140 [00:15<00:22,  3.93it/s]
Requesting API:  40%|████      | 56/140 [00:16<00:15,  5.42it/s]
Requesting API:  41%|████      | 57/140 [00:16<00:16,  4.95it/s]
Requesting API:  41%|████▏     | 58/140 [00:17<00:22,  3.62it/s]
Requesting API:  42%|████▏     | 59/140 [00:17<00:27,  2.92it/s]
Requesting API:  43%|████▎     | 60/140 [00:19<00:54,  1.48it/s]
Requesting API:  44%|████▎     | 61/140 [00:19<00:45,  1.73it/s]
Requesting API:  45%|████▌     | 63/140 [00:20<00:35,  2.19it/s]
Requesting API:  46%|████▋     | 65/140 [00:20<00:25,  2.94it/s]
Requesting API:  47%|████▋     | 66/140 [00:21<00:39,  1.89it/s]
Requesting API:  50%|█████     | 70/140 [00:21<00:18,  3.75it/s]
Requesting API:  51%|█████▏    | 72/140 [00:22<00:18,  3.65it/s]
Requesting API:  54%|█████▎    | 75/140 [00:22<00:13,  4.83it/s]
Requesting API:  54%|█████▍    | 76/140 [00:23<00:14,  4.55it/s]
Requesting API:  56%|█████▋    | 79/140 [00:24<00:17,  3.52it/s]
Requesting API:  57%|█████▋    | 80/140 [00:24<00:16,  3.60it/s]
Requesting API:  59%|█████▊    | 82/140 [00:25<00:16,  3.57it/s]
Requesting API:  59%|█████▉    | 83/140 [00:25<00:14,  3.93it/s]
Requesting API:  61%|██████    | 85/140 [00:25<00:11,  4.65it/s]
Requesting API:  62%|██████▏   | 87/140 [00:25<00:11,  4.69it/s]
Requesting API:  63%|██████▎   | 88/140 [00:26<00:13,  3.95it/s]
Requesting API:  64%|██████▍   | 90/140 [00:26<00:10,  4.68it/s]
Requesting API:  65%|██████▌   | 91/140 [00:27<00:12,  3.91it/s]
Requesting API:  66%|██████▋   | 93/140 [00:27<00:09,  4.95it/s]
Requesting API:  69%|██████▊   | 96/140 [00:27<00:07,  5.94it/s]
Requesting API:  69%|██████▉   | 97/140 [00:27<00:08,  5.29it/s]
Requesting API:  70%|███████   | 98/140 [00:28<00:11,  3.77it/s]
Requesting API:  71%|███████   | 99/140 [00:28<00:10,  3.88it/s]
Requesting API:  76%|███████▌  | 106/140 [00:29<00:04,  7.23it/s]
Requesting API:  76%|███████▋  | 107/140 [00:29<00:05,  6.37it/s]
Requesting API:  77%|███████▋  | 108/140 [00:29<00:05,  5.68it/s]
Requesting API:  78%|███████▊  | 109/140 [00:30<00:07,  4.06it/s]
Requesting API:  79%|███████▉  | 111/140 [00:30<00:05,  4.88it/s]
Requesting API:  81%|████████  | 113/140 [00:30<00:04,  5.94it/s]
Requesting API:  83%|████████▎ | 116/140 [00:31<00:03,  7.99it/s]
Requesting API:  85%|████████▌ | 119/140 [00:31<00:02,  9.76it/s]
Requesting API:  86%|████████▋ | 121/140 [00:31<00:01,  9.90it/s]
Requesting API:  88%|████████▊ | 123/140 [00:31<00:01, 10.12it/s]
Requesting API:  90%|█████████ | 126/140 [00:31<00:01, 10.35it/s]
Requesting API:  91%|█████████▏| 128/140 [00:32<00:01,  8.38it/s]
Requesting API:  92%|█████████▏| 129/140 [00:32<00:02,  5.34it/s]
Requesting API:  93%|█████████▎| 130/140 [00:33<00:02,  4.56it/s]
Requesting API:  94%|█████████▎| 131/140 [00:33<00:02,  4.30it/s]
Requesting API:  94%|█████████▍| 132/140 [00:33<00:01,  4.13it/s]
Requesting API:  95%|█████████▌| 133/140 [00:34<00:01,  4.01it/s]
Requesting API:  96%|█████████▋| 135/140 [00:34<00:01,  3.86it/s]
Requesting API:  97%|█████████▋| 136/140 [00:35<00:01,  2.05it/s]
Requesting API:  98%|█████████▊| 137/140 [00:37<00:01,  1.53it/s]
Requesting API:  99%|█████████▊| 138/140 [00:37<00:01,  1.73it/s]
Requesting API:  99%|█████████▉| 139/140 [00:40<00:01,  1.29s/it]
Requesting API: 100%|██████████| 140/140 [00:47<00:00,  2.95s/it]
Requesting API: 100%|██████████| 140/140 [00:47<00:00,  2.93it/s]

Requesting API:   0%|          | 0/10 [00:00<?, ?it/s]
Requesting API:  10%|█         | 1/10 [00:01<00:14,  1.66s/it]
Requesting API:  20%|██        | 2/10 [00:02<00:09,  1.21s/it]
Requesting API:  30%|███       | 3/10 [00:14<00:41,  5.89s/it]
Requesting API:  40%|████      | 4/10 [00:16<00:26,  4.48s/it]
Requesting API:  50%|█████     | 5/10 [00:22<00:24,  4.93s/it]
Requesting API:  60%|██████    | 6/10 [00:24<00:16,  4.18s/it]
Requesting API:  70%|███████   | 7/10 [01:54<01:36, 32.29s/it]
Requesting API: 100%|██████████| 10/10 [01:54<00:00, 11.49s/it]

Requesting API:   0%|          | 0/10 [00:00<?, ?it/s]
Requesting API:  10%|█         | 1/10 [00:00<00:02,  3.72it/s]
Requesting API:  20%|██        | 2/10 [00:00<00:03,  2.29it/s]
Requesting API:  40%|████      | 4/10 [00:01<00:01,  4.01it/s]
Requesting API:  60%|██████    | 6/10 [00:01<00:00,  5.10it/s]
Requesting API: 100%|██████████| 10/10 [00:01<00:00,  6.85it/s]

Requesting API:   0%|          | 0/10 [00:00<?, ?it/s]
Requesting API:  10%|█         | 1/10 [00:00<00:07,  1.16it/s]
Requesting API:  20%|██        | 2/10 [00:03<00:16,  2.10s/it]
Requesting API:  30%|███       | 3/10 [00:05<00:12,  1.72s/it]
Requesting API:  40%|████      | 4/10 [00:10<00:18,  3.10s/it]
Requesting API:  50%|█████     | 5/10 [00:20<00:28,  5.77s/it]
Requesting API:  60%|██████    | 6/10 [00:32<00:31,  7.91s/it]
Requesting API:  70%|███████   | 7/10 [01:55<01:36, 32.21s/it]
Requesting API: 100%|██████████| 10/10 [01:55<00:00, 11.51s/it]

Requesting API:   0%|          | 0/10 [00:00<?, ?it/s]
Requesting API:  10%|█         | 1/10 [00:00<00:04,  2.10it/s]
Requesting API:  40%|████      | 4/10 [00:01<00:01,  3.52it/s]
Requesting API:  60%|██████    | 6/10 [00:11<00:09,  2.49s/it]
Requesting API:  70%|███████   | 7/10 [00:12<00:05,  1.96s/it]
Requesting API:  80%|████████  | 8/10 [00:21<00:07,  3.92s/it]
Requesting API:  90%|█████████ | 9/10 [01:02<00:13, 13.71s/it]
Requesting API: 100%|██████████| 10/10 [01:53<00:00, 23.92s/it]
Requesting API: 100%|██████████| 10/10 [01:53<00:00, 11.34s/it]
2026-07-23:22:19:12 INFO     [loggers.evaluation_tracker:247] Saving results aggregated
2026-07-23:22:19:12 INFO     [loggers.evaluation_tracker:119] Saving per-task samples to outputs/evals/protocolF_gate3/student/*.jsonl
local-chat-completions ({'model': 'student', 'base_url': 'http://127.0.0.1:8399/v1/chat/completions', 'num_concurrent': 32, 'tokenized_requests': False, 'max_retries': 3}), gen_kwargs: ({'max_gen_toks': 1280}), limit: 10.0, num_fewshot: None, batch_size: 1
|          Tasks          |Version|     Filter     |n-shot|        Metric         |   |Value |   |Stderr|
|-------------------------|------:|----------------|-----:|-----------------------|---|-----:|---|------|
|mmlu_pro                 |    2.0|custom-extract  |     5|exact_match            |↑  |0.0571|±  |0.0190|
| - biology               |    3.1|custom-extract  |     5|exact_match            |↑  |0.0000|±  |     0|
| - business              |    3.1|custom-extract  |     5|exact_match            |↑  |0.3000|±  |0.1528|
| - chemistry             |    3.1|custom-extract  |     5|exact_match            |↑  |0.0000|±  |     0|
| - computer_science      |    3.1|custom-extract  |     5|exact_match            |↑  |0.1000|±  |0.1000|
| - economics             |    3.1|custom-extract  |     5|exact_match            |↑  |0.0000|±  |     0|
| - engineering           |    3.1|custom-extract  |     5|exact_match            |↑  |0.0000|±  |     0|
| - health                |    3.1|custom-extract  |     5|exact_match            |↑  |0.0000|±  |     0|
| - history               |    3.1|custom-extract  |     5|exact_match            |↑  |0.1000|±  |0.1000|
| - law                   |    3.1|custom-extract  |     5|exact_match            |↑  |0.1000|±  |0.1000|
| - math                  |    3.1|custom-extract  |     5|exact_match            |↑  |0.0000|±  |     0|
| - other                 |    3.1|custom-extract  |     5|exact_match            |↑  |0.2000|±  |0.1333|
| - philosophy            |    3.1|custom-extract  |     5|exact_match            |↑  |0.0000|±  |     0|
| - physics               |    3.1|custom-extract  |     5|exact_match            |↑  |0.0000|±  |     0|
| - psychology            |    3.1|custom-extract  |     5|exact_match            |↑  |0.0000|±  |     0|
|gpqa_diamond_cot_zeroshot|    2.2|flexible-extract|     0|exact_match            |↑  |0.3000|±  |0.1528|
|                         |       |strict-match    |     0|exact_match            |↑  |0.0000|±  |     0|
|gsm8k_cot_zeroshot       |    3.0|flexible-extract|     0|exact_match            |↑  |0.1000|±  |0.1000|
|                         |       |strict-match    |     0|exact_match            |↑  |0.0000|±  |     0|
|ifeval                   |    4.0|none            |     0|inst_level_loose_acc   |↑  |0.4444|±  |   N/A|
|                         |       |none            |     0|inst_level_strict_acc  |↑  |0.4444|±  |   N/A|
|                         |       |none            |     0|prompt_level_loose_acc |↑  |0.3000|±  |0.1528|
|                         |       |none            |     0|prompt_level_strict_acc|↑  |0.3000|±  |0.1528|
|minerva_math500          |    3.0|none            |     4|exact_match            |↑  |0.0000|±  |     0|
|                         |       |none            |     4|math_verify            |↑  |0.0000|±  |     0|

| Groups |Version|    Filter    |n-shot|  Metric   |   |Value |   |Stderr|
|--------|------:|--------------|-----:|-----------|---|-----:|---|-----:|
|mmlu_pro|      2|custom-extract|     5|exact_match|↑  |0.0571|±  | 0.019|