2026-07-17T11:15:57-07:00 serving allenai/OLMoE-1B-7B-0125-Instruct on GPU 0 port 8399
2026-07-17T11:15:57-07:00 waiting for server /health ...
2026-07-17T11:16:27-07:00 server up; chat pass [minerva_math500]
2026-07-17:11:16:27 WARNING  [config.evaluate_config:287] --limit SHOULD ONLY BE USED FOR TESTING. REAL METRICS SHOULD NOT BE COMPUTED USING LIMIT.
2026-07-17:11:16:34 INFO     [_cli.run:388] Selected Tasks: ['minerva_math500']
2026-07-17:11:16:35 INFO     [evaluator:214] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2026-07-17:11:16:35 WARNING  [evaluator:226] generation_kwargs: {'max_gen_toks': 1280} specified through cli, these settings will update set parameters in yaml tasks. Ensure 'do_sample=True' for non-greedy decoding!
2026-07-17:11:16:35 INFO     [evaluator:239] Initializing local-chat-completions model, with arguments: {'model': 'student', 'base_url': 'http://127.0.0.1:8399/v1/chat/completions', 'num_concurrent': 32, 'tokenized_requests': False, 'max_retries': 3}
2026-07-17:11:16:35 INFO     [models.api_models:179] Using max length 2048 - 1
2026-07-17:11:16:35 INFO     [models.api_models:200] Using tokenizer None
2026-07-17:11:16:37 INFO     [evaluator_utils:446] Selected tasks:
2026-07-17:11:16:37 INFO     [evaluator_utils:480] Task: minerva_math500 (minerva_math/minerva_math500.yaml)
2026-07-17:11:16:37 INFO     [evaluator:314] minerva_math500: Using gen_kwargs: {'until': ['Problem:'], 'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 1280}
2026-07-17:11:16:37 INFO     [api.task:312] Building contexts for minerva_math500 on rank 0...

  0%|          | 0/100 [00:00<?, ?it/s]
 40%|████      | 40/100 [00:00<00:00, 396.25it/s]
 82%|████████▏ | 82/100 [00:00<00:00, 409.76it/s]
100%|██████████| 100/100 [00:00<00:00, 410.31it/s]
2026-07-17:11:16:37 INFO     [evaluator:585] Running generate_until requests
2026-07-17:11:16:37 INFO     [models.api_models:747] Tokenized requests are disabled. Context + generation length is not checked.

Requesting API:   0%|          | 0/100 [00:00<?, ?it/s]
Requesting API:   1%|          | 1/100 [00:02<03:34,  2.16s/it]
Requesting API:   2%|▏         | 2/100 [00:03<02:21,  1.45s/it]
Requesting API:   3%|▎         | 3/100 [00:03<01:37,  1.00s/it]
Requesting API:   4%|▍         | 4/100 [00:04<01:21,  1.18it/s]
Requesting API:   5%|▌         | 5/100 [00:04<00:55,  1.71it/s]
Requesting API:   6%|▌         | 6/100 [00:04<00:41,  2.29it/s]
Requesting API:   8%|▊         | 8/100 [00:04<00:31,  2.93it/s]
Requesting API:   9%|▉         | 9/100 [00:05<00:40,  2.26it/s]
Requesting API:  11%|█         | 11/100 [00:06<00:32,  2.78it/s]
Requesting API:  12%|█▏        | 12/100 [00:06<00:26,  3.27it/s]
Requesting API:  13%|█▎        | 13/100 [00:07<00:37,  2.32it/s]
Requesting API:  15%|█▌        | 15/100 [00:07<00:29,  2.90it/s]
Requesting API:  17%|█▋        | 17/100 [00:07<00:22,  3.71it/s]
Requesting API:  18%|█▊        | 18/100 [00:07<00:19,  4.26it/s]
Requesting API:  19%|█▉        | 19/100 [00:08<00:17,  4.76it/s]
Requesting API:  20%|██        | 20/100 [00:08<00:18,  4.32it/s]
Requesting API:  22%|██▏       | 22/100 [00:08<00:19,  4.09it/s]
Requesting API:  23%|██▎       | 23/100 [00:09<00:19,  3.90it/s]
Requesting API:  24%|██▍       | 24/100 [00:09<00:20,  3.69it/s]
Requesting API:  25%|██▌       | 25/100 [00:10<00:38,  1.93it/s]
Requesting API:  26%|██▌       | 26/100 [00:10<00:30,  2.46it/s]
Requesting API:  28%|██▊       | 28/100 [00:10<00:18,  3.88it/s]
Requesting API:  30%|███       | 30/100 [00:11<00:15,  4.45it/s]
Requesting API:  31%|███       | 31/100 [00:12<00:25,  2.68it/s]
Requesting API:  32%|███▏      | 32/100 [00:13<00:44,  1.54it/s]
Requesting API:  33%|███▎      | 33/100 [00:13<00:35,  1.90it/s]
Requesting API:  35%|███▌      | 35/100 [00:14<00:22,  2.94it/s]
Requesting API:  37%|███▋      | 37/100 [00:14<00:16,  3.87it/s]
Requesting API:  38%|███▊      | 38/100 [00:14<00:16,  3.78it/s]
Requesting API:  39%|███▉      | 39/100 [00:14<00:17,  3.59it/s]
Requesting API:  43%|████▎     | 43/100 [00:15<00:07,  7.19it/s]
Requesting API:  45%|████▌     | 45/100 [00:15<00:09,  5.55it/s]
Requesting API:  46%|████▌     | 46/100 [00:16<00:11,  4.65it/s]
Requesting API:  47%|████▋     | 47/100 [00:16<00:11,  4.65it/s]
Requesting API:  48%|████▊     | 48/100 [00:16<00:10,  5.08it/s]
Requesting API:  49%|████▉     | 49/100 [00:16<00:10,  4.87it/s]
Requesting API:  50%|█████     | 50/100 [00:17<00:24,  2.04it/s]
Requesting API:  51%|█████     | 51/100 [00:19<00:31,  1.55it/s]
Requesting API:  52%|█████▏    | 52/100 [00:19<00:28,  1.70it/s]
Requesting API:  53%|█████▎    | 53/100 [00:19<00:26,  1.76it/s]
Requesting API:  54%|█████▍    | 54/100 [00:20<00:21,  2.19it/s]
Requesting API:  57%|█████▋    | 57/100 [00:20<00:13,  3.14it/s]
Requesting API:  58%|█████▊    | 58/100 [00:20<00:11,  3.62it/s]
Requesting API:  59%|█████▉    | 59/100 [00:21<00:13,  3.09it/s]
Requesting API:  61%|██████    | 61/100 [00:21<00:08,  4.59it/s]
Requesting API:  62%|██████▏   | 62/100 [00:21<00:07,  4.99it/s]
Requesting API:  63%|██████▎   | 63/100 [00:22<00:15,  2.38it/s]
Requesting API:  64%|██████▍   | 64/100 [00:23<00:21,  1.65it/s]
Requesting API:  67%|██████▋   | 67/100 [00:25<00:16,  1.98it/s]
Requesting API:  68%|██████▊   | 68/100 [00:25<00:13,  2.35it/s]
Requesting API:  70%|███████   | 70/100 [00:26<00:13,  2.20it/s]
Requesting API:  72%|███████▏  | 72/100 [00:27<00:12,  2.30it/s]
Requesting API:  73%|███████▎  | 73/100 [00:27<00:11,  2.27it/s]
Requesting API:  74%|███████▍  | 74/100 [00:27<00:10,  2.60it/s]
Requesting API:  78%|███████▊  | 78/100 [00:27<00:04,  5.15it/s]
Requesting API:  80%|████████  | 80/100 [00:28<00:03,  5.37it/s]
Requesting API:  82%|████████▏ | 82/100 [00:28<00:03,  5.72it/s]
Requesting API:  83%|████████▎ | 83/100 [00:30<00:07,  2.19it/s]
Requesting API:  84%|████████▍ | 84/100 [00:30<00:07,  2.15it/s]
Requesting API:  85%|████████▌ | 85/100 [00:30<00:06,  2.48it/s]
Requesting API:  86%|████████▌ | 86/100 [00:31<00:05,  2.66it/s]
Requesting API:  90%|█████████ | 90/100 [00:31<00:02,  4.35it/s]
Requesting API:  91%|█████████ | 91/100 [00:33<00:04,  1.91it/s]
Requesting API:  93%|█████████▎| 93/100 [00:34<00:03,  2.15it/s]
Requesting API:  94%|█████████▍| 94/100 [00:35<00:03,  1.58it/s]
Requesting API:  95%|█████████▌| 95/100 [00:36<00:02,  1.74it/s]
Requesting API:  96%|█████████▌| 96/100 [00:38<00:04,  1.02s/it]
Requesting API:  97%|█████████▋| 97/100 [00:39<00:02,  1.04it/s]
Requesting API:  98%|█████████▊| 98/100 [00:43<00:03,  1.69s/it]
Requesting API:  99%|█████████▉| 99/100 [00:44<00:01,  1.62s/it]
Requesting API: 100%|██████████| 100/100 [00:45<00:00,  1.40s/it]
Requesting API: 100%|██████████| 100/100 [00:45<00:00,  2.20it/s]
2026-07-17:11:17:27 INFO     [loggers.evaluation_tracker:247] Saving results aggregated
2026-07-17:11:17:27 INFO     [loggers.evaluation_tracker:119] Saving per-task samples to outputs/evals/general_suite/math_fix/student/*.jsonl
local-chat-completions ({'model': 'student', 'base_url': 'http://127.0.0.1:8399/v1/chat/completions', 'num_concurrent': 32, 'tokenized_requests': False, 'max_retries': 3}), gen_kwargs: ({'max_gen_toks': 1280}), limit: 100.0, num_fewshot: None, batch_size: 1
|     Tasks     |Version|Filter|n-shot|  Metric   |   |Value|   |Stderr|
|---------------|------:|------|-----:|-----------|---|----:|---|-----:|
|minerva_math500|      3|none  |     4|exact_match|↑  | 0.17|±  |0.0378|
|               |       |none  |     4|math_verify|↑  | 0.33|±  |0.0473|

2026-07-17T11:17:28-07:00 lm_eval exit=0 -> outputs/evals/general_suite/math_fix