Hugging Face's logo

hbfreed
/
variable-reap-archive

Safetensors
Model card Files
xet
 Community
variable-reap-archive / evals /general_suite /math_fix.log
hbfreed's picture
hbfreed
Add files using upload-large-folder tool
c7b14e3 verified
Raw
History Blame Contribute Delete
8.49 kB
2026-07-17T11:15:57-07:00 serving allenai/OLMoE-1B-7B-0125-Instruct on GPU 0 port 8399
2026-07-17T11:15:57-07:00 waiting for server /health ...
2026-07-17T11:16:27-07:00 server up; chat pass [minerva_math500]
2026-07-17:11:16:27 WARNING [config.evaluate_config:287] --limit SHOULD ONLY BE USED FOR TESTING. REAL METRICS SHOULD NOT BE COMPUTED USING LIMIT.
2026-07-17:11:16:34 INFO [_cli.run:388] Selected Tasks: ['minerva_math500']
2026-07-17:11:16:35 INFO [evaluator:214] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2026-07-17:11:16:35 WARNING [evaluator:226] generation_kwargs: {'max_gen_toks': 1280} specified through cli, these settings will update set parameters in yaml tasks. Ensure 'do_sample=True' for non-greedy decoding!
2026-07-17:11:16:35 INFO [evaluator:239] Initializing local-chat-completions model, with arguments: {'model': 'student', 'base_url': 'http://127.0.0.1:8399/v1/chat/completions', 'num_concurrent': 32, 'tokenized_requests': False, 'max_retries': 3}
2026-07-17:11:16:35 INFO [models.api_models:179] Using max length 2048 - 1
2026-07-17:11:16:35 INFO [models.api_models:200] Using tokenizer None
2026-07-17:11:16:37 INFO [evaluator_utils:446] Selected tasks:
2026-07-17:11:16:37 INFO [evaluator_utils:480] Task: minerva_math500 (minerva_math/minerva_math500.yaml)
2026-07-17:11:16:37 INFO [evaluator:314] minerva_math500: Using gen_kwargs: {'until': ['Problem:'], 'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 1280}
2026-07-17:11:16:37 INFO [api.task:312] Building contexts for minerva_math500 on rank 0...
 0%| | 0/100 [00:00<?, ?it/s] 40%|████ | 40/100 [00:00<00:00, 396.25it/s] 82%|████████▏ | 82/100 [00:00<00:00, 409.76it/s] 100%|██████████| 100/100 [00:00<00:00, 410.31it/s]
2026-07-17:11:16:37 INFO [evaluator:585] Running generate_until requests
2026-07-17:11:16:37 INFO [models.api_models:747] Tokenized requests are disabled. Context + generation length is not checked.
 Requesting API: 0%| | 0/100 [00:00<?, ?it/s] Requesting API: 1%| | 1/100 [00:02<03:34, 2.16s/it] Requesting API: 2%|▏ | 2/100 [00:03<02:21, 1.45s/it] Requesting API: 3%|▎ | 3/100 [00:03<01:37, 1.00s/it] Requesting API: 4%|▍ | 4/100 [00:04<01:21, 1.18it/s] Requesting API: 5%|▌ | 5/100 [00:04<00:55, 1.71it/s] Requesting API: 6%|▌ | 6/100 [00:04<00:41, 2.29it/s] Requesting API: 8%|▊ | 8/100 [00:04<00:31, 2.93it/s] Requesting API: 9%|▉ | 9/100 [00:05<00:40, 2.26it/s] Requesting API: 11%|█ | 11/100 [00:06<00:32, 2.78it/s] Requesting API: 12%|█▏ | 12/100 [00:06<00:26, 3.27it/s] Requesting API: 13%|█▎ | 13/100 [00:07<00:37, 2.32it/s] Requesting API: 15%|█▌ | 15/100 [00:07<00:29, 2.90it/s] Requesting API: 17%|█▋ | 17/100 [00:07<00:22, 3.71it/s] Requesting API: 18%|█▊ | 18/100 [00:07<00:19, 4.26it/s] Requesting API: 19%|█▉ | 19/100 [00:08<00:17, 4.76it/s] Requesting API: 20%|██ | 20/100 [00:08<00:18, 4.32it/s] Requesting API: 22%|██▏ | 22/100 [00:08<00:19, 4.09it/s] Requesting API: 23%|██▎ | 23/100 [00:09<00:19, 3.90it/s] Requesting API: 24%|██▍ | 24/100 [00:09<00:20, 3.69it/s] Requesting API: 25%|██▌ | 25/100 [00:10<00:38, 1.93it/s] Requesting API: 26%|██▌ | 26/100 [00:10<00:30, 2.46it/s] Requesting API: 28%|██▊ | 28/100 [00:10<00:18, 3.88it/s] Requesting API: 30%|███ | 30/100 [00:11<00:15, 4.45it/s] Requesting API: 31%|███ | 31/100 [00:12<00:25, 2.68it/s] Requesting API: 32%|███▏ | 32/100 [00:13<00:44, 1.54it/s] Requesting API: 33%|███▎ | 33/100 [00:13<00:35, 1.90it/s] Requesting API: 35%|███▌ | 35/100 [00:14<00:22, 2.94it/s] Requesting API: 37%|███▋ | 37/100 [00:14<00:16, 3.87it/s] Requesting API: 38%|███▊ | 38/100 [00:14<00:16, 3.78it/s] Requesting API: 39%|███▉ | 39/100 [00:14<00:17, 3.59it/s] Requesting API: 43%|████▎ | 43/100 [00:15<00:07, 7.19it/s] Requesting API: 45%|████▌ | 45/100 [00:15<00:09, 5.55it/s] Requesting API: 46%|████▌ | 46/100 [00:16<00:11, 4.65it/s] Requesting API: 47%|████▋ | 47/100 [00:16<00:11, 4.65it/s] Requesting API: 48%|████▊ | 48/100 [00:16<00:10, 5.08it/s] Requesting API: 49%|████▉ | 49/100 [00:16<00:10, 4.87it/s] Requesting API: 50%|█████ | 50/100 [00:17<00:24, 2.04it/s] Requesting API: 51%|█████ | 51/100 [00:19<00:31, 1.55it/s] Requesting API: 52%|█████▏ | 52/100 [00:19<00:28, 1.70it/s] Requesting API: 53%|█████▎ | 53/100 [00:19<00:26, 1.76it/s] Requesting API: 54%|█████▍ | 54/100 [00:20<00:21, 2.19it/s] Requesting API: 57%|█████▋ | 57/100 [00:20<00:13, 3.14it/s] Requesting API: 58%|█████▊ | 58/100 [00:20<00:11, 3.62it/s] Requesting API: 59%|█████▉ | 59/100 [00:21<00:13, 3.09it/s] Requesting API: 61%|██████ | 61/100 [00:21<00:08, 4.59it/s] Requesting API: 62%|██████▏ | 62/100 [00:21<00:07, 4.99it/s] Requesting API: 63%|██████▎ | 63/100 [00:22<00:15, 2.38it/s] Requesting API: 64%|██████▍ | 64/100 [00:23<00:21, 1.65it/s] Requesting API: 67%|██████▋ | 67/100 [00:25<00:16, 1.98it/s] Requesting API: 68%|██████▊ | 68/100 [00:25<00:13, 2.35it/s] Requesting API: 70%|███████ | 70/100 [00:26<00:13, 2.20it/s] Requesting API: 72%|███████▏ | 72/100 [00:27<00:12, 2.30it/s] Requesting API: 73%|███████▎ | 73/100 [00:27<00:11, 2.27it/s] Requesting API: 74%|███████▍ | 74/100 [00:27<00:10, 2.60it/s] Requesting API: 78%|███████▊ | 78/100 [00:27<00:04, 5.15it/s] Requesting API: 80%|████████ | 80/100 [00:28<00:03, 5.37it/s] Requesting API: 82%|████████▏ | 82/100 [00:28<00:03, 5.72it/s] Requesting API: 83%|████████▎ | 83/100 [00:30<00:07, 2.19it/s] Requesting API: 84%|████████▍ | 84/100 [00:30<00:07, 2.15it/s] Requesting API: 85%|████████▌ | 85/100 [00:30<00:06, 2.48it/s] Requesting API: 86%|████████▌ | 86/100 [00:31<00:05, 2.66it/s] Requesting API: 90%|█████████ | 90/100 [00:31<00:02, 4.35it/s] Requesting API: 91%|█████████ | 91/100 [00:33<00:04, 1.91it/s] Requesting API: 93%|█████████▎| 93/100 [00:34<00:03, 2.15it/s] Requesting API: 94%|█████████▍| 94/100 [00:35<00:03, 1.58it/s] Requesting API: 95%|█████████▌| 95/100 [00:36<00:02, 1.74it/s] Requesting API: 96%|█████████▌| 96/100 [00:38<00:04, 1.02s/it] Requesting API: 97%|█████████▋| 97/100 [00:39<00:02, 1.04it/s] Requesting API: 98%|█████████▊| 98/100 [00:43<00:03, 1.69s/it] Requesting API: 99%|█████████▉| 99/100 [00:44<00:01, 1.62s/it] Requesting API: 100%|██████████| 100/100 [00:45<00:00, 1.40s/it] Requesting API: 100%|██████████| 100/100 [00:45<00:00, 2.20it/s]
2026-07-17:11:17:27 INFO [loggers.evaluation_tracker:247] Saving results aggregated
2026-07-17:11:17:27 INFO [loggers.evaluation_tracker:119] Saving per-task samples to outputs/evals/general_suite/math_fix/student/*.jsonl
local-chat-completions ({'model': 'student', 'base_url': 'http://127.0.0.1:8399/v1/chat/completions', 'num_concurrent': 32, 'tokenized_requests': False, 'max_retries': 3}), gen_kwargs: ({'max_gen_toks': 1280}), limit: 100.0, num_fewshot: None, batch_size: 1
| Tasks |Version|Filter|n-shot| Metric | |Value| |Stderr|
|---------------|------:|------|-----:|-----------|---|----:|---|-----:|
|minerva_math500| 3|none | 4|exact_match|↑ | 0.17|± |0.0378|
| | |none | 4|math_verify|↑ | 0.33|± |0.0473|
2026-07-17T11:17:28-07:00 lm_eval exit=0 -> outputs/evals/general_suite/math_fix