File size: 3,743 Bytes
8717f59 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 | 2026-07-23:21:59:00 WARNING [config.evaluate_config:287] --limit SHOULD ONLY BE USED FOR TESTING. REAL METRICS SHOULD NOT BE COMPUTED USING LIMIT.
2026-07-23:21:59:07 INFO [_cli.run:388] Selected Tasks: ['humaneval', 'mbpp']
2026-07-23:21:59:08 INFO [evaluator:214] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2026-07-23:21:59:08 INFO [evaluator:239] Initializing local-completions model, with arguments: {'model': 'student', 'base_url': 'http://127.0.0.1:8399/v1/completions', 'tokenizer': 'outputs/qwen35_pruned_keep25', 'num_concurrent': 32, 'tokenized_requests': False, 'max_retries': 3}
2026-07-23:21:59:08 INFO [models.openai_completions:42] Remote tokenizer not supported. Using huggingface tokenizer backend.
2026-07-23:21:59:08 INFO [models.api_models:179] Using max length 2048 - 1
2026-07-23:21:59:08 INFO [models.api_models:200] Using tokenizer huggingface
2026-07-23:21:59:17 INFO [evaluator_utils:446] Selected tasks:
2026-07-23:21:59:17 INFO [evaluator_utils:480] Task: humaneval (humaneval/humaneval.yaml)
2026-07-23:21:59:17 INFO [evaluator_utils:480] Task: mbpp (mbpp/mbpp.yaml)
2026-07-23:21:59:17 INFO [evaluator:314] humaneval: Using gen_kwargs: {'until': ['\nclass', '\ndef', '\n#', '\nif', '\nprint'], 'max_gen_toks': 1024, 'do_sample': False}
2026-07-23:21:59:17 INFO [evaluator:314] mbpp: Using gen_kwargs: {'until': ['[DONE]'], 'do_sample': False}
2026-07-23:21:59:17 INFO [api.task:312] Building contexts for humaneval on rank 0...
0%| | 0/10 [00:00<?, ?it/s]
100%|██████████| 10/10 [00:00<00:00, 667.50it/s]
2026-07-23:21:59:17 INFO [api.task:312] Building contexts for mbpp on rank 0...
0%| | 0/10 [00:00<?, ?it/s]
100%|██████████| 10/10 [00:00<00:00, 135.02it/s]
2026-07-23:21:59:17 INFO [evaluator:585] Running generate_until requests
2026-07-23:21:59:17 INFO [models.api_models:747] Tokenized requests are disabled. Context + generation length is not checked.
Requesting API: 0%| | 0/10 [00:00<?, ?it/s]
Requesting API: 10%|█ | 1/10 [01:31<13:44, 91.61s/it]
Requesting API: 100%|██████████| 10/10 [01:31<00:00, 9.16s/it]
Requesting API: 0%| | 0/10 [00:00<?, ?it/s]
Requesting API: 10%|█ | 1/10 [00:03<00:28, 3.20s/it]
Requesting API: 20%|██ | 2/10 [00:03<00:11, 1.42s/it]
Requesting API: 40%|████ | 4/10 [00:03<00:03, 1.70it/s]
Requesting API: 50%|█████ | 5/10 [00:05<00:05, 1.07s/it]
Requesting API: 60%|██████ | 6/10 [00:05<00:03, 1.26it/s]
Requesting API: 90%|█████████ | 9/10 [00:06<00:00, 2.52it/s]
Requesting API: 100%|██████████| 10/10 [00:23<00:00, 3.94s/it]
Requesting API: 100%|██████████| 10/10 [00:23<00:00, 2.32s/it]
2026-07-23:22:01:17 INFO [loggers.evaluation_tracker:247] Saving results aggregated
2026-07-23:22:01:17 INFO [loggers.evaluation_tracker:119] Saving per-task samples to outputs/evals/protocolF_gate/student/*.jsonl
local-completions ({'model': 'student', 'base_url': 'http://127.0.0.1:8399/v1/completions', 'tokenizer': 'outputs/qwen35_pruned_keep25', 'num_concurrent': 32, 'tokenized_requests': False, 'max_retries': 3}), gen_kwargs: ({}), limit: 10.0, num_fewshot: None, batch_size: 1
| Tasks |Version| Filter |n-shot| Metric | |Value| |Stderr|
|---------|------:|-----------|-----:|---------|---|----:|---|-----:|
|humaneval| 1|create_test| 0|pass@1 |↑ | 0|± | 0|
|mbpp | 1|none | 3|pass_at_1|↑ | 0|± | 0|
|