[MMLU57-TOKEN] start Sun Aug  9 23:10:07 UTC 2026 token=hf_oUMhDnr...
[*] MicroLLM2 MMLU — model: /home/zeus/microllm2/microllm2-checkpoints/final_merged  shots=5  limit=20
[*] GPT2-XL 1.5B 1024ctx  vocab=50259 (ChatML) — MMLU via direct eval (no harness needed)
[*] lm-eval not installed — using lightweight direct MMLU eval (same logic, no harness)
[*] Install harness for official numbers: pip install lm-eval==0.4.4
[*] Loading tokenizer + model /home/zeus/microllm2/microllm2-checkpoints/final_merged ...
[+] Loaded on cuda:0 dtype=torch.bfloat16 — starting MMLU

MMLU subjects:   0%|          | 0/57 [00:00<?, ?it/s]
============================================================
[>] abstract_algebra  (shots=5)


abstract_algebra:   0%|          | 0/20 [00:00<?, ?it/s]

abstract_algebra:   5%|▌         | 1/20 [00:01<00:21,  1.15s/it]

abstract_algebra:  15%|█▌        | 3/20 [00:01<00:05,  2.92it/s]

abstract_algebra:  25%|██▌       | 5/20 [00:01<00:02,  5.08it/s]

abstract_algebra:  40%|████      | 8/20 [00:01<00:01,  8.67it/s]

abstract_algebra:  50%|█████     | 10/20 [00:01<00:00, 10.56it/s]

abstract_algebra:  70%|███████   | 14/20 [00:01<00:00, 15.27it/s]

abstract_algebra:  85%|████████▌ | 17/20 [00:01<00:00, 17.18it/s]

abstract_algebra: 100%|██████████| 20/20 [00:02<00:00, 17.57it/s]

                                                                 
MMLU subjects:   2%|▏         | 1/57 [00:03<03:29,  3.73s/it][=] abstract_algebra: 6/20 = 30.0%  (running avg 30.0%)

============================================================
[>] anatomy  (shots=5)


anatomy:   0%|          | 0/20 [00:00<?, ?it/s]

anatomy:  10%|█         | 2/20 [00:00<00:01, 13.20it/s]

anatomy:  30%|███       | 6/20 [00:00<00:00, 25.74it/s]

anatomy:  45%|████▌     | 9/20 [00:00<00:00, 24.91it/s]

anatomy:  60%|██████    | 12/20 [00:00<00:00, 24.58it/s]

anatomy:  75%|███████▌  | 15/20 [00:00<00:00, 23.82it/s]

anatomy:  90%|█████████ | 18/20 [00:00<00:00, 21.69it/s]

                                                        
MMLU subjects:   4%|▎         | 2/57 [00:06<02:43,  2.97s/it][=] anatomy: 5/20 = 25.0%  (running avg 27.5%)

============================================================
[>] astronomy  (shots=5)


astronomy:   0%|          | 0/20 [00:00<?, ?it/s]

astronomy:  10%|█         | 2/20 [00:00<00:01, 12.23it/s]

astronomy:  20%|██        | 4/20 [00:00<00:01, 14.44it/s]

astronomy:  30%|███       | 6/20 [00:00<00:00, 15.75it/s]

astronomy:  40%|████      | 8/20 [00:00<00:00, 16.45it/s]

astronomy:  50%|█████     | 10/20 [00:00<00:00, 16.60it/s]

astronomy:  60%|██████    | 12/20 [00:00<00:00, 17.09it/s]

astronomy:  70%|███████   | 14/20 [00:00<00:00, 17.48it/s]

astronomy:  85%|████████▌ | 17/20 [00:00<00:00, 19.61it/s]

astronomy:  95%|█████████▌| 19/20 [00:01<00:00, 19.15it/s]

                                                          
MMLU subjects:   5%|▌         | 3/57 [00:08<02:23,  2.66s/it][=] astronomy: 7/20 = 35.0%  (running avg 30.0%)

============================================================
[>] business_ethics  (shots=5)


business_ethics:   0%|          | 0/20 [00:00<?, ?it/s]

business_ethics:  10%|█         | 2/20 [00:00<00:01, 12.90it/s]

business_ethics:  20%|██        | 4/20 [00:00<00:01, 15.43it/s]

business_ethics:  40%|████      | 8/20 [00:00<00:00, 24.39it/s]

business_ethics:  55%|█████▌    | 11/20 [00:00<00:00, 24.16it/s]

business_ethics:  70%|███████   | 14/20 [00:00<00:00, 23.61it/s]

business_ethics:  85%|████████▌ | 17/20 [00:00<00:00, 23.27it/s]

business_ethics: 100%|██████████| 20/20 [00:00<00:00, 22.96it/s]

                                                                
MMLU subjects:   7%|▋         | 4/57 [00:10<02:10,  2.45s/it][=] business_ethics: 6/20 = 30.0%  (running avg 30.0%)

============================================================
[>] clinical_knowledge  (shots=5)


clinical_knowledge:   0%|          | 0/20 [00:00<?, ?it/s]

clinical_knowledge:  10%|█         | 2/20 [00:00<00:00, 18.14it/s]

clinical_knowledge:  20%|██        | 4/20 [00:00<00:00, 18.00it/s]

clinical_knowledge:  30%|███       | 6/20 [00:00<00:00, 17.94it/s]

clinical_knowledge:  40%|████      | 8/20 [00:00<00:00, 17.50it/s]

clinical_knowledge:  50%|█████     | 10/20 [00:00<00:00, 17.77it/s]

clinical_knowledge:  60%|██████    | 12/20 [00:00<00:00, 17.75it/s]

clinical_knowledge:  75%|███████▌  | 15/20 [00:00<00:00, 19.69it/s]

clinical_knowledge:  90%|█████████ | 18/20 [00:00<00:00, 20.96it/s]

                                                                   
MMLU subjects:   9%|▉         | 5/57 [00:12<02:05,  2.42s/it][=] clinical_knowledge: 9/20 = 45.0%  (running avg 33.0%)

============================================================
[>] college_biology  (shots=5)


college_biology:   0%|          | 0/20 [00:00<?, ?it/s]

college_biology:  10%|█         | 2/20 [00:00<00:01, 13.18it/s]

college_biology:  20%|██        | 4/20 [00:00<00:01, 15.56it/s]

college_biology:  35%|███▌      | 7/20 [00:00<00:00, 19.08it/s]

college_biology:  45%|████▌     | 9/20 [00:00<00:00, 18.60it/s]

college_biology:  65%|██████▌   | 13/20 [00:00<00:00, 21.91it/s]

college_biology:  80%|████████  | 16/20 [00:00<00:00, 22.47it/s]

college_biology:  95%|█████████▌| 19/20 [00:00<00:00, 22.98it/s]

                                                                
MMLU subjects:  11%|█         | 6/57 [00:15<01:59,  2.35s/it][=] college_biology: 9/20 = 45.0%  (running avg 35.0%)

============================================================
[>] college_chemistry  (shots=5)


college_chemistry:   0%|          | 0/20 [00:00<?, ?it/s]

college_chemistry:  10%|█         | 2/20 [00:00<00:01, 12.84it/s]

college_chemistry:  25%|██▌       | 5/20 [00:00<00:00, 18.26it/s]

college_chemistry:  40%|████      | 8/20 [00:00<00:00, 20.43it/s]

college_chemistry:  55%|█████▌    | 11/20 [00:00<00:00, 19.37it/s]

college_chemistry:  65%|██████▌   | 13/20 [00:00<00:00, 18.98it/s]

college_chemistry:  85%|████████▌ | 17/20 [00:00<00:00, 21.96it/s]

                                                                  
MMLU subjects:  12%|█▏        | 7/57 [00:17<01:56,  2.33s/it][=] college_chemistry: 3/20 = 15.0%  (running avg 32.1%)

============================================================
[>] college_computer_science  (shots=5)


college_computer_science:   0%|          | 0/20 [00:00<?, ?it/s]

college_computer_science:   5%|▌         | 1/20 [00:00<00:02,  9.28it/s]

college_computer_science:  15%|█▌        | 3/20 [00:00<00:01, 13.08it/s]

college_computer_science:  25%|██▌       | 5/20 [00:00<00:01, 14.86it/s]

college_computer_science:  35%|███▌      | 7/20 [00:00<00:00, 15.74it/s]

college_computer_science:  45%|████▌     | 9/20 [00:00<00:00, 16.53it/s]

college_computer_science:  55%|█████▌    | 11/20 [00:00<00:00, 17.01it/s]

college_computer_science:  65%|██████▌   | 13/20 [00:00<00:00, 17.24it/s]

college_computer_science:  75%|███████▌  | 15/20 [00:00<00:00, 17.33it/s]

college_computer_science:  85%|████████▌ | 17/20 [00:01<00:00, 17.21it/s]

college_computer_science:  95%|█████████▌| 19/20 [00:01<00:00, 17.23it/s]

                                                                         
MMLU subjects:  14%|█▍        | 8/57 [00:19<01:55,  2.37s/it][=] college_computer_science: 9/20 = 45.0%  (running avg 33.8%)

============================================================
[>] college_mathematics  (shots=5)


college_mathematics:   0%|          | 0/20 [00:00<?, ?it/s]

college_mathematics:  10%|█         | 2/20 [00:00<00:01, 12.83it/s]

college_mathematics:  20%|██        | 4/20 [00:00<00:01, 15.33it/s]

college_mathematics:  35%|███▌      | 7/20 [00:00<00:00, 18.80it/s]

college_mathematics:  60%|██████    | 12/20 [00:00<00:00, 26.76it/s]

college_mathematics:  80%|████████  | 16/20 [00:00<00:00, 26.72it/s]

college_mathematics:  95%|█████████▌| 19/20 [00:00<00:00, 23.14it/s]

                                                                    
MMLU subjects:  16%|█▌        | 9/57 [00:22<01:52,  2.34s/it][=] college_mathematics: 7/20 = 35.0%  (running avg 33.9%)

============================================================
[>] college_medicine  (shots=5)


college_medicine:   0%|          | 0/20 [00:00<?, ?it/s]

college_medicine:  10%|█         | 2/20 [00:00<00:01, 17.54it/s]

college_medicine:  20%|██        | 4/20 [00:00<00:00, 17.37it/s]

college_medicine:  30%|███       | 6/20 [00:00<00:00, 17.41it/s]

college_medicine:  45%|████▌     | 9/20 [00:00<00:00, 19.51it/s]

college_medicine:  55%|█████▌    | 11/20 [00:00<00:00, 18.71it/s]

college_medicine:  65%|██████▌   | 13/20 [00:00<00:00, 18.52it/s]

college_medicine:  75%|███████▌  | 15/20 [00:00<00:00, 18.49it/s]

college_medicine: 100%|██████████| 20/20 [00:00<00:00, 25.98it/s]

                                                                 
MMLU subjects:  18%|█▊        | 10/57 [00:24<01:47,  2.28s/it][=] college_medicine: 6/20 = 30.0%  (running avg 33.5%)

============================================================
[>] college_physics  (shots=5)


college_physics:   0%|          | 0/20 [00:00<?, ?it/s]

college_physics:  10%|█         | 2/20 [00:00<00:01, 12.81it/s]

college_physics:  20%|██        | 4/20 [00:00<00:01, 15.22it/s]

college_physics:  40%|████      | 8/20 [00:00<00:00, 21.09it/s]

college_physics:  60%|██████    | 12/20 [00:00<00:00, 23.72it/s]

college_physics:  80%|████████  | 16/20 [00:00<00:00, 27.98it/s]

college_physics:  95%|█████████▌| 19/20 [00:00<00:00, 26.64it/s]

                                                                
MMLU subjects:  19%|█▉        | 11/57 [00:26<01:42,  2.23s/it][=] college_physics: 3/20 = 15.0%  (running avg 31.8%)

============================================================
[>] computer_security  (shots=5)


computer_security:   0%|          | 0/20 [00:00<?, ?it/s]

computer_security:  10%|█         | 2/20 [00:00<00:00, 18.55it/s]

computer_security:  20%|██        | 4/20 [00:00<00:00, 18.25it/s]

computer_security:  35%|███▌      | 7/20 [00:00<00:00, 20.93it/s]

computer_security:  55%|█████▌    | 11/20 [00:00<00:00, 24.12it/s]

computer_security:  70%|███████   | 14/20 [00:00<00:00, 23.97it/s]

computer_security:  90%|█████████ | 18/20 [00:00<00:00, 28.34it/s]

                                                                  
MMLU subjects:  21%|██        | 12/57 [00:28<01:37,  2.17s/it][=] computer_security: 6/20 = 30.0%  (running avg 31.7%)

============================================================
[>] conceptual_physics  (shots=5)


conceptual_physics:   0%|          | 0/20 [00:00<?, ?it/s]

conceptual_physics:   5%|▌         | 1/20 [00:00<00:02,  9.44it/s]

conceptual_physics:  15%|█▌        | 3/20 [00:00<00:01, 14.31it/s]

conceptual_physics:  30%|███       | 6/20 [00:00<00:00, 18.53it/s]

conceptual_physics:  45%|████▌     | 9/20 [00:00<00:00, 20.38it/s]

conceptual_physics:  65%|██████▌   | 13/20 [00:00<00:00, 22.74it/s]

conceptual_physics:  85%|████████▌ | 17/20 [00:00<00:00, 24.44it/s]

conceptual_physics: 100%|██████████| 20/20 [00:00<00:00, 24.12it/s]

                                                                   
MMLU subjects:  23%|██▎       | 13/57 [00:30<01:37,  2.22s/it][=] conceptual_physics: 1/20 = 5.0%  (running avg 29.6%)

============================================================
[>] econometrics  (shots=5)


econometrics:   0%|          | 0/20 [00:00<?, ?it/s]

econometrics:  20%|██        | 4/20 [00:00<00:00, 27.82it/s]

econometrics:  35%|███▌      | 7/20 [00:00<00:00, 21.59it/s]

econometrics:  55%|█████▌    | 11/20 [00:00<00:00, 23.65it/s]

econometrics:  80%|████████  | 16/20 [00:00<00:00, 28.86it/s]

econometrics:  95%|█████████▌| 19/20 [00:00<00:00, 26.62it/s]

                                                             
MMLU subjects:  25%|██▍       | 14/57 [00:32<01:33,  2.18s/it][=] econometrics: 6/20 = 30.0%  (running avg 29.6%)

============================================================
[>] electrical_engineering  (shots=5)


electrical_engineering:   0%|          | 0/20 [00:00<?, ?it/s]

electrical_engineering:  10%|█         | 2/20 [00:00<00:01, 17.36it/s]

electrical_engineering:  30%|███       | 6/20 [00:00<00:00, 23.79it/s]

electrical_engineering:  45%|████▌     | 9/20 [00:00<00:00, 20.65it/s]

electrical_engineering:  65%|██████▌   | 13/20 [00:00<00:00, 25.92it/s]

electrical_engineering:  95%|█████████▌| 19/20 [00:00<00:00, 35.89it/s]

                                                                       
MMLU subjects:  26%|██▋       | 15/57 [00:34<01:30,  2.15s/it][=] electrical_engineering: 4/20 = 20.0%  (running avg 29.0%)

============================================================
[>] elementary_mathematics  (shots=5)


elementary_mathematics:   0%|          | 0/20 [00:00<?, ?it/s]

elementary_mathematics:  10%|█         | 2/20 [00:00<00:01, 12.90it/s]

elementary_mathematics:  20%|██        | 4/20 [00:00<00:01, 15.49it/s]

elementary_mathematics:  30%|███       | 6/20 [00:00<00:00, 16.54it/s]

elementary_mathematics:  50%|█████     | 10/20 [00:00<00:00, 23.85it/s]

elementary_mathematics:  70%|███████   | 14/20 [00:00<00:00, 28.23it/s]

elementary_mathematics:  85%|████████▌ | 17/20 [00:00<00:00, 26.39it/s]

                                                                       
MMLU subjects:  28%|██▊       | 16/57 [00:36<01:26,  2.11s/it][=] elementary_mathematics: 6/20 = 30.0%  (running avg 29.1%)

============================================================
[>] formal_logic  (shots=5)


formal_logic:   0%|          | 0/20 [00:00<?, ?it/s]

formal_logic:  10%|█         | 2/20 [00:00<00:01, 13.10it/s]

formal_logic:  20%|██        | 4/20 [00:00<00:01, 15.61it/s]

formal_logic:  30%|███       | 6/20 [00:00<00:00, 16.51it/s]

formal_logic:  45%|████▌     | 9/20 [00:00<00:00, 19.07it/s]

formal_logic:  65%|██████▌   | 13/20 [00:00<00:00, 22.24it/s]

formal_logic:  80%|████████  | 16/20 [00:00<00:00, 22.59it/s]

formal_logic:  95%|█████████▌| 19/20 [00:00<00:00, 22.94it/s]

                                                             
MMLU subjects:  30%|██▉       | 17/57 [00:39<01:24,  2.12s/it][=] formal_logic: 2/20 = 10.0%  (running avg 27.9%)

============================================================
[>] global_facts  (shots=5)


global_facts:   0%|          | 0/20 [00:00<?, ?it/s]

global_facts:  10%|█         | 2/20 [00:00<00:01, 17.09it/s]

global_facts:  30%|███       | 6/20 [00:00<00:00, 28.49it/s]

global_facts:  50%|█████     | 10/20 [00:00<00:00, 32.74it/s]

global_facts:  70%|███████   | 14/20 [00:00<00:00, 34.73it/s]

                                                             
MMLU subjects:  32%|███▏      | 18/57 [00:40<01:18,  2.00s/it][=] global_facts: 7/20 = 35.0%  (running avg 28.3%)

============================================================
[>] high_school_biology  (shots=5)


high_school_biology:   0%|          | 0/20 [00:00<?, ?it/s]

high_school_biology:  15%|█▌        | 3/20 [00:00<00:00, 24.19it/s]

high_school_biology:  35%|███▌      | 7/20 [00:00<00:00, 31.92it/s]

high_school_biology:  55%|█████▌    | 11/20 [00:00<00:00, 26.11it/s]

high_school_biology:  75%|███████▌  | 15/20 [00:00<00:00, 26.59it/s]

high_school_biology:  95%|█████████▌| 19/20 [00:00<00:00, 29.76it/s]

                                                                    
MMLU subjects:  33%|███▎      | 19/57 [00:43<01:17,  2.04s/it][=] high_school_biology: 9/20 = 45.0%  (running avg 29.2%)

============================================================
[>] high_school_chemistry  (shots=5)


high_school_chemistry:   0%|          | 0/20 [00:00<?, ?it/s]

high_school_chemistry:  10%|█         | 2/20 [00:00<00:01, 17.03it/s]

high_school_chemistry:  40%|████      | 8/20 [00:00<00:00, 40.46it/s]

high_school_chemistry:  65%|██████▌   | 13/20 [00:00<00:00, 40.38it/s]

high_school_chemistry:  90%|█████████ | 18/20 [00:00<00:00, 40.22it/s]

                                                                      
MMLU subjects:  35%|███▌      | 20/57 [00:44<01:13,  1.99s/it][=] high_school_chemistry: 7/20 = 35.0%  (running avg 29.5%)

============================================================
[>] high_school_computer_science  (shots=5)


high_school_computer_science:   0%|          | 0/20 [00:00<?, ?it/s]

high_school_computer_science:   5%|▌         | 1/20 [00:00<00:02,  8.41it/s]

high_school_computer_science:  25%|██▌       | 5/20 [00:00<00:00, 20.00it/s]

high_school_computer_science:  45%|████▌     | 9/20 [00:00<00:00, 26.45it/s]

high_school_computer_science:  60%|██████    | 12/20 [00:00<00:00, 22.36it/s]

high_school_computer_science:  80%|████████  | 16/20 [00:00<00:00, 23.75it/s]

high_school_computer_science:  95%|█████████▌| 19/20 [00:00<00:00, 23.32it/s]

                                                                             
MMLU subjects:  37%|███▋      | 21/57 [00:46<01:12,  2.01s/it][=] high_school_computer_science: 7/20 = 35.0%  (running avg 29.8%)

============================================================
[>] high_school_european_history  (shots=5)


high_school_european_history:   0%|          | 0/20 [00:00<?, ?it/s]

high_school_european_history:  15%|█▌        | 3/20 [00:00<00:00, 29.59it/s]

high_school_european_history:  40%|████      | 8/20 [00:00<00:00, 39.83it/s]

high_school_european_history:  65%|██████▌   | 13/20 [00:00<00:00, 42.99it/s]

high_school_european_history:  90%|█████████ | 18/20 [00:00<00:00, 44.36it/s]

                                                                             
MMLU subjects:  39%|███▊      | 22/57 [00:48<01:07,  1.94s/it][=] high_school_european_history: 4/20 = 20.0%  (running avg 29.3%)

============================================================
[>] high_school_geography  (shots=5)


high_school_geography:   0%|          | 0/20 [00:00<?, ?it/s]

high_school_geography:  15%|█▌        | 3/20 [00:00<00:00, 22.09it/s]

high_school_geography:  45%|████▌     | 9/20 [00:00<00:00, 34.56it/s]

high_school_geography:  65%|██████▌   | 13/20 [00:00<00:00, 30.54it/s]

high_school_geography:  85%|████████▌ | 17/20 [00:00<00:00, 32.49it/s]

                                                                      
MMLU subjects:  40%|████      | 23/57 [00:50<01:05,  1.91s/it][=] high_school_geography: 5/20 = 25.0%  (running avg 29.1%)

============================================================
[>] high_school_government_and_politics  (shots=5)


high_school_government_and_politics:   0%|          | 0/20 [00:00<?, ?it/s]

high_school_government_and_politics:  10%|█         | 2/20 [00:00<00:01, 14.50it/s]

high_school_government_and_politics:  30%|███       | 6/20 [00:00<00:00, 22.09it/s]

high_school_government_and_politics:  60%|██████    | 12/20 [00:00<00:00, 35.58it/s]

high_school_government_and_politics:  90%|█████████ | 18/20 [00:00<00:00, 38.62it/s]

                                                                                    
MMLU subjects:  42%|████▏     | 24/57 [00:52<01:02,  1.89s/it][=] high_school_government_and_politics: 4/20 = 20.0%  (running avg 28.7%)

============================================================
[>] high_school_macroeconomics  (shots=5)


high_school_macroeconomics:   0%|          | 0/20 [00:00<?, ?it/s]

high_school_macroeconomics:  20%|██        | 4/20 [00:00<00:00, 39.59it/s]

high_school_macroeconomics:  40%|████      | 8/20 [00:00<00:00, 38.27it/s]

high_school_macroeconomics:  60%|██████    | 12/20 [00:00<00:00, 32.61it/s]

high_school_macroeconomics:  90%|█████████ | 18/20 [00:00<00:00, 37.33it/s]

                                                                           
MMLU subjects:  44%|████▍     | 25/57 [00:54<00:58,  1.84s/it][=] high_school_macroeconomics: 0/20 = 0.0%  (running avg 27.6%)

============================================================
[>] high_school_mathematics  (shots=5)


high_school_mathematics:   0%|          | 0/20 [00:00<?, ?it/s]

high_school_mathematics:  10%|█         | 2/20 [00:00<00:01, 12.89it/s]

high_school_mathematics:  30%|███       | 6/20 [00:00<00:00, 21.24it/s]

high_school_mathematics:  60%|██████    | 12/20 [00:00<00:00, 35.06it/s]

high_school_mathematics:  90%|█████████ | 18/20 [00:00<00:00, 43.42it/s]

                                                                        
MMLU subjects:  46%|████▌     | 26/57 [00:55<00:57,  1.85s/it][=] high_school_mathematics: 4/20 = 20.0%  (running avg 27.3%)

============================================================
[>] high_school_microeconomics  (shots=5)


high_school_microeconomics:   0%|          | 0/20 [00:00<?, ?it/s]

high_school_microeconomics:  15%|█▌        | 3/20 [00:00<00:00, 18.39it/s]

high_school_microeconomics:  35%|███▌      | 7/20 [00:00<00:00, 27.64it/s]

high_school_microeconomics:  55%|█████▌    | 11/20 [00:00<00:00, 31.83it/s]

high_school_microeconomics:  75%|███████▌  | 15/20 [00:00<00:00, 33.78it/s]

high_school_microeconomics:  95%|█████████▌| 19/20 [00:00<00:00, 31.23it/s]

                                                                           
MMLU subjects:  47%|████▋     | 27/57 [00:57<00:55,  1.86s/it][=] high_school_microeconomics: 7/20 = 35.0%  (running avg 27.6%)

============================================================
[>] high_school_physics  (shots=5)


high_school_physics:   0%|          | 0/20 [00:00<?, ?it/s]

high_school_physics:  10%|█         | 2/20 [00:00<00:00, 18.31it/s]

high_school_physics:  30%|███       | 6/20 [00:00<00:00, 29.27it/s]

high_school_physics:  50%|█████     | 10/20 [00:00<00:00, 28.30it/s]

high_school_physics:  70%|███████   | 14/20 [00:00<00:00, 31.21it/s]

high_school_physics:  90%|█████████ | 18/20 [00:00<00:00, 33.23it/s]

                                                                    
MMLU subjects:  49%|████▉     | 28/57 [00:59<00:55,  1.90s/it][=] high_school_physics: 4/20 = 20.0%  (running avg 27.3%)

============================================================
[>] high_school_psychology  (shots=5)


high_school_psychology:   0%|          | 0/20 [00:00<?, ?it/s]

high_school_psychology:  20%|██        | 4/20 [00:00<00:00, 28.00it/s]

high_school_psychology:  40%|████      | 8/20 [00:00<00:00, 33.07it/s]

high_school_psychology:  60%|██████    | 12/20 [00:00<00:00, 34.99it/s]

high_school_psychology:  80%|████████  | 16/20 [00:00<00:00, 35.93it/s]

high_school_psychology: 100%|██████████| 20/20 [00:00<00:00, 36.42it/s]

                                                                       
MMLU subjects:  51%|█████     | 29/57 [01:01<00:52,  1.87s/it][=] high_school_psychology: 5/20 = 25.0%  (running avg 27.2%)

============================================================
[>] high_school_statistics  (shots=5)


high_school_statistics:   0%|          | 0/20 [00:00<?, ?it/s]

high_school_statistics:  10%|█         | 2/20 [00:00<00:01, 12.71it/s]

high_school_statistics:  25%|██▌       | 5/20 [00:00<00:00, 18.45it/s]

high_school_statistics:  35%|███▌      | 7/20 [00:00<00:00, 18.23it/s]

high_school_statistics:  45%|████▌     | 9/20 [00:00<00:00, 18.11it/s]

high_school_statistics:  55%|█████▌    | 11/20 [00:00<00:00, 17.87it/s]

high_school_statistics:  65%|██████▌   | 13/20 [00:00<00:00, 17.72it/s]

high_school_statistics:  75%|███████▌  | 15/20 [00:00<00:00, 17.67it/s]

high_school_statistics:  85%|████████▌ | 17/20 [00:00<00:00, 17.60it/s]

high_school_statistics:  95%|█████████▌| 19/20 [00:01<00:00, 17.72it/s]

                                                                       
MMLU subjects:  53%|█████▎    | 30/57 [01:04<00:54,  2.04s/it][=] high_school_statistics: 8/20 = 40.0%  (running avg 27.7%)

============================================================
[>] high_school_us_history  (shots=5)


high_school_us_history:   0%|          | 0/20 [00:00<?, ?it/s]

high_school_us_history:  20%|██        | 4/20 [00:00<00:00, 33.99it/s]

high_school_us_history:  45%|████▌     | 9/20 [00:00<00:00, 42.16it/s]

high_school_us_history:  70%|███████   | 14/20 [00:00<00:00, 44.91it/s]

high_school_us_history:  95%|█████████▌| 19/20 [00:00<00:00, 45.94it/s]

                                                                       
MMLU subjects:  54%|█████▍    | 31/57 [01:05<00:49,  1.92s/it][=] high_school_us_history: 4/20 = 20.0%  (running avg 27.4%)

============================================================
[>] high_school_world_history  (shots=5)


high_school_world_history:   0%|          | 0/20 [00:00<?, ?it/s]

high_school_world_history:  15%|█▌        | 3/20 [00:00<00:00, 20.31it/s]

high_school_world_history:  40%|████      | 8/20 [00:00<00:00, 32.41it/s]

high_school_world_history:  65%|██████▌   | 13/20 [00:00<00:00, 39.35it/s]

high_school_world_history:  90%|█████████ | 18/20 [00:00<00:00, 43.13it/s]

                                                                          
MMLU subjects:  56%|█████▌    | 32/57 [01:07<00:46,  1.86s/it][=] high_school_world_history: 7/20 = 35.0%  (running avg 27.7%)

============================================================
[>] human_aging  (shots=5)


human_aging:   0%|          | 0/20 [00:00<?, ?it/s]

human_aging:   5%|▌         | 1/20 [00:00<00:01,  9.91it/s]

human_aging:  20%|██        | 4/20 [00:00<00:00, 18.51it/s]

human_aging:  40%|████      | 8/20 [00:00<00:00, 23.30it/s]

human_aging:  55%|█████▌    | 11/20 [00:00<00:00, 23.62it/s]

human_aging:  70%|███████   | 14/20 [00:00<00:00, 23.48it/s]

human_aging:  90%|█████████ | 18/20 [00:00<00:00, 25.27it/s]

                                                            
MMLU subjects:  58%|█████▊    | 33/57 [01:09<00:45,  1.90s/it][=] human_aging: 8/20 = 40.0%  (running avg 28.0%)

============================================================
[>] human_sexuality  (shots=5)


human_sexuality:   0%|          | 0/20 [00:00<?, ?it/s]

human_sexuality:  10%|█         | 2/20 [00:00<00:01, 17.26it/s]

human_sexuality:  20%|██        | 4/20 [00:00<00:00, 17.97it/s]

human_sexuality:  50%|█████     | 10/20 [00:00<00:00, 31.12it/s]

human_sexuality:  70%|███████   | 14/20 [00:00<00:00, 33.83it/s]

human_sexuality:  90%|█████████ | 18/20 [00:00<00:00, 35.24it/s]

                                                                
MMLU subjects:  60%|█████▉    | 34/57 [01:11<00:44,  1.94s/it][=] human_sexuality: 3/20 = 15.0%  (running avg 27.6%)

============================================================
[>] international_law  (shots=5)


international_law:   0%|          | 0/20 [00:00<?, ?it/s]

international_law:  10%|█         | 2/20 [00:00<00:01, 13.41it/s]

international_law:  30%|███       | 6/20 [00:00<00:00, 21.45it/s]

international_law:  60%|██████    | 12/20 [00:00<00:00, 34.70it/s]

international_law:  80%|████████  | 16/20 [00:00<00:00, 28.56it/s]

international_law: 100%|██████████| 20/20 [00:00<00:00, 25.67it/s]

                                                                  
MMLU subjects:  61%|██████▏   | 35/57 [01:13<00:43,  1.99s/it][=] international_law: 7/20 = 35.0%  (running avg 27.9%)

============================================================
[>] jurisprudence  (shots=5)


jurisprudence:   0%|          | 0/20 [00:00<?, ?it/s]

jurisprudence:  10%|█         | 2/20 [00:00<00:01, 17.11it/s]

jurisprudence:  40%|████      | 8/20 [00:00<00:00, 32.97it/s]

jurisprudence:  60%|██████    | 12/20 [00:00<00:00, 34.56it/s]

jurisprudence:  90%|█████████ | 18/20 [00:00<00:00, 37.91it/s]

                                                              
MMLU subjects:  63%|██████▎   | 36/57 [01:15<00:40,  1.92s/it][=] jurisprudence: 8/20 = 40.0%  (running avg 28.2%)

============================================================
[>] logical_fallacies  (shots=5)


logical_fallacies:   0%|          | 0/20 [00:00<?, ?it/s]

logical_fallacies:  20%|██        | 4/20 [00:00<00:00, 38.49it/s]

logical_fallacies:  50%|█████     | 10/20 [00:00<00:00, 49.03it/s]

logical_fallacies:  75%|███████▌  | 15/20 [00:00<00:00, 38.28it/s]

logical_fallacies: 100%|██████████| 20/20 [00:00<00:00, 38.86it/s]

                                                                  
MMLU subjects:  65%|██████▍   | 37/57 [01:17<00:37,  1.85s/it][=] logical_fallacies: 7/20 = 35.0%  (running avg 28.4%)

============================================================
[>] machine_learning  (shots=5)


Downloading data:   0%|          | 0.00/5.25k [00:00<?, ?B/s]

Downloading data: 100%|██████████| 5.25k/5.25k [00:00<00:00, 19.1kB/s]
Downloading data: 100%|██████████| 5.25k/5.25k [00:00<00:00, 19.0kB/s]


Generating test split:   0%|          | 0/112 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 112/112 [00:00<00:00, 19509.20 examples/s]


Generating validation split:   0%|          | 0/11 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 11/11 [00:00<00:00, 3115.91 examples/s]


Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1538.18 examples/s]


machine_learning:   0%|          | 0/20 [00:00<?, ?it/s]

machine_learning:  10%|█         | 2/20 [00:00<00:01, 13.64it/s]

machine_learning:  30%|███       | 6/20 [00:00<00:00, 25.60it/s]

machine_learning:  50%|█████     | 10/20 [00:00<00:00, 30.54it/s]

machine_learning:  80%|████████  | 16/20 [00:00<00:00, 35.58it/s]

machine_learning: 100%|██████████| 20/20 [00:00<00:00, 32.40it/s]

                                                                 
MMLU subjects:  67%|██████▋   | 38/57 [01:19<00:38,  2.03s/it][=] machine_learning: 10/20 = 50.0%  (running avg 28.9%)

============================================================
[>] management  (shots=5)


Downloading data:   0%|          | 0.00/14.7k [00:00<?, ?B/s]

Downloading data: 100%|██████████| 14.7k/14.7k [00:00<00:00, 71.1kB/s]
Downloading data: 100%|██████████| 14.7k/14.7k [00:00<00:00, 70.8kB/s]


Downloading data:   0%|          | 0.00/4.50k [00:00<?, ?B/s]

Downloading data: 100%|██████████| 4.50k/4.50k [00:00<00:00, 25.8kB/s]
Downloading data: 100%|██████████| 4.50k/4.50k [00:00<00:00, 25.7kB/s]


Downloading data:   0%|          | 0.00/3.61k [00:00<?, ?B/s]

Downloading data: 100%|██████████| 3.61k/3.61k [00:00<00:00, 19.8kB/s]
Downloading data: 100%|██████████| 3.61k/3.61k [00:00<00:00, 19.7kB/s]


Generating test split:   0%|          | 0/103 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 103/103 [00:00<00:00, 23526.29 examples/s]


Generating validation split:   0%|          | 0/11 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 11/11 [00:00<00:00, 3129.65 examples/s]


Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1433.95 examples/s]


management:   0%|          | 0/20 [00:00<?, ?it/s]

management:  10%|█         | 2/20 [00:00<00:01, 14.08it/s]

management:  20%|██        | 4/20 [00:00<00:00, 16.16it/s]

management:  35%|███▌      | 7/20 [00:00<00:00, 19.49it/s]

management:  45%|████▌     | 9/20 [00:00<00:00, 18.65it/s]

management:  60%|██████    | 12/20 [00:00<00:00, 20.56it/s]

management:  75%|███████▌  | 15/20 [00:00<00:00, 19.77it/s]

                                                           
MMLU subjects:  68%|██████▊   | 39/57 [01:23<00:45,  2.55s/it][=] management: 4/20 = 20.0%  (running avg 28.7%)

============================================================
[>] marketing  (shots=5)


Downloading data:   0%|          | 0.00/37.3k [00:00<?, ?B/s]

Downloading data: 100%|██████████| 37.3k/37.3k [00:00<00:00, 216kB/s]
Downloading data: 100%|██████████| 37.3k/37.3k [00:00<00:00, 214kB/s]


Downloading data:   0%|          | 0.00/8.21k [00:00<?, ?B/s]

Downloading data: 100%|██████████| 8.21k/8.21k [00:00<00:00, 40.3kB/s]
Downloading data: 100%|██████████| 8.21k/8.21k [00:00<00:00, 40.1kB/s]


Downloading data:   0%|          | 0.00/4.28k [00:00<?, ?B/s]

Downloading data: 100%|██████████| 4.28k/4.28k [00:00<00:00, 13.5kB/s]
Downloading data: 100%|██████████| 4.28k/4.28k [00:00<00:00, 13.5kB/s]


Generating test split:   0%|          | 0/234 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 234/234 [00:00<00:00, 47706.56 examples/s]


Generating validation split:   0%|          | 0/25 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 25/25 [00:00<00:00, 6806.73 examples/s]


Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1449.31 examples/s]


marketing:   0%|          | 0/20 [00:00<?, ?it/s]

marketing:  25%|██▌       | 5/20 [00:00<00:00, 43.68it/s]

marketing:  55%|█████▌    | 11/20 [00:00<00:00, 52.50it/s]

marketing:  85%|████████▌ | 17/20 [00:00<00:00, 55.39it/s]

                                                          
MMLU subjects:  70%|███████   | 40/57 [01:26<00:47,  2.77s/it][=] marketing: 7/20 = 35.0%  (running avg 28.9%)

============================================================
[>] medical_genetics  (shots=5)


Downloading data:   0%|          | 0.00/16.4k [00:00<?, ?B/s]

Downloading data: 100%|██████████| 16.4k/16.4k [00:00<00:00, 84.6kB/s]
Downloading data: 100%|██████████| 16.4k/16.4k [00:00<00:00, 84.2kB/s]


Downloading data:   0%|          | 0.00/5.63k [00:00<?, ?B/s]

Downloading data: 100%|██████████| 5.63k/5.63k [00:00<00:00, 25.0kB/s]
Downloading data: 100%|██████████| 5.63k/5.63k [00:00<00:00, 24.9kB/s]


Downloading data:   0%|          | 0.00/3.77k [00:00<?, ?B/s]

Downloading data: 100%|██████████| 3.77k/3.77k [00:00<00:00, 22.5kB/s]
Downloading data: 100%|██████████| 3.77k/3.77k [00:00<00:00, 22.4kB/s]


Generating test split:   0%|          | 0/100 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 100/100 [00:00<00:00, 23050.69 examples/s]


Generating validation split:   0%|          | 0/11 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 11/11 [00:00<00:00, 3044.36 examples/s]


Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1133.23 examples/s]


medical_genetics:   0%|          | 0/20 [00:00<?, ?it/s]

medical_genetics:  15%|█▌        | 3/20 [00:00<00:00, 25.31it/s]

medical_genetics:  50%|█████     | 10/20 [00:00<00:00, 45.69it/s]

medical_genetics:  75%|███████▌  | 15/20 [00:00<00:00, 42.92it/s]

                                                                 
MMLU subjects:  72%|███████▏  | 41/57 [01:29<00:47,  2.94s/it][=] medical_genetics: 8/20 = 40.0%  (running avg 29.1%)

============================================================
[>] miscellaneous  (shots=5)


Downloading data:   0%|          | 0.00/98.6k [00:00<?, ?B/s]

Downloading data: 100%|██████████| 98.6k/98.6k [00:00<00:00, 387kB/s]
Downloading data: 100%|██████████| 98.6k/98.6k [00:00<00:00, 386kB/s]


Downloading data:   0%|          | 0.00/13.2k [00:00<?, ?B/s]

Downloading data: 100%|██████████| 13.2k/13.2k [00:00<00:00, 73.2kB/s]
Downloading data: 100%|██████████| 13.2k/13.2k [00:00<00:00, 72.8kB/s]


Downloading data:   0%|          | 0.00/3.37k [00:00<?, ?B/s]

Downloading data: 100%|██████████| 3.37k/3.37k [00:00<00:00, 18.5kB/s]
Downloading data: 100%|██████████| 3.37k/3.37k [00:00<00:00, 18.4kB/s]


Generating test split:   0%|          | 0/783 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 783/783 [00:00<00:00, 137394.47 examples/s]


Generating validation split:   0%|          | 0/86 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 86/86 [00:00<00:00, 21921.01 examples/s]


Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1380.52 examples/s]


miscellaneous:   0%|          | 0/20 [00:00<?, ?it/s]

miscellaneous:  15%|█▌        | 3/20 [00:00<00:00, 26.43it/s]

miscellaneous:  35%|███▌      | 7/20 [00:00<00:00, 33.14it/s]

miscellaneous:  55%|█████▌    | 11/20 [00:00<00:00, 23.92it/s]

miscellaneous:  70%|███████   | 14/20 [00:00<00:00, 23.88it/s]

miscellaneous:  90%|█████████ | 18/20 [00:00<00:00, 28.17it/s]

                                                              
MMLU subjects:  74%|███████▎  | 42/57 [01:33<00:46,  3.09s/it][=] miscellaneous: 6/20 = 30.0%  (running avg 29.2%)

============================================================
[>] moral_disputes  (shots=5)


Downloading data:   0%|          | 0.00/60.9k [00:00<?, ?B/s]

Downloading data: 100%|██████████| 60.9k/60.9k [00:00<00:00, 299kB/s]
Downloading data: 100%|██████████| 60.9k/60.9k [00:00<00:00, 298kB/s]


Downloading data:   0%|          | 0.00/10.7k [00:00<?, ?B/s]

Downloading data: 100%|██████████| 10.7k/10.7k [00:00<00:00, 51.6kB/s]
Downloading data: 100%|██████████| 10.7k/10.7k [00:00<00:00, 51.4kB/s]


Downloading data:   0%|          | 0.00/4.41k [00:00<?, ?B/s]

Downloading data: 100%|██████████| 4.41k/4.41k [00:00<00:00, 26.7kB/s]
Downloading data: 100%|██████████| 4.41k/4.41k [00:00<00:00, 26.5kB/s]


Generating test split:   0%|          | 0/346 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 346/346 [00:00<00:00, 62921.83 examples/s]


Generating validation split:   0%|          | 0/38 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 38/38 [00:00<00:00, 9795.56 examples/s]


Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1382.98 examples/s]


moral_disputes:   0%|          | 0/20 [00:00<?, ?it/s]

moral_disputes:  25%|██▌       | 5/20 [00:00<00:00, 43.64it/s]

moral_disputes:  50%|█████     | 10/20 [00:00<00:00, 41.80it/s]

moral_disputes:  80%|████████  | 16/20 [00:00<00:00, 49.01it/s]

                                                               
MMLU subjects:  75%|███████▌  | 43/57 [01:36<00:43,  3.11s/it][=] moral_disputes: 4/20 = 20.0%  (running avg 29.0%)

============================================================
[>] moral_scenarios  (shots=5)


Downloading data:   0%|          | 0.00/89.8k [00:00<?, ?B/s]

Downloading data: 100%|██████████| 89.8k/89.8k [00:00<00:00, 502kB/s]
Downloading data: 100%|██████████| 89.8k/89.8k [00:00<00:00, 499kB/s]


Downloading data:   0%|          | 0.00/14.9k [00:00<?, ?B/s]

Downloading data: 100%|██████████| 14.9k/14.9k [00:00<00:00, 97.2kB/s]
Downloading data: 100%|██████████| 14.9k/14.9k [00:00<00:00, 96.7kB/s]


Downloading data:   0%|          | 0.00/5.14k [00:00<?, ?B/s]

Downloading data: 100%|██████████| 5.14k/5.14k [00:00<00:00, 27.5kB/s]
Downloading data: 100%|██████████| 5.14k/5.14k [00:00<00:00, 27.4kB/s]


Generating test split:   0%|          | 0/895 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 895/895 [00:00<00:00, 133752.66 examples/s]


Generating validation split:   0%|          | 0/100 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 100/100 [00:00<00:00, 20927.57 examples/s]


Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1409.47 examples/s]


moral_scenarios:   0%|          | 0/20 [00:00<?, ?it/s]

moral_scenarios:  15%|█▌        | 3/20 [00:00<00:00, 24.61it/s]

moral_scenarios:  35%|███▌      | 7/20 [00:00<00:00, 31.12it/s]

moral_scenarios:  60%|██████    | 12/20 [00:00<00:00, 34.85it/s]

moral_scenarios:  80%|████████  | 16/20 [00:00<00:00, 35.45it/s]

                                                                
MMLU subjects:  77%|███████▋  | 44/57 [01:39<00:40,  3.15s/it][=] moral_scenarios: 3/20 = 15.0%  (running avg 28.6%)

============================================================
[>] nutrition  (shots=5)


Downloading data:   0%|          | 0.00/55.0k [00:00<?, ?B/s]

Downloading data: 100%|██████████| 55.0k/55.0k [00:00<00:00, 173kB/s]
Downloading data: 100%|██████████| 55.0k/55.0k [00:00<00:00, 172kB/s]


Downloading data:   0%|          | 0.00/9.02k [00:00<?, ?B/s]

Downloading data: 100%|██████████| 9.02k/9.02k [00:00<00:00, 58.6kB/s]
Downloading data: 100%|██████████| 9.02k/9.02k [00:00<00:00, 58.3kB/s]


Downloading data:   0%|          | 0.00/4.99k [00:00<?, ?B/s]

Downloading data: 100%|██████████| 4.99k/4.99k [00:00<00:00, 28.0kB/s]
Downloading data: 100%|██████████| 4.99k/4.99k [00:00<00:00, 27.9kB/s]


Generating test split:   0%|          | 0/306 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 306/306 [00:00<00:00, 60429.26 examples/s]


Generating validation split:   0%|          | 0/33 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 33/33 [00:00<00:00, 9086.33 examples/s]


Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1449.21 examples/s]


nutrition:   0%|          | 0/20 [00:00<?, ?it/s]

nutrition:  10%|█         | 2/20 [00:00<00:00, 19.34it/s]

nutrition:  30%|███       | 6/20 [00:00<00:00, 24.76it/s]

nutrition:  45%|████▌     | 9/20 [00:00<00:00, 24.27it/s]

nutrition:  65%|██████▌   | 13/20 [00:00<00:00, 25.76it/s]

nutrition:  85%|████████▌ | 17/20 [00:00<00:00, 29.38it/s]

                                                          
MMLU subjects:  79%|███████▉  | 45/57 [01:43<00:40,  3.34s/it][=] nutrition: 4/20 = 20.0%  (running avg 28.4%)

============================================================
[>] philosophy  (shots=5)


philosophy:   0%|          | 0/20 [00:00<?, ?it/s]

philosophy:  10%|█         | 2/20 [00:00<00:01, 17.21it/s]

philosophy:  45%|████▌     | 9/20 [00:00<00:00, 43.07it/s]

philosophy:  80%|████████  | 16/20 [00:00<00:00, 51.44it/s]

                                                           
MMLU subjects:  81%|████████  | 46/57 [01:45<00:31,  2.87s/it][=] philosophy: 3/20 = 15.0%  (running avg 28.2%)

============================================================
[>] prehistory  (shots=5)


Downloading data:   0%|          | 0.00/54.3k [00:00<?, ?B/s]

Downloading data: 100%|██████████| 54.3k/54.3k [00:00<00:00, 336kB/s]
Downloading data: 100%|██████████| 54.3k/54.3k [00:00<00:00, 334kB/s]


Downloading data:   0%|          | 0.00/9.89k [00:00<?, ?B/s]

Downloading data: 100%|██████████| 9.89k/9.89k [00:00<00:00, 50.2kB/s]
Downloading data: 100%|██████████| 9.89k/9.89k [00:00<00:00, 49.9kB/s]


Downloading data:   0%|          | 0.00/4.62k [00:00<?, ?B/s]

Downloading data: 100%|██████████| 4.62k/4.62k [00:00<00:00, 21.8kB/s]
Downloading data: 100%|██████████| 4.62k/4.62k [00:00<00:00, 21.7kB/s]


Generating test split:   0%|          | 0/324 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 324/324 [00:00<00:00, 63532.23 examples/s]


Generating validation split:   0%|          | 0/35 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 35/35 [00:00<00:00, 9094.33 examples/s]


Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1395.22 examples/s]


prehistory:   0%|          | 0/20 [00:00<?, ?it/s]

prehistory:  25%|██▌       | 5/20 [00:00<00:00, 43.02it/s]

prehistory:  50%|█████     | 10/20 [00:00<00:00, 41.15it/s]

prehistory:  75%|███████▌  | 15/20 [00:00<00:00, 35.37it/s]

                                                           
MMLU subjects:  82%|████████▏ | 47/57 [01:48<00:30,  3.04s/it][=] prehistory: 5/20 = 25.0%  (running avg 28.1%)

============================================================
[>] professional_accounting  (shots=5)


Downloading data:   0%|          | 0.00/69.5k [00:00<?, ?B/s]

Downloading data: 100%|██████████| 69.5k/69.5k [00:00<00:00, 356kB/s]
Downloading data: 100%|██████████| 69.5k/69.5k [00:00<00:00, 354kB/s]


Downloading data:   0%|          | 0.00/12.9k [00:00<?, ?B/s]

Downloading data: 100%|██████████| 12.9k/12.9k [00:00<00:00, 61.9kB/s]
Downloading data: 100%|██████████| 12.9k/12.9k [00:00<00:00, 61.7kB/s]


Downloading data:   0%|          | 0.00/4.89k [00:00<?, ?B/s]

Downloading data: 100%|██████████| 4.89k/4.89k [00:00<00:00, 27.4kB/s]
Downloading data: 100%|██████████| 4.89k/4.89k [00:00<00:00, 27.3kB/s]


Generating test split:   0%|          | 0/282 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 282/282 [00:00<00:00, 54398.83 examples/s]


Generating validation split:   0%|          | 0/31 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 31/31 [00:00<00:00, 8148.87 examples/s]


Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1535.93 examples/s]


professional_accounting:   0%|          | 0/20 [00:00<?, ?it/s]

professional_accounting:  10%|█         | 2/20 [00:00<00:00, 18.76it/s]

professional_accounting:  30%|███       | 6/20 [00:00<00:00, 24.19it/s]

professional_accounting:  50%|█████     | 10/20 [00:00<00:00, 29.18it/s]

professional_accounting:  70%|███████   | 14/20 [00:00<00:00, 31.16it/s]

professional_accounting:  95%|█████████▌| 19/20 [00:00<00:00, 33.78it/s]

                                                                        
MMLU subjects:  84%|████████▍ | 48/57 [01:52<00:28,  3.19s/it][=] professional_accounting: 6/20 = 30.0%  (running avg 28.1%)

============================================================
[>] professional_law  (shots=5)


Downloading data:   0%|          | 0.00/1.04M [00:00<?, ?B/s]

Downloading data: 100%|██████████| 1.04M/1.04M [00:00<00:00, 3.96MB/s]
Downloading data: 100%|██████████| 1.04M/1.04M [00:00<00:00, 3.93MB/s]


Downloading data:   0%|          | 0.00/116k [00:00<?, ?B/s]

Downloading data: 100%|██████████| 116k/116k [00:00<00:00, 603kB/s]
Downloading data: 100%|██████████| 116k/116k [00:00<00:00, 600kB/s]


Downloading data:   0%|          | 0.00/15.1k [00:00<?, ?B/s]

Downloading data: 100%|██████████| 15.1k/15.1k [00:00<00:00, 90.5kB/s]
Downloading data: 100%|██████████| 15.1k/15.1k [00:00<00:00, 90.1kB/s]


Generating test split:   0%|          | 0/1534 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 1534/1534 [00:00<00:00, 72607.74 examples/s]


Generating validation split:   0%|          | 0/170 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 170/170 [00:00<00:00, 29159.27 examples/s]


Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1370.87 examples/s]


professional_law:   0%|          | 0/20 [00:00<?, ?it/s]

professional_law:  20%|██        | 4/20 [00:00<00:00, 36.28it/s]

professional_law:  50%|█████     | 10/20 [00:00<00:00, 44.88it/s]

professional_law:  75%|███████▌  | 15/20 [00:00<00:00, 46.92it/s]

                                                                 
MMLU subjects:  86%|████████▌ | 49/57 [01:55<00:25,  3.17s/it][=] professional_law: 7/20 = 35.0%  (running avg 28.3%)

============================================================
[>] professional_medicine  (shots=5)


Downloading data:   0%|          | 0.00/125k [00:00<?, ?B/s]

Downloading data: 100%|██████████| 125k/125k [00:00<00:00, 666kB/s]
Downloading data: 100%|██████████| 125k/125k [00:00<00:00, 663kB/s]


Downloading data:   0%|          | 0.00/19.9k [00:00<?, ?B/s]

Downloading data: 100%|██████████| 19.9k/19.9k [00:00<00:00, 134kB/s]
Downloading data: 100%|██████████| 19.9k/19.9k [00:00<00:00, 134kB/s]


Downloading data:   0%|          | 0.00/8.45k [00:00<?, ?B/s]

Downloading data: 100%|██████████| 8.45k/8.45k [00:00<00:00, 52.0kB/s]
Downloading data: 100%|██████████| 8.45k/8.45k [00:00<00:00, 51.7kB/s]


Generating test split:   0%|          | 0/272 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 272/272 [00:00<00:00, 37004.56 examples/s]


Generating validation split:   0%|          | 0/31 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 31/31 [00:00<00:00, 7737.65 examples/s]


Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1409.19 examples/s]


professional_medicine:   0%|          | 0/20 [00:00<?, ?it/s]

professional_medicine:  20%|██        | 4/20 [00:00<00:00, 37.94it/s]

professional_medicine:  50%|█████     | 10/20 [00:00<00:00, 47.05it/s]

professional_medicine:  80%|████████  | 16/20 [00:00<00:00, 49.68it/s]

                                                                      
MMLU subjects:  88%|████████▊ | 50/57 [01:58<00:21,  3.13s/it][=] professional_medicine: 1/20 = 5.0%  (running avg 27.8%)

============================================================
[>] professional_psychology  (shots=5)


Downloading data:   0%|          | 0.00/133k [00:00<?, ?B/s]

Downloading data: 100%|██████████| 133k/133k [00:00<00:00, 599kB/s]
Downloading data: 100%|██████████| 133k/133k [00:00<00:00, 596kB/s]


Downloading data:   0%|          | 0.00/22.1k [00:00<?, ?B/s]

Downloading data: 100%|██████████| 22.1k/22.1k [00:00<00:00, 81.8kB/s]
Downloading data: 100%|██████████| 22.1k/22.1k [00:00<00:00, 81.6kB/s]


Downloading data:   0%|          | 0.00/4.69k [00:00<?, ?B/s]

Downloading data: 100%|██████████| 4.69k/4.69k [00:00<00:00, 30.9kB/s]
Downloading data: 100%|██████████| 4.69k/4.69k [00:00<00:00, 30.8kB/s]


Generating test split:   0%|          | 0/612 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 612/612 [00:00<00:00, 95884.13 examples/s]


Generating validation split:   0%|          | 0/69 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 69/69 [00:00<00:00, 16709.41 examples/s]


Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1311.87 examples/s]


professional_psychology:   0%|          | 0/20 [00:00<?, ?it/s]

professional_psychology:  20%|██        | 4/20 [00:00<00:00, 36.84it/s]

professional_psychology:  40%|████      | 8/20 [00:00<00:00, 36.65it/s]

professional_psychology:  60%|██████    | 12/20 [00:00<00:00, 27.62it/s]

professional_psychology:  80%|████████  | 16/20 [00:00<00:00, 30.48it/s]

professional_psychology: 100%|██████████| 20/20 [00:00<00:00, 29.12it/s]

                                                                        
MMLU subjects:  89%|████████▉ | 51/57 [02:01<00:19,  3.24s/it][=] professional_psychology: 9/20 = 45.0%  (running avg 28.1%)

============================================================
[>] public_relations  (shots=5)


Downloading data:   0%|          | 0.00/20.6k [00:00<?, ?B/s]

Downloading data: 100%|██████████| 20.6k/20.6k [00:00<00:00, 125kB/s]
Downloading data: 100%|██████████| 20.6k/20.6k [00:00<00:00, 125kB/s]


Downloading data:   0%|          | 0.00/6.45k [00:00<?, ?B/s]

Downloading data: 100%|██████████| 6.45k/6.45k [00:00<00:00, 35.2kB/s]
Downloading data: 100%|██████████| 6.45k/6.45k [00:00<00:00, 35.0kB/s]


Downloading data:   0%|          | 0.00/4.43k [00:00<?, ?B/s]

Downloading data: 100%|██████████| 4.43k/4.43k [00:00<00:00, 29.9kB/s]
Downloading data: 100%|██████████| 4.43k/4.43k [00:00<00:00, 29.7kB/s]


Generating test split:   0%|          | 0/110 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 110/110 [00:00<00:00, 24437.15 examples/s]


Generating validation split:   0%|          | 0/12 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 12/12 [00:00<00:00, 3341.85 examples/s]


Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1466.54 examples/s]


public_relations:   0%|          | 0/20 [00:00<?, ?it/s]

public_relations:  10%|█         | 2/20 [00:00<00:00, 19.12it/s]

public_relations:  30%|███       | 6/20 [00:00<00:00, 29.51it/s]

public_relations:  55%|█████▌    | 11/20 [00:00<00:00, 34.93it/s]

public_relations:  75%|███████▌  | 15/20 [00:00<00:00, 35.89it/s]

                                                                 
MMLU subjects:  91%|█████████ | 52/57 [02:05<00:16,  3.20s/it][=] public_relations: 9/20 = 45.0%  (running avg 28.5%)

============================================================
[>] security_studies  (shots=5)


Downloading data:   0%|          | 0.00/114k [00:00<?, ?B/s]

Downloading data: 100%|██████████| 114k/114k [00:00<00:00, 675kB/s]
Downloading data: 100%|██████████| 114k/114k [00:00<00:00, 671kB/s]


Downloading data:   0%|          | 0.00/18.7k [00:00<?, ?B/s]

Downloading data: 100%|██████████| 18.7k/18.7k [00:00<00:00, 106kB/s]
Downloading data: 100%|██████████| 18.7k/18.7k [00:00<00:00, 106kB/s]


Downloading data:   0%|          | 0.00/7.49k [00:00<?, ?B/s]

Downloading data: 100%|██████████| 7.49k/7.49k [00:00<00:00, 45.0kB/s]
Downloading data: 100%|██████████| 7.49k/7.49k [00:00<00:00, 44.8kB/s]


Generating test split:   0%|          | 0/245 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 245/245 [00:00<00:00, 37800.42 examples/s]


Generating validation split:   0%|          | 0/27 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 27/27 [00:00<00:00, 6508.78 examples/s]


Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1451.42 examples/s]


security_studies:   0%|          | 0/20 [00:00<?, ?it/s]

security_studies:  20%|██        | 4/20 [00:00<00:00, 37.50it/s]

security_studies:  50%|█████     | 10/20 [00:00<00:00, 46.09it/s]

security_studies:  80%|████████  | 16/20 [00:00<00:00, 48.87it/s]

                                                                 
MMLU subjects:  93%|█████████▎| 53/57 [02:08<00:12,  3.18s/it][=] security_studies: 5/20 = 25.0%  (running avg 28.4%)

============================================================
[>] sociology  (shots=5)


Downloading data:   0%|          | 0.00/43.9k [00:00<?, ?B/s]

Downloading data: 100%|██████████| 43.9k/43.9k [00:00<00:00, 228kB/s]
Downloading data: 100%|██████████| 43.9k/43.9k [00:00<00:00, 227kB/s]


Downloading data:   0%|          | 0.00/8.36k [00:00<?, ?B/s]

Downloading data: 100%|██████████| 8.36k/8.36k [00:00<00:00, 51.2kB/s]
Downloading data: 100%|██████████| 8.36k/8.36k [00:00<00:00, 50.9kB/s]


Downloading data:   0%|          | 0.00/4.21k [00:00<?, ?B/s]

Downloading data: 100%|██████████| 4.21k/4.21k [00:00<00:00, 27.3kB/s]
Downloading data: 100%|██████████| 4.21k/4.21k [00:00<00:00, 27.1kB/s]


Generating test split:   0%|          | 0/201 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 201/201 [00:00<00:00, 40145.48 examples/s]


Generating validation split:   0%|          | 0/22 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 22/22 [00:00<00:00, 5844.98 examples/s]


Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1353.79 examples/s]


sociology:   0%|          | 0/20 [00:00<?, ?it/s]

sociology:  10%|█         | 2/20 [00:00<00:01, 13.70it/s]

sociology:  30%|███       | 6/20 [00:00<00:00, 25.24it/s]

sociology:  60%|██████    | 12/20 [00:00<00:00, 38.57it/s]

sociology:  90%|█████████ | 18/20 [00:00<00:00, 45.52it/s]

                                                          
MMLU subjects:  95%|█████████▍| 54/57 [02:11<00:09,  3.13s/it][=] sociology: 4/20 = 20.0%  (running avg 28.2%)

============================================================
[>] us_foreign_policy  (shots=5)


Downloading data:   0%|          | 0.00/19.5k [00:00<?, ?B/s]

Downloading data: 100%|██████████| 19.5k/19.5k [00:00<00:00, 114kB/s]
Downloading data: 100%|██████████| 19.5k/19.5k [00:00<00:00, 113kB/s]


Downloading data:   0%|          | 0.00/5.27k [00:00<?, ?B/s]

Downloading data: 100%|██████████| 5.27k/5.27k [00:00<00:00, 34.0kB/s]
Downloading data: 100%|██████████| 5.27k/5.27k [00:00<00:00, 33.8kB/s]


Downloading data:   0%|          | 0.00/4.22k [00:00<?, ?B/s]

Downloading data: 100%|██████████| 4.22k/4.22k [00:00<00:00, 22.7kB/s]
Downloading data: 100%|██████████| 4.22k/4.22k [00:00<00:00, 22.6kB/s]


Generating test split:   0%|          | 0/100 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 100/100 [00:00<00:00, 23196.02 examples/s]


Generating validation split:   0%|          | 0/11 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 11/11 [00:00<00:00, 2971.81 examples/s]


Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1376.35 examples/s]


us_foreign_policy:   0%|          | 0/20 [00:00<?, ?it/s]

us_foreign_policy:  25%|██▌       | 5/20 [00:00<00:00, 44.09it/s]

us_foreign_policy:  55%|█████▌    | 11/20 [00:00<00:00, 52.64it/s]

us_foreign_policy:  85%|████████▌ | 17/20 [00:00<00:00, 55.54it/s]

                                                                  
MMLU subjects:  96%|█████████▋| 55/57 [02:14<00:06,  3.20s/it][=] us_foreign_policy: 5/20 = 25.0%  (running avg 28.2%)

============================================================
[>] virology  (shots=5)


Downloading data:   0%|          | 0.00/27.3k [00:00<?, ?B/s]

Downloading data: 100%|██████████| 27.3k/27.3k [00:00<00:00, 176kB/s]
Downloading data: 100%|██████████| 27.3k/27.3k [00:00<00:00, 175kB/s]


Downloading data:   0%|          | 0.00/7.05k [00:00<?, ?B/s]

Downloading data: 100%|██████████| 7.05k/7.05k [00:00<00:00, 42.4kB/s]
Downloading data: 100%|██████████| 7.05k/7.05k [00:00<00:00, 42.2kB/s]


Downloading data:   0%|          | 0.00/3.87k [00:00<?, ?B/s]

Downloading data: 100%|██████████| 3.87k/3.87k [00:00<00:00, 23.4kB/s]
Downloading data: 100%|██████████| 3.87k/3.87k [00:00<00:00, 23.3kB/s]


Generating test split:   0%|          | 0/166 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 166/166 [00:00<00:00, 42164.02 examples/s]


Generating validation split:   0%|          | 0/18 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 18/18 [00:00<00:00, 5863.43 examples/s]


Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1653.77 examples/s]


virology:   0%|          | 0/20 [00:00<?, ?it/s]

virology:  15%|█▌        | 3/20 [00:00<00:00, 27.33it/s]

virology:  45%|████▌     | 9/20 [00:00<00:00, 45.23it/s]

virology:  70%|███████   | 14/20 [00:00<00:00, 42.99it/s]

virology: 100%|██████████| 20/20 [00:00<00:00, 48.97it/s]

                                                         
MMLU subjects:  98%|█████████▊| 56/57 [02:17<00:03,  3.17s/it][=] virology: 5/20 = 25.0%  (running avg 28.1%)

============================================================
[>] world_religions  (shots=5)


Downloading data:   0%|          | 0.00/18.9k [00:00<?, ?B/s]

Downloading data: 100%|██████████| 18.9k/18.9k [00:00<00:00, 106kB/s]
Downloading data: 100%|██████████| 18.9k/18.9k [00:00<00:00, 106kB/s]


Downloading data:   0%|          | 0.00/4.94k [00:00<?, ?B/s]

Downloading data: 100%|██████████| 4.94k/4.94k [00:00<00:00, 30.3kB/s]
Downloading data: 100%|██████████| 4.94k/4.94k [00:00<00:00, 30.2kB/s]


Downloading data:   0%|          | 0.00/3.30k [00:00<?, ?B/s]

Downloading data: 100%|██████████| 3.30k/3.30k [00:00<00:00, 21.2kB/s]
Downloading data: 100%|██████████| 3.30k/3.30k [00:00<00:00, 21.1kB/s]


Generating test split:   0%|          | 0/171 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 171/171 [00:00<00:00, 39673.97 examples/s]


Generating validation split:   0%|          | 0/19 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 19/19 [00:00<00:00, 5365.37 examples/s]


Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1399.69 examples/s]


world_religions:   0%|          | 0/20 [00:00<?, ?it/s]

world_religions:  10%|█         | 2/20 [00:00<00:01, 13.20it/s]

world_religions:  25%|██▌       | 5/20 [00:00<00:00, 18.83it/s]

world_religions:  40%|████      | 8/20 [00:00<00:00, 20.93it/s]

world_religions:  60%|██████    | 12/20 [00:00<00:00, 23.89it/s]

world_religions:  75%|███████▌  | 15/20 [00:00<00:00, 23.91it/s]

world_religions:  95%|█████████▌| 19/20 [00:00<00:00, 28.23it/s]

                                                                
MMLU subjects: 100%|██████████| 57/57 [02:20<00:00,  3.22s/it]
MMLU subjects: 100%|██████████| 57/57 [02:20<00:00,  2.47s/it]
[=] world_religions: 3/20 = 15.0%  (running avg 27.9%)

============================================================
MMLU RESULT — /home/zeus/microllm2/microllm2-checkpoints/final_merged
Shots: 5  Subjects: 57/57
  abstract_algebra                     30.0%  (6/20)
  anatomy                              25.0%  (5/20)
  astronomy                            35.0%  (7/20)
  business_ethics                      30.0%  (6/20)
  clinical_knowledge                   45.0%  (9/20)
  college_biology                      45.0%  (9/20)
  college_chemistry                    15.0%  (3/20)
  college_computer_science             45.0%  (9/20)
  college_mathematics                  35.0%  (7/20)
  college_medicine                     30.0%  (6/20)
  college_physics                      15.0%  (3/20)
  computer_security                    30.0%  (6/20)
  conceptual_physics                    5.0%  (1/20)
  econometrics                         30.0%  (6/20)
  electrical_engineering               20.0%  (4/20)
  elementary_mathematics               30.0%  (6/20)
  formal_logic                         10.0%  (2/20)
  global_facts                         35.0%  (7/20)
  high_school_biology                  45.0%  (9/20)
  high_school_chemistry                35.0%  (7/20)
  high_school_computer_science         35.0%  (7/20)
  high_school_european_history         20.0%  (4/20)
  high_school_geography                25.0%  (5/20)
  high_school_government_and_politics  20.0%  (4/20)
  high_school_macroeconomics            0.0%  (0/20)
  high_school_mathematics              20.0%  (4/20)
  high_school_microeconomics           35.0%  (7/20)
  high_school_physics                  20.0%  (4/20)
  high_school_psychology               25.0%  (5/20)
  high_school_statistics               40.0%  (8/20)
  high_school_us_history               20.0%  (4/20)
  high_school_world_history            35.0%  (7/20)
  human_aging                          40.0%  (8/20)
  human_sexuality                      15.0%  (3/20)
  international_law                    35.0%  (7/20)
  jurisprudence                        40.0%  (8/20)
  logical_fallacies                    35.0%  (7/20)
  machine_learning                     50.0%  (10/20)
  management                           20.0%  (4/20)
  marketing                            35.0%  (7/20)
  medical_genetics                     40.0%  (8/20)
  miscellaneous                        30.0%  (6/20)
  moral_disputes                       20.0%  (4/20)
  moral_scenarios                      15.0%  (3/20)
  nutrition                            20.0%  (4/20)
  philosophy                           15.0%  (3/20)
  prehistory                           25.0%  (5/20)
  professional_accounting              30.0%  (6/20)
  professional_law                     35.0%  (7/20)
  professional_medicine                 5.0%  (1/20)
  professional_psychology              45.0%  (9/20)
  public_relations                     45.0%  (9/20)
  security_studies                     25.0%  (5/20)
  sociology                            20.0%  (4/20)
  us_foreign_policy                    25.0%  (5/20)
  virology                             25.0%  (5/20)
  world_religions                      15.0%  (3/20)

  OVERALL: 318/1140 = 27.89%
============================================================
[+] Saved mmlu_results.json

Note: GPT2-XL base ~24-26% MMLU (random 25%). MicroLLM2 distilled should be 25-30% —
MMLU is knowledge-heavy; GPT2 1.5B 1024ctx cannot match 7B+ models. Use as sanity check, not SOTA claim.
[MMLU57-TOKEN] exit 0 at Sun Aug  9 23:12:32 UTC 2026
{
  "model": "/home/zeus/microllm2/microllm2-checkpoints/final_merged",
  "shots": 5,
  "limit": 20,
  "overall": {
    "correct": 318,
    "total": 1140,
    "accuracy": 0.2789473684210526
  },
  "subjects": {
    "abstract_algebra": {
      "correct": 6,
      "total": 20,
      "accuracy": 0.3
    },
    "anatomy": {
      "correct": 5,
      "total": 20,
      "accuracy": 0.25
    },
    "astronomy": {
      "correct": 7,
      "total": 20,
      "accuracy": 0.35
    },
    "business_ethics": {
      "correct": 6,
      "total": 20,
      "accuracy": 0.3
    },
FINAL57 318 / 1140 27.89 57
'mmlu_results.json' -> 'mmlu57_results.json'
'mmlu_results.json' -> '/teamspace/studios/this_studio/microllm2/mmlu57_results.json'
'mmlu57_token.log' -> '/teamspace/studios/this_studio/microllm2/mmlu57_token.log'