Hugging Face's logo

MLVXN
/
MicroLLM2

Text Generation
Transformers
Safetensors
GGUF
English
gpt2
chatbot
lora
instruction-tuned
distilled
microllm2
conversational
text-generation-inference
Model card Files
xet
 Community
MicroLLM2 / mmlu57_token.log
MLVXN's picture
MLVXN
bench: add mmlu57_token.log full 57
683d31a verified
Raw
History Blame Contribute Delete
73.3 kB
[MMLU57-TOKEN] start Sun Aug 9 23:10:07 UTC 2026 token=hf_oUMhDnr...
[*] MicroLLM2 MMLU — model: /home/zeus/microllm2/microllm2-checkpoints/final_merged shots=5 limit=20
[*] GPT2-XL 1.5B 1024ctx vocab=50259 (ChatML) — MMLU via direct eval (no harness needed)
[*] lm-eval not installed — using lightweight direct MMLU eval (same logic, no harness)
[*] Install harness for official numbers: pip install lm-eval==0.4.4
[*] Loading tokenizer + model /home/zeus/microllm2/microllm2-checkpoints/final_merged ...
[+] Loaded on cuda:0 dtype=torch.bfloat16 — starting MMLU
 MMLU subjects: 0%| | 0/57 [00:00<?, ?it/s]
============================================================
[>] abstract_algebra (shots=5)
 abstract_algebra: 0%| | 0/20 [00:00<?, ?it/s]
 abstract_algebra: 5%|▌ | 1/20 [00:01<00:21, 1.15s/it]
 abstract_algebra: 15%|█▌ | 3/20 [00:01<00:05, 2.92it/s]
 abstract_algebra: 25%|██▌ | 5/20 [00:01<00:02, 5.08it/s]
 abstract_algebra: 40%|████ | 8/20 [00:01<00:01, 8.67it/s]
 abstract_algebra: 50%|█████ | 10/20 [00:01<00:00, 10.56it/s]
 abstract_algebra: 70%|███████ | 14/20 [00:01<00:00, 15.27it/s]
 abstract_algebra: 85%|████████▌ | 17/20 [00:01<00:00, 17.18it/s]
 abstract_algebra: 100%|██████████| 20/20 [00:02<00:00, 17.57it/s]
  MMLU subjects: 2%|▏ | 1/57 [00:03<03:29, 3.73s/it][=] abstract_algebra: 6/20 = 30.0% (running avg 30.0%)
============================================================
[>] anatomy (shots=5)
 anatomy: 0%| | 0/20 [00:00<?, ?it/s]
 anatomy: 10%|█ | 2/20 [00:00<00:01, 13.20it/s]
 anatomy: 30%|███ | 6/20 [00:00<00:00, 25.74it/s]
 anatomy: 45%|████▌ | 9/20 [00:00<00:00, 24.91it/s]
 anatomy: 60%|██████ | 12/20 [00:00<00:00, 24.58it/s]
 anatomy: 75%|███████▌ | 15/20 [00:00<00:00, 23.82it/s]
 anatomy: 90%|█████████ | 18/20 [00:00<00:00, 21.69it/s]
  MMLU subjects: 4%|▎ | 2/57 [00:06<02:43, 2.97s/it][=] anatomy: 5/20 = 25.0% (running avg 27.5%)
============================================================
[>] astronomy (shots=5)
 astronomy: 0%| | 0/20 [00:00<?, ?it/s]
 astronomy: 10%|█ | 2/20 [00:00<00:01, 12.23it/s]
 astronomy: 20%|██ | 4/20 [00:00<00:01, 14.44it/s]
 astronomy: 30%|███ | 6/20 [00:00<00:00, 15.75it/s]
 astronomy: 40%|████ | 8/20 [00:00<00:00, 16.45it/s]
 astronomy: 50%|█████ | 10/20 [00:00<00:00, 16.60it/s]
 astronomy: 60%|██████ | 12/20 [00:00<00:00, 17.09it/s]
 astronomy: 70%|███████ | 14/20 [00:00<00:00, 17.48it/s]
 astronomy: 85%|████████▌ | 17/20 [00:00<00:00, 19.61it/s]
 astronomy: 95%|█████████▌| 19/20 [00:01<00:00, 19.15it/s]
  MMLU subjects: 5%|▌ | 3/57 [00:08<02:23, 2.66s/it][=] astronomy: 7/20 = 35.0% (running avg 30.0%)
============================================================
[>] business_ethics (shots=5)
 business_ethics: 0%| | 0/20 [00:00<?, ?it/s]
 business_ethics: 10%|█ | 2/20 [00:00<00:01, 12.90it/s]
 business_ethics: 20%|██ | 4/20 [00:00<00:01, 15.43it/s]
 business_ethics: 40%|████ | 8/20 [00:00<00:00, 24.39it/s]
 business_ethics: 55%|█████▌ | 11/20 [00:00<00:00, 24.16it/s]
 business_ethics: 70%|███████ | 14/20 [00:00<00:00, 23.61it/s]
 business_ethics: 85%|████████▌ | 17/20 [00:00<00:00, 23.27it/s]
 business_ethics: 100%|██████████| 20/20 [00:00<00:00, 22.96it/s]
  MMLU subjects: 7%|▋ | 4/57 [00:10<02:10, 2.45s/it][=] business_ethics: 6/20 = 30.0% (running avg 30.0%)
============================================================
[>] clinical_knowledge (shots=5)
 clinical_knowledge: 0%| | 0/20 [00:00<?, ?it/s]
 clinical_knowledge: 10%|█ | 2/20 [00:00<00:00, 18.14it/s]
 clinical_knowledge: 20%|██ | 4/20 [00:00<00:00, 18.00it/s]
 clinical_knowledge: 30%|███ | 6/20 [00:00<00:00, 17.94it/s]
 clinical_knowledge: 40%|████ | 8/20 [00:00<00:00, 17.50it/s]
 clinical_knowledge: 50%|█████ | 10/20 [00:00<00:00, 17.77it/s]
 clinical_knowledge: 60%|██████ | 12/20 [00:00<00:00, 17.75it/s]
 clinical_knowledge: 75%|███████▌ | 15/20 [00:00<00:00, 19.69it/s]
 clinical_knowledge: 90%|█████████ | 18/20 [00:00<00:00, 20.96it/s]
  MMLU subjects: 9%|▉ | 5/57 [00:12<02:05, 2.42s/it][=] clinical_knowledge: 9/20 = 45.0% (running avg 33.0%)
============================================================
[>] college_biology (shots=5)
 college_biology: 0%| | 0/20 [00:00<?, ?it/s]
 college_biology: 10%|█ | 2/20 [00:00<00:01, 13.18it/s]
 college_biology: 20%|██ | 4/20 [00:00<00:01, 15.56it/s]
 college_biology: 35%|███▌ | 7/20 [00:00<00:00, 19.08it/s]
 college_biology: 45%|████▌ | 9/20 [00:00<00:00, 18.60it/s]
 college_biology: 65%|██████▌ | 13/20 [00:00<00:00, 21.91it/s]
 college_biology: 80%|████████ | 16/20 [00:00<00:00, 22.47it/s]
 college_biology: 95%|█████████▌| 19/20 [00:00<00:00, 22.98it/s]
  MMLU subjects: 11%|█ | 6/57 [00:15<01:59, 2.35s/it][=] college_biology: 9/20 = 45.0% (running avg 35.0%)
============================================================
[>] college_chemistry (shots=5)
 college_chemistry: 0%| | 0/20 [00:00<?, ?it/s]
 college_chemistry: 10%|█ | 2/20 [00:00<00:01, 12.84it/s]
 college_chemistry: 25%|██▌ | 5/20 [00:00<00:00, 18.26it/s]
 college_chemistry: 40%|████ | 8/20 [00:00<00:00, 20.43it/s]
 college_chemistry: 55%|█████▌ | 11/20 [00:00<00:00, 19.37it/s]
 college_chemistry: 65%|██████▌ | 13/20 [00:00<00:00, 18.98it/s]
 college_chemistry: 85%|████████▌ | 17/20 [00:00<00:00, 21.96it/s]
  MMLU subjects: 12%|█▏ | 7/57 [00:17<01:56, 2.33s/it][=] college_chemistry: 3/20 = 15.0% (running avg 32.1%)
============================================================
[>] college_computer_science (shots=5)
 college_computer_science: 0%| | 0/20 [00:00<?, ?it/s]
 college_computer_science: 5%|▌ | 1/20 [00:00<00:02, 9.28it/s]
 college_computer_science: 15%|█▌ | 3/20 [00:00<00:01, 13.08it/s]
 college_computer_science: 25%|██▌ | 5/20 [00:00<00:01, 14.86it/s]
 college_computer_science: 35%|███▌ | 7/20 [00:00<00:00, 15.74it/s]
 college_computer_science: 45%|████▌ | 9/20 [00:00<00:00, 16.53it/s]
 college_computer_science: 55%|█████▌ | 11/20 [00:00<00:00, 17.01it/s]
 college_computer_science: 65%|██████▌ | 13/20 [00:00<00:00, 17.24it/s]
 college_computer_science: 75%|███████▌ | 15/20 [00:00<00:00, 17.33it/s]
 college_computer_science: 85%|████████▌ | 17/20 [00:01<00:00, 17.21it/s]
 college_computer_science: 95%|█████████▌| 19/20 [00:01<00:00, 17.23it/s]
  MMLU subjects: 14%|█▍ | 8/57 [00:19<01:55, 2.37s/it][=] college_computer_science: 9/20 = 45.0% (running avg 33.8%)
============================================================
[>] college_mathematics (shots=5)
 college_mathematics: 0%| | 0/20 [00:00<?, ?it/s]
 college_mathematics: 10%|█ | 2/20 [00:00<00:01, 12.83it/s]
 college_mathematics: 20%|██ | 4/20 [00:00<00:01, 15.33it/s]
 college_mathematics: 35%|███▌ | 7/20 [00:00<00:00, 18.80it/s]
 college_mathematics: 60%|██████ | 12/20 [00:00<00:00, 26.76it/s]
 college_mathematics: 80%|████████ | 16/20 [00:00<00:00, 26.72it/s]
 college_mathematics: 95%|█████████▌| 19/20 [00:00<00:00, 23.14it/s]
  MMLU subjects: 16%|█▌ | 9/57 [00:22<01:52, 2.34s/it][=] college_mathematics: 7/20 = 35.0% (running avg 33.9%)
============================================================
[>] college_medicine (shots=5)
 college_medicine: 0%| | 0/20 [00:00<?, ?it/s]
 college_medicine: 10%|█ | 2/20 [00:00<00:01, 17.54it/s]
 college_medicine: 20%|██ | 4/20 [00:00<00:00, 17.37it/s]
 college_medicine: 30%|███ | 6/20 [00:00<00:00, 17.41it/s]
 college_medicine: 45%|████▌ | 9/20 [00:00<00:00, 19.51it/s]
 college_medicine: 55%|█████▌ | 11/20 [00:00<00:00, 18.71it/s]
 college_medicine: 65%|██████▌ | 13/20 [00:00<00:00, 18.52it/s]
 college_medicine: 75%|███████▌ | 15/20 [00:00<00:00, 18.49it/s]
 college_medicine: 100%|██████████| 20/20 [00:00<00:00, 25.98it/s]
  MMLU subjects: 18%|█▊ | 10/57 [00:24<01:47, 2.28s/it][=] college_medicine: 6/20 = 30.0% (running avg 33.5%)
============================================================
[>] college_physics (shots=5)
 college_physics: 0%| | 0/20 [00:00<?, ?it/s]
 college_physics: 10%|█ | 2/20 [00:00<00:01, 12.81it/s]
 college_physics: 20%|██ | 4/20 [00:00<00:01, 15.22it/s]
 college_physics: 40%|████ | 8/20 [00:00<00:00, 21.09it/s]
 college_physics: 60%|██████ | 12/20 [00:00<00:00, 23.72it/s]
 college_physics: 80%|████████ | 16/20 [00:00<00:00, 27.98it/s]
 college_physics: 95%|█████████▌| 19/20 [00:00<00:00, 26.64it/s]
  MMLU subjects: 19%|█▉ | 11/57 [00:26<01:42, 2.23s/it][=] college_physics: 3/20 = 15.0% (running avg 31.8%)
============================================================
[>] computer_security (shots=5)
 computer_security: 0%| | 0/20 [00:00<?, ?it/s]
 computer_security: 10%|█ | 2/20 [00:00<00:00, 18.55it/s]
 computer_security: 20%|██ | 4/20 [00:00<00:00, 18.25it/s]
 computer_security: 35%|███▌ | 7/20 [00:00<00:00, 20.93it/s]
 computer_security: 55%|█████▌ | 11/20 [00:00<00:00, 24.12it/s]
 computer_security: 70%|███████ | 14/20 [00:00<00:00, 23.97it/s]
 computer_security: 90%|█████████ | 18/20 [00:00<00:00, 28.34it/s]
  MMLU subjects: 21%|██ | 12/57 [00:28<01:37, 2.17s/it][=] computer_security: 6/20 = 30.0% (running avg 31.7%)
============================================================
[>] conceptual_physics (shots=5)
 conceptual_physics: 0%| | 0/20 [00:00<?, ?it/s]
 conceptual_physics: 5%|▌ | 1/20 [00:00<00:02, 9.44it/s]
 conceptual_physics: 15%|█▌ | 3/20 [00:00<00:01, 14.31it/s]
 conceptual_physics: 30%|███ | 6/20 [00:00<00:00, 18.53it/s]
 conceptual_physics: 45%|████▌ | 9/20 [00:00<00:00, 20.38it/s]
 conceptual_physics: 65%|██████▌ | 13/20 [00:00<00:00, 22.74it/s]
 conceptual_physics: 85%|████████▌ | 17/20 [00:00<00:00, 24.44it/s]
 conceptual_physics: 100%|██████████| 20/20 [00:00<00:00, 24.12it/s]
  MMLU subjects: 23%|██▎ | 13/57 [00:30<01:37, 2.22s/it][=] conceptual_physics: 1/20 = 5.0% (running avg 29.6%)
============================================================
[>] econometrics (shots=5)
 econometrics: 0%| | 0/20 [00:00<?, ?it/s]
 econometrics: 20%|██ | 4/20 [00:00<00:00, 27.82it/s]
 econometrics: 35%|███▌ | 7/20 [00:00<00:00, 21.59it/s]
 econometrics: 55%|█████▌ | 11/20 [00:00<00:00, 23.65it/s]
 econometrics: 80%|████████ | 16/20 [00:00<00:00, 28.86it/s]
 econometrics: 95%|█████████▌| 19/20 [00:00<00:00, 26.62it/s]
  MMLU subjects: 25%|██▍ | 14/57 [00:32<01:33, 2.18s/it][=] econometrics: 6/20 = 30.0% (running avg 29.6%)
============================================================
[>] electrical_engineering (shots=5)
 electrical_engineering: 0%| | 0/20 [00:00<?, ?it/s]
 electrical_engineering: 10%|█ | 2/20 [00:00<00:01, 17.36it/s]
 electrical_engineering: 30%|███ | 6/20 [00:00<00:00, 23.79it/s]
 electrical_engineering: 45%|████▌ | 9/20 [00:00<00:00, 20.65it/s]
 electrical_engineering: 65%|██████▌ | 13/20 [00:00<00:00, 25.92it/s]
 electrical_engineering: 95%|█████████▌| 19/20 [00:00<00:00, 35.89it/s]
  MMLU subjects: 26%|██▋ | 15/57 [00:34<01:30, 2.15s/it][=] electrical_engineering: 4/20 = 20.0% (running avg 29.0%)
============================================================
[>] elementary_mathematics (shots=5)
 elementary_mathematics: 0%| | 0/20 [00:00<?, ?it/s]
 elementary_mathematics: 10%|█ | 2/20 [00:00<00:01, 12.90it/s]
 elementary_mathematics: 20%|██ | 4/20 [00:00<00:01, 15.49it/s]
 elementary_mathematics: 30%|███ | 6/20 [00:00<00:00, 16.54it/s]
 elementary_mathematics: 50%|█████ | 10/20 [00:00<00:00, 23.85it/s]
 elementary_mathematics: 70%|███████ | 14/20 [00:00<00:00, 28.23it/s]
 elementary_mathematics: 85%|████████▌ | 17/20 [00:00<00:00, 26.39it/s]
  MMLU subjects: 28%|██▊ | 16/57 [00:36<01:26, 2.11s/it][=] elementary_mathematics: 6/20 = 30.0% (running avg 29.1%)
============================================================
[>] formal_logic (shots=5)
 formal_logic: 0%| | 0/20 [00:00<?, ?it/s]
 formal_logic: 10%|█ | 2/20 [00:00<00:01, 13.10it/s]
 formal_logic: 20%|██ | 4/20 [00:00<00:01, 15.61it/s]
 formal_logic: 30%|███ | 6/20 [00:00<00:00, 16.51it/s]
 formal_logic: 45%|████▌ | 9/20 [00:00<00:00, 19.07it/s]
 formal_logic: 65%|██████▌ | 13/20 [00:00<00:00, 22.24it/s]
 formal_logic: 80%|████████ | 16/20 [00:00<00:00, 22.59it/s]
 formal_logic: 95%|█████████▌| 19/20 [00:00<00:00, 22.94it/s]
  MMLU subjects: 30%|██▉ | 17/57 [00:39<01:24, 2.12s/it][=] formal_logic: 2/20 = 10.0% (running avg 27.9%)
============================================================
[>] global_facts (shots=5)
 global_facts: 0%| | 0/20 [00:00<?, ?it/s]
 global_facts: 10%|█ | 2/20 [00:00<00:01, 17.09it/s]
 global_facts: 30%|███ | 6/20 [00:00<00:00, 28.49it/s]
 global_facts: 50%|█████ | 10/20 [00:00<00:00, 32.74it/s]
 global_facts: 70%|███████ | 14/20 [00:00<00:00, 34.73it/s]
  MMLU subjects: 32%|███▏ | 18/57 [00:40<01:18, 2.00s/it][=] global_facts: 7/20 = 35.0% (running avg 28.3%)
============================================================
[>] high_school_biology (shots=5)
 high_school_biology: 0%| | 0/20 [00:00<?, ?it/s]
 high_school_biology: 15%|█▌ | 3/20 [00:00<00:00, 24.19it/s]
 high_school_biology: 35%|███▌ | 7/20 [00:00<00:00, 31.92it/s]
 high_school_biology: 55%|█████▌ | 11/20 [00:00<00:00, 26.11it/s]
 high_school_biology: 75%|███████▌ | 15/20 [00:00<00:00, 26.59it/s]
 high_school_biology: 95%|█████████▌| 19/20 [00:00<00:00, 29.76it/s]
  MMLU subjects: 33%|███▎ | 19/57 [00:43<01:17, 2.04s/it][=] high_school_biology: 9/20 = 45.0% (running avg 29.2%)
============================================================
[>] high_school_chemistry (shots=5)
 high_school_chemistry: 0%| | 0/20 [00:00<?, ?it/s]
 high_school_chemistry: 10%|█ | 2/20 [00:00<00:01, 17.03it/s]
 high_school_chemistry: 40%|████ | 8/20 [00:00<00:00, 40.46it/s]
 high_school_chemistry: 65%|██████▌ | 13/20 [00:00<00:00, 40.38it/s]
 high_school_chemistry: 90%|█████████ | 18/20 [00:00<00:00, 40.22it/s]
  MMLU subjects: 35%|███▌ | 20/57 [00:44<01:13, 1.99s/it][=] high_school_chemistry: 7/20 = 35.0% (running avg 29.5%)
============================================================
[>] high_school_computer_science (shots=5)
 high_school_computer_science: 0%| | 0/20 [00:00<?, ?it/s]
 high_school_computer_science: 5%|▌ | 1/20 [00:00<00:02, 8.41it/s]
 high_school_computer_science: 25%|██▌ | 5/20 [00:00<00:00, 20.00it/s]
 high_school_computer_science: 45%|████▌ | 9/20 [00:00<00:00, 26.45it/s]
 high_school_computer_science: 60%|██████ | 12/20 [00:00<00:00, 22.36it/s]
 high_school_computer_science: 80%|████████ | 16/20 [00:00<00:00, 23.75it/s]
 high_school_computer_science: 95%|█████████▌| 19/20 [00:00<00:00, 23.32it/s]
  MMLU subjects: 37%|███▋ | 21/57 [00:46<01:12, 2.01s/it][=] high_school_computer_science: 7/20 = 35.0% (running avg 29.8%)
============================================================
[>] high_school_european_history (shots=5)
 high_school_european_history: 0%| | 0/20 [00:00<?, ?it/s]
 high_school_european_history: 15%|█▌ | 3/20 [00:00<00:00, 29.59it/s]
 high_school_european_history: 40%|████ | 8/20 [00:00<00:00, 39.83it/s]
 high_school_european_history: 65%|██████▌ | 13/20 [00:00<00:00, 42.99it/s]
 high_school_european_history: 90%|█████████ | 18/20 [00:00<00:00, 44.36it/s]
  MMLU subjects: 39%|███▊ | 22/57 [00:48<01:07, 1.94s/it][=] high_school_european_history: 4/20 = 20.0% (running avg 29.3%)
============================================================
[>] high_school_geography (shots=5)
 high_school_geography: 0%| | 0/20 [00:00<?, ?it/s]
 high_school_geography: 15%|█▌ | 3/20 [00:00<00:00, 22.09it/s]
 high_school_geography: 45%|████▌ | 9/20 [00:00<00:00, 34.56it/s]
 high_school_geography: 65%|██████▌ | 13/20 [00:00<00:00, 30.54it/s]
 high_school_geography: 85%|████████▌ | 17/20 [00:00<00:00, 32.49it/s]
  MMLU subjects: 40%|████ | 23/57 [00:50<01:05, 1.91s/it][=] high_school_geography: 5/20 = 25.0% (running avg 29.1%)
============================================================
[>] high_school_government_and_politics (shots=5)
 high_school_government_and_politics: 0%| | 0/20 [00:00<?, ?it/s]
 high_school_government_and_politics: 10%|█ | 2/20 [00:00<00:01, 14.50it/s]
 high_school_government_and_politics: 30%|███ | 6/20 [00:00<00:00, 22.09it/s]
 high_school_government_and_politics: 60%|██████ | 12/20 [00:00<00:00, 35.58it/s]
 high_school_government_and_politics: 90%|█████████ | 18/20 [00:00<00:00, 38.62it/s]
  MMLU subjects: 42%|████▏ | 24/57 [00:52<01:02, 1.89s/it][=] high_school_government_and_politics: 4/20 = 20.0% (running avg 28.7%)
============================================================
[>] high_school_macroeconomics (shots=5)
 high_school_macroeconomics: 0%| | 0/20 [00:00<?, ?it/s]
 high_school_macroeconomics: 20%|██ | 4/20 [00:00<00:00, 39.59it/s]
 high_school_macroeconomics: 40%|████ | 8/20 [00:00<00:00, 38.27it/s]
 high_school_macroeconomics: 60%|██████ | 12/20 [00:00<00:00, 32.61it/s]
 high_school_macroeconomics: 90%|█████████ | 18/20 [00:00<00:00, 37.33it/s]
  MMLU subjects: 44%|████▍ | 25/57 [00:54<00:58, 1.84s/it][=] high_school_macroeconomics: 0/20 = 0.0% (running avg 27.6%)
============================================================
[>] high_school_mathematics (shots=5)
 high_school_mathematics: 0%| | 0/20 [00:00<?, ?it/s]
 high_school_mathematics: 10%|█ | 2/20 [00:00<00:01, 12.89it/s]
 high_school_mathematics: 30%|███ | 6/20 [00:00<00:00, 21.24it/s]
 high_school_mathematics: 60%|██████ | 12/20 [00:00<00:00, 35.06it/s]
 high_school_mathematics: 90%|█████████ | 18/20 [00:00<00:00, 43.42it/s]
  MMLU subjects: 46%|████▌ | 26/57 [00:55<00:57, 1.85s/it][=] high_school_mathematics: 4/20 = 20.0% (running avg 27.3%)
============================================================
[>] high_school_microeconomics (shots=5)
 high_school_microeconomics: 0%| | 0/20 [00:00<?, ?it/s]
 high_school_microeconomics: 15%|█▌ | 3/20 [00:00<00:00, 18.39it/s]
 high_school_microeconomics: 35%|███▌ | 7/20 [00:00<00:00, 27.64it/s]
 high_school_microeconomics: 55%|█████▌ | 11/20 [00:00<00:00, 31.83it/s]
 high_school_microeconomics: 75%|███████▌ | 15/20 [00:00<00:00, 33.78it/s]
 high_school_microeconomics: 95%|█████████▌| 19/20 [00:00<00:00, 31.23it/s]
  MMLU subjects: 47%|████▋ | 27/57 [00:57<00:55, 1.86s/it][=] high_school_microeconomics: 7/20 = 35.0% (running avg 27.6%)
============================================================
[>] high_school_physics (shots=5)
 high_school_physics: 0%| | 0/20 [00:00<?, ?it/s]
 high_school_physics: 10%|█ | 2/20 [00:00<00:00, 18.31it/s]
 high_school_physics: 30%|███ | 6/20 [00:00<00:00, 29.27it/s]
 high_school_physics: 50%|█████ | 10/20 [00:00<00:00, 28.30it/s]
 high_school_physics: 70%|███████ | 14/20 [00:00<00:00, 31.21it/s]
 high_school_physics: 90%|█████████ | 18/20 [00:00<00:00, 33.23it/s]
  MMLU subjects: 49%|████▉ | 28/57 [00:59<00:55, 1.90s/it][=] high_school_physics: 4/20 = 20.0% (running avg 27.3%)
============================================================
[>] high_school_psychology (shots=5)
 high_school_psychology: 0%| | 0/20 [00:00<?, ?it/s]
 high_school_psychology: 20%|██ | 4/20 [00:00<00:00, 28.00it/s]
 high_school_psychology: 40%|████ | 8/20 [00:00<00:00, 33.07it/s]
 high_school_psychology: 60%|██████ | 12/20 [00:00<00:00, 34.99it/s]
 high_school_psychology: 80%|████████ | 16/20 [00:00<00:00, 35.93it/s]
 high_school_psychology: 100%|██████████| 20/20 [00:00<00:00, 36.42it/s]
  MMLU subjects: 51%|█████ | 29/57 [01:01<00:52, 1.87s/it][=] high_school_psychology: 5/20 = 25.0% (running avg 27.2%)
============================================================
[>] high_school_statistics (shots=5)
 high_school_statistics: 0%| | 0/20 [00:00<?, ?it/s]
 high_school_statistics: 10%|█ | 2/20 [00:00<00:01, 12.71it/s]
 high_school_statistics: 25%|██▌ | 5/20 [00:00<00:00, 18.45it/s]
 high_school_statistics: 35%|███▌ | 7/20 [00:00<00:00, 18.23it/s]
 high_school_statistics: 45%|████▌ | 9/20 [00:00<00:00, 18.11it/s]
 high_school_statistics: 55%|█████▌ | 11/20 [00:00<00:00, 17.87it/s]
 high_school_statistics: 65%|██████▌ | 13/20 [00:00<00:00, 17.72it/s]
 high_school_statistics: 75%|███████▌ | 15/20 [00:00<00:00, 17.67it/s]
 high_school_statistics: 85%|████████▌ | 17/20 [00:00<00:00, 17.60it/s]
 high_school_statistics: 95%|█████████▌| 19/20 [00:01<00:00, 17.72it/s]
  MMLU subjects: 53%|█████▎ | 30/57 [01:04<00:54, 2.04s/it][=] high_school_statistics: 8/20 = 40.0% (running avg 27.7%)
============================================================
[>] high_school_us_history (shots=5)
 high_school_us_history: 0%| | 0/20 [00:00<?, ?it/s]
 high_school_us_history: 20%|██ | 4/20 [00:00<00:00, 33.99it/s]
 high_school_us_history: 45%|████▌ | 9/20 [00:00<00:00, 42.16it/s]
 high_school_us_history: 70%|███████ | 14/20 [00:00<00:00, 44.91it/s]
 high_school_us_history: 95%|█████████▌| 19/20 [00:00<00:00, 45.94it/s]
  MMLU subjects: 54%|█████▍ | 31/57 [01:05<00:49, 1.92s/it][=] high_school_us_history: 4/20 = 20.0% (running avg 27.4%)
============================================================
[>] high_school_world_history (shots=5)
 high_school_world_history: 0%| | 0/20 [00:00<?, ?it/s]
 high_school_world_history: 15%|█▌ | 3/20 [00:00<00:00, 20.31it/s]
 high_school_world_history: 40%|████ | 8/20 [00:00<00:00, 32.41it/s]
 high_school_world_history: 65%|██████▌ | 13/20 [00:00<00:00, 39.35it/s]
 high_school_world_history: 90%|█████████ | 18/20 [00:00<00:00, 43.13it/s]
  MMLU subjects: 56%|█████▌ | 32/57 [01:07<00:46, 1.86s/it][=] high_school_world_history: 7/20 = 35.0% (running avg 27.7%)
============================================================
[>] human_aging (shots=5)
 human_aging: 0%| | 0/20 [00:00<?, ?it/s]
 human_aging: 5%|▌ | 1/20 [00:00<00:01, 9.91it/s]
 human_aging: 20%|██ | 4/20 [00:00<00:00, 18.51it/s]
 human_aging: 40%|████ | 8/20 [00:00<00:00, 23.30it/s]
 human_aging: 55%|█████▌ | 11/20 [00:00<00:00, 23.62it/s]
 human_aging: 70%|███████ | 14/20 [00:00<00:00, 23.48it/s]
 human_aging: 90%|█████████ | 18/20 [00:00<00:00, 25.27it/s]
  MMLU subjects: 58%|█████▊ | 33/57 [01:09<00:45, 1.90s/it][=] human_aging: 8/20 = 40.0% (running avg 28.0%)
============================================================
[>] human_sexuality (shots=5)
 human_sexuality: 0%| | 0/20 [00:00<?, ?it/s]
 human_sexuality: 10%|█ | 2/20 [00:00<00:01, 17.26it/s]
 human_sexuality: 20%|██ | 4/20 [00:00<00:00, 17.97it/s]
 human_sexuality: 50%|█████ | 10/20 [00:00<00:00, 31.12it/s]
 human_sexuality: 70%|███████ | 14/20 [00:00<00:00, 33.83it/s]
 human_sexuality: 90%|█████████ | 18/20 [00:00<00:00, 35.24it/s]
  MMLU subjects: 60%|█████▉ | 34/57 [01:11<00:44, 1.94s/it][=] human_sexuality: 3/20 = 15.0% (running avg 27.6%)
============================================================
[>] international_law (shots=5)
 international_law: 0%| | 0/20 [00:00<?, ?it/s]
 international_law: 10%|█ | 2/20 [00:00<00:01, 13.41it/s]
 international_law: 30%|███ | 6/20 [00:00<00:00, 21.45it/s]
 international_law: 60%|██████ | 12/20 [00:00<00:00, 34.70it/s]
 international_law: 80%|████████ | 16/20 [00:00<00:00, 28.56it/s]
 international_law: 100%|██████████| 20/20 [00:00<00:00, 25.67it/s]
  MMLU subjects: 61%|██████▏ | 35/57 [01:13<00:43, 1.99s/it][=] international_law: 7/20 = 35.0% (running avg 27.9%)
============================================================
[>] jurisprudence (shots=5)
 jurisprudence: 0%| | 0/20 [00:00<?, ?it/s]
 jurisprudence: 10%|█ | 2/20 [00:00<00:01, 17.11it/s]
 jurisprudence: 40%|████ | 8/20 [00:00<00:00, 32.97it/s]
 jurisprudence: 60%|██████ | 12/20 [00:00<00:00, 34.56it/s]
 jurisprudence: 90%|█████████ | 18/20 [00:00<00:00, 37.91it/s]
  MMLU subjects: 63%|██████▎ | 36/57 [01:15<00:40, 1.92s/it][=] jurisprudence: 8/20 = 40.0% (running avg 28.2%)
============================================================
[>] logical_fallacies (shots=5)
 logical_fallacies: 0%| | 0/20 [00:00<?, ?it/s]
 logical_fallacies: 20%|██ | 4/20 [00:00<00:00, 38.49it/s]
 logical_fallacies: 50%|█████ | 10/20 [00:00<00:00, 49.03it/s]
 logical_fallacies: 75%|███████▌ | 15/20 [00:00<00:00, 38.28it/s]
 logical_fallacies: 100%|██████████| 20/20 [00:00<00:00, 38.86it/s]
  MMLU subjects: 65%|██████▍ | 37/57 [01:17<00:37, 1.85s/it][=] logical_fallacies: 7/20 = 35.0% (running avg 28.4%)
============================================================
[>] machine_learning (shots=5)
 Downloading data: 0%| | 0.00/5.25k [00:00<?, ?B/s]
 Downloading data: 100%|██████████| 5.25k/5.25k [00:00<00:00, 19.1kB/s] Downloading data: 100%|██████████| 5.25k/5.25k [00:00<00:00, 19.0kB/s]
 Generating test split: 0%| | 0/112 [00:00<?, ? examples/s] Generating test split: 100%|██████████| 112/112 [00:00<00:00, 19509.20 examples/s]
 Generating validation split: 0%| | 0/11 [00:00<?, ? examples/s] Generating validation split: 100%|██████████| 11/11 [00:00<00:00, 3115.91 examples/s]
 Generating dev split: 0%| | 0/5 [00:00<?, ? examples/s] Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1538.18 examples/s]
 machine_learning: 0%| | 0/20 [00:00<?, ?it/s]
 machine_learning: 10%|█ | 2/20 [00:00<00:01, 13.64it/s]
 machine_learning: 30%|███ | 6/20 [00:00<00:00, 25.60it/s]
 machine_learning: 50%|█████ | 10/20 [00:00<00:00, 30.54it/s]
 machine_learning: 80%|████████ | 16/20 [00:00<00:00, 35.58it/s]
 machine_learning: 100%|██████████| 20/20 [00:00<00:00, 32.40it/s]
  MMLU subjects: 67%|██████▋ | 38/57 [01:19<00:38, 2.03s/it][=] machine_learning: 10/20 = 50.0% (running avg 28.9%)
============================================================
[>] management (shots=5)
 Downloading data: 0%| | 0.00/14.7k [00:00<?, ?B/s]
 Downloading data: 100%|██████████| 14.7k/14.7k [00:00<00:00, 71.1kB/s] Downloading data: 100%|██████████| 14.7k/14.7k [00:00<00:00, 70.8kB/s]
 Downloading data: 0%| | 0.00/4.50k [00:00<?, ?B/s]
 Downloading data: 100%|██████████| 4.50k/4.50k [00:00<00:00, 25.8kB/s] Downloading data: 100%|██████████| 4.50k/4.50k [00:00<00:00, 25.7kB/s]
 Downloading data: 0%| | 0.00/3.61k [00:00<?, ?B/s]
 Downloading data: 100%|██████████| 3.61k/3.61k [00:00<00:00, 19.8kB/s] Downloading data: 100%|██████████| 3.61k/3.61k [00:00<00:00, 19.7kB/s]
 Generating test split: 0%| | 0/103 [00:00<?, ? examples/s] Generating test split: 100%|██████████| 103/103 [00:00<00:00, 23526.29 examples/s]
 Generating validation split: 0%| | 0/11 [00:00<?, ? examples/s] Generating validation split: 100%|██████████| 11/11 [00:00<00:00, 3129.65 examples/s]
 Generating dev split: 0%| | 0/5 [00:00<?, ? examples/s] Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1433.95 examples/s]
 management: 0%| | 0/20 [00:00<?, ?it/s]
 management: 10%|█ | 2/20 [00:00<00:01, 14.08it/s]
 management: 20%|██ | 4/20 [00:00<00:00, 16.16it/s]
 management: 35%|███▌ | 7/20 [00:00<00:00, 19.49it/s]
 management: 45%|████▌ | 9/20 [00:00<00:00, 18.65it/s]
 management: 60%|██████ | 12/20 [00:00<00:00, 20.56it/s]
 management: 75%|███████▌ | 15/20 [00:00<00:00, 19.77it/s]
  MMLU subjects: 68%|██████▊ | 39/57 [01:23<00:45, 2.55s/it][=] management: 4/20 = 20.0% (running avg 28.7%)
============================================================
[>] marketing (shots=5)
 Downloading data: 0%| | 0.00/37.3k [00:00<?, ?B/s]
 Downloading data: 100%|██████████| 37.3k/37.3k [00:00<00:00, 216kB/s] Downloading data: 100%|██████████| 37.3k/37.3k [00:00<00:00, 214kB/s]
 Downloading data: 0%| | 0.00/8.21k [00:00<?, ?B/s]
 Downloading data: 100%|██████████| 8.21k/8.21k [00:00<00:00, 40.3kB/s] Downloading data: 100%|██████████| 8.21k/8.21k [00:00<00:00, 40.1kB/s]
 Downloading data: 0%| | 0.00/4.28k [00:00<?, ?B/s]
 Downloading data: 100%|██████████| 4.28k/4.28k [00:00<00:00, 13.5kB/s] Downloading data: 100%|██████████| 4.28k/4.28k [00:00<00:00, 13.5kB/s]
 Generating test split: 0%| | 0/234 [00:00<?, ? examples/s] Generating test split: 100%|██████████| 234/234 [00:00<00:00, 47706.56 examples/s]
 Generating validation split: 0%| | 0/25 [00:00<?, ? examples/s] Generating validation split: 100%|██████████| 25/25 [00:00<00:00, 6806.73 examples/s]
 Generating dev split: 0%| | 0/5 [00:00<?, ? examples/s] Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1449.31 examples/s]
 marketing: 0%| | 0/20 [00:00<?, ?it/s]
 marketing: 25%|██▌ | 5/20 [00:00<00:00, 43.68it/s]
 marketing: 55%|█████▌ | 11/20 [00:00<00:00, 52.50it/s]
 marketing: 85%|████████▌ | 17/20 [00:00<00:00, 55.39it/s]
  MMLU subjects: 70%|███████ | 40/57 [01:26<00:47, 2.77s/it][=] marketing: 7/20 = 35.0% (running avg 28.9%)
============================================================
[>] medical_genetics (shots=5)
 Downloading data: 0%| | 0.00/16.4k [00:00<?, ?B/s]
 Downloading data: 100%|██████████| 16.4k/16.4k [00:00<00:00, 84.6kB/s] Downloading data: 100%|██████████| 16.4k/16.4k [00:00<00:00, 84.2kB/s]
 Downloading data: 0%| | 0.00/5.63k [00:00<?, ?B/s]
 Downloading data: 100%|██████████| 5.63k/5.63k [00:00<00:00, 25.0kB/s] Downloading data: 100%|██████████| 5.63k/5.63k [00:00<00:00, 24.9kB/s]
 Downloading data: 0%| | 0.00/3.77k [00:00<?, ?B/s]
 Downloading data: 100%|██████████| 3.77k/3.77k [00:00<00:00, 22.5kB/s] Downloading data: 100%|██████████| 3.77k/3.77k [00:00<00:00, 22.4kB/s]
 Generating test split: 0%| | 0/100 [00:00<?, ? examples/s] Generating test split: 100%|██████████| 100/100 [00:00<00:00, 23050.69 examples/s]
 Generating validation split: 0%| | 0/11 [00:00<?, ? examples/s] Generating validation split: 100%|██████████| 11/11 [00:00<00:00, 3044.36 examples/s]
 Generating dev split: 0%| | 0/5 [00:00<?, ? examples/s] Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1133.23 examples/s]
 medical_genetics: 0%| | 0/20 [00:00<?, ?it/s]
 medical_genetics: 15%|█▌ | 3/20 [00:00<00:00, 25.31it/s]
 medical_genetics: 50%|█████ | 10/20 [00:00<00:00, 45.69it/s]
 medical_genetics: 75%|███████▌ | 15/20 [00:00<00:00, 42.92it/s]
  MMLU subjects: 72%|███████▏ | 41/57 [01:29<00:47, 2.94s/it][=] medical_genetics: 8/20 = 40.0% (running avg 29.1%)
============================================================
[>] miscellaneous (shots=5)
 Downloading data: 0%| | 0.00/98.6k [00:00<?, ?B/s]
 Downloading data: 100%|██████████| 98.6k/98.6k [00:00<00:00, 387kB/s] Downloading data: 100%|██████████| 98.6k/98.6k [00:00<00:00, 386kB/s]
 Downloading data: 0%| | 0.00/13.2k [00:00<?, ?B/s]
 Downloading data: 100%|██████████| 13.2k/13.2k [00:00<00:00, 73.2kB/s] Downloading data: 100%|██████████| 13.2k/13.2k [00:00<00:00, 72.8kB/s]
 Downloading data: 0%| | 0.00/3.37k [00:00<?, ?B/s]
 Downloading data: 100%|██████████| 3.37k/3.37k [00:00<00:00, 18.5kB/s] Downloading data: 100%|██████████| 3.37k/3.37k [00:00<00:00, 18.4kB/s]
 Generating test split: 0%| | 0/783 [00:00<?, ? examples/s] Generating test split: 100%|██████████| 783/783 [00:00<00:00, 137394.47 examples/s]
 Generating validation split: 0%| | 0/86 [00:00<?, ? examples/s] Generating validation split: 100%|██████████| 86/86 [00:00<00:00, 21921.01 examples/s]
 Generating dev split: 0%| | 0/5 [00:00<?, ? examples/s] Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1380.52 examples/s]
 miscellaneous: 0%| | 0/20 [00:00<?, ?it/s]
 miscellaneous: 15%|█▌ | 3/20 [00:00<00:00, 26.43it/s]
 miscellaneous: 35%|███▌ | 7/20 [00:00<00:00, 33.14it/s]
 miscellaneous: 55%|█████▌ | 11/20 [00:00<00:00, 23.92it/s]
 miscellaneous: 70%|███████ | 14/20 [00:00<00:00, 23.88it/s]
 miscellaneous: 90%|█████████ | 18/20 [00:00<00:00, 28.17it/s]
  MMLU subjects: 74%|███████▎ | 42/57 [01:33<00:46, 3.09s/it][=] miscellaneous: 6/20 = 30.0% (running avg 29.2%)
============================================================
[>] moral_disputes (shots=5)
 Downloading data: 0%| | 0.00/60.9k [00:00<?, ?B/s]
 Downloading data: 100%|██████████| 60.9k/60.9k [00:00<00:00, 299kB/s] Downloading data: 100%|██████████| 60.9k/60.9k [00:00<00:00, 298kB/s]
 Downloading data: 0%| | 0.00/10.7k [00:00<?, ?B/s]
 Downloading data: 100%|██████████| 10.7k/10.7k [00:00<00:00, 51.6kB/s] Downloading data: 100%|██████████| 10.7k/10.7k [00:00<00:00, 51.4kB/s]
 Downloading data: 0%| | 0.00/4.41k [00:00<?, ?B/s]
 Downloading data: 100%|██████████| 4.41k/4.41k [00:00<00:00, 26.7kB/s] Downloading data: 100%|██████████| 4.41k/4.41k [00:00<00:00, 26.5kB/s]
 Generating test split: 0%| | 0/346 [00:00<?, ? examples/s] Generating test split: 100%|██████████| 346/346 [00:00<00:00, 62921.83 examples/s]
 Generating validation split: 0%| | 0/38 [00:00<?, ? examples/s] Generating validation split: 100%|██████████| 38/38 [00:00<00:00, 9795.56 examples/s]
 Generating dev split: 0%| | 0/5 [00:00<?, ? examples/s] Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1382.98 examples/s]
 moral_disputes: 0%| | 0/20 [00:00<?, ?it/s]
 moral_disputes: 25%|██▌ | 5/20 [00:00<00:00, 43.64it/s]
 moral_disputes: 50%|█████ | 10/20 [00:00<00:00, 41.80it/s]
 moral_disputes: 80%|████████ | 16/20 [00:00<00:00, 49.01it/s]
  MMLU subjects: 75%|███████▌ | 43/57 [01:36<00:43, 3.11s/it][=] moral_disputes: 4/20 = 20.0% (running avg 29.0%)
============================================================
[>] moral_scenarios (shots=5)
 Downloading data: 0%| | 0.00/89.8k [00:00<?, ?B/s]
 Downloading data: 100%|██████████| 89.8k/89.8k [00:00<00:00, 502kB/s] Downloading data: 100%|██████████| 89.8k/89.8k [00:00<00:00, 499kB/s]
 Downloading data: 0%| | 0.00/14.9k [00:00<?, ?B/s]
 Downloading data: 100%|██████████| 14.9k/14.9k [00:00<00:00, 97.2kB/s] Downloading data: 100%|██████████| 14.9k/14.9k [00:00<00:00, 96.7kB/s]
 Downloading data: 0%| | 0.00/5.14k [00:00<?, ?B/s]
 Downloading data: 100%|██████████| 5.14k/5.14k [00:00<00:00, 27.5kB/s] Downloading data: 100%|██████████| 5.14k/5.14k [00:00<00:00, 27.4kB/s]
 Generating test split: 0%| | 0/895 [00:00<?, ? examples/s] Generating test split: 100%|██████████| 895/895 [00:00<00:00, 133752.66 examples/s]
 Generating validation split: 0%| | 0/100 [00:00<?, ? examples/s] Generating validation split: 100%|██████████| 100/100 [00:00<00:00, 20927.57 examples/s]
 Generating dev split: 0%| | 0/5 [00:00<?, ? examples/s] Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1409.47 examples/s]
 moral_scenarios: 0%| | 0/20 [00:00<?, ?it/s]
 moral_scenarios: 15%|█▌ | 3/20 [00:00<00:00, 24.61it/s]
 moral_scenarios: 35%|███▌ | 7/20 [00:00<00:00, 31.12it/s]
 moral_scenarios: 60%|██████ | 12/20 [00:00<00:00, 34.85it/s]
 moral_scenarios: 80%|████████ | 16/20 [00:00<00:00, 35.45it/s]
  MMLU subjects: 77%|███████▋ | 44/57 [01:39<00:40, 3.15s/it][=] moral_scenarios: 3/20 = 15.0% (running avg 28.6%)
============================================================
[>] nutrition (shots=5)
 Downloading data: 0%| | 0.00/55.0k [00:00<?, ?B/s]
 Downloading data: 100%|██████████| 55.0k/55.0k [00:00<00:00, 173kB/s] Downloading data: 100%|██████████| 55.0k/55.0k [00:00<00:00, 172kB/s]
 Downloading data: 0%| | 0.00/9.02k [00:00<?, ?B/s]
 Downloading data: 100%|██████████| 9.02k/9.02k [00:00<00:00, 58.6kB/s] Downloading data: 100%|██████████| 9.02k/9.02k [00:00<00:00, 58.3kB/s]
 Downloading data: 0%| | 0.00/4.99k [00:00<?, ?B/s]
 Downloading data: 100%|██████████| 4.99k/4.99k [00:00<00:00, 28.0kB/s] Downloading data: 100%|██████████| 4.99k/4.99k [00:00<00:00, 27.9kB/s]
 Generating test split: 0%| | 0/306 [00:00<?, ? examples/s] Generating test split: 100%|██████████| 306/306 [00:00<00:00, 60429.26 examples/s]
 Generating validation split: 0%| | 0/33 [00:00<?, ? examples/s] Generating validation split: 100%|██████████| 33/33 [00:00<00:00, 9086.33 examples/s]
 Generating dev split: 0%| | 0/5 [00:00<?, ? examples/s] Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1449.21 examples/s]
 nutrition: 0%| | 0/20 [00:00<?, ?it/s]
 nutrition: 10%|█ | 2/20 [00:00<00:00, 19.34it/s]
 nutrition: 30%|███ | 6/20 [00:00<00:00, 24.76it/s]
 nutrition: 45%|████▌ | 9/20 [00:00<00:00, 24.27it/s]
 nutrition: 65%|██████▌ | 13/20 [00:00<00:00, 25.76it/s]
 nutrition: 85%|████████▌ | 17/20 [00:00<00:00, 29.38it/s]
  MMLU subjects: 79%|███████▉ | 45/57 [01:43<00:40, 3.34s/it][=] nutrition: 4/20 = 20.0% (running avg 28.4%)
============================================================
[>] philosophy (shots=5)
 philosophy: 0%| | 0/20 [00:00<?, ?it/s]
 philosophy: 10%|█ | 2/20 [00:00<00:01, 17.21it/s]
 philosophy: 45%|████▌ | 9/20 [00:00<00:00, 43.07it/s]
 philosophy: 80%|████████ | 16/20 [00:00<00:00, 51.44it/s]
  MMLU subjects: 81%|████████ | 46/57 [01:45<00:31, 2.87s/it][=] philosophy: 3/20 = 15.0% (running avg 28.2%)
============================================================
[>] prehistory (shots=5)
 Downloading data: 0%| | 0.00/54.3k [00:00<?, ?B/s]
 Downloading data: 100%|██████████| 54.3k/54.3k [00:00<00:00, 336kB/s] Downloading data: 100%|██████████| 54.3k/54.3k [00:00<00:00, 334kB/s]
 Downloading data: 0%| | 0.00/9.89k [00:00<?, ?B/s]
 Downloading data: 100%|██████████| 9.89k/9.89k [00:00<00:00, 50.2kB/s] Downloading data: 100%|██████████| 9.89k/9.89k [00:00<00:00, 49.9kB/s]
 Downloading data: 0%| | 0.00/4.62k [00:00<?, ?B/s]
 Downloading data: 100%|██████████| 4.62k/4.62k [00:00<00:00, 21.8kB/s] Downloading data: 100%|██████████| 4.62k/4.62k [00:00<00:00, 21.7kB/s]
 Generating test split: 0%| | 0/324 [00:00<?, ? examples/s] Generating test split: 100%|██████████| 324/324 [00:00<00:00, 63532.23 examples/s]
 Generating validation split: 0%| | 0/35 [00:00<?, ? examples/s] Generating validation split: 100%|██████████| 35/35 [00:00<00:00, 9094.33 examples/s]
 Generating dev split: 0%| | 0/5 [00:00<?, ? examples/s] Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1395.22 examples/s]
 prehistory: 0%| | 0/20 [00:00<?, ?it/s]
 prehistory: 25%|██▌ | 5/20 [00:00<00:00, 43.02it/s]
 prehistory: 50%|█████ | 10/20 [00:00<00:00, 41.15it/s]
 prehistory: 75%|███████▌ | 15/20 [00:00<00:00, 35.37it/s]
  MMLU subjects: 82%|████████▏ | 47/57 [01:48<00:30, 3.04s/it][=] prehistory: 5/20 = 25.0% (running avg 28.1%)
============================================================
[>] professional_accounting (shots=5)
 Downloading data: 0%| | 0.00/69.5k [00:00<?, ?B/s]
 Downloading data: 100%|██████████| 69.5k/69.5k [00:00<00:00, 356kB/s] Downloading data: 100%|██████████| 69.5k/69.5k [00:00<00:00, 354kB/s]
 Downloading data: 0%| | 0.00/12.9k [00:00<?, ?B/s]
 Downloading data: 100%|██████████| 12.9k/12.9k [00:00<00:00, 61.9kB/s] Downloading data: 100%|██████████| 12.9k/12.9k [00:00<00:00, 61.7kB/s]
 Downloading data: 0%| | 0.00/4.89k [00:00<?, ?B/s]
 Downloading data: 100%|██████████| 4.89k/4.89k [00:00<00:00, 27.4kB/s] Downloading data: 100%|██████████| 4.89k/4.89k [00:00<00:00, 27.3kB/s]
 Generating test split: 0%| | 0/282 [00:00<?, ? examples/s] Generating test split: 100%|██████████| 282/282 [00:00<00:00, 54398.83 examples/s]
 Generating validation split: 0%| | 0/31 [00:00<?, ? examples/s] Generating validation split: 100%|██████████| 31/31 [00:00<00:00, 8148.87 examples/s]
 Generating dev split: 0%| | 0/5 [00:00<?, ? examples/s] Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1535.93 examples/s]
 professional_accounting: 0%| | 0/20 [00:00<?, ?it/s]
 professional_accounting: 10%|█ | 2/20 [00:00<00:00, 18.76it/s]
 professional_accounting: 30%|███ | 6/20 [00:00<00:00, 24.19it/s]
 professional_accounting: 50%|█████ | 10/20 [00:00<00:00, 29.18it/s]
 professional_accounting: 70%|███████ | 14/20 [00:00<00:00, 31.16it/s]
 professional_accounting: 95%|█████████▌| 19/20 [00:00<00:00, 33.78it/s]
  MMLU subjects: 84%|████████▍ | 48/57 [01:52<00:28, 3.19s/it][=] professional_accounting: 6/20 = 30.0% (running avg 28.1%)
============================================================
[>] professional_law (shots=5)
 Downloading data: 0%| | 0.00/1.04M [00:00<?, ?B/s]
 Downloading data: 100%|██████████| 1.04M/1.04M [00:00<00:00, 3.96MB/s] Downloading data: 100%|██████████| 1.04M/1.04M [00:00<00:00, 3.93MB/s]
 Downloading data: 0%| | 0.00/116k [00:00<?, ?B/s]
 Downloading data: 100%|██████████| 116k/116k [00:00<00:00, 603kB/s] Downloading data: 100%|██████████| 116k/116k [00:00<00:00, 600kB/s]
 Downloading data: 0%| | 0.00/15.1k [00:00<?, ?B/s]
 Downloading data: 100%|██████████| 15.1k/15.1k [00:00<00:00, 90.5kB/s] Downloading data: 100%|██████████| 15.1k/15.1k [00:00<00:00, 90.1kB/s]
 Generating test split: 0%| | 0/1534 [00:00<?, ? examples/s] Generating test split: 100%|██████████| 1534/1534 [00:00<00:00, 72607.74 examples/s]
 Generating validation split: 0%| | 0/170 [00:00<?, ? examples/s] Generating validation split: 100%|██████████| 170/170 [00:00<00:00, 29159.27 examples/s]
 Generating dev split: 0%| | 0/5 [00:00<?, ? examples/s] Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1370.87 examples/s]
 professional_law: 0%| | 0/20 [00:00<?, ?it/s]
 professional_law: 20%|██ | 4/20 [00:00<00:00, 36.28it/s]
 professional_law: 50%|█████ | 10/20 [00:00<00:00, 44.88it/s]
 professional_law: 75%|███████▌ | 15/20 [00:00<00:00, 46.92it/s]
  MMLU subjects: 86%|████████▌ | 49/57 [01:55<00:25, 3.17s/it][=] professional_law: 7/20 = 35.0% (running avg 28.3%)
============================================================
[>] professional_medicine (shots=5)
 Downloading data: 0%| | 0.00/125k [00:00<?, ?B/s]
 Downloading data: 100%|██████████| 125k/125k [00:00<00:00, 666kB/s] Downloading data: 100%|██████████| 125k/125k [00:00<00:00, 663kB/s]
 Downloading data: 0%| | 0.00/19.9k [00:00<?, ?B/s]
 Downloading data: 100%|██████████| 19.9k/19.9k [00:00<00:00, 134kB/s] Downloading data: 100%|██████████| 19.9k/19.9k [00:00<00:00, 134kB/s]
 Downloading data: 0%| | 0.00/8.45k [00:00<?, ?B/s]
 Downloading data: 100%|██████████| 8.45k/8.45k [00:00<00:00, 52.0kB/s] Downloading data: 100%|██████████| 8.45k/8.45k [00:00<00:00, 51.7kB/s]
 Generating test split: 0%| | 0/272 [00:00<?, ? examples/s] Generating test split: 100%|██████████| 272/272 [00:00<00:00, 37004.56 examples/s]
 Generating validation split: 0%| | 0/31 [00:00<?, ? examples/s] Generating validation split: 100%|██████████| 31/31 [00:00<00:00, 7737.65 examples/s]
 Generating dev split: 0%| | 0/5 [00:00<?, ? examples/s] Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1409.19 examples/s]
 professional_medicine: 0%| | 0/20 [00:00<?, ?it/s]
 professional_medicine: 20%|██ | 4/20 [00:00<00:00, 37.94it/s]
 professional_medicine: 50%|█████ | 10/20 [00:00<00:00, 47.05it/s]
 professional_medicine: 80%|████████ | 16/20 [00:00<00:00, 49.68it/s]
  MMLU subjects: 88%|████████▊ | 50/57 [01:58<00:21, 3.13s/it][=] professional_medicine: 1/20 = 5.0% (running avg 27.8%)
============================================================
[>] professional_psychology (shots=5)
 Downloading data: 0%| | 0.00/133k [00:00<?, ?B/s]
 Downloading data: 100%|██████████| 133k/133k [00:00<00:00, 599kB/s] Downloading data: 100%|██████████| 133k/133k [00:00<00:00, 596kB/s]
 Downloading data: 0%| | 0.00/22.1k [00:00<?, ?B/s]
 Downloading data: 100%|██████████| 22.1k/22.1k [00:00<00:00, 81.8kB/s] Downloading data: 100%|██████████| 22.1k/22.1k [00:00<00:00, 81.6kB/s]
 Downloading data: 0%| | 0.00/4.69k [00:00<?, ?B/s]
 Downloading data: 100%|██████████| 4.69k/4.69k [00:00<00:00, 30.9kB/s] Downloading data: 100%|██████████| 4.69k/4.69k [00:00<00:00, 30.8kB/s]
 Generating test split: 0%| | 0/612 [00:00<?, ? examples/s] Generating test split: 100%|██████████| 612/612 [00:00<00:00, 95884.13 examples/s]
 Generating validation split: 0%| | 0/69 [00:00<?, ? examples/s] Generating validation split: 100%|██████████| 69/69 [00:00<00:00, 16709.41 examples/s]
 Generating dev split: 0%| | 0/5 [00:00<?, ? examples/s] Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1311.87 examples/s]
 professional_psychology: 0%| | 0/20 [00:00<?, ?it/s]
 professional_psychology: 20%|██ | 4/20 [00:00<00:00, 36.84it/s]
 professional_psychology: 40%|████ | 8/20 [00:00<00:00, 36.65it/s]
 professional_psychology: 60%|██████ | 12/20 [00:00<00:00, 27.62it/s]
 professional_psychology: 80%|████████ | 16/20 [00:00<00:00, 30.48it/s]
 professional_psychology: 100%|██████████| 20/20 [00:00<00:00, 29.12it/s]
  MMLU subjects: 89%|████████▉ | 51/57 [02:01<00:19, 3.24s/it][=] professional_psychology: 9/20 = 45.0% (running avg 28.1%)
============================================================
[>] public_relations (shots=5)
 Downloading data: 0%| | 0.00/20.6k [00:00<?, ?B/s]
 Downloading data: 100%|██████████| 20.6k/20.6k [00:00<00:00, 125kB/s] Downloading data: 100%|██████████| 20.6k/20.6k [00:00<00:00, 125kB/s]
 Downloading data: 0%| | 0.00/6.45k [00:00<?, ?B/s]
 Downloading data: 100%|██████████| 6.45k/6.45k [00:00<00:00, 35.2kB/s] Downloading data: 100%|██████████| 6.45k/6.45k [00:00<00:00, 35.0kB/s]
 Downloading data: 0%| | 0.00/4.43k [00:00<?, ?B/s]
 Downloading data: 100%|██████████| 4.43k/4.43k [00:00<00:00, 29.9kB/s] Downloading data: 100%|██████████| 4.43k/4.43k [00:00<00:00, 29.7kB/s]
 Generating test split: 0%| | 0/110 [00:00<?, ? examples/s] Generating test split: 100%|██████████| 110/110 [00:00<00:00, 24437.15 examples/s]
 Generating validation split: 0%| | 0/12 [00:00<?, ? examples/s] Generating validation split: 100%|██████████| 12/12 [00:00<00:00, 3341.85 examples/s]
 Generating dev split: 0%| | 0/5 [00:00<?, ? examples/s] Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1466.54 examples/s]
 public_relations: 0%| | 0/20 [00:00<?, ?it/s]
 public_relations: 10%|█ | 2/20 [00:00<00:00, 19.12it/s]
 public_relations: 30%|███ | 6/20 [00:00<00:00, 29.51it/s]
 public_relations: 55%|█████▌ | 11/20 [00:00<00:00, 34.93it/s]
 public_relations: 75%|███████▌ | 15/20 [00:00<00:00, 35.89it/s]
  MMLU subjects: 91%|█████████ | 52/57 [02:05<00:16, 3.20s/it][=] public_relations: 9/20 = 45.0% (running avg 28.5%)
============================================================
[>] security_studies (shots=5)
 Downloading data: 0%| | 0.00/114k [00:00<?, ?B/s]
 Downloading data: 100%|██████████| 114k/114k [00:00<00:00, 675kB/s] Downloading data: 100%|██████████| 114k/114k [00:00<00:00, 671kB/s]
 Downloading data: 0%| | 0.00/18.7k [00:00<?, ?B/s]
 Downloading data: 100%|██████████| 18.7k/18.7k [00:00<00:00, 106kB/s] Downloading data: 100%|██████████| 18.7k/18.7k [00:00<00:00, 106kB/s]
 Downloading data: 0%| | 0.00/7.49k [00:00<?, ?B/s]
 Downloading data: 100%|██████████| 7.49k/7.49k [00:00<00:00, 45.0kB/s] Downloading data: 100%|██████████| 7.49k/7.49k [00:00<00:00, 44.8kB/s]
 Generating test split: 0%| | 0/245 [00:00<?, ? examples/s] Generating test split: 100%|██████████| 245/245 [00:00<00:00, 37800.42 examples/s]
 Generating validation split: 0%| | 0/27 [00:00<?, ? examples/s] Generating validation split: 100%|██████████| 27/27 [00:00<00:00, 6508.78 examples/s]
 Generating dev split: 0%| | 0/5 [00:00<?, ? examples/s] Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1451.42 examples/s]
 security_studies: 0%| | 0/20 [00:00<?, ?it/s]
 security_studies: 20%|██ | 4/20 [00:00<00:00, 37.50it/s]
 security_studies: 50%|█████ | 10/20 [00:00<00:00, 46.09it/s]
 security_studies: 80%|████████ | 16/20 [00:00<00:00, 48.87it/s]
  MMLU subjects: 93%|█████████▎| 53/57 [02:08<00:12, 3.18s/it][=] security_studies: 5/20 = 25.0% (running avg 28.4%)
============================================================
[>] sociology (shots=5)
 Downloading data: 0%| | 0.00/43.9k [00:00<?, ?B/s]
 Downloading data: 100%|██████████| 43.9k/43.9k [00:00<00:00, 228kB/s] Downloading data: 100%|██████████| 43.9k/43.9k [00:00<00:00, 227kB/s]
 Downloading data: 0%| | 0.00/8.36k [00:00<?, ?B/s]
 Downloading data: 100%|██████████| 8.36k/8.36k [00:00<00:00, 51.2kB/s] Downloading data: 100%|██████████| 8.36k/8.36k [00:00<00:00, 50.9kB/s]
 Downloading data: 0%| | 0.00/4.21k [00:00<?, ?B/s]
 Downloading data: 100%|██████████| 4.21k/4.21k [00:00<00:00, 27.3kB/s] Downloading data: 100%|██████████| 4.21k/4.21k [00:00<00:00, 27.1kB/s]
 Generating test split: 0%| | 0/201 [00:00<?, ? examples/s] Generating test split: 100%|██████████| 201/201 [00:00<00:00, 40145.48 examples/s]
 Generating validation split: 0%| | 0/22 [00:00<?, ? examples/s] Generating validation split: 100%|██████████| 22/22 [00:00<00:00, 5844.98 examples/s]
 Generating dev split: 0%| | 0/5 [00:00<?, ? examples/s] Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1353.79 examples/s]
 sociology: 0%| | 0/20 [00:00<?, ?it/s]
 sociology: 10%|█ | 2/20 [00:00<00:01, 13.70it/s]
 sociology: 30%|███ | 6/20 [00:00<00:00, 25.24it/s]
 sociology: 60%|██████ | 12/20 [00:00<00:00, 38.57it/s]
 sociology: 90%|█████████ | 18/20 [00:00<00:00, 45.52it/s]
  MMLU subjects: 95%|█████████▍| 54/57 [02:11<00:09, 3.13s/it][=] sociology: 4/20 = 20.0% (running avg 28.2%)
============================================================
[>] us_foreign_policy (shots=5)
 Downloading data: 0%| | 0.00/19.5k [00:00<?, ?B/s]
 Downloading data: 100%|██████████| 19.5k/19.5k [00:00<00:00, 114kB/s] Downloading data: 100%|██████████| 19.5k/19.5k [00:00<00:00, 113kB/s]
 Downloading data: 0%| | 0.00/5.27k [00:00<?, ?B/s]
 Downloading data: 100%|██████████| 5.27k/5.27k [00:00<00:00, 34.0kB/s] Downloading data: 100%|██████████| 5.27k/5.27k [00:00<00:00, 33.8kB/s]
 Downloading data: 0%| | 0.00/4.22k [00:00<?, ?B/s]
 Downloading data: 100%|██████████| 4.22k/4.22k [00:00<00:00, 22.7kB/s] Downloading data: 100%|██████████| 4.22k/4.22k [00:00<00:00, 22.6kB/s]
 Generating test split: 0%| | 0/100 [00:00<?, ? examples/s] Generating test split: 100%|██████████| 100/100 [00:00<00:00, 23196.02 examples/s]
 Generating validation split: 0%| | 0/11 [00:00<?, ? examples/s] Generating validation split: 100%|██████████| 11/11 [00:00<00:00, 2971.81 examples/s]
 Generating dev split: 0%| | 0/5 [00:00<?, ? examples/s] Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1376.35 examples/s]
 us_foreign_policy: 0%| | 0/20 [00:00<?, ?it/s]
 us_foreign_policy: 25%|██▌ | 5/20 [00:00<00:00, 44.09it/s]
 us_foreign_policy: 55%|█████▌ | 11/20 [00:00<00:00, 52.64it/s]
 us_foreign_policy: 85%|████████▌ | 17/20 [00:00<00:00, 55.54it/s]
  MMLU subjects: 96%|█████████▋| 55/57 [02:14<00:06, 3.20s/it][=] us_foreign_policy: 5/20 = 25.0% (running avg 28.2%)
============================================================
[>] virology (shots=5)
 Downloading data: 0%| | 0.00/27.3k [00:00<?, ?B/s]
 Downloading data: 100%|██████████| 27.3k/27.3k [00:00<00:00, 176kB/s] Downloading data: 100%|██████████| 27.3k/27.3k [00:00<00:00, 175kB/s]
 Downloading data: 0%| | 0.00/7.05k [00:00<?, ?B/s]
 Downloading data: 100%|██████████| 7.05k/7.05k [00:00<00:00, 42.4kB/s] Downloading data: 100%|██████████| 7.05k/7.05k [00:00<00:00, 42.2kB/s]
 Downloading data: 0%| | 0.00/3.87k [00:00<?, ?B/s]
 Downloading data: 100%|██████████| 3.87k/3.87k [00:00<00:00, 23.4kB/s] Downloading data: 100%|██████████| 3.87k/3.87k [00:00<00:00, 23.3kB/s]
 Generating test split: 0%| | 0/166 [00:00<?, ? examples/s] Generating test split: 100%|██████████| 166/166 [00:00<00:00, 42164.02 examples/s]
 Generating validation split: 0%| | 0/18 [00:00<?, ? examples/s] Generating validation split: 100%|██████████| 18/18 [00:00<00:00, 5863.43 examples/s]
 Generating dev split: 0%| | 0/5 [00:00<?, ? examples/s] Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1653.77 examples/s]
 virology: 0%| | 0/20 [00:00<?, ?it/s]
 virology: 15%|█▌ | 3/20 [00:00<00:00, 27.33it/s]
 virology: 45%|████▌ | 9/20 [00:00<00:00, 45.23it/s]
 virology: 70%|███████ | 14/20 [00:00<00:00, 42.99it/s]
 virology: 100%|██████████| 20/20 [00:00<00:00, 48.97it/s]
  MMLU subjects: 98%|█████████▊| 56/57 [02:17<00:03, 3.17s/it][=] virology: 5/20 = 25.0% (running avg 28.1%)
============================================================
[>] world_religions (shots=5)
 Downloading data: 0%| | 0.00/18.9k [00:00<?, ?B/s]
 Downloading data: 100%|██████████| 18.9k/18.9k [00:00<00:00, 106kB/s] Downloading data: 100%|██████████| 18.9k/18.9k [00:00<00:00, 106kB/s]
 Downloading data: 0%| | 0.00/4.94k [00:00<?, ?B/s]
 Downloading data: 100%|██████████| 4.94k/4.94k [00:00<00:00, 30.3kB/s] Downloading data: 100%|██████████| 4.94k/4.94k [00:00<00:00, 30.2kB/s]
 Downloading data: 0%| | 0.00/3.30k [00:00<?, ?B/s]
 Downloading data: 100%|██████████| 3.30k/3.30k [00:00<00:00, 21.2kB/s] Downloading data: 100%|██████████| 3.30k/3.30k [00:00<00:00, 21.1kB/s]
 Generating test split: 0%| | 0/171 [00:00<?, ? examples/s] Generating test split: 100%|██████████| 171/171 [00:00<00:00, 39673.97 examples/s]
 Generating validation split: 0%| | 0/19 [00:00<?, ? examples/s] Generating validation split: 100%|██████████| 19/19 [00:00<00:00, 5365.37 examples/s]
 Generating dev split: 0%| | 0/5 [00:00<?, ? examples/s] Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1399.69 examples/s]
 world_religions: 0%| | 0/20 [00:00<?, ?it/s]
 world_religions: 10%|█ | 2/20 [00:00<00:01, 13.20it/s]
 world_religions: 25%|██▌ | 5/20 [00:00<00:00, 18.83it/s]
 world_religions: 40%|████ | 8/20 [00:00<00:00, 20.93it/s]
 world_religions: 60%|██████ | 12/20 [00:00<00:00, 23.89it/s]
 world_religions: 75%|███████▌ | 15/20 [00:00<00:00, 23.91it/s]
 world_religions: 95%|█████████▌| 19/20 [00:00<00:00, 28.23it/s]
  MMLU subjects: 100%|██████████| 57/57 [02:20<00:00, 3.22s/it] MMLU subjects: 100%|██████████| 57/57 [02:20<00:00, 2.47s/it]
[=] world_religions: 3/20 = 15.0% (running avg 27.9%)
============================================================
MMLU RESULT — /home/zeus/microllm2/microllm2-checkpoints/final_merged
Shots: 5 Subjects: 57/57
 abstract_algebra 30.0% (6/20)
 anatomy 25.0% (5/20)
 astronomy 35.0% (7/20)
 business_ethics 30.0% (6/20)
 clinical_knowledge 45.0% (9/20)
 college_biology 45.0% (9/20)
 college_chemistry 15.0% (3/20)
 college_computer_science 45.0% (9/20)
 college_mathematics 35.0% (7/20)
 college_medicine 30.0% (6/20)
 college_physics 15.0% (3/20)
 computer_security 30.0% (6/20)
 conceptual_physics 5.0% (1/20)
 econometrics 30.0% (6/20)
 electrical_engineering 20.0% (4/20)
 elementary_mathematics 30.0% (6/20)
 formal_logic 10.0% (2/20)
 global_facts 35.0% (7/20)
 high_school_biology 45.0% (9/20)
 high_school_chemistry 35.0% (7/20)
 high_school_computer_science 35.0% (7/20)
 high_school_european_history 20.0% (4/20)
 high_school_geography 25.0% (5/20)
 high_school_government_and_politics 20.0% (4/20)
 high_school_macroeconomics 0.0% (0/20)
 high_school_mathematics 20.0% (4/20)
 high_school_microeconomics 35.0% (7/20)
 high_school_physics 20.0% (4/20)
 high_school_psychology 25.0% (5/20)
 high_school_statistics 40.0% (8/20)
 high_school_us_history 20.0% (4/20)
 high_school_world_history 35.0% (7/20)
 human_aging 40.0% (8/20)
 human_sexuality 15.0% (3/20)
 international_law 35.0% (7/20)
 jurisprudence 40.0% (8/20)
 logical_fallacies 35.0% (7/20)
 machine_learning 50.0% (10/20)
 management 20.0% (4/20)
 marketing 35.0% (7/20)
 medical_genetics 40.0% (8/20)
 miscellaneous 30.0% (6/20)
 moral_disputes 20.0% (4/20)
 moral_scenarios 15.0% (3/20)
 nutrition 20.0% (4/20)
 philosophy 15.0% (3/20)
 prehistory 25.0% (5/20)
 professional_accounting 30.0% (6/20)
 professional_law 35.0% (7/20)
 professional_medicine 5.0% (1/20)
 professional_psychology 45.0% (9/20)
 public_relations 45.0% (9/20)
 security_studies 25.0% (5/20)
 sociology 20.0% (4/20)
 us_foreign_policy 25.0% (5/20)
 virology 25.0% (5/20)
 world_religions 15.0% (3/20)
 OVERALL: 318/1140 = 27.89%
============================================================
[+] Saved mmlu_results.json
Note: GPT2-XL base ~24-26% MMLU (random 25%). MicroLLM2 distilled should be 25-30% —
MMLU is knowledge-heavy; GPT2 1.5B 1024ctx cannot match 7B+ models. Use as sanity check, not SOTA claim.
[MMLU57-TOKEN] exit 0 at Sun Aug 9 23:12:32 UTC 2026
{
 "model": "/home/zeus/microllm2/microllm2-checkpoints/final_merged",
 "shots": 5,
 "limit": 20,
 "overall": {
 "correct": 318,
 "total": 1140,
 "accuracy": 0.2789473684210526
 },
 "subjects": {
 "abstract_algebra": {
 "correct": 6,
 "total": 20,
 "accuracy": 0.3
 },
 "anatomy": {
 "correct": 5,
 "total": 20,
 "accuracy": 0.25
 },
 "astronomy": {
 "correct": 7,
 "total": 20,
 "accuracy": 0.35
 },
 "business_ethics": {
 "correct": 6,
 "total": 20,
 "accuracy": 0.3
 },
FINAL57 318 / 1140 27.89 57
'mmlu_results.json' -> 'mmlu57_results.json'
'mmlu_results.json' -> '/teamspace/studios/this_studio/microllm2/mmlu57_results.json'
'mmlu57_token.log' -> '/teamspace/studios/this_studio/microllm2/mmlu57_token.log'