| Evaluating: hellaswag_zeroshot (0-shot, type: multiple_choice)... accuracy: 0.5511 | centered: 0.4014 | time: 58.63s |
| Evaluating: jeopardy (10-shot, type: language_modeling)... accuracy: 0.2518 | centered: 0.2518 | time: 12.50s |
| Evaluating: bigbench_qa_wikidata (10-shot, type: language_modeling)... accuracy: 0.5716 | centered: 0.5716 | time: 119.92s |
| Evaluating: arc_easy (10-shot, type: multiple_choice)... accuracy: 0.6860 | centered: 0.5814 | time: 14.57s |
| Evaluating: arc_challenge (10-shot, type: multiple_choice)... accuracy: 0.3985 | centered: 0.1980 | time: 7.18s |
| Evaluating: copa (0-shot, type: multiple_choice)... accuracy: 0.7000 | centered: 0.4000 | time: 0.58s |
| Evaluating: commonsense_qa (10-shot, type: multiple_choice)... accuracy: 0.2473 | centered: 0.0592 | time: 7.57s |
| Evaluating: piqa (10-shot, type: multiple_choice)... accuracy: 0.7323 | centered: 0.4646 | time: 11.02s |
| Evaluating: openbook_qa (0-shot, type: multiple_choice)... accuracy: 0.3880 | centered: 0.1840 | time: 2.90s |
| Evaluating: lambada_openai (0-shot, type: language_modeling)... accuracy: 0.4679 | centered: 0.4679 | time: 29.68s |
| Evaluating: hellaswag (10-shot, type: multiple_choice)... accuracy: 0.5595 | centered: 0.4126 | time: 65.66s |
| Evaluating: winograd (0-shot, type: schema)... accuracy: 0.7289 | centered: 0.4579 | time: 1.53s |
| Evaluating: winogrande (0-shot, type: schema)... accuracy: 0.5699 | centered: 0.1397 | time: 7.12s |
| Evaluating: bigbench_dyck_languages (10-shot, type: language_modeling)... accuracy: 0.1340 | centered: 0.1340 | time: 5.99s |
| Evaluating: agi_eval_lsat_ar (3-shot, type: multiple_choice)... accuracy: 0.2130 | centered: 0.0163 | time: 1.46s |
| Evaluating: bigbench_cs_algorithms (10-shot, type: language_modeling)... accuracy: 0.4114 | centered: 0.4114 | time: 7.78s |
| Evaluating: bigbench_operators (10-shot, type: language_modeling)... accuracy: 0.2000 | centered: 0.2000 | time: 1.23s |
| Evaluating: bigbench_repeat_copy_logic (10-shot, type: language_modeling)... accuracy: 0.0000 | centered: 0.0000 | time: 0.19s |
| Evaluating: squad (10-shot, type: language_modeling)... accuracy: 0.2967 | centered: 0.2967 | time: 67.21s |
| Evaluating: coqa (0-shot, type: language_modeling)... accuracy: 0.3024 | centered: 0.3024 | time: 47.20s |
| Evaluating: boolq (10-shot, type: multiple_choice)... accuracy: 0.5361 | centered: -0.2208 | time: 21.41s |
| Evaluating: bigbench_language_identification (10-shot, type: multiple_choice)... accuracy: 0.2500 | centered: 0.1749 | time: 77.11s |
| |
