| INFO: 2024-07-13 15:18:25,981: llmtf.base.evaluator: Starting eval on ['darumeru/multiq', 'darumeru/parus', 'darumeru/rcb', 'darumeru/ruopenbookqa', 'darumeru/rutie', 'darumeru/ruworldtree', 'darumeru/rwsd', 'darumeru/use', 'russiannlp/rucola_custom'] | |
| INFO: 2024-07-13 15:18:25,995: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [32000] | |
| INFO: 2024-07-13 15:18:25,996: llmtf.base.hfmodel: Updated generation_config.stop_strings: [] | |
| INFO: 2024-07-13 15:18:26,240: llmtf.base.evaluator: Starting eval on ['darumeru/rummlu'] | |
| INFO: 2024-07-13 15:18:26,240: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [32000] | |
| INFO: 2024-07-13 15:18:26,240: llmtf.base.hfmodel: Updated generation_config.stop_strings: [] | |
| INFO: 2024-07-13 15:18:27,990: llmtf.base.evaluator: Starting eval on ['nlpcoreteam/rummlu'] | |
| INFO: 2024-07-13 15:18:27,991: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [32000] | |
| INFO: 2024-07-13 15:18:27,991: llmtf.base.hfmodel: Updated generation_config.stop_strings: [] | |
| INFO: 2024-07-13 15:18:29,333: llmtf.base.evaluator: Starting eval on ['nlpcoreteam/enmmlu'] | |
| INFO: 2024-07-13 15:18:29,333: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [32000] | |
| INFO: 2024-07-13 15:18:29,333: llmtf.base.hfmodel: Updated generation_config.stop_strings: [] | |
| INFO: 2024-07-13 15:18:29,480: llmtf.base.darumeru/MultiQ: Loading Dataset: 3.48s | |
| INFO: 2024-07-13 15:18:30,985: llmtf.base.evaluator: Starting eval on ['daru/treewayabstractive'] | |
| INFO: 2024-07-13 15:18:30,985: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [32000] | |
| INFO: 2024-07-13 15:18:30,985: llmtf.base.hfmodel: Updated generation_config.stop_strings: [] | |
| INFO: 2024-07-13 15:18:33,199: llmtf.base.evaluator: Starting eval on ['daru/treewayextractive'] | |
| INFO: 2024-07-13 15:18:33,200: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [32000] | |
| INFO: 2024-07-13 15:18:33,200: llmtf.base.hfmodel: Updated generation_config.stop_strings: [] | |
| INFO: 2024-07-13 15:18:33,694: llmtf.base.darumeru/ruMMLU: Loading Dataset: 7.45s | |
| INFO: 2024-07-13 15:18:35,345: llmtf.base.daru/treewayabstractive: Loading Dataset: 4.36s | |
| INFO: 2024-07-13 15:18:35,432: llmtf.base.evaluator: Starting eval on ['darumeru/cp_sent_ru', 'darumeru/cp_sent_en', 'darumeru/cp_para_ru', 'darumeru/cp_para_en'] | |
| INFO: 2024-07-13 15:18:35,433: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [32000] | |
| INFO: 2024-07-13 15:18:35,433: llmtf.base.hfmodel: Updated generation_config.stop_strings: [] | |
| INFO: 2024-07-13 15:18:37,953: llmtf.base.darumeru/cp_sent_ru: Loading Dataset: 2.52s | |
| INFO: 2024-07-13 15:18:40,885: llmtf.base.daru/treewayextractive: Loading Dataset: 7.69s | |
| INFO: 2024-07-13 15:23:40,040: llmtf.base.evaluator: Starting eval on ['darumeru/parus', 'darumeru/rcb', 'darumeru/ruopenbookqa', 'darumeru/rutie', 'darumeru/ruworldtree', 'darumeru/rwsd', 'russiannlp/rucola_custom'] | |
| INFO: 2024-07-13 15:23:40,042: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [32000] | |
| INFO: 2024-07-13 15:23:40,042: llmtf.base.hfmodel: Updated generation_config.stop_strings: [] | |
| INFO: 2024-07-13 15:23:40,509: llmtf.base.evaluator: Starting eval on ['darumeru/rummlu', 'daru/treewayextractive'] | |
| INFO: 2024-07-13 15:23:40,510: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [32000] | |
| INFO: 2024-07-13 15:23:40,510: llmtf.base.hfmodel: Updated generation_config.stop_strings: [] | |
| INFO: 2024-07-13 15:23:41,090: llmtf.base.evaluator: Starting eval on ['nlpcoreteam/rummlu', 'nlpcoreteam/enmmlu'] | |
| INFO: 2024-07-13 15:23:41,091: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [32000] | |
| INFO: 2024-07-13 15:23:41,091: llmtf.base.hfmodel: Updated generation_config.stop_strings: [] | |
| INFO: 2024-07-13 15:23:42,369: llmtf.base.darumeru/PARus: Loading Dataset: 2.33s | |
| INFO: 2024-07-13 15:23:43,206: llmtf.base.evaluator: Starting eval on ['daru/treewayabstractive'] | |
| INFO: 2024-07-13 15:23:43,207: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [32000] | |
| INFO: 2024-07-13 15:23:43,207: llmtf.base.hfmodel: Updated generation_config.stop_strings: [] | |
| INFO: 2024-07-13 15:23:45,405: llmtf.base.evaluator: Starting eval on ['darumeru/multiq', 'darumeru/use'] | |
| INFO: 2024-07-13 15:23:45,405: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [32000] | |
| INFO: 2024-07-13 15:23:45,405: llmtf.base.hfmodel: Updated generation_config.stop_strings: [] | |
| INFO: 2024-07-13 15:23:45,786: llmtf.base.darumeru/PARus: Processing Dataset: 3.42s | |
| INFO: 2024-07-13 15:23:45,788: llmtf.base.darumeru/PARus: Results for darumeru/PARus: | |
| INFO: 2024-07-13 15:23:45,800: llmtf.base.darumeru/PARus: {'acc': 0.75} | |
| INFO: 2024-07-13 15:23:45,801: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [32000] | |
| INFO: 2024-07-13 15:23:45,801: llmtf.base.hfmodel: Updated generation_config.stop_strings: [] | |
| INFO: 2024-07-13 15:23:47,436: llmtf.base.daru/treewayabstractive: Loading Dataset: 4.23s | |
| INFO: 2024-07-13 15:23:47,479: llmtf.base.evaluator: Starting eval on ['darumeru/cp_sent_ru', 'darumeru/cp_para_ru'] | |
| INFO: 2024-07-13 15:23:47,479: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [32000] | |
| INFO: 2024-07-13 15:23:47,479: llmtf.base.hfmodel: Updated generation_config.stop_strings: [] | |
| INFO: 2024-07-13 15:23:47,668: llmtf.base.darumeru/ruMMLU: Loading Dataset: 7.16s | |
| INFO: 2024-07-13 15:23:47,812: llmtf.base.darumeru/RCB: Loading Dataset: 2.01s | |
| INFO: 2024-07-13 15:23:49,390: llmtf.base.evaluator: Starting eval on ['darumeru/cp_sent_en', 'darumeru/cp_para_en'] | |
| INFO: 2024-07-13 15:23:49,390: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [32000] | |
| INFO: 2024-07-13 15:23:49,390: llmtf.base.hfmodel: Updated generation_config.stop_strings: [] | |
| INFO: 2024-07-13 15:23:49,768: llmtf.base.darumeru/cp_sent_ru: Loading Dataset: 2.29s | |
| INFO: 2024-07-13 15:23:51,703: llmtf.base.darumeru/cp_sent_en: Loading Dataset: 2.31s | |
| INFO: 2024-07-13 15:23:53,877: llmtf.base.darumeru/RCB: Processing Dataset: 6.06s | |
| INFO: 2024-07-13 15:23:53,892: llmtf.base.darumeru/RCB: Results for darumeru/RCB: | |
| INFO: 2024-07-13 15:23:53,898: llmtf.base.darumeru/RCB: {'acc': 0.5227272727272727, 'f1_macro': 0.4428418803418803} | |
| INFO: 2024-07-13 15:23:53,899: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [32000] | |
| INFO: 2024-07-13 15:23:53,899: llmtf.base.hfmodel: Updated generation_config.stop_strings: [] | |
| INFO: 2024-07-13 15:23:56,231: llmtf.base.darumeru/ruOpenBookQA: Loading Dataset: 2.33s | |
| INFO: 2024-07-13 15:24:00,471: llmtf.base.darumeru/MultiQ: Loading Dataset: 15.07s | |
| INFO: 2024-07-13 15:24:35,821: llmtf.base.darumeru/ruOpenBookQA: Processing Dataset: 39.59s | |
| INFO: 2024-07-13 15:24:35,822: llmtf.base.darumeru/ruOpenBookQA: Results for darumeru/ruOpenBookQA: | |
| INFO: 2024-07-13 15:24:35,835: llmtf.base.darumeru/ruOpenBookQA: {'acc': 0.7323883161512027, 'f1_macro': 0.7329226353930633} | |
| INFO: 2024-07-13 15:24:35,842: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [32000] | |
| INFO: 2024-07-13 15:24:35,842: llmtf.base.hfmodel: Updated generation_config.stop_strings: [] | |
| INFO: 2024-07-13 15:24:40,146: llmtf.base.darumeru/ruTiE: Loading Dataset: 4.30s | |
| INFO: 2024-07-13 15:26:07,845: llmtf.base.nlpcoreteam/ruMMLU: Loading Dataset: 146.75s | |
| INFO: 2024-07-13 15:26:38,965: llmtf.base.darumeru/cp_sent_en: Processing Dataset: 167.26s | |
| INFO: 2024-07-13 15:26:38,982: llmtf.base.darumeru/cp_sent_en: Results for darumeru/cp_sent_en: | |
| INFO: 2024-07-13 15:26:38,999: llmtf.base.darumeru/cp_sent_en: {'symbol_per_token': 3.895358639670925, 'len': 0.9974420397589191, 'lcs': 0.9801922792969053} | |
| INFO: 2024-07-13 15:26:39,001: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [32000] | |
| INFO: 2024-07-13 15:26:39,001: llmtf.base.hfmodel: Updated generation_config.stop_strings: [] | |
| INFO: 2024-07-13 15:26:41,305: llmtf.base.darumeru/cp_para_en: Loading Dataset: 2.30s | |
| INFO: 2024-07-13 15:27:49,425: llmtf.base.darumeru/cp_sent_ru: Processing Dataset: 239.65s | |
| INFO: 2024-07-13 15:27:49,442: llmtf.base.darumeru/cp_sent_ru: Results for darumeru/cp_sent_ru: | |
| INFO: 2024-07-13 15:27:49,446: llmtf.base.darumeru/cp_sent_ru: {'symbol_per_token': 2.3734936335344794, 'len': 0.9922334558022529, 'lcs': 0.9153760193869099} | |
| INFO: 2024-07-13 15:27:49,448: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [32000] | |
| INFO: 2024-07-13 15:27:49,448: llmtf.base.hfmodel: Updated generation_config.stop_strings: [] | |
| INFO: 2024-07-13 15:27:51,495: llmtf.base.darumeru/cp_para_ru: Loading Dataset: 2.05s | |
| INFO: 2024-07-13 15:28:59,772: llmtf.base.darumeru/ruTiE: Processing Dataset: 259.62s | |
| INFO: 2024-07-13 15:28:59,774: llmtf.base.darumeru/ruTiE: Results for darumeru/ruTiE: | |
| INFO: 2024-07-13 15:28:59,815: llmtf.base.darumeru/ruTiE: {'acc': 0.5372093023255814} | |
| INFO: 2024-07-13 15:28:59,818: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [32000] | |
| INFO: 2024-07-13 15:28:59,819: llmtf.base.hfmodel: Updated generation_config.stop_strings: [] | |
| INFO: 2024-07-13 15:29:01,873: llmtf.base.darumeru/ruWorldTree: Loading Dataset: 2.05s | |
| INFO: 2024-07-13 15:29:04,102: llmtf.base.darumeru/ruWorldTree: Processing Dataset: 2.21s | |
| INFO: 2024-07-13 15:29:04,104: llmtf.base.darumeru/ruWorldTree: Results for darumeru/ruWorldTree: | |
| INFO: 2024-07-13 15:29:04,109: llmtf.base.darumeru/ruWorldTree: {'acc': 0.8761904761904762, 'f1_macro': 0.8744880624959789} | |
| INFO: 2024-07-13 15:29:04,110: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [32000] | |
| INFO: 2024-07-13 15:29:04,110: llmtf.base.hfmodel: Updated generation_config.stop_strings: [] | |
| INFO: 2024-07-13 15:29:06,342: llmtf.base.darumeru/RWSD: Loading Dataset: 2.23s | |
| INFO: 2024-07-13 15:29:13,292: llmtf.base.darumeru/RWSD: Processing Dataset: 6.95s | |
| INFO: 2024-07-13 15:29:13,294: llmtf.base.darumeru/RWSD: Results for darumeru/RWSD: | |
| INFO: 2024-07-13 15:29:13,298: llmtf.base.darumeru/RWSD: {'acc': 0.5392156862745098} | |
| INFO: 2024-07-13 15:29:13,299: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [32000] | |
| INFO: 2024-07-13 15:29:13,299: llmtf.base.hfmodel: Updated generation_config.stop_strings: [] | |
| INFO: 2024-07-13 15:29:16,878: llmtf.base.russiannlp/rucola_custom: Loading Dataset: 3.58s | |
| INFO: 2024-07-13 15:30:03,574: llmtf.base.russiannlp/rucola_custom: Processing Dataset: 46.69s | |
| INFO: 2024-07-13 15:30:03,578: llmtf.base.russiannlp/rucola_custom: Results for russiannlp/rucola_custom: | |
| INFO: 2024-07-13 15:30:03,605: llmtf.base.russiannlp/rucola_custom: {'acc': 0.7366343738787227, 'mcc': 0.34075509260259335} | |
| INFO: 2024-07-13 15:30:03,609: llmtf.base.evaluator: Ended eval | |
| INFO: 2024-07-13 15:30:03,635: llmtf.base.evaluator: | |
| mean darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/cp_sent_en darumeru/cp_sent_ru darumeru/ruOpenBookQA darumeru/ruTiE darumeru/ruWorldTree russiannlp/rucola_custom | |
| 0.716 0.750 0.483 0.539 0.997 0.992 0.733 0.537 0.875 0.539 | |
| INFO: 2024-07-13 15:30:38,848: llmtf.base.darumeru/cp_para_en: Processing Dataset: 237.54s | |
| INFO: 2024-07-13 15:30:38,850: llmtf.base.darumeru/cp_para_en: Results for darumeru/cp_para_en: | |
| INFO: 2024-07-13 15:30:38,867: llmtf.base.darumeru/cp_para_en: {'symbol_per_token': 3.964633931969539, 'len': 0.9963390331388082, 'lcs': 0.873438038674546} | |
| INFO: 2024-07-13 15:30:38,867: llmtf.base.evaluator: Ended eval | |
| INFO: 2024-07-13 15:30:38,877: llmtf.base.evaluator: | |
| mean darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/cp_para_en darumeru/cp_sent_en darumeru/cp_sent_ru darumeru/ruOpenBookQA darumeru/ruTiE darumeru/ruWorldTree russiannlp/rucola_custom | |
| 0.732 0.750 0.483 0.539 0.873 0.997 0.992 0.733 0.537 0.875 0.539 | |
| INFO: 2024-07-13 15:30:58,082: llmtf.base.darumeru/ruMMLU: Processing Dataset: 430.41s | |
| INFO: 2024-07-13 15:30:58,085: llmtf.base.darumeru/ruMMLU: Results for darumeru/ruMMLU: | |
| INFO: 2024-07-13 15:30:58,123: llmtf.base.darumeru/ruMMLU: {'acc': 0.4818916492068243} | |
| INFO: 2024-07-13 15:30:58,162: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [32000] | |
| INFO: 2024-07-13 15:30:58,163: llmtf.base.hfmodel: Updated generation_config.stop_strings: [] | |
| INFO: 2024-07-13 15:31:08,009: llmtf.base.daru/treewayextractive: Loading Dataset: 9.84s | |
| INFO: 2024-07-13 15:31:29,151: llmtf.base.darumeru/MultiQ: Processing Dataset: 448.68s | |
| INFO: 2024-07-13 15:31:29,154: llmtf.base.darumeru/MultiQ: Results for darumeru/MultiQ: | |
| INFO: 2024-07-13 15:31:29,174: llmtf.base.darumeru/MultiQ: {'f1': 0.2909161781249439, 'em': 0.16634799235181644} | |
| INFO: 2024-07-13 15:31:29,179: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [32000] | |
| INFO: 2024-07-13 15:31:29,179: llmtf.base.hfmodel: Updated generation_config.stop_strings: [] | |
| INFO: 2024-07-13 15:31:32,357: llmtf.base.darumeru/USE: Loading Dataset: 3.18s | |
| INFO: 2024-07-13 15:33:20,030: llmtf.base.nlpcoreteam/ruMMLU: Processing Dataset: 432.18s | |
| INFO: 2024-07-13 15:33:20,032: llmtf.base.nlpcoreteam/ruMMLU: Results for nlpcoreteam/ruMMLU: | |
| INFO: 2024-07-13 15:33:20,078: llmtf.base.nlpcoreteam/ruMMLU: metric | |
| subject | |
| abstract_algebra 0.360000 | |
| anatomy 0.414815 | |
| astronomy 0.565789 | |
| business_ethics 0.550000 | |
| clinical_knowledge 0.528302 | |
| college_biology 0.486111 | |
| college_chemistry 0.450000 | |
| college_computer_science 0.510000 | |
| college_mathematics 0.390000 | |
| college_medicine 0.508671 | |
| college_physics 0.254902 | |
| computer_security 0.580000 | |
| conceptual_physics 0.434043 | |
| econometrics 0.359649 | |
| electrical_engineering 0.489655 | |
| elementary_mathematics 0.370370 | |
| formal_logic 0.325397 | |
| global_facts 0.280000 | |
| high_school_biology 0.590323 | |
| high_school_chemistry 0.374384 | |
| high_school_computer_science 0.600000 | |
| high_school_european_history 0.666667 | |
| high_school_geography 0.666667 | |
| high_school_government_and_politics 0.580311 | |
| high_school_macroeconomics 0.438462 | |
| high_school_mathematics 0.359259 | |
| high_school_microeconomics 0.478992 | |
| high_school_physics 0.397351 | |
| high_school_psychology 0.625688 | |
| high_school_statistics 0.467593 | |
| high_school_us_history 0.681373 | |
| high_school_world_history 0.713080 | |
| human_aging 0.515695 | |
| human_sexuality 0.557252 | |
| international_law 0.652893 | |
| jurisprudence 0.527778 | |
| logical_fallacies 0.423313 | |
| machine_learning 0.321429 | |
| management 0.631068 | |
| marketing 0.675214 | |
| medical_genetics 0.530000 | |
| miscellaneous 0.624521 | |
| moral_disputes 0.528902 | |
| moral_scenarios 0.231285 | |
| nutrition 0.555556 | |
| philosophy 0.482315 | |
| prehistory 0.490741 | |
| professional_accounting 0.382979 | |
| professional_law 0.367014 | |
| professional_medicine 0.477941 | |
| professional_psychology 0.439542 | |
| public_relations 0.554545 | |
| security_studies 0.612245 | |
| sociology 0.686567 | |
| us_foreign_policy 0.730000 | |
| virology 0.487952 | |
| world_religions 0.690058 | |
| INFO: 2024-07-13 15:33:20,086: llmtf.base.nlpcoreteam/ruMMLU: metric | |
| subject | |
| STEM 0.444512 | |
| humanities 0.521601 | |
| other (business, health, misc.) 0.511622 | |
| social sciences 0.560827 | |
| INFO: 2024-07-13 15:33:20,093: llmtf.base.nlpcoreteam/ruMMLU: {'acc': 0.5096404231777287} | |
| INFO: 2024-07-13 15:33:20,128: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [32000] | |
| INFO: 2024-07-13 15:33:20,129: llmtf.base.hfmodel: Updated generation_config.stop_strings: [] | |
| INFO: 2024-07-13 15:34:10,886: llmtf.base.darumeru/cp_para_ru: Processing Dataset: 379.39s | |
| INFO: 2024-07-13 15:34:10,904: llmtf.base.darumeru/cp_para_ru: Results for darumeru/cp_para_ru: | |
| INFO: 2024-07-13 15:34:10,908: llmtf.base.darumeru/cp_para_ru: {'symbol_per_token': 2.469064241004292, 'len': 0.9929789601006123, 'lcs': 0.843045621556421} | |
| INFO: 2024-07-13 15:34:10,908: llmtf.base.evaluator: Ended eval | |
| INFO: 2024-07-13 15:34:10,938: llmtf.base.evaluator: | |
| mean darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/cp_para_en darumeru/cp_para_ru darumeru/cp_sent_en darumeru/cp_sent_ru darumeru/ruMMLU darumeru/ruOpenBookQA darumeru/ruTiE darumeru/ruWorldTree nlpcoreteam/ruMMLU russiannlp/rucola_custom | |
| 0.670 0.229 0.750 0.483 0.539 0.873 0.843 0.997 0.992 0.482 0.733 0.537 0.875 0.510 0.539 | |
| INFO: 2024-07-13 15:35:15,961: llmtf.base.nlpcoreteam/enMMLU: Loading Dataset: 115.83s | |
| INFO: 2024-07-13 15:36:14,977: llmtf.base.darumeru/USE: Processing Dataset: 282.62s | |
| INFO: 2024-07-13 15:36:14,978: llmtf.base.darumeru/USE: Results for darumeru/USE: | |
| INFO: 2024-07-13 15:36:14,982: llmtf.base.darumeru/USE: {'grade_norm': 0.06568627450980391} | |
| INFO: 2024-07-13 15:36:14,985: llmtf.base.evaluator: Ended eval | |
| INFO: 2024-07-13 15:36:14,994: llmtf.base.evaluator: | |
| mean darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/USE darumeru/cp_para_en darumeru/cp_para_ru darumeru/cp_sent_en darumeru/cp_sent_ru darumeru/ruMMLU darumeru/ruOpenBookQA darumeru/ruTiE darumeru/ruWorldTree nlpcoreteam/ruMMLU russiannlp/rucola_custom | |
| 0.630 0.229 0.750 0.483 0.539 0.066 0.873 0.843 0.997 0.992 0.482 0.733 0.537 0.875 0.510 0.539 | |
| INFO: 2024-07-13 15:39:22,754: llmtf.base.daru/treewayextractive: Processing Dataset: 494.73s | |
| INFO: 2024-07-13 15:39:22,774: llmtf.base.daru/treewayextractive: Results for daru/treewayextractive: | |
| INFO: 2024-07-13 15:39:23,003: llmtf.base.daru/treewayextractive: {'r-prec': 0.40769011544011546} | |
| INFO: 2024-07-13 15:39:23,057: llmtf.base.evaluator: Ended eval | |
| INFO: 2024-07-13 15:39:23,070: llmtf.base.evaluator: | |
| mean daru/treewayextractive darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/USE darumeru/cp_para_en darumeru/cp_para_ru darumeru/cp_sent_en darumeru/cp_sent_ru darumeru/ruMMLU darumeru/ruOpenBookQA darumeru/ruTiE darumeru/ruWorldTree nlpcoreteam/ruMMLU russiannlp/rucola_custom | |
| 0.616 0.408 0.229 0.750 0.483 0.539 0.066 0.873 0.843 0.997 0.992 0.482 0.733 0.537 0.875 0.510 0.539 | |
| INFO: 2024-07-13 15:40:06,403: llmtf.base.nlpcoreteam/enMMLU: Processing Dataset: 290.44s | |
| INFO: 2024-07-13 15:40:06,407: llmtf.base.nlpcoreteam/enMMLU: Results for nlpcoreteam/enMMLU: | |
| INFO: 2024-07-13 15:40:06,452: llmtf.base.nlpcoreteam/enMMLU: metric | |
| subject | |
| abstract_algebra 0.340000 | |
| anatomy 0.614815 | |
| astronomy 0.651316 | |
| business_ethics 0.650000 | |
| clinical_knowledge 0.675472 | |
| college_biology 0.729167 | |
| college_chemistry 0.470000 | |
| college_computer_science 0.550000 | |
| college_mathematics 0.380000 | |
| college_medicine 0.641618 | |
| college_physics 0.352941 | |
| computer_security 0.700000 | |
| conceptual_physics 0.553191 | |
| econometrics 0.456140 | |
| electrical_engineering 0.593103 | |
| elementary_mathematics 0.412698 | |
| formal_logic 0.476190 | |
| global_facts 0.310000 | |
| high_school_biology 0.790323 | |
| high_school_chemistry 0.458128 | |
| high_school_computer_science 0.670000 | |
| high_school_european_history 0.769697 | |
| high_school_geography 0.792929 | |
| high_school_government_and_politics 0.880829 | |
| high_school_macroeconomics 0.612821 | |
| high_school_mathematics 0.344444 | |
| high_school_microeconomics 0.642857 | |
| high_school_physics 0.337748 | |
| high_school_psychology 0.823853 | |
| high_school_statistics 0.467593 | |
| high_school_us_history 0.794118 | |
| high_school_world_history 0.814346 | |
| human_aging 0.721973 | |
| human_sexuality 0.732824 | |
| international_law 0.735537 | |
| jurisprudence 0.722222 | |
| logical_fallacies 0.760736 | |
| machine_learning 0.455357 | |
| management 0.776699 | |
| marketing 0.858974 | |
| medical_genetics 0.690000 | |
| miscellaneous 0.830140 | |
| moral_disputes 0.679191 | |
| moral_scenarios 0.232402 | |
| nutrition 0.709150 | |
| philosophy 0.655949 | |
| prehistory 0.672840 | |
| professional_accounting 0.460993 | |
| professional_law 0.468057 | |
| professional_medicine 0.709559 | |
| professional_psychology 0.619281 | |
| public_relations 0.645455 | |
| security_studies 0.673469 | |
| sociology 0.850746 | |
| us_foreign_policy 0.850000 | |
| virology 0.512048 | |
| world_religions 0.853801 | |
| INFO: 2024-07-13 15:40:06,459: llmtf.base.nlpcoreteam/enMMLU: metric | |
| subject | |
| STEM 0.514223 | |
| humanities 0.664237 | |
| other (business, health, misc.) 0.654389 | |
| social sciences 0.715100 | |
| INFO: 2024-07-13 15:40:06,466: llmtf.base.nlpcoreteam/enMMLU: {'acc': 0.6369873391991915} | |
| INFO: 2024-07-13 15:40:06,497: llmtf.base.evaluator: Ended eval | |
| INFO: 2024-07-13 15:40:06,508: llmtf.base.evaluator: | |
| mean daru/treewayextractive darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/USE darumeru/cp_para_en darumeru/cp_para_ru darumeru/cp_sent_en darumeru/cp_sent_ru darumeru/ruMMLU darumeru/ruOpenBookQA darumeru/ruTiE darumeru/ruWorldTree nlpcoreteam/enMMLU nlpcoreteam/ruMMLU russiannlp/rucola_custom | |
| 0.617 0.408 0.229 0.750 0.483 0.539 0.066 0.873 0.843 0.997 0.992 0.482 0.733 0.537 0.875 0.637 0.510 0.539 | |
| INFO: 2024-07-13 15:44:34,352: llmtf.base.daru/treewayabstractive: Processing Dataset: 1246.91s | |
| INFO: 2024-07-13 15:44:34,354: llmtf.base.daru/treewayabstractive: Results for daru/treewayabstractive: | |
| INFO: 2024-07-13 15:44:34,373: llmtf.base.daru/treewayabstractive: {'rouge1': 0.34479017541198337, 'rouge2': 0.12451437402782907} | |
| INFO: 2024-07-13 15:44:34,376: llmtf.base.evaluator: Ended eval | |
| INFO: 2024-07-13 15:44:34,403: llmtf.base.evaluator: | |
| mean daru/treewayabstractive daru/treewayextractive darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/USE darumeru/cp_para_en darumeru/cp_para_ru darumeru/cp_sent_en darumeru/cp_sent_ru darumeru/ruMMLU darumeru/ruOpenBookQA darumeru/ruTiE darumeru/ruWorldTree nlpcoreteam/enMMLU nlpcoreteam/ruMMLU russiannlp/rucola_custom | |
| 0.596 0.235 0.408 0.229 0.750 0.483 0.539 0.066 0.873 0.843 0.997 0.992 0.482 0.733 0.537 0.875 0.637 0.510 0.539 | |