| INFO: 2024-07-12 11:14:00,454: llmtf.base.evaluator: Starting eval on ['darumeru/multiq', 'darumeru/parus', 'darumeru/rcb', 'darumeru/ruopenbookqa', 'darumeru/rutie', 'darumeru/ruworldtree', 'darumeru/rwsd', 'darumeru/use', 'russiannlp/rucola_custom'] |
| INFO: 2024-07-12 11:14:00,456: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [32000, 13] |
| INFO: 2024-07-12 11:14:00,456: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['\n', '\n\n'] |
| INFO: 2024-07-12 11:14:02,095: llmtf.base.evaluator: Starting eval on ['darumeru/rummlu'] |
| INFO: 2024-07-12 11:14:02,095: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [32000, 13] |
| INFO: 2024-07-12 11:14:02,095: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['\n', '\n\n'] |
| INFO: 2024-07-12 11:14:03,625: llmtf.base.darumeru/MultiQ: Loading Dataset: 3.17s |
| INFO: 2024-07-12 11:14:04,544: llmtf.base.evaluator: Starting eval on ['nlpcoreteam/rummlu'] |
| INFO: 2024-07-12 11:14:04,545: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [32000] |
| INFO: 2024-07-12 11:14:04,545: llmtf.base.hfmodel: Updated generation_config.stop_strings: [] |
| INFO: 2024-07-12 11:14:08,211: llmtf.base.evaluator: Starting eval on ['daru/treewayabstractive'] |
| INFO: 2024-07-12 11:14:08,212: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [32000, 13] |
| INFO: 2024-07-12 11:14:08,212: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['\n', '\n\n'] |
| INFO: 2024-07-12 11:14:09,060: llmtf.base.darumeru/ruMMLU: Loading Dataset: 6.96s |
| INFO: 2024-07-12 11:14:10,849: llmtf.base.evaluator: Starting eval on ['nlpcoreteam/enmmlu'] |
| INFO: 2024-07-12 11:14:10,850: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [32000] |
| INFO: 2024-07-12 11:14:10,850: llmtf.base.hfmodel: Updated generation_config.stop_strings: [] |
| INFO: 2024-07-12 11:14:11,432: llmtf.base.evaluator: Starting eval on ['daru/treewayextractive'] |
| INFO: 2024-07-12 11:14:11,433: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [32000] |
| INFO: 2024-07-12 11:14:11,433: llmtf.base.hfmodel: Updated generation_config.stop_strings: [] |
| INFO: 2024-07-12 11:14:12,547: llmtf.base.daru/treewayabstractive: Loading Dataset: 4.34s |
| INFO: 2024-07-12 11:14:12,606: llmtf.base.evaluator: Starting eval on ['darumeru/cp_sent_ru', 'darumeru/cp_sent_en', 'darumeru/cp_para_ru', 'darumeru/cp_para_en'] |
| INFO: 2024-07-12 11:14:12,607: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [32000, 13] |
| INFO: 2024-07-12 11:14:12,607: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['\n', '\n\n'] |
| INFO: 2024-07-12 11:14:15,201: llmtf.base.darumeru/cp_sent_ru: Loading Dataset: 2.59s |
| INFO: 2024-07-12 11:14:19,195: llmtf.base.daru/treewayextractive: Loading Dataset: 7.76s |
| INFO: 2024-07-12 11:16:12,870: llmtf.base.nlpcoreteam/ruMMLU: Loading Dataset: 128.32s |
| INFO: 2024-07-12 11:16:16,430: llmtf.base.nlpcoreteam/enMMLU: Loading Dataset: 125.58s |
| INFO: 2024-07-12 11:21:53,306: llmtf.base.darumeru/ruMMLU: Processing Dataset: 464.24s |
| INFO: 2024-07-12 11:21:53,309: llmtf.base.darumeru/ruMMLU: Results for darumeru/ruMMLU: |
| INFO: 2024-07-12 11:21:53,316: llmtf.base.darumeru/ruMMLU: {'acc': 0.4817918786790382} |
| INFO: 2024-07-12 11:21:53,356: llmtf.base.evaluator: Ended eval |
| INFO: 2024-07-12 11:21:53,362: llmtf.base.evaluator: |
| mean darumeru/ruMMLU |
| 0.482 0.482 |
| INFO: 2024-07-12 11:25:10,236: llmtf.base.nlpcoreteam/enMMLU: Processing Dataset: 533.80s |
| INFO: 2024-07-12 11:25:10,239: llmtf.base.nlpcoreteam/enMMLU: Results for nlpcoreteam/enMMLU: |
| INFO: 2024-07-12 11:25:10,278: llmtf.base.nlpcoreteam/enMMLU: metric |
| subject |
| abstract_algebra 0.330000 |
| anatomy 0.629630 |
| astronomy 0.657895 |
| business_ethics 0.650000 |
| clinical_knowledge 0.675472 |
| college_biology 0.729167 |
| college_chemistry 0.490000 |
| college_computer_science 0.540000 |
| college_mathematics 0.380000 |
| college_medicine 0.641618 |
| college_physics 0.343137 |
| computer_security 0.700000 |
| conceptual_physics 0.561702 |
| econometrics 0.456140 |
| electrical_engineering 0.600000 |
| elementary_mathematics 0.412698 |
| formal_logic 0.468254 |
| global_facts 0.310000 |
| high_school_biology 0.787097 |
| high_school_chemistry 0.443350 |
| high_school_computer_science 0.670000 |
| high_school_european_history 0.763636 |
| high_school_geography 0.792929 |
| high_school_government_and_politics 0.880829 |
| high_school_macroeconomics 0.612821 |
| high_school_mathematics 0.337037 |
| high_school_microeconomics 0.642857 |
| high_school_physics 0.344371 |
| high_school_psychology 0.823853 |
| high_school_statistics 0.472222 |
| high_school_us_history 0.794118 |
| high_school_world_history 0.818565 |
| human_aging 0.726457 |
| human_sexuality 0.740458 |
| international_law 0.727273 |
| jurisprudence 0.722222 |
| logical_fallacies 0.760736 |
| machine_learning 0.455357 |
| management 0.776699 |
| marketing 0.858974 |
| medical_genetics 0.690000 |
| miscellaneous 0.828863 |
| moral_disputes 0.679191 |
| moral_scenarios 0.234637 |
| nutrition 0.715686 |
| philosophy 0.655949 |
| prehistory 0.672840 |
| professional_accounting 0.464539 |
| professional_law 0.470013 |
| professional_medicine 0.705882 |
| professional_psychology 0.617647 |
| public_relations 0.636364 |
| security_studies 0.673469 |
| sociology 0.855721 |
| us_foreign_policy 0.850000 |
| virology 0.524096 |
| world_religions 0.859649 |
| INFO: 2024-07-12 11:25:10,286: llmtf.base.nlpcoreteam/enMMLU: metric |
| subject |
| STEM 0.514113 |
| humanities 0.663622 |
| other (business, health, misc.) 0.656994 |
| social sciences 0.715257 |
| INFO: 2024-07-12 11:25:10,293: llmtf.base.nlpcoreteam/enMMLU: {'acc': 0.6374965559822189} |
| INFO: 2024-07-12 11:25:10,326: llmtf.base.evaluator: Ended eval |
| INFO: 2024-07-12 11:25:10,331: llmtf.base.evaluator: |
| mean darumeru/ruMMLU nlpcoreteam/enMMLU |
| 0.560 0.482 0.637 |
| INFO: 2024-07-12 11:25:17,845: llmtf.base.darumeru/cp_sent_ru: Processing Dataset: 662.64s |
| INFO: 2024-07-12 11:25:17,847: llmtf.base.darumeru/cp_sent_ru: Results for darumeru/cp_sent_ru: |
| INFO: 2024-07-12 11:25:17,851: llmtf.base.darumeru/cp_sent_ru: {'symbol_per_token': 2.373538037297123, 'len': 0.9934467912858197, 'lcs': 0.920592765079274} |
| INFO: 2024-07-12 11:25:17,853: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [32000, 13] |
| INFO: 2024-07-12 11:25:17,853: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['\n', '\n\n'] |
| INFO: 2024-07-12 11:25:20,290: llmtf.base.darumeru/cp_sent_en: Loading Dataset: 2.44s |
| INFO: 2024-07-12 11:26:26,223: llmtf.base.nlpcoreteam/ruMMLU: Processing Dataset: 613.35s |
| INFO: 2024-07-12 11:26:26,224: llmtf.base.nlpcoreteam/ruMMLU: Results for nlpcoreteam/ruMMLU: |
| INFO: 2024-07-12 11:26:26,262: llmtf.base.nlpcoreteam/ruMMLU: metric |
| subject |
| abstract_algebra 0.360000 |
| anatomy 0.414815 |
| astronomy 0.565789 |
| business_ethics 0.560000 |
| clinical_knowledge 0.520755 |
| college_biology 0.479167 |
| college_chemistry 0.450000 |
| college_computer_science 0.500000 |
| college_mathematics 0.380000 |
| college_medicine 0.508671 |
| college_physics 0.264706 |
| computer_security 0.580000 |
| conceptual_physics 0.438298 |
| econometrics 0.368421 |
| electrical_engineering 0.475862 |
| elementary_mathematics 0.370370 |
| formal_logic 0.325397 |
| global_facts 0.280000 |
| high_school_biology 0.596774 |
| high_school_chemistry 0.374384 |
| high_school_computer_science 0.600000 |
| high_school_european_history 0.678788 |
| high_school_geography 0.666667 |
| high_school_government_and_politics 0.580311 |
| high_school_macroeconomics 0.441026 |
| high_school_mathematics 0.359259 |
| high_school_microeconomics 0.478992 |
| high_school_physics 0.390728 |
| high_school_psychology 0.623853 |
| high_school_statistics 0.458333 |
| high_school_us_history 0.686275 |
| high_school_world_history 0.713080 |
| human_aging 0.511211 |
| human_sexuality 0.564885 |
| international_law 0.652893 |
| jurisprudence 0.527778 |
| logical_fallacies 0.417178 |
| machine_learning 0.312500 |
| management 0.631068 |
| marketing 0.675214 |
| medical_genetics 0.520000 |
| miscellaneous 0.627075 |
| moral_disputes 0.531792 |
| moral_scenarios 0.233520 |
| nutrition 0.562092 |
| philosophy 0.485531 |
| prehistory 0.487654 |
| professional_accounting 0.386525 |
| professional_law 0.366362 |
| professional_medicine 0.481618 |
| professional_psychology 0.441176 |
| public_relations 0.554545 |
| security_studies 0.600000 |
| sociology 0.696517 |
| us_foreign_policy 0.740000 |
| virology 0.481928 |
| world_religions 0.684211 |
| INFO: 2024-07-12 11:26:26,270: llmtf.base.nlpcoreteam/ruMMLU: metric |
| subject |
| STEM 0.442010 |
| humanities 0.522343 |
| other (business, health, misc.) 0.511498 |
| social sciences 0.563033 |
| INFO: 2024-07-12 11:26:26,278: llmtf.base.nlpcoreteam/ruMMLU: {'acc': 0.5097207580093521} |
| INFO: 2024-07-12 11:26:26,312: llmtf.base.evaluator: Ended eval |
| INFO: 2024-07-12 11:26:26,318: llmtf.base.evaluator: |
| mean darumeru/cp_sent_ru darumeru/ruMMLU nlpcoreteam/enMMLU nlpcoreteam/ruMMLU |
| 0.656 0.993 0.482 0.637 0.510 |
| INFO: 2024-07-12 11:26:59,107: llmtf.base.darumeru/MultiQ: Processing Dataset: 775.48s |
| INFO: 2024-07-12 11:26:59,112: llmtf.base.darumeru/MultiQ: Results for darumeru/MultiQ: |
| INFO: 2024-07-12 11:26:59,146: llmtf.base.darumeru/MultiQ: {'f1': 0.28867687620463706, 'em': 0.16730401529636713} |
| INFO: 2024-07-12 11:26:59,151: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [32000, 13] |
| INFO: 2024-07-12 11:26:59,151: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['\n', '\n\n'] |
| INFO: 2024-07-12 11:27:01,342: llmtf.base.darumeru/PARus: Loading Dataset: 2.19s |
| INFO: 2024-07-12 11:27:09,303: llmtf.base.darumeru/PARus: Processing Dataset: 7.96s |
| INFO: 2024-07-12 11:27:09,304: llmtf.base.darumeru/PARus: Results for darumeru/PARus: |
| INFO: 2024-07-12 11:27:09,316: llmtf.base.darumeru/PARus: {'acc': 0.74} |
| INFO: 2024-07-12 11:27:09,317: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [32000, 13] |
| INFO: 2024-07-12 11:27:09,317: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['\n', '\n\n'] |
| INFO: 2024-07-12 11:27:11,444: llmtf.base.darumeru/RCB: Loading Dataset: 2.13s |
| INFO: 2024-07-12 11:27:20,933: llmtf.base.darumeru/RCB: Processing Dataset: 9.49s |
| INFO: 2024-07-12 11:27:20,934: llmtf.base.darumeru/RCB: Results for darumeru/RCB: |
| INFO: 2024-07-12 11:27:20,940: llmtf.base.darumeru/RCB: {'acc': 0.5227272727272727, 'f1_macro': 0.4428418803418803} |
| INFO: 2024-07-12 11:27:20,941: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [32000, 13] |
| INFO: 2024-07-12 11:27:20,941: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['\n', '\n\n'] |
| INFO: 2024-07-12 11:27:26,755: llmtf.base.darumeru/ruOpenBookQA: Loading Dataset: 5.81s |
| INFO: 2024-07-12 11:27:42,200: llmtf.base.daru/treewayextractive: Processing Dataset: 803.00s |
| INFO: 2024-07-12 11:27:42,202: llmtf.base.daru/treewayextractive: Results for daru/treewayextractive: |
| INFO: 2024-07-12 11:27:42,503: llmtf.base.daru/treewayextractive: {'r-prec': 0.4038567821067821} |
| INFO: 2024-07-12 11:27:43,066: llmtf.base.evaluator: Ended eval |
| INFO: 2024-07-12 11:27:43,074: llmtf.base.evaluator: |
| mean daru/treewayextractive darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/cp_sent_ru darumeru/ruMMLU nlpcoreteam/enMMLU nlpcoreteam/ruMMLU |
| 0.560 0.404 0.228 0.740 0.483 0.993 0.482 0.637 0.510 |
| INFO: 2024-07-12 11:28:53,021: llmtf.base.darumeru/ruOpenBookQA: Processing Dataset: 86.26s |
| INFO: 2024-07-12 11:28:53,023: llmtf.base.darumeru/ruOpenBookQA: Results for darumeru/ruOpenBookQA: |
| INFO: 2024-07-12 11:28:53,035: llmtf.base.darumeru/ruOpenBookQA: {'acc': 0.7319587628865979, 'f1_macro': 0.7326117311318767} |
| INFO: 2024-07-12 11:28:53,043: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [32000, 13] |
| INFO: 2024-07-12 11:28:53,043: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['\n', '\n\n'] |
| INFO: 2024-07-12 11:28:57,375: llmtf.base.darumeru/ruTiE: Loading Dataset: 4.33s |
| INFO: 2024-07-12 11:33:18,433: llmtf.base.darumeru/ruTiE: Processing Dataset: 261.04s |
| INFO: 2024-07-12 11:33:18,436: llmtf.base.darumeru/ruTiE: Results for darumeru/ruTiE: |
| INFO: 2024-07-12 11:33:18,465: llmtf.base.darumeru/ruTiE: {'acc': 0.5395348837209303} |
| INFO: 2024-07-12 11:33:18,468: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [32000, 13] |
| INFO: 2024-07-12 11:33:18,468: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['\n', '\n\n'] |
| INFO: 2024-07-12 11:33:20,755: llmtf.base.darumeru/ruWorldTree: Loading Dataset: 2.29s |
| INFO: 2024-07-12 11:33:24,890: llmtf.base.darumeru/ruWorldTree: Processing Dataset: 4.13s |
| INFO: 2024-07-12 11:33:24,904: llmtf.base.darumeru/ruWorldTree: Results for darumeru/ruWorldTree: |
| INFO: 2024-07-12 11:33:24,910: llmtf.base.darumeru/ruWorldTree: {'acc': 0.8857142857142857, 'f1_macro': 0.8835438024831281} |
| INFO: 2024-07-12 11:33:24,911: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [32000, 13] |
| INFO: 2024-07-12 11:33:24,911: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['\n', '\n\n'] |
| INFO: 2024-07-12 11:33:26,881: llmtf.base.darumeru/RWSD: Loading Dataset: 1.97s |
| INFO: 2024-07-12 11:33:35,928: llmtf.base.darumeru/RWSD: Processing Dataset: 9.05s |
| INFO: 2024-07-12 11:33:35,944: llmtf.base.darumeru/RWSD: Results for darumeru/RWSD: |
| INFO: 2024-07-12 11:33:35,978: llmtf.base.darumeru/RWSD: {'acc': 0.5441176470588235} |
| INFO: 2024-07-12 11:33:35,979: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [32000, 13] |
| INFO: 2024-07-12 11:33:35,979: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['\n', '\n\n'] |
| INFO: 2024-07-12 11:33:38,817: llmtf.base.darumeru/USE: Loading Dataset: 2.84s |
| INFO: 2024-07-12 11:34:04,752: llmtf.base.darumeru/cp_sent_en: Processing Dataset: 524.46s |
| INFO: 2024-07-12 11:34:04,754: llmtf.base.darumeru/cp_sent_en: Results for darumeru/cp_sent_en: |
| INFO: 2024-07-12 11:34:04,758: llmtf.base.darumeru/cp_sent_en: {'symbol_per_token': 3.8962352972297385, 'len': 0.997653558022427, 'lcs': 0.9800192820829826} |
| INFO: 2024-07-12 11:34:04,759: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [32000, 13] |
| INFO: 2024-07-12 11:34:04,759: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['\n', '\n\n'] |
| INFO: 2024-07-12 11:34:07,018: llmtf.base.darumeru/cp_para_ru: Loading Dataset: 2.26s |
| INFO: 2024-07-12 11:43:37,087: llmtf.base.darumeru/USE: Processing Dataset: 598.27s |
| INFO: 2024-07-12 11:43:37,090: llmtf.base.darumeru/USE: Results for darumeru/USE: |
| INFO: 2024-07-12 11:43:37,124: llmtf.base.darumeru/USE: {'grade_norm': 0.06274509803921567} |
| INFO: 2024-07-12 11:43:37,127: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [32000] |
| INFO: 2024-07-12 11:43:37,128: llmtf.base.hfmodel: Updated generation_config.stop_strings: [] |
| INFO: 2024-07-12 11:43:41,113: llmtf.base.russiannlp/rucola_custom: Loading Dataset: 3.98s |
| INFO: 2024-07-12 11:45:35,417: llmtf.base.russiannlp/rucola_custom: Processing Dataset: 114.30s |
| INFO: 2024-07-12 11:45:35,422: llmtf.base.russiannlp/rucola_custom: Results for russiannlp/rucola_custom: |
| INFO: 2024-07-12 11:45:35,446: llmtf.base.russiannlp/rucola_custom: {'acc': 0.7369931826336562, 'mcc': 0.3418673882542275} |
| INFO: 2024-07-12 11:45:35,450: llmtf.base.evaluator: Ended eval |
| INFO: 2024-07-12 11:45:35,484: llmtf.base.evaluator: |
| mean daru/treewayextractive darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/USE darumeru/cp_sent_en darumeru/cp_sent_ru darumeru/ruMMLU darumeru/ruOpenBookQA darumeru/ruTiE darumeru/ruWorldTree nlpcoreteam/enMMLU nlpcoreteam/ruMMLU russiannlp/rucola_custom |
| 0.585 0.404 0.228 0.740 0.483 0.544 0.063 0.998 0.993 0.482 0.732 0.540 0.885 0.637 0.510 0.539 |
| INFO: 2024-07-12 11:49:02,906: llmtf.base.darumeru/cp_para_ru: Processing Dataset: 895.89s |
| INFO: 2024-07-12 11:49:02,917: llmtf.base.darumeru/cp_para_ru: Results for darumeru/cp_para_ru: |
| INFO: 2024-07-12 11:49:02,921: llmtf.base.darumeru/cp_para_ru: {'symbol_per_token': 2.4736956938033994, 'len': 0.9938656254701991, 'lcs': 0.8313599424096164} |
| INFO: 2024-07-12 11:49:02,922: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [32000, 13] |
| INFO: 2024-07-12 11:49:02,922: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['\n', '\n\n'] |
| INFO: 2024-07-12 11:49:05,017: llmtf.base.darumeru/cp_para_en: Loading Dataset: 2.09s |
| INFO: 2024-07-12 11:59:16,449: llmtf.base.daru/treewayabstractive: Processing Dataset: 2703.90s |
| INFO: 2024-07-12 11:59:16,451: llmtf.base.daru/treewayabstractive: Results for daru/treewayabstractive: |
| INFO: 2024-07-12 11:59:16,482: llmtf.base.daru/treewayabstractive: {'rouge1': 0.34439020370710943, 'rouge2': 0.12428286477373612} |
| INFO: 2024-07-12 11:59:16,484: llmtf.base.evaluator: Ended eval |
| INFO: 2024-07-12 11:59:16,496: llmtf.base.evaluator: |
| mean daru/treewayabstractive daru/treewayextractive darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/USE darumeru/cp_para_ru darumeru/cp_sent_en darumeru/cp_sent_ru darumeru/ruMMLU darumeru/ruOpenBookQA darumeru/ruTiE darumeru/ruWorldTree nlpcoreteam/enMMLU nlpcoreteam/ruMMLU russiannlp/rucola_custom |
| 0.579 0.234 0.404 0.228 0.740 0.483 0.544 0.063 0.831 0.998 0.993 0.482 0.732 0.540 0.885 0.637 0.510 0.539 |
| INFO: 2024-07-12 12:00:13,938: llmtf.base.darumeru/cp_para_en: Processing Dataset: 668.92s |
| INFO: 2024-07-12 12:00:13,954: llmtf.base.darumeru/cp_para_en: Results for darumeru/cp_para_en: |
| INFO: 2024-07-12 12:00:13,960: llmtf.base.darumeru/cp_para_en: {'symbol_per_token': 3.9519089902445974, 'len': 0.9916598586547556, 'lcs': 0.8689672911613434} |
| INFO: 2024-07-12 12:00:13,960: llmtf.base.evaluator: Ended eval |
| INFO: 2024-07-12 12:00:13,985: llmtf.base.evaluator: |
| mean daru/treewayabstractive daru/treewayextractive darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/USE darumeru/cp_para_en darumeru/cp_para_ru darumeru/cp_sent_en darumeru/cp_sent_ru darumeru/ruMMLU darumeru/ruOpenBookQA darumeru/ruTiE darumeru/ruWorldTree nlpcoreteam/enMMLU nlpcoreteam/ruMMLU russiannlp/rucola_custom |
| 0.595 0.234 0.404 0.228 0.740 0.483 0.544 0.063 0.869 0.831 0.998 0.993 0.482 0.732 0.540 0.885 0.637 0.510 0.539 |
|
|
