| INFO: 2024-10-17 07:12:53,947: llmtf.base.evaluator: Starting eval on ['darumeru/multiq'] |
| INFO: 2024-10-17 07:12:53,947: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [131508] |
| INFO: 2024-10-17 07:12:53,947: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] |
| INFO: 2024-10-17 07:13:01,539: llmtf.base.darumeru/MultiQ: Loading Dataset: 7.59s |
| INFO: 2024-10-17 07:18:20,829: llmtf.base.darumeru/MultiQ: Processing Dataset: 319.29s |
| INFO: 2024-10-17 07:18:20,829: llmtf.base.darumeru/MultiQ: Results for darumeru/MultiQ: |
| INFO: 2024-10-17 07:18:20,830: llmtf.base.darumeru/MultiQ: {'f1': 0.3485719410941241, 'em': 0.24282982791587} |
| INFO: 2024-10-17 07:18:20,835: llmtf.base.evaluator: Ended eval |
| INFO: 2024-10-17 07:18:20,835: llmtf.base.evaluator: |
| mean darumeru/MultiQ |
| 0.296 0.296 |
| INFO: 2024-10-17 07:18:30,261: llmtf.base.evaluator: Starting eval on ['darumeru/parus'] |
| INFO: 2024-10-17 07:18:30,261: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [131508] |
| INFO: 2024-10-17 07:18:30,261: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] |
| INFO: 2024-10-17 07:18:34,809: llmtf.base.darumeru/PARus: Loading Dataset: 4.55s |
| INFO: 2024-10-17 07:18:39,184: llmtf.base.darumeru/PARus: Processing Dataset: 4.37s |
| INFO: 2024-10-17 07:18:39,184: llmtf.base.darumeru/PARus: Results for darumeru/PARus: |
| INFO: 2024-10-17 07:18:39,194: llmtf.base.darumeru/PARus: {'acc': 0.68} |
| INFO: 2024-10-17 07:18:39,194: llmtf.base.evaluator: Ended eval |
| INFO: 2024-10-17 07:18:39,195: llmtf.base.evaluator: |
| mean darumeru/MultiQ darumeru/PARus |
| 0.488 0.296 0.680 |
| INFO: 2024-10-17 07:18:48,257: llmtf.base.evaluator: Starting eval on ['darumeru/rcb'] |
| INFO: 2024-10-17 07:18:48,258: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [131508] |
| INFO: 2024-10-17 07:18:48,258: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] |
| INFO: 2024-10-17 07:18:52,169: llmtf.base.darumeru/RCB: Loading Dataset: 3.91s |
| INFO: 2024-10-17 07:18:57,742: llmtf.base.darumeru/RCB: Processing Dataset: 5.57s |
| INFO: 2024-10-17 07:18:57,742: llmtf.base.darumeru/RCB: Results for darumeru/RCB: |
| INFO: 2024-10-17 07:18:57,745: llmtf.base.darumeru/RCB: {'acc': 0.5272727272727272, 'f1_macro': 0.47584611730940257} |
| INFO: 2024-10-17 07:18:57,746: llmtf.base.evaluator: Ended eval |
| INFO: 2024-10-17 07:18:57,747: llmtf.base.evaluator: |
| mean darumeru/MultiQ darumeru/PARus darumeru/RCB |
| 0.492 0.296 0.680 0.502 |
| INFO: 2024-10-17 07:19:07,388: llmtf.base.evaluator: Starting eval on ['darumeru/ruopenbookqa'] |
| INFO: 2024-10-17 07:19:07,388: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [131508] |
| INFO: 2024-10-17 07:19:07,388: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] |
| INFO: 2024-10-17 07:19:13,124: llmtf.base.darumeru/ruOpenBookQA: Loading Dataset: 5.74s |
| INFO: 2024-10-17 07:20:12,666: llmtf.base.darumeru/ruOpenBookQA: Processing Dataset: 59.54s |
| INFO: 2024-10-17 07:20:12,666: llmtf.base.darumeru/ruOpenBookQA: Results for darumeru/ruOpenBookQA: |
| INFO: 2024-10-17 07:20:12,678: llmtf.base.darumeru/ruOpenBookQA: {'acc': 0.7207903780068728, 'f1_macro': 0.7206838429510474} |
| INFO: 2024-10-17 07:20:12,689: llmtf.base.evaluator: Ended eval |
| INFO: 2024-10-17 07:20:12,690: llmtf.base.evaluator: |
| mean darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/ruOpenBookQA |
| 0.549 0.296 0.680 0.502 0.721 |
| INFO: 2024-10-17 07:20:21,945: llmtf.base.evaluator: Starting eval on ['darumeru/ruworldtree'] |
| INFO: 2024-10-17 07:20:21,945: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [131508] |
| INFO: 2024-10-17 07:20:21,945: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] |
| INFO: 2024-10-17 07:20:25,640: llmtf.base.darumeru/ruWorldTree: Loading Dataset: 3.69s |
| INFO: 2024-10-17 07:20:28,309: llmtf.base.darumeru/ruWorldTree: Processing Dataset: 2.67s |
| INFO: 2024-10-17 07:20:28,310: llmtf.base.darumeru/ruWorldTree: Results for darumeru/ruWorldTree: |
| INFO: 2024-10-17 07:20:28,312: llmtf.base.darumeru/ruWorldTree: {'acc': 0.8952380952380953, 'f1_macro': 0.8944916936662219} |
| INFO: 2024-10-17 07:20:28,313: llmtf.base.evaluator: Ended eval |
| INFO: 2024-10-17 07:20:28,314: llmtf.base.evaluator: |
| mean darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/ruOpenBookQA darumeru/ruWorldTree |
| 0.619 0.296 0.680 0.502 0.721 0.895 |
| INFO: 2024-10-17 07:20:37,966: llmtf.base.evaluator: Starting eval on ['darumeru/rwsd'] |
| INFO: 2024-10-17 07:20:37,967: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [131508] |
| INFO: 2024-10-17 07:20:37,967: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] |
| INFO: 2024-10-17 07:20:42,582: llmtf.base.darumeru/RWSD: Loading Dataset: 4.62s |
| INFO: 2024-10-17 07:20:47,988: llmtf.base.darumeru/RWSD: Processing Dataset: 5.41s |
| INFO: 2024-10-17 07:20:47,988: llmtf.base.darumeru/RWSD: Results for darumeru/RWSD: |
| INFO: 2024-10-17 07:20:47,989: llmtf.base.darumeru/RWSD: {'acc': 0.5343137254901961} |
| INFO: 2024-10-17 07:20:47,989: llmtf.base.evaluator: Ended eval |
| INFO: 2024-10-17 07:20:47,990: llmtf.base.evaluator: |
| mean darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/ruOpenBookQA darumeru/ruWorldTree |
| 0.605 0.296 0.680 0.502 0.534 0.721 0.895 |
| INFO: 2024-10-17 07:20:57,317: llmtf.base.evaluator: Starting eval on ['daru/treewayextractive'] |
| INFO: 2024-10-17 07:20:57,317: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [131508] |
| INFO: 2024-10-17 07:20:57,317: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] |
| INFO: 2024-10-17 07:21:13,664: llmtf.base.daru/treewayextractive: Loading Dataset: 16.35s |
| INFO: 2024-10-17 07:24:01,803: llmtf.base.daru/treewayextractive: Processing Dataset: 168.14s |
| INFO: 2024-10-17 07:24:01,803: llmtf.base.daru/treewayextractive: Results for daru/treewayextractive: |
| INFO: 2024-10-17 07:24:02,038: llmtf.base.daru/treewayextractive: {'r-prec': 0.3983020202020202} |
| INFO: 2024-10-17 07:24:02,084: llmtf.base.evaluator: Ended eval |
| INFO: 2024-10-17 07:24:02,085: llmtf.base.evaluator: |
| mean daru/treewayextractive darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/ruOpenBookQA darumeru/ruWorldTree |
| 0.575 0.398 0.296 0.680 0.502 0.534 0.721 0.895 |
| INFO: 2024-10-17 07:24:11,344: llmtf.base.evaluator: Starting eval on ['nlpcoreteam/rummlu'] |
| INFO: 2024-10-17 07:24:11,345: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [131508] |
| INFO: 2024-10-17 07:24:11,345: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] |
| INFO: 2024-10-17 07:29:12,497: llmtf.base.nlpcoreteam/ruMMLU: Loading Dataset: 301.15s |
| INFO: 2024-10-17 07:35:18,210: llmtf.base.nlpcoreteam/ruMMLU: Processing Dataset: 365.71s |
| INFO: 2024-10-17 07:35:18,210: llmtf.base.nlpcoreteam/ruMMLU: Results for nlpcoreteam/ruMMLU: |
| INFO: 2024-10-17 07:35:18,279: llmtf.base.nlpcoreteam/ruMMLU: metric |
| subject |
| abstract_algebra 0.330000 |
| anatomy 0.422222 |
| astronomy 0.625000 |
| business_ethics 0.580000 |
| clinical_knowledge 0.592453 |
| college_biology 0.506944 |
| college_chemistry 0.340000 |
| college_computer_science 0.540000 |
| college_mathematics 0.370000 |
| college_medicine 0.549133 |
| college_physics 0.431373 |
| computer_security 0.570000 |
| conceptual_physics 0.536170 |
| econometrics 0.385965 |
| electrical_engineering 0.531034 |
| elementary_mathematics 0.515873 |
| formal_logic 0.333333 |
| global_facts 0.390000 |
| high_school_biology 0.670968 |
| high_school_chemistry 0.487685 |
| high_school_computer_science 0.660000 |
| high_school_european_history 0.733333 |
| high_school_geography 0.696970 |
| high_school_government_and_politics 0.569948 |
| high_school_macroeconomics 0.523077 |
| high_school_mathematics 0.429630 |
| high_school_microeconomics 0.521008 |
| high_school_physics 0.443709 |
| high_school_psychology 0.706422 |
| high_school_statistics 0.523148 |
| high_school_us_history 0.642157 |
| high_school_world_history 0.729958 |
| human_aging 0.587444 |
| human_sexuality 0.641221 |
| international_law 0.694215 |
| jurisprudence 0.638889 |
| logical_fallacies 0.533742 |
| machine_learning 0.419643 |
| management 0.650485 |
| marketing 0.726496 |
| medical_genetics 0.550000 |
| miscellaneous 0.629630 |
| moral_disputes 0.575145 |
| moral_scenarios 0.248045 |
| nutrition 0.614379 |
| philosophy 0.643087 |
| prehistory 0.546296 |
| professional_accounting 0.358156 |
| professional_law 0.373533 |
| professional_medicine 0.500000 |
| professional_psychology 0.495098 |
| public_relations 0.500000 |
| security_studies 0.665306 |
| sociology 0.701493 |
| us_foreign_policy 0.700000 |
| virology 0.433735 |
| world_religions 0.672515 |
| INFO: 2024-10-17 07:35:18,289: llmtf.base.nlpcoreteam/ruMMLU: metric |
| subject |
| STEM 0.496176 |
| humanities 0.566481 |
| other (business, health, misc.) 0.541724 |
| social sciences 0.592209 |
| INFO: 2024-10-17 07:35:18,294: llmtf.base.nlpcoreteam/ruMMLU: {'acc': 0.549147460511024} |
| INFO: 2024-10-17 07:35:18,341: llmtf.base.evaluator: Ended eval |
| INFO: 2024-10-17 07:35:18,343: llmtf.base.evaluator: |
| mean daru/treewayextractive darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/ruOpenBookQA darumeru/ruWorldTree nlpcoreteam/ruMMLU |
| 0.572 0.398 0.296 0.680 0.502 0.534 0.721 0.895 0.549 |
| INFO: 2024-10-17 07:35:27,953: llmtf.base.evaluator: Starting eval on ['nlpcoreteam/enmmlu'] |
| INFO: 2024-10-17 07:35:27,953: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [131508] |
| INFO: 2024-10-17 07:35:27,953: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] |
| INFO: 2024-10-17 07:37:30,758: llmtf.base.nlpcoreteam/enMMLU: Loading Dataset: 122.80s |
| INFO: 2024-10-17 07:43:10,625: llmtf.base.nlpcoreteam/enMMLU: Processing Dataset: 339.87s |
| INFO: 2024-10-17 07:43:10,626: llmtf.base.nlpcoreteam/enMMLU: Results for nlpcoreteam/enMMLU: |
| INFO: 2024-10-17 07:43:10,691: llmtf.base.nlpcoreteam/enMMLU: metric |
| subject |
| abstract_algebra 0.370000 |
| anatomy 0.622222 |
| astronomy 0.697368 |
| business_ethics 0.670000 |
| clinical_knowledge 0.709434 |
| college_biology 0.701389 |
| college_chemistry 0.450000 |
| college_computer_science 0.570000 |
| college_mathematics 0.360000 |
| college_medicine 0.670520 |
| college_physics 0.480392 |
| computer_security 0.720000 |
| conceptual_physics 0.655319 |
| econometrics 0.500000 |
| electrical_engineering 0.565517 |
| elementary_mathematics 0.539683 |
| formal_logic 0.357143 |
| global_facts 0.370000 |
| high_school_biology 0.800000 |
| high_school_chemistry 0.561576 |
| high_school_computer_science 0.670000 |
| high_school_european_history 0.763636 |
| high_school_geography 0.772727 |
| high_school_government_and_politics 0.849741 |
| high_school_macroeconomics 0.679487 |
| high_school_mathematics 0.440741 |
| high_school_microeconomics 0.756303 |
| high_school_physics 0.450331 |
| high_school_psychology 0.849541 |
| high_school_statistics 0.643519 |
| high_school_us_history 0.813725 |
| high_school_world_history 0.835443 |
| human_aging 0.695067 |
| human_sexuality 0.763359 |
| international_law 0.768595 |
| jurisprudence 0.787037 |
| logical_fallacies 0.779141 |
| machine_learning 0.464286 |
| management 0.805825 |
| marketing 0.884615 |
| medical_genetics 0.750000 |
| miscellaneous 0.784163 |
| moral_disputes 0.650289 |
| moral_scenarios 0.270391 |
| nutrition 0.718954 |
| philosophy 0.717042 |
| prehistory 0.737654 |
| professional_accounting 0.496454 |
| professional_law 0.458931 |
| professional_medicine 0.672794 |
| professional_psychology 0.668301 |
| public_relations 0.681818 |
| security_studies 0.718367 |
| sociology 0.810945 |
| us_foreign_policy 0.790000 |
| virology 0.487952 |
| world_religions 0.812865 |
| INFO: 2024-10-17 07:43:10,700: llmtf.base.nlpcoreteam/enMMLU: metric |
| subject |
| STEM 0.563340 |
| humanities 0.673223 |
| other (business, health, misc.) 0.667000 |
| social sciences 0.736716 |
| INFO: 2024-10-17 07:43:10,705: llmtf.base.nlpcoreteam/enMMLU: {'acc': 0.6600696360837558} |
| INFO: 2024-10-17 07:43:10,741: llmtf.base.evaluator: Ended eval |
| INFO: 2024-10-17 07:43:10,743: llmtf.base.evaluator: |
| mean daru/treewayextractive darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/ruOpenBookQA darumeru/ruWorldTree nlpcoreteam/enMMLU nlpcoreteam/ruMMLU |
| 0.582 0.398 0.296 0.680 0.502 0.534 0.721 0.895 0.660 0.549 |
| INFO: 2024-10-17 07:43:20,115: llmtf.base.evaluator: Starting eval on ['daru/treewayabstractive'] |
| INFO: 2024-10-17 07:43:20,115: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [131508] |
| INFO: 2024-10-17 07:43:20,115: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] |
| INFO: 2024-10-17 07:43:24,372: llmtf.base.daru/treewayabstractive: Loading Dataset: 4.26s |
| INFO: 2024-10-17 07:47:01,407: llmtf.base.daru/treewayabstractive: Processing Dataset: 217.03s |
| INFO: 2024-10-17 07:47:01,407: llmtf.base.daru/treewayabstractive: Results for daru/treewayabstractive: |
| INFO: 2024-10-17 07:47:01,408: llmtf.base.daru/treewayabstractive: {'rouge1': 0.32720307606797727, 'rouge2': 0.10857945570692258} |
| INFO: 2024-10-17 07:47:01,409: llmtf.base.evaluator: Ended eval |
| INFO: 2024-10-17 07:47:01,410: llmtf.base.evaluator: |
| mean daru/treewayabstractive daru/treewayextractive darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/ruOpenBookQA darumeru/ruWorldTree nlpcoreteam/enMMLU nlpcoreteam/ruMMLU |
| 0.545 0.218 0.398 0.296 0.680 0.502 0.534 0.721 0.895 0.660 0.549 |
| INFO: 2024-10-17 07:47:10,811: llmtf.base.evaluator: Starting eval on ['darumeru/cp_para_ru'] |
| INFO: 2024-10-17 07:47:10,811: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [131508] |
| INFO: 2024-10-17 07:47:10,811: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] |
| INFO: 2024-10-17 07:47:15,676: llmtf.base.darumeru/cp_para_ru: Loading Dataset: 4.86s |
| INFO: 2024-10-17 07:49:51,029: llmtf.base.darumeru/cp_para_ru: Processing Dataset: 155.35s |
| INFO: 2024-10-17 07:49:51,030: llmtf.base.darumeru/cp_para_ru: Results for darumeru/cp_para_ru: |
| INFO: 2024-10-17 07:49:51,031: llmtf.base.darumeru/cp_para_ru: {'symbol_per_token': 3.76859951568896, 'len': 0.9950709951674359, 'lcs': 0.9} |
| INFO: 2024-10-17 07:49:51,031: llmtf.base.evaluator: Ended eval |
| INFO: 2024-10-17 07:49:51,032: llmtf.base.evaluator: |
| mean daru/treewayabstractive daru/treewayextractive darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/cp_para_ru darumeru/ruOpenBookQA darumeru/ruWorldTree nlpcoreteam/enMMLU nlpcoreteam/ruMMLU |
| 0.578 0.218 0.398 0.296 0.680 0.502 0.534 0.900 0.721 0.895 0.660 0.549 |
|
|
