| INFO: 2024-11-15 16:53:45,193: llmtf.base.evaluator: Starting eval on ['darumeru/multiq'] |
| INFO: 2024-11-15 16:53:45,193: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [145111] |
| INFO: 2024-11-15 16:53:45,193: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] |
| INFO: 2024-11-15 16:53:45,368: llmtf.base.evaluator: Starting eval on ['darumeru/parus'] |
| INFO: 2024-11-15 16:53:45,368: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [145111] |
| INFO: 2024-11-15 16:53:45,368: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] |
| INFO: 2024-11-15 16:53:47,651: llmtf.base.darumeru/PARus: Loading Dataset: 2.28s |
| INFO: 2024-11-15 16:53:48,903: llmtf.base.darumeru/MultiQ: Loading Dataset: 3.71s |
| INFO: 2024-11-15 16:53:49,849: llmtf.base.darumeru/PARus: Processing Dataset: 2.20s |
| INFO: 2024-11-15 16:53:49,849: llmtf.base.darumeru/PARus: Results for darumeru/PARus: |
| INFO: 2024-11-15 16:53:49,859: llmtf.base.darumeru/PARus: {'acc': 0.21} |
| INFO: 2024-11-15 16:53:49,859: llmtf.base.evaluator: Ended eval |
| INFO: 2024-11-15 16:53:49,859: llmtf.base.evaluator: |
| mean darumeru/PARus |
| 0.210 0.210 |
| INFO: 2024-11-15 16:53:58,225: llmtf.base.evaluator: Starting eval on ['darumeru/ruopenbookqa'] |
| INFO: 2024-11-15 16:53:58,225: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [145111] |
| INFO: 2024-11-15 16:53:58,225: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] |
| INFO: 2024-11-15 16:54:01,621: llmtf.base.darumeru/ruOpenBookQA: Loading Dataset: 3.40s |
| INFO: 2024-11-15 16:54:27,891: llmtf.base.darumeru/ruOpenBookQA: Processing Dataset: 26.27s |
| INFO: 2024-11-15 16:54:27,892: llmtf.base.darumeru/ruOpenBookQA: Results for darumeru/ruOpenBookQA: |
| INFO: 2024-11-15 16:54:27,903: llmtf.base.darumeru/ruOpenBookQA: {'acc': 0.46134020618556704, 'f1_macro': 0.4607981854644543} |
| INFO: 2024-11-15 16:54:27,909: llmtf.base.evaluator: Ended eval |
| INFO: 2024-11-15 16:54:27,910: llmtf.base.evaluator: |
| mean darumeru/PARus darumeru/ruOpenBookQA |
| 0.336 0.210 0.461 |
| INFO: 2024-11-15 16:54:36,808: llmtf.base.evaluator: Starting eval on ['darumeru/rwsd'] |
| INFO: 2024-11-15 16:54:36,809: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [145111] |
| INFO: 2024-11-15 16:54:36,809: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] |
| INFO: 2024-11-15 16:54:41,755: llmtf.base.darumeru/RWSD: Loading Dataset: 4.95s |
| INFO: 2024-11-15 16:54:44,248: llmtf.base.darumeru/RWSD: Processing Dataset: 2.49s |
| INFO: 2024-11-15 16:54:44,248: llmtf.base.darumeru/RWSD: Results for darumeru/RWSD: |
| INFO: 2024-11-15 16:54:44,248: llmtf.base.darumeru/RWSD: {'acc': 0.45098039215686275} |
| INFO: 2024-11-15 16:54:44,249: llmtf.base.evaluator: Ended eval |
| INFO: 2024-11-15 16:54:44,250: llmtf.base.evaluator: |
| mean darumeru/PARus darumeru/RWSD darumeru/ruOpenBookQA |
| 0.374 0.210 0.451 0.461 |
| INFO: 2024-11-15 16:54:52,912: llmtf.base.evaluator: Starting eval on ['nlpcoreteam/rummlu'] |
| INFO: 2024-11-15 16:54:52,912: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [145111] |
| INFO: 2024-11-15 16:54:52,912: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] |
| INFO: 2024-11-15 16:56:43,744: llmtf.base.nlpcoreteam/ruMMLU: Loading Dataset: 110.83s |
| INFO: 2024-11-15 16:59:38,517: llmtf.base.nlpcoreteam/ruMMLU: Processing Dataset: 174.77s |
| INFO: 2024-11-15 16:59:38,517: llmtf.base.nlpcoreteam/ruMMLU: Results for nlpcoreteam/ruMMLU: |
| INFO: 2024-11-15 16:59:38,578: llmtf.base.nlpcoreteam/ruMMLU: metric |
| subject |
| abstract_algebra 0.240000 |
| anatomy 0.370370 |
| astronomy 0.453947 |
| business_ethics 0.350000 |
| clinical_knowledge 0.475472 |
| college_biology 0.340278 |
| college_chemistry 0.250000 |
| college_computer_science 0.350000 |
| college_mathematics 0.310000 |
| college_medicine 0.427746 |
| college_physics 0.274510 |
| computer_security 0.540000 |
| conceptual_physics 0.331915 |
| econometrics 0.350877 |
| electrical_engineering 0.496552 |
| elementary_mathematics 0.357143 |
| formal_logic 0.317460 |
| global_facts 0.350000 |
| high_school_biology 0.416129 |
| high_school_chemistry 0.438424 |
| high_school_computer_science 0.550000 |
| high_school_european_history 0.472727 |
| high_school_geography 0.484848 |
| high_school_government_and_politics 0.398964 |
| high_school_macroeconomics 0.325641 |
| high_school_mathematics 0.318519 |
| high_school_microeconomics 0.407563 |
| high_school_physics 0.337748 |
| high_school_psychology 0.453211 |
| high_school_statistics 0.342593 |
| high_school_us_history 0.387255 |
| high_school_world_history 0.468354 |
| human_aging 0.448430 |
| human_sexuality 0.396947 |
| international_law 0.661157 |
| jurisprudence 0.490741 |
| logical_fallacies 0.312883 |
| machine_learning 0.339286 |
| management 0.446602 |
| marketing 0.615385 |
| medical_genetics 0.390000 |
| miscellaneous 0.441890 |
| moral_disputes 0.419075 |
| moral_scenarios 0.242458 |
| nutrition 0.411765 |
| philosophy 0.456592 |
| prehistory 0.373457 |
| professional_accounting 0.336879 |
| professional_law 0.314211 |
| professional_medicine 0.330882 |
| professional_psychology 0.364379 |
| public_relations 0.463636 |
| security_studies 0.395918 |
| sociology 0.452736 |
| us_foreign_policy 0.640000 |
| virology 0.361446 |
| world_religions 0.403509 |
| INFO: 2024-11-15 16:59:38,586: llmtf.base.nlpcoreteam/ruMMLU: metric |
| subject |
| STEM 0.371502 |
| humanities 0.409222 |
| other (business, health, misc.) 0.411205 |
| social sciences 0.427893 |
| INFO: 2024-11-15 16:59:38,591: llmtf.base.nlpcoreteam/ruMMLU: {'acc': 0.4049555256540374} |
| INFO: 2024-11-15 16:59:38,625: llmtf.base.evaluator: Ended eval |
| INFO: 2024-11-15 16:59:38,627: llmtf.base.evaluator: |
| mean darumeru/PARus darumeru/RWSD darumeru/ruOpenBookQA nlpcoreteam/ruMMLU |
| 0.382 0.210 0.451 0.461 0.405 |
| INFO: 2024-11-15 16:59:47,415: llmtf.base.evaluator: Starting eval on ['daru/treewayabstractive'] |
| INFO: 2024-11-15 16:59:47,416: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [145111] |
| INFO: 2024-11-15 16:59:47,416: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] |
| INFO: 2024-11-15 16:59:50,756: llmtf.base.daru/treewayabstractive: Loading Dataset: 3.34s |
| INFO: 2024-11-15 17:00:04,037: llmtf.base.darumeru/MultiQ: Processing Dataset: 375.13s |
| INFO: 2024-11-15 17:00:04,037: llmtf.base.darumeru/MultiQ: Results for darumeru/MultiQ: |
| INFO: 2024-11-15 17:00:04,038: llmtf.base.darumeru/MultiQ: {'f1': 0.2166827508040571, 'em': 0.11567877629063097} |
| INFO: 2024-11-15 17:00:04,043: llmtf.base.evaluator: Ended eval |
| INFO: 2024-11-15 17:00:04,044: llmtf.base.evaluator: |
| mean darumeru/MultiQ darumeru/PARus darumeru/RWSD darumeru/ruOpenBookQA nlpcoreteam/ruMMLU |
| 0.339 0.166 0.210 0.451 0.461 0.405 |
| INFO: 2024-11-15 17:00:12,617: llmtf.base.evaluator: Starting eval on ['darumeru/rcb'] |
| INFO: 2024-11-15 17:00:12,617: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [145111] |
| INFO: 2024-11-15 17:00:12,617: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] |
| INFO: 2024-11-15 17:00:15,035: llmtf.base.darumeru/RCB: Loading Dataset: 2.42s |
| INFO: 2024-11-15 17:00:17,696: llmtf.base.darumeru/RCB: Processing Dataset: 2.66s |
| INFO: 2024-11-15 17:00:17,696: llmtf.base.darumeru/RCB: Results for darumeru/RCB: |
| INFO: 2024-11-15 17:00:17,699: llmtf.base.darumeru/RCB: {'acc': 0.4681818181818182, 'f1_macro': 0.3981025874347421} |
| INFO: 2024-11-15 17:00:17,700: llmtf.base.evaluator: Ended eval |
| INFO: 2024-11-15 17:00:17,701: llmtf.base.evaluator: |
| mean darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/ruOpenBookQA nlpcoreteam/ruMMLU |
| 0.354 0.166 0.210 0.433 0.451 0.461 0.405 |
| INFO: 2024-11-15 17:00:26,448: llmtf.base.evaluator: Starting eval on ['darumeru/ruworldtree'] |
| INFO: 2024-11-15 17:00:26,448: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [145111] |
| INFO: 2024-11-15 17:00:26,448: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] |
| INFO: 2024-11-15 17:00:28,717: llmtf.base.darumeru/ruWorldTree: Loading Dataset: 2.27s |
| INFO: 2024-11-15 17:00:30,112: llmtf.base.darumeru/ruWorldTree: Processing Dataset: 1.39s |
| INFO: 2024-11-15 17:00:30,112: llmtf.base.darumeru/ruWorldTree: Results for darumeru/ruWorldTree: |
| INFO: 2024-11-15 17:00:30,116: llmtf.base.darumeru/ruWorldTree: {'acc': 0.5619047619047619, 'f1_macro': 0.5621556341232892} |
| INFO: 2024-11-15 17:00:30,116: llmtf.base.evaluator: Ended eval |
| INFO: 2024-11-15 17:00:30,116: llmtf.base.evaluator: |
| mean darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/ruOpenBookQA darumeru/ruWorldTree nlpcoreteam/ruMMLU |
| 0.384 0.166 0.210 0.433 0.451 0.461 0.562 0.405 |
| INFO: 2024-11-15 17:00:38,826: llmtf.base.evaluator: Starting eval on ['daru/treewayextractive'] |
| INFO: 2024-11-15 17:00:38,826: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [145111] |
| INFO: 2024-11-15 17:00:38,826: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] |
| INFO: 2024-11-15 17:00:51,532: llmtf.base.daru/treewayextractive: Loading Dataset: 12.71s |
| INFO: 2024-11-15 17:02:39,091: llmtf.base.daru/treewayextractive: Processing Dataset: 107.56s |
| INFO: 2024-11-15 17:02:39,091: llmtf.base.daru/treewayextractive: Results for daru/treewayextractive: |
| INFO: 2024-11-15 17:02:39,318: llmtf.base.daru/treewayextractive: {'r-prec': 0.3703670274170274} |
| INFO: 2024-11-15 17:02:39,357: llmtf.base.evaluator: Ended eval |
| INFO: 2024-11-15 17:02:39,359: llmtf.base.evaluator: |
| mean daru/treewayextractive darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/ruOpenBookQA darumeru/ruWorldTree nlpcoreteam/ruMMLU |
| 0.382 0.370 0.166 0.210 0.433 0.451 0.461 0.562 0.405 |
| INFO: 2024-11-15 17:02:47,961: llmtf.base.evaluator: Starting eval on ['nlpcoreteam/enmmlu'] |
| INFO: 2024-11-15 17:02:47,961: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [145111] |
| INFO: 2024-11-15 17:02:47,961: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] |
| INFO: 2024-11-15 17:03:40,869: llmtf.base.daru/treewayabstractive: Processing Dataset: 230.11s |
| INFO: 2024-11-15 17:03:40,869: llmtf.base.daru/treewayabstractive: Results for daru/treewayabstractive: |
| INFO: 2024-11-15 17:03:40,870: llmtf.base.daru/treewayabstractive: {'rouge1': 0.30478241492489133, 'rouge2': 0.09260347651798656} |
| INFO: 2024-11-15 17:03:40,872: llmtf.base.evaluator: Ended eval |
| INFO: 2024-11-15 17:03:40,872: llmtf.base.evaluator: |
| mean daru/treewayabstractive daru/treewayextractive darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/ruOpenBookQA darumeru/ruWorldTree nlpcoreteam/ruMMLU |
| 0.362 0.199 0.370 0.166 0.210 0.433 0.451 0.461 0.562 0.405 |
| INFO: 2024-11-15 17:04:33,322: llmtf.base.nlpcoreteam/enMMLU: Loading Dataset: 105.36s |
| INFO: 2024-11-15 17:07:11,362: llmtf.base.nlpcoreteam/enMMLU: Processing Dataset: 158.04s |
| INFO: 2024-11-15 17:07:11,362: llmtf.base.nlpcoreteam/enMMLU: Results for nlpcoreteam/enMMLU: |
| INFO: 2024-11-15 17:07:11,425: llmtf.base.nlpcoreteam/enMMLU: metric |
| subject |
| abstract_algebra 0.320000 |
| anatomy 0.481481 |
| astronomy 0.480263 |
| business_ethics 0.470000 |
| clinical_knowledge 0.573585 |
| college_biology 0.506944 |
| college_chemistry 0.310000 |
| college_computer_science 0.380000 |
| college_mathematics 0.290000 |
| college_medicine 0.485549 |
| college_physics 0.303922 |
| computer_security 0.640000 |
| conceptual_physics 0.489362 |
| econometrics 0.412281 |
| electrical_engineering 0.551724 |
| elementary_mathematics 0.402116 |
| formal_logic 0.285714 |
| global_facts 0.350000 |
| high_school_biology 0.541935 |
| high_school_chemistry 0.453202 |
| high_school_computer_science 0.670000 |
| high_school_european_history 0.587879 |
| high_school_geography 0.621212 |
| high_school_government_and_politics 0.616580 |
| high_school_macroeconomics 0.443590 |
| high_school_mathematics 0.348148 |
| high_school_microeconomics 0.529412 |
| high_school_physics 0.344371 |
| high_school_psychology 0.642202 |
| high_school_statistics 0.430556 |
| high_school_us_history 0.573529 |
| high_school_world_history 0.594937 |
| human_aging 0.542601 |
| human_sexuality 0.534351 |
| international_law 0.619835 |
| jurisprudence 0.555556 |
| logical_fallacies 0.558282 |
| machine_learning 0.366071 |
| management 0.582524 |
| marketing 0.782051 |
| medical_genetics 0.430000 |
| miscellaneous 0.579821 |
| moral_disputes 0.546243 |
| moral_scenarios 0.241341 |
| nutrition 0.562092 |
| philosophy 0.517685 |
| prehistory 0.518519 |
| professional_accounting 0.414894 |
| professional_law 0.365711 |
| professional_medicine 0.367647 |
| professional_psychology 0.446078 |
| public_relations 0.536364 |
| security_studies 0.538776 |
| sociology 0.592040 |
| us_foreign_policy 0.700000 |
| virology 0.433735 |
| world_religions 0.631579 |
| INFO: 2024-11-15 17:07:11,433: llmtf.base.nlpcoreteam/enMMLU: metric |
| subject |
| STEM 0.434923 |
| humanities 0.507447 |
| other (business, health, misc.) 0.503999 |
| social sciences 0.551074 |
| INFO: 2024-11-15 17:07:11,438: llmtf.base.nlpcoreteam/enMMLU: {'acc': 0.4993605425093448} |
| INFO: 2024-11-15 17:07:11,470: llmtf.base.evaluator: Ended eval |
| INFO: 2024-11-15 17:07:11,472: llmtf.base.evaluator: |
| mean daru/treewayabstractive daru/treewayextractive darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/ruOpenBookQA darumeru/ruWorldTree nlpcoreteam/enMMLU nlpcoreteam/ruMMLU |
| 0.376 0.199 0.370 0.166 0.210 0.433 0.451 0.461 0.562 0.499 0.405 |
| INFO: 2024-11-15 17:07:20,297: llmtf.base.evaluator: Starting eval on ['darumeru/cp_para_ru'] |
| INFO: 2024-11-15 17:07:20,298: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [145111] |
| INFO: 2024-11-15 17:07:20,298: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] |
| INFO: 2024-11-15 17:07:22,739: llmtf.base.darumeru/cp_para_ru: Loading Dataset: 2.44s |
| INFO: 2024-11-15 17:10:45,613: llmtf.base.darumeru/cp_para_ru: Processing Dataset: 202.87s |
| INFO: 2024-11-15 17:10:45,613: llmtf.base.darumeru/cp_para_ru: Results for darumeru/cp_para_ru: |
| INFO: 2024-11-15 17:10:45,614: llmtf.base.darumeru/cp_para_ru: {'tokens_per_word': 1.8729616000663136, 'symbol_per_token': 4.043905520913176, 'len': 0.927490542336371, 'lcs': 0.05} |
| INFO: 2024-11-15 17:10:45,615: llmtf.base.evaluator: Ended eval |
| INFO: 2024-11-15 17:10:45,615: llmtf.base.evaluator: |
| mean daru/treewayabstractive daru/treewayextractive darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/cp_para_ru darumeru/ruOpenBookQA darumeru/ruWorldTree nlpcoreteam/enMMLU nlpcoreteam/ruMMLU |
| 0.346 0.199 0.370 0.166 0.210 0.433 0.451 0.050 0.461 0.562 0.499 0.405 |
|
|
