Add files using upload-large-folder tool

scripts/yans/lm-evaluation-harness/lm_eval/tasks/bertaqa/README.md

@@ -0,0 +1,50 @@
+# BertaQA
+
+### Paper
+
+Title: BertaQA: How Much Do Language Models Know About Local Culture?
+
+Abstract: https://arxiv.org/abs/2406.07302
+
+Large Language Models (LLMs) exhibit extensive knowledge about the world, but most evaluations have been limited to global or anglocentric subjects. This raises the question of how well these models perform on topics relevant to other cultures, whose presence on the web is not that prominent. To address this gap, we introduce BertaQA, a multiple-choice trivia dataset that is parallel in English and Basque. The dataset consists of a local subset with questions pertinent to the Basque culture, and a global subset with questions of broader interest. We find that state-of-the-art LLMs struggle with local cultural knowledge, even as they excel on global topics. However, we show that continued pre-training in Basque significantly improves the models' performance on Basque culture, even when queried in English. To our knowledge, this is the first solid evidence of knowledge transfer from a low-resource to a high-resource language. Our analysis sheds light on the complex interplay between language and knowledge, and reveals that some prior findings do not fully hold when reassessed on local topics. Our dataset and evaluation code are available under open licenses at https://github.com/juletx/BertaQA.
+
+Homepage: https://github.com/juletx/BertaQA
+
+### Citation
+
+```
+@misc{etxaniz2024bertaqa,
+      title={BertaQA: How Much Do Language Models Know About Local Culture?},
+      author={Julen Etxaniz and Gorka Azkune and Aitor Soroa and Oier Lopez de Lacalle and Mikel Artetxe},
+      year={2024},
+      eprint={2406.07302},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL}
+}
+```
+
+### Groups and Tasks
+
+#### Groups
+
+- `bertaqa`: Group of BertaQA tasks.
+
+#### Tasks
+
+- `bertaqa_eu`: Trivia questions in Basque.
+- `bertaqa_en`: Trivia questions in English, human-translated from Basque.
+- `bertaqa_en_mt_*`: Trivia questions in English, machine-translated from Basque with different models.
+
+### Checklist
+
+For adding novel benchmarks/datasets to the library:
+
+- [ ] Is the task an existing benchmark in the literature?
+  - [ ] Have you referenced the original paper that introduced the task?
+  - [ ] If yes, does the original paper provide a reference implementation? If so, have you checked against the reference implementation and documented how to run such a test?
+
+If other tasks on this dataset are already supported:
+
+- [ ] Is the "Main" variant of this task clearly denoted?
+- [ ] Have you provided a short sentence in a README on what each new variant adds / evaluates?
+- [ ] Have you noted which, if any, published evaluation setups are matched by this variant?

scripts/yans/lm-evaluation-harness/lm_eval/tasks/bertaqa/_bertaqa_template

@@ -0,0 +1,15 @@
+tag: bertaqa
+dataset_path: HiTZ/BertaQA
+dataset_name: null
+validation_split: null
+test_split: test
+fewshot_split: test
+output_type: multiple_choice
+doc_to_choice: ["A", "B", "C"]
+doc_to_target: answer
+metric_list:
+  - metric: acc
+    aggregation: mean
+    higher_is_better: true
+metadata:
+  version: 0.0

scripts/yans/lm-evaluation-harness/lm_eval/tasks/bertaqa/bertaqa_en.yaml

@@ -0,0 +1,4 @@
+task: bertaqa_en
+include: _bertaqa_template
+dataset_name: en
+doc_to_text: "Question: {{question}}\nA: {{candidates[0]}}\nB: {{candidates[1]}}\nC: {{candidates[2]}}\nAnswer:"

scripts/yans/lm-evaluation-harness/lm_eval/tasks/bertaqa/bertaqa_en_mt_gemma-7b.yaml

@@ -0,0 +1,4 @@
+task: bertaqa_en_mt_gemma-7b
+include: _bertaqa_template
+dataset_name: en_mt_gemma-7b
+doc_to_text: "Question: {{question}}\nA: {{candidates[0]}}\nB: {{candidates[1]}}\nC: {{candidates[2]}}\nAnswer:"

scripts/yans/lm-evaluation-harness/lm_eval/tasks/bertaqa/bertaqa_en_mt_hitz.yaml

@@ -0,0 +1,4 @@
+task: bertaqa_en_mt_hitz
+include: _bertaqa_template
+dataset_name: en_mt_hitz
+doc_to_text: "Question: {{question}}\nA: {{candidates[0]}}\nB: {{candidates[1]}}\nC: {{candidates[2]}}\nAnswer:"

scripts/yans/lm-evaluation-harness/lm_eval/tasks/bertaqa/bertaqa_en_mt_itzuli.yaml

@@ -0,0 +1,4 @@
+task: bertaqa_en_mt_itzuli
+include: _bertaqa_template
+dataset_name: en_mt_itzuli
+doc_to_text: "Question: {{question}}\nA: {{candidates[0]}}\nB: {{candidates[1]}}\nC: {{candidates[2]}}\nAnswer:"

scripts/yans/lm-evaluation-harness/lm_eval/tasks/bertaqa/bertaqa_en_mt_latxa-13b-v1.1.yaml

@@ -0,0 +1,4 @@
+task: bertaqa_en_mt_latxa-13b-v1.1
+include: _bertaqa_template
+dataset_name: en_mt_latxa-13b-v1.1
+doc_to_text: "Question: {{question}}\nA: {{candidates[0]}}\nB: {{candidates[1]}}\nC: {{candidates[2]}}\nAnswer:"

scripts/yans/lm-evaluation-harness/lm_eval/tasks/bertaqa/bertaqa_en_mt_latxa-13b-v1.yaml

@@ -0,0 +1,4 @@
+task: bertaqa_en_mt_latxa-13b-v1
+include: _bertaqa_template
+dataset_name: en_mt_latxa-13b-v1
+doc_to_text: "Question: {{question}}\nA: {{candidates[0]}}\nB: {{candidates[1]}}\nC: {{candidates[2]}}\nAnswer:"

scripts/yans/lm-evaluation-harness/lm_eval/tasks/bertaqa/bertaqa_en_mt_latxa-70b-v1.1.yaml

@@ -0,0 +1,4 @@
+task: bertaqa_en_mt_latxa-70b-v1.1
+include: _bertaqa_template
+dataset_name: en_mt_latxa-70b-v1.1
+doc_to_text: "Question: {{question}}\nA: {{candidates[0]}}\nB: {{candidates[1]}}\nC: {{candidates[2]}}\nAnswer:"

scripts/yans/lm-evaluation-harness/lm_eval/tasks/bertaqa/bertaqa_en_mt_latxa-70b-v1.yaml

@@ -0,0 +1,4 @@
+task: bertaqa_en_mt_latxa-70b-v1
+include: _bertaqa_template
+dataset_name: en_mt_latxa-70b-v1
+doc_to_text: "Question: {{question}}\nA: {{candidates[0]}}\nB: {{candidates[1]}}\nC: {{candidates[2]}}\nAnswer:"

scripts/yans/lm-evaluation-harness/lm_eval/tasks/bertaqa/bertaqa_en_mt_latxa-7b-v1.1.yaml

@@ -0,0 +1,4 @@
+task: bertaqa_en_mt_latxa-7b-v1.1
+include: _bertaqa_template
+dataset_name: en_mt_latxa-7b-v1.1
+doc_to_text: "Question: {{question}}\nA: {{candidates[0]}}\nB: {{candidates[1]}}\nC: {{candidates[2]}}\nAnswer:"

scripts/yans/lm-evaluation-harness/lm_eval/tasks/bertaqa/bertaqa_en_mt_latxa-7b-v1.yaml

@@ -0,0 +1,4 @@
+task: bertaqa_en_mt_latxa-7b-v1
+include: _bertaqa_template
+dataset_name: en_mt_latxa-7b-v1
+doc_to_text: "Question: {{question}}\nA: {{candidates[0]}}\nB: {{candidates[1]}}\nC: {{candidates[2]}}\nAnswer:"

scripts/yans/lm-evaluation-harness/lm_eval/tasks/bertaqa/bertaqa_en_mt_llama-2-13b.yaml

@@ -0,0 +1,4 @@
+task: bertaqa_en_mt_llama-2-13b
+include: _bertaqa_template
+dataset_name: en_mt_llama-2-13b
+doc_to_text: "Question: {{question}}\nA: {{candidates[0]}}\nB: {{candidates[1]}}\nC: {{candidates[2]}}\nAnswer:"

scripts/yans/lm-evaluation-harness/lm_eval/tasks/bertaqa/bertaqa_en_mt_llama-2-70b.yaml

@@ -0,0 +1,4 @@
+task: bertaqa_en_mt_llama-2-70b
+include: _bertaqa_template
+dataset_name: en_mt_llama-2-70b
+doc_to_text: "Question: {{question}}\nA: {{candidates[0]}}\nB: {{candidates[1]}}\nC: {{candidates[2]}}\nAnswer:"

scripts/yans/lm-evaluation-harness/lm_eval/tasks/bertaqa/bertaqa_en_mt_llama-2-7b.yaml

@@ -0,0 +1,4 @@
+task: bertaqa_en_mt_llama-2-7b
+include: _bertaqa_template
+dataset_name: en_mt_llama-2-7b
+doc_to_text: "Question: {{question}}\nA: {{candidates[0]}}\nB: {{candidates[1]}}\nC: {{candidates[2]}}\nAnswer:"

scripts/yans/lm-evaluation-harness/lm_eval/tasks/bertaqa/bertaqa_en_mt_madlad.yaml

@@ -0,0 +1,4 @@
+task: bertaqa_en_mt_madlad
+include: _bertaqa_template
+dataset_name: en_mt_madlad
+doc_to_text: "Question: {{question}}\nA: {{candidates[0]}}\nB: {{candidates[1]}}\nC: {{candidates[2]}}\nAnswer:"

scripts/yans/lm-evaluation-harness/lm_eval/tasks/bertaqa/bertaqa_en_mt_nllb.yaml

@@ -0,0 +1,4 @@
+task: bertaqa_en_mt_nllb
+include: _bertaqa_template
+dataset_name: en_mt_nllb
+doc_to_text: "Question: {{question}}\nA: {{candidates[0]}}\nB: {{candidates[1]}}\nC: {{candidates[2]}}\nAnswer:"

scripts/yans/lm-evaluation-harness/lm_eval/tasks/bertaqa/bertaqa_eu.yaml

@@ -0,0 +1,4 @@
+task: bertaqa_eu
+include: _bertaqa_template
+dataset_name: eu
+doc_to_text: "Galdera: {{question}}\nA: {{candidates[0]}}\nB: {{candidates[1]}}\nC: {{candidates[2]}}\nErantzuna:"

scripts/yans/lm-evaluation-harness/lm_eval/tasks/cmmlu/_generate_configs.py

@@ -0,0 +1,164 @@
+"""
+Take in a YAML, and output all other splits with this YAML
+"""
+
+import argparse
+import os
+
+import yaml
+from tqdm import tqdm
+
+from lm_eval.utils import eval_logger
+
+
+SUBJECTS = {
+    "agronomy": "农学",
+    "anatomy": "解剖学",
+    "ancient_chinese": "古汉语",
+    "arts": "艺术学",
+    "astronomy": "天文学",
+    "business_ethics": "商业伦理",
+    "chinese_civil_service_exam": "中国公务员考试",
+    "chinese_driving_rule": "中国驾驶规则",
+    "chinese_food_culture": "中国饮食文化",
+    "chinese_foreign_policy": "中国外交政策",
+    "chinese_history": "中国历史",
+    "chinese_literature": "中国文学",
+    "chinese_teacher_qualification": "中国教师资格",
+    "clinical_knowledge": "临床知识",
+    "college_actuarial_science": "大学精算学",
+    "college_education": "大学教育学",
+    "college_engineering_hydrology": "大学工程水文学",
+    "college_law": "大学法律",
+    "college_mathematics": "大学数学",
+    "college_medical_statistics": "大学医学统计",
+    "college_medicine": "大学医学",
+    "computer_science": "计算机科学",
+    "computer_security": "计算机安全",
+    "conceptual_physics": "概念物理学",
+    "construction_project_management": "建设工程管理",
+    "economics": "经济学",
+    "education": "教育学",
+    "electrical_engineering": "电气工程",
+    "elementary_chinese": "小学语文",
+    "elementary_commonsense": "小学常识",
+    "elementary_information_and_technology": "小学信息技术",
+    "elementary_mathematics": "初等数学",
+    "ethnology": "民族学",
+    "food_science": "食品科学",
+    "genetics": "遗传学",
+    "global_facts": "全球事实",
+    "high_school_biology": "高中生物",
+    "high_school_chemistry": "高中化学",
+    "high_school_geography": "高中地理",
+    "high_school_mathematics": "高中数学",
+    "high_school_physics": "高中物理学",
+    "high_school_politics": "高中政治",
+    "human_sexuality": "人类性行为",
+    "international_law": "国际法学",
+    "journalism": "新闻学",
+    "jurisprudence": "法理学",
+    "legal_and_moral_basis": "法律与道德基础",
+    "logical": "逻辑学",
+    "machine_learning": "机器学习",
+    "management": "管理学",
+    "marketing": "市场营销",
+    "marxist_theory": "马克思主义理论",
+    "modern_chinese": "现代汉语",
+    "nutrition": "营养学",
+    "philosophy": "哲学",
+    "professional_accounting": "专业会计",
+    "professional_law": "专业法学",
+    "professional_medicine": "专业医学",
+    "professional_psychology": "专业心理学",
+    "public_relations": "公共关系",
+    "security_study": "安全研究",
+    "sociology": "社会学",
+    "sports_science": "体育学",
+    "traditional_chinese_medicine": "中医中药",
+    "virology": "病毒学",
+    "world_history": "世界历史",
+    "world_religions": "世界宗教",
+}
+
+
+def parse_args():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--base_yaml_path", required=True)
+    parser.add_argument("--save_prefix_path", default="cmmlu")
+    parser.add_argument("--cot_prompt_path", default=None)
+    parser.add_argument("--task_prefix", default="")
+    return parser.parse_args()
+
+
+if __name__ == "__main__":
+    args = parse_args()
+
+    # get filename of base_yaml so we can `"include": ` it in our other YAMLs.
+    base_yaml_name = os.path.split(args.base_yaml_path)[-1]
+    with open(args.base_yaml_path, encoding="utf-8") as f:
+        base_yaml = yaml.full_load(f)
+
+    if args.cot_prompt_path is not None:
+        import json
+
+        with open(args.cot_prompt_path, encoding="utf-8") as f:
+            cot_file = json.load(f)
+
+    for subject_eng, subject_zh in tqdm(SUBJECTS.items()):
+        if args.cot_prompt_path is not None:
+            description = cot_file[subject_eng]
+        else:
+            description = (
+                f"以下是关于{subject_zh}的单项选择题，请直接给出正确答案的选项。\n\n"
+            )
+
+        yaml_dict = {
+            "include": base_yaml_name,
+            "task": f"cmmlu_{args.task_prefix}_{subject_eng}"
+            if args.task_prefix != ""
+            else f"cmmlu_{subject_eng}",
+            "dataset_name": subject_eng,
+            "description": description,
+        }
+
+        file_save_path = args.save_prefix_path + f"_{subject_eng}.yaml"
+        eval_logger.info(f"Saving yaml for subset {subject_eng} to {file_save_path}")
+        with open(file_save_path, "w", encoding="utf-8") as yaml_file:
+            yaml.dump(
+                yaml_dict,
+                yaml_file,
+                width=float("inf"),
+                allow_unicode=True,
+                default_style='"',
+            )
+
+    # write group config out
+
+    group_yaml_dict = {
+        "group": "cmmlu",
+        "task": [
+            (
+                f"cmmlu_{args.task_prefix}_{subject_eng}"
+                if args.task_prefix != ""
+                else f"cmmlu_{subject_eng}"
+            )
+            for subject_eng in SUBJECTS.keys()
+        ],
+        "aggregate_metric_list": [
+            {"metric": "acc", "aggregation": "mean", "weight_by_size": True},
+            {"metric": "acc_norm", "aggregation": "mean", "weight_by_size": True},
+        ],
+        "metadata": {"version": 0.0},
+    }
+
+    file_save_path = "_" + args.save_prefix_path + ".yaml"
+
+    with open(file_save_path, "w", encoding="utf-8") as group_yaml_file:
+        yaml.dump(
+            group_yaml_dict,
+            group_yaml_file,
+            width=float("inf"),
+            allow_unicode=True,
+            default_style='"',
+        )

scripts/yans/lm-evaluation-harness/lm_eval/tasks/cmmlu/cmmlu_agronomy.yaml

@@ -0,0 +1,4 @@
+"dataset_name": "agronomy"
+"description": "以下是关于农学的单项选择题，请直接给出正确答案的选项。\n\n"
+"include": "_default_template_yaml"
+"task": "cmmlu_agronomy"

scripts/yans/lm-evaluation-harness/lm_eval/tasks/cmmlu/cmmlu_chinese_food_culture.yaml

@@ -0,0 +1,4 @@
+"dataset_name": "chinese_food_culture"
+"description": "以下是关于中国饮食文化的单项选择题，请直接给出正确答案的选项。\n\n"
+"include": "_default_template_yaml"
+"task": "cmmlu_chinese_food_culture"

scripts/yans/lm-evaluation-harness/lm_eval/tasks/cmmlu/cmmlu_conceptual_physics.yaml

@@ -0,0 +1,4 @@
+"dataset_name": "conceptual_physics"
+"description": "以下是关于概念物理学的单项选择题，请直接给出正确答案的选项。\n\n"
+"include": "_default_template_yaml"
+"task": "cmmlu_conceptual_physics"

scripts/yans/lm-evaluation-harness/lm_eval/tasks/cmmlu/cmmlu_default_business_ethics.yaml

@@ -0,0 +1,4 @@
+"dataset_name": "business_ethics"
+"description": "以下是关于商业伦理的单项选择题，请直接给出正确答案的选项。\n\n"
+"include": "_default_template_yaml"
+"task": "cmmlu_business_ethics"

scripts/yans/lm-evaluation-harness/lm_eval/tasks/cmmlu/cmmlu_default_chinese_foreign_policy.yaml

@@ -0,0 +1,4 @@
+"dataset_name": "chinese_foreign_policy"
+"description": "以下是关于中国外交政策的单项选择题，请直接给出正确答案的选项。\n\n"
+"include": "_default_template_yaml"
+"task": "cmmlu_chinese_foreign_policy"

scripts/yans/lm-evaluation-harness/lm_eval/tasks/cmmlu/cmmlu_default_chinese_teacher_qualification.yaml

@@ -0,0 +1,4 @@
+"dataset_name": "chinese_teacher_qualification"
+"description": "以下是关于中国教师资格的单项选择题，请直接给出正确答案的选项。\n\n"
+"include": "_default_template_yaml"
+"task": "cmmlu_chinese_teacher_qualification"

scripts/yans/lm-evaluation-harness/lm_eval/tasks/cmmlu/cmmlu_default_construction_project_management.yaml

@@ -0,0 +1,4 @@
+"dataset_name": "construction_project_management"
+"description": "以下是关于建设工程管理的单项选择题，请直接给出正确答案的选项。\n\n"
+"include": "_default_template_yaml"
+"task": "cmmlu_construction_project_management"

scripts/yans/lm-evaluation-harness/lm_eval/tasks/cmmlu/cmmlu_default_high_school_geography.yaml

@@ -0,0 +1,4 @@
+"dataset_name": "high_school_geography"
+"description": "以下是关于高中地理的单项选择题，请直接给出正确答案的选项。\n\n"
+"include": "_default_template_yaml"
+"task": "cmmlu_high_school_geography"

scripts/yans/lm-evaluation-harness/lm_eval/tasks/cmmlu/cmmlu_default_jurisprudence.yaml

@@ -0,0 +1,4 @@
+"dataset_name": "jurisprudence"
+"description": "以下是关于法理学的单项选择题，请直接给出正确答案的选项。\n\n"
+"include": "_default_template_yaml"
+"task": "cmmlu_jurisprudence"

scripts/yans/lm-evaluation-harness/lm_eval/tasks/cmmlu/cmmlu_global_facts.yaml

@@ -0,0 +1,4 @@
+"dataset_name": "global_facts"
+"description": "以下是关于全球事实的单项选择题，请直接给出正确答案的选项。\n\n"
+"include": "_default_template_yaml"
+"task": "cmmlu_global_facts"

scripts/yans/lm-evaluation-harness/lm_eval/tasks/cmmlu/cmmlu_human_sexuality.yaml

@@ -0,0 +1,4 @@
+"dataset_name": "human_sexuality"
+"description": "以下是关于人类性行为的单项选择题，请直接给出正确答案的选项。\n\n"
+"include": "_default_template_yaml"
+"task": "cmmlu_human_sexuality"

scripts/yans/lm-evaluation-harness/lm_eval/tasks/cmmlu/cmmlu_management.yaml

@@ -0,0 +1,4 @@
+"dataset_name": "management"
+"description": "以下是关于管理学的单项选择题，请直接给出正确答案的选项。\n\n"
+"include": "_default_template_yaml"
+"task": "cmmlu_management"

scripts/yans/lm-evaluation-harness/lm_eval/tasks/eus_trivia/README.md

@@ -0,0 +1,54 @@
+# EusTrivia
+
+### Paper
+
+Title: Latxa: An Open Language Model and Evaluation Suite for Basque
+
+Abstract: https://arxiv.org/abs/2403.20266
+
+EusTrivia consists of 1,715 trivia questions from multiple online sources. 56.3\% of the questions are elementary level (grades 3-6), while the rest are considered challenging. A significant portion of the questions focus specifically on the Basque Country, its language and culture. Each multiple-choice question contains two, three or four choices (3.84 on average) and a single correct answer. Five areas of knowledge are covered:
+
+- **Humanities and Natural Sciences** (27.8%): This category encompasses questions about history, geography, biology, ecology and other social and natural sciences.
+- **Leisure and Art** (24.5%): This category includes questions on sports and athletes, performative and plastic arts and artists, architecture, cultural events, and related topics.
+- **Music** (16.0%): Here are grouped all the questions about music and musicians, both classical and contemporary.
+- **Language and Literature** (17.1%): This category is concerned with all kinds of literature productions and writers, as well as metalinguistic questions (e.g., definitions, synonyms, and word usage).
+- **Mathematics and ICT** (14.5%): This category covers mathematical problems and questions about ICT, as well as questions about people known for their contributions to these fields of knowledge.
+
+Homepage: https://github.com/hitz-zentroa/latxa
+
+
+### Citation
+
+```
+@misc{etxaniz2024latxa,
+      title={Latxa: An Open Language Model and Evaluation Suite for Basque},
+      author={Julen Etxaniz and Oscar Sainz and Naiara Perez and Itziar Aldabe and German Rigau and Eneko Agirre and Aitor Ormazabal and Mikel Artetxe and Aitor Soroa},
+      year={2024},
+      eprint={2403.20266},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL}
+}
+```
+
+### Groups and Tasks
+
+#### Groups
+
+There are no groups.
+
+#### Tasks
+
+* `eus_trivia`: EusTrivia consists of 1,715 trivia questions from multiple online sources.
+
+### Checklist
+
+For adding novel benchmarks/datasets to the library:
+* [ ] Is the task an existing benchmark in the literature?
+  * [ ] Have you referenced the original paper that introduced the task?
+  * [ ] If yes, does the original paper provide a reference implementation? If so, have you checked against the reference implementation and documented how to run such a test?
+
+
+If other tasks on this dataset are already supported:
+* [ ] Is the "Main" variant of this task clearly denoted?
+* [ ] Have you provided a short sentence in a README on what each new variant adds / evaluates?
+* [ ] Have you noted which, if any, published evaluation setups are matched by this variant?

scripts/yans/lm-evaluation-harness/lm_eval/tasks/eus_trivia/eus_trivia.yaml

@@ -0,0 +1,16 @@
+dataset_path: HiTZ/EusTrivia
+dataset_name: default
+task: eus_trivia
+doc_to_text: !function utils.doc_to_text
+doc_to_choice: !function utils.doc_to_choice
+validation_split: null
+test_split: test
+fewshot_split: test
+output_type: multiple_choice
+doc_to_target: answer
+metric_list:
+  - metric: acc
+    aggregation: mean
+    higher_is_better: true
+metadata:
+  version: 0.0

scripts/yans/lm-evaluation-harness/lm_eval/tasks/eus_trivia/utils.py

@@ -0,0 +1,41 @@
+from typing import List
+
+
+letters = ["A", "B", "C", "D"]
+
+
+def doc_to_text(doc) -> str:
+    """
+    Converts a document to a formatted string.
+
+    Args:
+        doc (dict): A dictionary containing the document information.
+
+    Returns:
+        str: A formatted string containing the question and answer choices.
+    """
+    candidates = doc["candidates"]
+    num_choices = len(candidates)
+    if num_choices < 2:
+        raise ValueError("Invalid number of candidates")
+    choices = letters[:num_choices]
+    formatted_choices = "\n".join(
+        [f"{choice}: {candidates[i]}" for i, choice in enumerate(choices)]
+    )
+    return f"Galdera: {doc['question']}\n{formatted_choices}\nErantzuna:"
+
+
+def doc_to_choice(doc) -> List[str]:
+    """
+    Returns the answer choices for a document.
+
+    Args:
+        doc (dict): A dictionary containing the document information.
+
+    Returns:
+        list: A list of strings containing the answer choices.
+    """
+    num_choices = len(doc["candidates"])
+    if num_choices < 2:
+        raise ValueError("Invalid number of candidates")
+    return letters[:num_choices]

scripts/yans/lm-evaluation-harness/lm_eval/tasks/kormedmcqa/README.md

@@ -0,0 +1,47 @@
+# KorMedMCQA
+
+### Paper
+
+Title: `KorMedMCQA: Multi-Choice Question Answering Benchmark for Korean Healthcare Professional Licensing Examinations`
+
+Abstract: `We introduce KorMedMCQA, the first Korean multiple-choice question answering (MCQA) benchmark derived from Korean healthcare professional licensing examinations, covering from the year 2012 to year 2023. This dataset consists of a selection of questions from the license examinations for doctors, nurses, and pharmacists, featuring a diverse array of subjects. We conduct baseline experiments on various large language models, including proprietary/open-source, multilingual/Korean-additional pretrained, and clinical context pretrained models, highlighting the potential for further enhancements. We make our data publicly available on HuggingFace and provide a evaluation script via LM-Harness, inviting further exploration and advancement in Korean healthcare environments.`
+
+
+Paper : https://arxiv.org/abs/2403.01469
+
+Homepage: https://huggingface.co/datasets/sean0042/KorMedMCQA
+
+
+### Citation
+
+```
+@article{kweon2024kormedmcqa,
+      title={KorMedMCQA: Multi-Choice Question Answering Benchmark for Korean Healthcare Professional Licensing Examinations},
+      author={Sunjun Kweon and Byungjin Choi and Minkyu Kim and Rae Woong Park and Edward Choi},
+      journal={arXiv preprint arXiv:2403.01469},
+      year={2024}
+}
+```
+
+### Groups and Tasks
+
+* `kormedmcqa`: Runs `kormedmcqa_doctor`, `kormedmcqa_nurse`, and `kormedmcqa_pharm`.
+
+#### Tasks
+
+* `kormedmcqa_doctor`: `Official Korean Doctor Examination`
+* `kormedmcqa_nurse`: `Official Korean Nurse Examination`
+* `kormedmcqa_pharm`: `Official Korean Pharmacist Examination`
+
+### Checklist
+
+For adding novel benchmarks/datasets to the library:
+* [x] Is the task an existing benchmark in the literature?
+  * [x] Have you referenced the original paper that introduced the task?
+  * [x] If yes, does the original paper provide a reference implementation? If so, have you checked against the reference implementation and documented how to run such a test?
+
+
+If other tasks on this dataset are already supported:
+* [ ] Is the "Main" variant of this task clearly denoted?
+* [ ] Have you provided a short sentence in a README on what each new variant adds / evaluates?
+* [ ] Have you noted which, if any, published evaluation setups are matched by this variant?

scripts/yans/lm-evaluation-harness/lm_eval/tasks/kormedmcqa/_kormedmcqa.yaml

@@ -0,0 +1,11 @@
+group: kormedmcqa
+task:
+  - kormedmcqa_doctor
+  - kormedmcqa_nurse
+  - kormedmcqa_pharm
+aggregate_metric_list:
+  - metric: exact_match
+    aggregation: mean
+    weight_by_size: true
+metadata:
+  version: 0.0

scripts/yans/lm-evaluation-harness/lm_eval/tasks/kormedmcqa/kormedmcqa_doctor.yaml

@@ -0,0 +1,26 @@
+task : kormedmcqa_doctor
+dataset_path : sean0042/KorMedMCQA
+dataset_name : doctor
+test_split : test
+fewshot_split : dev
+fewshot_config:
+  sampler: first_n
+output_type: generate_until
+doc_to_text: "{{question.strip()}}\nA. {{A}}\nB. {{B}}\nC. {{C}}\nD. {{D}}\nE. {{E}}\n정답："
+doc_to_target: "{{['A', 'B', 'C', 'D', 'E'][answer-1]}}"
+metric_list:
+  - metric: exact_match
+    aggregation: mean
+    higher_is_better: true
+    ignore_case: true
+    ignore_punctuation: true
+    regexes_to_ignore:
+          - " "
+generation_kwargs:
+  until:
+    - "Q:"
+    - "\n\n"
+    - "</s>"
+    - "."
+  do_sample: false
+  temperature: 0.0

scripts/yans/lm-evaluation-harness/lm_eval/tasks/kormedmcqa/kormedmcqa_nurse.yaml

@@ -0,0 +1,26 @@
+task : kormedmcqa_nurse
+dataset_path : sean0042/KorMedMCQA
+dataset_name : nurse
+test_split : test
+fewshot_split : dev
+fewshot_config:
+  sampler: first_n
+output_type: generate_until
+doc_to_text: "{{question.strip()}}\nA. {{A}}\nB. {{B}}\nC. {{C}}\nD. {{D}}\nE. {{E}}\n정답："
+doc_to_target: "{{['A', 'B', 'C', 'D', 'E'][answer-1]}}"
+metric_list:
+  - metric: exact_match
+    aggregation: mean
+    higher_is_better: true
+    ignore_case: true
+    ignore_punctuation: true
+    regexes_to_ignore:
+          - " "
+generation_kwargs:
+  until:
+    - "Q:"
+    - "\n\n"
+    - "</s>"
+    - "."
+  do_sample: false
+  temperature: 0.0

scripts/yans/lm-evaluation-harness/lm_eval/tasks/kormedmcqa/kormedmcqa_pharm.yaml

@@ -0,0 +1,26 @@
+task : kormedmcqa_pharm
+dataset_path : sean0042/KorMedMCQA
+dataset_name : pharm
+test_split : test
+fewshot_split : dev
+fewshot_config:
+  sampler: first_n
+output_type: generate_until
+doc_to_text: "{{question.strip()}}\nA. {{A}}\nB. {{B}}\nC. {{C}}\nD. {{D}}\nE. {{E}}\n정답："
+doc_to_target: "{{['A', 'B', 'C', 'D', 'E'][answer-1]}}"
+metric_list:
+  - metric: exact_match
+    aggregation: mean
+    higher_is_better: true
+    ignore_case: true
+    ignore_punctuation: true
+    regexes_to_ignore:
+          - " "
+generation_kwargs:
+  until:
+    - "Q:"
+    - "\n\n"
+    - "</s>"
+    - "."
+  do_sample: false
+  temperature: 0.0

scripts/yans/lm-evaluation-harness/lm_eval/tasks/logiqa2/README.md

@@ -0,0 +1,52 @@
+# LogiQA 2.0
+
+### Paper
+
+LogiQA 2.0 — An Improved Dataset for Logical Reasoning in Natural Language Understanding https://ieeexplore.ieee.org/document/10174688
+
+
+The dataset is an amendment and re-annotation of LogiQA in 2020, a large-scale logical reasoning reading comprehension dataset adapted from the Chinese Civil Service Examination. This new version has an increased data size, the texts are refined with manual translation by professionals, and improved by removing items with distinctive cultural features like Chinese idioms.
+
+Furthermore, a two-way natural language inference (NLI) task is introduced, resulting in 35k premise-hypothesis pairs with gold labels, making it the first large-scale NLI dataset for complex logical reasoning
+
+Homepage: https://github.com/csitfun/LogiQA2.0
+
+### Citation
+
+```bibtex
+@ARTICLE{10174688,
+  author={Liu, Hanmeng and Liu, Jian and Cui, Leyang and Teng, Zhiyang and Duan, Nan and Zhou, Ming and Zhang, Yue},
+  journal={IEEE/ACM Transactions on Audio, Speech, and Language Processing},
+  title={LogiQA 2.0 — An Improved Dataset for Logical Reasoning in Natural Language Understanding},
+  year={2023},
+  volume={},
+  number={},
+  pages={1-16},
+  doi={10.1109/TASLP.2023.3293046}}
+```
+
+### Groups and Tasks
+
+#### Groups
+
+* Not part of a group yet
+
+#### Tasks
+
+* `logiqa2_zh`: The original dataset in Chinese.
+* `logiqa2_NLI`: The NLI version of the dataset converted from the MRC version.
+* `logieval`: Prompt based; https://github.com/csitfun/LogiEval
+
+NOTE! The subtasks have not been verified yet.
+
+### Checklist
+
+* [x] Is the task an existing benchmark in the literature?
+  * [x] Have you referenced the original paper that introduced the task?
+  * [x] If yes, does the original paper provide a reference implementation?
+    * [x] The original paper does not. There is another implementation of this task, but it designed for instruction tuned models: https://github.com/csitfun/LogiEval
+
+If other tasks on this dataset are already supported:
+* [x] Is the "Main" variant of this task clearly denoted?
+* [x] Have you provided a short sentence in a README on what each new variant adds / evaluates?
+* [ ] Have you noted which, if any, published evaluation setups are matched by this variant?

scripts/yans/lm-evaluation-harness/lm_eval/tasks/logiqa2/logieval.yaml

@@ -0,0 +1,29 @@
+task: logieval
+dataset_path: baber/logiqa2
+dataset_name: logieval
+output_type: generate_until
+training_split: train
+test_split: test
+# Instructions + {content}
+doc_to_text: "Instructions: You will be presented with a passage and a question about that passage. There are four options to be chosen from, you need to choose the only correct option to answer that question. If the first option is right, you generate the answer 'A', if the second option is right, you generate the answer 'B', if the third option is right, you generate the answer 'C', if the fourth option is right, you generate the answer 'D'. Read the question and options thoroughly and select the correct answer from the four answer labels. Read the passage thoroughly to ensure you know what the passage entails.\n{{content}}"
+doc_to_target: "{{ideal}}"
+metric_list:
+  - metric: exact_match
+    aggregation: mean
+    higher_is_better: true
+generation_kwargs:
+  do_sample: false
+num_fewshot: 1
+filter_list:
+  - name: "get-answer"
+    filter:
+      - function: "regex"
+        # starts with A-D excluding leading spaces
+        # original implementation uses a.startswith(b)
+        # https://github.com/openai/evals/blob/305b237cdb3884c7ddb6a5d12cb184a83551fcba/evals/api.py#L84
+        regex_pattern: "^\\s*([A-D])"
+      - function: "take_first"
+metadata:
+  version: 0.0
+dataset_kwargs:
+  trust_remote_code: true

scripts/yans/lm-evaluation-harness/lm_eval/tasks/logiqa2/logiqa2.yaml

@@ -0,0 +1,21 @@
+task: logiqa2
+dataset_path: baber/logiqa2
+dataset_name: logiqa2
+output_type: multiple_choice
+training_split: train
+validation_split: validation
+test_split: test
+doc_to_choice: "{{options}}"
+doc_to_text: !function utils_logiqa2.doc_to_text
+doc_to_target: "{{answer}}"
+doc_to_decontamination_query: "{{context}}"
+should_decontaminate: false
+metric_list:
+  - metric: acc
+    aggregation: mean
+    higher_is_better: true
+  - metric: acc_norm
+    aggregation: mean
+    higher_is_better: true
+metadata:
+  version: 0.0

scripts/yans/lm-evaluation-harness/lm_eval/tasks/logiqa2/utils_logiqa2.py

@@ -0,0 +1,27 @@
+# Copied from Master
+def doc_to_text(doc) -> str:
+    """
+    Passage: <passage>
+    Question: <question>
+    A. <choice1>
+    B. <choice2>
+    C. <choice3>
+    D. <choice4>
+    Answer:
+    """
+    choices = ["a", "b", "c", "d"]
+    prompt = "Passage: " + doc["text"] + "\n"
+    prompt += "Question: " + doc["question"] + "\n"
+    for choice, option in zip(choices, doc["options"]):
+        prompt += f"{choice.upper()}. {option}\n"
+    prompt += "Answer:"
+    return prompt
+
+
+# # https://github.com/csitfun/LogiQA2.0/blob/main/logiqa2nli/nli-prompt.py
+# def doc_to_textNLI(doc):
+#     maj_premise = ' '.join(list(doc['major_premise']))
+#     min_premise = ' '.join(list(doc['minor_premise']))
+#     hypo = doc['conclusion']
+#     prompt_input = "Given the fact: " + maj_premise + ' ' + min_premise + " Does it follow that: " + hypo + " Yes or no?"
+#     return prompt_input

scripts/yans/lm-evaluation-harness/lm_eval/tasks/openbookqa/README.md

@@ -0,0 +1,54 @@
+# OpenBookQA
+
+### Paper
+
+Title: `Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering`
+
+Abstract: https://arxiv.org/abs/1809.02789
+
+OpenBookQA is a question-answering dataset modeled after open book exams for
+assessing human understanding of a subject. It consists of 5,957 multiple-choice
+elementary-level science questions (4,957 train, 500 dev, 500 test), which probe
+the understanding of a small “book” of 1,326 core science facts and the application
+of these facts to novel situations. For training, the dataset includes a mapping
+from each question to the core science fact it was designed to probe. Answering
+OpenBookQA questions requires additional broad common knowledge, not contained
+in the book. The questions, by design, are answered incorrectly by both a retrieval-
+based algorithm and a word co-occurrence algorithm.
+
+Homepage: https://allenai.org/data/open-book-qa
+
+
+### Citation
+
+```
+@inproceedings{OpenBookQA2018,
+    title={Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering},
+    author={Todor Mihaylov and Peter Clark and Tushar Khot and Ashish Sabharwal},
+    booktitle={EMNLP},
+    year={2018}
+}
+```
+
+### Groups and Tasks
+
+#### Groups
+
+* Not part of a group yet
+
+#### Tasks
+
+* `openbookqa`
+
+### Checklist
+
+For adding novel benchmarks/datasets to the library:
+* [ ] Is the task an existing benchmark in the literature?
+  * [ ] Have you referenced the original paper that introduced the task?
+  * [ ] If yes, does the original paper provide a reference implementation? If so, have you checked against the reference implementation and documented how to run such a test?
+
+
+If other tasks on this dataset are already supported:
+* [ ] Is the "Main" variant of this task clearly denoted?
+* [ ] Have you provided a short sentence in a README on what each new variant adds / evaluates?
+* [ ] Have you noted which, if any, published evaluation setups are matched by this variant?

scripts/yans/lm-evaluation-harness/lm_eval/tasks/openbookqa/openbookqa.yaml

@@ -0,0 +1,21 @@
+task: openbookqa
+dataset_path: openbookqa
+dataset_name: main
+output_type: multiple_choice
+training_split: train
+validation_split: validation
+test_split: test
+doc_to_text: question_stem
+doc_to_target: "{{choices.label.index(answerKey.lstrip())}}"
+doc_to_choice: "{{choices.text}}"
+should_decontaminate: true
+doc_to_decontamination_query: question_stem
+metric_list:
+  - metric: acc
+    aggregation: mean
+    higher_is_better: true
+  - metric: acc_norm
+    aggregation: mean
+    higher_is_better: true
+metadata:
+  version: 1.0

scripts/yans/lm-evaluation-harness/lm_eval/tasks/scrolls/README.md

@@ -0,0 +1,31 @@
+"""
+SCROLLS: Standardized CompaRison Over Long Language Sequences
+https://arxiv.org/abs/2201.03533
+
+SCROLLS is a suite of datasets that require synthesizing information over long texts.
+The benchmark includes seven natural language tasks across multiple domains,
+including summarization, question answering, and natural language inference.
+
+Homepage: https://www.scrolls-benchmark.com/
+
+Since SCROLLS tasks are generally longer than the maximum sequence length of many models,
+it is possible to create "subset" tasks that contain only those samples whose tokenized length
+is less than some pre-defined limit. For example, to create a subset of "Qasper" that would
+be suitable for a model using the GPTNeoX tokenizer and a 4K maximium sequence length:
+
+```
+class QasperGPTNeoX4K(Qasper):
+    PRUNE_TOKENIZERS = ["EleutherAI/pythia-410m-deduped"]
+    PRUNE_MAX_TOKENS = 4096
+    PRUNE_NUM_PROC = _num_cpu_cores() # optional, to speed up pruning of large datasets like NarrativeQA
+```
+
+`PRUNE_TOKENIZERS` can contain more than one tokenizer; this will include only samples that are
+less than `PRUNE_MAX_TOKENS` for ALL of the tokenizers. This can be useful to comparing models
+that use different tokenizers but the same maximum sequence length.
+
+Once the subset task class has been defined in this file, it can be used by adding the class
+to `lm_eval/tasks/__init__.py`.
+
+NOTE: GovReport may need `max_gen_toks` set larger for causal models.
+"""

scripts/yans/lm-evaluation-harness/lm_eval/tasks/scrolls/scrolls_contractnli.yaml

@@ -0,0 +1,3 @@
+group: scrolls
+task: scrolls_contractnli
+class: !function task.ContractNLI

scripts/yans/lm-evaluation-harness/lm_eval/tasks/scrolls/scrolls_govreport.yaml

@@ -0,0 +1,3 @@
+group: scrolls
+task: scrolls_govreport
+class: !function task.GovReport

scripts/yans/lm-evaluation-harness/lm_eval/tasks/scrolls/scrolls_narrativeqa.yaml

@@ -0,0 +1,3 @@
+group: scrolls
+task: scrolls_narrativeqa
+class: !function task.NarrativeQA

scripts/yans/lm-evaluation-harness/lm_eval/tasks/scrolls/scrolls_qasper.yaml

@@ -0,0 +1,3 @@
+group: scrolls
+task: scrolls_qasper
+class: !function task.Qasper