Add files using upload-large-folder tool

scripts/yans/lm-evaluation-harness/lm_eval/tasks/afrimgsm/README.md

@@ -0,0 +1,52 @@
+# MathQA
+
+### Paper
+
+IrokoBench: A New Benchmark for African Languages in the Age of Large Language Models
+https://arxiv.org/pdf/2406.03368
+
+IrokoBench is a human-translated benchmark dataset for 16 typologically diverse
+low-resource African languages covering three tasks: natural language inference (AfriXNLI),
+mathematical reasoning (AfriMGSM), and multi-choice knowledge-based QA (AfriMMLU).
+
+
+### Citation
+
+```
+@misc{adelani2024irokobenchnewbenchmarkafrican,
+      title={IrokoBench: A New Benchmark for African Languages in the Age of Large Language Models},
+      author={David Ifeoluwa Adelani and Jessica Ojo and Israel Abebe Azime and Jian Yun Zhuang and Jesujoba O. Alabi and Xuanli He and Millicent Ochieng and Sara Hooker and Andiswa Bukula and En-Shiun Annie Lee and Chiamaka Chukwuneke and Happy Buzaaba and Blessing Sibanda and Godson Kalipe and Jonathan Mukiibi and Salomon Kabongo and Foutse Yuehgoh and Mmasibidi Setaka and Lolwethu Ndolela and Nkiruka Odu and Rooweither Mabuya and Shamsuddeen Hassan Muhammad and Salomey Osei and Sokhar Samb and Tadesse Kebede Guge and Pontus Stenetorp},
+      year={2024},
+      eprint={2406.03368},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL},
+      url={https://arxiv.org/abs/2406.03368},
+}
+```
+
+### Groups and Tasks
+
+#### Groups
+
+* `afrimgsm`: All afrimgsm tasks
+* `afrimgsm_direct`: afrimgsm_direct evaluates models performance on the curated dataset
+* `afrimgsm_en_cot`: afrimgsm_en_cot includes 5-shot of exemplars for chain-of-thought approach
+* `afrimgsm_translate`: afrimgsm_translate evaluates models in translate-test setting
+
+#### Tasks
+* `afrimgsm_direct_{language_code}`: each task evaluates for one language
+* `afrimgsm_en_cot_{language_code}`: each task evaluates for one language
+* `afrimgsm_translate_{language_code}`: each task evaluates for one language
+
+### Checklist
+
+For adding novel benchmarks/datasets to the library:
+* [x] Is the task an existing benchmark in the literature?
+  * [x] Have you referenced the original paper that introduced the task?
+  * [ ] If yes, does the original paper provide a reference implementation? If so, have you checked against the reference implementation and documented how to run such a test?
+
+If other tasks on this dataset are already supported:
+* [x] Is the "Main" variant of this task clearly denoted?
+* [x] Have you provided a short sentence in a README on what each new variant adds / evaluates?
+* [x] Have you noted which, if any, published evaluation setups are matched by this variant?
+  * [x] Checked for equivalence with v0.3.0 LM Evaluation Harness

scripts/yans/lm-evaluation-harness/lm_eval/tasks/afrimgsm/direct/afrimgsm_direct_amh.yaml

@@ -0,0 +1,12 @@
+# Generated by utils.py
+dataset_name: amh
+doc_to_target: '{% if answer is not none %}{{answer[15:]}}{% else %}{{answer_number|string}}{% endif %}'
+doc_to_text: '{% if answer is not none %}{{question+"\nAnswer:"}}{% else %}{{"Question: "+question+"\nAnswer:"}}{% endif %}'
+generation_kwargs:
+  do_sample: false
+  until:
+  - 'Question:'
+  - </s>
+  - <|im_end|>
+include: direct_yaml
+task: afrimgsm_direct_amh

scripts/yans/lm-evaluation-harness/lm_eval/tasks/afrimgsm/direct/afrimgsm_direct_eng.yaml

@@ -0,0 +1,12 @@
+# Generated by utils.py
+dataset_name: eng
+doc_to_target: '{% if answer is not none %}{{answer[21:]}}{% else %}{{answer_number|string}}{% endif %}'
+doc_to_text: '{% if answer is not none %}{{question+"\nAnswer:"}}{% else %}{{"Question: "+question+"\nAnswer:"}}{% endif %}'
+generation_kwargs:
+  do_sample: false
+  until:
+  - 'Question:'
+  - </s>
+  - <|im_end|>
+include: direct_yaml
+task: afrimgsm_direct_eng

scripts/yans/lm-evaluation-harness/lm_eval/tasks/afrimgsm/direct/afrimgsm_direct_ewe.yaml

@@ -0,0 +1,12 @@
+# Generated by utils.py
+dataset_name: ewe
+doc_to_target: '{% if answer is not none %}{{answer[21:]}}{% else %}{{answer_number|string}}{% endif %}'
+doc_to_text: '{% if answer is not none %}{{question+"\nAnswer:"}}{% else %}{{"Question: "+question+"\nAnswer:"}}{% endif %}'
+generation_kwargs:
+  do_sample: false
+  until:
+  - 'Question:'
+  - </s>
+  - <|im_end|>
+include: direct_yaml
+task: afrimgsm_direct_ewe

scripts/yans/lm-evaluation-harness/lm_eval/tasks/afrimgsm/direct/afrimgsm_direct_fra.yaml

@@ -0,0 +1,12 @@
+# Generated by utils.py
+dataset_name: fra
+doc_to_target: '{% if answer is not none %}{{answer[21:]}}{% else %}{{answer_number|string}}{% endif %}'
+doc_to_text: '{% if answer is not none %}{{question+"\nAnswer:"}}{% else %}{{"Question: "+question+"\nAnswer:"}}{% endif %}'
+generation_kwargs:
+  do_sample: false
+  until:
+  - 'Question:'
+  - </s>
+  - <|im_end|>
+include: direct_yaml
+task: afrimgsm_direct_fra

scripts/yans/lm-evaluation-harness/lm_eval/tasks/afrimgsm/direct/afrimgsm_direct_hau.yaml

@@ -0,0 +1,12 @@
+# Generated by utils.py
+dataset_name: hau
+doc_to_target: '{% if answer is not none %}{{answer[21:]}}{% else %}{{answer_number|string}}{% endif %}'
+doc_to_text: '{% if answer is not none %}{{question+"\nAnswer:"}}{% else %}{{"Question: "+question+"\nAnswer:"}}{% endif %}'
+generation_kwargs:
+  do_sample: false
+  until:
+  - 'Question:'
+  - </s>
+  - <|im_end|>
+include: direct_yaml
+task: afrimgsm_direct_hau

scripts/yans/lm-evaluation-harness/lm_eval/tasks/afrimgsm/direct/afrimgsm_direct_ibo.yaml

@@ -0,0 +1,12 @@
+# Generated by utils.py
+dataset_name: ibo
+doc_to_target: '{% if answer is not none %}{{answer[21:]}}{% else %}{{answer_number|string}}{% endif %}'
+doc_to_text: '{% if answer is not none %}{{question+"\nAnswer:"}}{% else %}{{"Question: "+question+"\nAnswer:"}}{% endif %}'
+generation_kwargs:
+  do_sample: false
+  until:
+  - 'Question:'
+  - </s>
+  - <|im_end|>
+include: direct_yaml
+task: afrimgsm_direct_ibo

scripts/yans/lm-evaluation-harness/lm_eval/tasks/afrimgsm/direct/afrimgsm_direct_kin.yaml

@@ -0,0 +1,12 @@
+# Generated by utils.py
+dataset_name: kin
+doc_to_target: '{% if answer is not none %}{{answer[21:]}}{% else %}{{answer_number|string}}{% endif %}'
+doc_to_text: '{% if answer is not none %}{{question+"\nAnswer:"}}{% else %}{{"Question: "+question+"\nAnswer:"}}{% endif %}'
+generation_kwargs:
+  do_sample: false
+  until:
+  - 'Question:'
+  - </s>
+  - <|im_end|>
+include: direct_yaml
+task: afrimgsm_direct_kin

scripts/yans/lm-evaluation-harness/lm_eval/tasks/afrimgsm/direct/afrimgsm_direct_lin.yaml

@@ -0,0 +1,12 @@
+# Generated by utils.py
+dataset_name: lin
+doc_to_target: '{% if answer is not none %}{{answer[21:]}}{% else %}{{answer_number|string}}{% endif %}'
+doc_to_text: '{% if answer is not none %}{{question+"\nAnswer:"}}{% else %}{{"Question: "+question+"\nAnswer:"}}{% endif %}'
+generation_kwargs:
+  do_sample: false
+  until:
+  - 'Question:'
+  - </s>
+  - <|im_end|>
+include: direct_yaml
+task: afrimgsm_direct_lin

scripts/yans/lm-evaluation-harness/lm_eval/tasks/afrimgsm/direct/afrimgsm_direct_lug.yaml

@@ -0,0 +1,12 @@
+# Generated by utils.py
+dataset_name: lug
+doc_to_target: '{% if answer is not none %}{{answer[21:]}}{% else %}{{answer_number|string}}{% endif %}'
+doc_to_text: '{% if answer is not none %}{{question+"\nAnswer:"}}{% else %}{{"Question: "+question+"\nAnswer:"}}{% endif %}'
+generation_kwargs:
+  do_sample: false
+  until:
+  - 'Question:'
+  - </s>
+  - <|im_end|>
+include: direct_yaml
+task: afrimgsm_direct_lug

scripts/yans/lm-evaluation-harness/lm_eval/tasks/afrimgsm/direct/afrimgsm_direct_orm.yaml

@@ -0,0 +1,12 @@
+# Generated by utils.py
+dataset_name: orm
+doc_to_target: '{% if answer is not none %}{{answer[21:]}}{% else %}{{answer_number|string}}{% endif %}'
+doc_to_text: '{% if answer is not none %}{{question+"\nAnswer:"}}{% else %}{{"Question: "+question+"\nAnswer:"}}{% endif %}'
+generation_kwargs:
+  do_sample: false
+  until:
+  - 'Question:'
+  - </s>
+  - <|im_end|>
+include: direct_yaml
+task: afrimgsm_direct_orm

scripts/yans/lm-evaluation-harness/lm_eval/tasks/afrimgsm/direct/afrimgsm_direct_sna.yaml

@@ -0,0 +1,12 @@
+# Generated by utils.py
+dataset_name: sna
+doc_to_target: '{% if answer is not none %}{{answer[21:]}}{% else %}{{answer_number|string}}{% endif %}'
+doc_to_text: '{% if answer is not none %}{{question+"\nAnswer:"}}{% else %}{{"Question: "+question+"\nAnswer:"}}{% endif %}'
+generation_kwargs:
+  do_sample: false
+  until:
+  - 'Question:'
+  - </s>
+  - <|im_end|>
+include: direct_yaml
+task: afrimgsm_direct_sna

scripts/yans/lm-evaluation-harness/lm_eval/tasks/afrimgsm/direct/afrimgsm_direct_sot.yaml

@@ -0,0 +1,12 @@
+# Generated by utils.py
+dataset_name: sot
+doc_to_target: '{% if answer is not none %}{{answer[21:]}}{% else %}{{answer_number|string}}{% endif %}'
+doc_to_text: '{% if answer is not none %}{{question+"\nAnswer:"}}{% else %}{{"Question: "+question+"\nAnswer:"}}{% endif %}'
+generation_kwargs:
+  do_sample: false
+  until:
+  - 'Question:'
+  - </s>
+  - <|im_end|>
+include: direct_yaml
+task: afrimgsm_direct_sot

scripts/yans/lm-evaluation-harness/lm_eval/tasks/afrimgsm/direct/afrimgsm_direct_swa.yaml

@@ -0,0 +1,12 @@
+# Generated by utils.py
+dataset_name: swa
+doc_to_target: '{% if answer is not none %}{{answer[21:]}}{% else %}{{answer_number|string}}{% endif %}'
+doc_to_text: '{% if answer is not none %}{{question+"\nAnswer:"}}{% else %}{{"Question: "+question+"\nAnswer:"}}{% endif %}'
+generation_kwargs:
+  do_sample: false
+  until:
+  - 'Question:'
+  - </s>
+  - <|im_end|>
+include: direct_yaml
+task: afrimgsm_direct_swa

scripts/yans/lm-evaluation-harness/lm_eval/tasks/afrimgsm/direct/afrimgsm_direct_twi.yaml

@@ -0,0 +1,12 @@
+# Generated by utils.py
+dataset_name: twi
+doc_to_target: '{% if answer is not none %}{{answer[21:]}}{% else %}{{answer_number|string}}{% endif %}'
+doc_to_text: '{% if answer is not none %}{{question+"\nAnswer:"}}{% else %}{{"Question: "+question+"\nAnswer:"}}{% endif %}'
+generation_kwargs:
+  do_sample: false
+  until:
+  - 'Question:'
+  - </s>
+  - <|im_end|>
+include: direct_yaml
+task: afrimgsm_direct_twi

scripts/yans/lm-evaluation-harness/lm_eval/tasks/afrimgsm/direct/afrimgsm_direct_wol.yaml

@@ -0,0 +1,12 @@
+# Generated by utils.py
+dataset_name: wol
+doc_to_target: '{% if answer is not none %}{{answer[21:]}}{% else %}{{answer_number|string}}{% endif %}'
+doc_to_text: '{% if answer is not none %}{{question+"\nAnswer:"}}{% else %}{{"Question: "+question+"\nAnswer:"}}{% endif %}'
+generation_kwargs:
+  do_sample: false
+  until:
+  - 'Question:'
+  - </s>
+  - <|im_end|>
+include: direct_yaml
+task: afrimgsm_direct_wol

scripts/yans/lm-evaluation-harness/lm_eval/tasks/afrimgsm/direct/afrimgsm_direct_xho.yaml

@@ -0,0 +1,12 @@
+# Generated by utils.py
+dataset_name: xho
+doc_to_target: '{% if answer is not none %}{{answer[21:]}}{% else %}{{answer_number|string}}{% endif %}'
+doc_to_text: '{% if answer is not none %}{{question+"\nAnswer:"}}{% else %}{{"Question: "+question+"\nAnswer:"}}{% endif %}'
+generation_kwargs:
+  do_sample: false
+  until:
+  - 'Question:'
+  - </s>
+  - <|im_end|>
+include: direct_yaml
+task: afrimgsm_direct_xho

scripts/yans/lm-evaluation-harness/lm_eval/tasks/afrimgsm/direct/afrimgsm_direct_yor.yaml

@@ -0,0 +1,12 @@
+# Generated by utils.py
+dataset_name: yor
+doc_to_target: '{% if answer is not none %}{{answer[16:]}}{% else %}{{answer_number|string}}{% endif %}'
+doc_to_text: '{% if answer is not none %}{{question+"\nAnswer:"}}{% else %}{{"Question: "+question+"\nAnswer:"}}{% endif %}'
+generation_kwargs:
+  do_sample: false
+  until:
+  - 'Question:'
+  - </s>
+  - <|im_end|>
+include: direct_yaml
+task: afrimgsm_direct_yor

scripts/yans/lm-evaluation-harness/lm_eval/tasks/afrimgsm/direct/afrimgsm_direct_zul.yaml

@@ -0,0 +1,12 @@
+# Generated by utils.py
+dataset_name: zul
+doc_to_target: '{% if answer is not none %}{{answer[21:]}}{% else %}{{answer_number|string}}{% endif %}'
+doc_to_text: '{% if answer is not none %}{{question+"\nAnswer:"}}{% else %}{{"Question: "+question+"\nAnswer:"}}{% endif %}'
+generation_kwargs:
+  do_sample: false
+  until:
+  - 'Question:'
+  - </s>
+  - <|im_end|>
+include: direct_yaml
+task: afrimgsm_direct_zul

scripts/yans/lm-evaluation-harness/lm_eval/tasks/afrimgsm/direct/direct_yaml

@@ -0,0 +1,37 @@
+# This file will be included in the generated language-specific task configs.
+# It doesn't have a yaml file extension as it is not meant to be imported directly
+# by the harness.
+group:
+    - afrimgsm
+    - afrimgsm_direct
+dataset_path: masakhane/afrimgsm
+dataset_name: null  # Overridden by language-specific config.
+output_type: generate_until
+# training_split: train
+test_split: test
+target_delimiter: ""
+generation_kwargs:
+  until:
+    - "\n\n"
+    - "\n"
+  do_sample: false
+  temperature: 0.0
+filter_list:
+  - name: remove_whitespace
+    filter:
+      - function: remove_whitespace
+      - function: take_first
+  - filter:
+    - function: regex
+      group_select: -1
+      regex_pattern: (-?[$0-9.,]{2,})|(-?[0-9]+)
+    - function: take_first
+    name: flexible-extract
+metric_list:
+  - metric: exact_match
+    aggregation: mean
+    higher_is_better: true
+    ignore_case: true
+    ignore_punctuation: true
+metadata:
+  version: 2.0

scripts/yans/lm-evaluation-harness/lm_eval/tasks/afrimgsm/en_cot/afrimgsm_en_cot_amh.yaml

@@ -0,0 +1,12 @@
+# Generated by utils.py
+dataset_name: amh
+doc_to_target: '{% if answer is not none %}{{answer[15:]}}{% else %}{{answer_number|string}}{% endif %}'
+doc_to_text: '{% if answer is not none %}{{question+"\nStep-by-Step Answer:"}}{% else %}{{"Question: "+question+"\nStep-by-Step Answer:"}}{% endif %}'
+generation_kwargs:
+  do_sample: false
+  until:
+  - 'Question:'
+  - </s>
+  - <|im_end|>
+include: cot_yaml
+task: afrimgsm_en_cot_amh

scripts/yans/lm-evaluation-harness/lm_eval/tasks/afrimgsm/en_cot/afrimgsm_en_cot_ewe.yaml

@@ -0,0 +1,12 @@
+# Generated by utils.py
+dataset_name: ewe
+doc_to_target: '{% if answer is not none %}{{answer[21:]}}{% else %}{{answer_number|string}}{% endif %}'
+doc_to_text: '{% if answer is not none %}{{question+"\nStep-by-Step Answer:"}}{% else %}{{"Question: "+question+"\nStep-by-Step Answer:"}}{% endif %}'
+generation_kwargs:
+  do_sample: false
+  until:
+  - 'Question:'
+  - </s>
+  - <|im_end|>
+include: cot_yaml
+task: afrimgsm_en_cot_ewe

scripts/yans/lm-evaluation-harness/lm_eval/tasks/afrimgsm/en_cot/afrimgsm_en_cot_fra.yaml

@@ -0,0 +1,12 @@
+# Generated by utils.py
+dataset_name: fra
+doc_to_target: '{% if answer is not none %}{{answer[21:]}}{% else %}{{answer_number|string}}{% endif %}'
+doc_to_text: '{% if answer is not none %}{{question+"\nStep-by-Step Answer:"}}{% else %}{{"Question: "+question+"\nStep-by-Step Answer:"}}{% endif %}'
+generation_kwargs:
+  do_sample: false
+  until:
+  - 'Question:'
+  - </s>
+  - <|im_end|>
+include: cot_yaml
+task: afrimgsm_en_cot_fra

scripts/yans/lm-evaluation-harness/lm_eval/tasks/afrimgsm/en_cot/afrimgsm_en_cot_kin.yaml

@@ -0,0 +1,12 @@
+# Generated by utils.py
+dataset_name: kin
+doc_to_target: '{% if answer is not none %}{{answer[21:]}}{% else %}{{answer_number|string}}{% endif %}'
+doc_to_text: '{% if answer is not none %}{{question+"\nStep-by-Step Answer:"}}{% else %}{{"Question: "+question+"\nStep-by-Step Answer:"}}{% endif %}'
+generation_kwargs:
+  do_sample: false
+  until:
+  - 'Question:'
+  - </s>
+  - <|im_end|>
+include: cot_yaml
+task: afrimgsm_en_cot_kin

scripts/yans/lm-evaluation-harness/lm_eval/tasks/afrimgsm/en_cot/afrimgsm_en_cot_lin.yaml

@@ -0,0 +1,12 @@
+# Generated by utils.py
+dataset_name: lin
+doc_to_target: '{% if answer is not none %}{{answer[21:]}}{% else %}{{answer_number|string}}{% endif %}'
+doc_to_text: '{% if answer is not none %}{{question+"\nStep-by-Step Answer:"}}{% else %}{{"Question: "+question+"\nStep-by-Step Answer:"}}{% endif %}'
+generation_kwargs:
+  do_sample: false
+  until:
+  - 'Question:'
+  - </s>
+  - <|im_end|>
+include: cot_yaml
+task: afrimgsm_en_cot_lin

scripts/yans/lm-evaluation-harness/lm_eval/tasks/afrimgsm/en_cot/afrimgsm_en_cot_orm.yaml

@@ -0,0 +1,12 @@
+# Generated by utils.py
+dataset_name: orm
+doc_to_target: '{% if answer is not none %}{{answer[21:]}}{% else %}{{answer_number|string}}{% endif %}'
+doc_to_text: '{% if answer is not none %}{{question+"\nStep-by-Step Answer:"}}{% else %}{{"Question: "+question+"\nStep-by-Step Answer:"}}{% endif %}'
+generation_kwargs:
+  do_sample: false
+  until:
+  - 'Question:'
+  - </s>
+  - <|im_end|>
+include: cot_yaml
+task: afrimgsm_en_cot_orm

scripts/yans/lm-evaluation-harness/lm_eval/tasks/afrimgsm/en_cot/afrimgsm_en_cot_sna.yaml

@@ -0,0 +1,12 @@
+# Generated by utils.py
+dataset_name: sna
+doc_to_target: '{% if answer is not none %}{{answer[21:]}}{% else %}{{answer_number|string}}{% endif %}'
+doc_to_text: '{% if answer is not none %}{{question+"\nStep-by-Step Answer:"}}{% else %}{{"Question: "+question+"\nStep-by-Step Answer:"}}{% endif %}'
+generation_kwargs:
+  do_sample: false
+  until:
+  - 'Question:'
+  - </s>
+  - <|im_end|>
+include: cot_yaml
+task: afrimgsm_en_cot_sna

scripts/yans/lm-evaluation-harness/lm_eval/tasks/afrimgsm/en_cot/afrimgsm_en_cot_wol.yaml

@@ -0,0 +1,12 @@
+# Generated by utils.py
+dataset_name: wol
+doc_to_target: '{% if answer is not none %}{{answer[21:]}}{% else %}{{answer_number|string}}{% endif %}'
+doc_to_text: '{% if answer is not none %}{{question+"\nStep-by-Step Answer:"}}{% else %}{{"Question: "+question+"\nStep-by-Step Answer:"}}{% endif %}'
+generation_kwargs:
+  do_sample: false
+  until:
+  - 'Question:'
+  - </s>
+  - <|im_end|>
+include: cot_yaml
+task: afrimgsm_en_cot_wol

scripts/yans/lm-evaluation-harness/lm_eval/tasks/afrimgsm/en_cot/afrimgsm_en_cot_xho.yaml

@@ -0,0 +1,12 @@
+# Generated by utils.py
+dataset_name: xho
+doc_to_target: '{% if answer is not none %}{{answer[21:]}}{% else %}{{answer_number|string}}{% endif %}'
+doc_to_text: '{% if answer is not none %}{{question+"\nStep-by-Step Answer:"}}{% else %}{{"Question: "+question+"\nStep-by-Step Answer:"}}{% endif %}'
+generation_kwargs:
+  do_sample: false
+  until:
+  - 'Question:'
+  - </s>
+  - <|im_end|>
+include: cot_yaml
+task: afrimgsm_en_cot_xho

scripts/yans/lm-evaluation-harness/lm_eval/tasks/afrimgsm/en_cot/afrimgsm_en_cot_yor.yaml

@@ -0,0 +1,12 @@
+# Generated by utils.py
+dataset_name: yor
+doc_to_target: '{% if answer is not none %}{{answer[16:]}}{% else %}{{answer_number|string}}{% endif %}'
+doc_to_text: '{% if answer is not none %}{{question+"\nStep-by-Step Answer:"}}{% else %}{{"Question: "+question+"\nStep-by-Step Answer:"}}{% endif %}'
+generation_kwargs:
+  do_sample: false
+  until:
+  - 'Question:'
+  - </s>
+  - <|im_end|>
+include: cot_yaml
+task: afrimgsm_en_cot_yor

scripts/yans/lm-evaluation-harness/lm_eval/tasks/afrimgsm/en_cot/afrimgsm_en_cot_zul.yaml

@@ -0,0 +1,12 @@
+# Generated by utils.py
+dataset_name: zul
+doc_to_target: '{% if answer is not none %}{{answer[21:]}}{% else %}{{answer_number|string}}{% endif %}'
+doc_to_text: '{% if answer is not none %}{{question+"\nStep-by-Step Answer:"}}{% else %}{{"Question: "+question+"\nStep-by-Step Answer:"}}{% endif %}'
+generation_kwargs:
+  do_sample: false
+  until:
+  - 'Question:'
+  - </s>
+  - <|im_end|>
+include: cot_yaml
+task: afrimgsm_en_cot_zul

scripts/yans/lm-evaluation-harness/lm_eval/tasks/afrimgsm/gen_yaml.sh

@@ -0,0 +1,7 @@
+#!/bin/bash
+
+# python utils.py --overwrite --output-dir direct --mode direct
+# python utils.py --overwrite --output-dir direct_native --mode direct-native
+# python utils.py --overwrite --output-dir en_cot --mode en-cot
+# python utils.py --overwrite --output-dir native_cot --mode native-cot
+python utils.py --overwrite --output-dir translate_direct --mode translate-direct

scripts/yans/lm-evaluation-harness/lm_eval/tasks/afrimgsm/run.sh

@@ -0,0 +1,6 @@
+lm_eval --model hf   \
+        --model_args pretrained="google/gemma-7b"  --tasks afrimgsm_en_cot_eng,mgsm_en_cot_en,afrimgsm_native_cot_eng,mgsm_native_cot_en,afrimgsm_direct_eng,mgsm_direct_en,afrimgsm_direct_native_eng  \
+        --device cuda:0     \
+        --batch_size 1  \
+        --verbosity DEBUG \
+        --limit 5

scripts/yans/lm-evaluation-harness/lm_eval/tasks/afrimgsm/translate/afrimgsm_translate_amh.yaml

@@ -0,0 +1,12 @@
+# Generated by utils.py
+dataset_name: amh
+doc_to_target: '{% if answer is not none %}{{answer[15:]}}{% else %}{{answer_number|string}}{% endif %}'
+doc_to_text: '{% if answer is not none %}{{question+"\nAnswer:"}}{% else %}{{"Question: "+question+"\nAnswer:"}}{% endif %}'
+generation_kwargs:
+  do_sample: false
+  until:
+  - 'Question:'
+  - </s>
+  - <|im_end|>
+include: translate_direct_yaml
+task: afrimgsm_translate_direct_amh

scripts/yans/lm-evaluation-harness/lm_eval/tasks/afrimgsm/translate/afrimgsm_translate_eng.yaml

@@ -0,0 +1,12 @@
+# Generated by utils.py
+dataset_name: eng
+doc_to_target: '{% if answer is not none %}{{answer[21:]}}{% else %}{{answer_number|string}}{% endif %}'
+doc_to_text: '{% if answer is not none %}{{question+"\nAnswer:"}}{% else %}{{"Question: "+question+"\nAnswer:"}}{% endif %}'
+generation_kwargs:
+  do_sample: false
+  until:
+  - 'Question:'
+  - </s>
+  - <|im_end|>
+include: translate_direct_yaml
+task: afrimgsm_translate_direct_eng

scripts/yans/lm-evaluation-harness/lm_eval/tasks/afrimgsm/translate/afrimgsm_translate_ewe.yaml

@@ -0,0 +1,12 @@
+# Generated by utils.py
+dataset_name: ewe
+doc_to_target: '{% if answer is not none %}{{answer[21:]}}{% else %}{{answer_number|string}}{% endif %}'
+doc_to_text: '{% if answer is not none %}{{question+"\nAnswer:"}}{% else %}{{"Question: "+question+"\nAnswer:"}}{% endif %}'
+generation_kwargs:
+  do_sample: false
+  until:
+  - 'Question:'
+  - </s>
+  - <|im_end|>
+include: translate_direct_yaml
+task: afrimgsm_translate_direct_ewe

scripts/yans/lm-evaluation-harness/lm_eval/tasks/afrimgsm/translate/afrimgsm_translate_fra.yaml

@@ -0,0 +1,12 @@
+# Generated by utils.py
+dataset_name: fra
+doc_to_target: '{% if answer is not none %}{{answer[21:]}}{% else %}{{answer_number|string}}{% endif %}'
+doc_to_text: '{% if answer is not none %}{{question+"\nAnswer:"}}{% else %}{{"Question: "+question+"\nAnswer:"}}{% endif %}'
+generation_kwargs:
+  do_sample: false
+  until:
+  - 'Question:'
+  - </s>
+  - <|im_end|>
+include: translate_direct_yaml
+task: afrimgsm_translate_direct_fra

scripts/yans/lm-evaluation-harness/lm_eval/tasks/afrimgsm/translate/afrimgsm_translate_hau.yaml

@@ -0,0 +1,12 @@
+# Generated by utils.py
+dataset_name: hau
+doc_to_target: '{% if answer is not none %}{{answer[21:]}}{% else %}{{answer_number|string}}{% endif %}'
+doc_to_text: '{% if answer is not none %}{{question+"\nAnswer:"}}{% else %}{{"Question: "+question+"\nAnswer:"}}{% endif %}'
+generation_kwargs:
+  do_sample: false
+  until:
+  - 'Question:'
+  - </s>
+  - <|im_end|>
+include: translate_direct_yaml
+task: afrimgsm_translate_direct_hau

scripts/yans/lm-evaluation-harness/lm_eval/tasks/afrimgsm/translate/afrimgsm_translate_ibo.yaml

@@ -0,0 +1,12 @@
+# Generated by utils.py
+dataset_name: ibo
+doc_to_target: '{% if answer is not none %}{{answer[21:]}}{% else %}{{answer_number|string}}{% endif %}'
+doc_to_text: '{% if answer is not none %}{{question+"\nAnswer:"}}{% else %}{{"Question: "+question+"\nAnswer:"}}{% endif %}'
+generation_kwargs:
+  do_sample: false
+  until:
+  - 'Question:'
+  - </s>
+  - <|im_end|>
+include: translate_direct_yaml
+task: afrimgsm_translate_direct_ibo

scripts/yans/lm-evaluation-harness/lm_eval/tasks/afrimgsm/translate/afrimgsm_translate_kin.yaml

@@ -0,0 +1,12 @@
+# Generated by utils.py
+dataset_name: kin
+doc_to_target: '{% if answer is not none %}{{answer[21:]}}{% else %}{{answer_number|string}}{% endif %}'
+doc_to_text: '{% if answer is not none %}{{question+"\nAnswer:"}}{% else %}{{"Question: "+question+"\nAnswer:"}}{% endif %}'
+generation_kwargs:
+  do_sample: false
+  until:
+  - 'Question:'
+  - </s>
+  - <|im_end|>
+include: translate_direct_yaml
+task: afrimgsm_translate_direct_kin

scripts/yans/lm-evaluation-harness/lm_eval/tasks/afrimgsm/translate/afrimgsm_translate_lin.yaml

@@ -0,0 +1,12 @@
+# Generated by utils.py
+dataset_name: lin
+doc_to_target: '{% if answer is not none %}{{answer[21:]}}{% else %}{{answer_number|string}}{% endif %}'
+doc_to_text: '{% if answer is not none %}{{question+"\nAnswer:"}}{% else %}{{"Question: "+question+"\nAnswer:"}}{% endif %}'
+generation_kwargs:
+  do_sample: false
+  until:
+  - 'Question:'
+  - </s>
+  - <|im_end|>
+include: translate_direct_yaml
+task: afrimgsm_translate_direct_lin

scripts/yans/lm-evaluation-harness/lm_eval/tasks/afrimgsm/translate/afrimgsm_translate_lug.yaml

@@ -0,0 +1,12 @@
+# Generated by utils.py
+dataset_name: lug
+doc_to_target: '{% if answer is not none %}{{answer[21:]}}{% else %}{{answer_number|string}}{% endif %}'
+doc_to_text: '{% if answer is not none %}{{question+"\nAnswer:"}}{% else %}{{"Question: "+question+"\nAnswer:"}}{% endif %}'
+generation_kwargs:
+  do_sample: false
+  until:
+  - 'Question:'
+  - </s>
+  - <|im_end|>
+include: translate_direct_yaml
+task: afrimgsm_translate_direct_lug

scripts/yans/lm-evaluation-harness/lm_eval/tasks/afrimgsm/translate/afrimgsm_translate_orm.yaml

@@ -0,0 +1,12 @@
+# Generated by utils.py
+dataset_name: orm
+doc_to_target: '{% if answer is not none %}{{answer[21:]}}{% else %}{{answer_number|string}}{% endif %}'
+doc_to_text: '{% if answer is not none %}{{question+"\nAnswer:"}}{% else %}{{"Question: "+question+"\nAnswer:"}}{% endif %}'
+generation_kwargs:
+  do_sample: false
+  until:
+  - 'Question:'
+  - </s>
+  - <|im_end|>
+include: translate_direct_yaml
+task: afrimgsm_translate_direct_orm

scripts/yans/lm-evaluation-harness/lm_eval/tasks/afrimgsm/translate/afrimgsm_translate_sna.yaml

@@ -0,0 +1,12 @@
+# Generated by utils.py
+dataset_name: sna
+doc_to_target: '{% if answer is not none %}{{answer[21:]}}{% else %}{{answer_number|string}}{% endif %}'
+doc_to_text: '{% if answer is not none %}{{question+"\nAnswer:"}}{% else %}{{"Question: "+question+"\nAnswer:"}}{% endif %}'
+generation_kwargs:
+  do_sample: false
+  until:
+  - 'Question:'
+  - </s>
+  - <|im_end|>
+include: translate_direct_yaml
+task: afrimgsm_translate_direct_sna

scripts/yans/lm-evaluation-harness/lm_eval/tasks/afrimgsm/translate/afrimgsm_translate_sot.yaml

@@ -0,0 +1,12 @@
+# Generated by utils.py
+dataset_name: sot
+doc_to_target: '{% if answer is not none %}{{answer[21:]}}{% else %}{{answer_number|string}}{% endif %}'
+doc_to_text: '{% if answer is not none %}{{question+"\nAnswer:"}}{% else %}{{"Question: "+question+"\nAnswer:"}}{% endif %}'
+generation_kwargs:
+  do_sample: false
+  until:
+  - 'Question:'
+  - </s>
+  - <|im_end|>
+include: translate_direct_yaml
+task: afrimgsm_translate_direct_sot

scripts/yans/lm-evaluation-harness/lm_eval/tasks/afrimgsm/translate/afrimgsm_translate_swa.yaml

@@ -0,0 +1,12 @@
+# Generated by utils.py
+dataset_name: swa
+doc_to_target: '{% if answer is not none %}{{answer[21:]}}{% else %}{{answer_number|string}}{% endif %}'
+doc_to_text: '{% if answer is not none %}{{question+"\nAnswer:"}}{% else %}{{"Question: "+question+"\nAnswer:"}}{% endif %}'
+generation_kwargs:
+  do_sample: false
+  until:
+  - 'Question:'
+  - </s>
+  - <|im_end|>
+include: translate_direct_yaml
+task: afrimgsm_translate_direct_swa

scripts/yans/lm-evaluation-harness/lm_eval/tasks/afrimgsm/translate/afrimgsm_translate_twi.yaml

@@ -0,0 +1,12 @@
+# Generated by utils.py
+dataset_name: twi
+doc_to_target: '{% if answer is not none %}{{answer[21:]}}{% else %}{{answer_number|string}}{% endif %}'
+doc_to_text: '{% if answer is not none %}{{question+"\nAnswer:"}}{% else %}{{"Question: "+question+"\nAnswer:"}}{% endif %}'
+generation_kwargs:
+  do_sample: false
+  until:
+  - 'Question:'
+  - </s>
+  - <|im_end|>
+include: translate_direct_yaml
+task: afrimgsm_translate_direct_twi

scripts/yans/lm-evaluation-harness/lm_eval/tasks/afrimgsm/translate/afrimgsm_translate_wol.yaml

@@ -0,0 +1,12 @@
+# Generated by utils.py
+dataset_name: wol
+doc_to_target: '{% if answer is not none %}{{answer[21:]}}{% else %}{{answer_number|string}}{% endif %}'
+doc_to_text: '{% if answer is not none %}{{question+"\nAnswer:"}}{% else %}{{"Question: "+question+"\nAnswer:"}}{% endif %}'
+generation_kwargs:
+  do_sample: false
+  until:
+  - 'Question:'
+  - </s>
+  - <|im_end|>
+include: translate_direct_yaml
+task: afrimgsm_translate_direct_wol

scripts/yans/lm-evaluation-harness/lm_eval/tasks/afrimgsm/translate/afrimgsm_translate_xho.yaml

@@ -0,0 +1,12 @@
+# Generated by utils.py
+dataset_name: xho
+doc_to_target: '{% if answer is not none %}{{answer[21:]}}{% else %}{{answer_number|string}}{% endif %}'
+doc_to_text: '{% if answer is not none %}{{question+"\nAnswer:"}}{% else %}{{"Question: "+question+"\nAnswer:"}}{% endif %}'
+generation_kwargs:
+  do_sample: false
+  until:
+  - 'Question:'
+  - </s>
+  - <|im_end|>
+include: translate_direct_yaml
+task: afrimgsm_translate_direct_xho

scripts/yans/lm-evaluation-harness/lm_eval/tasks/afrimgsm/translate/afrimgsm_translate_yor.yaml

@@ -0,0 +1,12 @@
+# Generated by utils.py
+dataset_name: yor
+doc_to_target: '{% if answer is not none %}{{answer[16:]}}{% else %}{{answer_number|string}}{% endif %}'
+doc_to_text: '{% if answer is not none %}{{question+"\nAnswer:"}}{% else %}{{"Question: "+question+"\nAnswer:"}}{% endif %}'
+generation_kwargs:
+  do_sample: false
+  until:
+  - 'Question:'
+  - </s>
+  - <|im_end|>
+include: translate_direct_yaml
+task: afrimgsm_translate_direct_yor