Adding Evaluation Results

README.md

@@ -1,5 +1,4 @@
 ---
-license: apache-2.0
 language:
 - ar
 - he
@@ -62,8 +61,7 @@ language:
 - et
 - fi
 - hu
-
-pipeline_tag: text-generation
+license: apache-2.0
 tags:
 - multilingual
 - PyTorch
@@ -75,7 +73,111 @@ tags:
 datasets:
 - mc4
 - wikipedia
-thumbnail: "https://github.com/sberbank-ai/mgpt"
+pipeline_tag: text-generation
+thumbnail: https://github.com/sberbank-ai/mgpt
+model-index:
+- name: mGPT
+  results:
+  - task:
+      type: text-generation
+      name: Text Generation
+    dataset:
+      name: AI2 Reasoning Challenge (25-Shot)
+      type: ai2_arc
+      config: ARC-Challenge
+      split: test
+      args:
+        num_few_shot: 25
+    metrics:
+    - type: acc_norm
+      value: 23.81
+      name: normalized accuracy
+    source:
+      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=ai-forever/mGPT
+      name: Open LLM Leaderboard
+  - task:
+      type: text-generation
+      name: Text Generation
+    dataset:
+      name: HellaSwag (10-Shot)
+      type: hellaswag
+      split: validation
+      args:
+        num_few_shot: 10
+    metrics:
+    - type: acc_norm
+      value: 26.37
+      name: normalized accuracy
+    source:
+      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=ai-forever/mGPT
+      name: Open LLM Leaderboard
+  - task:
+      type: text-generation
+      name: Text Generation
+    dataset:
+      name: MMLU (5-Shot)
+      type: cais/mmlu
+      config: all
+      split: test
+      args:
+        num_few_shot: 5
+    metrics:
+    - type: acc
+      value: 25.17
+      name: accuracy
+    source:
+      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=ai-forever/mGPT
+      name: Open LLM Leaderboard
+  - task:
+      type: text-generation
+      name: Text Generation
+    dataset:
+      name: TruthfulQA (0-shot)
+      type: truthful_qa
+      config: multiple_choice
+      split: validation
+      args:
+        num_few_shot: 0
+    metrics:
+    - type: mc2
+      value: 39.62
+    source:
+      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=ai-forever/mGPT
+      name: Open LLM Leaderboard
+  - task:
+      type: text-generation
+      name: Text Generation
+    dataset:
+      name: Winogrande (5-shot)
+      type: winogrande
+      config: winogrande_xl
+      split: validation
+      args:
+        num_few_shot: 5
+    metrics:
+    - type: acc
+      value: 50.67
+      name: accuracy
+    source:
+      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=ai-forever/mGPT
+      name: Open LLM Leaderboard
+  - task:
+      type: text-generation
+      name: Text Generation
+    dataset:
+      name: GSM8k (5-shot)
+      type: gsm8k
+      config: main
+      split: test
+      args:
+        num_few_shot: 5
+    metrics:
+    - type: acc
+      value: 0.0
+      name: accuracy
+    source:
+      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=ai-forever/mGPT
+      name: Open LLM Leaderboard
 ---
 
 # Multilingual GPT model
@@ -141,3 +243,17 @@ Languages:
 The model was trained with sequence length 512 using Megatron and Deepspeed libs by [SberDevices](https://sberdevices.ru/) team on a dataset of 600 GB of texts in 61 languages. The model has seen 440 billion BPE tokens in total.
 
 Total training time was around 14 days on 256 Nvidia V100 GPUs.  
+
+# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
+Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_ai-forever__mGPT)
+
+|             Metric              |Value|
+|---------------------------------|----:|
+|Avg.                             |27.61|
+|AI2 Reasoning Challenge (25-Shot)|23.81|
+|HellaSwag (10-Shot)              |26.37|
+|MMLU (5-Shot)                    |25.17|
+|TruthfulQA (0-shot)              |39.62|
+|Winogrande (5-shot)              |50.67|
+|GSM8k (5-shot)                   | 0.00|
+