Update README.md

README.md

@@ -65,61 +65,62 @@ Our data encompasses examples of a length up to 16384 tokens, further enhancing
 ## Evaluation
 
 We ran all benchmarks using [lm-eval](https://github.com/EleutherAI/lm-evaluation-harness) with `--apply_chat_template`.
-For comparison, we performed the same benchmarks on the base model as well, in the exact same environment with the same parameters.
+For comparison, we performed the same benchmarks on the base model and Llama-3.1-8B-Instruct as well, in the exact same environment with the same parameters.
 
 ### English Benchmarks
 
-| Benchmark | Mistral-Nemo-Instruct-2407 | educa-ai-nemo-dpo |
-| --- | --- | --- |
-| hellaswag (acc_norm) | 71.9% | **77.6%** |
-| winogrande (acc) | 69.8% | **75.2%** |
-| openbookqa (acc_norm) | 45.8% | **47.0%** |
-| commonsense_qa (acc) | 74.4% | **75.4%** |
-| truthfulqa_mc1 (acc) | 39.66% | **41.5%** |
-| mmlu (acc) | 64.9% | **66.5%** |
-| triviaqa (exact_match) | 12.3% | **23.99%** |
-| agieval (acc) | 36.6% | **39.1%** |
-| arc_challenge (acc_norm) | 52.5% | **54.4%** |
-| arc_easy (acc_norm) | 74.1% | **76.0%** |
-| piqa (acc_norm) | 78.9% | **81.5%** |
-| leaderboard_bbh (acc_norm) | 49.1% | **53.0%** |
-| leaderboard_gpqa (acc_norm) | **30.6%** | 29.4% |
-| leaderboard_ifeval (inst_level_loose_acc) | 72.8% | **75.1%** |
-| leaderboard_mmlu_pro (acc) | **35.1%** | 33.67% |
-| leaderboard_musr (acc_norm) | 39.3% | **40.2%** |
+| Benchmark | Llama-3.1-8B-Instruct | Mistral-Nemo-Instruct-2407 | educa-ai-nemo-dpo |
+| --- | --- | --- | --- |
+| hellaswag (acc_norm) | 72.6% | 71.9% | **77.6%** |
+| winogrande (acc) | 68.0% | 69.8% | **75.2%** |
+| openbookqa (acc_norm) | **49.0%** | 45.8% | 47.0% |
+| commonsense_qa (acc) | 64.9% | 74.4% | **75.4%** |
+| truthfulqa_mc1 (acc) | 40.4% | 39.66% | **41.5%** |
+| mmlu (acc) | 63.2% | 64.9% | **66.5%** |
+| triviaqa (exact_match) | 5.3% | 12.3% | **23.99%** |
+| agieval (acc) | 36.3% | 36.6% | **39.1%** |
+| arc_challenge (acc_norm) | 54.1% | 52.5% | **54.4%** |
+| arc_easy (acc_norm) | 75.7% | 74.1% | **76.0%** |
+| piqa (acc_norm) | 79.6% | 78.9% | **81.5%** |
+| leaderboard_bbh (acc_norm) | 37.4% | 49.1% | **53.0%** |
+| leaderboard_gpqa (acc_norm) | 28.5% | **30.6%** | 29.4% |
+| leaderboard_ifeval (inst_level_loose_acc) | **84.7%** | 72.8% | 75.1% |
+| leaderboard_mmlu_pro (acc) | 16.2% | **35.1%** | 33.67% |
+| leaderboard_musr (acc_norm) | 38.8% | 39.3% | **40.2%** |
 
 ### Multilingual Benchmarks
 
-| Benchmark | Mistral-Nemo-Instruct-2407 | educa-ai-nemo-dpo |
-| --- | --- | --- |
-| global_mmlu_full (acc) | | |
-|  * de | 55.8% | **57.5%** |
-|  * en | 63.1% | **63.8%** |
-|  * es | 58.1% | **58.9%** |
-|  * fr | 56.3% | **58.1%** |
-|  * it | 58.1% | **59.6%** |
-|  * ja | 50.0% | **51.0%** |
-|  * pt | 43.5% | **55.7%** |
-|  * ru | 54.9% | **55.0%** |
-|  * zh | 52.2% | **55.6%** |
-| arc_challenge_mt (acc_norm) | | |
-|  * de | 42.6% | **46.8%** |
-|  * es | 45.6% | **47.3%** |
-|  * it | 44.3% | **46.7%** |
-|  * pt | 42.3% | **46.8%** |
-| xnli (acc) | | |
-|  * de | **47.6%** | 47.1% |
-|  * en | 57.3% | **57.8%** |
-|  * es | 45.0% | **47.0%** |
-|  * fr | 38.5% | **40.0%** |
-|  * ru | **41.8%** | 38.6% |
-|  * zh | **36.3%** | 36.1% |
-| xquad (f1) | | |
-|  * de | 22.7% | **35.6%** |
-|  * en | 21.8% | **29.9%** |
-|  * es | 17.6% | **29.6%** |
-|  * ru | 24.6% | **37.3%** |
-|  * zh | 10.0% | **16.7%** |
+| Benchmark | Llama-3.1-8B-Instruct | Mistral-Nemo-Instruct-2407 | educa-ai-nemo-dpo |
+| --- | --- | --- | --- |
+| global_mmlu_full (acc) | | | |
+|  - de | 48.2% | 55.8% | **57.5%** |
+|  - en | 60.0% | 63.1% | **63.8%** |
+|  - es | 54.7% | 58.1% | **58.9%** |
+|  - fr | 48.3% | 56.3% | **58.1%** |
+|  - it | 51.0% | 58.1% | **59.6%** |
+|  - ja | 47.4% | 50.0% | **51.0%** |
+|  - pt | 23.0% | 43.5% | **55.7%** |
+|  - ru | 41.4% | 54.9% | **55.0%** |
+|  - zh | 49.7% | 52.2% | **55.6%** |
+| arc_challenge_mt (acc_norm) | | | |
+|  - de | 39.9% | 42.6% | **46.8%** |
+|  - es | 42.8% | 45.6% | **47.3%** |
+|  - it | 43.9% | 44.3% | **46.7%** |
+|  - pt | 41.9% | 42.3% | **46.8%** |
+| xnli (acc) | | | |
+|  - de | **48.1%** | 47.6% | 47.1% |
+|  - en | 52.4% | 57.3% | **57.8%** |
+|  - es | 46.3% | 45.0% | **47.0%** |
+|  - fr | **51.6%** | 38.5% | 40.0% |
+|  - ru | **48.1%** | 41.8% | 38.6% |
+|  - zh | **40.3%** | 36.3% | 36.1% |
+| xquad (f1) | | | |
+|  - de | 30.4% | 22.7% | **35.6%** |
+|  - en | **35.0%** | 21.8% | 29.9% |
+|  - es | **31.2%** | 17.6% | 29.6% |
+|  - ru | **39.6%** | 24.6% | 37.3% |
+|  - zh | **28.8%** | 10.0% | 16.7% |
+
 
 ## Model Card Authors [optional]