Adding the Open Portuguese LLM Leaderboard Evaluation Results
#4
by
leaderboard-pt-pr-bot
- opened
README.md
CHANGED
|
@@ -1,5 +1,8 @@
|
|
| 1 |
---
|
| 2 |
-
|
|
|
|
|
|
|
|
|
|
| 3 |
tags:
|
| 4 |
- mistral
|
| 5 |
- instruct
|
|
@@ -8,13 +11,10 @@ tags:
|
|
| 8 |
- gpt4
|
| 9 |
- synthetic data
|
| 10 |
- distillation
|
|
|
|
| 11 |
model-index:
|
| 12 |
- name: OpenHermes-2-Mistral-7B-PTBR
|
| 13 |
results: []
|
| 14 |
-
license: apache-2.0
|
| 15 |
-
language:
|
| 16 |
-
- en
|
| 17 |
-
- pt
|
| 18 |
---
|
| 19 |
|
| 20 |
# OpenHermes 2 - Mistral 7B - PT BR
|
|
@@ -103,4 +103,23 @@ To utilize the prompt format without a system prompt, simply leave the line out.
|
|
| 103 |
Currently, I recommend using LM Studio for chatting with Hermes 2. It is a GUI application that utilizes GGUF models with a llama.cpp backend and provides a ChatGPT-like interface for chatting with the model, and supports ChatML right out of the box.
|
| 104 |
In LM-Studio, simply select the ChatML Prefix on the settings side pane:
|
| 105 |
|
| 106 |
-

|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
+
language:
|
| 3 |
+
- en
|
| 4 |
+
- pt
|
| 5 |
+
license: apache-2.0
|
| 6 |
tags:
|
| 7 |
- mistral
|
| 8 |
- instruct
|
|
|
|
| 11 |
- gpt4
|
| 12 |
- synthetic data
|
| 13 |
- distillation
|
| 14 |
+
base_model: mistralai/Mistral-7B-v0.1
|
| 15 |
model-index:
|
| 16 |
- name: OpenHermes-2-Mistral-7B-PTBR
|
| 17 |
results: []
|
|
|
|
|
|
|
|
|
|
|
|
|
| 18 |
---
|
| 19 |
|
| 20 |
# OpenHermes 2 - Mistral 7B - PT BR
|
|
|
|
| 103 |
Currently, I recommend using LM Studio for chatting with Hermes 2. It is a GUI application that utilizes GGUF models with a llama.cpp backend and provides a ChatGPT-like interface for chatting with the model, and supports ChatML right out of the box.
|
| 104 |
In LM-Studio, simply select the ChatML Prefix on the settings side pane:
|
| 105 |
|
| 106 |
+

|
| 107 |
+
|
| 108 |
+
|
| 109 |
+
# Open Portuguese LLM Leaderboard Evaluation Results
|
| 110 |
+
|
| 111 |
+
Detailed results can be found [here](https://huggingface.co/datasets/eduagarcia-temp/llm_pt_leaderboard_raw_results/tree/main/artificialguybr/OpenHermesV2-PTBR) and on the [๐ Open Portuguese LLM Leaderboard](https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard)
|
| 112 |
+
|
| 113 |
+
| Metric | Value |
|
| 114 |
+
|--------------------------|---------|
|
| 115 |
+
|Average |**61.94**|
|
| 116 |
+
|ENEM Challenge (No Images)| 59.90|
|
| 117 |
+
|BLUEX (No Images) | 48.40|
|
| 118 |
+
|OAB Exams | 41.41|
|
| 119 |
+
|Assin2 RTE | 88.89|
|
| 120 |
+
|Assin2 STS | 67.61|
|
| 121 |
+
|FaQuAD NLI | 53.06|
|
| 122 |
+
|HateBR Binary | 74.77|
|
| 123 |
+
|PT Hate Speech Binary | 64.38|
|
| 124 |
+
|tweetSentBR | 59.01|
|
| 125 |
+
|