Hugging Face's logo

sethuiyer
/
MedleyMD

Text Generation
Transformers
Safetensors
English
mixtral
Mixture of Experts
Merge
medical
mergekit
conversational
Eval Results (legacy)
text-generation-inference
Model card Files
xet
 Community
1

Adding Evaluation Results

#1
by leaderboard-pr-bot - opened
base: refs/heads/main
←
from: refs/pr/1
Discussion Files changed
+123
-7
Files changed (1) hide show
  1. README.md +123 -7
README.md CHANGED
@@ -1,21 +1,124 @@
1
  ---
 
 
2
  license: cc-by-nc-nd-4.0
 
3
  tags:
4
  - moe
5
  - merge
6
  - medical
7
  - mergekit
8
- base_model:
9
- - sethuiyer/Dr_Samantha_7b_mistral
10
- - fblgit/UNA-TheBeagle-7b-v1
11
- language:
12
- - en
13
  datasets:
14
  - medmcqa
15
  - cognitivecomputations/samantha-data
16
  - jondurbin/bagel-v0.3
17
- library_name: transformers
 
 
18
  pipeline_tag: text-generation
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
19
  ---
20
 
21
  # MedleyMD
@@ -114,4 +217,17 @@ It is composed of a topmost gating network that assigns weights to each expert n
114
  The expert networks are trained independently, and the gating network learns to choose the best combination of these experts to make the final prediction.
115
  Mixout demonstrates a stronger ability to handle complex data distributions and is more efficient in terms of training time and memory usage compared to a
116
  traditional ensemble approach.
117
- ```
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
+ language:
3
+ - en
4
  license: cc-by-nc-nd-4.0
5
+ library_name: transformers
6
  tags:
7
  - moe
8
  - merge
9
  - medical
10
  - mergekit
 
 
 
 
 
11
  datasets:
12
  - medmcqa
13
  - cognitivecomputations/samantha-data
14
  - jondurbin/bagel-v0.3
15
+ base_model:
16
+ - sethuiyer/Dr_Samantha_7b_mistral
17
+ - fblgit/UNA-TheBeagle-7b-v1
18
  pipeline_tag: text-generation
19
+ model-index:
20
+ - name: MedleyMD
21
+ results:
22
+ - task:
23
+ type: text-generation
24
+ name: Text Generation
25
+ dataset:
26
+ name: AI2 Reasoning Challenge (25-Shot)
27
+ type: ai2_arc
28
+ config: ARC-Challenge
29
+ split: test
30
+ args:
31
+ num_few_shot: 25
32
+ metrics:
33
+ - type: acc_norm
34
+ value: 66.47
35
+ name: normalized accuracy
36
+ source:
37
+ url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=sethuiyer/MedleyMD
38
+ name: Open LLM Leaderboard
39
+ - task:
40
+ type: text-generation
41
+ name: Text Generation
42
+ dataset:
43
+ name: HellaSwag (10-Shot)
44
+ type: hellaswag
45
+ split: validation
46
+ args:
47
+ num_few_shot: 10
48
+ metrics:
49
+ - type: acc_norm
50
+ value: 86.06
51
+ name: normalized accuracy
52
+ source:
53
+ url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=sethuiyer/MedleyMD
54
+ name: Open LLM Leaderboard
55
+ - task:
56
+ type: text-generation
57
+ name: Text Generation
58
+ dataset:
59
+ name: MMLU (5-Shot)
60
+ type: cais/mmlu
61
+ config: all
62
+ split: test
63
+ args:
64
+ num_few_shot: 5
65
+ metrics:
66
+ - type: acc
67
+ value: 65.1
68
+ name: accuracy
69
+ source:
70
+ url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=sethuiyer/MedleyMD
71
+ name: Open LLM Leaderboard
72
+ - task:
73
+ type: text-generation
74
+ name: Text Generation
75
+ dataset:
76
+ name: TruthfulQA (0-shot)
77
+ type: truthful_qa
78
+ config: multiple_choice
79
+ split: validation
80
+ args:
81
+ num_few_shot: 0
82
+ metrics:
83
+ - type: mc2
84
+ value: 52.46
85
+ source:
86
+ url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=sethuiyer/MedleyMD
87
+ name: Open LLM Leaderboard
88
+ - task:
89
+ type: text-generation
90
+ name: Text Generation
91
+ dataset:
92
+ name: Winogrande (5-shot)
93
+ type: winogrande
94
+ config: winogrande_xl
95
+ split: validation
96
+ args:
97
+ num_few_shot: 5
98
+ metrics:
99
+ - type: acc
100
+ value: 80.27
101
+ name: accuracy
102
+ source:
103
+ url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=sethuiyer/MedleyMD
104
+ name: Open LLM Leaderboard
105
+ - task:
106
+ type: text-generation
107
+ name: Text Generation
108
+ dataset:
109
+ name: GSM8k (5-shot)
110
+ type: gsm8k
111
+ config: main
112
+ split: test
113
+ args:
114
+ num_few_shot: 5
115
+ metrics:
116
+ - type: acc
117
+ value: 68.99
118
+ name: accuracy
119
+ source:
120
+ url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=sethuiyer/MedleyMD
121
+ name: Open LLM Leaderboard
122
  ---
123
 
124
  # MedleyMD
 
217
  The expert networks are trained independently, and the gating network learns to choose the best combination of these experts to make the final prediction.
218
  Mixout demonstrates a stronger ability to handle complex data distributions and is more efficient in terms of training time and memory usage compared to a
219
  traditional ensemble approach.
220
+ ```
221
+ # [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
222
+ Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_sethuiyer__MedleyMD)
223
+
224
+ | Metric |Value|
225
+ |---------------------------------|----:|
226
+ |Avg. |69.89|
227
+ |AI2 Reasoning Challenge (25-Shot)|66.47|
228
+ |HellaSwag (10-Shot) |86.06|
229
+ |MMLU (5-Shot) |65.10|
230
+ |TruthfulQA (0-shot) |52.46|
231
+ |Winogrande (5-shot) |80.27|
232
+ |GSM8k (5-shot) |68.99|
233
+