Hugging Face's logo

CarrotAI
/
Rabbit-Ko-15B-Instruct

Text Generation
Safetensors
Korean
qwen2
Carrot
Korea
mergekit
conversational
Model card Files
xet
Community
CarrotAI commited on
Commit
5aad3c1
·
verified ·
1 Parent(s): 1582e05

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +10 -32
README.md CHANGED
@@ -20,38 +20,16 @@ tags:
20
 
21
  ### Score
22
 
23
- Kmmlu
24
- | 분야 주제 | 수 | 평균 점수 | 최고 점수 | 최저 점수 |
25
- |-----------|---|-----------|-----------|-----------|
26
- | 공학 및 기술 | 15 | 0.4503 | 0.7320 (컴퓨터 과학) | 0.3210 (전기 공학) |
27
- | 자연 과학 | 5 | 0.4013 | 0.4860 (생태학) | 0.3090 (환경 과학) |
28
- | 사회 과학 및 인문학 | 11 | 0.5443 | 0.8010 (마케팅) | 0.3500 (한국 역사) |
29
- | 비즈니스 및 경영 | 7 | 0.5310 | 0.6190 (경영학) | 0.3950 (세금) |
30
- | 의료 및 건강 | 2 | 0.5495 | 0.6200 (건강) | 0.4790 (심리학) |
31
- | 기타 | 5 | 0.4414 | 0.5970 (인테리어 건축 및 디자인) | 0.3620 (건설) |
32
-
33
- 전체 평균 점수: 0.5059 (50.59%)
34
-
35
-
36
-
37
- |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr|
38
- |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
39
- |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.7854|± |0.0113|
40
- | | |strict-match | 5|exact_match|↑ |0.7597|± |0.0118|
41
-
42
- | Tasks |Version|Filter|n-shot| Metric | |Value | |Stderr|
43
- |----------------|------:|------|-----:|--------|---|-----:|---|------|
44
- |kobest_boolq | 1|none | 5|acc |↑ |0.9145|± |0.0075|
45
- | | |none | 5|f1 |↑ |0.9145|± | N/A|
46
- |kobest_copa | 1|none | 5|acc |↑ |0.7050|± |0.0144|
47
- | | |none | 5|f1 |↑ |0.7045|± | N/A|
48
- |kobest_hellaswag| 1|none | 5|acc |↑ |0.4600|± |0.0223|
49
- | | |none | 5|acc_norm|↑ |0.5540|± |0.0223|
50
- | | |none | 5|f1 |↑ |0.4564|± | N/A|
51
- |kobest_sentineg | 1|none | 5|acc |↑ |0.9496|± |0.0110|
52
- | | |none | 5|f1 |↑ |0.9496|± | N/A|
53
- |kobest_wic | 1|none | 5|acc |↑ |0.7294|± |0.0125|
54
- | | |none | 5|f1 |↑ |0.7258|± | N/A|
55
 
56
  ## Quickstart
57
 
 
20
 
21
  ### Score
22
 
23
+ | Benchmark | CarrotAI/Rabbit-Ko-15B-Instruct | Llama 3.1 8B Inst. | Gemma 2 9B Inst. | QWEN 2 7B Inst. | Phi 3 7B Inst. | Mistral 7B | Shot |
24
+ |-----------|:-------------------------------:|:------------------:|:-----------------:|:----------------:|:--------------:|:----------:|:----:|
25
+ | GSM8K | 80.29 | 75.9 | 77.2 | 62.3 | 86.4 | 47.5 | 5 |
26
+ | KMMLU | 47.95 | 41.8 | 40.3 | 46.5 | 37.2 | 31.4 | 5 |
27
+ | KoBEST-BoolQ | 91.67 | 87.6 | 89.9 | 90.2 | 76.9 | 84.3 | 5 |
28
+ | KoBEST-COPA | 71.30 | 72.8 | 60.6 | 70.3 | 54.5 | 62.9 | 5 |
29
+ | KoBEST-WiC | 71.11 | 41.7 | 54.3 | 65.9 | 56.0 | 44.6 | 5 |
30
+ | KoBEST-HellaSwag | 45.40 | 44.5 | 42.6 | 46.8 | 34.8 | 42.4 | 5 |
31
+ | KoBEST-SentiNeg | 94.96 | 95.2 | 72.0 | 92.9 | 81.0 | 84.7 | 5 |
32
+ | Average | 71.81 | 65.64 | 62.41 | 67.84 | 60.97 | 56.83 | - |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
33
 
34
  ## Quickstart
35