Hugging Face's logo

Spaces:
steampunque
/
benchlm
Running

App Files Community
steampunque commited on
Commit
eae8382
·
1 Parent(s): 71856e3

Add madlad400-10b-mt Q4_K_H MT result

Browse files
Files changed (1) hide show
  1. README.md +45 -45
README.md CHANGED
@@ -730,49 +730,49 @@ AUDIO MODELS:
730
 
731
  MT MODELS:
732
 
733
- MODEL | HY-MT1.5-7B | madlad400-10b-mt | plamo-2-translate | translategemma-4b-it | translategemma-12b-it | translategemma-12b-it | translategemma-27b-it |
734
- ---------------------------------------------|-------------|------------------|-------------------|----------------------|-----------------------|-----------------------|-----------------------|
735
- params | 7.50B | 10.71B | 9.53B | 3.88B | 11.77B | 11.77B | 27.01B |
736
- quant | Q6_K_H | Q6_K_H | Q6_K_H | Q6_K_H | Q4_K_H | Q6_K_H | Q4_K_H |
737
- engine | llama.cpp version: 7789 | llama.cpp version: 7772 | llama.cpp version: 7762 | llama.cpp version: 7760 | llama.cpp version: 7779 | llama.cpp version: 7779 | llama.cpp version: 7789 |
738
- **TEST** | **acc** | **acc** | **acc** | **acc** | **acc** | **acc** | **acc** |
739
- FLORES200_de_en | 32.2 | 42.3 | 35.5 | 37.8 | 40.3 | 40.1 | 42.2 |
740
- FLORES200_en_de | 3.0 | 36.9 | 24.1 | 31.0 | 34.5 | 34.2 | 35.3 |
741
- FLORES200_en_es | 25.5 | 26.5 | 21.3 | 25.1 | 27.0 | 27.3 | 27.2 |
742
- FLORES200_en_fr | 37.2 | 49.8 | 36.5 | 40.9 | 44.7 | 44.2 | 45.2 |
743
- FLORES200_en_ja | 31.0 | 22.3 | 28.2 | 27.7 | 30.7 | 30.8 | 31.0 |
744
- FLORES200_en_ru | 22.6 | 28.8 | 18.2 | 25.4 | 27.5 | 28.1 | 28.2 |
745
- FLORES200_en_zh | 37.0 | 37.3 | 33.6 | 36.6 | 39.7 | 39.5 | 41.9 |
746
- FLORES200_es_en | 23.9 | 29.2 | 24.7 | 28.0 | 28.7 | 29.0 | 29.9 |
747
- FLORES200_fr_en | 32.6 | 44.3 | 36.4 | 38.8 | 41.2 | 41.7 | 43.4 |
748
- FLORES200_ja_en | 20.5 | 25.9 | 23.4 | 21.5 | 24.1 | 23.9 | 26.6 |
749
- FLORES200_ru_en | 26.7 | 34.9 | 29.0 | 30.7 | 33.0 | 32.7 | 34.8 |
750
- FLORES200_zh_en | 22.9 | 27.3 | 23.4 | 23.5 | 25.7 | 25.6 | 27.8 |
751
- FLORES200 | 26.3 | 33.8 | 27.9 | 30.6 | 33.1 | 33.1 | 34.4 |
752
- OPUS_de_en | 25.9 | 27.1 | 21.3 | 28.2 | 29.8 | 29.2 | 30.5 |
753
- OPUS_en_de | 11.0 | 30.7 | 21.1 | 25.5 | 26.5 | 25.5 | 27.1 |
754
- OPUS_en_es | 29.0 | 37.1 | 27.9 | 31.1 | 32.0 | 31.3 | 32.4 |
755
- OPUS_en_fr | 24.7 | 34.3 | 26.7 | 27.0 | 28.8 | 28.8 | 29.0 |
756
- OPUS_en_ja | 10.2 | 15.9 | 11.9 | 9.8 | 10.5 | 10.5 | 11.7 |
757
- OPUS_en_ru | 22.0 | 31.6 | 19.7 | 23.1 | 24.9 | 24.4 | 25.5 |
758
- OPUS_en_zh | 28.8 | 41.3 | 26.7 | 26.4 | 28.9 | 28.8 | 29.9 |
759
- OPUS_es_en | 28.4 | 40.3 | 26.7 | 33.2 | 35.0 | 34.5 | 36.4 |
760
- OPUS_fr_en | 25.8 | 35.8 | 29.6 | 29.2 | 30.9 | 30.6 | 32.3 |
761
- OPUS_ja_en | 14.8 | 18.8 | 16.0 | 15.1 | 17.0 | 16.2 | 16.8 |
762
- OPUS_ru_en | 24.5 | 34.8 | 27.1 | 27.1 | 29.0 | 28.3 | 30.2 |
763
- OPUS_zh_en | 26.1 | 38.8 | 23.8 | 22.1 | 25.5 | 24.9 | 27.6 |
764
- OPUS | 22.6 | 32.2 | 23.2 | 24.8 | 26.6 | 26.1 | 27.4 |
765
- DE_EN | 28.0 | 32.1 | 26.0 | 31.4 | 33.3 | 32.8 | 34.4 |
766
- EN_DE | 8.3 | 32.7 | 22.1 | 27.3 | 29.2 | 28.3 | 29.8 |
767
- ES_EN | 26.8 | 36.5 | 26.0 | 31.4 | 32.8 | 32.6 | 34.2 |
768
- EN_ES | 27.8 | 33.5 | 25.6 | 29.0 | 30.2 | 29.9 | 30.6 |
769
- FR_EN | 28.0 | 38.6 | 31.9 | 32.4 | 34.3 | 34.3 | 36.0 |
770
- EN_FR | 28.9 | 39.5 | 30.0 | 31.6 | 34.1 | 33.9 | 34.4 |
771
- RU_EN | 25.2 | 34.8 | 27.7 | 28.2 | 30.3 | 29.8 | 31.7 |
772
- EN_RU | 22.2 | 30.6 | 19.1 | 23.8 | 25.7 | 25.6 | 26.3 |
773
- JA_EN | 16.6 | 21.2 | 18.4 | 17.2 | 19.3 | 18.8 | 20.1 |
774
- EN_JA | 17.2 | 18.0 | 17.3 | 15.8 | 17.2 | 17.3 | 18.1 |
775
- ZH_EN | 25.0 | 34.9 | 23.6 | 22.5 | 25.5 | 25.1 | 27.6 |
776
- EN_ZH | 31.5 | 39.9 | 29.0 | 29.8 | 32.4 | 32.3 | 33.9 |
777
  COMPOSITE AVERAGE
778
- AVG | 23.8 | 32.7 | 24.7 | 26.7 | 28.7 | 28.4 | 29.8 |
 
730
 
731
  MT MODELS:
732
 
733
+ MODEL | HY-MT1.5-7B | madlad400-10b-mt | madlad400-10b-mt | plamo-2-translate | translategemma-4b-it | translategemma-12b-it | translategemma-12b-it | translategemma-27b-it |
734
+ ---------------------------------------------|-------------|------------------|------------------|-------------------|----------------------|-----------------------|-----------------------|-----------------------|
735
+ params | 7.50B | 10.71B | 10.71B | 9.53B | 3.88B | 11.77B | 11.77B | 27.01B |
736
+ quant | Q6_K_H | Q4_K_H | Q6_K_H | Q6_K_H | Q6_K_H | Q4_K_H | Q6_K_H | Q4_K_H |
737
+ engine | llama.cpp version: 7789 | llama.cpp version: 7830 | llama.cpp version: 7772 | llama.cpp version: 7762 | llama.cpp version: 7760 | llama.cpp version: 7779 | llama.cpp version: 7779 | llama.cpp version: 7789 |
738
+ **TEST** | **acc** | **acc** | **acc** | **acc** | **acc** | **acc** | **acc** | **acc** |
739
+ FLORES200_de_en | 32.2 | 42.3 | 42.3 | 35.5 | 37.8 | 40.3 | 40.1 | 42.2 |
740
+ FLORES200_en_de | 3.0 | 36.6 | 36.9 | 24.1 | 31.0 | 34.5 | 34.2 | 35.3 |
741
+ FLORES200_en_es | 25.5 | 26.5 | 26.5 | 21.3 | 25.1 | 27.0 | 27.3 | 27.2 |
742
+ FLORES200_en_fr | 37.2 | 49.7 | 49.8 | 36.5 | 40.9 | 44.7 | 44.2 | 45.2 |
743
+ FLORES200_en_ja | 31.0 | 22.2 | 22.3 | 28.2 | 27.7 | 30.7 | 30.8 | 31.0 |
744
+ FLORES200_en_ru | 22.6 | 28.7 | 28.8 | 18.2 | 25.4 | 27.5 | 28.1 | 28.2 |
745
+ FLORES200_en_zh | 37.0 | 37.2 | 37.3 | 33.6 | 36.6 | 39.7 | 39.5 | 41.9 |
746
+ FLORES200_es_en | 23.9 | 29.1 | 29.2 | 24.7 | 28.0 | 28.7 | 29.0 | 29.9 |
747
+ FLORES200_fr_en | 32.6 | 44.1 | 44.3 | 36.4 | 38.8 | 41.2 | 41.7 | 43.4 |
748
+ FLORES200_ja_en | 20.5 | 25.9 | 25.9 | 23.4 | 21.5 | 24.1 | 23.9 | 26.6 |
749
+ FLORES200_ru_en | 26.7 | 34.7 | 34.9 | 29.0 | 30.7 | 33.0 | 32.7 | 34.8 |
750
+ FLORES200_zh_en | 22.9 | 27.3 | 27.3 | 23.4 | 23.5 | 25.7 | 25.6 | 27.8 |
751
+ FLORES200 | 26.3 | 33.7 | 33.8 | 27.9 | 30.6 | 33.1 | 33.1 | 34.4 |
752
+ OPUS_de_en | 25.9 | 35.6 | 27.1 | 21.3 | 28.2 | 29.8 | 29.2 | 30.5 |
753
+ OPUS_en_de | 11.0 | 32.9 | 30.7 | 21.1 | 25.5 | 26.5 | 25.5 | 27.1 |
754
+ OPUS_en_es | 29.0 | 37.1 | 37.1 | 27.9 | 31.1 | 32.0 | 31.3 | 32.4 |
755
+ OPUS_en_fr | 24.7 | 34.2 | 34.3 | 26.7 | 27.0 | 28.8 | 28.8 | 29.0 |
756
+ OPUS_en_ja | 10.2 | 16.0 | 15.9 | 11.9 | 9.8 | 10.5 | 10.5 | 11.7 |
757
+ OPUS_en_ru | 22.0 | 31.7 | 31.6 | 19.7 | 23.1 | 24.9 | 24.4 | 25.5 |
758
+ OPUS_en_zh | 28.8 | 41.1 | 41.3 | 26.7 | 26.4 | 28.9 | 28.8 | 29.9 |
759
+ OPUS_es_en | 28.4 | 40.2 | 40.3 | 26.7 | 33.2 | 35.0 | 34.5 | 36.4 |
760
+ OPUS_fr_en | 25.8 | 35.8 | 35.8 | 29.6 | 29.2 | 30.9 | 30.6 | 32.3 |
761
+ OPUS_ja_en | 14.8 | 18.9 | 18.8 | 16.0 | 15.1 | 17.0 | 16.2 | 16.8 |
762
+ OPUS_ru_en | 24.5 | 34.7 | 34.8 | 27.1 | 27.1 | 29.0 | 28.3 | 30.2 |
763
+ OPUS_zh_en | 26.1 | 38.5 | 38.8 | 23.8 | 22.1 | 25.5 | 24.9 | 27.6 |
764
+ OPUS | 22.6 | 33.1 | 32.2 | 23.2 | 24.8 | 26.6 | 26.1 | 27.4 |
765
+ DE_EN | 28.0 | 37.8 | 32.1 | 26.0 | 31.4 | 33.3 | 32.8 | 34.4 |
766
+ EN_DE | 8.3 | 34.1 | 32.7 | 22.1 | 27.3 | 29.2 | 28.3 | 29.8 |
767
+ ES_EN | 26.8 | 36.4 | 36.5 | 26.0 | 31.4 | 32.8 | 32.6 | 34.2 |
768
+ EN_ES | 27.8 | 33.5 | 33.5 | 25.6 | 29.0 | 30.2 | 29.9 | 30.6 |
769
+ FR_EN | 28.0 | 38.6 | 38.6 | 31.9 | 32.4 | 34.3 | 34.3 | 36.0 |
770
+ EN_FR | 28.9 | 39.4 | 39.5 | 30.0 | 31.6 | 34.1 | 33.9 | 34.4 |
771
+ RU_EN | 25.2 | 34.7 | 34.8 | 27.7 | 28.2 | 30.3 | 29.8 | 31.7 |
772
+ EN_RU | 22.2 | 30.7 | 30.6 | 19.1 | 23.8 | 25.7 | 25.6 | 26.3 |
773
+ JA_EN | 16.6 | 21.2 | 21.2 | 18.4 | 17.2 | 19.3 | 18.8 | 20.1 |
774
+ EN_JA | 17.2 | 18.0 | 18.0 | 17.3 | 15.8 | 17.2 | 17.3 | 18.1 |
775
+ ZH_EN | 25.0 | 34.7 | 34.9 | 23.6 | 22.5 | 25.5 | 25.1 | 27.6 |
776
+ EN_ZH | 31.5 | 39.8 | 39.9 | 29.0 | 29.8 | 32.4 | 32.3 | 33.9 |
777
  COMPOSITE AVERAGE
778
+ AVG | 23.8 | 33.2 | 32.7 | 24.7 | 26.7 | 28.7 | 28.4 | 29.8 |