Translation
English
Japanese
Eval Results (legacy)
radinplaid commited on
Commit
80ea7de
·
verified ·
1 Parent(s): 4f9134c

Update README.md

Browse files

Add Aya Expanse 8b and Hunyuan-MT metrics

Files changed (1) hide show
  1. README.md +10 -8
README.md CHANGED
@@ -106,11 +106,13 @@ The model is in `ctranslate2` format, and the tokenizers are `sentencepiece`, so
106
  `bleu` and `chrf2` are calculated with [sacrebleu](https://github.com/mjpost/sacrebleu) on the [Flores200 `devtest` test set](https://huggingface.co/datasets/facebook/flores), `comet22` with the [`comet`](https://github.com/Unbabel/COMET) library and the [default model](https://huggingface.co/Unbabel/wmt22-comet-da). "Time (s)" is the time in seconds to translate the flores-devtest dataset (1012 sentences) on an RTX 4070s GPU with batch size 32.
107
 
108
 
109
- | | bleu | chrf2 | comet22 | Time (s) |
110
- |:---------------------------------|-------:|--------:|----------:|-----------:|
111
- | quickmt/quickmt-ja-en | 28.87 | 58.56 | 87.83 | 1.10 |
112
- | Helsink-NLP/opus-mt-ja-en | 19.22 | 49.15 | 82.64 | 3.54 |
113
- | facebook/nllb-200-distilled-600M | 24.05 | 53.39 | 85.84 | 22.5 |
114
- | facebook/nllb-200-distilled-1.3B | 28.4 | 56.96 | 87.47 | 37.15 |
115
- | facebook/m2m100_418M | 18.82 | 49.55 | 82.59 | 18.27 |
116
- | facebook/m2m100_1.2B | 23.32 | 53.46 | 85.43 | 35.44 |
 
 
 
106
  `bleu` and `chrf2` are calculated with [sacrebleu](https://github.com/mjpost/sacrebleu) on the [Flores200 `devtest` test set](https://huggingface.co/datasets/facebook/flores), `comet22` with the [`comet`](https://github.com/Unbabel/COMET) library and the [default model](https://huggingface.co/Unbabel/wmt22-comet-da). "Time (s)" is the time in seconds to translate the flores-devtest dataset (1012 sentences) on an RTX 4070s GPU with batch size 32.
107
 
108
 
109
+ | | bleu | chrf2 | comet22 | Time (s) |
110
+ |:---------------------------------------|-------:|--------:|----------:|-----------:|
111
+ | quickmt/quickmt-ja-en | 28.87 | 58.56 | 87.83 | 1.10 |
112
+ | Helsink-NLP/opus-mt-ja-en | 19.22 | 49.15 | 82.64 | 3.54 |
113
+ | tencent/Hunyuan-MT-7B-fp8 | 22.79 | 56.73 | 87.94 | 22 |
114
+ | facebook/nllb-200-distilled-600M | 24.05 | 53.39 | 85.84 | 22.5 |
115
+ | facebook/nllb-200-distilled-1.3B | 28.4 | 56.96 | 87.47 | 37.15 |
116
+ | facebook/m2m100_418M | 18.82 | 49.55 | 82.59 | 18.27 |
117
+ | facebook/m2m100_1.2B | 23.32 | 53.46 | 85.43 | 35.44 |
118
+ | CohereLabs/aya-expanse-8b (bnb quant) | 26.40 | 56.92 | 87.97 | 73 |