davidmeikle commited on
Commit
ddbf03d
·
verified ·
1 Parent(s): f38b06a

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +106 -0
README.md ADDED
@@ -0,0 +1,106 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: eole
3
+ tags:
4
+ - eole
5
+ - comet
6
+ - machine-translation-evaluation
7
+ - quality-estimation
8
+ - converted
9
+ - fp16
10
+ base_model: Unbabel/wmt22-comet-da
11
+ license: apache-2.0
12
+ ---
13
+
14
+ # WMT22 COMET DA FP16 (EOLE)
15
+
16
+ This is [Unbabel/wmt22-comet-da](https://huggingface.co/Unbabel/wmt22-comet-da) converted to [EOLE](https://github.com/eole-nlp/eole) format.
17
+
18
+ Model weights were dtype-cast to fp16 during EOLE conversion. This repository contains a format and dtype conversion for use with EOLE's native `transformer_encoder_scorer` implementation.
19
+
20
+ This is not an upstream `unbabel-comet` checkpoint layout. Use it with EOLE.
21
+
22
+ ## Model Details
23
+
24
+ | | |
25
+ |---|---|
26
+ | Original model | [Unbabel/wmt22-comet-da](https://huggingface.co/Unbabel/wmt22-comet-da) |
27
+ | EOLE architecture | `transformer_encoder_scorer` |
28
+ | Weight dtype | `fp16` |
29
+ | Scoring type | `comet` |
30
+ | Class identifier | `regression_metric` |
31
+ | EOLE scorer | `EOLE-COMET` |
32
+ | Requires reference | Yes |
33
+ | Encoder | XLM-R large style encoder |
34
+
35
+ ## Usage
36
+
37
+ Requires an EOLE version with native COMET scorer support.
38
+
39
+ ### Validation Metric
40
+
41
+ ```yaml
42
+ valid_metrics: ["EOLE-COMET"]
43
+ comet_model: eole-nlp/wmt22-comet-da-eole-fp16
44
+ comet_batch_size: 64
45
+ ```
46
+
47
+ ### Direct Scoring
48
+
49
+ ```bash
50
+ eole predict \
51
+ --model_path eole-nlp/wmt22-comet-da-eole-fp16 \
52
+ --src /path/to/src.txt \
53
+ --tgt /path/to/mt.txt \
54
+ --ref /path/to/ref.txt \
55
+ --output /path/to/scores.txt \
56
+ --with_score
57
+ ```
58
+
59
+ By default, direct scoring writes one segment score per input line. To emit a
60
+ single aggregate system score, add `--score_level system`:
61
+
62
+ ```bash
63
+ eole predict \
64
+ --model_path eole-nlp/wmt22-comet-da-eole-fp16 \
65
+ --src /path/to/src.txt \
66
+ --tgt /path/to/mt.txt \
67
+ --ref /path/to/ref.txt \
68
+ --output /path/to/system-score.txt \
69
+ --with_score \
70
+ --score_level system
71
+ ```
72
+
73
+ `score_level: system` writes one numeric line containing the arithmetic mean of
74
+ the segment scores, similar to Unbabel COMET's `--only_system` mode but using
75
+ EOLE's score-file format.
76
+
77
+ ## Conversion
78
+
79
+ Converted with:
80
+
81
+ ```bash
82
+ eole convert COMET \
83
+ --model Unbabel/wmt22-comet-da \
84
+ --dtype fp16 \
85
+ --output wmt22-comet-da-eole-fp16
86
+ ```
87
+
88
+ ## Parity
89
+
90
+ The fp32 EOLE conversion was parity-checked against the upstream Unbabel COMET runtime. On a 100-line WMT17 slice, native EOLE scoring matched the upstream baseline within floating point tolerance:
91
+
92
+ | Metric | Value |
93
+ |---|---:|
94
+ | System score delta | `1.85e-08` |
95
+ | Sentence MAE | `5.42e-08` |
96
+ | Sentence max abs | `7.15e-07` |
97
+
98
+ The fp16 artifacts are intended to provide smaller EOLE model files and fp16 runtime behavior. Run the EOLE COMET parity harness against these fp16 repositories if exact fp16 deltas are required.
99
+
100
+ ## Original Model and Attribution
101
+
102
+ This model is a conversion of [Unbabel/wmt22-comet-da](https://huggingface.co/Unbabel/wmt22-comet-da). The model weights were converted to EOLE format and dtype-cast to fp16.
103
+
104
+ This converted repository follows the upstream model license: Apache-2.0. Please refer to the original model card and license for training data, intended use, limitations, and citation details.
105
+
106
+ Unbabel COMET: https://github.com/Unbabel/COMET