| --- |
| pipeline_tag: translation |
| library_name: comet |
| language: |
| - multilingual |
| - af |
| - am |
| - ar |
| - as |
| - az |
| - be |
| - bg |
| - bn |
| - br |
| - bs |
| - ca |
| - cs |
| - cy |
| - da |
| - de |
| - el |
| - en |
| - eo |
| - es |
| - et |
| - eu |
| - fa |
| - fi |
| - fr |
| - fy |
| - ga |
| - gd |
| - gl |
| - gu |
| - ha |
| - he |
| - hi |
| - hr |
| - hu |
| - hy |
| - id |
| - is |
| - it |
| - ja |
| - jv |
| - ka |
| - kk |
| - km |
| - kn |
| - ko |
| - ku |
| - ky |
| - la |
| - lo |
| - lt |
| - lv |
| - mg |
| - mk |
| - ml |
| - mn |
| - mr |
| - ms |
| - my |
| - ne |
| - nl |
| - 'no' |
| - om |
| - or |
| - pa |
| - pl |
| - ps |
| - pt |
| - ro |
| - ru |
| - sa |
| - sd |
| - si |
| - sk |
| - sl |
| - so |
| - sq |
| - sr |
| - su |
| - sv |
| - sw |
| - ta |
| - te |
| - th |
| - tl |
| - tr |
| - ug |
| - uk |
| - ur |
| - uz |
| - vi |
| - xh |
| - yi |
| - zh |
| license: apache-2.0 |
| base_model: |
| - FacebookAI/xlm-roberta-large |
| --- |
| |
| # COMET-poly-base-wmt25 |
|
|
| This model is based on [COMET-poly](https://github.com/zouharvi/COMET-poly), which is a fork but not compatible with original Unbabel's COMET. |
| To run the model, you need to first install this version of COMET either with: |
| ```bash |
| pip install "git+https://github.com/zouharvi/COMET-poly#egg=comet-poly&subdirectory=comet_poly" |
| ``` |
| or in editable mode: |
| ```bash |
| git clone https://github.com/zouharvi/COMET-poly.git |
| cd COMET-poly |
| pip3 install -e comet_poly |
| ``` |
|
|
| This model scores the translation `mt` given its source. It is a baseline model that other COMET-poly models are compared to. |
| ```python |
| import comet_poly |
| model = comet_poly.load_from_checkpoint(comet_poly.download_model("zouharvi/COMET-poly-base-wmt25")) |
| data = [ |
| { |
| "src": "Iceberg lettuce got its name in the 1920s when it was shipped packed in ice to stay fresh.", |
| "mt": "Eisbergsalat erhielt seinen Namen in den 1920er-Jahren, als er in Eis verpackt verschickt wurde, um frisch zu bleiben.", |
| }, |
| { |
| "src": "Goats have rectangular pupils, which give them a wide field of vision—up to 320 degrees!", |
| "mt": "Kozy mají obdélníkové zornice, což jim umožňuje vidět skoro všude kolem sebe, aniž by musely otáčet hlavou.", |
| }, |
| { |
| "src": "This helps them spot predators from almost all directions without moving their heads.", |
| "mt": "Điều này giúp chúng phát hiện kẻ săn mồi từ gần như mọi hướng mà không cần quay đầu.", |
| } |
| ] |
| print("scores", model.predict(data, batch_size=8, gpus=1).scores) |
| ``` |
| Outputs: |
| ``` |
| scores [94.98790740966797, 77.56731414794922, 90.77655029296875] |
| ``` |
|
|
| The training data is WMT up to 2024 (inclusive) with DA/ESA/MQM merged on a single scale. |
| This model is based on the work [COMET-poly: Machine Translation Metric Grounded in Other Candidates](https://aclanthology.org/2025.wmt-1.63/) which can be cited as: |
| ``` |
| @inproceedings{zufle-etal-2025-comet, |
| title = "{COMET}-poly: Machine Translation Metric Grounded in Other Candidates", |
| author = {Z{\"u}fle, Maike and |
| Zouhar, Vil{\'e}m and |
| Dinh, Tu Anh and |
| Maia Polo, Felipe and |
| Niehues, Jan and |
| Sachan, Mrinmaya}, |
| editor = "Haddow, Barry and |
| Kocmi, Tom and |
| Koehn, Philipp and |
| Monz, Christof", |
| booktitle = "Proceedings of the Tenth Conference on Machine Translation", |
| month = nov, |
| year = "2025", |
| address = "Suzhou, China", |
| publisher = "Association for Computational Linguistics", |
| url = "https://aclanthology.org/2025.wmt-1.63/", |
| doi = "10.18653/v1/2025.wmt-1.63", |
| pages = "887--904", |
| ISBN = "979-8-89176-341-8", |
| abstract = "Automated metrics for machine translation attempt to replicate human judgment. Unlike humans, who often assess a translation in the context of multiple alternatives, these metrics typically consider only the source sentence and a single translation. This discrepancy in the evaluation setup may negatively impact the performance of automated metrics. We propose two automated metrics that incorporate additional information beyond the single translation. COMET-polycand uses alternative translations of the same source sentence to compare and contrast with the translation at hand, thereby providing a more informed assessment of its quality. COMET-polyic, inspired by retrieval-based in-context learning, takes in translations of similar source texts along with their human-labeled quality scores to guide the evaluation. We find that including a single additional translation in COMET-polycand improves the segment-level metric performance (0.079 to 0.118 Kendall{'}s tau-b correlation), with further gains when more translations are added. Incorporating retrieved examples in COMET-polyic yields similar improvements (0.079 to 0.116 Kendall{'}s tau-b correlation). We release our models publicly." |
| } |
| ``` |
