README.md

---
language:
- en
- tr
license: cc-by-nc-sa-4.0
library_name: comet
pipeline_tag: translation
base_model: Unbabel/XCOMET-XL
tags:
- machine-translation
- mt-evaluation
- quality-estimation
- comet
- xcomet
- turkish
---

# xCOMET-XL-TR (v2) — English↔Turkish MT evaluation

A fine-tune of Unbabel's **[xCOMET-XL](https://huggingface.co/Unbabel/XCOMET-XL)**
(3.5B params) specialised for **English↔Turkish** machine-translation quality
estimation, with 5 lexical features fused through a small residual bottleneck
(per *BLEU Meets COMET*, Glushkova et al. 2023).

Given a `(source, machine-translation, reference)` triplet it returns a quality
score, roughly in `[0, 1]` — **higher is better**.

- **Weights:** this repo — `xcomet-xl-tr-v2.bf16.ckpt` (BF16, ~7 GB).
- **Code:** *[code repository — anonymized for review]* — you need both this repo and the code repo.
- Anonymized repository URL: https://anonymous.4open.science/r/TurCOMET-8A79/README.md

## Performance

On a held-out Turkish WMT-DA test split (1,768 rows) it beats baseline xCOMET-XL
on every correlation metric against human DA scores:

| Metric | Baseline xCOMET-XL | This model |
|---|---:|---:|
| Pearson (regression-only)  | 0.473 | **0.547** |
| Spearman (regression-only) | 0.531 | **0.562** |
| Kendall (regression-only)  | 0.368 | **0.394** |
| Pearson (full predict_step) | 0.479 | **0.515** |

It cleanly ranks hand-crafted PERFECT > GOOD > BAD > TERRIBLE translations (8/8
groups) in both directions, with PERFECT translations scoring ~0.94–0.98.

## Quick start

**1. Install** (Python ≥ 3.10, CUDA GPU recommended). The order matters —
`unbabel-comet` over-pins numpy/protobuf, so they are restored afterwards:

```bash
# clone the (anonymized) code repository, then:
cd xcomet-xl-tr
bash install.sh
# install.sh runs:
#   pip install "unbabel-comet==2.2.7" "sentence-transformers>=3.0.0" \
#               "sacrebleu>=2.4.0" "zemberek-python>=0.2.3" "huggingface_hub>=0.23"
#   pip install "numpy==2.0.2" "protobuf>=5.29,<6"
```

**2. Authenticate** (this model is private):

```bash
huggingface-cli login        # or:  export HF_TOKEN=hf_xxx
```

**3. Score a triplet:**

```python
from huggingface_hub import hf_hub_download
from xcomet_tr import load_model, score

ckpt = hf_hub_download("XCOMETTR/XCOMET-XL-TR", "xcomet-xl-tr-v2.bf16.ckpt")
model = load_model(ckpt)          # bf16, GPU if available

# (source, machine_translation, reference, direction)  — direction: "en-tr" | "tr-en"
triplets = [
    ("Istanbul is the largest city in Turkey.",
     "İstanbul, Türkiye'nin en büyük şehridir.",
     "İstanbul, Türkiye'nin en büyük şehridir.", "en-tr"),
    ("Hava bugün çok güzel.",
     "The weather is very nice today.",
     "The weather is very nice today.", "tr-en"),
]
print(score(model, triplets))     # e.g. [0.97, 0.96]
```

`python example.py` in the code repo runs exactly this end-to-end.

## How it works

`XCOMETFeatures` (an `XCOMETMetric` subclass) adds one module — a
`[encoder_dim + 5] → 64 → encoder_dim` bottleneck added **residually** (zero-init,
so it starts identical to xCOMET-XL) to the pooled sentence embedding. The 5
features are: chrF++(mt,ref), LaBSE cos(src,mt), length-ratio z-score, lemma-TER
(Turkish lemmatised via Zemberek), and a direction flag. Load it via the code
repo's `load_model`, which uses `load_pretrained_weights=False` so the
self-contained checkpoint needs **no** extra base-encoder download.

## Notes & limitations

- **BF16** — published/loaded in bfloat16 (xCOMET-XL was trained bf16-mixed);
  matches fp32 to 2–3 decimals.
- **512-token window** — XLM-R-XL caps at 512 tokens; xCOMET encodes `mt+src`,
  `mt+ref`, and `mt+src+ref`, so long documents are truncated. Best used per
  sentence / short paragraph; for documents, score sentences and average.
- **Domain** — fine-tuned on news-domain WMT-DA (2017–2018); expect some shift
  elsewhere. Turkish word-level supervision is heuristic (no human MQM spans
  exist for Turkish).

## License

**CC-BY-NC-SA 4.0**, inherited from [Unbabel/XCOMET-XL](https://huggingface.co/Unbabel/XCOMET-XL).
Non-commercial use only; derivatives must use the same license. Built on
[Unbabel/COMET](https://github.com/Unbabel/COMET); lexical fusion from Glushkova
et al. 2023; Turkish morphology via Zemberek.