---
license: other
license_name: custom-model-license
license_link: LICENSE
library_name: transformers
pipeline_tag: text-generation
language:
- en
- ru
- kk
- be
- hy
tags:
- translation
- wmt26
- machine-translation
- finetune
---

# 8B submission to the WMT26 General MT task

This repository contains our constrained submission to the WMT26 General
Machine Translation task. The model translates English into Russian,
Belarusian, Kazakh, and Armenian. It is an approximately 8B-parameter
decoder-only causal language model.

## Model details

| Property | Value |
|---|---|
| Model type | Decoder-only causal language model |
| Parameters | Approximately 8B |
| Source language | English |
| Target languages | Russian, Belarusian, Kazakh, Armenian |
| Context length | 32,768 tokens |
| License | See [`LICENSE`](LICENSE) |

## Usage

The repository includes [`inference.py`](inference.py), which supports plain
translation prompts, the four WMT26 domain prompts, and custom instructions.
It uses greedy decoding and prints only the generated translation to standard
output.

Install the required packages:

```bash
python -m pip install "torch>=2.1" "transformers>=4.46.3,<5" accelerate sentencepiece packaging
```

Translate a string into Russian:

```bash
python inference.py \
  --target ru \
  --text "The agreement will enter into force next month."
```

Use one of the WMT26 domain instructions:

```bash
python inference.py \
  --target kk \
  --domain news \
  --text "The committee announced the results on Tuesday."
```

The supported domain values are `social`, `speech`, `news`, and `software`.
The source text can also be supplied through standard input. Use `--prompt` or
`--prompt-file` to provide a custom instruction.

### Transformers example

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "foksly/wmt26-constrained-submission"
tokenizer = AutoTokenizer.from_pretrained(model_id, use_fast=False)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
    device_map="auto",
).eval()

prompt = """Переведи с английского на казахский:
The committee announced the results on Tuesday."""
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=512, do_sample=False)
print(tokenizer.decode(
    output[0, inputs["input_ids"].shape[1]:],
    skip_special_tokens=True,
))
```

## Evaluation

We evaluate the released checkpoint against five public models with at most
20B parameters. These are local evaluation results, not official WMT26 scores.

### BOUQuET

The following paragraph-level results use the BOUQuET references. MetricX
uses the
[`google/metricx-24-hybrid-xxl-v2p6`](https://huggingface.co/google/metricx-24-hybrid-xxl-v2p6)
checkpoint.

#### ChrF++ (higher is better)

| System | en-ru | en-be | en-kk | en-hy |
|---|---:|---:|---:|---:|
| **Our Model** | **62.3** | **58.0** | **55.3** | **53.5** |
| TranslateGemma-12B | 62.2 | 54.9 | 47.9 | 50.1 |
| Qwen-3.5-9B | 60.1 | 49.1 | 48.3 | 49.6 |
| MADLAD-400-10B | 59.4 | 54.2 | 46.8 | 50.3 |
| NLLB-200-3.3B | 57.4 | 52.1 | 47.8 | 52.8 |
| GPT-OSS-20B | 51.9 | 29.3 | 49.0 | 12.9 |

#### MetricX-24-XXL (lower is better)

| System | en-ru | en-be | en-kk | en-hy |
|---|---:|---:|---:|---:|
| **Our Model** | 1.58 | 3.87 | **3.44** | **5.41** |
| TranslateGemma-12B | **1.52** | **3.83** | 5.60 | 6.30 |
| Qwen-3.5-9B | 2.22 | 5.73 | 5.52 | 7.32 |
| MADLAD-400-10B | 4.79 | 5.56 | 5.86 | 7.13 |
| NLLB-200-3.3B | 6.01 | 7.63 | 7.10 | 6.35 |
| GPT-OSS-20B | 2.72 | 6.16 | 5.82 | 9.18 |

### WMT25 General MT source paragraphs

We also translate the official English source paragraphs from the WMT25
General MT task and evaluate them with ORBIT-SC using GPT-5.4 as a single
judge. The MQM score is computed as `5 × Major + Minor` from the predicted
error spans.

#### Accuracy (higher is better)

| System | en-ru | en-be | en-kk | en-hy |
|---|---:|---:|---:|---:|
| **Our Model** | **81.6** | **69.1** | **69.9** | **52.2** |
| TranslateGemma-12B | 73.9 | 59.1 | 47.2 | 41.8 |
| Qwen-3.5-9B | 69.6 | 51.1 | 46.2 | 39.1 |
| MADLAD-400-10B | 32.5 | 40.5 | 40.8 | 28.3 |
| NLLB-200-3.3B | 39.7 | 33.8 | 32.2 | 35.0 |
| GPT-OSS-20B | 62.4 | 40.9 | 40.8 | 30.2 |

#### Fluency (higher is better)

| System | en-ru | en-be | en-kk | en-hy |
|---|---:|---:|---:|---:|
| **Our Model** | **85.3** | **69.5** | **72.0** | **54.0** |
| TranslateGemma-12B | 81.2 | 64.4 | 52.5 | 48.1 |
| Qwen-3.5-9B | 72.7 | 51.8 | 51.4 | 44.8 |
| MADLAD-400-10B | 32.5 | 44.8 | 48.4 | 37.5 |
| NLLB-200-3.3B | 40.3 | 33.8 | 35.7 | 36.4 |
| GPT-OSS-20B | 63.5 | 39.3 | 42.3 | 29.4 |

#### MQM (lower is better)

| System | en-ru | en-be | en-kk | en-hy |
|---|---:|---:|---:|---:|
| **Our Model** | **11.3** | **24.2** | **22.3** | 37.6 |
| TranslateGemma-12B | 17.9 | 30.8 | 40.9 | 43.8 |
| Qwen-3.5-9B | 21.4 | 38.5 | 40.8 | 46.4 |
| MADLAD-400-10B | 34.5 | 38.6 | 40.1 | **31.7**\* |
| NLLB-200-3.3B | 39.5 | 46.0 | 45.5 | 46.2 |
| GPT-OSS-20B | 27.2 | 46.8 | 44.5 | 52.4 |

\* The MADLAD English-to-Armenian MQM value is affected by the count-based
aggregation of a small number of long critical spans. Its low value should
not be interpreted as strong translation quality.

## License

The model is distributed under the terms in [`LICENSE`](LICENSE). Review the
license before using or redistributing the model.