README.md

---
license: apache-2.0
---

# Lumen — WMT26 General MT Submission

This repository contains **Lumen**, a submission to the **WMT 2026 General Machine
Translation Shared Task**.

Lumen is a 14.8B-parameter dense translation model built on the Qwen3-14B architecture.
It covers all 23 official WMT26 translation directions, including several
low-resource and regional varieties (Ligurian, Ladin, Northern Sámi, Egyptian Arabic).

At 14.8B parameters the model is eligible for the WMT26 **constrained** track, which
requires open-weight models below 20B parameters.

## Model details

| | |
|---|---|
| Architecture | `Qwen3ForCausalLM` (dense, 40 layers, GQA with 40 Q / 8 KV heads) |
| Parameters | 14.77 B |
| Hidden size / FFN | 5120 / 17408 |
| Vocabulary | 152064 (Qwen2 BPE tokenizer) |
| Weights dtype | bfloat16, single `model.safetensors` (~29.5 GB) |
| Context length | **4096 tokens** (`max_position_embeddings`) |
| RoPE theta | 10000 |

## Supported language pairs

All 23 official WMT26 General MT directions:

| Direction | Languages | Direction | Languages |
|---|---|---|---|
| en→zh | English → Chinese (Simplified) | en→is | English → Icelandic |
| en→zh-Hant-TW | English → Chinese (Traditional, Taiwan) | en→id | English → Indonesian |
| en→cs | English → Czech | en→kk | English → Kazakh |
| en→de | English → German | en→be | English → Belarusian |
| en→ru | English → Russian | en→hy | English → Eastern Armenian |
| en→uk | English → Ukrainian | en→arz | English → Egyptian Arabic |
| en→ja | English → Japanese | en→lij | English → Ligurian (Italy) |
| en→ko | English → Korean | en→lld | English → Ladin (Val Badia, Italy) |
| en→et | English → Estonian | en→se | English → Northern Sámi |
| en→th | English → Thai | cs→de | Czech → German |
| zh→ja | Chinese (Simplified) → Japanese | cs→uk | Czech → Ukrainian |
| | | cs→vi | Czech → Vietnamese |

## Repository layout

```
.
├── model/                       # HF-format weights (upload target for Hugging Face)
│   ├── config.json
│   ├── generation_config.json
│   ├── chat_template.jinja
│   ├── special_tokens_map.json
│   ├── tokenizer.json
│   ├── tokenizer_config.json
│   └── model.safetensors        # 29,539,287,712 bytes, 443 tensors, bf16
├── inference/
│   └── run_eval_ruled_check.py  # submission decoding pipeline
├── requirements.txt
└── README.md
```

## Prompt format

The model expects a `Human:` / `Assistant:` template (shipped as
`model/chat_template.jinja`). A rendered prompt looks like:

```
Please translate the following text into {tgt_lang}:
#guidelines
{guidelines}
#src_text
{src_text}
```

The `#guidelines` block carries the per-domain instructions supplied with the WMT26
test sets (register, HTML preservation, handling of URLs and hashtags, and so on).

## Usage

### 1. Dependencies

```bash
pip install -r requirements.txt
```

### 2. Minimal example (vLLM)

```python
from transformers import AutoTokenizer
from vllm import LLM, SamplingParams

model_path = "model"  # local path, or the Hugging Face repo id once published

tokenizer = AutoTokenizer.from_pretrained(model_path)
llm = LLM(model=model_path, max_model_len=4096)

lang_name = {
    "zh": "chinese", "zh-tw": "traditional chinese", "cs": "czech", "de": "german",
    "uk": "ukraine", "vi": "vietnamese", "ja": "japanese", "ko": "korean",
    "ru": "russian", "th": "thai", "id": "indonesian", "et": "estonian",
    "is": "icelandic", "kk": "kazakh", "be": "belarusian", "hy": "armenian",
    "arz": "egyptian arabic", "lij": "ligurian", "lld": "ladin", "se": "northern sami",
}

source_text = "This paper presents the Lumen system, our submission to the WMT 2026."
target = "zh"

content = (
    f"Please translate the following text into {lang_name[target].title()}.\n#guidelines\n"
    f"#src_text\n{source_text}"
)
prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": content}],
    tokenize=False,
    add_generation_prompt=True,
)

outputs = llm.generate(
    [prompt],
    SamplingParams(temperature=0.0, top_p=0.001, top_k=1, max_tokens=2048),
)
print(outputs[0].outputs[0].text.strip())
```

### 3. Reproducing the submission pipeline

`inference/run_eval_ruled_check.py` is the decoding pipeline used for the submission.
It goes beyond plain greedy decoding:

1. **Structure-aware splitting.** Each source is classified as `html`, `json` or
   `plain`. HTML text nodes and translatable attributes (`alt`, `title`,
   `placeholder`, `aria-label`) and JSON string leaves are translated
   independently, then the original structure is rebuilt. Long plain text is split
   into sentence-aware chunks.
2. **Rule-based quality checks.** Every output is screened for empty results,
   length-ratio outliers, character-level decoder loops, repeated n-grams,
   insufficient target-script coverage, and source copying.
3. **Anti-loop retries.** Flagged samples are regenerated with
   `repetition_penalty=1.3` and `frequency_penalty=0.5` rather than a higher
   temperature, which was empirically ineffective against character-level runaway
   (`"оооо…"`, `"ііі…"`). The better of the two candidates is kept.
4. **Structural validation.** Rebuilt HTML is compared against the source tag-count
   signature and JSON against the source shape, so structure-breaking outputs are
   rejected.

Input is a JSONL file where each line provides `prompt`, `src`, `src_lang` and
`tgt_lang`:

```bash
python inference/run_eval_ruled_check.py \
    --model model \
    --input  testset/en-zh.jsonl \
    --output out/en-zh.jsonl \
    --max-tokens 2048 \
    --tensor-parallel-size 1
```

Two files are written: the main output with an added `hypothesis` field (plus
`_rule_flags`, `_retried`, `_kind` diagnostics), and a `<output>.failed.jsonl`
companion listing samples that were still flagged after retrying.

## Notes and limitations

- **Context length is 4096 tokens.** The released config sets
  `max_position_embeddings=4096` with `rope_theta=10000`, so this is a hard limit
  rather than a soft recommendation. Documents longer than this must be chunked; the
  pipeline above does so automatically. `--max-tokens` must leave room for the prompt
  inside this budget.
- **Differences from stock Qwen3-14B.** All layer dimensions match the official
  Qwen3-14B (5120 hidden, 40 layers, 40 Q / 8 KV heads, 17408 FFN, untied
  embeddings), but three config fields differ: `vocab_size` is 152064 rather than
  151936, `rope_theta` is 10000 rather than 1000000, and `max_position_embeddings` is
  4096 rather than 40960. The tokenizer holds 151669 real tokens, so the vocabulary
  difference is padding only.
- **Compatibility.** `config.json` follows the transformers 5.x schema, using `dtype`
  and a nested `rope_parameters` block. transformers 4.x does not recognise these
  keys, so pass the dtype explicitly (`dtype=torch.bfloat16`) to avoid loading the
  model in fp32.
- **Target-script check on JSON.** The script-coverage rule measures the ratio of
  target-script characters over the whole output. For JSON payloads the untranslated
  ASCII keys dilute this ratio, so short dictionaries can be flagged even when the
  translation is correct. These land in `.failed.jsonl` for review rather than being
  discarded.

## License

Released under the Apache License 2.0
(https://www.apache.org/licenses/LICENSE-2.0.txt, SPDX-License-Identifier: Apache-2.0).