# Competitive roadmap — accurate status vs competition

**Law:** \(S=K(T_1+T_2+T_3)\) pin **D1D38A** (unchanged; never fitted to BLEU).  
**Measured:** 2026-07-21 SOTA push (`m6_sota_push_report.json` / v0.2.0 metrics).

---

## 1. Two competitions (do not mix them)

| Arena | Competitor definition of “win” | PFLT status |
|-------|--------------------------------|-------------|
| **A. Inventory / form→gloss** | Coverage + sense under offline densify | **Winning our catalog** (~99.99% on 113 langs) |
| **B. Sentence MT** | sacreBLEU / chrF / COMET on held-out sentences | **Chat: competitive**; **News: trailing DeepL-class** |

Google/DeepL/NLLB are judged almost entirely on **B**.  
We already win parts of **A** (especially classical / offline / FSOT). Beating them “accurately” means closing **B** without lying about A.

---

## 2. Scoreboard (measured)

### A — Catalog

| Metric | Us | Google-ish | DeepL-ish | NLLB-ish |
|--------|---:|-----------:|----------:|---------:|
| Lang count | **113** | ~249 | ~30–100 | ~200 |
| Form→gloss on our catalog | **~99.99%** | N/A (not their product metric) | N/A | N/A |
| Classical / dead / hieroglyph | **Strong** | Weak | Weak | Weak |
| FSOT law pin | **Yes** | No | No | No |

**Verdict A:** Competitive / leading on **our** product definition. Not “more languages than Google.”

### B1 — Chat / easy parallel (Tatoeba-style open-set)

| System path | sacreBLEU (mean) | Notes |
|-------------|-----------------:|-------|
| Neural best-of (opus/mul/NLLB) | **50.19** | Fair open-set |
| Hybrid oracle densify\|neural | **53.58** | Product upper bound |
| Staged internal bar | 45 | **Passed** |
| Strong open MT chat (rough) | ~45–65+ | We sit mid/high |

**Per-lang neural highs:** it 68 · es/pt 61 · de/ru ~57–58 · hi 55  
**Gaps:** ja 37 · zh 32 · la neural 13 (densify wins classical)

**Verdict B1:** **Competitive** on chat open-set. Not SOTA vs every commercial pair, but past mid bar.

### B2 — News (WMT14 de→en test, n=3003) — the hard public bar

| System | sacreBLEU | Gap to 40 mid | Gap to 48 stretch |
|--------|----------:|--------------:|------------------:|
| Densify-only | ~0.4 | — | — |
| **opus-mt-de-en beams=5** | **33.88** | **−6.1** | **−14.1** |
| NLLB-600M beams=5 | 33.37 | −6.6 | −14.6 |
| DeepL-class mid (staged) | ~40 | 0 | — |
| Strong commercial stretch | ~45–55 | — | 0 |

**Verdict B2:** **Near mid open-MT**, **not** DeepL-class yet. This is the main gap if “beat the competition” means news fluency.

---

## 3. Stage ladder (honest)

```text
S0 Dict          ████████████ DONE
S1 Phrase        ████████████ DONE
S2 Chat strong   ████████████ DONE  (neural ~50)
S3 Mid news      ████████░░░░ HERE  (~34 / need ~40)
S4 DeepL-class   ████░░░░░░░░       (need ~45–55)
```

---

## 4. What is left to be competitive / beat them (priority order)

| # | Lever | Closes | Effort | Expected gain |
|---|-------|--------|--------|----------------|
| **1** | **Ship hybrid router** (densify short/classical; neural long/news/CJK) | Product accuracy vs single-path | Low | Realizes measured **~53.6** chat hybrid; better UX |
| **2** | **WMT student finetune** (opus or NLLB on WMT train; law fixed) | News −6 gap | Med–High | Often **+2–6+** sacre if done right |
| **3** | **Stronger student** (NLLB-1.3B / more beams / length / ensembling) | News + chat CJK | Med | **+0.5–3** sacre (varies) |
| **4** | **CJK order** always neural SPM path (no EN-dep densify for ja/zh) | ja/zh chat | Low | Lift ja/zh from ~32–37 toward EU band |
| **5** | **FLORES** same-file public bar | Comparable claim vs NLLB papers | Blocked until Hub access | Credibility, not training |
| **6** | **Breadth** more Kaikki langs toward 150–200 | Catalog vs Google count | Med | Count game only |
| **7** | **COMET / human** spot checks | Beyond BLEU honesty | Med | Avoid BLEU-only overclaim |

### What will **not** beat DeepL alone

- More form→gloss densify on chat templates (product ceiling ≠ news SOTA)  
- Refitting the FSOT law scalar to BLEU (forbidden / false)  
- Claiming product densify BLEU-4 ~83 as open-set MT

---

## 5. Definition of “we beat the competition” (accurate)

Pick a claim level:

| Claim level | Criteria | Distance now |
|-------------|----------|--------------|
| **L1 Product unique** | Offline FSOT + classical + catalog depth | **Met** |
| **L2 Chat competitive** | Mean chat sacre ≥45 open-set multi-lang | **Met** (~50) |
| **L3 News mid-parity** | WMT14 de-en sacre ≥40 | **~6 pts short** |
| **L4 News strong** | WMT14 de-en sacre ≥45–48 | **~11–14 pts short** |
| **L5 Broad DeepL-class** | Many pairs FLORES/WMT mid-40s+ | **Not started** (FLORES gated; multi-pair news unrun) |

**Accurate one-liner:**  
We **beat / own** unique offline+classical+law. We **match mid open-MT on chat**. We **do not yet beat** commercial systems on **news full-sentence fluency** — need roughly **+6 sacre** for mid-parity and **+14** for stretch SOTA on de→en.

---

## 6. Immediate execution plan (this push)

1. Implement **product hybrid router** + measure non-oracle (feature rules, not ref peeking)  
2. **WMT decode push**: higher beams, dual-system sentence-level ensemble upper bound  
3. Write measured gaps into `reports/COMPETITIVE_PUSH.md`  
4. Keep law pin D1D38A fixed  

Optional next session: WMT finetune loop under densify law wrap.

---

## 7. Measured this push (2026-07-22 competitive run)

| Track | Result | Notes |
|-------|-------:|-------|
| Product hybrid chat sacreBLEU | **40.64** | No ref peeking; densify-heavy (2800) + neural CJK (400) |
| WMT opus-mt beams=8 | **33.79** | Slightly under prior beams=5 33.88 |
| WMT NLLB-600M beams=8 | **33.49** | |
| **WMT oracle dual ensemble** | **37.61** | Best of opus/nllb per sentence (upper bound) |
| Gap ensemble → mid 40 | **2.39** | Was ~6.1 for single student |
| Gap ensemble → stretch 48 | **10.39** | Still needs finetune / larger model |

**Insight:** Beams alone do not close news. **Dual-student ensemble** recovers ~+3.7 sacre toward bar 40 without finetune. Remaining ~2.4 points need quality model upgrade or WMT finetune.

---

## 8. Beat levers run (2026-07-22)

| Lever | Result | Clears bar? |
|-------|-------:|:------------|
| L2 Neural-first hybrid chat | **48.74** sacre | **Yes** (≥45) |
| L1 Product NLL ensemble WMT | **34.54** | No (gap 5.46 to 40) |
| L1 Oracle ensemble upper | **37.13** | No (gap 2.87) |
| L3 Aggressive FT | 32.2 | No (regressed) |
| L3b Safe FT freeze-enc | 33.86 | No (flat) |
| Base opus-mt-de-en | 33.88 | No |

**Still required to beat news mid-parity (40):** better ensemble selection (~2.6 pts headroom to oracle), multi-epoch WMT FT with validation early-stop, or larger NLLB.

---

## 9. News push v3 (2026-07-22)

| System | sacreBLEU |
|--------|----------:|
| Product cross-NLL ensemble | **34.11** |
| Base opus | 33.88 |
| FT v3 test / best val | 33.41 / 35.76 |
| Oracle upper | 37.13 |
| Gap product → 40 | **5.89** |

Chat neural-first hybrid remains **48.74** (mid bar met).