jonasaise/laya-swedish-preference-gate

Laya Swedish Preference Gate (v4.1)

A ~10 ms judge for Swedish text. One forward pass of a fine-tuned Laya decision model (mmBERT-base backbone) answers one of five questions, as a "System 1" filter in front of a slow LLM judge.

Question	Input	Output
preference	prompt (or English source) + two responses	calibrated P(response A is better)
naturalness	two responses (prompt optional)	P(A is more natural, idiomatic Swedish)
quality	prompt + one response	1–5 score
natural_swedish	one response	P(natural Swedish vs translationese)
grammatical (new)	one text	P(grammatically correct); compare two texts, see limitations

Earlier releases: revision="v4", revision="v3", revision="v2", revision="v1".

Code, the full ablation and every number below: github.com/joenaess/laya-swedish-preference-gate. Training and evaluation pairs: jonasaise/laya-swedish-gate-v4-data.

What changed from v3

v4 and v4.1 add training data chosen in a controlled ablation (same recipe, seed and hyperparameters; only the data differs; all tests held out):

• DaLAJ-GED (Superlim 2): learner sentences vs the same sentence corrected, for naturalness and the new grammatical question.
• SweFAQ (Superlim 2): a question's own answer vs other answers from the same FAQ page, for preference.
• Europarl (Språkbanken): professional Swedish translations vs Kimi-k3 machine translations of English speeches, plus adequacy negatives (another paragraph's translation, a half translation, a word-shuffled translation, the untranslated English).
• v4.1 only: correct text from four registers against corrupted copies (a word deleted or two neighbours swapped), and wrong-source negatives from the adjacent paragraph of the same debate, to reduce grammatical over-flagging and fluent-but-wrong translations winning.

Held-out test (share where the reference wins)	v3	v4	v4.1
Fluent-vs-fluent probe, preference	79.9%	78.6%	79.3%
DPO validation, preference	98.9%	98.7%	98.8%
liu-nlp native vs OPUS-MT, naturalness	44.3%	52.2%	51.0%
FLORES+ human vs OPUS-MT, preference with source	48.5%	73.8%	71.9%
WMT24++ human vs OPUS-MT, preference with source	54.3%	74.6%	72.8%
WMT24++ human vs Kimi-k3, preference with source	43.1%	58.9%	59.8%
DaLAJ corrected vs learner, naturalness	55.9%	89.7%	88.0%
GerLangMod-sv original vs verb misplaced, naturalness	83.3%	95.5%	95.7%
SweFAQ own answer vs other, preference	60.3%	88.9%	88.6%
BLiMPPI-sv polarity items, naturalness	40.7%	86.4%	89.8%
MultiBLiMP-sv agreement, naturalness	83.1%	97.0%	97.5%
ScaLA-sv MCC (EuroEval's metric), single-response	0.146	0.341	0.466

Calibration: preference T = 1.674, naturalness T = 0.485, quality T = 1.226, natural_swedish T = 1.495, grammatical T = 1.427. ECE on the DPO validation pairs 0.0164.

A trap we hit on the way. Human-vs-MT pairs alone teach a gate that "the candidate least like the source wins": the intermediate arms preferred a fluent translation of the wrong sentence, half translations and nonsense. The adequacy negatives fix it (FLORES+ devtest, English source in the prompt, share where the reference wins):

The reference against …	v3	without adequacy negatives	v4	v4.1
the reference of a different sentence	91.0%	0.0%	78.0%	89.5%
LFM2-350M's translation (pilot's lowest chrF)	97.0%	3.5%	98.0%	97.5%
only the first half of the reference	72.0%	7.5%	100.0%	100.0%
the untranslated English	20.5%	100.0%	100.0%	100.0%
the untranslated English (naturalness, no prompt)	87.5%	38.5%	99.5%	99.5%

On 5600 translations by 28 LLMs, v4.1's translation score agrees with chrF at Spearman 0.38 per translation (v3: 0.30).

Pilot: a quick proxy for EuroEval Swedish

Scoring 100 Swedish answers per model with quality and averaging, over 28 open LLMs with Swedish results on the EuroEval leaderboard: Spearman with the Swedish rank 0.66 [0.33, 0.85], 0.63 [0.25, 0.81] after controlling for model size. A pilot (models up to 11B parameters, 6 of them in 4-bit), not a calibrated predictor.

Quickstart

uv pip install "git+https://github.com/joenaess/laya-swedish-preference-gate"

from laya_gate.predictor import SwedishPreferenceGate

gate = SwedishPreferenceGate("jonasaise/laya-swedish-preference-gate")

r = gate.predict_preference(
    prompt="Hur fungerar fotosyntesen?",
    chosen_candidate="Fotosyntesen är den process där växter omvandlar ljus till energi.",
    rejected_candidate="Det är när solen lyser på träd och gör magi.",
)
# {"preference": "candidate_a", "p_chosen": <float>, "is_valid_margin": <bool>, ...}

# translation quality: put the English source in the prompt
t = gate.predict_preference(
    prompt="Översätt följande text till svenska:
The meeting was postponed.",
    chosen_candidate="Mötet sköts upp.",
    rejected_candidate="Mötet ställdes in.",
)

s = gate.score_quality(prompt="Hur fungerar fotosyntesen?", response="Växter gör mat av solljus.")
g = gate.score_grammar("Människor är snälla där och alla varor är billiga.")
a, b = (gate.score_grammar(x)["p_correct"] for x in ("Hon har bott här.", "Hon har bo här."))

Limitations

• grammatical over-flags as a stand-alone verdict. At 0.5 it marks 25.2% of fluent LLM answers and 47.9% of WMT24++ references as incorrect (it was trained on single learner sentences). Compare two versions of a text instead. On lowercased text without punctuation (GerLangMod-sv) it is at chance (86.3%); v4 was at 47.8%.
• Wrong-content translations still win sometimes: v4 prefers a fluent translation of a different sentence in 10.5% of FLORES+ pairs (v3: 9.0%).
• Answer preference largely tracks length. On 129 human Chatbot Arena votes on Swedish answers, preference agrees with people in 58.9% of battles, close to "the longer answer wins" (64.3%); Kimi-k3 reaches 74.8%. Use the gate for Swedish form (grammar, translationese, translation adequacy, relevance), not to pick between two fluent answers.
• Pairwise preference, not fact-checking; 1,024-token window; one seed per training arm.
• Kimi-k3 is not a neutral reference for its own output: on WMT24++ it prefers the human translation over OPUS-MT in 81.0% of pairs but over its own translation in only 24.5%.

Training data and licenses

Weights: Apache-2.0. Base model convaiinnovations/laya: apache-2.0. Training data:

• jonasaise/dpo-swedish-interpreted (mit) and argilla/ultrafeedback-binarized-preferences-cleaned (mit).
• Europarl Swedish-English: Koehn (2005), Europarl: A Parallel Corpus for Statistical Machine Translation, MT Summit X; enriched and distributed by Språkbanken, https://doi.org/10.23695/mdp4-2p72, CC BY 4.0.
• DaLAJ-GED (CC BY 4.0) and SweFAQ (CC BY-SA 4.0) from Superlim 2 (sbx/superlim-2), Språkbanken.
• Machine translations by Helsinki-NLP/opus-mt-en-sv (evaluation only) and Kimi-k3.

Evaluation only: liu-nlp/translationese-opensubtitles (Apache-2.0), FLORES+, WMT24++ (Apache-2.0), ScaLA-sv (mteb/scala_sv_classification), GerLangMod (Apache-2.0), BLiMPPI (arXiv:2610.13607; Swedish subset from UD SweLL/Talbanken CC BY-SA 4.0 and LinES CC BY-NC-SA 4.0), MultiBLiMP (jumelet/multiblimp, CC BY 4.0).

@software{laya_swedish_preference_gate,
  title  = {Laya Swedish Preference Gate},
  author = {Lind, Jonas},
  year   = {2026},
  url    = {https://huggingface.co/jonasaise/laya-swedish-preference-gate}
}