KIEFERSA/Sophea-OSS-v1

Sophea-OSS-v1

Greek and English reasoning fine-tune of gpt-oss-20b: harmony-channel traces, format-repaired

📄 Released with the paper Thinking in a Low-Resource Language: What SFT Builds, What RL Fixes, What Accuracy Cannot See (arXiv:2608.17744). Every number on this card is measured, contextualized, and caveated in the paper; read it for the full evaluation protocol.

• Creator: Kiefer SA (Sophea AI Lab, Athens)
• Base model: openai/gpt-oss-20b (sparse MoE transformer, 20.9B total / 3.60B active, 4-of-32 routing, MXFP4-packed experts)
• Languages: Greek and English (the reasoning trace follows the question's language)
• Decoding: thinking, always on: the base opens its analysis channel on 100% of rows even at reasoning_effort=none, and this fine-tune does not add a switch the base never had

Harmony channels, not <think>. The trace arrives in the analysis channel and the answer in assistantfinal; a scorer or client written for <think> carriers silently mis-reads this family. Split on the channel markers whenever they appear, and read the final answer from the final channel only. A detailed thinking off system message does not disable reasoning (it returns content=null with the text in reasoning).

Intended use

• Greek and English reasoning assistant with the study's best instructed trace-language steerability (62.5% el→EN, the highest of the five releases; 93.3% en→EL)
• Deployments that read the trace: it is on-language at 98.1% and never switches mid-question

Two family costs to weigh: the answer-channel leak is 10.3% (the trace-register boundary holds at different tightness per family; Qwen sits at 3.5%, Nano at 0.0%), and serving is more expensive than the base (see the budget note below). Not evaluated for safety-critical, legal, or medical use.

Fine-tuned from gpt-oss-20b, then format-repaired; the pre-repair arm is reported as reference only.

---

Evaluation

Arrows mark the desirable direction for every metric: ↑ higher is better, ↓ lower is better.

Headline scorecard

Axis	gpt-oss-20b (base)	Sophea-OSS-v1
Greek-trace fidelity, Greek questions (ratio ≥ 0.9, n=5,156) ↑	0.0% (0/5,156)	98.1%
English traces on English questions ↑	100%	100% (measured on the pre-repair arm)
In-question language switches / 100 sentences ↓	9.7	0.0
Answer-channel leak ↓	0.5% (English control)	10.3%
Answer-format fallback, Greek think lane ↓	0.7–5.2% per domain	26% (70% before the format repair)
Instructed override, el→EN / en→EL ↑	n/a (already reasons in English)	62.5% / 93.3%
Median trace length, tokens (own tokenizer) ↓	396	640 (1.6× more)
Greek NLU macro, forgetting check (vs own base) ↑	61.2	58.0 (−3.2)
English NLU macro, forgetting check (vs own base) ↑	75.7	76.7 (+1.0)

Budget, disclosed honestly: the token saving changes sign on this family. The release spends 1.6× more tokens than its English-reasoning base (median 640 vs 396): its traces are already slightly longer in words than the terse base's, and Greek's 2.3–2.5× token fertility widens the difference. The word-level shortening the recipe buys elsewhere does not survive translation into tokens here.

The format story: before repair, the Greek-lane anchored accuracy read 56.2 with 70.2% of rows never emitting the requested answer line: a format floor, not a reasoning one (the traces were present and on-language). The repair dose collapses fallback 70% → 26% and recovers 4.1 of the 7.3 Greek NLU macro points the pre-repair arm had lost; English retention recovers to +1.0 above base.

Per-benchmark detail

Per-domain, Greek think lane

Domain (Greek think lane)	Base acc ↑	Pre-repair acc ↑	Sophea-OSS-v1 acc ↑	Base fallback ↓	Pre-repair fallback ↓	Sophea-OSS-v1 fallback ↓
Math (two math slices)	90.7	74.6	78.6	2.4%	34.9%	17.2%
Commonsense (HellaSwag + Winogrande)	63.2	53.7	53.3	5.2%	83.5%	25.7%
Logic (ProofWriter, macro-recall)	32.2	25.2	26.5	0.7%	77.4%	50.9%

The pre-repair fallback concentrated in the open-form domains (83.5% commonsense, 77.4% logic) while math's short numeric answer stayed format-robust (34.9%): the shape a format defect takes, not a capability loss. Conditioning on rows that emit the requested answer line, the release reads 94.4 / 58.2 / 30.0 (math / commonsense / logic) against raw 78.6 / 53.3 / 26.5.

Forgetting check: Titan-1 suite (non-reasoning mode)

This suite exists to answer one question: did the Greek fine-tune damage general ability? Against its own base, across 14 bilingual NLU benchmarks: −3.2 Greek / +1.0 English. English is fully retained (recovered above base by the format repair), and the paper localises the Greek residual to answer-format compliance rather than lost knowledge. Read the numbers below as a forgetting check, not a capability leaderboard.

Benchmark (Titan-1 suite)	Base	Sophea-OSS-v1
ARC-Challenge-el	77.1	68.6
ARC-Easy-el	85.4	79.9
Belebele-el	86.0	81.3
GreekMMLU	66.3	61.3
HellaSwag-el	42.9	52.1
Medical-MCQA-el	42.1	38.7
TruthfulQA-el	39.7	35.5
Winogrande-el	55.4	59.6
MMLU-el	55.5	45.4
Greek macro	61.2	58.0
ARC-Challenge (en)	90.0	86.8
ARC-Easy (en)	96.0	95.7
HellaSwag (en)	57.0	68.4
Winogrande (en)	64.6	66.7
MMLU (en)	70.8	65.8
English macro	75.7	76.7

Register control is flat at 39/46; grammaticality improves 32 → 40 of 58 (judge-scored counts, directional evidence only).

Usage

Serve with vLLM (OpenAI-compatible):

vllm serve KIEFERSA/Sophea-OSS-v1 --served-model-name sophea-oss-v1

Client (OpenAI SDK):

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
resp = client.chat.completions.create(
    model="sophea-oss-v1",
    messages=[{"role": "user", "content": "Ένα κατάστημα πουλά στυλό προς 1,50 € το ένα. Πόσο κοστίζουν 12 στυλό;"}],
    temperature=0.7,
)
print(resp.choices[0].message.reasoning)  # analysis channel: the trace, in the question's language
print(resp.choices[0].message.content)    # assistantfinal channel: the answer

Transformers: load with attn_implementation="eager" (this family rejects sdpa) and apply the harmony chat template via tok.apply_chat_template(...) as usual; reasoning cannot be switched off.

Steering the trace language (partial, measured per checkpoint): append "Think step by step IN ENGLISH." to a Greek question, or "Σκέψου βήμα προς βήμα ΣΤΑ ΕΛΛΗΝΙΚΑ." to an English one. This checkpoint follows the English suffix on 62.5% of Greek items (the best of the five releases) and the Greek suffix on 93.3% of English items.

License

Apache-2.0. The base model, openai/gpt-oss-20b, is released under Apache-2.0 and this fine-tune inherits those terms. Verify base-model terms before use.

Citation

This model is a release artifact of arXiv:2608.17744. If you use it, please cite:

@misc{kirouane2026thinking,
      title={Thinking in a Low-Resource Language: What SFT Builds, What RL Fixes, What Accuracy Cannot See},
      author={Ayoub Kirouane and Christos Petrocheilos},
      year={2026},
      eprint={2608.17744},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2608.17744},
}