XERON-0.2 🎯

XERON-0.2 is the second release of the XERON family: a typed-decision (System 1) model forked from XERON-0.1 (itself a fine-tune of convaiinnovations/laya's multilingual checkpoint, backbone jhu-clsp/mmBERT-base, 322M params).

It answers typed questions over a state — choice / score / noul (boolean) — in a single forward pass, returning calibrated probabilities. It never generates text, so it cannot hallucinate and cannot emit malformed schemas.

What changed vs 0.1: XERON-0.2 adds a fine-tuning pass over JevBench-style decision data — 54,000 sequences from the Jevify jev-bench corpus (22 public datasets reformatted into System One questions: emotion, NLI, MCQ, sentiment/rating rubrics, grounded yes/no, paraphrase, toxicity, spam, implicit reasoning) plus the English typed-decisions set. Training: 2 epochs · ~0.7 h · 1×A100 40GB (bf16) · post-hoc temperature calibration [0.96, 1.098, 0.569].

📈 Results

JevBench v1.3 (public items only, matched subset)

JevBench's frozen set has 534 decisions; only 231 are public (the judge tier is entirely held out), so these are not directly comparable to the published board ranks. Every system below ran the same 231 items with the official harness (fstandhartinger/jevbench, laya_local adapter); Jev/Laya rows come from the benchmark's own per-task artifact.

시스템	easy (48)	standard (72)	hard (111)	전체 (231)	Intelligence
Jev 1.13.0 (TypeSafe, API)	1.000	0.986	0.730	0.866	82.2
XERON-0.2 (ours)	0.979	0.583	0.324	0.541	34.1
laya-typed-decisions (Convai, 421M, tuned)	0.979	0.653	0.270	0.537	38.0
XERON-0.1 (ours)	0.875	0.444	0.306	0.468	23.3
laya-multilingual (base, untuned)	0.896	0.403	0.324	0.468	21.5

XERON-0.1 → XERON-0.2

항목	0.1	0.2	Δ
JevBench overall (231)	0.468	0.541	+7.3 %p
standard tier	0.444	0.583	+13.9 %p
easy tier	0.875	0.979	+10.4 %p
hard tier	0.306	0.324	+1.8 %p
typed-decisions test (400 cases)	0.700	0.713	+1.3 %p
Brier (typed-decisions)	0.449	0.424	−0.025
score MAE (typed-decisions)	0.421	0.373	−0.049
hard-tier ECE	0.211	0.187	−0.024
hard-tier prob. fidelity (TVD)	0.389	0.465	+0.076 (악화)

• XERON-0.2 now edges out the vendor's tuned Laya checkpoint on overall JevBench accuracy (0.541 vs 0.537) and beats it on the hard tier (0.324 vs 0.270), while remaining 322M multilingual rather than 421M English-only.
• Trade-off: training on largely single-label corpora sharpens the model, which improves ECE but worsens fidelity to exact gold distributions (TVD 0.389 → 0.465). If you need soft, human-disagreement-shaped distributions, prefer XERON-0.1 or add soft-label data.

typedd-decisions benchmark

LocalLLaMA/typed-decisions test split, 400 cases / 1,400 decisions, GPU:

모델	choice acc	soft acc	Brier	ECE	score MAE
XERON-0.2	0.7133	0.5353	0.4242	0.2104	0.3725
XERON-0.1	0.7000	0.5171	0.4493	0.2143	0.4213
laya-typed-decisions	0.7333	0.4460	0.4669	0.2380	0.2963

🚀 사용법

pip install laya

import laya

agent = laya.load("PIXELZX/XERON-0.2")

state = "Policy: refunds require a receipt and purchase within 30 days. A customer bought 12 days ago but has no receipt."
questions = {
    "permitted": {"type": "noul", "instructions": "Under the stated policy, is the requested action permitted?"},
    "urgency":   {"type": "score",  "levels": ["0 — no pressure", "1 — routine", "2 — elevated", "3 — critical"]},
}

print(agent.predict(state, questions)["answers"])

📈 재현

git clone https://github.com/PIXELZX0/XERON && cd XERON
# JevBench 공개 231건 (동일 하네스·어댑터)
git clone --depth 1 https://github.com/fstandhartinger/jevbench /tmp/jevbench
python results/jevbench-public/run_public_jevbench.py PIXELZX/XERON-0.2 XERON-0.2 /tmp/jb02
python results/jevbench-public/score_public_jevbench.py /tmp/jb02 XERON-0.2
# typed-decisions
python scripts/evaluate.py --model PIXELZX/XERON-0.2 --split test --device cuda --output eval.json

⚠️ 한계

• 4,096 토큰 학습 — 초장문은 CTX_CAP 확장 후 재학습 필요.
• 선택지 개수 상한: Laya 계열 헤드는 head_max_len=256이라 선택지가 많은 태스크(예: 77/151개 intent)는 옵션 텍스트가 잘려 성능이 급락한다. 학습 데이터에서도 그런 config는 제외했다.
• JevBench hard tier(0.324)와 Jev(0.730)의 격차는 여전히 크다. hard tier는 장문 정책 문서·모호한 트레이드오프· 함정 문항 위주로, 우리 학습 데이터에 그런 저작(authored) 루브릭 데이터가 거의 없다.
• 확률 충실도(TVD)가 0.1보다 나빠졌다 — soft label 데이터를 섞으면 개선 여지가 있다.

Built on Laya by Convai Innovations (Apache-2.0) and jhu-clsp/mmBERT-base. Data: Jevify jev-bench (mixed licenses, see its manifest).