Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
savoji 
posted an update 5 days ago
Post
104
SO-Arena scored 21 System One models, and the most accurate one ranks only seventh on reliability.

System One models pick from fixed options in one forward pass, so teams use them for routing, tool choice and verification. A leaderboard score does not say whether the pick survives shuffled options or a missing right answer. SO-Arena (2,919 tasks, 11 domains) measures that separately.

🔍 What's new
• Reliability R = sqrt(S*N). S is stability over 12 option reorderings. N = 2AUC-1 on spotting that the correct option was removed
• JevAny-27B is the most reliable System One model yet ranks eleventh in capability
• 14 of 18 models reach S above 0.8, but N runs from near chance to about 0.8

📊 The numbers
• GPT-6 Sol (82.3) and Luna (79.7) lead. The best System One models, GLiDE and KnowLine-31B, score 70.2
• Open-weight System One models answer within 0.3 s per task on one H100
• Knowledge, sentiment, retrieval: comparable to Luna, with little gain from a bigger backbone or thinking
• Math: thinking lifts StartLux Auto from 50.8 to 92.5. In intent, safety, tools and web, the larger backbone helps far more

⚠️ Caveats
• The backbone gain mixes size with Qwen3.5 vs Qwen3.8. Thinking is tested on one model
• Two closed System Two references only, and no independent replication yet

Before adopting a model, check per-domain scores, then reorder options and drop the correct one on your own task. The MIT repo ships the scoring and reliability tooling.

SO-Arena: Benchmarking the Capability Spectrum and Reliability of System One Models (2611.15777)
https://github.com/holdrain/SO-Arena

Do you test decision models for order sensitivity before putting them in an agent?

📚 Read it elsewhere
• The deep dive on Medium: https://medium.com/p/e923d2f58e85
• The long read on Substack: https://sajjadpakdamansavoji.substack.com/p/so-arena-finds-the-most-accurate
• The thread on X: https://x.com/saavoji/status/2121665100981444943
• Every version on sajjadpsavoji.github.io: https://sajjadpsavoji.github.io/blog#so-arena-finds

R = sqrt(S*N) makes stability and discrimination substitutable: S=1.0, N=0.25 and S=0.5, N=0.5 both score 0.5. For routing, those failures differ — a stable model that can't spot a missing option will pick confidently when it should abstain. Report the pair, or weight N by the decision cost.

·

Fair point. Both of your examples give 0.5, so R alone hides which failure you have. The reliability figure in the README plots S and N next to R, so you can read the pair there. For routing I would look at N first, since a stable model that never flags a missing option is the risky one.