Seb-9B vs Jev 1.13 vs Laya on the 17 public suites from Laya's benchmark report (identical questions, self-run)
Disclosure: I built Seb-9B (https://huggingface.co/ironbcc/seb-9b), so this is a self-run comparison, not an independent one.
Setup. I rebuilt the test cases from Laya's benchmark report with Laya's own sampling code. The rebuilt cases match Laya's case for case, and my runs of this checkpoint reproduce its published numbers. Every model answered byte-identical questions.
- Laya: the default English checkpoint at the repo root, no router.
- Jev: TypeSafe Jev 1.13.0 through its API.
- Seb: Seb-9B, one question per request.
Accuracy (400 rows per suite unless noted):
| Suite | Seb-9B | Jev 1.13 | Laya |
|---|---|---|---|
| Typed decisions: 4 business workflows (2,000 decisions) ¹ | 0.753 | 0.733 | 0.361 |
| AG News (4 topics) | 0.885 | 0.863 | 0.950 |
| DAIR Emotion (6 labels) | 0.590 | 0.595 | 0.595 |
| banking77 (gold + 19 candidates) | 0.907 | 0.920 | 0.765 |
| Email spam | 0.950 | 0.970 | 0.993 |
| Phishing | 0.863 | 0.900 | 0.980 |
| Jailbreak detection | 0.868 | 0.943 | 0.708 |
| Toxicity moderation | 0.807 | 0.660 | 0.530 |
| RAG passage relevance | 0.627 | 0.613 | 0.625 |
| Support triage (10 queues) | 0.378 | 0.362 | 0.502 |
| Model routing (math / code / news) | 0.945 | 0.975 | 0.639 |
| MASSIVE intent (20 options) | 0.943 | 0.930 | 0.783 |
| MASSIVE scenario | 0.780 | 0.753 | 0.603 |
| XNLI (English) | 0.890 | 0.870 | 0.860 |
| SST-5 sentiment (5 levels) | 0.630 | 0.578 | 0.372 |
| BoolQ | 0.893 | 0.917 | 0.830 |
| Prompt-injection detection | 0.759 | 0.776 | 0.698 |
| Mean, 17 suites | 0.792 | 0.786 | 0.694 |
| Mean, 16 suites (without typed decisions) | 0.795 | 0.789 | 0.715 |
| Mean ECE, 17 suites (lower is better) | 0.088 | 0.124 | 0.170 |
Answer changes when the options are shuffled (intent / emotion / NLI): Seb 0.030 / 0.055 / 0.005, Jev 0.015 / 0.035 / 0.005, Laya 0.150 / 0.040 / 0.000.
Reading it:
- Seb and Jev are close on mean accuracy (+0.006). I haven't run a significance test on that gap, so I wouldn't call either one ahead on accuracy from this table. Seb's calibration error is lower.
- Laya leads on AG News, spam, phishing and support triage. AG News, spam and phishing are in Laya's training mix. Jev leads on jailbreak detection, model routing, BoolQ, banking77 and prompt injection.
- Support triage is weak for all three models (0.36–0.50).
Caveats:
- ¹ Seb's training data included the typed-decisions benchmark's own
trainsplit (not its test cases), so that row is not zero-shot for Seb. Laya's checkpoint fine-tuned on that split, laya-typed-decisions, scores 0.766. The 16-suite mean leaves the row out. - Seb's choice questions take at most 20 options. On banking77 with all 77 labels in one question, Seb needs two steps (0.820, vs Jev 0.843 and Laya 0.425 answering all 77 at once), so that row isn't in the means.
- English only. Laya's multilingual checkpoint and its router aren't tested here.
Seb's image results and its own sealed-suite comparison with Jev are on the Seb card. If anyone reruns these cases independently, I'd like to see where the numbers differ.