add README.md
Browse files
README.md
CHANGED
|
@@ -73,6 +73,27 @@ picking the next action for a robot: same call, same model.
|
|
| 73 |
|
| 74 |
<img src="https://huggingface.co/ironbcc/Seb-9B-v2/resolve/main/assets/di-wins-vs-jev.png" width="100%">
|
| 75 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 76 |
## Against general LLMs
|
| 77 |
|
| 78 |
<img src="https://huggingface.co/ironbcc/Seb-9B-v2/resolve/main/assets/llm-table.png" alt="Accuracy on 420 public decision rows: Claude Opus 0.855, Claude Sonnet low effort 0.817, Seb-9B-v2 0.798, Claude Haiku medium effort 0.795, gpt-6-luna high 0.779" width="100%">
|
|
|
|
| 73 |
|
| 74 |
<img src="https://huggingface.co/ironbcc/Seb-9B-v2/resolve/main/assets/di-wins-vs-jev.png" width="100%">
|
| 75 |
|
| 76 |
+
## Against Cloudflare's clef family
|
| 77 |
+
|
| 78 |
+
Decision Index 0.3 public index, per area (chance-corrected, higher is better). Seb-9B-v2 is our full run on the 0.3 kit (self-run); the clef numbers are the live board's (2026-10-22).
|
| 79 |
+
|
| 80 |
+
| | **Seb-9B-v2** (9B) | clef-flash (9B) | clef-omni (30B-A3B) | clef (27B) |
|
| 81 |
+
|---|---|---|---|---|
|
| 82 |
+
| **Public index** | **59.16** | 56.15 | 56.54 | 61.71 |
|
| 83 |
+
| Knowledge & reasoning | 39.9 | 53.0 | **53.8** | 52.8 |
|
| 84 |
+
| Language understanding | **62.2** | 47.2 | 49.8 | 61.3 |
|
| 85 |
+
| Retrieval & classification | **63.5** | 52.2 | 59.3 | 62.8 |
|
| 86 |
+
| Tools & automation | **83.1** | 81.9 | 72.6 | 80.7 |
|
| 87 |
+
| Arts & human taste | **48.9** | 48.4 | 46.4 | **48.9** |
|
| 88 |
+
| Benchmarks where Seb is ahead (of 37) | | 15 | 15 | 7 |
|
| 89 |
+
| Board Full score (includes private tests) | not scored | 44.43 | 45.04 | 54.92 |
|
| 90 |
+
|
| 91 |
+
- **Where Seb-9B-v2 is ahead of clef-flash:** POP909 (79.2 vs 0.7), RAGTruth (64.3 vs 0), HoVer (76.2 vs 22.4), iSarcasmEval (45.0 vs 0), VAST (61.2 vs 24.4), When2Call (88.1 vs 53.8), CLINC150 (93.1 vs 66.7), GSM8K, NLI4CT, ANLI, BRIGHT, BBH, BANKING77, ToolRet and ContractNLI.
|
| 92 |
+
- **Against clef-omni:** the same group plus Home appliances (80.7 vs 68.2). **Against clef (27B):** POP909, HoVer, When2Call, VAST, iSarcasmEval, RAGTruth and ContractNLI (78.0 vs 73.0).
|
| 93 |
+
- **Where the clef models are ahead:** knowledge and reasoning (CRUXEval, MuSR, CLadder, WinoGrande among the largest gaps), and clef (27B) is ahead overall (61.71 vs 59.16).
|
| 94 |
+
- **How to read the 56.15 gap:** most of Seb-9B-v2's lead over clef-flash comes from three benchmarks where clef-flash scores at or near chance (POP909, RAGTruth, iSarcasmEval). The public benchmarks can be trained on, which is why the board's headline is the Full score with private tests. Seb-9B-v2 has not been scored on those.
|
| 95 |
+
- **Speed is not compared here:** the board measured clef-flash at 53.4 ms median on its own request mix and Seb-9B-v2 was measured on short rows (21.7 ms), so the two are not like for like.
|
| 96 |
+
|
| 97 |
## Against general LLMs
|
| 98 |
|
| 99 |
<img src="https://huggingface.co/ironbcc/Seb-9B-v2/resolve/main/assets/llm-table.png" alt="Accuracy on 420 public decision rows: Claude Opus 0.855, Claude Sonnet low effort 0.817, Seb-9B-v2 0.798, Claude Haiku medium effort 0.795, gpt-6-luna high 0.779" width="100%">
|