add README.md
Browse files
README.md
CHANGED
|
@@ -61,45 +61,29 @@ picking the next action for a robot: same call, same model.
|
|
| 61 |
| `choice` | `A`…`T` (up to 20) | "Which team should handle this ticket?" · "Which action should the robot take next?" |
|
| 62 |
| `score` | `0`…`4` | "How urgent is the message?" |
|
| 63 |
|
| 64 |
-
## Results
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 65 |
|
| 66 |
-
| | **Seb-9B-v2** | Seb-9B | Jev 1.13 | Laya |
|
| 67 |
-
|---|---|---|---|---|
|
| 68 |
-
| Decision Index 0.3, public index | **59.2** | | | |
|
| 69 |
-
| Laya public suites, mean accuracy (17 suites) | **0.801** | 0.793 | 0.786 | 0.694 |
|
| 70 |
-
| Calibration error, ECE (lower is better) | 0.110 | **0.088** | 0.124 | 0.170 |
|
| 71 |
-
| Typed decisions, sealed suite (12 suites, pooled) | 93.1 | **94.6** | 94.3 | |
|
| 72 |
-
| Trading decisions suite | **0.749** | 0.732 | 0.707 | |
|
| 73 |
-
|
| 74 |
-
- **Biggest gains over Seb-9B:** BANKING77 0.973 (Seb-9B 0.907, Jev 0.920), jailbreak detection 0.940 (0.868), model
|
| 75 |
-
routing 0.980 (0.945), phishing 0.895 (0.863), MASSIVE scenario 0.833 (0.780).
|
| 76 |
-
|
| 77 |
-
## Decision Index vs hosted Jev
|
| 78 |
-
|
| 79 |
-
Decision Index 0.3 public index: 37 benchmarks in 5 areas, chance-corrected, higher is better. Seb-9B-v2 is a full run on the 0.3 kit
|
| 80 |
-
(110,520 requests); the Jev numbers are the live board's (2026-10-22).
|
| 81 |
-
|
| 82 |
-
| | Public index |
|
| 83 |
-
|---|---|
|
| 84 |
-
| **Seb-9B-v2 (9B)** | **59.16** |
|
| 85 |
-
| Jev 1.14 (hosted) | 60.70 |
|
| 86 |
-
| Jev 1.13 (hosted) | 57.96 |
|
| 87 |
-
|
| 88 |
-
| Area | **Seb-9B-v2** | Jev 1.13 | Jev 1.14 |
|
| 89 |
-
|---|---|---|---|
|
| 90 |
-
| Knowledge & reasoning | 39.9 | 53.9 | 57.1 |
|
| 91 |
-
| Language understanding | 62.2 | 59.2 | 61.2 |
|
| 92 |
-
| Retrieval & classification | 63.5 | 55.4 | 63.5 |
|
| 93 |
-
| Tools & automation | 83.1 | 75.1 | 73.8 |
|
| 94 |
-
| Arts & human taste | 48.9 | 39.1 | 39.2 |
|
| 95 |
-
|
| 96 |
-
Seb-9B-v2 is ahead of Jev 1.13 on 22 of 37 benchmarks (biggest: POP909 +63.3, HoVer +30.5, Home appliances +28.4, ContractNLI +19.0, CLadder +16.6, VAST +14.2) and
|
| 97 |
-
behind on the rest (largest gaps: GPQA Diamond -48.3, MMLU-Pro -33.2, BBH -30.7). Against Jev 1.14 it is ahead on 20 of 37.
|
| 98 |
-
The board also lists newer open entries with higher public-index scores, for example OneJev1.6-9B 65.3; entries near Seb-9B-v2's size and
|
| 99 |
-
score: Drex v1.5 (9B) 58.1, Bespoke Nimble 9B v3 57.2. Per-benchmark numbers: `di_public_index_vs_jev.json` in the demo folder.
|
| 100 |
-
|
| 101 |
-
<img src="https://huggingface.co/ironbcc/Seb-9B-v2/resolve/main/assets/di-index-vs-jev.png" width="100%">
|
| 102 |
-
<img src="https://huggingface.co/ironbcc/Seb-9B-v2/resolve/main/assets/di-areas-vs-jev.png" width="100%">
|
| 103 |
<img src="https://huggingface.co/ironbcc/Seb-9B-v2/resolve/main/assets/di-wins-vs-jev.png" width="100%">
|
| 104 |
|
| 105 |
## Against general LLMs
|
|
|
|
| 61 |
| `choice` | `A`…`T` (up to 20) | "Which team should handle this ticket?" · "Which action should the robot take next?" |
|
| 62 |
| `score` | `0`…`4` | "How urgent is the message?" |
|
| 63 |
|
| 64 |
+
## Results
|
| 65 |
+
|
| 66 |
+
| | **Seb-9B-v2** | Seb-9B (v1) | Jev 1.13 | Jev 1.14 | Laya |
|
| 67 |
+
|---|---|---|---|---|---|
|
| 68 |
+
| **Decision Index 0.3, public index** (37 benchmarks, chance-corrected) | **59.16** | | 57.96 | 60.70 | |
|
| 69 |
+
| Knowledge & reasoning | 39.9 | | 53.9 | 57.1 | |
|
| 70 |
+
| Language understanding | 62.2 | | 59.2 | 61.2 | |
|
| 71 |
+
| Retrieval & classification | 63.5 | | 55.4 | 63.5 | |
|
| 72 |
+
| Tools & automation | 83.1 | | 75.1 | 73.8 | |
|
| 73 |
+
| Arts & human taste | 48.9 | | 39.1 | 39.2 | |
|
| 74 |
+
| **Laya public suites, mean accuracy** (17 suites) | **0.801** | 0.793 | 0.786 | | 0.694 |
|
| 75 |
+
| Calibration error, ECE (lower is better) | 0.110 | **0.088** | 0.124 | | 0.170 |
|
| 76 |
+
| Typed decisions, sealed suite (12 suites, pooled) | 93.1 | **94.6** | 94.3 | | |
|
| 77 |
+
| Trading decisions suite | **0.749** | 0.732 | 0.707 | | |
|
| 78 |
+
|
| 79 |
+
- **Decision Index:** Seb-9B-v2 is a full run on the 0.3 kit (110,520 requests); the Jev numbers are the live board's (2026-10-22). It is ahead of Jev 1.13
|
| 80 |
+
on 22 of 37 benchmarks and ahead of Jev 1.14 on 20 of 37. Biggest wins over Jev 1.13: POP909 +63.3, HoVer +30.5, Home appliances +28.4,
|
| 81 |
+
ContractNLI +19.0, CLadder +16.6, VAST +14.2. Largest gaps: GPQA Diamond -48.3, MMLU-Pro -33.2, BBH -30.7. The board also lists newer open entries with higher
|
| 82 |
+
public-index scores, for example OneJev1.6-9B 65.3; near Seb-9B-v2's size and score are Drex v1.5 (9B) 58.1 and Bespoke Nimble 9B v3 57.2. Per-benchmark numbers:
|
| 83 |
+
`di_public_index_vs_jev.json` in the demo folder.
|
| 84 |
+
- **Biggest gains over Seb-9B:** BANKING77 0.973 (Seb-9B 0.907, Jev 0.920), jailbreak detection 0.940 (0.868), model routing 0.980 (0.945), phishing 0.895 (0.863),
|
| 85 |
+
MASSIVE scenario 0.833 (0.780).
|
| 86 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 87 |
<img src="https://huggingface.co/ironbcc/Seb-9B-v2/resolve/main/assets/di-wins-vs-jev.png" width="100%">
|
| 88 |
|
| 89 |
## Against general LLMs
|