docs: pin matched live Jev API benchmark (+13.52% lead on SWE-bench)
Browse files
README.md
CHANGED
|
@@ -51,6 +51,13 @@ widget:
|
|
| 51 |
* **AI / ML Runtime Triage:** **99.4% (164 / 165 correct)**
|
| 52 |
* **Cybersecurity CWE Triage:** **95.2% (300 / 315 correct)**
|
| 53 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 54 |
### Set 2: Developer PR Autopilot Gating (Held-Out)
|
| 55 |
* **Top-1 Accuracy:** **93.66% (399 / 426 correct)**
|
| 56 |
* **Sensitivity (Safe Actions):** **100.0% (71 / 71)**
|
|
|
|
| 51 |
* **AI / ML Runtime Triage:** **99.4% (164 / 165 correct)**
|
| 52 |
* **Cybersecurity CWE Triage:** **95.2% (300 / 315 correct)**
|
| 53 |
|
| 54 |
+
### Matched Head-to-Head vs. TypeSafe Jev (Live API: `jev-1.13.0`)
|
| 55 |
+
Evaluated across 222 identical technical decisions from Princeton SWE-bench and CyberNative CVEs queried live via TypeSafe Jev's official API (`https://jevmodel.org/v1/systemone`):
|
| 56 |
+
* **TypeSafe Jev (`jev-1.13.0` Live API):** **43.69%** (97 / 222 correct, Wilson 95% CI: $[37.33\%, 50.27\%]$)
|
| 57 |
+
* **Cortex-1 Large (This Work):** **57.21%** (127 / 222 correct, Wilson 95% CI: $[50.62\%, 63.56\%]$)
|
| 58 |
+
* **Net Empirical Margin:** **+13.52% Lead ($p < 0.005$)**
|
| 59 |
+
* **Decision Verification Log:** [`evaluation/matched_jev_head_to_head.csv`](https://github.com/mukti-sys/cortex-1-large/blob/main/evaluation/matched_jev_head_to_head.csv)
|
| 60 |
+
|
| 61 |
### Set 2: Developer PR Autopilot Gating (Held-Out)
|
| 62 |
* **Top-1 Accuracy:** **93.66% (399 / 426 correct)**
|
| 63 |
* **Sensitivity (Safe Actions):** **100.0% (71 / 71)**
|