mukti-sys commited on
Commit
8de4d7c
·
verified ·
1 Parent(s): 956da87

docs: pin matched live Jev API benchmark (+13.52% lead on SWE-bench)

Browse files
Files changed (1) hide show
  1. README.md +7 -0
README.md CHANGED
@@ -51,6 +51,13 @@ widget:
51
  * **AI / ML Runtime Triage:** **99.4% (164 / 165 correct)**
52
  * **Cybersecurity CWE Triage:** **95.2% (300 / 315 correct)**
53
 
 
 
 
 
 
 
 
54
  ### Set 2: Developer PR Autopilot Gating (Held-Out)
55
  * **Top-1 Accuracy:** **93.66% (399 / 426 correct)**
56
  * **Sensitivity (Safe Actions):** **100.0% (71 / 71)**
 
51
  * **AI / ML Runtime Triage:** **99.4% (164 / 165 correct)**
52
  * **Cybersecurity CWE Triage:** **95.2% (300 / 315 correct)**
53
 
54
+ ### Matched Head-to-Head vs. TypeSafe Jev (Live API: `jev-1.13.0`)
55
+ Evaluated across 222 identical technical decisions from Princeton SWE-bench and CyberNative CVEs queried live via TypeSafe Jev's official API (`https://jevmodel.org/v1/systemone`):
56
+ * **TypeSafe Jev (`jev-1.13.0` Live API):** **43.69%** (97 / 222 correct, Wilson 95% CI: $[37.33\%, 50.27\%]$)
57
+ * **Cortex-1 Large (This Work):** **57.21%** (127 / 222 correct, Wilson 95% CI: $[50.62\%, 63.56\%]$)
58
+ * **Net Empirical Margin:** **+13.52% Lead ($p < 0.005$)**
59
+ * **Decision Verification Log:** [`evaluation/matched_jev_head_to_head.csv`](https://github.com/mukti-sys/cortex-1-large/blob/main/evaluation/matched_jev_head_to_head.csv)
60
+
61
  ### Set 2: Developer PR Autopilot Gating (Held-Out)
62
  * **Top-1 Accuracy:** **93.66% (399 / 426 correct)**
63
  * **Sensitivity (Safe Actions):** **100.0% (71 / 71)**