Verm1ion commited on
Commit
a35f9b2
Β·
verified Β·
1 Parent(s): 1c06730

Add ensemble evaluation results

Browse files
Files changed (1) hide show
  1. README.md +22 -0
README.md CHANGED
@@ -96,3 +96,25 @@ print(f"Injection: {is_injection} (confidence: {probs[0, 1].item():.4f})")
96
  url={https://github.com/lakeraai/pint-benchmark/pull/35}
97
  }
98
  ```
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
96
  url={https://github.com/lakeraai/pint-benchmark/pull/35}
97
  }
98
  ```
99
+
100
+ ## Evaluation
101
+
102
+ > Injection Sentry is a 3-model ensemble; **this repo is one component**. Numbers below are for the **full ensemble** β€” reproduce via [Verm1lion/InjectionSentry](https://github.com/Verm1lion/InjectionSentry).
103
+
104
+ Tested on 9 public prompt-injection / jailbreak datasets (pinned revisions, threshold `0.57`):
105
+
106
+ | Dataset | n | Recall | FPR | Bal. Acc | AUC |
107
+ |---|--:|--:|--:|--:|--:|
108
+ | deepset/prompt-injections (test) | 116 | 0.867 | 0.000 | 0.933 | 0.970 |
109
+ | jackhhao/jailbreak (test) | 262 | 0.971 | 0.008 | 0.982 | 0.997 |
110
+ | xTRam1/safe-guard (test) | 2060 | 0.998 | 0.001 | 0.999 | 1.000 |
111
+ | GenTel-Bench (8k) | 8000 | 0.927 | 0.033 | 0.947 | 0.993 |
112
+ | InjecGuard/PIGuard (valid) | 144 | 0.938 | 0.021 | 0.958 | 0.989 |
113
+ | NotInject (over-defense) | 339 | β€” | 0.000 | β€” | β€” |
114
+ | BIPIA (injection) | 125 | 0.856 | β€” | β€” | β€” |
115
+ | Lakera/gandalf (test) | 112 | 0.982 | β€” | β€” | β€” |
116
+
117
+ - **0% false positives on NotInject** (benign prompts with injection trigger-words) β€” not fooled by surface keywords.
118
+ - **Estimated Lakera PINT β‰ˆ 92%** (PINT is gated; estimated from category-weighted balanced accuracy) β€” roughly #2 on the public leaderboard, behind Lakera Guard (95.2%).
119
+
120
+ *Note: xTRam1 / deepset / gandalf / BIPIA overlap common training data, so GenTel-Bench (0.93) is the cleaner signal. WildGuard-benign FPR is high, but those prompts use jailbreak / role-play framing.*