Add ensemble evaluation results
Browse files
README.md
CHANGED
|
@@ -96,3 +96,25 @@ print(f"Injection: {is_injection} (confidence: {probs[0, 1].item():.4f})")
|
|
| 96 |
url={https://github.com/lakeraai/pint-benchmark/pull/35}
|
| 97 |
}
|
| 98 |
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 96 |
url={https://github.com/lakeraai/pint-benchmark/pull/35}
|
| 97 |
}
|
| 98 |
```
|
| 99 |
+
|
| 100 |
+
## Evaluation
|
| 101 |
+
|
| 102 |
+
> Injection Sentry is a 3-model ensemble; **this repo is one component**. Numbers below are for the **full ensemble** β reproduce via [Verm1lion/InjectionSentry](https://github.com/Verm1lion/InjectionSentry).
|
| 103 |
+
|
| 104 |
+
Tested on 9 public prompt-injection / jailbreak datasets (pinned revisions, threshold `0.57`):
|
| 105 |
+
|
| 106 |
+
| Dataset | n | Recall | FPR | Bal. Acc | AUC |
|
| 107 |
+
|---|--:|--:|--:|--:|--:|
|
| 108 |
+
| deepset/prompt-injections (test) | 116 | 0.867 | 0.000 | 0.933 | 0.970 |
|
| 109 |
+
| jackhhao/jailbreak (test) | 262 | 0.971 | 0.008 | 0.982 | 0.997 |
|
| 110 |
+
| xTRam1/safe-guard (test) | 2060 | 0.998 | 0.001 | 0.999 | 1.000 |
|
| 111 |
+
| GenTel-Bench (8k) | 8000 | 0.927 | 0.033 | 0.947 | 0.993 |
|
| 112 |
+
| InjecGuard/PIGuard (valid) | 144 | 0.938 | 0.021 | 0.958 | 0.989 |
|
| 113 |
+
| NotInject (over-defense) | 339 | β | 0.000 | β | β |
|
| 114 |
+
| BIPIA (injection) | 125 | 0.856 | β | β | β |
|
| 115 |
+
| Lakera/gandalf (test) | 112 | 0.982 | β | β | β |
|
| 116 |
+
|
| 117 |
+
- **0% false positives on NotInject** (benign prompts with injection trigger-words) β not fooled by surface keywords.
|
| 118 |
+
- **Estimated Lakera PINT β 92%** (PINT is gated; estimated from category-weighted balanced accuracy) β roughly #2 on the public leaderboard, behind Lakera Guard (95.2%).
|
| 119 |
+
|
| 120 |
+
*Note: xTRam1 / deepset / gandalf / BIPIA overlap common training data, so GenTel-Bench (0.93) is the cleaner signal. WildGuard-benign FPR is high, but those prompts use jailbreak / role-play framing.*
|