Upload eval/injection_eval.md with huggingface_hub

eval/injection_eval.md

@@ -0,0 +1,63 @@
+# injection evaluation
+
+**Threshold 0.19 (calibrated on the validation split, objective recall_at_fpr).** The config default of 0.5 scored 0.938 against 0.961 for the calibrated value, on validation. Every table below is on test, at the calibrated threshold.
+
+## Per language
+
+| Language | Support | P | R | F1 | Notes |
+|---|---|---|---|---|---|
+| `bg` Bulgarian | 13 | 1.000 | 1.000 | 1.000 |  |
+| `cs` Czech | 13 | 1.000 | 0.923 | 0.960 |  |
+| `da` Danish | 14 | 0.933 | 1.000 | 0.966 |  |
+| `de` German | 14 | 0.933 | 1.000 | 0.966 |  |
+| `el` Greek | 14 | 1.000 | 1.000 | 1.000 |  |
+| `en` English | 14 | 0.933 | 1.000 | 0.966 |  |
+| `es` Spanish | 12 | 1.000 | 1.000 | 1.000 |  |
+| `et` Estonian | 14 | 0.933 | 1.000 | 0.966 |  |
+| `fi` Finnish | 14 | 0.933 | 1.000 | 0.966 |  |
+| `fr` French | 13 | 0.923 | 0.923 | 0.923 |  |
+| `ga` Irish | 14 | 1.000 | 1.000 | 1.000 |  |
+| `hr` Croatian | 13 | 1.000 | 1.000 | 1.000 |  |
+| `hu` Hungarian | 14 | 1.000 | 1.000 | 1.000 |  |
+| `it` Italian | 14 | 1.000 | 1.000 | 1.000 |  |
+| `lt` Lithuanian | 13 | 1.000 | 1.000 | 1.000 |  |
+| `lv` Latvian | 14 | 1.000 | 1.000 | 1.000 |  |
+| `mt` Maltese | 14 | 1.000 | 0.571 | 0.727 | not in base model pretraining |
+| `nl` Dutch | 14 | 1.000 | 1.000 | 1.000 |  |
+| `pl` Polish | 14 | 1.000 | 1.000 | 1.000 |  |
+| `pt` Portuguese | 14 | 0.875 | 1.000 | 0.933 |  |
+| `ro` Romanian | 14 | 0.933 | 1.000 | 0.966 |  |
+| `sk` Slovak | 14 | 1.000 | 0.929 | 0.963 |  |
+| `sl` Slovenian | 14 | 1.000 | 1.000 | 1.000 |  |
+| `sv` Swedish | 14 | 1.000 | 1.000 | 1.000 |  |
+| `tr` Turkish | 14 | 0.933 | 1.000 | 0.966 |  |
+| `az` Azerbaijani | 14 | 1.000 | 0.929 | 0.963 |  |
+
+The base-model note is a fact about pretraining, not a cause of the score beside it. `nsfw` Maltese carried the same note at 0.000 and reached 1.000 on corpus size alone, with nothing about the base model changed. Check how many examples a weak score rests on before reaching for this.
+
+## Per register
+
+| Register | Support | P | R | F1 | FPR |
+|---|---|---|---|---|---|
+| `encoded_payload` | 50 | 1.000 | 0.980 | 0.990 | 0.000 |
+| `hypothetical_framing` | 52 | 1.000 | 0.942 | 0.970 | 0.000 |
+| `ignore_instructions` | 51 | 1.000 | 0.980 | 0.990 | 0.000 |
+| `instruction_in_document` | 52 | 1.000 | 1.000 | 1.000 | 0.000 |
+| `instruction_in_tool_output` | 48 | 1.000 | 1.000 | 1.000 | 0.000 |
+| `meta_question` | 0 | 0.000 | 0.000 | 0.000 | 0.029 |
+| `mundane_informational` | 0 | 0.000 | 0.000 | 0.000 | 0.000 |
+| `mundane_operational` | 0 | 0.000 | 0.000 | 0.000 | 0.000 |
+| `mundane_transactional` | 0 | 0.000 | 0.000 | 0.000 | 0.000 |
+| `ordinary_instruction` | 0 | 0.000 | 0.000 | 0.000 | 0.000 |
+| `ordinary_question` | 0 | 0.000 | 0.000 | 0.000 | 0.000 |
+| `persona_override` | 52 | 1.000 | 0.962 | 0.980 | 0.000 |
+| `quoted_attack` | 0 | 0.000 | 0.000 | 0.000 | 0.019 |
+| `roleplay_benign` | 0 | 0.000 | 0.000 | 0.000 | 0.019 |
+| `security_discussion` | 0 | 0.000 | 0.000 | 0.000 | 0.029 |
+| `system_prompt_extraction` | 52 | 1.000 | 0.942 | 0.970 | 0.000 |
+
+## Known weaknesses
+
+The three weakest languages by F1: `mt` at 0.727, `fr` at 0.923, `pt` at 0.933.
+
+These are published rather than dropped. A coverage table with the bad rows removed is not a coverage table.