Upload eval/groundedness_eval.md with huggingface_hub
Browse files- eval/groundedness_eval.md +56 -0
eval/groundedness_eval.md
ADDED
|
@@ -0,0 +1,56 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# groundedness evaluation
|
| 2 |
+
|
| 3 |
+
**Threshold 0.5, taken from the config.**
|
| 4 |
+
|
| 5 |
+
**Pair accuracy: 0.899** (both halves of a minimal pair correct. This is the headline number for this model.)
|
| 6 |
+
|
| 7 |
+
**Single-label head, so this table reports exact-match accuracy.** Precision, recall and F1 over `did the detector fire` are meaningless here: argmax always names a class, so every row would count as a true positive and every language would read 1.000 whether or not a single prediction was right.
|
| 8 |
+
|
| 9 |
+
## Per language
|
| 10 |
+
|
| 11 |
+
| Language | N | Accuracy | Notes |
|
| 12 |
+
|---|---|---|---|
|
| 13 |
+
| `bg` Bulgarian | 80 | 0.975 | |
|
| 14 |
+
| `cs` Czech | 78 | 0.962 | |
|
| 15 |
+
| `da` Danish | 80 | 0.963 | |
|
| 16 |
+
| `de` German | 78 | 0.936 | |
|
| 17 |
+
| `el` Greek | 80 | 0.950 | |
|
| 18 |
+
| `en` English | 78 | 0.897 | |
|
| 19 |
+
| `es` Spanish | 80 | 0.950 | |
|
| 20 |
+
| `et` Estonian | 76 | 0.961 | |
|
| 21 |
+
| `fi` Finnish | 78 | 0.936 | |
|
| 22 |
+
| `fr` French | 80 | 0.938 | |
|
| 23 |
+
| `ga` Irish | 78 | 0.974 | |
|
| 24 |
+
| `hr` Croatian | 78 | 0.923 | |
|
| 25 |
+
| `hu` Hungarian | 80 | 0.950 | |
|
| 26 |
+
| `it` Italian | 80 | 0.988 | |
|
| 27 |
+
| `lt` Lithuanian | 80 | 0.963 | |
|
| 28 |
+
| `lv` Latvian | 80 | 0.950 | |
|
| 29 |
+
| `mt` Maltese | 80 | 0.938 | not in base model pretraining |
|
| 30 |
+
| `nl` Dutch | 80 | 0.938 | |
|
| 31 |
+
| `pl` Polish | 80 | 0.912 | |
|
| 32 |
+
| `pt` Portuguese | 80 | 0.963 | |
|
| 33 |
+
| `ro` Romanian | 80 | 1.000 | |
|
| 34 |
+
| `sk` Slovak | 80 | 0.925 | |
|
| 35 |
+
| `sl` Slovenian | 80 | 0.963 | |
|
| 36 |
+
| `sv` Swedish | 80 | 0.912 | |
|
| 37 |
+
| `tr` Turkish | 78 | 0.962 | |
|
| 38 |
+
| `az` Azerbaijani | 80 | 0.912 | |
|
| 39 |
+
|
| 40 |
+
The base-model note is a fact about pretraining, not a cause of the score beside it. `nsfw` Maltese carried the same note at 0.000 and reached 1.000 on corpus size alone, with nothing about the base model changed. Check how many examples a weak score rests on before reaching for this.
|
| 41 |
+
|
| 42 |
+
## Per register
|
| 43 |
+
|
| 44 |
+
| Register | N | Accuracy |
|
| 45 |
+
|---|---|---|
|
| 46 |
+
| `lexical_overlap` | 412 | 0.949 |
|
| 47 |
+
| `negation_conflict` | 414 | 0.947 |
|
| 48 |
+
| `numeric_conflict` | 404 | 0.908 |
|
| 49 |
+
| `scope_conflict` | 416 | 0.966 |
|
| 50 |
+
| `unstated` | 416 | 0.966 |
|
| 51 |
+
|
| 52 |
+
## Known weaknesses
|
| 53 |
+
|
| 54 |
+
The three weakest languages by accuracy: `en` at 0.897, `az` at 0.912, `pl` at 0.912.
|
| 55 |
+
|
| 56 |
+
These are published rather than dropped. A coverage table with the bad rows removed is not a coverage table.
|