groundedness evaluation
Threshold 0.5, taken from the config.
Pair accuracy: 0.899 (both halves of a minimal pair correct. This is the headline number for this model.)
Single-label head, so this table reports exact-match accuracy. Precision, recall and F1 over did the detector fire are meaningless here: argmax always names a class, so every row would count as a true positive and every language would read 1.000 whether or not a single prediction was right.
Per language
| Language | N | Accuracy | Notes |
|---|---|---|---|
bg Bulgarian |
80 | 0.975 | |
cs Czech |
78 | 0.962 | |
da Danish |
80 | 0.963 | |
de German |
78 | 0.936 | |
el Greek |
80 | 0.950 | |
en English |
78 | 0.897 | |
es Spanish |
80 | 0.950 | |
et Estonian |
76 | 0.961 | |
fi Finnish |
78 | 0.936 | |
fr French |
80 | 0.938 | |
ga Irish |
78 | 0.974 | |
hr Croatian |
78 | 0.923 | |
hu Hungarian |
80 | 0.950 | |
it Italian |
80 | 0.988 | |
lt Lithuanian |
80 | 0.963 | |
lv Latvian |
80 | 0.950 | |
mt Maltese |
80 | 0.938 | not in base model pretraining |
nl Dutch |
80 | 0.938 | |
pl Polish |
80 | 0.912 | |
pt Portuguese |
80 | 0.963 | |
ro Romanian |
80 | 1.000 | |
sk Slovak |
80 | 0.925 | |
sl Slovenian |
80 | 0.963 | |
sv Swedish |
80 | 0.912 | |
tr Turkish |
78 | 0.962 | |
az Azerbaijani |
80 | 0.912 |
The base-model note is a fact about pretraining, not a cause of the score beside it. nsfw Maltese carried the same note at 0.000 and reached 1.000 on corpus size alone, with nothing about the base model changed. Check how many examples a weak score rests on before reaching for this.
Per register
| Register | N | Accuracy |
|---|---|---|
lexical_overlap |
412 | 0.949 |
negation_conflict |
414 | 0.947 |
numeric_conflict |
404 | 0.908 |
scope_conflict |
416 | 0.966 |
unstated |
416 | 0.966 |
Known weaknesses
The three weakest languages by accuracy: en at 0.897, az at 0.912, pl at 0.912.
These are published rather than dropped. A coverage table with the bad rows removed is not a coverage table.