File size: 3,940 Bytes
6b7d0ec | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 | # moderation evaluation
**Threshold 0.84 (calibrated on the validation split, objective macro_f1).** The config default of 0.5 scored 0.923 against 0.931 for the calibrated value, on validation. Every table below is on test, at the calibrated threshold.
## Per language
| Language | Support | P | R | F1 | Notes |
|---|---|---|---|---|---|
| `bg` Bulgarian | 60 | 1.000 | 0.967 | 0.983 | |
| `cs` Czech | 60 | 1.000 | 1.000 | 1.000 | |
| `da` Danish | 60 | 1.000 | 0.983 | 0.992 | |
| `de` German | 60 | 1.000 | 0.983 | 0.992 | |
| `el` Greek | 60 | 1.000 | 1.000 | 1.000 | |
| `en` English | 59 | 0.983 | 1.000 | 0.992 | |
| `es` Spanish | 59 | 1.000 | 0.983 | 0.991 | |
| `et` Estonian | 60 | 1.000 | 1.000 | 1.000 | |
| `fi` Finnish | 60 | 1.000 | 1.000 | 1.000 | |
| `fr` French | 59 | 1.000 | 1.000 | 1.000 | |
| `ga` Irish | 60 | 0.952 | 0.983 | 0.967 | |
| `hr` Croatian | 58 | 0.983 | 1.000 | 0.991 | |
| `hu` Hungarian | 60 | 1.000 | 1.000 | 1.000 | |
| `it` Italian | 59 | 1.000 | 1.000 | 1.000 | |
| `lt` Lithuanian | 60 | 0.984 | 1.000 | 0.992 | |
| `lv` Latvian | 60 | 1.000 | 0.967 | 0.983 | |
| `mt` Maltese | 59 | 0.982 | 0.949 | 0.966 | not in base model pretraining |
| `nl` Dutch | 60 | 1.000 | 0.983 | 0.992 | |
| `pl` Polish | 60 | 1.000 | 1.000 | 1.000 | |
| `pt` Portuguese | 60 | 0.968 | 1.000 | 0.984 | |
| `ro` Romanian | 60 | 1.000 | 1.000 | 1.000 | |
| `sk` Slovak | 60 | 1.000 | 0.983 | 0.992 | |
| `sl` Slovenian | 59 | 1.000 | 1.000 | 1.000 | |
| `sv` Swedish | 60 | 1.000 | 1.000 | 1.000 | |
| `tr` Turkish | 60 | 0.984 | 1.000 | 0.992 | |
| `az` Azerbaijani | 60 | 1.000 | 0.967 | 0.983 | |
The base-model note is a fact about pretraining, not a cause of the score beside it. `nsfw` Maltese carried the same note at 0.000 and reached 1.000 on corpus size alone, with nothing about the base model changed. Check how many examples a weak score rests on before reaching for this.
## Per register
| Register | Support | P | R | F1 | FPR |
|---|---|---|---|---|---|
| `cyber_intrusion` | 130 | 1.000 | 1.000 | 1.000 | 0.000 |
| `cyber_intrusion_near_miss` | 0 | 0.000 | 0.000 | 0.000 | 0.000 |
| `defamation` | 130 | 1.000 | 1.000 | 1.000 | 0.000 |
| `defamation_near_miss` | 0 | 0.000 | 0.000 | 0.000 | 0.019 |
| `election_integrity` | 130 | 1.000 | 0.992 | 0.996 | 0.000 |
| `election_integrity_near_miss` | 0 | 0.000 | 0.000 | 0.000 | 0.000 |
| `extremism` | 129 | 1.000 | 0.984 | 0.992 | 0.000 |
| `extremism_near_miss` | 0 | 0.000 | 0.000 | 0.000 | 0.038 |
| `fraud_deception` | 130 | 1.000 | 1.000 | 1.000 | 0.000 |
| `fraud_deception_near_miss` | 0 | 0.000 | 0.000 | 0.000 | 0.058 |
| `hate_incitement` | 126 | 1.000 | 1.000 | 1.000 | 0.000 |
| `hate_incitement_near_miss` | 0 | 0.000 | 0.000 | 0.000 | 0.000 |
| `illicit_drugs` | 130 | 1.000 | 1.000 | 1.000 | 0.000 |
| `illicit_drugs_near_miss` | 0 | 0.000 | 0.000 | 0.000 | 0.000 |
| `mundane_informational` | 0 | 0.000 | 0.000 | 0.000 | 0.000 |
| `mundane_operational` | 0 | 0.000 | 0.000 | 0.000 | 0.000 |
| `mundane_transactional` | 0 | 0.000 | 0.000 | 0.000 | 0.000 |
| `property_crime` | 129 | 1.000 | 0.977 | 0.988 | 0.000 |
| `property_crime_near_miss` | 0 | 0.000 | 0.000 | 0.000 | 0.019 |
| `self_harm` | 129 | 1.000 | 0.992 | 0.996 | 0.000 |
| `self_harm_near_miss` | 0 | 0.000 | 0.000 | 0.000 | 0.019 |
| `sexual_exploitation` | 129 | 1.000 | 0.969 | 0.984 | 0.000 |
| `sexual_exploitation_near_miss` | 0 | 0.000 | 0.000 | 0.000 | 0.019 |
| `violent_facilitation` | 130 | 1.000 | 0.977 | 0.988 | 0.000 |
| `violent_facilitation_near_miss` | 0 | 0.000 | 0.000 | 0.000 | 0.019 |
| `weapons_cbrn` | 130 | 1.000 | 0.992 | 0.996 | 0.000 |
| `weapons_cbrn_near_miss` | 0 | 0.000 | 0.000 | 0.000 | 0.000 |
## Known weaknesses
The three weakest languages by F1: `mt` at 0.966, `ga` at 0.967, `az` at 0.983.
These are published rather than dropped. A coverage table with the bad rows removed is not a coverage table. |