Card: role-noun bench numbers for 2.0.4
Browse files
README.md
CHANGED
|
@@ -101,6 +101,18 @@ Unlabelled phones and identifiers (the rules take a bare phone only in the group
|
|
| 101 |
|
| 102 |
Names off hides no name. A component is the independent unit, covered when every name in it is whole: on the random panel 18 of 116 keep part of a name. The second pass is a same-day blind self-review of 100 passages by the same reviewer (11 disagreed), not independent agreement. Since 2.0.4 the scores are the frozen 2.0.0 predictions with person spans refiltered through 2.0.4's role-noun rule (2.0.3: 1,051 random false characters). `hybrid.json` `eval_names` has the full numbers, including float16 and INT8 scored for the record; neither ships.
|
| 103 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 104 |
**Cost:** on one RTX 4090 with 3 processes, names took 2.0.0 from 74,520 to 66,492 chars/s (−11%) and peak device memory from 9,006 to 15,416 MiB; with one process, −9%. Local measurements are in `CHANGELOG.md`.
|
| 105 |
|
| 106 |
## Cue-less phone test
|
|
|
|
| 101 |
|
| 102 |
Names off hides no name. A component is the independent unit, covered when every name in it is whole: on the random panel 18 of 116 keep part of a name. The second pass is a same-day blind self-review of 100 passages by the same reviewer (11 disagreed), not independent agreement. Since 2.0.4 the scores are the frozen 2.0.0 predictions with person spans refiltered through 2.0.4's role-noun rule (2.0.3: 1,051 random false characters). `hybrid.json` `eval_names` has the full numbers, including float16 and INT8 scored for the record; neither ships.
|
| 103 |
|
| 104 |
+
**Role-noun bench** (`nergal_role_bench_v1`, 2026-10-19): how often a capitalised procedural role is masked as a person. A role noun is never a name, so every one masked is a false mask and no gold is needed. 4,615 Dynaword legal documents, about 5M characters each from UZP appeal rulings, TED notices, local-government and gov.pl pages, EUR-Lex and Dziennik Ustaw, excluding the documents the 2.0.4 roles were chosen on. Every column runs the same FastPDN weights, so the change is NERGAL's span filter, not the model.
|
| 105 |
+
|
| 106 |
+
| Role (capitalised) | Occurrences | FastPDN tokens | 2.0.3 | 2.0.4 |
|
| 107 |
+
|---|---:|---:|---:|---:|
|
| 108 |
+
| Odwołujący | 4,171 | 1,361 | 1,361 | **0** |
|
| 109 |
+
| Przystępujący | 534 | 164 | 164 | **0** |
|
| 110 |
+
| Zamawiający | 12,631 | 16 | 16 | **1** |
|
| 111 |
+
| Skarżący | 50 | 3 | 3 | **0** |
|
| 112 |
+
| Wykonawca, Prezes, Inwestor | 14,547 | 6 | 6 | 6 |
|
| 113 |
+
|
| 114 |
+
The other nine roles tracked (wnioskodawca, uczestnik, powód, pozwany, przewodniczący, protokolant, pełnomocnik, oferent, protestujący) have no mask. The one 2.0.4 `Zamawiający` is a truncated `Zamawiając`. Both versions mask all 69,834 other words FastPDN tags, so the filter removes only role nouns; name recall is the name test above. The texts hold real names and are not released.
|
| 115 |
+
|
| 116 |
**Cost:** on one RTX 4090 with 3 processes, names took 2.0.0 from 74,520 to 66,492 chars/s (−11%) and peak device memory from 9,006 to 15,416 MiB; with one process, −9%. Local measurements are in `CHANGELOG.md`.
|
| 117 |
|
| 118 |
## Cue-less phone test
|