2.0.5: the names layer leaves more role nouns, and a number right after one, out

CHANGELOG.md

@@ -8,6 +8,25 @@ Semver for this island:
 
 Accuracy is 841-dev, union at 0.95. Through 1.1.2 the gold has 354 spans; from 1.2.0 it is restated to phone policy v3 (315 spans), and the two are not comparable. A version that changes those numbers must update `hybrid.json` `eval` and the tables below.
 
+## 2.0.5
+
+Same NERGAL weights, threshold, union, rules (SHA `4d054723…`) and API. One change in `nergal.py` `person_spans`: the names layer (`names=True`) leaves more role nouns, and a number right after one, out of a person span. Names off is unchanged.
+
+- **More role nouns.** Besides the five participles of 2.0.4, a model word that is przewodniczący, pozwany, wykonawca, wnioskodawca (wnioskodawczyni), powód (powódka), prezes, inwestor, uczestnik (uczestniczka), pełnomocnik (pełnomocniczka), oferent (oferentka) or protokolant (protokolantka), in any case and number, is dropped before words are merged. Each was added only on dev evidence: masked as a person in the UZP rulings outside the role bench or in Dynaword 841-dev, and matching no gold person word in name tests v1 and v2.
+- **Cut and numbered roles.** The bare participle stem (`Zamawiając`: text cut short, or the adverbial participle) and a role with a number glued on (`Odwołujący3`, `wykonawcy_1`) are dropped too.
+- **A number right after a role is not a name.** `Odwołujący 1 Jan Nowak` → `Odwołujący 1 [PERSON]`: a model word that is only digits, right after a dropped role noun with only spaces, tabs or no-break spaces between, is dropped. A letterless word anywhere else keeps its mask (the name test gold holds 4 letterless person words).
+- **Measured on every UZP ruling.** G1's NERGAL stage and rules (polish-dynaword `05c12f32`) rebuilt from stored FastPDN spans for all 34,715 `uzp_orzeczenia_pl` rulings, 2.0.4 → 2.0.5 occurrences inside a person span: capitalised `Odwołujący` 1,885 → 33 of 903,658, `Przystępujący` 130 → 10, `Zamawiający` 26 → 1, `Skarżący` 1 → 0, `Wykonawca` 214 → 6 and lowercase `wykonawca` 524 → 5, `Inwestor` 74 → 15, bare stems 129 → 0, a number after a role 789 → 4. `[PERSON]` placeholders in G1 output fall 258,212 → 255,086 in 780 rulings.
+- **Correction to 2.0.4.** Its entry says Wnioskodawca, Uczestnik, Powód, Pozwany, Przewodniczący, Protokolant, Pełnomocnik, Prezes, Inwestor and Oferent are never inside a person span in the surveyed samples. Over all UZP rulings 2.0.4 masks 190 of them (Inwestor 74, Uczestnik 38, Oferent 23, Prezes 15, Powód 13, Protokolant 8, Wnioskodawca 6, Pozwany 5, Pełnomocnik 4, Przewodniczący 4); 2.0.5 masks none.
+
+Still missed: a role glued to another word or cut short in damaged text (most of the 33 `Odwołujący` and 10 `Przystępujący` left); `inwestorski` and other words that only start with a role stem, kept on purpose.
+
+841-dev, names off: unchanged from 2.0.4. Release gates (`artifacts/nergal_2_0_5_role_nouns/`, frozen before scoring, stored spans, no new inference): the 1,837 bench rows, 999 documents of the Dynaword runtime sample (names off and on; one is in the role bench) and 41,204 FineWeb miss-hunt passages, output identical. UZP G1: the 2.0.4 rebuild equals the stored output of the G1 check for all 289 cached rulings whose text Hub PR #149 left unchanged; no character gains a person span, and all 33,857 that lose one are in a role noun or a number right after one. Name test v1 and v2 (1,050 passages): a live CPU float32 run of the 2.0.5 `Names` equals the stored spans refiltered, 0 mismatches; no passage changes against 2.0.4 and no gold person character is lost. `hybrid.json` `eval_names` is unchanged. Label pool v3 `bench`, names off and `--names`: every cohort as 2.0.4. Role bench, held out until the freeze: every role 0, guard unchanged; it holds no tagged number after a role, so the number rule is measured on UZP alone.
+
+| Version | Whole /315 | Residual | Rules FP | Union FP | Char P | Char R | What changed |
+|---|---:|---:|---:|---:|---:|---:|---|
+| 2.0.4, names off | 303 | 9 | 24 | 80 | 98.59% | 97.83% | Names layer leaves procedural role nouns out |
+| 2.0.5, names off | 303 | 9 | 24 | 80 | 98.59% | 97.83% | Names layer leaves more role nouns, and a number right after one, out |
+
 ## 2.0.4
 
 Same NERGAL weights, threshold, union, rules (SHA `4d054723…`) and API. One change in `nergal.py` `person_spans`: the names layer (`names=True`) leaves procedural role nouns out. Names off is unchanged.

README.md

@@ -16,7 +16,7 @@ tags:
 - hybrid
 ---
 
-# NERGAL 2.0.4
+# NERGAL 2.0.5
 
 **Named Entity Recognition with Grounded Additive Labels**
 
@@ -66,9 +66,9 @@ NERGAL is not designed to remove:
 
 These are intended exclusions; false positives can still mask some of this content.
 
-### Known gaps in 2.0.4
+### Known gaps in 2.0.5
 
-Unlabelled phones and identifiers (the rules take a bare phone only in the grouped national forms `601 234 567` and `22 123 45 67`; a plain `601234567` needs a label or the model), unusual formatting and damaged text can escape detection. **VINs and obfuscated emails** (such as `name (at) domain.pl`) are approved annotation targets, but that approval alone does not establish reliable support in the released 2.0.4 model. Do not rely on it to remove them consistently. An email with a lowercase word glued onto its domain (`jan@firma.plkontakt`) is masked together with that word, and text glued before an address can be masked with it. A phone written as short parts joined by two slashes (`12 / 345 / 678`) is left as text. Input that already holds a `[PII]` / `[PHONE]` / `[Telefon]` placeholder: a NIP or REGON label reaches past one placeholder to a checksum-valid value, and a value cut by a placeholder is masked digit group by digit group only after a `PESEL` / `NIP` / `REGON` label (`[PII]`) or a phone cue (`[PHONE]`). Without either (`8503[PII]1212343`, `601 [PII] 200 300`), with fewer than 5 digits, or when the run starts with a placeholder (`tel. [PHONE] 200 300`), the fragments stay. Names-on gaps are under [Names](#names).
+Unlabelled phones and identifiers (the rules take a bare phone only in the grouped national forms `601 234 567` and `22 123 45 67`; a plain `601234567` needs a label or the model), unusual formatting and damaged text can escape detection. **VINs and obfuscated emails** (such as `name (at) domain.pl`) are approved annotation targets, but that approval alone does not establish reliable support in the released 2.0.5 model. Do not rely on it to remove them consistently. An email with a lowercase word glued onto its domain (`jan@firma.plkontakt`) is masked together with that word, and text glued before an address can be masked with it. A phone written as short parts joined by two slashes (`12 / 345 / 678`) is left as text. Input that already holds a `[PII]` / `[PHONE]` / `[Telefon]` placeholder: a NIP or REGON label reaches past one placeholder to a checksum-valid value, and a value cut by a placeholder is masked digit group by digit group only after a `PESEL` / `NIP` / `REGON` label (`[PII]`) or a phone cue (`[PHONE]`). Without either (`8503[PII]1212343`, `601 [PII] 200 300`), with fewer than 5 digits, or when the run starts with a placeholder (`tel. [PHONE] 200 300`), the fragments stay. Names-on gaps are under [Names](#names).
 
 ## Versions
 
@@ -82,13 +82,14 @@ Unlabelled phones and identifiers (the rules take a bare phone only in the group
 | 2.0.1, names off | 303 | 9 | 24 | 80 | 98.59% | 97.83% | A `[PERSON]` placeholder ends an other-number label |
 | 2.0.2, names off | 303 | 9 | 24 | 80 | 98.59% | 97.83% | A rule's label wins on its characters |
 | 2.0.3, names off | 303 | 9 | 24 | 80 | 98.59% | 97.83% | Values cut or preceded by a placeholder |
-| **2.0.4, names off** | **303** | **9** | **24** | **80** | **98.59%** | **97.83%** | Names layer leaves procedural role nouns out |
+| 2.0.4, names off | 303 | 9 | 24 | 80 | 98.59% | 97.83% | Names layer leaves procedural role nouns out |
+| **2.0.5, names off** | **303** | **9** | **24** | **80** | **98.59%** | **97.83%** | Names layer leaves more role nouns, and a number right after one, out |
 
 1.0.0–1.1.2 were scored on the earlier 354-span gold (whole 323 → 326, union FP 133 → 80); per-version numbers are in `CHANGELOG.md`.
 
 ## Names
 
-`names=True` (CLI `--names`) adds a second model, [FastPDN NER — Polish PII](https://huggingface.co/ArkadiuszPawlak/fastpdn-ner-polish-pii) by ArkadiuszPawlak, revision `636c57fe…`, fine-tuned from [clarin-pl/FastPDN](https://huggingface.co/clarin-pl/FastPDN) on filtered KPWr plus LLM-synthetic data. `names/` holds it converted from ONNX to safetensors with no value changed. **`names/` is licensed [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/), not MIT**; `names/NOTICE.md` has the attribution, the changes and the file hashes. NERGAL uses only its person tags (`PERSON`, `PERSON_F`, `PERSON_L`), masks whole words, and always runs it in float32. Since 2.0.4 it leaves procedural role nouns out of a person span: odwołujący, przystępujący, zamawiający, protestujący and skarżący in any inflection (`Odwołujący Jan Nowak` → `Odwołujący [PERSON]`). The number after a role (`Odwołujący 1`) and role nouns outside these five can still be masked.
+`names=True` (CLI `--names`) adds a second model, [FastPDN NER — Polish PII](https://huggingface.co/ArkadiuszPawlak/fastpdn-ner-polish-pii) by ArkadiuszPawlak, revision `636c57fe…`, fine-tuned from [clarin-pl/FastPDN](https://huggingface.co/clarin-pl/FastPDN) on filtered KPWr plus LLM-synthetic data. `names/` holds it converted from ONNX to safetensors with no value changed. **`names/` is licensed [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/), not MIT**; `names/NOTICE.md` has the attribution, the changes and the file hashes. NERGAL uses only its person tags (`PERSON`, `PERSON_F`, `PERSON_L`), masks whole words, and always runs it in float32. It leaves procedural role nouns out of a person span (`Odwołujący Jan Nowak` → `Odwołujący [PERSON]`): since 2.0.5 the parties, representatives and offices odwołujący, przystępujący, zamawiający, protestujący, skarżący, przewodniczący, pozwany, powód, wykonawca, wnioskodawca, uczestnik, pełnomocnik, oferent, protokolant, inwestor and prezes, in any inflection and the feminine forms, also cut to the bare participle stem (`Zamawiając`) or with a number glued on (`Odwołujący3`). A number right after one (`Odwołujący 1`) is left out too. A role glued to another word in damaged text can still be masked.
 
 **Scope with names on.** The name test labels every person: private individuals, public officials, politicians and historical figures alike, plus fictional characters, nicknames, partial names and initials. Titles and roles stay outside the span. Names inside a street, patronage, organisation or work title, eponyms, usernames, and names inside emails or URLs are not person spans. Masking public figures removes who said, signed or wrote what, which is why names are opt-in. This is the labelling target; the model does not meet it everywhere.
 
@@ -99,19 +100,19 @@ Unlabelled phones and identifiers (the rules take a bare phone only in the group
 | Random | 300 | 614 | 586 (95.44%) | 98/116 (76.6–90.5%) | 1,021 | 34 (11.3%) |
 | Targeted | 150 | 292 | 286 (97.95%) | 94/99 (88.6–98.3%) | 976 | 42 (28.0%) |
 
-Names off hides no name. A component is the independent unit, covered when every name in it is whole: on the random panel 18 of 116 keep part of a name. The second pass is a same-day blind self-review of 100 passages by the same reviewer (11 disagreed), not independent agreement. Since 2.0.4 the scores are the frozen 2.0.0 predictions with person spans refiltered through 2.0.4's role-noun rule (2.0.3: 1,051 random false characters). `hybrid.json` `eval_names` has the full numbers, including float16 and INT8 scored for the record; neither ships.
+Names off hides no name. A component is the independent unit, covered when every name in it is whole: on the random panel 18 of 116 keep part of a name. The second pass is a same-day blind self-review of 100 passages by the same reviewer (11 disagreed), not independent agreement. Since 2.0.4 the scores are the frozen 2.0.0 predictions with person spans refiltered through the release's role-noun rule (2.0.3: 1,051 random false characters); 2.0.5 gives the same spans as 2.0.4 on both name tests. `hybrid.json` `eval_names` has the full numbers, including float16 and INT8 scored for the record; neither ships.
 
-**Role-noun bench** (`nergal_role_bench_v1`, 2026-10-19): how often a capitalised procedural role is masked as a person. A role noun is never a name, so every one masked is a false mask and no gold is needed. 4,615 Dynaword legal documents, about 5M characters each from UZP appeal rulings, TED notices, local-government and gov.pl pages, EUR-Lex and Dziennik Ustaw, excluding the documents the 2.0.4 roles were chosen on. Every column runs the same FastPDN weights, so the change is NERGAL's span filter, not the model.
+**Role-noun bench** (`nergal_role_bench_v1`, 2026-10-19): how often a capitalised procedural role is masked as a person. A role noun is never a name, so every one masked is a false mask and no gold is needed. 4,615 Dynaword legal documents, about 5M characters each from UZP appeal rulings, TED notices, local-government and gov.pl pages, EUR-Lex and Dziennik Ustaw, excluding the documents the 2.0.4 roles were chosen on; the 2.0.5 roles were chosen on the UZP rulings outside the bench. Every column runs the same FastPDN weights, so the change is NERGAL's span filter, not the model.
 
-| Role (capitalised) | Occurrences | FastPDN tokens | 2.0.3 | 2.0.4 |
-|---|---:|---:|---:|---:|
-| Odwołujący | 4,171 | 1,361 | 1,361 | **0** |
-| Przystępujący | 534 | 164 | 164 | **0** |
-| Zamawiający | 12,631 | 16 | 16 | **1** |
-| Skarżący | 50 | 3 | 3 | **0** |
-| Wykonawca, Prezes, Inwestor | 14,547 | 6 | 6 | 6 |
+| Role (capitalised) | Occurrences | FastPDN tokens | 2.0.3 | 2.0.4 | 2.0.5 |
+|---|---:|---:|---:|---:|---:|
+| Odwołujący | 4,171 | 1,361 | 1,361 | 0 | **0** |
+| Przystępujący | 534 | 164 | 164 | 0 | **0** |
+| Zamawiający | 12,631 | 16 | 16 | 1 | **0** |
+| Skarżący | 50 | 3 | 3 | 0 | **0** |
+| Wykonawca, Prezes, Inwestor | 14,547 | 6 | 6 | 6 | **0** |
 
-The other nine roles tracked (wnioskodawca, uczestnik, powód, pozwany, przewodniczący, protokolant, pełnomocnik, oferent, protestujący) have no mask. The one 2.0.4 `Zamawiający` is a truncated `Zamawiając`. Both versions mask all 69,834 other words FastPDN tags, so the filter removes only role nouns; name recall is the name test above. The texts hold real names and are not released.
+The other nine roles tracked (wnioskodawca, uczestnik, powód, pozwany, przewodniczący, protokolant, pełnomocnik, oferent, protestujący) have no mask in any version. The one 2.0.4 `Zamawiający` is a truncated `Zamawiając`; it was known from this table before 2.0.5 added the bare stem, so its 1 → 0 is not a held-out result. Every version masks all 69,834 other words FastPDN tags, so the filter removes only role nouns; name recall is the name test above. The bench holds no tagged number after a role, so the 2.0.5 number rule is measured on UZP rulings (`CHANGELOG.md`). The texts hold real names and are not released.
 
 **Cost:** on one RTX 4090 with 3 processes, names took 2.0.0 from 74,520 to 66,492 chars/s (−11%) and peak device memory from 9,006 to 15,416 MiB; with one process, −9%. Local measurements are in `CHANGELOG.md`.
 
@@ -125,7 +126,7 @@ The other nine roles tracked (wnioskodawca, uczestnik, powód, pozwany, przewodn
 
 ## Compared with other systems
 
-Same split and gold. Every model runs through the NERGAL recipe with names off: its spans at 0.95, phone spans cut by phone policy v3, unioned with the rules. The rows were scored with the 1.2.0 rules; 2.0.0 through 2.0.4 give the same rule spans on this split. **Naked** is the model alone. **Residual** is passages with any gold character left unmasked. Character scores are gold vs masked characters.
+Same split and gold. Every model runs through the NERGAL recipe with names off: its spans at 0.95, phone spans cut by phone policy v3, unioned with the rules. The rows were scored with the 1.2.0 rules; 2.0.0 through 2.0.5 give the same rule spans on this split. **Naked** is the model alone. **Residual** is passages with any gold character left unmasked. Character scores are gold vs masked characters.
 
 | System | Mode | Whole /315 | Residual | False chars | Char P | Char R |
 |---|---|---:|---:|---:|---:|---:|
@@ -135,9 +136,9 @@ Same split and gold. Every model runs through the NERGAL recipe with names off:
 | Fine-tuned GLiNER (previous incumbent) | naked | 263 | 28 | 15 | 99.70% | 88.80% |
 | Fine-tuned GLiNER (previous incumbent) | ∪ regex | 299 | 12 | 39 | 99.30% | 97.18% |
 | XLM-R epoch 5 | naked | 275 | 28 | 67 | 98.72% | 90.25% |
-| **NERGAL 2.0.4, names off** | **∪ regex** | **303** | **9** | **80** | **98.59%** | **97.83%** |
+| **NERGAL 2.0.5, names off** | **∪ regex** | **303** | **9** | **80** | **98.59%** | **97.83%** |
 
-Zero-shot [GLiNER 2.5](https://huggingface.co/fastino/gliner2.5-multi-v1) is not competitive, especially on non-phone PII (7/185 whole, naked). The fine-tuned GLiNER that NERGAL replaced is the more precise union (39 false characters vs 80) and the stronger phone model (128/130 whole, naked, vs 115), but it masks 4 fewer values whole and leaves 12 residual passages vs 9; XLM-R is the stronger model on other PII (160/185 vs 135). NERGAL 2.0.4 wholly masks 124/130 phones and 179/185 other PII, each under its own placeholder (`[PHONE]` / `[PII]`); exact-span precision 91.05%, recall 93.65%, F1 92.33%.
+Zero-shot [GLiNER 2.5](https://huggingface.co/fastino/gliner2.5-multi-v1) is not competitive, especially on non-phone PII (7/185 whole, naked). The fine-tuned GLiNER that NERGAL replaced is the more precise union (39 false characters vs 80) and the stronger phone model (128/130 whole, naked, vs 115), but it masks 4 fewer values whole and leaves 12 residual passages vs 9; XLM-R is the stronger model on other PII (160/185 vs 135). NERGAL 2.0.5 wholly masks 124/130 phones and 179/185 other PII, each under its own placeholder (`[PHONE]` / `[PII]`); exact-span precision 91.05%, recall 93.65%, F1 92.33%.
 
 XLM-R and epoch 5 were chosen in September 2026, on the earlier gold and before the phone-policy cut. Fine-tuned GLiNER, HerBERT-large and XLM-R-large were trained on the same split; only XLM-R passed the content-preservation gate, and epoch 5 at 0.95 (seed `202609160`) was the only one of 133 epoch/threshold points that covered more than the fine-tuned GLiNER union of that time while adding no false-mask characters it did not already make. Under the 1.2.0 rules, gold and phone cut, that no longer holds: the GLiNER union makes fewer false masks (table above).
 

hybrid.json

@@ -1,7 +1,7 @@
 {
   "full_name": "Named Entity Recognition with Grounded Additive Labels",
   "hub_id": "SlayerLab/NERGAL",
-  "version": "2.0.4",
+  "version": "2.0.5",
   "mode": "rules_union",
   "epoch": 5,
   "seed": 202609160,
@@ -54,7 +54,7 @@
     "max_length": 512,
     "stride": 128,
     "dtype": "float32",
-    "decode": "argmax, union over windows, expanded to whole words, procedural role nouns left out",
+    "decode": "argmax, union over windows, expanded to whole words, procedural role nouns (parties, representatives, offices; a number glued to one included) and a number right after one left out",
     "converted_from": {
       "model.onnx": "11924bfd20eb478ee614e5e1184536825db4747753d341af7c04b940efa84645",
       "convert.py": "a698bc993b253fc70d960aa14af37a5088afb79c77dae70557c7056ce394333e"
@@ -82,7 +82,7 @@
       "fp16": "mps",
       "int8": "cpu"
     },
-    "score_report_sha256": "161c10a872efebbea44a57af0f87004b6c443c2490240b3e41cc93ffeabd1ff8",
+    "score_report_sha256": "2e9b4ac1dfc4d672b8988454f8bd64ae4d54b78bd77fecbdc9397b28ea865a36",
     "panels": {
       "random": {
         "passages": 300,

nergal.py

@@ -17,7 +17,7 @@ import scrub_pii
 from scrub_pii import LEGACY_PHONE_TAG, PERSON_TAG, PHONE_TAG, PII_TAG
 
 HUB_ID = 'SlayerLab/NERGAL'
-VERSION = '2.0.4'
+VERSION = '2.0.5'
 GAPS = ['[PII_SPACE]', '[PII_BREAK]']
 GAP_IDS = [250002, 250003]
 BIO_LABELS = ['O', 'B-phone', 'I-phone', 'B-pii', 'I-pii']
@@ -31,12 +31,22 @@ RANK = {label: i for i, label in enumerate(TAGS)}
 PERSON_LABELS = ('PERSON', 'PERSON_F', 'PERSON_L')
 _WORD = re.compile(r"[\w'’-]")
 _PLACEHOLDER = re.compile('|'.join(map(re.escape, MARKERS)))
-# Procedural party roles the names model tags as persons (artifacts/nergal_2_0_4_role_nouns): the noun in any
-# inflection. Names policy keeps roles outside a person span.
-_PARTICIPLE = '(?:y|ego|emu|ym|ych|ymi|a|ej|ą|e)'
-ROLE_NOUNS = tuple(stem + _PARTICIPLE for stem in ('odwołując', 'przystępując', 'zamawiając', 'protestując',
-                                                  'skarżąc'))
-_ROLE_NOUN = re.compile(r'\W*(?:%s)\W*' % '|'.join(ROLE_NOUNS), re.IGNORECASE)
+# Procedural roles the names model tags as persons (artifacts/nergal_2_0_4_role_nouns,
+# artifacts/nergal_2_0_5_role_nouns): parties, representatives and offices, the noun in any inflection, the bare
+# participle stem (`Odwołując` cut short, or the adverbial participle), and a number glued to it (`Odwołujący3`,
+# `wykonawcy_1`). Names policy keeps roles outside a person span (`prezes Jan Wróbel` → `Jan Wróbel`).
+_PARTICIPLE = '(?:y|ego|emu|ym|ych|ymi|a|ej|ą|e)?'
+_CA = '(?:a|y|ę|ą|o|ów|om|ami|ach)'  # wykonawca
+_NIK = '(?:k|ka|kowi|kiem|ku|cy|ków|kom|kami|kach|czk(?:a|i|ę|ą|o|om|ami|ach)|czce|czek)'  # uczestnik, -niczka
+_NT = '(?:t|ta|towi|tem|cie|ci|tów|tom|tami|tach|tk(?:a|i|ę|ą|o|om|ami|ach)|tce|tek)'  # oferent, -entka
+ROLE_NOUNS = tuple(stem + _PARTICIPLE for stem in ('odwołując', 'przystępując', 'zamawiając', 'protestując', 'skarżąc',
+                                                  'przewodnicząc')) + (
+    'pozwan(?:y|ego|emu|ym|i|ych|ymi|a|ej|ą|e)', 'wykonawc' + _CA, 'wnioskodawc' + _CA,
+    'wnioskodawczy(?:ni|ń|niom|niami|niach)', 'pow(?:ód|od(?:a|owi|em|zie|owie|ów|om|ami|ach))',
+    'powódk(?:a|i|ce|ę|ą|o|ek|om|ami|ach)', 'inwestor(?:a|owi|em|ze|zy|ów|om|ami|ach)?',
+    'prezes(?:a|owi|em|ie|i|ów|om|ami|ach)?', 'uczestni' + _NIK, 'pełnomocni' + _NIK, 'oferen' + _NT, 'protokolan' + _NT)
+_ROLE_NOUN = re.compile(r'\W*(?:%s)_?\d*\W*' % '|'.join(ROLE_NOUNS), re.IGNORECASE)
+_NUMBER = re.compile(r'\W*\d+\W*')  # with the punctuation a tagged token grows to, as for _ROLE_NOUN
 
 
 def sha(path):
@@ -204,12 +214,17 @@ def expand(text, start, end):
 
 def person_spans(text, ranges):
     """Person spans from FastPDN person-token ranges: whole words, joined across spaces, tabs and no-break spaces
-    (not punctuation or line breaks); words touching a placeholder and procedural role nouns (ROLE_NOUNS) are
-    skipped."""
+    (not punctuation or line breaks); words touching a placeholder, role nouns (ROLE_NOUNS) and a number
+    right after a skipped role noun (`Odwołujący 1`) are skipped."""
     blocked = [m.span() for m in _PLACEHOLDER.finditer(text)]
-    merged = []
+    merged, role_end = [], None
     for start, end in sorted(expand(text, a, b) for a, b in ranges):
-        if any(start < b and a < end for a, b in blocked) or _ROLE_NOUN.fullmatch(text, start, end):
+        if any(start < b and a < end for a, b in blocked):
+            continue
+        if _ROLE_NOUN.fullmatch(text, start, end):
+            role_end = end
+            continue
+        if role_end is not None and _NUMBER.fullmatch(text, start, end) and not text[role_end:start].strip(' \t\u00a0'):
             continue
         if merged and not text[merged[-1][1]:start].strip(' \t\u00a0'):
             merged[-1][1] = max(merged[-1][1], end)

test_nergal.py

@@ -9,11 +9,33 @@ from pathlib import Path
 
 HERE = Path(__file__).resolve().parent
 RULES_SHA = '4d0547238131908ca498a23607a7877512b2fb6022c484171334af1e604fc95a'
-# Procedural party-role nouns in every inflection the names layer leaves out, built here, not from ROLE_NOUNS
-PARTICIPLE_ENDINGS = ('y', 'ego', 'emu', 'ym', 'ych', 'ymi', 'a', 'ej', 'ą', 'e')
-ROLE_NOUN_FORMS = tuple(stem + e for stem in ('Odwołując', 'Przystępując', 'Zamawiając', 'Protestując', 'Skarżąc')
-                        for e in PARTICIPLE_ENDINGS)
-STEM_NAMES = ('Odwołującyński', 'Zamawiającek', 'Skarżącki')
+# Party and office role nouns in every inflection the names layer leaves out, built here, not from ROLE_NOUNS; the
+# empty participle ending is the bare stem (2.0.5)
+PARTICIPLE_ENDINGS = ('', 'y', 'ego', 'emu', 'ym', 'ych', 'ymi', 'a', 'ej', 'ą', 'e')
+CA = ('a', 'y', 'ę', 'ą', 'o', 'ów', 'om', 'ami', 'ach')
+FEMININE = ('a', 'i', 'ę', 'ą', 'o', 'om', 'ami', 'ach')
+NIK = ('k', 'ka', 'kowi', 'kiem', 'ku', 'cy', 'ków', 'kom', 'kami', 'kach', 'czce', 'czek') + tuple('czk' + e for e in FEMININE)
+NT = ('t', 'ta', 'towi', 'tem', 'cie', 'ci', 'tów', 'tom', 'tami', 'tach', 'tce', 'tek') + tuple('tk' + e for e in FEMININE)
+NOUN_ENDINGS = {
+    'Wykonawc': CA,
+    'Wnioskodawc': CA,
+    'Wnioskodawczy': ('ni', 'ń', 'niom', 'niami', 'niach'),
+    'Uczestni': NIK,
+    'Pełnomocni': NIK,
+    'Oferen': NT,
+    'Protokolan': NT,
+    'Pozwan': ('y', 'ego', 'emu', 'ym', 'i', 'ych', 'ymi', 'a', 'ej', 'ą', 'e'),
+    'Powod': ('a', 'owi', 'em', 'zie', 'owie', 'ów', 'om', 'ami', 'ach'),
+    'Powódk': ('a', 'i', 'ce', 'ę', 'ą', 'o', 'ek', 'om', 'ami', 'ach'),
+    'Inwestor': ('', 'a', 'owi', 'em', 'ze', 'zy', 'ów', 'om', 'ami', 'ach'),
+    'Prezes': ('', 'a', 'owi', 'em', 'ie', 'i', 'ów', 'om', 'ami', 'ach'),
+}
+ROLE_NOUN_FORMS = tuple(stem + e for stem in ('Odwołując', 'Przystępując', 'Zamawiając', 'Protestując', 'Skarżąc',
+                                              'Przewodnicząc') for e in PARTICIPLE_ENDINGS) + tuple(
+    stem + e for stem, endings in NOUN_ENDINGS.items() for e in endings) + ('Powód',)
+STEM_NAMES = ('Odwołującyński', 'Zamawiającek', 'Skarżącki', 'Wykonawczyk', 'Wykonawcówna', 'Powodowski', 'Prezesik',
+              'Inwestorski', 'Pozwański', 'Oferentowski', 'Uczestnikowski', 'Pełnomocnikiewicz', 'Protokolantowicz',
+              'Wnioskodawczak')
 
 
 class NergalTests(unittest.TestCase):
@@ -21,7 +43,7 @@ class NergalTests(unittest.TestCase):
         from nergal import GAP_IDS, GAPS, HUB_ID, RULES_SHA as PINNED, THRESHOLD, VERSION
         card = json.loads((HERE / 'hybrid.json').read_text())
         self.assertEqual(HUB_ID, 'SlayerLab/NERGAL')
-        self.assertEqual(VERSION, '2.0.4')
+        self.assertEqual(VERSION, '2.0.5')
         self.assertEqual(card['version'], VERSION)
         self.assertEqual(card['eval']['union_fp'], 80)
         self.assertEqual(card['eval']['rules_fp'], 24)
@@ -268,6 +290,43 @@ class NergalTests(unittest.TestCase):
         for name in STEM_NAMES:  # invented surnames that start like a role noun
             self.assertEqual(persons(name), [name])
 
+    def test_person_spans_leave_a_number_right_after_a_role_noun_out(self):
+        from nergal import person_spans
+
+        def persons(text):
+            # every word tagged; punctuation is its own token, as the model tokenises it, and left untagged
+            return [text[s['start']:s['end']] for s in person_spans(text, [m.span() for m in re.finditer(r"[\w'’-]+", text)])]
+        for text in ('Odwołujący 1', 'Odwołującego 12', 'odwołujący\t3', 'Wykonawca 2', '(Odwołujący 1)'):
+            self.assertEqual(persons(text), [], text)
+        self.assertEqual(persons('Odwołujący 1 Jan Nowak'), ['Jan Nowak'])
+        self.assertEqual(persons('Jan Nowak 1'), ['Jan Nowak 1'])  # no role noun before it: the number stays
+        self.assertEqual(persons('Odwołujący, 1'), ['1'])  # punctuation between: not right after
+        self.assertEqual(persons('Odwołujący\n1'), ['1'])  # nor a line break
+        self.assertEqual(persons('Odwołujący 1a'), ['1a'])  # only a bare number
+        self.assertEqual(persons('Odwołujący 1 2'), ['2'])  # only the first
+        self.assertEqual(persons('Odwołujący Jan 1'), ['Jan 1'])  # a name between
+        self.assertEqual(persons('Odwołujący [PII] 1'), ['1'])  # a placeholder between
+        self.assertEqual(person_spans('Odwołujący 1).', [(0, 10), (11, 12), (12, 13)]), [])  # a tagged ) grows to 1)
+        self.assertEqual(person_spans('Odwołujący 1 Jan', [(11, 12), (13, 16)]),
+                         [{'start': 11, 'end': 16, 'label': 'person', 'score': 1.0}])  # role noun untagged: the number stays
+
+    def test_person_spans_leave_a_role_noun_with_a_glued_number_out(self):
+        from nergal import person_spans
+
+        def persons(text):
+            return [text[s['start']:s['end']] for s in person_spans(text, [m.span() for m in re.finditer(r"[\w'’-]+", text)])]
+        for text in ('Odwołujący3', 'odwołującego_2', 'Wykonawcy_1', 'Przystępujący_', 'Zamawiającemu12'):
+            self.assertEqual(persons(text), [], text)
+        self.assertEqual(persons('Odwołujący3 Jan Nowak'), ['Jan Nowak'])
+        self.assertEqual(persons('Odwołujący_2 1'), [])  # a number after it too
+        for text in ('Odwołujący3a', 'Odwołujący__2', 'Odwołującegozarzut', 'Powodowski2'):  # glued to more than a number
+            self.assertEqual(persons(text), [text], text)
+
+    def test_person_spans_leave_the_adverbial_participle_out(self):
+        from nergal import person_spans
+        text = 'Odwołując się do wyroku, Przystępując do sporu'
+        self.assertEqual(person_spans(text, [m.span() for m in re.finditer(r'\S+', text) if m[0][0].isupper()]), [])
+
     def test_model_phone_spans_follow_the_phone_policy(self):
         from nergal import model_keep, scrub_spans
         span = lambda text, part, label='phone', score=0.99: dict(