Release 1.0.1: prefix-only glued-email trim.

Same epoch-5 weights; 841-dev union FP 133→123 with no gold loss.

Co-authored-by: Cursor <cursoragent@cursor.com>

CHANGELOG.md

@@ -0,0 +1,24 @@
+# NERGAL versions
+
+Semver for this island:
+
+- **MAJOR** — API, threshold, or weight recipe changes
+- **MINOR** — new capability, same API
+- **PATCH** — rules or card fix, same weights and API
+
+Accuracy is 841-dev, union at 0.95, 354 gold spans. A version that changes those numbers must update `hybrid.json` `eval` and the tables below.
+
+## 1.0.1
+
+Prefix-only glued-email trim. Cluster-gated title-case 5–11 letter prefixes are dropped from the redaction when the remainder is already a lowercase-local email. Same epoch-5 weights. Lost gold 0. New clean-passage damage 0.
+
+| Version | Whole /354 | Residual | Rules FP | Union FP | Char P | Char R | What changed |
+|---|---:|---:|---:|---:|---:|---:|---|
+| 1.0.0 | 323 | 25 | 133 | 133 | 97.76% | 95.95% | First Hub snapshot |
+| 1.0.1 | 323 | 25 | 98 | 123 | 97.93% | 95.95% | Prefix-only glued-email trim |
+
+Rules SHA `4dcc441c…`. Seven emails trimmed, 35 prefix characters dropped; the model still covers 25 of those positions. Suffix glue is unchanged.
+
+## 1.0.0
+
+First Hub snapshot. Seed `202609160`, epoch 5, rules `547c0428…`. Union FP 133 was the glued-email regex floor. This seed added no new false characters versus the historical incumbent.

README.md

@@ -13,7 +13,7 @@ tags:
 - hybrid
 ---
 
-# NERGAL
+# NERGAL 1.0.1
 
 **Named Entity Recognition with Grounded Additive Labels**
 
@@ -23,10 +23,20 @@ SlayerLab hybrid PII cleaner for Polish. Not a chat model. Not a drop-in `pipeli
 
 Python rules do the identifiers they can prove. A transformer NER head adds phone and other PII spans the regex misses. The cleaner **unions** the two on the original text, then replaces hits with `[Telefon]` or `[PII]`.
 
-- **Ground:** frozen `scrub_pii` regex
+- **Version:** `1.0.1` (`hybrid.json`, `CHANGELOG.md`)
+- **Ground:** `scrub_pii` regex (SHA256 `4dcc441c…`)
 - **Additive labels:** XLM-RoBERTa-large token classifier, BIO tags `phone` / `pii`, threshold 0.95
 - **This snapshot:** seed `202609160`, **epoch 5** of a seven-epoch schedule
 
+## Versions
+
+841-dev, union at 0.95, 354 gold spans. Same weights and API is a patch; new capability is minor; API, threshold, or weight recipe is major. A release that changes these numbers updates `hybrid.json` `eval` and this table.
+
+| Version | Whole /354 | Residual | Rules FP | Union FP | Char P | Char R | What changed |
+|---|---:|---:|---:|---:|---:|---:|---|
+| 1.0.0 | 323 | 25 | 133 | 133 | 97.76% | 95.95% | First Hub snapshot |
+| **1.0.1** | **323** | **25** | **98** | **123** | **97.93%** | **95.95%** | Prefix-only glued-email trim |
+
 ## 841-dev
 
 Tables use one development split: 841 passages, 215 with gold PII, **354 spans** (169 phone, 185 other PII). It is the `dev` side of a 4,500-passage labelled tranche (3,655 train / 841 dev). Sources match Dynaword (EUR-Lex, HPLT, Wikipedia, parliamentary and government text, plus smaller news/literary slices). Labels mix unchanged silver with human review.
@@ -57,19 +67,19 @@ Fresh XLM-R, seven epochs. 133 epoch/threshold combinations. Epoch 5 at 0.95 was
 
 ## Compared with other systems
 
-Same 841-dev split, threshold **0.95**. **Naked** is the transformer alone. **∪ regex** is that model unioned with the frozen rules (the NERGAL recipe). Character scores are gold vs masked characters.
+Same 841-dev split, threshold **0.95**. **Naked** is the transformer alone. **∪ regex** is that model unioned with the current rules. Character scores are gold vs masked characters.
 
 | System | Mode | Whole /354 | Residual | False chars | Char P | Char R |
 |---|---|---:|---:|---:|---:|---:|
-| Regex (`scrub_pii`) | rules | 245 | 73 | 133 | 97.36% | 81.01% |
+| Regex (`scrub_pii`) | rules | 245 | 73 | 98 | 98.04% | 81.01% |
 | GLiNER 2.5-multi zero-shot | naked | 81 | 188 | 970 | 56.98% | 21.22% |
 | GLiNER 2.5-multi zero-shot | ∪ regex | 262 | 63 | 1,103 | 82.36% | 85.02% |
 | Historical GLiNER email12 | naked | 249 | 70 | 10 | 99.81% | 85.39% |
 | Historical GLiNER email12 | ∪ regex | 289 | 51 | 143 | 97.54% | 93.48% |
 | XLM-R epoch 5 | naked | 298 | 42 | 57 | 98.96% | 89.38% |
-| **NERGAL (this snapshot)** | **∪ regex** | **323** | **25** | **133** | **97.76%** | **95.95%** |
+| **NERGAL 1.0.1** | **∪ regex** | **323** | **25** | **123** | **97.93%** | **95.95%** |
 
-Zero-shot [GLiNER 2.5](https://huggingface.co/fastino/gliner2.5-multi-v1) is not competitive here, especially on non-phone PII (7/185 whole vs 160 naked / 179 union for this snapshot). Fine-tuned historical GLiNER is the precise naked baseline (10 false characters) and still trails XLM-R on coverage. NERGAL is XLM-R epoch 5 plus the regex: 144/169 phone, 179/185 other PII. Exact-span precision 87.50%, recall 88.98%, F1 88.24%. The 133 false characters are the glued-email regex error; this seed does not add to that floor.
+Zero-shot [GLiNER 2.5](https://huggingface.co/fastino/gliner2.5-multi-v1) is not competitive here, especially on non-phone PII (7/185 whole vs 160 naked / 179 union). Fine-tuned historical GLiNER is the precise naked baseline (10 false characters) and still trails XLM-R on coverage. NERGAL is XLM-R epoch 5 plus the regex: 144/169 phone, 179/185 other PII. Exact-span precision 87.50%, recall 88.98%, F1 88.24%.
 
 Trained GLiNER, HerBERT-large, and XLM-R-large were also compared on this split (plot above). GLiNER’s 0.50 coverage lead is the precision collapse in that figure; no GLiNER or HerBERT operating point passed the content-preservation gate.
 
@@ -77,13 +87,15 @@ Trained GLiNER, HerBERT-large, and XLM-R-large were also compared on this split
 
 | Seed | Whole /354 | False chars | New false vs historical union |
 |---|---:|---:|---:|
-| 202609160 (this repo) | 323 | 133 | 0 |
+| 202609160 (this snapshot) | 323 | 123 | 0 |
 | 202609161 | 322 | 134 | 1 |
 | 202609162 | 316 | 151 | 18 |
 
+Seed 160 is the published weights. 161 and 162 were confirmation runs of the same recipe.
+
 ## Load
 
-This repo is the PII island: frozen `scrub_pii.py` plus `nergal.py`. `pipeline("token-classification")` will not match. Regex runs on the original text, the model adds spans at 0.95, then the two are unioned and replaced with `[Telefon]` / `[PII]`.
+This repo is the PII island: `scrub_pii.py` plus `nergal.py`. `pipeline("token-classification")` will not match. Regex runs on the original text, the model adds spans at 0.95, then the two are unioned and replaced with `[Telefon]` / `[PII]`.
 
 ```python
 from pathlib import Path
@@ -98,6 +110,6 @@ nergal = Nergal.from_pretrained(root, local_files_only=True)
 masked, counts = nergal.scrub(text)
 ```
 
-`hybrid.json` records threshold 0.95 and gap ids `250002` / `250003`. `test_nergal.py` is synthetic (no corpus text). From this snapshot: `python -m unittest test_nergal`.
+`hybrid.json` records version `1.0.1`, threshold 0.95, gap ids `250002` / `250003`, and the 841-dev `eval` block. `test_nergal.py` is synthetic (no corpus text). From this snapshot: `python -m unittest test_nergal`.
 
 Base weights: [`FacebookAI/xlm-roberta-large`](https://huggingface.co/FacebookAI/xlm-roberta-large) revision `c23d21b0620b635a76227c604d44e43a9f0ee389` (MIT).

hybrid.json

@@ -1,11 +1,29 @@
 {
   "full_name": "Named Entity Recognition with Grounded Additive Labels",
   "hub_id": "SlayerLab/NERGAL",
+  "version": "1.0.1",
   "mode": "rules_union",
   "epoch": 5,
   "seed": 202609160,
   "threshold": 0.95,
-  "rules_sha256": "547c0428b0799bf051566d6ac489987eff27f09e1bda36a5665452fe155b3966",
+  "rules_sha256": "4dcc441c39b653b1e8360928c9e36f6299c1f17e81e7566dd2d252cc5dbbb03c",
+  "eval": {
+    "split": "841-dev",
+    "gold_entities": 354,
+    "whole_entities": 323,
+    "residual_passages": 25,
+    "union_fp": 123,
+    "rules_fp": 98,
+    "character_precision": 0.9793,
+    "character_recall": 0.9595,
+    "exact_precision": 0.875,
+    "exact_recall": 0.8898,
+    "exact_f1": 0.8824,
+    "phone_whole": 144,
+    "phone_gold": 169,
+    "pii_whole": 179,
+    "pii_gold": 185
+  },
   "weights_sha256": "063ee5f9782c1b4d99e838be5a836328718e1372e213d5b87098b371b5c162af",
   "gaps": [
     "[PII_SPACE]",

nergal.py

@@ -17,12 +17,13 @@ import scrub_pii
 from scrub_pii import PHONE_TAG, PII_TAG
 
 HUB_ID = 'SlayerLab/NERGAL'
+VERSION = '1.0.1'
 GAPS = ['[PII_SPACE]', '[PII_BREAK]']
 GAP_IDS = [250002, 250003]
 BIO_LABELS = ['O', 'B-phone', 'I-phone', 'B-pii', 'I-pii']
 LABELS = ['phone', 'pii']
 THRESHOLD = 0.95
-RULES_SHA = '547c0428b0799bf051566d6ac489987eff27f09e1bda36a5665452fe155b3966'
+RULES_SHA = '4dcc441c39b653b1e8360928c9e36f6299c1f17e81e7566dd2d252cc5dbbb03c'
 
 
 def sha(path):

scrub_pii.py

@@ -11,7 +11,9 @@ valid check digit. Passport
 variants include one-letter and diplomatic IDs. Foreign IBANs use country
 lengths and MOD-97. ePUAP paths require a label; e-Doreczenia uses its AE:PL form.
 Wrapped email domains, local parts hyphenated across one line break and small
-extraction gaps around @/hyphens are supported.
+extraction gaps around @/hyphens are supported. A title-case alphabetic prefix
+of 5–11 letters is dropped from the redaction when the remainder is already a
+complete lowercase-local email and the same passage has at least two such glues.
 Explicit phone extensions and terminal suffix ranges are included; room numbers are not. Labelled numeric
 PINs (including URL pin= values) map to [PII] before phone detection.
 Contact/helpline headings cover consecutive descriptive phone-list entries;
@@ -509,9 +511,51 @@ def _phone_ok(raw: str, short: bool = False) -> bool:
             or (short and 9 <= len(d) <= 12 and d.startswith('0')))
 
 
+def _trim_glued_email_prefix(text: str, start: int, end: int) -> int:
+    """Drop a title-case letter prefix when the remainder is already a complete email.
+
+    Prefix-only: do not touch dotted locals or suffix glue. Leftmost 5–11 letters.
+    Remainder local must be lowercase letters, length at least 5.
+    """
+    frag = text[start:end]
+    at = frag.find('@')
+    if at < 0:
+        return start
+    loc = frag[:at]
+    if '.' in loc or '-' in loc or sum(c.isupper() for c in loc) != 1:
+        return start
+    for i in range(5, min(12, at)):
+        prefix, rest = frag[:i], frag[i:]
+        if not prefix.isalpha() or not (prefix[0].isupper() and prefix[1:].islower()):
+            continue
+        r_at = rest.find('@')
+        rloc = rest[:r_at] if r_at >= 0 else ''
+        if r_at < 5 or not rloc.isalpha() or not rloc.islower():
+            continue
+        if _EMAIL_RE.fullmatch(rest):
+            return start + i
+    return start
+
+
+def _glued_email_trim_starts(text: str) -> dict[int, int]:
+    """Map match starts to trimmed starts when a passage has two or more glues.
+
+    A lone title-case local is a real mailbox (Lostandfound@…). Two or more
+    CELEX-shaped prefix+mailbox tokens in one passage are column glue.
+    """
+    starts = {}
+    for pattern in (_EMAIL_WRAP_RE, _EMAIL_RE):
+        for m in pattern.finditer(text):
+            new = _trim_glued_email_prefix(text, m.start(), m.end())
+            if new != m.start():
+                starts.setdefault(m.start(), new)
+    return starts if len(starts) >= 2 else {}
+
+
 def _replace_checked(text: str, pattern: re.Pattern, tag: str, ok,
                      label: re.Pattern | None = None, *, mask=_tag) -> tuple[str, int]:
     n = 0
+    trims = _glued_email_trim_starts(text) if pattern in (_EMAIL_RE, _EMAIL_WRAP_RE) else {}
 
     def _sub(m):
         nonlocal n
@@ -522,8 +566,9 @@ def _replace_checked(text: str, pattern: re.Pattern, tag: str, ok,
         if not ok(m.group(0)) or (pattern not in (_EMAIL_RE, _EMAIL_WRAP_RE, _EMAIL_FRAGMENT_RE)
                                  and _is_amount(text, m.start(), m.end())):
             return m.group(0)
+        start = trims.get(m.start(), m.start())
         n += 1
-        return mask(m.start(), m.end(), tag)
+        return text[m.start():start] + mask(start, m.end(), tag)
 
     return pattern.sub(_sub, text), n
 

test_nergal.py

@@ -5,14 +5,18 @@ import unittest
 from pathlib import Path
 
 HERE = Path(__file__).resolve().parent
-RULES_SHA = '547c0428b0799bf051566d6ac489987eff27f09e1bda36a5665452fe155b3966'
+RULES_SHA = '4dcc441c39b653b1e8360928c9e36f6299c1f17e81e7566dd2d252cc5dbbb03c'
 
 
 class NergalTests(unittest.TestCase):
     def test_card_and_rules_hash(self):
-        from nergal import GAP_IDS, GAPS, HUB_ID, RULES_SHA as PINNED, THRESHOLD
+        from nergal import GAP_IDS, GAPS, HUB_ID, RULES_SHA as PINNED, THRESHOLD, VERSION
         card = json.loads((HERE / 'hybrid.json').read_text())
         self.assertEqual(HUB_ID, 'SlayerLab/NERGAL')
+        self.assertEqual(VERSION, '1.0.1')
+        self.assertEqual(card['version'], VERSION)
+        self.assertEqual(card['eval']['union_fp'], 123)
+        self.assertEqual(card['eval']['rules_fp'], 98)
         self.assertEqual(GAPS, card['gaps'])
         self.assertEqual(GAP_IDS, card['gap_ids'])
         self.assertEqual(THRESHOLD, card['threshold'])