v0.3.0 model card — sync to 12-class schema + 9×16 bench + latency

README.md

@@ -26,48 +26,55 @@ pipeline_tag: token-classification
 
 Multilingual PII detection. ONNX-exported GLiNER built on
 [`urchade/gliner_multi_pii-v1`](https://huggingface.co/urchade/gliner_multi_pii-v1)
-(mDeBERTa-v3 base + GLiNER head, ~278M params). ~1.2 GB FP32.
-12-category span output.
+(mDeBERTa-v3 base + GLiNER head, ~278M params). ~1.2 GB FP32. 12-class
+span output.
 
-**Hobby / experiment.** A nights-and-weekends project. No SLA, no
-roadmap commitments. If it helps you, great.
+🧪 **Hobby / experiment.** Nights-and-weekends project. No SLA.
 
-Companion runtime: [`nullpii`](https://www.npmjs.com/package/nullpii)
-(npm) + [`@lbroth/nullpii-gateway`](https://www.npmjs.com/package/@lbroth/nullpii-gateway)
-(Anthropic Messages API proxy).
-Source: [github.com/lBroth/nullpii](https://github.com/lBroth/nullpii).
+Attribution: this model includes **NVIDIA Nemotron-PII (CC-BY-4.0)**
+derivative content.
 
-## What gets detected
+## Two F1 columns
 
-12 categories. ML-trained (8): `account_number`, `private_address`,
-`private_date`, `private_email`, `private_person`, `private_phone`,
-`private_url`, `secret`. Zero-shot prompted + regex post-pass (4):
-`private_passport`, `private_driver_license`, `private_vehicle_id`,
-`private_geolocation`. Pure regex (2): `private_ip`, `private_mac`.
+The repo ships the raw ONNX + tokenizer. F1 depends on which runtime you pair it with.
 
-Full label table + adversarial-input coverage matrix:
-[GitHub README §What gets caught](https://github.com/lBroth/nullpii#what-gets-caught).
+| Mode | What it is | F1 leader |
+|---|---|---|
+| `nullpii-bare` | this ONNX + GLiNER decoder + 1400-char chunking. No post-processing. | clean OOD splits |
+| `nullpii` (full runtime) | npm package: this model + 70-pattern recognizer pack (AWS / GitHub / Stripe / IBAN / SSN / …) + adversarial-input preprocessor (NFKC / any-ascii / URL `%XX` / HTML entity / zero-width / spaced PII) + base64 decoder + never-PII filter + reversible vault | adversarial / token-shape PII / production round-trip |
 
-## Two runtime modes
+Both numbers published below so the model-vs-pipeline delta is explicit.
 
-This repo ships the raw ONNX + tokenizer + `gliner_config.json`. F1
-depends on which runtime you pair it with:
+## Benchmark
 
-| Mode | What it is | Best for |
-|---|---|---|
-| **`nullpii-bare`** | this ONNX + GLiNER decoder + 1400-char chunking. No post-processing. | benchmark parity with other GLiNER family models. |
-| **`nullpii`** | the npm package full runtime: adversarial-input preprocessor (NFKC + transliteration + URL/HTML decode + zero-width strip), 50+ recognizer regex pack with validators (IBAN mod-97, Luhn, VIN ISO 3779, etc.), base64 decode-then-classify, never-PII filter, boundary refinement, reversible in-memory vault. | production PII sanitization, OOD generalization. |
+Mac M5 Pro CPU, single seed, macro F1 at IoU ≥ 0.5 partial-match span scoring. Cap 5,000 / dataset (less where the dataset is smaller). `--parallel-tools 1` fair-serial. Third-party tools run bare (no nullpii post-processing on competitor rows). Full matrix CSV: [`packages/eval/published-bench/matrix.csv`](https://github.com/lBroth/nullpii/blob/main/packages/eval/published-bench/matrix.csv). Run: `packages/eval/scripts/bench_full.py`.
+
+v0.3.0 (M5 Pro CPU, full 9×16 matrix). OOD macro F1 for `nullpii` = **0.7784** (`presidio-synthetic` + `isotonic-{en,de,fr,it}-heldout` + `ai4privacy-300k-heldout` + `tab-echr`).
+
+| Dataset | n | **`nullpii`** | **`nullpii-bare`** | `nemotron-pii-raw` | `gliner-pii-large-v1` | `gliner-onnx-pii-fp32` | `deberta` | `piiranha` | `presidio` | `opf` |
+|---|---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
+| `presidio-synthetic` | 5,000 | **0.9137** | 0.8487 | 0.7154 | 0.6749 | 0.5254 | 0.5111 | 0.3853 | 0.5511 § | 0.6530 |
+| `isotonic-en-heldout` | 1,900 | 0.7197 | 0.5969 | **0.7518** | 0.6662 | 0.5485 | 0.6224 | 0.4124 | 0.4472 | 0.4095 |
+| `isotonic-de-heldout` | 2,400 | **0.7297** | 0.6191 | 0.7271 | 0.6325 | 0.5432 | 0.3969 | 0.4112 | 0.3859 | 0.4155 |
+| `isotonic-fr-heldout` | 2,800 | 0.7254 | 0.6001 | **0.7276** | 0.6663 | 0.5393 | 0.4824 | 0.4172 | 0.4042 | 0.4257 |
+| `isotonic-it-heldout` | 2,200 | **0.7395** | 0.6148 | 0.7273 | 0.6605 | 0.5519 | 0.4509 | 0.4176 | 0.4057 | 0.4420 |
+| `ai4privacy-300k-heldout` | 5,000 | **0.6966** | 0.5241 | 0.6608 | 0.4306 | 0.5131 | 0.2183 | 0.3266 | 0.4882 | 0.4630 |
+| `tab-echr` ⚠ | 127 | 0.9239 | **0.9275** | 0.6026 | 0.6346 | 0.6463 | 0.2908 | 0.3163 | 0.7761 | 0.4166 |
+| `nemotron-pii-test` ⚠ | 5,000 | 0.8063 | 0.6814 | **0.9286** ‡ | 0.7675 | 0.7352 | 0.4153 | 0.3286 | 0.4236 | 0.4005 |
+| `nullpii-internal-bench` ⚐ | 2,361 | **0.4228** | 0.3090 | 0.3065 | 0.2851 | 0.2936 | 0.1711 | 0.1669 | 0.1436 | 0.2488 |
 
-Held-out OOD macro F1 (`nullpii`): **0.7784** across `presidio-synthetic`,
-`isotonic-{en,de,fr,it}-heldout`, `ai4privacy-300k-heldout`, `tab-echr`.
+Full 16-row matrix at [github.com/lBroth/nullpii/tree/main/packages/eval/published-bench](https://github.com/lBroth/nullpii/tree/main/packages/eval/published-bench).
 
-Full 9-tool × 16-dataset matrix vs Piiranha, DeBERTa-PII, Presidio,
-Nemotron-PII, OpenAI privacy-filter, GLiNER native:
-[github.com/lBroth/nullpii/tree/main/packages/eval/published-bench](https://github.com/lBroth/nullpii/tree/main/packages/eval/published-bench).
+Legend:
+- **bold** = row max
+- ⚠ training-distribution overlap with at least one competitor in the row
+- ⚐ in-distribution for `nullpii` itself (regression cell, **not** counted in the OOD headline)
+- ‡ competitor on its own training distribution (best-case self-report)
+- § Presidio benched on its own evaluator dataset (best-case self-report)
 
 ## Latency
 
-M5 Pro CPU, Node 24, `nullpii` runtime:
+M5 Pro CPU, Node 24, `nullpii` runtime full pipeline:
 
 | Input | p50 | p95 | p99 |
 |---:|---:|---:|---:|
@@ -75,60 +82,80 @@ M5 Pro CPU, Node 24, `nullpii` runtime:
 | 1,000 chars | 95 ms | 113 ms | 114 ms |
 | 10,000 chars | 938 ms | 972 ms | 1,122 ms |
 
-Cold start (first call, includes ONNX load): ~756 ms.
+Cold start (first `sanitize()`, ONNX load included): ~756 ms.
+
+## When to pick which
+
+- **`nullpii-bare`** — clean OOD splits, raw F1 priority, integrate directly via Python / your own runtime.
+- **`nullpii` (npm full runtime)** — production LLM proxy. Token-shape PII (Stripe / IBAN / SSN / 50+ secret patterns), adversarial inputs (zero-width / base64 / URL-encoded), reversible-vault round-trip with session-bound placeholders. `npm i nullpii`.
+
+## Schema (12 classes)
+
+ML-trained (8): `account_number` · `private_address` · `private_date` · `private_email` · `private_person` · `private_phone` · `private_url` · `secret`
+
+Zero-shot prompted + regex post-pass (4): `private_passport` · `private_driver_license` · `private_vehicle_id` · `private_geolocation`
+
+Pure regex post-pass (2): `private_ip` · `private_mac` (GLiNER head not trained on them).
+
+## Intended use
+
+- Pre-LLM PII redaction for prompts / RAG corpora / log scrubbing.
+- Span-level PII tagging for batch redaction.
+- Geographic scope: EU + Romance + English. Limited coverage outside.
+
+## Out-of-scope
 
-## Usage
+- **GDPR Article 9 special categories** (health, biometric, genetic, religious, political, trade-union, sexual orientation). Not represented in the schema.
+- **HIPAA PHI** — `account_number` catches MRN-shaped digit runs but the model is not a HIPAA de-identifier.
+- **CJK / RTL / Indic scripts** — limited coverage; treat as out-of-scope.
+- **Air-gapped first-run** — point at a local mirror via `NULLPII_MODEL_DIR` or `modelDir` config.
+
+## Limitations
+
+- Adversarial robustness comes from the npm runtime pipeline, not the model alone. The bare-model column does not include the adversarial preprocessor or the recognizer pack.
+- Long-input chunking at 512-token boundaries (npm word-chunker 140 words / 30 overlap; bare GLiNER chunker 1400 / 200 chars). Boundary spans dedupe via IoU.
+- `nullpii-bench` is in-distribution for the project pipeline — treat as regression test, not OOD claim.
+
+## How to use
+
+### npm (production path)
 
 ```bash
 npm install nullpii onnxruntime-node
 ```
 
 ```ts
-import { sanitize, restore } from 'nullpii';
-
-const safe = await sanitize('Email John Smith at john@acme.io');
-// → 'Email {{PII_PRIVATE_PERSON_0_…}} at {{PII_PRIVATE_EMAIL_0_…}}'
-
-// ... your LLM call (OpenAI, Anthropic, Gemini, anything) ...
+import { sanitize, restore, wrapForLLM } from 'nullpii';
 
+const safe = await sanitize('Email John Smith at john@acme.io about SSN 123-45-6789');
+const prompt = wrapForLLM(safe, 'Translate to Italian');
+// … LLM call …
 const back = restore(reply, safe.sessionId);
-// → original text restored
 ```
 
-The npm package downloads this model on first `sanitize()` into
-`~/.cache/nullpii/`. Pre-warm with `npx nullpii prefetch`.
+First call downloads the artifacts here into `~/.cache/nullpii/`. Pre-warm with `npx nullpii prefetch`.
 
-## Direct ONNX (no runtime)
+### Python (bare model)
 
-```py
-# Python — minimum viable: tokenizer + ONNX inference + decoder
+```python
 from gliner import GLiNER
-m = GLiNER.from_pretrained('lBroth/nullpii', load_onnx_model=True)
-spans = m.predict_entities(
-    text='Email John at john@acme.io',
-    labels=['private_person', 'private_email'],
-)
+m = GLiNER.from_pretrained("lBroth/nullpii", load_onnx_model=True)
+labels = [
+    "account_number", "private_address", "private_date", "private_email",
+    "private_person", "private_phone", "private_url", "secret",
+    # zero-shot prompted (recall lower; pair with regex pack in production)
+    "private_passport", "private_driver_license",
+    "private_vehicle_id", "private_geolocation",
+]
+m.predict_entities("Email John at john@acme.io", labels, threshold=0.5)
 ```
 
-## Limitations
-
-- Detector misses (no model is 100% accurate).
-- Not a HIPAA de-identifier — diagnoses, ICD codes, dosages,
-  biometric / genetic identifiers are out of scope.
-- `private_ip` / `private_mac` come from the regex pack, not the model.
-- Detection is best-effort. Defence in depth, not the sole privacy
-  control.
-
-## Attribution
+## License
 
-- Base model: [`urchade/gliner_multi_pii-v1`](https://huggingface.co/urchade/gliner_multi_pii-v1)
-  (GLiNER, Zaratiana et al., NAACL 2024) — Apache-2.0.
-- mDeBERTa-v3 base — MIT.
-- Includes Nemotron-PII derivative content — CC-BY-4.0.
+Apache-2.0. Combined upstream attribution: Apache-2.0 (base model) + CC-BY-4.0 (Nemotron-PII derivative content — attribution required, see header). Commercial redistribution permitted subject to the Nemotron attribution.
 
-See [`NOTICE`](https://github.com/lBroth/nullpii/blob/main/NOTICE) for the full
-attribution + license posture.
+## Citation
 
-## License
+> nullpii contributors (2026). *nullpii — multilingual PII detection.* https://huggingface.co/lBroth/nullpii
 
-Apache-2.0.
+Built on [`urchade/gliner_multi_pii-v1`](https://huggingface.co/urchade/gliner_multi_pii-v1) (Zaratiana et al., NAACL 2024).