--- license: apache-2.0 language: - bg - cs - da - de - el - en - es - et - fi - fr - hr - hu - id - it - ja - ko - lt - lv - ms - nl - pl - pt - ro - sk - sl - sr - sv - tl - vi - zh tags: - token-classification - pii - pii-detection - named-entity-recognition - privacy - redaction - multilingual - onnx pipeline_tag: token-classification library_name: onnxruntime --- # PiiScan — multilingual PII detection with exact character offsets For any input text, PiiScan returns **every personal-data span it finds, with the exact character offsets into your original string**. It ships as a CPU-only **INT8 ONNX** model (~139 MB with tokenizer) that runs fully offline — no GPU, no network. The guarantee everything rests on: `original_text[entity.start:entity.end] == entity.value`. - **Base model:** `distilbert-base-multilingual-cased` - **Task:** token classification, BIO over 27 labels (13 entity types) - **Types:** 13 model-trained + 11 rule-only = **24 usable**; 3 experimental, 12 blocked - **Languages:** 30 (no Indic scripts — see limitations) - **Runtime:** `onnxruntime` (CPU), ~13 ms / document - **Offsets:** Unicode code points, zero-based, end-exclusive ## Results — held-out test set (policy `balanced`, exact-span match) | Type | Precision | Recall | F1 | Support | |---|---:|---:|---:|---:| | `POSTAL_CODE` | 0.997 | 0.995 | **0.996** | 2,566 | | `PAYMENT_CARD_NUMBER` | 0.995 | 0.993 | **0.994** | 1,894 | | `CITY` | 0.993 | 0.991 | **0.992** | 4,146 | | `PHONE_NUMBER` | 0.988 | 0.996 | **0.992** | 3,361 | | `EMAIL_ADDRESS` | 0.987 | 0.988 | **0.988** | 4,400 | | `STREET_ADDRESS` | 0.989 | 0.978 | **0.983** | 3,909 | | `AGE` | 0.992 | 0.962 | **0.977** | 2,991 | | `TAX_ID` | 0.990 | 0.941 | **0.965** | 1,540 | | `DRIVER_LICENSE_NUMBER` | 0.984 | 0.940 | **0.961** | 1,662 | | `PERSON_NAME` | 0.976 | 0.931 | **0.953** | 8,810 | | `DATE_TIME` | 0.980 | 0.922 | **0.950** | 9,592 | | `PASSPORT_NUMBER` | 0.977 | 0.917 | **0.946** | 1,260 | | `GOVERNMENT_ID` | 0.977 | 0.912 | **0.943** | 3,439 | | **macro-F1** | | | **0.972** | | | **micro-F1** | 0.985 | 0.954 | **0.969** | 49,570 | Scored at document level on 8,328 held-out documents, exact `[start, end)` and type match. **These are in-distribution numbers.** The test split comes from the same generator as the training data. On 127 human-annotated real court judgments in the same split, the two types those documents annotate score considerably lower: | Slice | Precision | Recall | F1 | |---|---:|---:|---:| | Same generator as training | 0.990 | 0.978 | 0.984 | | Real human-annotated documents | 0.889 | 0.641 | 0.745 | Expect the lower number on your own data. Validate in your domain before relying on it. ## Entity tiers — what is actually supported Not all types are equal, and this card says so rather than advertising one big number. | Tier | Count | Types | |---|---:|---| | **Model-trained** | 13 | `AGE`, `CITY`, `DATE_TIME`, `DRIVER_LICENSE_NUMBER`, `EMAIL_ADDRESS`, `GOVERNMENT_ID`, `PASSPORT_NUMBER`, `PAYMENT_CARD_NUMBER`, `PERSON_NAME`, `PHONE_NUMBER`, `POSTAL_CODE`, `STREET_ADDRESS`, `TAX_ID` | | **Rule-only** (checksums/format) | 11 | `AADHAAR_NUMBER`, `CRYPTO_WALLET`, `DEVICE_IDENTIFIER`, `FAX_NUMBER`, `GEO_COORDINATES`, `IBAN`, `IP_ADDRESS`, `MAC_ADDRESS`, `SOCIAL_SECURITY_NUMBER`, `SWIFT_BIC`, `VEHICLE_IDENTIFIER` | | **Experimental** (opt-in) | 3 | `COUNTRY`, `URL`, `USERNAME` | | **Blocked** (insufficient data) | 12 | `ADDRESS`, `BANK_ACCOUNT_NUMBER`, `BIOMETRIC_IDENTIFIER`, `COUNTY_DISTRICT`, `DATE_OF_BIRTH`, `HEALTH_PLAN_ID`, `LICENSE_PLATE`, `MEDICAL_RECORD_NUMBER`, `OTHER_UNIQUE_ID`, `PROVIDER_ID`, `STATE_PROVINCE`, `VOTER_ID` | Rule-only types are excluded from the model **by design**. MOD-97 validates an IBAN exactly; training a network to approximate a checksum would be strictly worse. ## Quick start (Python + onnxruntime) ```bash pip install onnxruntime tokenizers huggingface_hub numpy ``` ```python import json, numpy as np, onnxruntime as ort from huggingface_hub import hf_hub_download from tokenizers import Tokenizer REPO = "vectorsense/piiscan" sess = ort.InferenceSession(hf_hub_download(REPO, "model.int8.onnx"), providers=["CPUExecutionProvider"]) tok = Tokenizer.from_file(hf_hub_download(REPO, "tokenizer.json")) labels = json.loads(open(hf_hub_download(REPO, "labels.json"), encoding="utf-8").read()) def detect(text: str, threshold: float = 0.75) -> list[dict]: enc = tok.encode(text) ids = np.array([enc.ids], dtype=np.int64) mask = np.array([enc.attention_mask], dtype=np.int64) feeds = {i.name: (mask if "mask" in i.name else ids) for i in sess.get_inputs()} logits = sess.run(None, feeds)[0][0] e = np.exp(logits - logits.max(-1, keepdims=True)) probs = e / e.sum(-1, keepdims=True) spans, current = [], None for i, (start, end) in enumerate(enc.offsets): if start == end: # special token continue label = labels[int(probs[i].argmax())] if label == "O": current = None continue prefix, entity_type = label.split("-", 1) if prefix == "B" or current is None or current["type"] != entity_type: current = {"type": entity_type, "start": start, "end": end, "scores": [float(probs[i].max())]} spans.append(current) else: current["end"] = end current["scores"].append(float(probs[i].max())) out = [] for s in spans: score = sum(s["scores"]) / len(s["scores"]) if score >= threshold: out.append({"type": s["type"], "start": s["start"], "end": s["end"], "value": text[s["start"]:s["end"]], "score": round(score, 4)}) return out for entity in detect("Patient John Smith was seen on 01/15/2026 at 80331 München"): print(entity) # {'type': 'PERSON_NAME', 'start': 8, 'end': 18, 'value': 'John Smith', 'score': 0.8427} # {'type': 'DATE_TIME', 'start': 31, 'end': 41, 'value': '01/15/2026', 'score': 0.9948} # {'type': 'POSTAL_CODE', 'start': 45, 'end': 50, 'value': '80331', 'score': 0.9925} ``` This snippet is the **model only**, single window. It does not include the rule layer (IBAN, Aadhaar, payment-card checksums), sliding windows for long documents, or the hybrid resolver. For those, use the package below. ## Run it as a REST API (request / response service) The companion [PiiScan repo](https://github.com/palarnab) ships a FastAPI service that wraps this model with the rule layer, policy presets, nested spans, and redaction. ```bash pip install fastapi "uvicorn[standard]" onnxruntime tokenizers numpy pyyaml uvicorn service.app:app --port 5001 # auto-loads model.int8.onnx from ./models ``` ```bash curl -s http://127.0.0.1:5001/detect_pii \ -H "Content-Type: application/json" \ -d '{"texts":["Contact: jsmith@email.com or 555-123-4567"]}' ``` ```powershell # PowerShell — send UTF-8 bytes, or non-ASCII input is mangled by Windows PowerShell 5.1 $body = '{"texts":["Contact: jsmith@email.com or 555-123-4567"]}' $bytes = [Text.Encoding]::UTF8.GetBytes($body) Invoke-RestMethod -Uri http://127.0.0.1:5001/detect_pii -Method Post ` -ContentType 'application/json; charset=utf-8' -Body $bytes ``` Response fields per entity: `entitytype`, `start`, `end`, `value`, `score`, `length`, `subtype`, `source` (`rule` / `model` / `hybrid`), `policy_tags`, `parent_id`. Endpoints: `GET /`, `POST /detect_pii`, `GET /health/ready`, `GET /metadata`, `POST /redact`. ## Policy presets `strict` (0.50) / `balanced` (0.75) / `precision` (0.90) set the score threshold; override per request with `options.score_threshold`. Published results use `balanced`. Scores are **not calibrated probabilities** — 0.90 does not mean 90% correct. Rule-layer spans carry a fixed confidence by evidence strength, not a learned score. See `thresholds.json`. ## Training record Full provenance lives in this repo under `training/` (best checkpoint, configs, dataset manifest, source licences, rule-layer benchmark). Trained 3 epochs on 1,299,487 rows, 4 h 20 m on one RTX 2000 Ada. ## Limitations & licensing - **No PHI claim.** `MEDICAL_RECORD_NUMBER` and `HEALTH_PLAN_ID` are blocked for lack of data. This is not a HIPAA de-identification tool. - **No Indic support.** The corpus has 30 languages but zero Devanagari or Bengali script. Aadhaar and PAN still work inside Indic text because those formats are script-independent. - **Recall is never 1.0.** High-stakes redaction needs human review. - **Long-form real-world prose is weaker than the headline table.** Person names in legal documents are the known worst case; initials and single-letter anonymised parties are frequently missed. - **Place names can be tagged as people**, occasionally with high confidence. - CJK has a small span-alignment ceiling from wordpiece merging across entity boundaries. - Training data derived from **Ai4Privacy OpenPII 1.5M (CC-BY-4.0), © Ai Suisse SA** — https://huggingface.co/datasets/ai4privacy/pii-masking-openpii-1.5m Evaluation also uses the **Text Anonymization Benchmark** (MIT). - Only a redistributable subset of the corpus is published — see the dataset card. ## Author Arnab Pal — [LinkedIn](https://www.linkedin.com/in/pal-arnab) · [GitHub](https://github.com/palarnab)