pvv5385/pii-detector

PII Detection Model

A token-classification (NER) model that detects personally identifiable information in free text. Fine-tuned from microsoft/deberta-v3-base (183.9M parameters) over 56 entity types.

On an independent holdout it scores micro F1 0.7148 across all shipped labels — but that single number hides a wide spread, from 29 labels above 0.70 down to 7 that do not work. The per-label table below is the one to read before relying on any specific label.

Model Details

Field	Value
Base model	microsoft/deberta-v3-base (MIT)
Task	Token classification (BIO NER)
Parameters	183,921,525 (183.9M)
Released weights dtype	F16
Output labels	117 (BIO over 58 types, 56 shipped)
Max sequence length	512 tokens
Training examples	44,814 train / 3,915 val, from a 51,101-row corpus
Epochs	8
Learning rate	1e-06
Batch size	16
Training precision	fp32
Final validation F1	0.9869 (in-domain — see Metrics)
License	Apache 2.0

Metrics

Evaluated on 43,501 English rows of an independent, publicly available PII corpus (137,093 annotated entities), which contributed zero rows to training. Entity-level strict scoring: a prediction counts only if both its span boundaries and its label are exactly right.

Metric	Value
Micro F1 (shipped labels)	0.7148
Macro F1 (over the 53 labels this holdout has ground truth for)	0.6606
Micro F1, excluding labels the holdout cannot score	0.7228
In-domain validation F1 (own test split)	0.9869

The in-domain number is not a performance claim. 0.987 on our own split versus 0.7148 on held-out data is the gap between fitting the generators that produced the training data and generalizing to text written by someone else. The holdout number is the honest one; the in-domain number is reported only so the size of that gap is visible.

Comparison with other PII models

Scored on the Text Anonymization Benchmark — 127 real European Court of Human Rights judgments with 5,424 human-annotated entities. TAB is evaluation-only for every model in this table: none of them trained on it, so unlike most published PII numbers this is not a model being graded on its own coursework.

Each model's own taxonomy is collapsed onto TAB's 8 categories by the same harness, and all of them are decoded and scored identically. Entity-level strict scoring, as everywhere else in this card.

Model	Micro F1	Macro F1
This model	0.6194	0.2683
This model, previous release	0.5048	0.2112
Isotonic/deberta-v3-base_finetuned_ai4privacy_v2	0.4570	0.1719
LiquidAI/LFM2.5-Encoder-350M-PII-Detector	0.3597	0.2407
llm-semantic-router/mmbert32k-pii-detector-merged	0.0700	0.0381

Per category

Micro F1 alone would be misleading here, so the categories are broken out. n is the number of gold entities in that category.

Model	PERSON n=987	DATETIME n=2,468	LOC n=391	ORG n=653	CODE n=329	DEM n=233	QUANTITY n=171	MISC n=192
This model	0.642	0.889	0.408	0.109	0.018	0.080	0.000	0.000
This model, previous release	0.254	0.857	0.404	0.092	0.034	0.047	0.000	0.000
Isotonic/deberta-v3-base_finetuned_ai4privacy_v2	0.184	0.821	0.270	0.047	0.013	0.028	0.013	0.000
LiquidAI/LFM2.5-Encoder-350M-PII-Detector	0.807	0.000	0.645	0.474	0.000	0.000	0.000	0.000
llm-semantic-router/mmbert32k-pii-detector-merged	0.116	0.077	0.083	0.021	0.001	0.005	0.000	0.000

Read the per-category table before the micro F1. DATETIME is 2,468 of TAB's 5,424 gold entities (46%), and this model scores 0.889 on it. That single category is what carries the micro number. On PERSON, LOC and ORG the LiquidAI encoder is the stronger detector — it scores 0 on DATETIME only because its taxonomy has no date label at all, which the micro column punishes and the macro column largely does not. The two macro numbers are much closer than the two micro numbers, and that gap is the honest description of the difference.

Four categories are near zero for this model (CODE, DEM, QUANTITY, MISC). Part of that is unreachable by construction: TAB's ORG and DEM cover courts, ministries and nationality, for which this taxonomy has no corresponding label, so no amount of model quality would score them. The rest is a real limitation on legal prose.

Do not read this against the human ceiling. TAB reports an inter-annotator agreement of 0.5443 — trained annotators agree with each other only that often on exact spans. That number is not comparable to the F1 column above: it is Jaccard over TAB's full 56-type taxonomy, while these are micro F1 over an 8-way collapse. A higher number here does not mean this model exceeds human agreement.

Decode convention matters more than it should. Models that emit B- on continuation subwords fragment every multi-token entity under plain BIO decoding, and exact-span scoring charges that twice — once as a miss, once as a false positive. The table above merges adjacent same-label spans for every model, which is the only way the comparison means anything.

Model	Plain decode	Merged decode
This model	0.4362	0.6194
This model, previous release	0.4191	0.5048
Isotonic/deberta-v3-base_finetuned_ai4privacy_v2	0.3320	0.4570
LiquidAI/LFM2.5-Encoder-350M-PII-Detector	0.3557	0.3597
llm-semantic-router/mmbert32k-pii-detector-merged	0.0498	0.0700

The convention changes the ranking, not just the magnitudes — LiquidAI/LFM2.5-Encoder-350M-PII-Detector and Isotonic/deberta-v3-base_finetuned_ai4privacy_v2 trade places between the two columns. A plain-decode leaderboard would order these models differently from a merged-decode one, which is the whole reason both are published.

Where this model loses

On the primary holdout in the Metrics section above, Isotonic/deberta-v3-base_finetuned_ai4privacy_v2 scores 0.7551 against this model's 0.7096 — same 43,501 rows, same merged-decode convention, a real loss of 0.0455. (This model's 0.7096 differs slightly from the headline figure above because that one is restricted to shipped labels; both models are scored here over all labels, which is the only way the two are comparable.)

It is reported here rather than omitted, with the reason it is not the number this card leads with: that corpus is the other model's own training data. A model measured on the set it was fine-tuned on is not doing the same task as a model seeing it for the first time, and the gap above is the expected size of that advantage, not evidence about either model's behaviour on text it has not seen. TAB is uncontaminated for every model in the table above, which is why it leads.

Entity Types

All 56 shipped labels, with per-label holdout scores. Tier is the release claim:

• STRONG (29 labels) — F1 >= 0.70 on the independent holdout.
• FAIR (20 labels) — F1 0.40-0.70. Usable with review; expect misses.
• WEAK (7 labels) — F1 0.10-0.40. Do not rely on these.

Support is the number of true entities in the holdout; a label with low support has a noisy score in either direction. Examples are drawn from this repo's own value generators.

Entity	Tier	F1	Precision	Recall	Support	Examples
URL	STRONG	0.996	0.993	0.998	2,910	https://example.com/reset-password, https://acme-corp.com/profile
EMAIL	STRONG	0.995	0.992	0.999	4,092	bianca.perez.37@gmail.com, laura.jones.127@icloud.com
GPS_COORDINATE	STRONG	0.989	0.984	0.994	1,651	-36.486405, -29.543152, -83.971175, 46.107185
TAX_NUMBER *	STRONG	0.960	0.961	0.959	7,374	75-6048680, 55-4966286
MAC_ADDRESS	STRONG	0.959	0.938	0.980	1,114	e5:48:38:84:d3:56, c2:a8:ff:4e:cc:42
IP_ADDRESS	STRONG	0.953	0.950	0.956	7,106	4273:10c6:e387:d075:06cb:af6b:9ca5:522d, 28.17.138.12
ETHEREUM_ADDRESS	STRONG	0.948	0.905	0.995	1,624	0x165760232900cfdb788a5cf79c308c8ed9538acb, 0xf7c5200f0ba57fde4020b601b48e0a8d70a0b0d6
ACCOUNT_NAME	STRONG	0.934	0.947	0.921	2,793	Tara Harris Checking, Joint Savings Account
USERNAME	STRONG	0.931	0.914	0.948	3,056	Melissa_Holland36, guest_8357
PASSPORT_NUMBER *	STRONG	0.924	0.928	0.921	5,992	674804396, 222269181
ID_CARD_NUMBER *	STRONG	0.865	0.862	0.869	9,771	BU7926938, XI8412359
SSN	STRONG	0.865	0.894	0.838	2,045	110-89-1978, 410-58-0483
DRIVER_LICENSE_NUMBER *	STRONG	0.863	0.867	0.859	8,466	B9894753, X3011706
CREDIT_CARD_ISSUER	STRONG	0.862	0.865	0.859	1,510	American Express, jcb
STREET_ADDRESS	STRONG	0.860	0.886	0.835	2,799	Joseph Ramp Apt. 4B, Deanna Fort Unit 12
GIVEN_NAME	STRONG	0.848	0.815	0.885	13,323	Marie-Claire, Alan
COMPANY_NAME	STRONG	0.827	0.858	0.799	2,502	Reed-Fitzgerald, Cameron, Flores and Sparks
SURNAME	STRONG	0.818	0.744	0.909	4,817	Lee, Johnson
BITCOIN_ADDRESS	STRONG	0.812	0.769	0.861	2,403	1jbyFnDJe5MEM5YuNPLXFjQP25B1YqAdhZ, 1x26cH9eAZtLeGqKc3XtgVoaRKmrp61JY3
PHONE_NUMBER	STRONG	0.809	0.684	0.989	2,425	(839) 263-6911, +1 (784) 911-6201
ORDINAL_DIRECTION	STRONG	0.806	0.963	0.693	1,109	Southwest, East
BANK_ACCOUNT	STRONG	0.801	0.914	0.714	2,831	37592364, 99896148
ZIP_CODE	STRONG	0.773	0.666	0.922	2,322	23226, 44307
PASSWORD	STRONG	0.766	0.887	0.674	2,622	vVx6PWvSO2, PBd4dMwVV0Ae
VEHICLE_VIN	STRONG	0.760	0.776	0.744	849	XEYDPAZWBFU9V13TC, WHX3JC1TRBH8XXEXY
DATE	STRONG	0.755	0.796	0.718	6,909	22nd February, 05/01/2015
JOB_TITLE	STRONG	0.740	0.647	0.863	2,954	Product Manager, Investor Solutions Coordinator
HEIGHT	STRONG	0.714	0.761	0.671	837	6'2", 5'9"
BIC	STRONG	0.708	0.604	0.856	793	JCVIGBPO, PIZGUSCW
CREDIT_CARD_CVV	FAIR	0.660	0.933	0.511	869	790, 923
BUILDING_NUMBER	FAIR	0.650	0.579	0.742	2,714	7199, 8606
JOB_AREA	FAIR	0.644	0.916	0.497	2,882	Factors, Finance
TITLE	FAIR	0.644	0.676	0.614	3,446	Mx., Mr.
CITY	FAIR	0.638	0.480	0.951	2,787	Port Janice, Port Michaelhaven
CREDIT_CARD	FAIR	0.637	0.478	0.951	2,574	4613 1252 8584 1440, 4674 0903 8304 4114
SEX	FAIR	0.632	0.721	0.563	2,606	M, F
VEHICLE_VRM	FAIR	0.627	0.852	0.496	950	RF15IAX, MAE-5547
STATE	FAIR	0.621	0.626	0.617	2,735	IL, NV
CURRENCY_SYMBOL	FAIR	0.595	0.762	0.488	2,678	₨, £
IBAN	FAIR	0.592	0.633	0.555	1,973	BE26992095962771, LT566041696972063678
PIN	FAIR	0.588	0.533	0.656	738	4826, 7554
LITECOIN_ADDRESS	FAIR	0.574	0.570	0.578	850	LL8jRyKS9JPYMypg26stdyyRatHykWeCXo, LNyN1WtVEqTjckB7W73UWvFB9YRKjYTAcf
AMOUNT	FAIR	0.570	0.700	0.481	3,103	77738.58, 18878.50
USER_AGENT	FAIR	0.561	0.511	0.623	2,084	Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36, Mozilla/5.0 (iPhone; CPU iPhone OS 17_4 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Mobile/15E148 Safari/604.1
PHONE_IMEI	FAIR	0.560	0.715	0.460	1,875	515094143886215, 519604210126012
COUNTY	FAIR	0.552	0.860	0.406	2,745	Maricopa County, Anglesey
TIME	FAIR	0.544	0.661	0.462	2,883	15:26:51.017, 05:34
JOB_TYPE	FAIR	0.467	0.906	0.314	2,710	Representative, Engineer
CURRENCY_NAME	FAIR	0.403	0.274	0.756	817	United States Dollar, Canadian Dollar
SECONDARY_ADDRESS	WEAK	0.368	0.512	0.287	2,307	Unit 637, Rm 853
CURRENCY_CODE	WEAK	0.362	0.500	0.284	828	EUR, CHF
EYE_COLOR	WEAK	0.359	0.453	0.297	868	Gray, Hazel
GENDER	WEAK	0.346	0.290	0.428	2,385	Multigender, Androgyne
AGE	WEAK	0.329	0.289	0.381	2,370	60, 58
MIDDLE_NAME	WEAK	0.155	0.341	0.101	2,881	Kellie, Natasha
CURRENCY	WEAK	0.102	0.419	0.058	1,908	EUR, Djiboutian Franc

* The 4 starred labels have zero ground truth in the primary holdout — it does not annotate them, so their F1 there would be 0.00 by construction no matter how well the detector works. Their scores above come from a second, never-trained holdout of 20,000 rows from a different public PII corpus that does annotate them. Because those are different rows, these numbers are not included in the micro F1 above.

Intended Use

Built for PII detection and redaction assistance in text pipelines: flagging candidate spans before storage, filtering user input, anonymizing logs — with a human or a second check in the loop.

Not a compliance guarantee. It misses PII and it invents PII. At 0.7148 micro F1, roughly one in four entity decisions on held-out text is wrong in one direction or the other. Do not make it the only thing standing between personal data and disclosure, and do not present its output as evidence that a document has been cleared.

Known Limitations

7 labels do not work well enough to rely on (CURRENCY, MIDDLE_NAME, AGE, GENDER, EYE_COLOR, CURRENCY_CODE, SECONDARY_ADDRESS). They are shipped because they are part of the taxonomy and occasionally right, not because they are usable. They cluster on soft and demographic attributes, where the boundary of what counts as the entity is genuinely ambiguous and annotators disagree with each other as much as the model disagrees with them.

Some labels are false-positive heavy. Precision below 0.50 means more than half of what they flag is not the entity claimed:

Entity	Precision	Recall
CURRENCY_NAME	0.274	0.756
AGE	0.289	0.381
GENDER	0.290	0.428
MIDDLE_NAME	0.341	0.101
CURRENCY	0.419	0.058
EYE_COLOR	0.453	0.297
CREDIT_CARD	0.478	0.951
CITY	0.480	0.951

CURRENCY_NAME, CREDIT_CARD, CITY trade precision for coverage — they still catch most true entities. That is the safer direction for redaction, but not for anything that acts on a detection (routing, blocking, alerting).

AGE, GENDER, MIDDLE_NAME, CURRENCY, EYE_COLOR are not even that trade: low precision and low recall, so they both miss the entity and misfire elsewhere.

2 labels are retired and must be filtered out (MASKED_NUMBER, US_DRIVER_LICENSE). They remain in the weights' id2label — retiring them by renumbering would invalidate the checkpoint — so raw pipeline() output can still contain them. PIIPredictor drops them for you; if you use the model directly, filter them yourself (see Usage). MASKED_NUMBER was retired for poor quality; US_DRIVER_LICENSE because no external dataset annotates it, so nothing independent can confirm it generalizes.

English only. The base checkpoint is the English-vocabulary deberta-v3-base and the corpus is English; other languages are not supported and were not measured.

Synthetic training data bounds generalization. No real PII was used, which is a deliberate privacy choice with a real cost: formats absent from the generators and the two supplementary corpora are unlikely to be recognized. Expect degradation on non-US formats, domain jargon that resembles identifiers, and creatively obfuscated PII.

Placeholder-looking values are suppressed, and the rule is not the obvious one. support@example.com and (415) 555-0132 are not detected, while sarah.kim@example.com and (415) 203-0132 in the same sentences are — and so is sarah@example.com. It is not the reserved domain or the reserved exchange by itself that suppresses a detection, but its pairing with a generic, role-style local part or number. This is the hard-negative training working as designed on placeholder-looking values, but it has a sharp edge: in text dense with such values, entities that are not placeholders can be suppressed along with them. Synthetic test fixtures built from example.com and 555 numbers will therefore understate the model — evaluate on realistic values.

Partially masked PII is inconsistently handled. Masked numeric identifiers are left alone as intended (SSN ending in 1234, ***-**-1234, Card ending ****-1234). Masked emails are not: j***@gmail.com is tagged EMAIL, despite the training data treating it as a negative. Do not rely on either behaviour — if your use case turns on masked PII, test it against your own data.

Long documents need windowing, and only PIIPredictor does it for you. The model reads 512 tokens at a time. PIIPredictor.predict() slides overlapping windows across longer text (stride=64) and warns when it does, so entities past the limit are still found — though one straddling a window boundary may still be missed or duplicated. Raw pipeline() has no such handling: it truncates, and everything past the limit is silently invisible. Split the input yourself if you go that route.

Training Data

Synthetic data only — a 51,101-row corpus, no real personal information. That claim rests on each source's own statement about its contents, not on an assumption:

Source	Rows	License	Attribution
openai/gpt-oss-20b (run locally to generate text)	37,845	Apache-2.0	OpenAI
ai4privacy/pii-masking-openpii-1m	6,279	CC-BY-4.0	Ai4Privacy / Ai Suisse SA, © 2026 Ai Suisse SA
nvidia/Nemotron-PII	6,227	CC-BY-4.0	NVIDIA Corporation
generate_long_documents.py (this repository — template prose, Faker-backed PII)	750	Apache-2.0	This repository

• The LLM never invents PII. It writes carrier text containing literal {ENTITY_TYPE} placeholders; every PII value is spliced in afterwards from Faker-backed generators, which is also why the character offsets are exact rather than substring-matched.
• Ai4Privacy states its dataset "contains synthetic PII only — no real personal data is included."
• NVIDIA describes Nemotron-PII as "fully synthetic and designed to avoid real personal data" (NeMo Data Designer over Census-grounded personas).
• 4,717 examples contain no PII at all, alongside distractors (order numbers, SKUs, mnemonic phone words, partial SSNs, obfuscated emails), to teach the model what is not PII.

The two CC-BY-4.0 sources are attributed here and in the repository's NOTICE file, as that license requires.

Evaluation data

Headline metrics come from an independent, publicly available PII corpus that contributed zero rows to training. Overlap was checked directly rather than assumed: 0 of the 51,101 corpus rows share exact text with the 43,501 English holdout rows.

That corpus carries a custom tiered license prohibiting redistribution, so it is not included in this repository or in any published artifact — neither its rows nor anything derived from them.

Usage

Some placeholder-looking values are deliberately suppressed by the hard-negative training: support@example.com and (415) 555-0132 return nothing, while sarah.kim@example.com and (415) 203-0132 are detected. Use realistic values when testing — see Known Limitations.

Python API

from pii_model.infer import PIIPredictor

predictor = PIIPredictor("models/best_model")
results = predictor.predict(
    "Contact Sarah Kim at sarah.kim@example.com, phone (415) 203-0132"
)
for ent in results:
    print(f"[{ent['label']}] {ent['text']!r} ({ent['start']}:{ent['end']})")

HuggingFace pipeline

Raw pipeline() output includes the retired labels, which PIIPredictor would have filtered — so filter them:

from transformers import pipeline

RETIRED = {'MASKED_NUMBER', 'US_DRIVER_LICENSE'}

pipe = pipeline(
    "token-classification",
    model="pvv5385/pii-detector",
    aggregation_strategy="simple",
)
results = [
    e for e in pipe("Contact John Reyes at john.reyes@acme-corp.com")
    if e["entity_group"] not in RETIRED
]

CLI

python -m src.pii_model.infer --text "Contact John Reyes at john.reyes@acme-corp.com"
python -m src.pii_model.infer --text "..." --json

License

Apache 2.0 — see LICENSE for the full text and NOTICE for third-party attributions.

The base model, microsoft/deberta-v3-base, is MIT-licensed; these weights are a fine-tune of it and therefore a derivative work. Every upstream license in the chain — MIT for the base model, Apache-2.0 for the text generator, CC-BY-4.0 for the two supplementary datasets — permits commercial use and redistribution with attribution, which is what allows this model to be released under Apache 2.0.

Disclaimer

Provided "as is", without warranty of any kind. PII detection is inherently imperfect and this model is measurably imperfect at it; the authors are not responsible for consequences arising from its use. It should never be the sole mechanism protecting personal data.