Card: comparison charts, CPU figures, detect() keeps spans to one line (weights unchanged)

a293bea verified about 24 hours ago

11.9 kB

---
license: apache-2.0
base_model: FacebookAI/xlm-roberta-base
library_name: transformers
pipeline_tag: token-classification
language:
- en
- zh
tags:
- pii
- pii-detection
- de-identification
- anonymization
- redaction
- legal
- ner
datasets:
- nvidia/Nemotron-PII
- gravitee-io/pii-detection-dataset
- gretelai/synthetic_pii_finance_multilingual
- xusenlin/clue-ner
- ildpil/text-anonymization-benchmark
---
# veil-detect

**veil-detect** finds personal information in text: names, organizations, places, contact details and usernames,
ID and case numbers, financial and credential strings, dates and dates of birth. It is the span detector from
*veil*, the de-identification pipeline we use at Libra OS, released on its own under Apache-2.0.

- **Built for recall.** It proposes everything that might need masking. On three blind samples of US court opinions it
  misses 3–5% of mentions, against 21–24% for the strongest open alternative.
- **Small and fast.** 278M parameters (xlm-roberta-base), 1.9 GiB of GPU memory, about ten times the throughput of
  the GLiNER- and LLM-based detectors we compared it with. It also runs on CPU alone: about 0.6 s per 1,000
  characters on 2 cores, in 1.6 GB of RAM.
- **English and Chinese**, including mainland formats (身份证, mobile numbers, 统一社会信用代码, 案号).
- **Clean training lineage.** Permissively licensed datasets and our own template generators only. No user or chat
  logs, and no output of a third-party language model was used to label anything.

Version **1.0.0** (2026-10-21; card revised 2026-10-23, weights unchanged). Use the `v1.0.0` tag to pin the weights:
`from_pretrained("libraos/veil-detect", revision="v1.0.0")`.

## Quick start

The model reads documents in 384-token windows, the way it was trained. The helper below handles documents of any
length and returns character spans. The `transformers` token-classification pipeline is not recommended: with this
tokenizer its aggregation modes split or merge spans wrongly.

```python
import torch
from transformers import AutoTokenizer, AutoModelForTokenClassification
REPO = "libraos/veil-detect"
tok = AutoTokenizer.from_pretrained(REPO)
model = AutoModelForTokenClassification.from_pretrained(REPO).eval()
def detect(text, window=384, stride=64):
    """Character spans [{start, end, label, text}] for one document of any length."""
    enc = tok(text, add_special_tokens=False, return_offsets_mapping=True, verbose=False)
    ids, offs = enc["input_ids"], enc["offset_mapping"]
    body = window - 2; tag_at = {}
    for w0 in range(0, max(1, len(ids) - stride), body - stride):
        x = torch.tensor([[tok.cls_token_id] + ids[w0:w0 + body] + [tok.sep_token_id]])
        with torch.no_grad():
            pred = model(input_ids=x).logits[0, 1:-1].argmax(-1).tolist()
        for (a, b), p in zip(offs[w0:w0 + body], pred):
            if a < b: tag_at[(a, b)] = model.config.id2label[p]
    spans = []
    for (a, b), tag in sorted(tag_at.items()):
        if tag == "O": continue
        label = tag.split("-", 1)[1]
        prev = spans[-1] if spans else None
        # continue the previous span when the label matches and only spaces separate them (a line break ends a span)
        if prev and prev["label"] == label and not text[prev["end"]:a].strip() and "\n" not in text[prev["end"]:a]:
            prev["end"] = b
        else:
            spans.append({"start": a, "end": b, "label": label})
    for s in spans: s["text"] = text[s["start"]:s["end"]]
    return spans
email = ("Hi team,\n\nMaria Gonzalez from Brightline Hosting called about the lease at 12 Baker Street, London. "
         "She can be reached at +1 415 555 0132 or maria.gonzalez@example.com.\n\nThanks,\nOmar")
for span in detect(email):
    print(span["label"], span["text"])
```

```
person Maria Gonzalez
organization Brightline Hosting
location 12 Baker Street
location London
contact +1 415 555 0132
contact maria.gonzalez@example.com
person Omar
```

Masking is a separate decision. The model tags judges, courts, public bodies and well-known names as readily as
private ones, by design. Mask everything it returns for maximum privacy, or put your own policy or review step after it.

## Labels

| label | covers | examples |
|---|---|---|
| `person` | names, including surname-only and initials | Maria Gonzalez, Mr Kowalski, A.L.Y., 张伟 |
| `organization` | companies, firms, courts, agencies, schools | Brightline Hosting, Fla. 4th DCA |
| `location` | addresses, streets, cities, regions | 12 Baker Street, 杭州市西湖区文三路100号 |
| `contact` | e-mail, phone, URL, usernames and handles | +1 415 555 0132, kai_rivers92, wxid_maple199 |
| `id` | national IDs, passports, account, docket and application numbers | 110101199003071234, No. 4D17-248, 12345/09 |
| `financial` | card numbers, IBANs, bank accounts | DE89 3704 0044 0532 0130 00 |
| `credential` | passwords, API keys, PINs, card security codes, recovery phrases | |
| `date` | dates and times | 3 May 2009, 2024-08-06 07:27 |
| `date_of_birth` | dates that are a person's birth date | born 14 March 1987 |
| `demographic` | occupation, nationality, age, gender, religion, health conditions and similar attributes | Senior Manager, aged 42 |

Tags are BIO over these ten labels (21 classes with `O`).

## Evaluation

### Blind legal samples

![Share of personal-data mentions missed on three blind samples of US court opinions: veil-detect 4.1%, nvidia/gliner-PII 22.3%, OpenMed nemotron 41.4%, openai/privacy-filter 50.3%](https://huggingface.co/libraos/veil-detect/resolve/main/assets/blind-miss-rate.png)

Three samples of US court opinions that no training source contains, from the Caselaw Access Project. Gold is the
mentions two independent annotators agreed on. *Miss rate* counts a gold mention as found when any predicted span
overlaps it. *Must-mask recall* is the same over the mentions that identify someone directly or indirectly. veil-detect
was scored together with veil's rule layer for exact formats, as it runs in our pipeline (the rules are not part of
this release). The other detectors were scored as published.

| detector | miss rate, samples 1 / 2 / 3 | must-mask recall, samples 1 / 2 / 3 |
|---|---|---|
| **veil-detect 1.0** | **4.9% / 3.9% / 3.4%** | **0.949 / 0.955 / 0.972** |
| nvidia/gliner-PII | 21.2% / 24.2% / 21.4% | 0.844 / 0.841 / 0.832 |
| OpenMed nemotron | 42.5% / 43.5% / 38.1% | 0.701 / 0.686 / 0.703 |
| openai/privacy-filter | 55.2% / 56.3% / 39.3% | 0.618 / 0.575 / 0.715 |

Sample 1 is 1973 federal appeals (1,865 mentions), sample 2 is 1979–80 federal appeals (1,694) and sample 3 is
2017–18 state courts (1,842).

### Mixed corpus

81,864 gold mentions from the test splits of Nemotron-PII, gravitee, Gretel finance, pii-bench-zh and TAB. The model
was trained on the train splits of several of these, so read this as an in-distribution figure, not a blind one.

| | all | person | contact | address | numbers | TAB (ECHR) |
|---|---:|---:|---:|---:|---:|---:|
| miss rate | 3.2% | 6.4% | 0.4% | 2.5% | 1.0% | 2.1% |

Throughput on the same corpus, one RTX 3080 (veil with its rule layer):

![Throughput in MB/s and GPU memory in GiB on one RTX 3080: veil-detect 0.24 MB/s and 1.9 GiB; nvidia/gliner-PII 0.019 and 6.8; OpenMed nemotron 0.024 and 8.8; openai/privacy-filter 0.015 and 8.8](https://huggingface.co/libraos/veil-detect/resolve/main/assets/throughput-memory.png)

| detector | MB/s | GPU memory |
|---|---:|---:|
| **veil-detect** | **0.24** | **1.9 GiB** |
| nvidia/gliner-PII | 0.019 | 6.8 GiB |
| OpenMed nemotron | 0.024 | 8.8 GiB |
| openai/privacy-filter | 0.015 | 8.8 GiB |

### Running on CPU

No GPU is needed. Measured with the same 384-token windows as `detect()` above (batched, with veil's rule layer), fp32
PyTorch, one AMD EPYC 7402 and a 7,900-character bilingual contract, one document at a time:

| CPU cores | seconds per 1,000 characters | peak memory |
|---:|---:|---:|
| 1 | 1.07 | 1.6 GB |
| 2 | 0.57 | 1.6 GB |
| 4 | 0.42 | 1.6 GB |

A typical 1,500-character contract takes about 0.9 s on 2 cores. Set `torch.set_num_threads()` to the cores the process
is given; otherwise PyTorch sizes its thread pool to the whole machine. We do not recommend dynamic int8 quantization of
the linear layers: it cut CPU time by about a quarter, but on our contract test set the share of direct identifiers
found fell from 0.94 to 0.71.

## Intended use

- First stage of a redaction or pseudonymization pipeline over documents, support tickets, chat and logs, before text
  is stored, shared or sent to a third-party model.
- Finding personal data in a corpus for review, deletion requests or data mapping.

Not intended for:
- **Guaranteed anonymization on its own.** A 3–5% miss rate means a long document will usually contain at least one
  missed mention. For legal or regulated release, keep a human review step.
- Decisions about the people it finds. It labels spans of text; it does not identify anyone.

## Limitations

- **Give it context.** Fed one sentence at a time instead of whole documents, recall on a 1,287-mention sample falls
  from 0.95 to 0.88, most of it on phone numbers, e-mails and case numbers. Pass paragraphs or whole documents.
- **Usernames in fiction or roleplay** are found less reliably than in chat, support or forum text (0.94 recall on a
  synthetic set of out-of-character inserts).
- **Addresses in older legal prose** are the weakest class on the blind samples (7–11% missed).
- **No label for quantities or miscellaneous identifiers** (TAB's QUANTITY / MISC categories).
- **Languages.** English and Chinese are evaluated. The finance data adds French, German, Dutch, Spanish and Italian, which
  are not separately measured. Other languages rely on the base model alone.
- **Over-tagging is expected.** Judges, counsel, courts and public figures are tagged as people and organizations.

## Training

Fine-tuned from `FacebookAI/xlm-roberta-base` in stages: general PII and Chinese data from scratch, then legal text,
then a `date_of_birth` label, then the final mix (83,337 documents in 100,603 windows of 384 tokens, 8,000 steps,
batch 16, learning rate 1.5e-5, 19 minutes on one RTX 3080). Long documents are cut into overlapping windows by hand.
Upper-case augmentation (p = 0.3) covers captions like "Peter J. BRENNAN, Secretary of Labor".

| source | licence | used for |
|---|---|---|
| nvidia/Nemotron-PII | CC-BY-4.0 | English PII across industries |
| gravitee-io/pii-detection-dataset | Apache-2.0 | English PII, harmonised from several public sets |
| gretelai/synthetic_pii_finance_multilingual | Apache-2.0 | finance documents in six languages |
| CLUENER2020 (xusenlin/clue-ner) | Apache-2.0 | Chinese names, places, organizations |
| Text Anonymization Benchmark, train split | MIT | ECHR judgments |
| template generators | written for this model | Chinese forms and legal documents, US court opinions and dockets, exchange and wallet text, structured birth-date records, usernames in chat and support text, ECHR application numbers |
Blind test documents, the pii-bench-zh benchmark and every test split were excluded from training.
## Versions
| version | date | notes |
|---|---|---|
| 1.0.0 | 2026-10-21 | first public release |
| 1.0.0, card revision | 2026-10-23 | comparison charts, CPU figures; `detect()` no longer joins spans across a line break (a signature line and the role line under it came out as one name). Weights unchanged. |
## Licence and citation
Apache-2.0. The NOTICE file carries the attribution that Nemotron-PII's CC-BY-4.0 licence requires.
```bibtex
@misc{veil_detect_2026,
  title  = {veil-detect: a recall-first PII span detector},
  author = {{Libra OS}},
  year   = {2026},
  url    = {https://huggingface.co/libraos/veil-detect},
  note   = {Version 1.0.0}
}
```

	---
	license: apache-2.0
	base_model: FacebookAI/xlm-roberta-base
	library_name: transformers
	pipeline_tag: token-classification
	language:
	- en
	- zh
	tags:
	- pii
	- pii-detection
	- de-identification
	- anonymization
	- redaction
	- legal
	- ner
	datasets:
	- nvidia/Nemotron-PII
	- gravitee-io/pii-detection-dataset
	- gretelai/synthetic_pii_finance_multilingual
	- xusenlin/clue-ner
	- ildpil/text-anonymization-benchmark
	---

	# veil-detect

	veil-detect finds personal information in text: names, organizations, places, contact details and usernames,
	ID and case numbers, financial and credential strings, dates and dates of birth. It is the span detector from
	veil, the de-identification pipeline we use at Libra OS, released on its own under Apache-2.0.

	- Built for recall. It proposes everything that might need masking. On three blind samples of US court opinions it
	misses 3–5% of mentions, against 21–24% for the strongest open alternative.
	- Small and fast. 278M parameters (xlm-roberta-base), 1.9 GiB of GPU memory, about ten times the throughput of
	the GLiNER- and LLM-based detectors we compared it with. It also runs on CPU alone: about 0.6 s per 1,000
	characters on 2 cores, in 1.6 GB of RAM.
	- English and Chinese, including mainland formats (身份证, mobile numbers, 统一社会信用代码, 案号).
	- Clean training lineage. Permissively licensed datasets and our own template generators only. No user or chat
	logs, and no output of a third-party language model was used to label anything.

	Version 1.0.0 (2026-10-21; card revised 2026-10-23, weights unchanged). Use the `v1.0.0` tag to pin the weights:
	`from_pretrained("libraos/veil-detect", revision="v1.0.0")`.

	## Quick start

	The model reads documents in 384-token windows, the way it was trained. The helper below handles documents of any
	length and returns character spans. The `transformers` token-classification pipeline is not recommended: with this
	tokenizer its aggregation modes split or merge spans wrongly.

	```python
	import torch
	from transformers import AutoTokenizer, AutoModelForTokenClassification

	REPO = "libraos/veil-detect"
	tok = AutoTokenizer.from_pretrained(REPO)
	model = AutoModelForTokenClassification.from_pretrained(REPO).eval()


	def detect(text, window=384, stride=64):
	"""Character spans [{start, end, label, text}] for one document of any length."""
	enc = tok(text, add_special_tokens=False, return_offsets_mapping=True, verbose=False)
	ids, offs = enc["input_ids"], enc["offset_mapping"]
	body = window - 2; tag_at = {}
	for w0 in range(0, max(1, len(ids) - stride), body - stride):
	x = torch.tensor([[tok.cls_token_id] + ids[w0:w0 + body] + [tok.sep_token_id]])
	with torch.no_grad():
	pred = model(input_ids=x).logits[0, 1:-1].argmax(-1).tolist()
	for (a, b), p in zip(offs[w0:w0 + body], pred):
	if a < b: tag_at[(a, b)] = model.config.id2label[p]
	spans = []
	for (a, b), tag in sorted(tag_at.items()):
	if tag == "O": continue
	label = tag.split("-", 1)[1]
	prev = spans[-1] if spans else None
	# continue the previous span when the label matches and only spaces separate them (a line break ends a span)
	if prev and prev["label"] == label and not text[prev["end"]:a].strip() and "\n" not in text[prev["end"]:a]:
	prev["end"] = b
	else:
	spans.append({"start": a, "end": b, "label": label})
	for s in spans: s["text"] = text[s["start"]:s["end"]]
	return spans


	email = ("Hi team,\n\nMaria Gonzalez from Brightline Hosting called about the lease at 12 Baker Street, London. "
	"She can be reached at +1 415 555 0132 or maria.gonzalez@example.com.\n\nThanks,\nOmar")
	for span in detect(email):
	print(span["label"], span["text"])
	```

	```
	person Maria Gonzalez
	organization Brightline Hosting
	location 12 Baker Street
	location London
	contact +1 415 555 0132
	contact maria.gonzalez@example.com
	person Omar
	```

	Masking is a separate decision. The model tags judges, courts, public bodies and well-known names as readily as
	private ones, by design. Mask everything it returns for maximum privacy, or put your own policy or review step after it.

	## Labels

	\| label \| covers \| examples \|
	\|---\|---\|---\|
	\| `person` \| names, including surname-only and initials \| Maria Gonzalez, Mr Kowalski, A.L.Y., 张伟 \|
	\| `organization` \| companies, firms, courts, agencies, schools \| Brightline Hosting, Fla. 4th DCA \|
	\| `location` \| addresses, streets, cities, regions \| 12 Baker Street, 杭州市西湖区文三路100号 \|
	\| `contact` \| e-mail, phone, URL, usernames and handles \| +1 415 555 0132, kai_rivers92, wxid_maple199 \|
	\| `id` \| national IDs, passports, account, docket and application numbers \| 110101199003071234, No. 4D17-248, 12345/09 \|
	\| `financial` \| card numbers, IBANs, bank accounts \| DE89 3704 0044 0532 0130 00 \|
	\| `credential` \| passwords, API keys, PINs, card security codes, recovery phrases \| \|
	\| `date` \| dates and times \| 3 May 2009, 2024-08-06 07:27 \|
	\| `date_of_birth` \| dates that are a person's birth date \| born 14 March 1987 \|
	\| `demographic` \| occupation, nationality, age, gender, religion, health conditions and similar attributes \| Senior Manager, aged 42 \|

	Tags are BIO over these ten labels (21 classes with `O`).

	## Evaluation

	### Blind legal samples

	![Share of personal-data mentions missed on three blind samples of US court opinions: veil-detect 4.1%, nvidia/gliner-PII 22.3%, OpenMed nemotron 41.4%, openai/privacy-filter 50.3%](https://huggingface.co/libraos/veil-detect/resolve/main/assets/blind-miss-rate.png)

	Three samples of US court opinions that no training source contains, from the Caselaw Access Project. Gold is the
	mentions two independent annotators agreed on. Miss rate counts a gold mention as found when any predicted span
	overlaps it. Must-mask recall is the same over the mentions that identify someone directly or indirectly. veil-detect
	was scored together with veil's rule layer for exact formats, as it runs in our pipeline (the rules are not part of
	this release). The other detectors were scored as published.

	\| detector \| miss rate, samples 1 / 2 / 3 \| must-mask recall, samples 1 / 2 / 3 \|
	\|---\|---\|---\|
	\| veil-detect 1.0 \| 4.9% / 3.9% / 3.4% \| 0.949 / 0.955 / 0.972 \|
	\| nvidia/gliner-PII \| 21.2% / 24.2% / 21.4% \| 0.844 / 0.841 / 0.832 \|
	\| OpenMed nemotron \| 42.5% / 43.5% / 38.1% \| 0.701 / 0.686 / 0.703 \|
	\| openai/privacy-filter \| 55.2% / 56.3% / 39.3% \| 0.618 / 0.575 / 0.715 \|

	Sample 1 is 1973 federal appeals (1,865 mentions), sample 2 is 1979–80 federal appeals (1,694) and sample 3 is
	2017–18 state courts (1,842).

	### Mixed corpus

	81,864 gold mentions from the test splits of Nemotron-PII, gravitee, Gretel finance, pii-bench-zh and TAB. The model
	was trained on the train splits of several of these, so read this as an in-distribution figure, not a blind one.

	\| \| all \| person \| contact \| address \| numbers \| TAB (ECHR) \|
	\|---\|---:\|---:\|---:\|---:\|---:\|---:\|
	\| miss rate \| 3.2% \| 6.4% \| 0.4% \| 2.5% \| 1.0% \| 2.1% \|

	Throughput on the same corpus, one RTX 3080 (veil with its rule layer):

	![Throughput in MB/s and GPU memory in GiB on one RTX 3080: veil-detect 0.24 MB/s and 1.9 GiB; nvidia/gliner-PII 0.019 and 6.8; OpenMed nemotron 0.024 and 8.8; openai/privacy-filter 0.015 and 8.8](https://huggingface.co/libraos/veil-detect/resolve/main/assets/throughput-memory.png)

	\| detector \| MB/s \| GPU memory \|
	\|---\|---:\|---:\|
	\| veil-detect \| 0.24 \| 1.9 GiB \|
	\| nvidia/gliner-PII \| 0.019 \| 6.8 GiB \|
	\| OpenMed nemotron \| 0.024 \| 8.8 GiB \|
	\| openai/privacy-filter \| 0.015 \| 8.8 GiB \|

	### Running on CPU

	No GPU is needed. Measured with the same 384-token windows as `detect()` above (batched, with veil's rule layer), fp32
	PyTorch, one AMD EPYC 7402 and a 7,900-character bilingual contract, one document at a time:

	\| CPU cores \| seconds per 1,000 characters \| peak memory \|
	\|---:\|---:\|---:\|
	\| 1 \| 1.07 \| 1.6 GB \|
	\| 2 \| 0.57 \| 1.6 GB \|
	\| 4 \| 0.42 \| 1.6 GB \|

	A typical 1,500-character contract takes about 0.9 s on 2 cores. Set `torch.set_num_threads()` to the cores the process
	is given; otherwise PyTorch sizes its thread pool to the whole machine. We do not recommend dynamic int8 quantization of
	the linear layers: it cut CPU time by about a quarter, but on our contract test set the share of direct identifiers
	found fell from 0.94 to 0.71.

	## Intended use

	- First stage of a redaction or pseudonymization pipeline over documents, support tickets, chat and logs, before text
	is stored, shared or sent to a third-party model.
	- Finding personal data in a corpus for review, deletion requests or data mapping.

	Not intended for:
	- Guaranteed anonymization on its own. A 3–5% miss rate means a long document will usually contain at least one
	missed mention. For legal or regulated release, keep a human review step.
	- Decisions about the people it finds. It labels spans of text; it does not identify anyone.

	## Limitations

	- Give it context. Fed one sentence at a time instead of whole documents, recall on a 1,287-mention sample falls
	from 0.95 to 0.88, most of it on phone numbers, e-mails and case numbers. Pass paragraphs or whole documents.
	- Usernames in fiction or roleplay are found less reliably than in chat, support or forum text (0.94 recall on a
	synthetic set of out-of-character inserts).
	- Addresses in older legal prose are the weakest class on the blind samples (7–11% missed).
	- No label for quantities or miscellaneous identifiers (TAB's QUANTITY / MISC categories).
	- Languages. English and Chinese are evaluated. The finance data adds French, German, Dutch, Spanish and Italian, which
	are not separately measured. Other languages rely on the base model alone.
	- Over-tagging is expected. Judges, counsel, courts and public figures are tagged as people and organizations.

	## Training

	Fine-tuned from `FacebookAI/xlm-roberta-base` in stages: general PII and Chinese data from scratch, then legal text,
	then a `date_of_birth` label, then the final mix (83,337 documents in 100,603 windows of 384 tokens, 8,000 steps,
	batch 16, learning rate 1.5e-5, 19 minutes on one RTX 3080). Long documents are cut into overlapping windows by hand.
	Upper-case augmentation (p = 0.3) covers captions like "Peter J. BRENNAN, Secretary of Labor".

	\| source \| licence \| used for \|
	\|---\|---\|---\|
	\| nvidia/Nemotron-PII \| CC-BY-4.0 \| English PII across industries \|
	\| gravitee-io/pii-detection-dataset \| Apache-2.0 \| English PII, harmonised from several public sets \|
	\| gretelai/synthetic_pii_finance_multilingual \| Apache-2.0 \| finance documents in six languages \|
	\| CLUENER2020 (xusenlin/clue-ner) \| Apache-2.0 \| Chinese names, places, organizations \|
	\| Text Anonymization Benchmark, train split \| MIT \| ECHR judgments \|
	\| template generators \| written for this model \| Chinese forms and legal documents, US court opinions and dockets, exchange and wallet text, structured birth-date records, usernames in chat and support text, ECHR application numbers \|

	Blind test documents, the pii-bench-zh benchmark and every test split were excluded from training.

	## Versions

	\| version \| date \| notes \|
	\|---\|---\|---\|
	\| 1.0.0 \| 2026-10-21 \| first public release \|
	\| 1.0.0, card revision \| 2026-10-23 \| comparison charts, CPU figures; `detect()` no longer joins spans across a line break (a signature line and the role line under it came out as one name). Weights unchanged. \|

	## Licence and citation

	Apache-2.0. The NOTICE file carries the attribution that Nemotron-PII's CC-BY-4.0 licence requires.

	```bibtex
	@misc{veil_detect_2026,
	title = {veil-detect: a recall-first PII span detector},
	author = {{Libra OS}},
	year = {2026},
	url = {https://huggingface.co/libraos/veil-detect},
	note = {Version 1.0.0}
	}
	```