TSchachi/pii-token-classifier-de-en-code

PII token classifier for German and English text, code-aware

A fine-tuned version of bardsai/eu-pii-anonimization-multilang (revision 62ca5712368705bac11b8fea19e42e65cce088dd) that marks personal data in text, mainly names of people. It was adjusted for one situation the base model handles poorly: text that mixes prose with source code, logs and documentation, where the base model marks tool names, identifiers and dates as personal data.

This is a modified version of the base model. The base model is licensed under Apache-2.0, and so is this one.

What was changed

The network, the tokenizer and the list of labels are the base model's. Only weights were adjusted: the embeddings and the lower 6 of 12 encoder layers were kept frozen, the upper 6 layers and the classification layer were fine-tuned for two passes over about 10,800 short examples (learning rate 2e-5, batch size 16, maximum 320 tokens).

Training examples, none of which contain real personal data:

• Source code and package descriptions of open-source Python packages (about 1.6 MB), as examples of text with nothing to find. Each window was labelled with the base model's own prediction, except that a name, place or birth date the base model marked and a second detector (urchade/gliner_multi_pii-v1) did not mark was relabelled as nothing.
• Invented people in German and English sentences (clinical notes, chat messages, code and data lines): 1,200 on their own and 3,200 placed in the middle of open-source code or docs. All names are invented combinations of common first names and surnames.
• 8,000 German sentences from the training part of GermEval 2014 (news and Wikipedia, labelled by people). Labelled persons were used as persons; a "person" or "place" prediction of the base model on a token the human labels do not mark as such was relabelled as nothing.

Labels

Unchanged from the base model: 34 types in BIO form, among them PERSON_NAME, POSTAL_ADDRESS, LOCATION, DATE_OF_BIRTH, EMAIL_ADDRESS, PHONE_NUMBER and HEALTH_DATA. The fine-tuning targeted PERSON_NAME, LOCATION and DATE_OF_BIRTH. The other types were not trained on and not evaluated here.

How to use

from transformers import pipeline

pipe = pipeline(
    "token-classification",
    model="<user>/<repository>",
    revision="<commit hash>",          # pin it: results belong to one set of weights
    aggregation_strategy="simple",
)
for entity in pipe("Rücksprache mit Dr. Huber wegen der Entlassung am Montag."):
    print(entity["entity_group"], entity["start"], entity["end"], round(float(entity["score"]), 2))

The model reads at most 512 tokens at a time. For longer text, cut it into overlapping windows and merge the results; the measurements below used 512-character windows with a 128-character overlap and kept spans with a score of at least 0.6.

Results

All numbers come from one pipeline: 512-character windows, score threshold 0.6, LOCATION counted as an address, plus regular-expression rules for e-mail addresses, phone numbers, IBANs and numeric birth dates, and a filter that drops implausible spans. "Base" is the unchanged base model, "GLiNER" is urchade/gliner_multi_pii-v1 with eleven zero-shot labels. Processor only, Apple Silicon, four performance cores.

Test	GLiNER	Base	This model
Seconds per kB of text	0.22–0.25	0.09	0.08
Memory after load / highest over 151 scans	2.1 / 4.0 GB	1.2 / 1.8 GB	1.2 / 1.8 GB
German clinical letters (GraSCCo PII v2, 63 synthetic letters): patient names found, of 166	158	161	159
GraSCCo: doctor names found, of 154	128	141	143
GraSCCo: name spans on something that is not a person	116	42	38
German news and Wikipedia (GermEval 2014 test part, 5,100 sentences): persons found, of 1,639	1,493	1,547	1,543
GermEval: name spans on something that is not a person	353	172	137
Invented names, sentence on its own, of 140 (80 German, 60 English)	139	137	140
Invented names placed in the middle of code, of 140	139	134	139
One-word names next to punctuation, of 20	20	19	19
Code and docs of an unseen software project (490 kB): spans marked by the model	62	145	58
Unseen open-source Python code (238 kB): spans marked by the model	35	60	19

A span counts as found when any detected span overlaps it. On the unseen open-source code, 13 of this model's 19 spans have the shape "First Last"; that text contains real author names.

Limits

• Evaluated for names, and only in German and English. The base model lists 24 languages. What the fine-tuning did to the others, and to the label types it did not target, was not measured.
• The invented-name tests were written by the same author as the invented training sentences. Names and wording differ, but the style is shared, so those rows are optimistic. The GraSCCo and GermEval rows do not have this weakness.
• It still misses names. On the two German sets with human labels it misses about 4 to 7 in 100. It is one layer of a privacy tool, not a guarantee.
• Short, context-free input is hard. A single capitalised word without a sentence around it is often a guess.
• Phone numbers, e-mail addresses and IBANs were found by regular expressions in these measurements, not by the model.
• GPU on Apple Silicon (MPS): results are identical and long texts run about 1.6 times faster, but the GPU allocator's memory grew from 1.8 GB to 9.7 GB over 1,000 scans of texts of different lengths. Measured on the processor, memory does not grow.
• One machine. Speed and memory were measured on a single Apple Silicon Mac.

Files

• model.safetensors — sha256 08552c982992a3284f2e72e019a67d5596de18406d826606d45aff4891f52556
• config.json, tokenizer.json, tokenizer_config.json

Licence and attribution

• This model: Apache-2.0. It is a modified version of bardsai/eu-pii-anonimization-multilang (Apache-2.0); the weights of the upper six encoder layers and the classification layer were changed by fine-tuning as described above.
• Training data: GermEval 2014 Named Entity Recognition Shared Task data, by Darina Benikova, Chris Biemann and Marc Reznicek, licensed under CC-BY 4.0. Source code and metadata of open-source Python packages, used as unlabelled text.
• Evaluation data: GraSCCo PII v2 (Graz Synthetic Clinical text Corpus with PII annotations, CC-BY 4.0) and the test part of GermEval 2014 (CC-BY 4.0).