Hugging Face's logo

Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
Back to feed
crp4222 
posted an update 5 days ago
Post
2,732
I trained a small PII detector for PrivAiTe, my local proxy that masks personal data before a prompt reaches an LLM: crp4222/privaite-pii-minilm.

It is a multilingual MiniLM (117M parameters, 15 types from names and contacts to IDs, bank details and secrets), trained only on datasets whose licenses allow commercial use, released under BSD 3-Clause, with PyTorch, ONNX fp32 and int8, and Core ML builds.

On 9 public test sets it never saw, it masks as many person names as the best of GLiNER PII, NVIDIA GLiNER-PII, GLiNER2, Piiranha and OpenAI privacy-filter, or more, on 6 of them, usually with higher precision. It trails GLiNER PII on short lowercase voice requests and NVIDIA on court rulings.

It is also 2.5 to 7 times smaller and much faster: 9 ms per 1,000 characters on an M1 Pro Neural Engine and 23 ms on CPU in int8, against 186 to 621 ms for the others on the same machine and documents. Numbers, test sets and limits are on the card: crp4222/privaite-pii-minilm
Kreout

Counting a half-masked name as a miss is a useful choice for a proxy: a good token score can still leave the identifying part intact. Could the proxy surface which entity classes have held-out evaluation coverage? The card gives PERSON a much stronger evidence base than SECRET, which hasn't been tested on unseen data yet. That distinction would help someone decide which traffic they can reasonably route through it.

·
crp4222

Measured on Samsung's CredData, split by repository so 154 repos stay unseen. Alone, the model fully masks 22% of credentials: fine on passwords (61%) and tokens (56%), near zero on private keys, JWTs and bare keys assigned in code. With PrivAiTe's rules (0.8.1) the minilm preset reaches 35%, against 60% for the default onnx preset; the gap is almost all generic keys and secrets under arbitrary names (7% vs 34%, 13% vs 86%), where the Privacy Filter model is much stronger. Two fine-tunes closed much of it but flagged far more placeholders, so I didn't ship them. The breakdown is on the card now.
Twu31

The M1 Pro comparison makes the deployment trade-off concrete: Core ML fp16 is faster, while ONNX int8 has the smaller artifact. Are the nine held-out PERSON results from the PyTorch/ONNX fp32 version, or were they also checked on the Core ML build? I'd be interested in whether the short lowercase requests remain the main weak spot after conversion.

·
crp4222

Core ML fp16 on the Neural Engine is tied with fp32 on all nine sets, within 0.1 point of persons fully masked. int8 is the build that loses a little: 0.4 to 3 points on five of the nine, most on the short voice requests (MASSIVE 80.6 against 83.6). So short lowercase requests stay the main weak spot after conversion; int8 makes them slightly worse, Core ML does not. I'll add the per-build table to the card.
In this post