|
|
--- |
|
|
license: mit |
|
|
datasets: |
|
|
- ai4privacy/open-pii-masking-500k-ai4privacy |
|
|
language: |
|
|
- fr |
|
|
- en |
|
|
- de |
|
|
- te |
|
|
- hi |
|
|
- it |
|
|
- es |
|
|
- nl |
|
|
base_model: |
|
|
- answerdotai/ModernBERT-base |
|
|
library_name: transformers |
|
|
tags: |
|
|
- PII |
|
|
--- |
|
|
|
|
|
## Evaluation Metrics |
|
|
|
|
|
The table below summarizes the detailed evaluation results per PII label: |
|
|
|
|
|
| **Label** | **TP** | **FP** | **FN** | **Accuracy** | **Precision** | **Recall** | **F1 Score** | |
|
|
|--------------------|:------:|:------:|:------|:------------:|:-------------:|:----------:|:------------:| |
|
|
| SURNAME | 3722 | 0 | 28 | 99.25% | 100.0% | 99.25% | 99.63% | |
|
|
| O (Non-PII) | 0 | 400 | 0 | 99.30% | n/a | n/a | n/a | |
|
|
| TIME | 1936 | 0 | 0 | 100.0% | 100.0% | 100.0% | 100.0% | |
|
|
| DRIVERLICENSENUM | 505 | 0 | 2 | 99.61% | 100.0% | 99.61% | 99.80% | |
|
|
| PASSPORTNUM | 564 | 0 | 2 | 99.65% | 100.0% | 99.65% | 99.82% | |
|
|
| GIVENNAME | 7548 | 0 | 172 | 97.77% | 100.0% | 97.77% | 98.87% | |
|
|
| TELEPHONENUM | 3641 | 0 | 0 | 100.0% | 100.0% | 100.0% | 100.0% | |
|
|
| BUILDINGNUM | 407 | 0 | 19 | 95.54% | 100.0% | 95.54% | 97.72% | |
|
|
| AGE | 168 | 0 | 1 | 99.41% | 100.0% | 99.41% | 99.70% | |
|
|
| DATE | 2335 | 0 | 0 | 100.0% | 100.0% | 100.0% | 100.0% | |
|
|
| CITY | 1672 | 0 | 130 | 92.79% | 100.0% | 92.79% | 96.26% | |
|
|
| TITLE | 349 | 0 | 35 | 90.89% | 100.0% | 90.89% | 95.23% | |
|
|
| IDCARDNUM | 1998 | 0 | 22 | 98.91% | 100.0% | 98.91% | 99.45% | |
|
|
| GENDER | 121 | 0 | 0 | 100.0% | 100.0% | 100.0% | 100.0% | |
|
|
| CREDITCARDNUMBER | 557 | 0 | 1 | 99.82% | 100.0% | 99.82% | 99.91% | |
|
|
| SEX | 78 | 0 | 1 | 98.73% | 100.0% | 98.73% | 99.36% | |
|
|
| STREET | 1368 | 0 | 19 | 98.63% | 100.0% | 98.63% | 99.31% | |
|
|
| TAXNUM | 345 | 0 | 12 | 96.64% | 100.0% | 96.64% | 98.29% | |
|
|
| EMAIL | 2606 | 0 | 2 | 99.92% | 100.0% | 99.92% | 99.96% | |
|
|
| SOCIALNUM | 411 | 0 | 11 | 97.39% | 100.0% | 97.39% | 98.68% | |
|
|
| ZIPCODE | 406 | 0 | 20 | 95.31% | 100.0% | 95.31% | 97.60% | |
|
|
|
|
|
### Overall Evaluation |
|
|
- **Accuracy:** 99.01% |
|
|
- **Precision:** 98.72% |
|
|
- **Recall:** 98.47% |
|
|
- **F1 Score:** 98.59% |
|
|
|
|
|
- **Total True Positives (TP):** 30,737 |
|
|
- **Total False Positives (FP):** 400 |
|
|
- **Total False Negatives (FN):** 477 |
|
|
|
|
|
### Macro-Averaged Metrics |
|
|
- **Accuracy:** 98.35% |
|
|
- **Precision:** 95.24% |
|
|
- **Recall:** 93.35% |
|
|
- **F1 Score:** 94.29% |
|
|
|
|
|
--- |
|
|
|
|
|
## Model Behavior & Limitations |
|
|
|
|
|
- **Evaluation Focus:** |
|
|
The metrics shown above reflect performance on the test split of the [open-pii-masking-500k-ai4privacy](https://huggingface.co/datasets/ai4privacy/open-pii-masking-500k-ai4privacy) dataset. Real-world performance may vary and requires additional measures. Feel free to contact support@ai4privacy.com for assistance. |
|
|
|
|
|
--- |
|
|
|
|
|
## Disclaimer |
|
|
|
|
|
This model card details the evaluation metrics and fine-tuning parameters for the multilingual anonymiser. **Please note:** |
|
|
- The model is provided **as-is** under the MIT License. |
|
|
- It is intended solely for redaction purposes and does not perform full PII classification. |
|
|
- Users should carefully test and evaluate its performance on their own data before deploying in production environments. |
|
|
|
|
|
--- |
|
|
|
|
|
*Ai4Privacy – Committed to protecting personal data in the age of AI.* |
|
|
|
|
|
--- |
