DataFog
/

pii-small-en

@@ -1,175 +1,179 @@
----
-library_name: transformers
-license: apache-2.0
-language:
-- en
-tags:
-- token-classification
-- ner
-- pii
-- privacy
-- deberta
-- crf
-datasets:
-- ai4privacy/internationalised_pii_dataset
-- gretelai/gretel-pii-masking-en-v1
-pipeline_tag: token-classification
-model-index:
-- name: datafog-pii-ner-v1
-  results:
-  - task:
-      type: token-classification
-      name: Named Entity Recognition
-    metrics:
-    - type: f1
-      value: 0.904
-      name: Overall F1
-    - type: precision
-      value: 0.907
-      name: Overall Precision
-    - type: recall
-      value: 0.902
-      name: Overall Recall
----
-# DataFog PII-NER v1
-A token classification model for detecting **Personally Identifiable Information (PII)** in English text. Built on DeBERTa-v3-xsmall with character-level CNN features and a CRF decoding head for structured BIO tag prediction.
-## Model Details
-| Property | Value |
-|----------|-------|
-| Architecture | DeBERTa-v3-xsmall + CharCNN + CRF |
-| Parameters | ~22.7M total |
-| Labels | 89 BIO tags (40 entity types) |
-| Max sequence length | 256 tokens |
-| Training data | ~135K examples from 3 datasets |
-| Training hardware | NVIDIA A100 (Colab), BF16 mixed precision |
-| Framework | Transformers 5.0, PyTorch 2.x |
-## Architecture
-```
-Input text
-    |
-    v
-DeBERTa-v3-xsmall (70.7M pretrained params)
-    |
-    v
-Character CNN (3/4/5-gram filters)
-    |
-    v
-Gating Fusion (learned weighted combination)
-    |
-    v
-CRF Head (sequence-level decoding)
-    |
-    v
-89 BIO tag predictions
-```
-The CRF head enforces valid BIO tag sequences (e.g., I-PERSON can only follow B-PERSON or I-PERSON), which improves entity boundary detection compared to independent per-token classification.
-## Supported Entity Types (40 types, 4 tiers)
-### Tier 1 -- Critical PII
-SSN, Credit Card, Bank Account, Passport Number, Drivers License, Tax ID
-### Tier 2 -- High Sensitivity
-Person, Email, Phone, Date of Birth, Street Address, IP Address
-### Tier 3 -- Moderate Sensitivity
-Username, Date, Location, Organization, URL, License Plate, Age, Nationality, Gender, Ethnicity, Religion, Marital Status
-### Tier 4 -- Domain-Specific
-Medical Record, Employee ID, Student ID, Account Number, PIN, Password, Biometric, Vehicle ID, Device ID, Crypto Wallet, IBAN, Swift Code, Insurance Number, Salary, Criminal Record, Political Affiliation, Sexual Orientation, Health Condition, Genetic Data, Trade Union
-## Test Set Results
-| Metric | Value |
-|--------|-------|
-| **Overall F1** | **0.904** |
-| Overall Precision | 0.907 |
-| Overall Recall | 0.902 |
-### Tier Recall
-| Tier | Recall | Target |
-|------|--------|--------|
-| Tier 1 (Critical) | 0.722 | 0.98 |
-| Tier 2 (High) | 0.934 | 0.95 |
-| Tier 3 (Moderate) | 0.919 | 0.90 |
-| Tier 4 (Domain) | 0.866 | 0.85 |
-### Per-Entity F1 (All Types)
-| Entity Type | F1 | Recall |
-|-------------|-----|--------|
-| Biometric | 0.996 | 0.996 |
-| URL | 0.994 | 0.995 |
-| Email | 0.991 | 0.987 |
-| IP Address | 0.988 | 0.992 |
-| Date of Birth | 0.978 | 0.980 |
-| Vehicle ID | 0.964 | 0.989 |
-| Phone | 0.963 | 0.961 |
-| Employee ID | 0.962 | 0.959 |
-| License Plate | 0.960 | 0.952 |
-| Gender | 0.952 | 0.949 |
-| IBAN | 0.930 | 0.898 |
-| Swift Code | 0.926 | 0.980 |
-| Username | 0.924 | 0.912 |
-| Location | 0.922 | 0.908 |
-| Account Number | 0.908 | 0.917 |
-| Organization | 0.898 | 0.903 |
-| SSN | 0.891 | 0.858 |
-| Drivers License | 0.885 | 0.881 |
-| Password | 0.878 | 0.885 |
-| Date | 0.875 | 0.869 |
-| Person | 0.861 | 0.868 |
-| Credit Card | 0.862 | 0.839 |
-| Age | 0.851 | 0.861 |
-| Street Address | 0.834 | 0.817 |
-| Bank Account | 0.791 | 0.746 |
-| Tax ID | 0.665 | 0.624 |
-| Passport Number | 0.469 | 0.385 |
-| PIN | 0.432 | 0.302 |
-## Training Details
-- **Backbone LR:** 2e-5 (with AdamW eps=1.0 to prevent NaN)
-- **Head LR:** 1e-3 (50x faster than backbone)
-- **Warmup:** 10% of steps
-- **Epochs:** 10 (best checkpoint at epoch 5)
-- **Effective batch size:** 32
-- **Mixed precision:** BF16
-## Training Data
-Trained on a combined dataset of ~135K examples from:
-- [AI4Privacy PII Dataset](https://huggingface.co/datasets/ai4privacy/internationalised_pii_dataset)
-- [Nemotron PII](https://huggingface.co/datasets/ai4privacy/pii-masking-400k)
-- [Gretel PII Masking](https://huggingface.co/datasets/gretelai/gretel-pii-masking-en-v1)
-## Limitations
-- Tier 1 recall (0.722) is below the 0.98 target -- critical PII types like SSN, Credit Card, and Passport Number need improvement
-- Rare entity types (PIN, Passport Number, Tax ID) have low F1 due to limited training examples
-- English-only
-- Max 256 tokens per input (longer documents need chunking)
-- Custom architecture requires the `datafog-pii-ner` package for loading (not a standard HuggingFace token classifier)
-## Citation
-```bibtex
-@software{datafog_pii_ner_v1,
-  title={DataFog PII-NER v1: Token Classification for PII Detection},
-  author={DataFog},
-  year={2026},
-  url={https://github.com/DataFog/datafog-labs}
-}
-```
-## License
-Apache 2.0

+---
+library_name: transformers
+license: apache-2.0
+language:
+- en
+tags:
+- token-classification
+- ner
+- pii
+- privacy
+- deberta
+- crf
+datasets:
+- ai4privacy/internationalised_pii_dataset
+- gretelai/gretel-pii-masking-en-v1
+pipeline_tag: token-classification
+model-index:
+- name: datafog-pii-small-en
+  results:
+  - task:
+      type: token-classification
+      name: Named Entity Recognition
+    metrics:
+    - type: f1
+      value: 0.9071
+      name: Overall F1
+    - type: precision
+      value: 0.8981
+      name: Overall Precision
+    - type: recall
+      value: 0.9162
+      name: Overall Recall
+---
+# DataFog PII-NER v1.3
+A lightweight token classification model for detecting **Personally Identifiable Information (PII)** in English text. Built on DeBERTa-v3-xsmall with character-level CNN features and a CRF decoding head for structured BIO tag prediction.
+**v1.3** is the fourth iteration, achieving the best overall F1 (0.9071) across all versions through early backbone freezing and progressive tier weight reduction.
+## Model Details
+| Property | Value |
+|----------|-------|
+| Architecture | DeBERTa-v3-xsmall + CharCNN + GatingFusion + CRF |
+| Parameters | ~22.7M total |
+| Labels | 89 BIO tags (44 entity types) |
+| Max sequence length | 256 tokens |
+| Training data | ~169K examples from 3 datasets (with Tier 1 oversampling) |
+| Training hardware | NVIDIA H100 PCIe (80GB), BF16 mixed precision |
+| Training time | 20 hours (10 epochs) |
+| Framework | Transformers 4.49, PyTorch 2.7 |
+## Architecture
+The **CharCNN** captures structural PII patterns (SSN: XXX-XX-XXXX, credit cards: XXXX-XXXX-XXXX-XXXX) while **DeBERTa** provides contextual understanding. The **gating fusion** dynamically weights character vs. contextual features per token. The **CRF head** enforces valid BIO tag sequences at the sequence level.
+## Supported Entity Types (44 types, 4 tiers)
+### Tier 1 -- Critical PII (target: 0.98 recall)
+SSN, Credit Card, Bank Account, Passport Number, Drivers License, Tax ID
+### Tier 2 -- High Sensitivity (target: 0.95 recall)
+Person, Email, Phone, Date of Birth, Street Address, IP Address
+### Tier 3 -- Moderate Sensitivity (target: 0.90 recall)
+Username, Date, Location, Organization, URL, License Plate, Age, Nationality, Gender, Ethnicity, Religion, Marital Status
+### Tier 4 -- Domain-Specific (target: 0.85 recall)
+Medical Record, Employee ID, Student ID, Account Number, PIN, Password, Biometric, Vehicle ID, Device ID, Crypto Wallet, IBAN, Swift Code, Insurance Number, Salary, Criminal Record, Political Affiliation, Sexual Orientation, Health Condition, Genetic Data, Trade Union
+## Test Set Results
+### Overall Metrics
+| Metric | V1.3 | V1.2 | V1.1 | V1 |
+|--------|------|------|------|-----|
+| **Overall F1** | **0.9071** | 0.9005 | 0.9005 | 0.904 |
+| Precision | 0.8981 | 0.9050 | 0.9062 | 0.907 |
+| **Recall** | **0.9162** | 0.8960 | 0.8950 | 0.902 |
+### Tier Recall
+| Tier | V1.3 | V1.2 | Target | Status |
+|------|------|------|--------|--------|
+| Tier 1 (Critical) | 0.823 | 0.841 | 0.98 | FAIL |
+| Tier 2 (High) | **0.945** | 0.936 | 0.95 | FAIL |
+| Tier 3 (Moderate) | **0.930** | 0.911 | 0.90 | PASS |
+| Tier 4 (Domain) | **0.868** | 0.845 | 0.85 | PASS |
+### Per-Entity F1 (Top 20)
+| Entity Type | F1 |
+|-------------|------|
+| URL | 0.994 |
+| Biometric | 0.992 |
+| IP Address | 0.988 |
+| Date of Birth | 0.981 |
+| Vehicle ID | 0.976 |
+| Email | 0.968 |
+| Phone | 0.966 |
+| License Plate | 0.952 |
+| Gender | 0.946 |
+| Employee ID | 0.940 |
+| IBAN | 0.935 |
+| Username | 0.930 |
+| SSN | 0.930 |
+| Location | 0.929 |
+| Account Number | 0.923 |
+| Organization | 0.902 |
+| Drivers License | 0.881 |
+| Password | 0.880 |
+| Date | 0.877 |
+| Person | 0.875 |
+## Training Details
+### V1.3 Approach: Early Freeze + Progressive Tier Weights
+Two key innovations based on learnings from V1-V1.2:
+1. **Backbone freeze after epoch 3**: DeBERTa weights are frozen after epoch 3 to preserve clean representations before training instability occurs.
+2. **Progressive tier weight reduction**: CRF loss weights start at 3x/2x/1.5x/1x (Tier 1-4) for epochs 1-2, then reduce to 2x/1.5x/1.25x/1x from epoch 3 onward. This limits gradient amplification buildup while giving a strong initial learning signal.
+### Hyperparameters
+| Parameter | Value |
+|-----------|-------|
+| Backbone LR | 1e-5 (with AdamW eps=1.0) |
+| Head LR | 1e-3 (100x faster) |
+| LR Schedule | Cosine |
+| Warmup | 500 steps |
+| Epochs | 10 (3 full + 7 head-only) |
+| Effective batch size | 32 (8 x 4 gradient accumulation) |
+| Mixed precision | BF16 |
+| Best checkpoint | Epoch 3 |
+### Training Data
+~169K examples from three open-licensed datasets:
+- [AI4Privacy PII Dataset](https://huggingface.co/datasets/ai4privacy/internationalised_pii_dataset) (~43K English examples, Apache 2.0)
+- [NVIDIA Nemotron PII](https://huggingface.co/datasets/ai4privacy/pii-masking-400k) (~100K examples, CC-BY-4.0)
+- [Gretel PII Masking](https://huggingface.co/datasets/gretelai/gretel-pii-masking-en-v1) (~26K examples, Apache 2.0)
+Tier 1 entity examples are oversampled 3x to address the 323x frequency imbalance between common entities (DATE: 170K) and rare critical entities (PASSPORT: 526).
+## Version History
+| Version | F1 | Tier 1 Recall | Key Change |
+|---------|------|--------------|------------|
+| V1 | 0.904 | 0.722 | Baseline |
+| V1.1 | 0.9005 | 0.771 | Tier-weighted loss + oversampling |
+| V1.2 | 0.9005 | 0.841 | Backbone freeze after epoch 4 |
+| **V1.3** | **0.907** | 0.823 | Early freeze (epoch 3) + progressive tier weights |
+## Limitations
+- Tier 1 recall (0.823) is below the 0.98 target -- critical PII types like Passport Number (only 526 training examples) remain challenging
+- 16 entity types have zero training examples (Nationality, Ethnicity, Religion, etc.) and cannot be detected
+- English-only
+- Max 256 tokens per input (longer documents need chunking)
+- Custom architecture requires the  source code for loading
+## Links
+- **Code**: [github.com/DataFog/datafog-labs](https://github.com/DataFog/datafog-labs)
+- **Training Chronicle**: [Full training log](https://github.com/DataFog/datafog-labs/blob/main/pii-ner-v1/docs/training_chronicle.md)
+- **WandB Run**: [V1.3 training metrics](https://wandb.ai/datafog/huggingface/runs/a66aw6sb)
+## Citation
+## License
+Apache 2.0