Upload bf16 release artifact

.gitattributes

@@ -33,3 +33,6 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+figures/evaluation-test-1-1.png filter=lfs diff=lfs merge=lfs -text
+figures/evaluation-train-1-1.png filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,228 @@
+---
+language:
+- ko
+- en
+license: apache-2.0
+library_name: transformers
+pipeline_tag: token-classification
+base_model: openai/privacy-filter
+tags:
+- token-classification
+- ner
+- pii
+- privacy
+- pii-masking
+- korean
+- finance
+- bioes
+- viterbi
+- mixture-of-experts
+datasets:
+- BCCard/pii-masking-openpii-finance
+metrics:
+- f1
+- precision
+- recall
+---
+
+# 1. Overview
+A Korean/English **PII detection model for the finance domain**, built by full fine-tuning
+[`openai/privacy-filter`](https://huggingface.co/openai/privacy-filter) (1.4B MoE, 50M active) on synthetic finance-domain PII data. It tags **18 PII entity types** (73 BIOES classes) at token level and is intended as the **NER layer of a multi-layer PII-masking gateway** in front of LLM services —
+behind a regex backstop for fully structured identifiers, never as a standalone compliance guarantee.
+
+On held-out validation it reaches **strict span-F1 0.956 (ko) / 0.969 (en)**. On an independent, adversarially-hardened Golden Set it holds **0.944 (ko) / 0.907 (en)** with **masking coverage 0.996 (ko) / 0.998 (en)** — i.e. ≥99.5% of gold PII characters are covered by predicted spans.
+
+## 1.1. TL;DR
+* **Base model**: [`openai/privacy-filter`](https://huggingface.co/openai/privacy-filter) — 1.4B-parameter MoE (128 experts, 50M active), 8 layers, hidden 640, bidirectional banded attention (±128), o200k tokenizer
+* **Domain / Language**: Finance (BC Card — cards, accounts, national IDs, customer service text) / Korean + English
+* **Task**: Token classification (BIOES) → character-offset PII spans → masking
+* **Labels (18)**: `PERSON, RRN, FRN, CARD_NUMBER, ACCOUNT_NUMBER, SECRET, USER_ID, EMAIL, PHONE, PASSPORT, DRIVER_LICENSE, GENERIC_ID, ADDRESS, ZIPCODE, DATE, CARD_EXPIRY, CVC, IPIN`
+* **Method**: Full fine-tuning (all parameters incl. experts & router) with a re-initialized 73-class head (rows copied from the base head by taxonomy mapping)
+* **Decoding**: **constrained BIOES Viterbi** (not per-token argmax) + whitespace span refinement — the bundled `viterbi_calibration.json` exposes precision↔recall operating-point biases without retraining
+* **Format**: BF16 (attention `sinks` kept FP32), single safetensors + tokenizer + label taxonomy + Viterbi calibration sidecar
+* **Sequence length**: trained on sequences ≤768 tokens — chunk longer inputs
+* **Intended use**
+  - In-house **PII masking gateway** (detect → mask before text reaches an LLM)
+  - Korean-centric finance text with mixed English (IDs, e-mails, card numbers)
+
+## 1.2. Label Taxonomy (N=18)
+The 18 labels re-map the upstream ai4privacy source labels to the granularity a Korean financial masking policy needs - merging fragments into single spans (`GIVENNAME`/`SURNAME` → `PERSON`, `CITY`/`STREET`/`BUILDINGNUM` → `ADDRESS`) and adding Korea-specific classes absent upstream (`RRN`, `FRN`, `IPIN`, `CARD_EXPIRY`, `CVC`, `SECRET`). `data source` records the row-source buckets in which each label occurs: `ko` means `openpii-1.5m-ko`, `en` means `openpii-1.5m-en`, and `domain` means locally synthesized rows.
+
+| label | description | data source |
+|-------|-------------|-------------|
+| `PERSON` | full name (surname + given, single span) | ko, en, domain |
+| `RRN` | resident registration number (Korea) | ko, domain |
+| `FRN` | foreign registration number | domain |
+| `CARD_NUMBER` | credit/debit card PAN | ko, en, domain |
+| `ACCOUNT_NUMBER` | bank account number | ko, domain |
+| `SECRET` | auth secret (password / API key / token) | ko, domain |
+| `USER_ID` | online member ID | ko, en, domain |
+| `EMAIL` | email address | ko, en, domain |
+| `PHONE` | phone number (mobile / landline) | ko, en, domain |
+| `PASSPORT` | passport number | ko, en, domain |
+| `DRIVER_LICENSE` | driver's license number | ko, en, domain |
+| `GENERIC_ID` | generic identifier (no KO counterpart) | ko, en, domain |
+| `ADDRESS` | address (city / street / building, single span) | ko, en, domain |
+| `ZIPCODE` | postal code | ko, en, domain |
+| `DATE` | date / time | ko, en, domain |
+| `CARD_EXPIRY` | card expiry date | domain |
+| `CVC` | card verification code | domain |
+| `IPIN` | I-PIN number | domain |
+
+Each entity type has `B-`, `I-`, `E-` and `S-` boundary classes, plus the background class `O`. This yields 73 output classes. The bundled `label-taxonomy.yaml` and `config.json` must remain in the same label order.
+
+## 1.3. Usage
+
+```python
+import torch
+from transformers import AutoModelForTokenClassification, AutoTokenizer
+
+model_id = "BCCard/MoAI-Privacy-Filter"
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+model = AutoModelForTokenClassification.from_pretrained(model_id)
+model.eval()
+
+text = "고객 모아이님(000000-0000000)께서 010-0000-0000로 연락 요청하셨습니다."
+enc = tokenizer(text, return_offsets_mapping=True, add_special_tokens=False, return_tensors="pt")
+offsets = enc.pop("offset_mapping")[0].tolist()
+
+with torch.no_grad():
+    logits = model(**enc).logits[0]          # [T, 73] — raw output
+
+# 1) Decode the logits with constrained BIOES Viterbi (recommended; see note below)
+#    and map token paths to character spans via `offsets` — the detector's output:
+# -> [{"start": 3, "end": 6, "label": "PERSON"},   '모아이'
+#     {"start": 8, "end": 22, "label": "RRN"},     '000000-0000000'
+#     {"start": 26, "end": 39, "label": "PHONE"}]  '010-0000-0000'
+# 2) Masking is downstream application logic — replace each span according to
+#    your masking policy, e.g.:
+# -> "고객 [PERSON]님([RRN])께서 [PHONE]로 연락 요청하셨습니다."
+```
+
+> **Decoding note** — this model (like its base) is post-trained for **constrained Viterbi decoding**
+> over the BIOES transition grammar, *not* independent per-token argmax. Argmax can emit invalid tag
+> sequences (span splits / orphan tags) and measurably lowers span-F1. The bundled
+> `viterbi_calibration.json` follows the upstream operating-point schema: its six transition biases
+> shift the precision↔recall trade-off at deploy time without retraining (all `0.0` = neutral).
+
+## 1.4. Training Data
+| Dataset | Role | Size |
+|---------|------|------|
+| (Public) [BCCard/pii-masking-openpii-finance](https://huggingface.co/datasets/BCCard/pii-masking-openpii-finance) (v2) | Training / validation | ~58.5k train rows · ~14.5k validation rows |
+| (Private) BCCard/pii-masking-openpii-finance-test (v2) | Golden Set (release gate; never used for training/tuning) | 2,000 rows (ko 1,460 / en 540) |
+
+* Sources: curated Korean subset of `ai4privacy/pii-masking-openpii-1.5m` (label taxonomy remapped, name spans merged & naturalized) + finance-domain synthetic templates + **~30% English replay** (forgetting guard)
+* Hard-example design baked into v2: surface-similar non-PII decoys (FP suppression), label-confusion pairs in one sentence (RRN↔FRN, DRIVER_LICENSE↔GENERIC_ID), weak-context true PII (FN suppression), long-span address boundary variants
+* All values are synthetic; validity-pattern collisions with real identifiers are removed at generation time (e.g. card numbers are forced to fail Luhn)
+
+## 1.5. Training Procedure
+| Item | Value |
+|------|-------|
+| Method | Full fine-tuning (1.4B params — experts and router included) |
+| Head | 33-class base head → 73-class head, initialized by copying base rows via taxonomy mapping |
+| Loss | Token-level cross-entropy |
+| Batch | effective 16 (per-device × world × accum), fixed across hardware layouts |
+| LR / scheduler | 1e-4 / linear decay, warmup 3% |
+| Optimizer | AdamW (fused), weight decay 0.0, max_grad_norm 1.0 |
+| Epochs | 5 — best checkpoint by validation span micro-F1, **decoded with the same constrained Viterbi as deployment** |
+| Precision | FP32 master weights + BF16 autocast; MoE router/experts explicitly kept FP32 during compute |
+| Hardware | 1× NVIDIA H100 (~5h) |
+
+<div align="center">
+  <img src="figures/evaluation-train-1-1.png" alt="Training loss, learning-rate and gradient-norm curves for the v1 and v2 models" >
+</div>
+
+<div align="center">
+  <img src="figures/evaluation-test-1-1.png" alt="Training-time validation metric curves for the v1 and v2 models" >
+</div>
+
+<br>
+
+# 2. Evaluation
+## 2.1. Setup
+* **Golden Set**: independently generated 2,000-row test set (ko 1,460 / en 540), **adversarially hardened** — weak-context PII, decoys, confusion pairs and boundary variants are deliberately over-represented, so scores here read *lower* than typical in-distribution synthetic benchmarks
+* **Protocol**: strict exact-match span P/R/F1 (CoNLL-style; boundary and label must both match) + **masking coverage** (share of gold PII *characters* covered by predicted spans, label-agnostic — the leakage-oriented metric)
+* **Decoding**: constrained Viterbi + whitespace refinement — identical to the deployment chain
+
+## 2.2. Results
+### `validation` dataset
+
+| Metric | v2 model / v2 validation |
+|---|---:|
+| micro F1 | 0.9599 |
+| macro F1 | 0.9603 |
+| ko strict micro F1 | 0.9562 |
+| ko macro F1 | 0.9568 |
+| en strict micro F1 | 0.9688 |
+| en macro F1 | 0.9631 |
+| **masking coverage** | **0.9979** |
+| **ko masking coverage** | **0.9987** |
+| **en masking coverage** | **0.9965** |
+
+* These values were measured post-hoc by running the exported `final-bf16` artifact over all 14,543 v2 validation rows (ko 10,460 / en 4,083) through the deployment-equivalent chain: constrained Viterbi, actual tokenizer character offsets and whitespace refinement.
+* Overall micro F1 and masking coverage pool all ko/en spans or characters before scoring. Overall macro F1 pools per-label TP/FP/FN across both languages and then averages the 18 label F1 values.
+* Character coverage counts are ko **482,861 / 483,488** and en **264,979 / 265,898** gold PII characters.
+* The training-time checkpoint-selection metrics remain ko micro F1 **0.9820**, en micro F1 **0.9739** and global macro F1 **0.9764** at epoch 5. They compare entity spans on token indices, so they are not interchangeable with the character-span values above and do not include masking coverage.
+
+### `test` dataset
+Independently generated Golden Set — deliberately harder than validation: weak-context PII, surface-similar decoys, label-confusion pairs and long-span boundary variants are over-represented.
+**Δ = vs. the post-hoc validation baseline above** using the same `final-bf16` artifact and character-span evaluation chain. The difference measures test hardening and distribution shift, not model regression.
+
+| Metric | v2 model / v2 test | Δ |
+|---|---:|---:|
+| micro F1 | 0.9336 | -2.63%p |
+| macro F1 | 0.9308 | -2.94%p |
+| ko strict micro F1 | 0.9441 | -1.21%p |
+| ko macro F1 | 0.9416 | -1.53%p |
+| en strict micro F1 | 0.9065 | -6.24%p |
+| en macro F1 | 0.9017 | -6.14%p |
+| **masking coverage** | **0.9964** | -0.16%p |
+| **ko masking coverage** | **0.9956** | -0.31%p |
+| **en masking coverage** | **0.9984** | +0.18%p |
+
+* **Masking coverage stays ≥0.9956 on the adversarial set** — only 0.44% (ko) / 0.16% (en) of gold PII characters are uncovered; most strict-F1 losses are boundary or label-name errors, not leaks
+* **English ADDRESS holds on hard boundary variants**: strict recall **0.983** on long-span address forms (state suffixes, unit/floor tails) that are heavily represented in this set
+* **Weak-context person names are the main remaining leak channel**: ko `PERSON` strict recall 0.875 with 82 full-span misses (see Limitations)
+* Label-swap errors (e.g. en `ACCOUNT_NUMBER` predicted as `GENERIC_ID`/`CARD_NUMBER`) keep **coverage 1.0** — the value is still masked; only the label name is wrong
+
+## 2.3. Reading the numbers
+Strict exact-match span-F1 on an adversarial test is a deliberately harsh score: a one-character boundary miss or a swapped label counts as a full error. For the product question — *"how much PII text leaks through?"* — masking coverage is the operative metric: **0.44% (ko) / 0.16% (en) of gold PII characters uncovered**, concentrated in weak-context person names.
+
+<br>
+
+## 2.4. Limitations
+* **One layer of defense** — inherits the base model's positioning: not an anonymization or compliance guarantee. Deploy behind a regex backstop for fully structured identifiers (RRN patterns, card numbers, phones) and combine with policy-level controls.
+* **Weak-context person names** — Korean names without honorifics/particles or list-form values are the main miss channel (ko `PERSON` recall 0.875 on the adversarial set). Consider a recall-leaning Viterbi operating point in high-sensitivity deployments.
+* **Alphanumeric ID confusion** — `USER_ID`/`SECRET`/`GENERIC_ID`/`ACCOUNT_NUMBER` share surface forms; without cue words the label may swap (masking still applies — coverage stays ~1.0).
+* **Synthetic-only training & evaluation** — no real customer text was used or evaluated. Real-world robustness (typos, slang, OCR noise) is unvalidated; shadow-mode rollout is recommended before enforcement.
+* **Fixed label policy** — the 18-label taxonomy is baked in at fine-tuning time; changing masking policy granularity requires re-fine-tuning (runtime keep/mask toggles must operate on these labels).
+* **Context window** — banded attention limits each token's context to ±128 tokens; trained sequence regime is ≤768 tokens (chunk longer documents).
+
+<br>
+
+# 3. Future Work
+* **v3 data** — weak-context person-name hard positives, cue-word diversification for the alphanumeric ID group, privacy-safe failure-collection loop from shadow operation
+* **Operating point** - recall-leaning Viterbi transition biases tuned on validation or a dedicated calibration set without Golden Set feedback
+* **Serving** - target-hardware latency, throughput and memory benchmarks for the separately published INT8 weight-only ONNX artifact
+
+<br>
+
+# 4. Meta Info
+## 4.1. Citation
+```bibtex
+@misc{bccard2026moaiprivacyfilter,
+  title        = {MoAI-Privacy-Filter: A Korean Finance-Domain PII Detection Model},
+  author       = {BC Card AX Team},
+  year         = {2026},
+  howpublished = {https://huggingface.co/BCCard/MoAI-Privacy-Filter},
+  note         = {Full fine-tune of openai/privacy-filter for Korean/English PII masking in the BC Card domain}
+}
+```
+
+## 4.2. See Also
+* **Base model**: [`openai/privacy-filter`](https://huggingface.co/openai/privacy-filter)
+* **INT8 ONNX artifact**: [`BCCard/MoAI-Privacy-Filter-INT8`](https://huggingface.co/BCCard/MoAI-Privacy-Filter-INT8)
+* **Training dataset**: [`BCCard/pii-masking-openpii-finance`](https://huggingface.co/datasets/BCCard/pii-masking-openpii-finance)
+* **Source data attribution**: `ai4privacy/pii-masking-openpii-1.5m` (CC-BY-4.0)
+
+<br>

config.json

@@ -0,0 +1,200 @@
+{
+  "architectures": [
+    "OpenAIPrivacyFilterForTokenClassification"
+  ],
+  "attention_bias": true,
+  "attention_dropout": 0.0,
+  "bos_token_id": null,
+  "classifier_dropout": 0.0,
+  "default_n_ctx": 128000,
+  "dtype": "bfloat16",
+  "eos_token_id": 199999,
+  "head_dim": 64,
+  "hidden_act": "silu",
+  "hidden_size": 640,
+  "id2label": {
+    "0": "O",
+    "1": "B-PERSON",
+    "2": "I-PERSON",
+    "3": "E-PERSON",
+    "4": "S-PERSON",
+    "5": "B-RRN",
+    "6": "I-RRN",
+    "7": "E-RRN",
+    "8": "S-RRN",
+    "9": "B-FRN",
+    "10": "I-FRN",
+    "11": "E-FRN",
+    "12": "S-FRN",
+    "13": "B-CARD_NUMBER",
+    "14": "I-CARD_NUMBER",
+    "15": "E-CARD_NUMBER",
+    "16": "S-CARD_NUMBER",
+    "17": "B-ACCOUNT_NUMBER",
+    "18": "I-ACCOUNT_NUMBER",
+    "19": "E-ACCOUNT_NUMBER",
+    "20": "S-ACCOUNT_NUMBER",
+    "21": "B-SECRET",
+    "22": "I-SECRET",
+    "23": "E-SECRET",
+    "24": "S-SECRET",
+    "25": "B-USER_ID",
+    "26": "I-USER_ID",
+    "27": "E-USER_ID",
+    "28": "S-USER_ID",
+    "29": "B-EMAIL",
+    "30": "I-EMAIL",
+    "31": "E-EMAIL",
+    "32": "S-EMAIL",
+    "33": "B-PHONE",
+    "34": "I-PHONE",
+    "35": "E-PHONE",
+    "36": "S-PHONE",
+    "37": "B-PASSPORT",
+    "38": "I-PASSPORT",
+    "39": "E-PASSPORT",
+    "40": "S-PASSPORT",
+    "41": "B-DRIVER_LICENSE",
+    "42": "I-DRIVER_LICENSE",
+    "43": "E-DRIVER_LICENSE",
+    "44": "S-DRIVER_LICENSE",
+    "45": "B-GENERIC_ID",
+    "46": "I-GENERIC_ID",
+    "47": "E-GENERIC_ID",
+    "48": "S-GENERIC_ID",
+    "49": "B-ADDRESS",
+    "50": "I-ADDRESS",
+    "51": "E-ADDRESS",
+    "52": "S-ADDRESS",
+    "53": "B-ZIPCODE",
+    "54": "I-ZIPCODE",
+    "55": "E-ZIPCODE",
+    "56": "S-ZIPCODE",
+    "57": "B-DATE",
+    "58": "I-DATE",
+    "59": "E-DATE",
+    "60": "S-DATE",
+    "61": "B-CARD_EXPIRY",
+    "62": "I-CARD_EXPIRY",
+    "63": "E-CARD_EXPIRY",
+    "64": "S-CARD_EXPIRY",
+    "65": "B-CVC",
+    "66": "I-CVC",
+    "67": "E-CVC",
+    "68": "S-CVC",
+    "69": "B-IPIN",
+    "70": "I-IPIN",
+    "71": "E-IPIN",
+    "72": "S-IPIN"
+  },
+  "initial_context_length": 4096,
+  "initializer_range": 0.02,
+  "intermediate_size": 640,
+  "label2id": {
+    "B-ACCOUNT_NUMBER": 17,
+    "B-ADDRESS": 49,
+    "B-CARD_EXPIRY": 61,
+    "B-CARD_NUMBER": 13,
+    "B-CVC": 65,
+    "B-DATE": 57,
+    "B-DRIVER_LICENSE": 41,
+    "B-EMAIL": 29,
+    "B-FRN": 9,
+    "B-GENERIC_ID": 45,
+    "B-IPIN": 69,
+    "B-PASSPORT": 37,
+    "B-PERSON": 1,
+    "B-PHONE": 33,
+    "B-RRN": 5,
+    "B-SECRET": 21,
+    "B-USER_ID": 25,
+    "B-ZIPCODE": 53,
+    "E-ACCOUNT_NUMBER": 19,
+    "E-ADDRESS": 51,
+    "E-CARD_EXPIRY": 63,
+    "E-CARD_NUMBER": 15,
+    "E-CVC": 67,
+    "E-DATE": 59,
+    "E-DRIVER_LICENSE": 43,
+    "E-EMAIL": 31,
+    "E-FRN": 11,
+    "E-GENERIC_ID": 47,
+    "E-IPIN": 71,
+    "E-PASSPORT": 39,
+    "E-PERSON": 3,
+    "E-PHONE": 35,
+    "E-RRN": 7,
+    "E-SECRET": 23,
+    "E-USER_ID": 27,
+    "E-ZIPCODE": 55,
+    "I-ACCOUNT_NUMBER": 18,
+    "I-ADDRESS": 50,
+    "I-CARD_EXPIRY": 62,
+    "I-CARD_NUMBER": 14,
+    "I-CVC": 66,
+    "I-DATE": 58,
+    "I-DRIVER_LICENSE": 42,
+    "I-EMAIL": 30,
+    "I-FRN": 10,
+    "I-GENERIC_ID": 46,
+    "I-IPIN": 70,
+    "I-PASSPORT": 38,
+    "I-PERSON": 2,
+    "I-PHONE": 34,
+    "I-RRN": 6,
+    "I-SECRET": 22,
+    "I-USER_ID": 26,
+    "I-ZIPCODE": 54,
+    "O": 0,
+    "S-ACCOUNT_NUMBER": 20,
+    "S-ADDRESS": 52,
+    "S-CARD_EXPIRY": 64,
+    "S-CARD_NUMBER": 16,
+    "S-CVC": 68,
+    "S-DATE": 60,
+    "S-DRIVER_LICENSE": 44,
+    "S-EMAIL": 32,
+    "S-FRN": 12,
+    "S-GENERIC_ID": 48,
+    "S-IPIN": 72,
+    "S-PASSPORT": 40,
+    "S-PERSON": 4,
+    "S-PHONE": 36,
+    "S-RRN": 8,
+    "S-SECRET": 24,
+    "S-USER_ID": 28,
+    "S-ZIPCODE": 56
+  },
+  "max_position_embeddings": 131072,
+  "model_type": "openai_privacy_filter",
+  "num_attention_heads": 14,
+  "num_experts_per_tok": 4,
+  "num_hidden_layers": 8,
+  "num_key_value_heads": 2,
+  "num_local_experts": 128,
+  "output_router_logits": false,
+  "pad_token_id": 199999,
+  "rms_norm_eps": 1e-05,
+  "rope_parameters": {
+    "beta_fast": 32.0,
+    "beta_slow": 1.0,
+    "factor": 32.0,
+    "original_max_position_embeddings": 4096,
+    "rope_theta": 150000.0,
+    "rope_type": "yarn",
+    "truncate": false
+  },
+  "router_aux_loss_coef": 0.001,
+  "sliding_window": 128,
+  "tie_word_embeddings": false,
+  "transformers.js_config": {
+    "use_external_data_format": {
+      "model": 1,
+      "model.onnx": 3,
+      "model_fp16.onnx": 2
+    }
+  },
+  "transformers_version": "5.13.1",
+  "use_cache": false,
+  "vocab_size": 200064
+}

label-taxonomy.yaml

@@ -0,0 +1,242 @@
+# =============================================================================
+# Label Mapping - ai4privacy 1.5m 라벨 -> 내부 학습 라벨 매핑 테이블
+#
+# * 역할: 모델 학습 라벨 = 로깅 스키마 키 = 정책 테이블 키의 단일 소스
+#   (docs/handoff/2026-07-03-pii-masking-architecture-qna.md [13])
+# * 역산 근거: playbooks/privacy_filter/privacy-filter-policy.md (표 1 정형 패턴 / 표 2 마스킹 기준)
+# * 실측 근거: docs/handoff/2026-07-03-ai4privacy-1p5m-ko-audit.md (v0 초안 + ko 서브셋 감사)
+# * v0 -> v1 변경: 주소 처리 확정(단일+서브마스킹) / FRN 신설 / USER_ID·ACCOUNT_NUMBER
+#   드롭 철회 / 여권·운전면허 분리 유지 / regex 전담 항목 분리
+# * 2026-07-21: AGE·GENDER 드롭 확정 (표1·표2 무근거 - 정책 역산 원칙 관철, N=17 -> 15)
+# * 2026-07-21: 표2 문맥 항목 3종 모델 승격 (CARD_EXPIRY·CVC·IPIN - 문맥 검출은 모델 레인
+#   본령, 키워드 regex는 이중 레인 병행. N=15 -> 18, 61 -> 73클래스)
+# =============================================================================
+
+version: v1
+date: 2026-07-21
+basis:
+  policy: playbooks/privacy_filter/privacy-filter-policy.md
+  audit: docs/handoff/2026-07-03-ai4privacy-1p5m-ko-audit.md
+
+# -----------------------------------------------------------------------------
+# 1. 모델 학습 라벨 (N=18 -> BIOES 4N+1 = 73클래스)
+#    * data_source: 라벨 표본의 출처 리스트 - ko(1.5m-ko 정제본) / en(1.5m-en 리플레이) /
+#      domain(도메인 부트스트랩 대기 - 헤드는 유사 라벨 행 복사로 init)
+#    * RRN의 1.5m 유래분은 ko 한정 (en TAXNUM은 GENERIC_ID fallback) - domain은 증강 방식 B
+#    * en 분포 실측 (2026-07-13, 163,740행/1,247,393스팬): 상위 19종 집중 - PASSWORD 0,
+#      ACCOUNTNUM 1, USERNAME 20 -> "희소 라벨은 en 담당" 가설 기각, [domain] 재배정
+# -----------------------------------------------------------------------------
+model_labels:
+  - name: PERSON
+    description: 성명 (성+이름 병합 단일 스팬)
+    policy_ref: "표2 성명 (김*용 - 첫·끝 글자 제외) / 표2 성명(영문) (앞 4자리 노출) - 치환 시 스크립트(한글/영문)로 규칙 분기, full-span 경계 필수"
+    data_source: [ko, en]
+    head_init_base: private_person
+  - name: RRN
+    description: 주민등록번호
+    policy_ref: "표1 주민번호 / 표2 주민등록번호 (뒤 7자리 마스킹)"
+    data_source: [ko, domain]     # domain = 증강 방식 B 신규 생성분 (전략문 §3.3.1)
+    head_init_base: account_number
+  - name: FRN
+    description: 외국인등록번호 (신설 - 1.5m 소스 0건)
+    policy_ref: "표1 외국인등록번호 ([5-8] 시작) / 표2 주민등록번호(외국인등록번호 포함) (뒤 7자리)"
+    data_source: [domain]
+    head_init_from: RRN
+  - name: CARD_NUMBER
+    description: 신용/체크카드 번호
+    policy_ref: "표1 카드번호 / 표2 카드번호 (7~12번째 자리, PCI-DSS)"
+    data_source: [ko, en, domain] # domain = 증강 방식 B 신규 생성분 (전략문 §3.3.1)
+    head_init_base: account_number
+  - name: ACCOUNT_NUMBER
+    description: 계좌번호 (v0 드롭 철회 - 표2 마스킹 기준 존재)
+    policy_ref: "표1 계좌번호 (전화번호 흡수 위험 - recognizer 우선순위 필요) / 표2 계좌번호 (뒤 5자리)"
+    data_source: [domain]          # ko 5·en 1건 실측 - 합성 증강 필수 (은행별 포맷 규칙 생성 용이)
+    head_init_base: account_number
+  - name: SECRET
+    description: 인증 시크릿 (비밀번호·API 키·토큰 통합 - 카드/회원/ISP 구분은 문맥 불가 + 액션 동일)
+    policy_ref: "표2 카드비밀번호·온라인 회원 패스워드·ISP비밀번호 (처리 금지 -> 게이트웨이에서는 LLM 노출 금지 = 전체 마스킹으로 해석)"
+    data_source: [domain]          # 1.5m 전무 (ko 4·en 0 실측) - 도메인/합성 증강 전담
+    head_init_base: secret         # base 8종 중 secret 행 정확 복사 (인접 아닌 직계 상속)
+  - name: USER_ID
+    description: 온라인 회원 ID (v0 드롭 철회 - 표2 마스킹 기준 존재)
+    policy_ref: "표2 온라인 회원 ID (앞 2자리 제외)"
+    data_source: [domain]          # ko 4·en 20건 실측 - 증강 필수
+    head_init_base: account_number
+  - name: EMAIL
+    description: 이메일 주소
+    policy_ref: "표1 이메일 / 표2 이메일주소 (ID 앞 2자리 제외 마스킹)"
+    data_source: [ko, en]
+    head_init_base: private_email
+  - name: PHONE
+    description: 전화번호 (휴대폰/일반전화 통합 - 치환 단계에서 프리픽스로 재분류)
+    policy_ref: "표1 휴대폰번호·전화번호 / 표2 기본 뒤 6자리 공통 - 내부망 채널 한정 휴대폰
+      뒤 4자리 완화 (치환 시 값 프리픽스로 휴대폰 판별 + 채널 축 분기)"
+    data_source: [ko, en]
+    head_init_base: private_phone
+  - name: PASSPORT
+    description: 여권번호 (분리 유지 - 마스킹 기준 상이 + ko 3,211건)
+    policy_ref: "표1 여권번호 / 표2 여권번호 (뒤 4자리)"
+    data_source: [ko, en]
+    head_init_base: account_number
+  - name: DRIVER_LICENSE
+    description: 운전면허번호 (분리 유지 - 마스킹 기준 상이 + ko 4,092건)
+    policy_ref: "표1 운전면허번호 / 표2 운전면허번호 (중간 6자리)"
+    data_source: [ko, en]
+    head_init_base: account_number
+  - name: GENERIC_ID
+    description: 범용 식별자 (한국 대응물 없는 ID류 - recall-first로 O 대신 유지)
+    policy_ref: "audit §5 (SOCIALNUM 10자리·IDCARDNUM 영숫자)"
+    data_source: [ko, en]
+    head_init_base: account_number
+  - name: ADDRESS
+    description: 주소 (시·도로명·건물번호 통합 스팬 - 치환 단계에서 숫자 서브마스킹)
+    policy_ref: "표2 주소 (지번: 읍/면/동 미만 숫자 / 도로명: 건물번호·상세주소 숫자 마스킹 -> 스팬 내 규칙 치환)"
+    data_source: [ko, en]
+    head_init_base: private_address
+  - name: ZIPCODE
+    description: 우편번호 (분리 유지 - keep 토글 세분성 전제)
+    policy_ref: "handoff [10] ZIPCODE keep 실증"
+    data_source: [ko, en]
+    head_init_base: private_address
+  - name: DATE
+    description: 날짜·시각 (생년월일 미분리 - ko 데이터 DOB 0건 + 과잉 마스킹 준수 인정 조항)
+    policy_ref: "표2 생년월일 (노출 금지 - DATE 전체 마스킹으로 초과 준수)"
+    data_source: [ko, en]
+    head_init_base: private_date
+  - name: CARD_EXPIRY
+    description: 카드유효기한 (표2 문맥 항목 승격 2026-07-21 - 키워드 regex 이중 레인 병행)
+    policy_ref: "표2 카드유효기한 (**/** 전체 마스킹)"
+    data_source: [domain]          # 1.5m 원천 0건 - 증강 전담 (PAN 동반 문맥 생성)
+    head_init_base: private_date
+  - name: CVC
+    description: 카드검증코드 (CVC/CVV/CAV 통합 - 표2 문맥 항목 승격 2026-07-21)
+    policy_ref: "표2 카드검증코드 (저장·출력 금지 -> 게이트웨이 전체 마스킹 해석)"
+    data_source: [domain]          # 1.5m 원천 0건 - 증강 전담 (단독 생성 금지 - PAN 동반 필수)
+    head_init_base: account_number
+  - name: IPIN
+    description: I-PIN 번호 (표2 문맥 항목 승격 2026-07-21 - 값 규격 pending)
+    policy_ref: "표2 I-PIN (뒤 5자리 - 하이픈 무관 숫자 기준)"
+    data_source: [domain]          # 1.5m 원천 0건 - 규격 확정 전 파일럿 수량만
+    head_init_base: account_number
+
+# -----------------------------------------------------------------------------
+# 2. 소스 라벨 매핑 (ai4privacy 1.5m -> 내부 라벨)
+#    * "O" = 학습에서 비엔티티 처리 (드롭 리스트 - 정책팀 컨펌 후 확정)
+#    * 조건부 매핑은 locale + value_pattern 안전장치 동반
+# -----------------------------------------------------------------------------
+source_mapping: # <source_mapping>
+  GIVENNAME: PERSON
+  SURNAME: PERSON
+  TITLE: O                        # 군/양/gun/yang 직역 존칭 88% - 전량 드롭
+  TAXNUM:
+    label: RRN  # <conditional_mapping>
+    condition:
+      locale: ko
+      value_pattern: '^\d{6}-[1-4]\d{6}$'   # 실측 3,975/3,975 정합 - 안전장치
+    fallback: GENERIC_ID          # 비정합 값 / 타 로케일(en 리플레이)은 세무 ID -> 범용 ID
+  SOCIALNUM: GENERIC_ID
+  IDCARDNUM: GENERIC_ID
+  DRIVERLICENSENUM: DRIVER_LICENSE
+  PASSPORTNUM: PASSPORT
+  TELEPHONENUM: PHONE
+  EMAIL: EMAIL
+  CREDITCARDNUMBER: CARD_NUMBER
+  ACCOUNTNUM: ACCOUNT_NUMBER
+  CITY: ADDRESS
+  STREET: ADDRESS
+  BUILDINGNUM: ADDRESS
+  ZIPCODE: ZIPCODE
+  DATE: DATE
+  TIME: DATE
+  AGE: O                          # 2026-07-21 드롭 확정 - 표1·표2 무근거 (quasi-identifier 존치안 폐기)
+  GENDER: O                       # 〃
+  SEX: O                          # 〃 (GENDER 중복 소스)
+  USERNAME: USER_ID
+  PASSWORD: SECRET                # ko 4·en 0건 - 시드 미미, 본 표본은 도메인/합성 증강
+  # 롱테일 (ko <= 40건, 학습 라벨 부적격) - 정책팀 컨펌 대기 드롭 리스트
+  ORGANISATION: O
+  URL: O
+  AMOUNT: O
+  COUNTRY: O
+  CURRENCY: O
+  BANKNAME: O
+  TIMEZONE: O
+  SALARY: O
+  IPV4: O                         # 고객 IP는 표2 항목이나 ko 2건 - 1차 regex(표1 IP주소) 전담
+  JOBTITLE: O
+  HOSPITALNAME: O
+  ALLERGIES: O
+  WEIGHT: O                       # en 전용 라벨 (ko 0건) - 2026-07-13 en 전수 검증에서 발견, 정책 외
+  HEIGHT: O                       # validation 전용 희소 라벨 (ko 1건 + en 1건) - 2026-07-22 refine fail-fast로 발견, 정책 외
+
+# -----------------------------------------------------------------------------
+# 3. 병합 규칙 (정제 스크립트 2단계 - 인접 스팬 병합, ko·en 공통)
+#    * 병합은 언어 공통 (en "John Smith"도 단일 스팬이어야 표2 영문 성명 규칙 성립)
+#    * ko 전용은 병합이 아니라 그 다음의 "자연화"(성+명 붙여쓰기 - 전��문 §3.4.2 (1) 2단계)
+#    * 순서 강제 금지: ko 주소는 큰 -> 작, en 주소는 작 -> 큰 역순 - 인접성만 조건
+#
+# * gap_allowed : 두 엔티티 사이의 인식·허용이 가능한 구조 및 패턴
+#     - whitespace (공백·탭)
+#     - punct (구두점 : `, . - · / ( )` 등 문장 부호)
+# -----------------------------------------------------------------------------
+merge_rules:  # <merge_rules>
+  person:
+    source_labels: [GIVENNAME, SURNAME]
+    max_gap_chars: 4
+    gap_allowed: whitespace_or_punct
+    output_label: PERSON
+  address:
+    source_labels: [CITY, STREET, BUILDINGNUM]
+    max_gap_chars: 4
+    gap_allowed: whitespace_or_punct
+    output_label: ADDRESS
+
+# -----------------------------------------------------------------------------
+# 4. 1차 regex 전담 항목 (모델 라벨 제외 - 로깅 스키마 키에는 포함)
+#    * 표1 정형 패턴 recognizer가 검출, 치환 정책은 표2 준용
+# -----------------------------------------------------------------------------
+regex_only:
+  - key: CI
+    policy_ref: "표1 CI (86자+`==` 고정, lookaround 경계 - 검수 보강 2026-07-21) / 표2 CI (앞 7자리 노출)"
+    # SHA-512 base64 동형 과탐 허용 - 정밀도 필요시 ci·연계정보 키워드 가점
+  - key: IP_ADDRESS
+    policy_ref: "표1 IP주소 (옥텟 0~255 엄격형 - 검수 보강 2026-07-21) / 표2 고객의 IP주소 (앞 3자리 = 첫 옥텟 마스킹 해석, IPv4 한정)"
+
+# -----------------------------------------------------------------------------
+# 4-1. 1차 regex 병행 (이중 레인) - 모델 라벨이면서 키워드 regex도 병행
+#    * 2026-07-21 모델 승격 3종: 키워드 regex(고정밀 저recall - 명시 문형) + 모델(무키워드
+#      맥락 recall) union - 문맥 의존 검출은 모델 레인의 존재 이유
+#    * 패턴 엔진 전제: Python re (lookbehind 사용 - RE2/Hyperscan 계열 비호환)
+#    * 치환은 값 캡처그룹만 (키워드 보존 = 추적성)
+# -----------------------------------------------------------------------------
+regex_assist:
+  - key: CARD_EXPIRY
+    patterns:
+      keyword: '(?i)(유효\s*기[간한]|expir\w*|valid\s*thru)\D{0,12}(0[1-9]|1[0-2])\s*[/.\-]\s*((?:20)?\d\d)(?!\d)'
+      pan_adjacent: '(0[1-9]|1[0-2])\s*/\s*\d{2}(?!\d)'   # 카드번호 매치 직후 \D{0,20} 윈도우 내에서만 적용
+  - key: CVC
+    patterns:
+      keyword: '(?i)(\b(?:cv[vc]2?|security\s*code)\b|카드\s*검증\s*(?:번호|코드|값)?|보안\s*코드)\W{0,6}(?!(?:19|20)\d\d(?!\d))(\d{3,4})(?!\d)'
+      # CID·CSC 키워드 기본 제외 (correlation ID 충돌). 정밀도 옵션: PAN co-occurrence 게이트
+  - key: IPIN
+    patterns:
+      keyword: '(?i)(아이핀|i[-\s]?pin)\s*(?:번호|no\.?)?\D{0,8}(\d{6}[-\s]?\d{7})(?!\d)'
+      # 주민번호 규칙 선행 적용 후 잔여분만 IPIN. 값 규격(13자리 가정) 확인 필요 (pending)
+
+# -----------------------------------------------------------------------------
+# 5. 스코프 외 - 텍스트 게이트웨이가 다루지 않는 표2 항목 (선언적 기록)
+# -----------------------------------------------------------------------------
+out_of_scope:
+  - key: VIDEO_PERSONAL_INFO
+    policy_ref: "표2 개인영상정보 (검정 목적 외 처리 금지)"
+    reason: "텍스트 엔티티가 아닌 모달리티(이미지·영상) - 멀티모달 입력 경로가 열리면
+      텍스트 필터를 우회하므로 별도 통제(입력 차단 or 비전 필터) 필요"
+
+# -----------------------------------------------------------------------------
+# 6. 미결 - 정책팀 컨펌 항목, TBD
+# -----------------------------------------------------------------------------
+pending_confirmation:
+  - "드롭 리스트(O 매핑) 전체 = 비PII 선언 - 특히 ORGANISATION·AMOUNT·SALARY"
+  - "IPIN 값 규격 (13자리 가정) - 제공기관 계약 확인 (CVC 검출 규칙은 2026-07-21 키워드 동반 패턴 채택으로 해소)"
+  - "SECRET 신설 채택 (2026-07-13) - 표2 처리금지류의 '게이트웨이 전체 마스킹' 해석은
+    정책팀 사후 컨펌 대상"

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9beec7575336b7c05ee59d7da9dfd25027b241ae5507e1ac4f7a04d7a10d83cb
+size 2799040778

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0614fe83cadab421296e664e1f48f4261fa8fef6e03e63bb75c20f38e37d07d3
+size 27868174

tokenizer_config.json

@@ -0,0 +1,13 @@
+{
+  "backend": "tokenizers",
+  "eos_token": "<|endoftext|>",
+  "is_local": false,
+  "local_files_only": false,
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "model_max_length": 128000,
+  "pad_token": "<|endoftext|>",
+  "tokenizer_class": "TokenizersBackend"
+}

viterbi_calibration.json

@@ -0,0 +1,14 @@
+{
+  "operating_points": {
+    "default": {
+      "biases": {
+        "transition_bias_background_stay": 0.0,
+        "transition_bias_background_to_start": 0.0,
+        "transition_bias_end_to_background": 0.0,
+        "transition_bias_end_to_start": 0.0,
+        "transition_bias_inside_to_continue": 0.0,
+        "transition_bias_inside_to_end": 0.0
+      }
+    }
+  }
+}