Release v2.1 · model + card

.gitattributes

@@ -0,0 +1,3 @@
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Digitflow
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -0,0 +1,176 @@
+---
+license: mit
+language:
+- de
+base_model: openai/privacy-filter
+pipeline_tag: token-classification
+library_name: opf
+tags:
+- pii
+- privacy
+- ner
+- token-classification
+- german
+- de
+- privacy-filter
+- opf
+datasets:
+- ai4privacy/open-pii-masking-500k-ai4privacy
+metrics:
+- f1
+model-index:
+- name: digitflow/privacy-filter-de-ft
+  results:
+  - task:
+      type: token-classification
+      name: PII detection (German)
+    dataset:
+      name: ai4privacy/open-pii-masking-500k-ai4privacy (de validation, n=1,000)
+      type: ai4privacy/open-pii-masking-500k-ai4privacy
+      split: validation
+      args:
+        language: de
+    metrics:
+    - type: f1
+      value: 0.8706
+      name: OPF-containment F1 (char-level, label-agnostic)
+    - type: f1
+      value: 0.8368
+      name: Char-coverage F1 (label-aware)
+    - type: f1
+      value: 0.6445
+      name: Strict span F1
+---
+
+# digitflow/privacy-filter-de-ft
+
+A German-language fine-tune of [`openai/privacy-filter`](https://huggingface.co/openai/privacy-filter).
+It exposes the same inference API and OPF label space as the base
+model, so existing OPF call sites work without changes on German
+input.
+
+**Caveat.** This model is not a perfect redactor for German PII. No
+warranty is provided and Digitflow accepts no legal responsibility
+for decisions made on its output. Use at your own risk. For
+non-German text, use [`openai/privacy-filter`](https://huggingface.co/openai/privacy-filter)
+directly.
+
+## Benchmark
+
+Evaluated on the German subset (`language == 'de'`, n = 1,000) of the
+[`ai4privacy/open-pii-masking-500k-ai4privacy`](https://huggingface.co/datasets/ai4privacy/open-pii-masking-500k-ai4privacy)
+validation split, scored with OPF-containment F1 (the char-level,
+label-agnostic completeness metric from the OPF reference scoring
+code). 95 % confidence intervals are estimated by 1,000-sample
+bootstrap resampling with replacement, taking the 2.5th and 97.5th
+percentiles of the resulting F1 distribution.
+
+| Metric | `openai/privacy-filter` | `digitflow/privacy-filter-de-ft` | Δ |
+|---|---:|---:|---:|
+| **OPF-containment F1** | 0.8437 | **0.8706** | **+0.027** |
+| Leak rate (1 − char recall, label-agnostic) | 23.05 % | **20.49 %** | **−2.56 pp** |
+| Char-coverage F1, label-aware | 0.6791 | **0.8368** | **+0.158** |
+| Strict span F1 | 0.4348 | **0.6445** | **+0.210** |
+| Strict span precision | 0.5645 | **0.7518** | +0.187 |
+| Strict span recall | 0.3536 | **0.5640** | +0.210 |
+
+| Model | OPF-containment F1 | 95 % bootstrap CI |
+|---|---:|---|
+| `openai/privacy-filter` | 0.8437 | [0.8294, 0.8579] |
+| `digitflow/privacy-filter-de-ft` | 0.8706 | [0.8585, 0.8812] |
+
+The intervals do not overlap; the +0.027 lift is significant against
+single-slice sampling noise.
+
+## Examples
+
+Output of `m.redact(text)`, formatted as `label:'redacted text'`.
+`(none)` means the model returned no spans.
+
+| Input | `openai/privacy-filter` | `digitflow/privacy-filter-de-ft` |
+|---|---|---|
+| Mein Name ist Jürgen Müller und ich wohne in Hamburg. | `(none)` | `private_person:'Jürgen Müller'`, `private_address:'Hamburg'` |
+| Mein Passwort lautet SicherPasswort123! | `(none)` | `secret:'SicherPasswort123!'` |
+| Senden Sie das Paket an Hauptstraße 25, 10115 Berlin. | `(none)` | `private_address:'Hauptstraße 25, 10115 Berlin'` |
+| Hans-Jürgen Brömmelmeyer hat den Termin bestätigt. | `(none)` | `private_person:'Hans-Jürgen Brömmelmeyer'` |
+| Server-Status: https://intern.firma.de/health. | `(none)` | `private_url:'https://intern.firma.de/health'` |
+| Termin mit Mariella von Schönefeld-Brixius um 15:00. | `private_person:'Mariella von Schönefeld-Brixius'` | `private_person:'Mariella von Schönefeld-Brixius'`, `private_date:'15:00'` |
+
+## How it was built
+
+The fine-tune adapts the base model to German PII through slot-filled
+augmentation of public German carriers.
+
+It is supplemented by a hand-authored curriculum spanning real-world
+text registers, and trained on a single NVIDIA Jetson Orin.
+
+The training set is screened against the evaluation slice for
+contamination before training begins.
+
+## How to use it
+
+The OPF Python API is unchanged. Fetch the checkpoint with
+`huggingface_hub.snapshot_download(...)` and pass the resulting local
+path to `opf.OPF`.
+
+```python
+from huggingface_hub import snapshot_download
+import opf
+
+path = snapshot_download("digitflow/privacy-filter-de-ft")
+
+m = opf.OPF(
+    model=path,
+    device="cuda",
+    output_mode="typed",
+    decode_mode="viterbi",
+)
+
+text = "Mein Name ist Jürgen Müller und ich wohne in Hamburg."
+result = m.redact(text)
+for span in result.detected_spans:
+    print(f"{span.label}: {text[span.start:span.end]!r}")
+# private_person: 'Jürgen Müller'
+# private_address: 'Hamburg'
+```
+
+`snapshot_download` caches the weights under `~/.cache/huggingface/`
+so subsequent calls are free. The current `opf` release does not
+resolve a Hub repo id directly; it expects a local checkpoint
+directory.
+
+### Reproducing the benchmark
+
+```python
+from datasets import load_dataset
+from huggingface_hub import snapshot_download
+import opf
+# ... plus shared.span_prf and metrics.char_coverage_prf from the
+# openai/privacy-filter reference scoring code.
+
+ds = load_dataset(
+    "ai4privacy/open-pii-masking-500k-ai4privacy",
+    split="validation",
+)
+de = ds.filter(lambda r: r["language"] == "de").select(range(1000))
+
+ft_path = snapshot_download("digitflow/privacy-filter-de-ft")
+m_base = opf.OPF(device="cuda", output_mode="typed", decode_mode="viterbi")
+m_ft   = opf.OPF(model=ft_path,
+                 device="cuda", output_mode="typed", decode_mode="viterbi")
+
+# Run m.redact() per row, collect predicted spans, score against gold
+# with `char_coverage_prf(predictions, golds, label_aware=False)`.
+# Report the __micro__.f1 as OPF-containment F1.
+```
+
+## License and citations
+
+**License.** [MIT](./LICENSE).
+
+[`ai4privacy/open-pii-masking-500k-ai4privacy`](https://huggingface.co/datasets/ai4privacy/open-pii-masking-500k-ai4privacy)
+was used as the source of training carriers (with augmentation) and
+as the validation slice for the benchmark above.
+
+[`openai/privacy-filter`](https://huggingface.co/openai/privacy-filter)
+is the base model (Apache 2.0).

USAGE.txt

@@ -0,0 +1,27 @@
+digitflow/privacy-filter-de-ft
+A German fine-tune of openai/privacy-filter. Drop-in on German text.
+
+Quick start (Python, opf):
+    from huggingface_hub import snapshot_download
+    import opf
+
+    path = snapshot_download("digitflow/privacy-filter-de-ft")
+    m = opf.OPF(
+        model=path,
+        device="cuda",
+        output_mode="typed",
+        decode_mode="viterbi",
+    )
+    result = m.redact("Mein Name ist Jürgen Müller und ich wohne in Hamburg.")
+    for span in result.detected_spans:
+        print(span.label, repr(span.text))
+    # private_person 'Jürgen Müller'
+    # private_address 'Hamburg'
+
+Note: the current opf release loads from a local checkpoint directory,
+not from a Hub repo id directly. Fetch with snapshot_download first.
+The cache lives under ~/.cache/huggingface/ so subsequent runs are
+free.
+
+See README.md for the full model card, benchmark numbers, examples,
+and when not to use this model.

config.json

@@ -0,0 +1,74 @@
+{
+  "bidirectional_context": true,
+  "bidirectional_left_context": 128,
+  "bidirectional_right_context": 128,
+  "category_version": "v2",
+  "default_n_ctx": 128000,
+  "encoding": "o200k_base",
+  "experts_per_token": 4,
+  "head_dim": 64,
+  "hidden_size": 640,
+  "inference_contract_version": 1,
+  "initial_context_length": 4096,
+  "intermediate_size": 640,
+  "max_position_embeddings": 131072,
+  "model_type": "privacy_filter",
+  "ner_class_names": [
+    "O",
+    "B-account_number",
+    "I-account_number",
+    "E-account_number",
+    "S-account_number",
+    "B-private_address",
+    "I-private_address",
+    "E-private_address",
+    "S-private_address",
+    "B-private_date",
+    "I-private_date",
+    "E-private_date",
+    "S-private_date",
+    "B-private_email",
+    "I-private_email",
+    "E-private_email",
+    "S-private_email",
+    "B-private_person",
+    "I-private_person",
+    "E-private_person",
+    "S-private_person",
+    "B-private_phone",
+    "I-private_phone",
+    "E-private_phone",
+    "S-private_phone",
+    "B-private_url",
+    "I-private_url",
+    "E-private_url",
+    "S-private_url",
+    "B-secret",
+    "I-secret",
+    "E-secret",
+    "S-secret"
+  ],
+  "num_attention_heads": 14,
+  "num_experts": 128,
+  "num_hidden_layers": 8,
+  "num_key_value_heads": 2,
+  "num_labels": 33,
+  "param_dtype": "bfloat16",
+  "rope_ntk_alpha": 1.0,
+  "rope_ntk_beta": 32.0,
+  "rope_scaling_factor": 32.0,
+  "rope_theta": 150000,
+  "sliding_window": 257,
+  "span_class_names": [
+    "O",
+    "account_number",
+    "private_address",
+    "private_date",
+    "private_email",
+    "private_person",
+    "private_phone",
+    "private_url",
+    "secret"
+  ],
+  "vocab_size": 200064
+}

dtypes.json

@@ -0,0 +1,92 @@
+{
+  "embedding.weight": "torch.bfloat16",
+  "block.0.attn.qkv.weight": "torch.bfloat16",
+  "block.0.attn.qkv.bias": "torch.bfloat16",
+  "block.0.attn.sinks": "torch.float32",
+  "block.0.attn.out.weight": "torch.bfloat16",
+  "block.0.attn.out.bias": "torch.bfloat16",
+  "block.0.mlp.gate.weight": "torch.bfloat16",
+  "block.0.mlp.gate.bias": "torch.bfloat16",
+  "block.0.mlp.swiglu.weight": "torch.bfloat16",
+  "block.0.mlp.swiglu.bias": "torch.bfloat16",
+  "block.0.mlp.out.weight": "torch.bfloat16",
+  "block.0.mlp.out.bias": "torch.bfloat16",
+  "block.1.attn.qkv.weight": "torch.bfloat16",
+  "block.1.attn.qkv.bias": "torch.bfloat16",
+  "block.1.attn.sinks": "torch.float32",
+  "block.1.attn.out.weight": "torch.bfloat16",
+  "block.1.attn.out.bias": "torch.bfloat16",
+  "block.1.mlp.gate.weight": "torch.bfloat16",
+  "block.1.mlp.gate.bias": "torch.bfloat16",
+  "block.1.mlp.swiglu.weight": "torch.bfloat16",
+  "block.1.mlp.swiglu.bias": "torch.bfloat16",
+  "block.1.mlp.out.weight": "torch.bfloat16",
+  "block.1.mlp.out.bias": "torch.bfloat16",
+  "block.2.attn.qkv.weight": "torch.bfloat16",
+  "block.2.attn.qkv.bias": "torch.bfloat16",
+  "block.2.attn.sinks": "torch.float32",
+  "block.2.attn.out.weight": "torch.bfloat16",
+  "block.2.attn.out.bias": "torch.bfloat16",
+  "block.2.mlp.gate.weight": "torch.bfloat16",
+  "block.2.mlp.gate.bias": "torch.bfloat16",
+  "block.2.mlp.swiglu.weight": "torch.bfloat16",
+  "block.2.mlp.swiglu.bias": "torch.bfloat16",
+  "block.2.mlp.out.weight": "torch.bfloat16",
+  "block.2.mlp.out.bias": "torch.bfloat16",
+  "block.3.attn.qkv.weight": "torch.bfloat16",
+  "block.3.attn.qkv.bias": "torch.bfloat16",
+  "block.3.attn.sinks": "torch.float32",
+  "block.3.attn.out.weight": "torch.bfloat16",
+  "block.3.attn.out.bias": "torch.bfloat16",
+  "block.3.mlp.gate.weight": "torch.bfloat16",
+  "block.3.mlp.gate.bias": "torch.bfloat16",
+  "block.3.mlp.swiglu.weight": "torch.bfloat16",
+  "block.3.mlp.swiglu.bias": "torch.bfloat16",
+  "block.3.mlp.out.weight": "torch.bfloat16",
+  "block.3.mlp.out.bias": "torch.bfloat16",
+  "block.4.attn.qkv.weight": "torch.bfloat16",
+  "block.4.attn.qkv.bias": "torch.bfloat16",
+  "block.4.attn.sinks": "torch.float32",
+  "block.4.attn.out.weight": "torch.bfloat16",
+  "block.4.attn.out.bias": "torch.bfloat16",
+  "block.4.mlp.gate.weight": "torch.bfloat16",
+  "block.4.mlp.gate.bias": "torch.bfloat16",
+  "block.4.mlp.swiglu.weight": "torch.bfloat16",
+  "block.4.mlp.swiglu.bias": "torch.bfloat16",
+  "block.4.mlp.out.weight": "torch.bfloat16",
+  "block.4.mlp.out.bias": "torch.bfloat16",
+  "block.5.attn.qkv.weight": "torch.bfloat16",
+  "block.5.attn.qkv.bias": "torch.bfloat16",
+  "block.5.attn.sinks": "torch.float32",
+  "block.5.attn.out.weight": "torch.bfloat16",
+  "block.5.attn.out.bias": "torch.bfloat16",
+  "block.5.mlp.gate.weight": "torch.bfloat16",
+  "block.5.mlp.gate.bias": "torch.bfloat16",
+  "block.5.mlp.swiglu.weight": "torch.bfloat16",
+  "block.5.mlp.swiglu.bias": "torch.bfloat16",
+  "block.5.mlp.out.weight": "torch.bfloat16",
+  "block.5.mlp.out.bias": "torch.bfloat16",
+  "block.6.attn.qkv.weight": "torch.bfloat16",
+  "block.6.attn.qkv.bias": "torch.bfloat16",
+  "block.6.attn.sinks": "torch.float32",
+  "block.6.attn.out.weight": "torch.bfloat16",
+  "block.6.attn.out.bias": "torch.bfloat16",
+  "block.6.mlp.gate.weight": "torch.bfloat16",
+  "block.6.mlp.gate.bias": "torch.bfloat16",
+  "block.6.mlp.swiglu.weight": "torch.bfloat16",
+  "block.6.mlp.swiglu.bias": "torch.bfloat16",
+  "block.6.mlp.out.weight": "torch.bfloat16",
+  "block.6.mlp.out.bias": "torch.bfloat16",
+  "block.7.attn.qkv.weight": "torch.bfloat16",
+  "block.7.attn.qkv.bias": "torch.bfloat16",
+  "block.7.attn.sinks": "torch.float32",
+  "block.7.attn.out.weight": "torch.bfloat16",
+  "block.7.attn.out.bias": "torch.bfloat16",
+  "block.7.mlp.gate.weight": "torch.bfloat16",
+  "block.7.mlp.gate.bias": "torch.bfloat16",
+  "block.7.mlp.swiglu.weight": "torch.bfloat16",
+  "block.7.mlp.swiglu.bias": "torch.bfloat16",
+  "block.7.mlp.out.weight": "torch.bfloat16",
+  "block.7.mlp.out.bias": "torch.bfloat16",
+  "unembedding.weight": "torch.bfloat16"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6ef1dd109ba977e8baa611ae2f0fcb037e1782d342fe388318d9bde21882d14e
+size 2798983976

viterbi_calibration.json

@@ -0,0 +1,14 @@
+{
+  "operating_points": {
+    "default": {
+      "biases": {
+        "transition_bias_background_stay": 0.0,
+        "transition_bias_background_to_start": 0.0,
+        "transition_bias_end_to_background": 0.0,
+        "transition_bias_end_to_start": 0.0,
+        "transition_bias_inside_to_continue": 0.0,
+        "transition_bias_inside_to_end": 0.0
+      }
+    }
+  }
+}