Add Excel redaction demo kit

README.md

@@ -14,23 +14,21 @@ pipeline_tag: token-classification
 
 # privacy-filter-finetuned
 
-A finetuned version of [openai/privacy-filter](https://huggingface.co/openai/privacy-filter) with three additional detection categories on top of the original eight — built for procurement, finance, and sales datasets where company names, prices, and business reference IDs are as sensitive as personal data.
+A finetuned version of [openai/privacy-filter](https://huggingface.co/openai/privacy-filter) with three additional detection categories for procurement, finance, and sales datasets where company names, prices, and business reference IDs can be sensitive.
 
-Runs entirely locally. No data leaves the device during inference.
+The model runs locally for inference. Input data is processed on the user's machine when using the included Excel demo script.
 
----
-
-## New Categories
+## Added Categories
 
-This checkpoint adds three categories to the base model's original eight:
+This checkpoint keeps the original OPF privacy labels and adds:
 
 | Category | What it detects | Examples |
 |---|---|---|
 | `company_name` | Supplier names, client account names, corporate entities | `Meridian Logistics Ltd`, `Apex Manufacturing PLC`, `TechStart Solutions Inc` |
-| `price` | Monetary values with currency symbols or codes | `£4,250.00`, `$12,500`, `€5,000`, `2,500 GBP` |
+| `price` | Monetary values with currency symbols or codes | `GBP 4,250.00`, `$12,500`, `EUR 5,000`, `2,500 GBP` |
 | `id_number` | Purchase orders, invoice numbers, reference IDs, case numbers, internal codes | `PO-00442`, `INV/2024/00567`, `REF-2024-001`, `CASE-20240315-001`, `ORD-78542` |
 
-## Full Label Space (11 categories)
+## Label Space
 
 ```json
 {
@@ -52,95 +50,116 @@ This checkpoint adds three categories to the base model's original eight:
 }
 ```
 
----
-
-## Usage
+## Quick Inference Demo
 
-Install the official `opf` package from the [openai/privacy-filter](https://github.com/openai/privacy-filter) repo:
+Install dependencies:
 
 ```bash
-git clone https://github.com/openai/privacy-filter
-cd privacy-filter
-pip install -e .
+pip install git+https://github.com/openai/privacy-filter.git huggingface_hub
 ```
 
-Download this checkpoint:
+Download and run the checkpoint:
 
 ```python
 from huggingface_hub import snapshot_download
+from opf import OPF
+
+checkpoint = snapshot_download("galexdav/privacy-filter-finetuned")
+model = OPF(model=checkpoint, device="cpu")
 
-snapshot_download(
-    repo_id="galexdav/privacy-filter-finetuned",
-    local_dir="./privacy-filter-finetuned"
-)
+result = model.redact("PO-00442 raised for Meridian Logistics Ltd, total GBP 4,250.00.")
+for span in result.detected_spans:
+    print(f"{span.label}: {span.text}")
 ```
 
-Run inference:
+CLI example:
 
-```python
-from opf import OPF
+```bash
+opf --checkpoint ./privacy-filter-finetuned --device cpu "Invoice INV/2024/00567 from Apex Manufacturing PLC, amount GBP 9,600.00"
+```
 
-model = OPF(model="./privacy-filter-finetuned", device="cpu")
-result = model.redact("PO-00442 raised for Meridian Logistics Ltd, total £4,250.00.")
+## Excel Redaction Demo
 
-for span in result.detected_spans:
-    print(f"{span.label}: '{span.text}'")
+The included `clean_excel.py` script:
+
+1. Loads an Excel workbook.
+2. Detects sensitive spans in selected text columns.
+3. Replaces each detected value with synthetic data.
+4. Reuses the same synthetic value when the same original value appears again in the run.
+5. Writes a new workbook with `_clean` columns and a `PII_Audit_Log` sheet.
+
+Install the demo requirements locally:
 
-# id_number: 'PO-00442'
-# company_name: 'Meridian Logistics Ltd'
-# price: '£4,250.00'
+```bash
+pip install -r requirements.txt
 ```
 
-Or via the CLI:
+Run with the included sample workbook:
 
 ```bash
-opf --checkpoint ./privacy-filter-finetuned --device cpu \
-  "Invoice INV/2024/00567 from Apex Manufacturing PLC — amount £9,600.00"
+python clean_excel.py dummy_procurement_data.xlsx --device cpu
 ```
 
----
+Run on your own workbook:
 
-## Training Details
+```bash
+python clean_excel.py your_data.xlsx --device cpu
+```
 
-| | |
-|---|---|
-| Base model | openai/privacy-filter |
-| Training examples | ~830 |
-| Validation examples | ~130 |
-| Epochs | 3 |
-| Hardware | 1× NVIDIA L4 (24 GB) via Hugging Face Jobs |
-| Training time | ~9 minutes |
+Choose specific columns:
 
-Training data was generated programmatically using a template × entity cross-product approach, covering all 11 categories including the original 8 (to prevent catastrophic forgetting).
+```bash
+python clean_excel.py your_data.xlsx --columns Supplier Notes ContactEmail --device cpu
+```
 
----
+Process every sheet:
+
+```bash
+python clean_excel.py your_data.xlsx --all-sheets --device cpu
+```
+
+The script auto-downloads this checkpoint from Hugging Face when `--checkpoint` is not supplied. To use a local checkpoint explicitly:
+
+```bash
+python clean_excel.py your_data.xlsx --checkpoint ./models/privacy-filter-finetuned --device cpu
+```
+
+Use `--device cuda` if you have a working CUDA GPU setup.
 
-## Test Results
+## Output Workbook
 
-22/24 tests passing on the held-out test suite:
+For an input called `dummy_procurement_data.xlsx`, the script creates:
 
-| Category | Result |
+```text
+dummy_procurement_data_cleaned.xlsx
+```
+
+The output includes:
+
+- Original columns unchanged.
+- New `<column>_clean` columns containing synthetic replacements.
+- `PII_Audit_Log` with label, original value, replacement value, sheet, row, and column.
+
+## Training Details
+
+| Field | Value |
 |---|---|
-| company_name (Ltd, PLC, Inc, LLP) | ✅ All pass |
-| price (£, $, €, written codes) | ✅ All pass |
-| id_number (PO, INV, REF, CASE, ORD) | ✅ All pass |
-| Multi-category (company + id + price in one sentence) | ✅ Pass |
-| private_person, private_email, private_phone | ✅ Pass |
-| private_address, account_number, private_date | ✅ Pass |
-| secret (API keys) | ⚠️ Occasional bleed into id_number |
-| Negative (no PII) | ⚠️ Occasional company_name false positive |
+| Base model | `openai/privacy-filter` |
+| Training examples | about 830 |
+| Validation examples | about 130 |
+| Epochs | 3 |
+| Hardware | 1x NVIDIA L4, 24 GB, via Hugging Face Jobs |
+| Training time | about 41 minutes |
 
----
+Training data was generated programmatically using templates and entity cross-products, covering all 11 categories including the original OPF categories to reduce catastrophic forgetting.
 
 ## Known Limitations
 
-- **Company names in isolated cells** — the model relies on surrounding sentence context to classify noun phrases as company names. A bare cell value with no surrounding text may not trigger detection.
-- **Price span boundaries** — currency symbols are occasionally left outside the detected span boundary in some formats.
-- **secret vs id_number** — API keys with alphanumeric patterns (e.g. `sk-live-...`) can be tagged as `id_number` rather than `secret`.
-- **English only** — inherited from the base model; performance on non-English text and regional naming conventions is limited.
-
----
+- Company names in isolated cells may be harder to detect without sentence context.
+- Price span boundaries may occasionally exclude a currency symbol.
+- API-key-like values can occasionally be classified as `id_number` rather than `secret`.
+- English only, inherited from the base model.
 
-## Licence
+## License
 
-Apache 2.0 — same as the base model. Commercial use permitted.
+Apache 2.0, same as the base model. Commercial use is permitted under the license terms.

clean_excel.py

@@ -0,0 +1,369 @@
+"""
+clean_excel.py - Run OpenAI Privacy Filter over an Excel file.
+Replaces detected PII with consistent synthetic fake data using Faker.
+
+Usage:
+    python clean_excel.py input.xlsx
+    python clean_excel.py input.xlsx --columns name email notes
+    python clean_excel.py input.xlsx --checkpoint /path/to/model
+    python clean_excel.py input.xlsx --sheet "Sheet2"
+    python clean_excel.py input.xlsx --all-sheets
+"""
+
+import argparse
+import os
+import re
+import sys
+from pathlib import Path
+from faker import Faker
+from openpyxl import load_workbook
+from openpyxl.styles import PatternFill, Font
+from tqdm import tqdm
+import pandas as pd
+from opf import OPF
+
+fake = Faker("en_GB")
+
+DEFAULT_HF_MODEL_REPO = "galexdav/privacy-filter-finetuned"
+DEFAULT_LOCAL_CHECKPOINT = Path("models") / "privacy-filter-finetuned"
+
+# Colour constants for the audit sheet
+HIGHLIGHT   = PatternFill("solid", start_color="FFF2CC", end_color="FFF2CC")
+HEADER_FILL = PatternFill("solid", start_color="D9E1F2", end_color="D9E1F2")
+BOLD        = Font(bold=True)
+
+# Consistency map: same real value maps to same fake value across entire run
+_replacement_cache: dict[str, str] = {}
+
+
+def _checkpoint_has_weights(path: Path) -> bool:
+    return (
+        path.is_dir()
+        and (path / "config.json").is_file()
+        and any(path.glob("*.safetensors"))
+    )
+
+
+def resolve_checkpoint(checkpoint: str | None) -> str:
+    """
+    Resolve the model checkpoint to use.
+
+    Priority:
+      1. --checkpoint argument
+      2. OPF_CHECKPOINT environment variable
+      3. local models/privacy-filter-finetuned directory
+      4. auto-download from the Hugging Face model repo
+    """
+    if checkpoint:
+        return str(Path(checkpoint).expanduser())
+
+    env_checkpoint = os.environ.get("OPF_CHECKPOINT")
+    if env_checkpoint:
+        return str(Path(env_checkpoint).expanduser())
+
+    if _checkpoint_has_weights(DEFAULT_LOCAL_CHECKPOINT):
+        return str(DEFAULT_LOCAL_CHECKPOINT)
+
+    try:
+        from huggingface_hub import snapshot_download
+    except ImportError:
+        print(
+            "huggingface_hub is required to auto-download the model. "
+            "Install demo dependencies with: pip install -r requirements.txt"
+        )
+        sys.exit(1)
+
+    print(f"Checkpoint not found locally. Downloading {DEFAULT_HF_MODEL_REPO}...")
+    return snapshot_download(
+        repo_id=DEFAULT_HF_MODEL_REPO,
+        repo_type="model",
+        local_dir=str(DEFAULT_LOCAL_CHECKPOINT),
+    )
+
+
+def _random_digits(length: int) -> str:
+    return "".join(str(fake.random_int(min=0, max=9)) for _ in range(length))
+
+
+def fake_company_name(original: str) -> str:
+    suffix_match = re.search(
+        r"\b(Ltd|Limited|PLC|LLP|Inc|Corp|Corporation)\.?$",
+        original.strip(),
+        flags=re.IGNORECASE,
+    )
+    if not suffix_match:
+        return fake.company()
+
+    suffix = suffix_match.group(1)
+    company = re.sub(
+        r"\s+(Ltd|Limited|PLC|LLP|Inc|Corp|Corporation)\.?$",
+        "",
+        fake.company(),
+        flags=re.IGNORECASE,
+    ).strip(" ,.")
+    return f"{company} {suffix}"
+
+
+def fake_price(original: str) -> str:
+    text = original.strip()
+    has_decimals = "." in text
+    whole = fake.random_int(min=100, max=999_999)
+    amount = f"{whole:,}"
+    if has_decimals:
+        amount = f"{amount}.{fake.random_int(min=0, max=99):02d}"
+
+    pound = "\u00a3"
+    euro = "\u20ac"
+    if pound in text:
+        return f"{pound}{amount}"
+    if "$" in text:
+        return f"${amount}"
+    if euro in text:
+        return f"{euro}{amount}"
+
+    currency_code = re.search(r"\b(GBP|USD|EUR)\b", text, flags=re.IGNORECASE)
+    if currency_code:
+        code = currency_code.group(1).upper()
+        if text.upper().startswith(code):
+            return f"{code} {amount}"
+        return f"{amount} {code}"
+
+    return amount
+
+
+def fake_id_number(original: str) -> str:
+    return re.sub(r"\d+", lambda match: _random_digits(len(match.group(0))), original)
+
+
+def get_fake(label: str, original: str) -> str:
+    """
+    Return a synthetic replacement for a detected PII span.
+    The same original value always maps to the same fake value (within a run).
+    """
+    key = f"{label}:{original}"
+    if key in _replacement_cache:
+        return _replacement_cache[key]
+
+    generators = {
+        "private_person":  fake.name,
+        "private_email":   fake.email,
+        "private_phone":   fake.phone_number,
+        "private_address": lambda: fake.address().replace("\n", ", "),
+        "account_number":  fake.bban,
+        "private_url":     fake.url,
+        "private_date":    lambda: fake.date(pattern="%d/%m/%Y"),
+        "secret":          lambda: "[REDACTED]",
+        "company_name":    lambda: fake_company_name(original),
+        "price":           lambda: fake_price(original),
+        "id_number":       lambda: fake_id_number(original),
+    }
+
+    fn = generators.get(label, lambda: f"[{label.upper()}]")
+    synthetic = fn()
+    _replacement_cache[key] = synthetic
+    return synthetic
+
+
+def load_model(checkpoint: str | None, device: str) -> OPF:
+    path = resolve_checkpoint(checkpoint)
+    print(f"Loading model from: {path}")
+    model = OPF(model=path, device=device)
+    print("Model ready.\n")
+    return model
+
+
+def detect_text_columns(df: pd.DataFrame) -> list[str]:
+    """Auto-detect columns containing meaningful text (not IDs or numbers)."""
+    text_cols = []
+    for col in df.columns:
+        sample = df[col].dropna().astype(str)
+        if len(sample) == 0:
+            continue
+        long_strings = sample[sample.str.len() > 5]
+        if len(long_strings) / len(sample) > 0.5:
+            text_cols.append(col)
+    return text_cols
+
+
+def apply_synthetic(text: str, result) -> tuple[str, list[dict]]:
+    """
+    Replace detected spans in text with consistent synthetic values.
+    Works character by character from the end so offsets stay valid.
+    """
+    spans = sorted(result.detected_spans, key=lambda s: s.start, reverse=True)
+    synthetic_text = text
+    replacements = []
+
+    for span in spans:
+        original_value = text[span.start:span.end]
+        fake_value = get_fake(span.label, original_value)
+        synthetic_text = synthetic_text[:span.start] + fake_value + synthetic_text[span.end:]
+        replacements.append({
+            "label":         span.label,
+            "original":      original_value,
+            "replacement":   fake_value,
+        })
+
+    return synthetic_text, replacements
+
+
+def clean_sheet(
+    df: pd.DataFrame,
+    model: OPF,
+    columns: list[str],
+    sheet_name: str,
+) -> tuple[pd.DataFrame, list[dict]]:
+    audit_log = []
+    df_clean = df.copy()
+
+    for col in columns:
+        if col not in df.columns:
+            print(f"  Warning: column '{col}' not found in sheet '{sheet_name}', skipping.")
+            continue
+
+        clean_col = f"{col}_clean"
+        df_clean[clean_col] = df[col].astype(str)
+
+        print(f"  Processing column: '{col}'")
+        for idx, cell_value in tqdm(df[col].items(), total=len(df), leave=False):
+            if pd.isna(cell_value) or str(cell_value).strip() == "":
+                continue
+
+            text = str(cell_value)
+            result = model.redact(text)
+
+            if not result.detected_spans:
+                df_clean.at[idx, clean_col] = text
+                continue
+
+            synthetic_text, replacements = apply_synthetic(text, result)
+            df_clean.at[idx, clean_col] = synthetic_text
+
+            for r in replacements:
+                audit_log.append({
+                    "sheet":            sheet_name,
+                    "row":              idx + 2,
+                    "column":           col,
+                    "pii_type":         r["label"],
+                    "original_value":   r["original"],
+                    "replaced_with":    r["replacement"],
+                    "original_text":    text[:120],
+                    "synthetic_text":   synthetic_text[:120],
+                })
+
+    return df_clean, audit_log
+
+
+def write_output(
+    input_path: Path,
+    sheet_data: dict[str, pd.DataFrame],
+    audit_log: list[dict],
+    output_path: Path,
+):
+    with pd.ExcelWriter(output_path, engine="openpyxl") as writer:
+        for sheet_name, df in sheet_data.items():
+            df.to_excel(writer, sheet_name=sheet_name, index=False)
+
+        if audit_log:
+            audit_df = pd.DataFrame(audit_log)
+            audit_df.to_excel(writer, sheet_name="PII_Audit_Log", index=False)
+
+    _format_audit_sheet(output_path)
+    print(f"\nSaved: {output_path}")
+
+
+def _format_audit_sheet(output_path: Path):
+    wb = load_workbook(output_path)
+    if "PII_Audit_Log" not in wb.sheetnames:
+        wb.save(output_path)
+        return
+
+    ws = wb["PII_Audit_Log"]
+
+    for cell in ws[1]:
+        cell.font = BOLD
+        cell.fill = HEADER_FILL
+
+    headers = [cell.value for cell in ws[1]]
+    for col_name in ("original_value", "replaced_with"):
+        if col_name in headers:
+            col_idx = headers.index(col_name) + 1
+            for row in ws.iter_rows(min_row=2, min_col=col_idx, max_col=col_idx):
+                for cell in row:
+                    cell.fill = HIGHLIGHT
+
+    for col_cells in ws.columns:
+        max_len = max((len(str(c.value or "")) for c in col_cells), default=10)
+        ws.column_dimensions[col_cells[0].column_letter].width = min(max_len + 4, 60)
+
+    wb.save(output_path)
+
+
+def main():
+    parser = argparse.ArgumentParser(description="Clean PII from Excel files using OPF + Faker.")
+    parser.add_argument("input",          help="Path to input .xlsx file")
+    parser.add_argument("--columns",      nargs="+", help="Column names to process (default: auto-detect)")
+    parser.add_argument("--sheet",        help="Process a specific sheet only")
+    parser.add_argument("--all-sheets",   action="store_true", help="Process all sheets")
+    parser.add_argument("--checkpoint",   help="Path to model checkpoint directory")
+    parser.add_argument("--device",       default="cuda", choices=["cuda", "cpu"])
+    parser.add_argument("--output",       help="Output file path (default: input_cleaned.xlsx)")
+    args = parser.parse_args()
+
+    input_path = Path(args.input)
+    if not input_path.exists():
+        print(f"Error: file not found: {input_path}")
+        sys.exit(1)
+
+    output_path = Path(args.output) if args.output else input_path.with_stem(input_path.stem + "_cleaned")
+
+    model = load_model(args.checkpoint, args.device)
+
+    all_sheets = pd.read_excel(input_path, sheet_name=None)
+
+    if args.all_sheets:
+        sheets_to_process = list(all_sheets.keys())
+    elif args.sheet:
+        if args.sheet not in all_sheets:
+            print(f"Error: sheet '{args.sheet}' not found. Available: {list(all_sheets.keys())}")
+            sys.exit(1)
+        sheets_to_process = [args.sheet]
+    else:
+        sheets_to_process = [list(all_sheets.keys())[0]]
+
+    cleaned_sheets = {}
+    full_audit_log = []
+
+    for sheet_name in sheets_to_process:
+        df = all_sheets[sheet_name]
+        print(f"\nSheet: '{sheet_name}' - {len(df)} rows, {len(df.columns)} columns")
+
+        cols = args.columns if args.columns else detect_text_columns(df)
+        if not args.columns:
+            print(f"  Auto-detected text columns: {cols}")
+
+        if not cols:
+            print("  No text columns found, skipping.")
+            cleaned_sheets[sheet_name] = df
+            continue
+
+        df_clean, audit_log = clean_sheet(df, model, cols, sheet_name)
+        cleaned_sheets[sheet_name] = df_clean
+        full_audit_log.extend(audit_log)
+
+    write_output(input_path, cleaned_sheets, full_audit_log, output_path)
+
+    print(f"\n{'-'*50}")
+    print(f"PII spans detected and replaced: {len(full_audit_log)}")
+    if full_audit_log:
+        from collections import Counter
+        counts = Counter(entry["pii_type"] for entry in full_audit_log)
+        for label, count in counts.most_common():
+            print(f"  {label:<25} {count}")
+    print(f"{'-'*50}")
+    print(f"Clean file:  {output_path}")
+    print(f"Unique PII values replaced: {len(_replacement_cache)}")
+
+
+if __name__ == "__main__":
+    main()

requirements.txt

@@ -0,0 +1,6 @@
+git+https://github.com/openai/privacy-filter.git
+huggingface_hub
+pandas
+openpyxl
+faker
+tqdm