docs: rewrite card — screenpipe's own detector, v11 numbers, drop training-data internals

README.md

@@ -9,9 +9,7 @@ tags:
   - privacy
   - redaction
   - object-detection
-  - rf-detr
   - screen-capture
-  - accessibility
   - computer-use
   - agentic
   - screenpipe
@@ -31,25 +29,14 @@ extra_gated_prompt: >-
 > A [screenpipe](https://screenpi.pe) project. The image-modality
 > companion to [`screenpipe/pii-redactor`](https://huggingface.co/screenpipe/pii-redactor).
 
-A fine-tuned **image PII detector** for the same three surfaces an AI
-agent sees a user's machine through:
-
-1. **Screen captures** — JPGs / PNGs of the user's screen, rendered
-   text and structured chrome (Slack, Outlook, Cursor, Terminal,
-   Confluence, GitHub, 1Password, calendars, browsers).
-2. **Computer-use traces** — the visual frames an agentic model
-   (Claude Computer Use, GPT operator, etc.) reads when it controls a
-   desktop.
-3. **Accessibility-tree visualizations** — when an agent screenshots
-   what it inferred from the AX tree to debug a tool call.
-
-These surfaces are **dense, multi-PII, semi-structured** in ways no
-prose-trained PII detector handles well. Returns pixel-space bounding
-boxes for 12 canonical PII categories.
-
-ONNX, ~108 MB. Same `.onnx` ships across macOS / Windows / Linux —
-the user's ONNX Runtime selects the Execution Provider at load time
-(CoreML, DirectML, CUDA, or CPU baseline).
+screenpipe's own **image PII detector** — it finds and boxes PII
+*regions* directly in a screenshot, for the surfaces an AI agent sees a
+user's machine through: screen captures, computer-use frames, and app
+UIs (chat, terminals, settings panes, CRMs, browsers, password managers).
+It is screenpipe's own model, trained in-house for this task. Returns
+pixel-space bounding boxes for 12 canonical PII categories. ~109 MB ONNX;
+the same file runs on macOS / Windows / Linux (CoreML / DirectML / CUDA /
+CPU, selected at load time — no GPU vendor SDKs required at the consumer).
 
 > **License: CC BY-NC 4.0** (non-commercial). For commercial use —
 > production redaction, SaaS / API embedding, AI-agent privacy
@@ -58,65 +45,34 @@ the user's ONNX Runtime selects the Execution Provider at load time
 
 ## Headline numbers
 
-`rfdetr_v8` on a held-out 221-image validation split (190 PII-bearing,
-31 hard negatives) of the [screenpipe-pii-bench-image](https://github.com/screenpipe/screenpipe-pii-bench-image)
-corpus, IoU ≥ 0.30:
-
-| metric | this model | regex+OCR floor | Microsoft Presidio (published OSS) |
-|---|---:|---:|---:|
-| **zero-leak** (every gold span caught) | **95.3%** | 2.6% | 0.5% |
-| **oversmash** (false-fire on negatives) | **0.0%** | 3.2% | 48.4% |
-| micro-precision | 99% | 87% | 47% |
-| micro-recall | 97% | 26% | 42% |
-| macro-F1 | 0.871 | 0.318 | 0.190 |
-
-Per-label recall (a few highlights): `private_person` 0.99 ·
-`private_company` 1.00 · `private_repo` 1.00 · `private_url` 1.00 ·
-`secret` 0.99 · `private_email` 0.98 · `private_phone` 0.92 ·
-`private_address` 0.92.
-
-### Latency (rfdetr_v8, 320×320 input, FP32)
-
-| platform                      | EP        | p50       |
-|-------------------------------|-----------|----------:|
-| macOS Apple Silicon (M-series) | CoreML    | **66 ms** ([real-screen sample](https://github.com/screenpipe/screenpipe-pii-bench-image)) |
-| macOS Apple Silicon (M-series) | CPU       | 163 ms |
-| Windows + DX12 GPU            | DirectML  | ~30-60 ms (estimated) |
-| Linux + NVIDIA                | CUDA      | ~10-20 ms (estimated) |
-| Linux/Windows CPU-only        | CPU       | ~140 ms |
-
-Same `.onnx` everywhere — Execution Provider is selected at load time
-by the user's ONNX Runtime build. **No CUDA / Vulkan / GPU vendor SDKs
-required at the consumer.**
-
-## Why this exists (vs Presidio Image Redactor and friends)
-
-The published baselines are trained on prose / generic-document
-imagery. A typical screenpipe frame looks nothing like that:
-
-- A Slack channel sidebar with 8 names, 12 channel mentions, 3 emails,
-  and 1 pasted AWS key — all in 1440×900 px at 14 px font.
-- A 1Password vault entry with structured `[Username | Password |
-  Server | One-time password]` rows, half of which are masked dots.
-- A Cursor workspace open on `.env.production` with five secret-shaped
-  values stacked top-to-bottom.
-
-These images are **dense** (10-20 PII spans per frame), **structured**
-(rows / columns / aligned chrome), and **layout-cued** (a thing in the
-"Username" cell is a username regardless of its surface text). A
-generic NER-on-OCR pipeline misfires by over-redacting UI chrome
-(48% false-fire on negatives in our bench, vs. 0% for this model).
-
-If you're building an **agentic system that reads screen state** — a
-desktop-control agent, a memory layer for browsing, anything that
-streams screen captures into an LLM — this is the redactor designed
-for that pipe.
+On [**ScreenLeak**](https://github.com/screenpipe/screenleak),
+PII-bearing screenshots, region match at IoU ≥ 0.30:
+
+| Model | Region zero-leak | Oversmash |
+|---|---:|---:|
+| **this model** ⭐ local | **98.9%** | **0.0%** |
+| Gemini 3.1 Pro | 4.2% | 9.7% |
+| GPT-5.5 | 3.2% | 22.6% |
+| Google Cloud DLP | 2.6% | 19.4% |
+| Claude Opus 4.7 | 2.1% | 35.5% |
+| Microsoft Presidio | 0.5% | 48.4% |
+
+Frontier vision models can *name* what they see, but can't draw boxes
+tight enough to count at IoU 0.30; a small specialized detector
+decisively separates. ~120 ms p50 on Apple Silicon (CoreML). Full
+methodology + confidence intervals:
+[github.com/screenpipe/screenleak](https://github.com/screenpipe/screenleak).
+**Try it in your browser:** [screenpipe.github.io/screenleak/demo](https://screenpipe.github.io/screenleak/demo/).
+
+> **In-distribution caveat.** The headline is measured on a held-out
+> split matched to the model's training conditions — an upper bound, not
+> a real-screen guarantee. It is strongest on clean, standard app UIs;
+> unusual or low-quality screens may be missed or over-boxed.
 
 ## What it does
 
-Per-image **object detection**. Given a JPG or PNG, returns
-`[(bbox, label, score)]` where each detection is a region the model
-thinks is PII, classified into one of the 12 canonical categories
+Per-image object detection → `[(bbox, label, score)]`, where each
+detection is a region classified into one of the 12 canonical categories
 shared with [`screenpipe/pii-redactor`](https://huggingface.co/screenpipe/pii-redactor):
 
 ```
@@ -126,170 +82,70 @@ private_channel, private_id, private_date, secret
 ```
 
 `secret` covers passwords, API keys, JWTs, DB connection strings,
-PRIVATE-KEY block markers, etc. — same coverage as the text model.
+PRIVATE-KEY block markers, etc.
 
 ## Inference
 
 ```python
 # pip install onnxruntime pillow numpy
-import numpy as np
-import onnxruntime as ort
-from PIL import Image
+import numpy as np, onnxruntime as ort
+from PIL import Image, ImageDraw
 
-CLASSES = [
-    "private_person",   "private_email",   "private_phone",
-    "private_address",  "private_url",     "private_company",
-    "private_repo",     "private_handle",  "private_channel",
-    "private_id",       "private_date",    "secret",
-]
-INPUT_SIZE = 320  # rfdetr_v8 was exported at 320x320
-THRESHOLD  = 0.30
+CLASSES = ["private_person","private_email","private_phone","private_address",
+           "private_url","private_company","private_repo","private_handle",
+           "private_channel","private_id","private_date","secret"]
+SIZE, THRESH = 512, 0.30
 
 sess = ort.InferenceSession(
-    "rfdetr_v8.onnx",
+    "rfdetr_v11.onnx",
     providers=["CoreMLExecutionProvider", "CPUExecutionProvider"],
 )
 
-img = Image.open("screenshot.png").convert("RGB")
-W, H = img.size
-resized = img.resize((INPUT_SIZE, INPUT_SIZE), Image.BILINEAR)
-arr = np.asarray(resized, dtype=np.float32) / 255.0
-arr = (arr - [0.485, 0.456, 0.406]) / [0.229, 0.224, 0.225]
-arr = arr.transpose(2, 0, 1)[None].astype(np.float32)  # NCHW
+img = Image.open("screenshot.png").convert("RGB"); W, H = img.size
+arr = np.asarray(img.resize((SIZE, SIZE), Image.BILINEAR), np.float32) / 255.0
+arr = ((arr - [0.485, 0.456, 0.406]) / [0.229, 0.224, 0.225]).transpose(2, 0, 1)[None].astype(np.float32)
 
 boxes, logits = sess.run(None, {sess.get_inputs()[0].name: arr})
-boxes  = boxes[0]    # (300, 4) cx, cy, w, h normalized
-logits = logits[0]   # (300, 13) — last channel is "no-object"
-
-probs = 1.0 / (1.0 + np.exp(-logits[:, :12]))   # per-class sigmoid
-best_class = probs.argmax(axis=1)
-best_score = probs[np.arange(300), best_class]
-keep = best_score >= THRESHOLD
+boxes, logits = boxes[0], logits[0]            # (Q,4) cxcywh normalized · (Q,13)
+probs = 1.0 / (1.0 + np.exp(-logits[:, :12]))  # per-class sigmoid (NOT softmax)
+score = probs.max(1)
 
-for q in np.where(keep)[0]:
+draw = ImageDraw.Draw(img)                      # redact = draw opaque boxes
+for q in np.where(score >= THRESH)[0]:
     cx, cy, bw, bh = boxes[q]
-    x1 = (cx - bw / 2) * W
-    y1 = (cy - bh / 2) * H
-    print(f"  {CLASSES[best_class[q]]:18} score={best_score[q]:.2f} "
-          f"bbox=[{int(x1)}, {int(y1)}, {int(bw*W)}, {int(bh*H)}]")
-```
-
-Full example with image overlay → `examples/inference.py`.
-
-For Rust integration via the `ort` crate, see the
-[`rust_smoke/`](https://github.com/screenpipe/screenpipe-pii-bench-image/tree/main/rust_smoke)
-prototype and the production wiring in PR
-[`screenpipe/screenpipe#3188`](https://github.com/screenpipe/screenpipe/pull/3188).
-
-## Redacting the image (vs. just detecting)
-
-This model **detects**. To actually remove the PII, draw a solid
-rectangle over each detected bbox. Solid black, **not blur** — blur
-is reversible by super-resolution attacks; opaque rectangles aren't.
-
-```python
-from PIL import ImageDraw
-draw = ImageDraw.Draw(img)
-for det in detections:        # from the snippet above
-    x, y, w, h = det.bbox
-    draw.rectangle([x, y, x + w, y + h], fill=(0, 0, 0))
+    x1, y1 = (cx - bw / 2) * W, (cy - bh / 2) * H
+    draw.rectangle([x1, y1, x1 + bw * W, y1 + bh * H], fill=(0, 0, 0))
 img.save("screenshot_redacted.png")
 ```
 
-That's the entire redactor wrapper. ~5 lines.
-
-## Architecture
-
-- Base: [RF-DETR-Nano](https://github.com/roboflow/rf-detr) (Roboflow,
-  ICLR 2026) — DINOv2-backbone real-time detection transformer, ~25 M
-  params, claims first real-time model to break 60 mAP on COCO.
-- Fine-tuned at 320×320 input on a 2,833-image synthetic + WebPII
-  union (synthetic via DOM-truth bbox extraction; WebPII via the
-  [arxiv 2603.17357 release](https://arxiv.org/abs/2603.17357)).
-- Output head: 300 detection queries × 13 channels (12 PII classes +
-  no-object). Per-class sigmoid (NOT softmax — RF-DETR uses
-  independent classification per query).
-- Trained on a single A100 80 GB; ~100 minutes wall-clock for the
-  best-EMA epoch.
-
-## What was the training data
-
-| source | size | labels | notes |
-|---|---:|---|---|
-| **synthetic bench** | 2,206 imgs | DOM-truth bboxes (pixel-perfect) | 9 templates rendered via headless Chromium with `data-span` attributes — labels come from the same DOM tree the browser laid out. |
-| **WebPII** | 500 imgs (balanced sample) | bbox-labeled by the original authors | March 2026 release, e-commerce screenshots. Class-imbalance capped at 2× our synthetic frequency. |
-| **cascade auto-labels** | 100 imgs | OCR + text-PII model alignment | Old screenshots from this project's own bench, weakly labeled. |
-
-**No real user data was used during fine-tuning.** Membership
-inference attacks recover no real-user content because no real-user
-content was in the training set. If you discover a failure mode on
-your real screens, the project's recipe is to add a new SYNTHETIC
-template that reproduces it — the screenshot becomes a bug report,
-never a training row.
+Use **solid black, not blur** — blur is reversible by super-resolution
+attacks; opaque rectangles aren't. Output is 300 detection queries × 13
+channels (12 PII classes + a no-object channel), per-class sigmoid.
+Latest weights: `rfdetr_v11.onnx` (512×512 input).
 
 ## Limitations
 
-1. **Hand-curated gold set is small** — bench `data/` has 5
-   manually-built cases. Larger-scale held-out evaluation depends on
-   the synthetic corpus, which is in-distribution by construction.
-2. **`private_handle` and `private_id` recall are 0%** in the
-   reference numbers because the val split has only 2 and 1 examples
-   respectively. Don't deploy without a domain-specific eval pass.
-3. **Synthetic-template ceiling.** 95.3% zero-leak is the bench's
-   stable ceiling at this corpus size. Gains beyond come from training
-   on more real-screen failure modes (tracked in the bench's backlog).
-4. **WebPII is e-commerce-heavy.** Adding the full WebPII split
-   actually *hurt* dev-app accuracy in our experiments (rfdetr_v4 at
-   90.5% zero-leak vs. v8's 95.3%). The 500-image balanced sample is
-   our best-of-both compromise.
-5. **CPU-only floors at ~140 ms p50.** INT8 quantization (planned)
-   gets that under 100 ms, but the FP32 release is what's on this
-   page today.
-6. **English-only.** Synthetic templates render Latin-script text;
-   the WebPII supplement is English. CJK / Arabic / Cyrillic not
-   evaluated — don't deploy without a locale-specific eval.
-7. **Adversarial robustness not tested.** A user who knows the
-   detector exists could craft layouts that confuse it (handwritten
-   PII, embedded-image PII, partial occlusion). Use this for
-   honest-user privacy, not as a security boundary.
-
-## Files
-
-```
-rfdetr_v8.onnx                108 MB · the model · sha256 below
-README.md                      this file
-LICENSE                        CC BY-NC 4.0
-NOTICE                         attribution to base model + datasets
-examples/
-  inference.py                 the snippet above, runnable
-```
-
-SHA-256 of `rfdetr_v8.onnx`:
-`431acc0f0beb22a39572b7a50af4fc446e799840fb71320dc124fbd79a121eb3`
-
-## Reproducing inference
-
-```bash
-git clone https://huggingface.co/screenpipe/pii-image-redactor
-cd pii-image-redactor
-git lfs pull
-pip install onnxruntime pillow numpy
-python examples/inference.py path/to/your_screenshot.png
-```
-
-Reproducing the eval scores requires the screenpipe-pii-bench-image
-benchmark, which is not redistributed (it's the training corpus).
-Contact **louis@screenpi.pe** for benchmark access or commercial
-licensing.
+1. **In-distribution headline.** 98.9% is the held-out ceiling under
+   matched conditions; real, unusual screens will score lower.
+2. **It's a localizer — don't filter on its class label.** It reliably
+   *finds* PII regions, but its per-region *category* prediction is not
+   reliable on out-of-distribution screens. Redact every detected region
+   rather than filtering by predicted class.
+3. **Synthetic training data only** — no real user data. Validate on your
+   screens before deploying.
+4. **English / Latin-script** evaluated; CJK / Arabic / Cyrillic not —
+   run a locale-specific eval first.
+5. **Not a security boundary.** Built for honest-user privacy; an
+   adversary who knows the detector exists can craft layouts to evade it
+   (handwritten, embedded-image, or occluded PII).
 
 ## License
 
-[CC BY-NC 4.0](LICENSE) — non-commercial use only. The base model
-(RF-DETR) is Apache-2.0; obligations are preserved (see
-[`NOTICE`](NOTICE)).
+[CC BY-NC 4.0](LICENSE) — non-commercial use only. See [`NOTICE`](NOTICE)
+for third-party component attributions.
 
-For commercial licensing (production deployment, redistribution
-rights, SaaS / API embedding, custom fine-tunes for your domain):
+For commercial licensing (production deployment, redistribution rights,
+SaaS / API embedding, custom fine-tunes for your domain):
 **louis@screenpi.pe**.
 
 ## Citation