# Changelog

Big changes since the fork of [`NoteGuard/Automatic-PII-preprocessing-tool`](https://github.com/NoteGuard/Automatic-PII-preprocessing-tool),
grouped by pull request / milestone (newest first). Encode Vibe Coding Hackathon — FLock Sovereign AI Challenge (hosted by Encode Hub).

## Live-demo polish — 2026-06-21
Direct UI / docs changes on `main` for the public demo.
- **Download de-identified output** from the Try-it tab: a single sanitised note *or* the full
  sanitised dataset, as JSON or CSV — only the de-identified text is exported (raw PHI never leaves).
- Removed FLock.io as a technical reference throughout (UI + docs); kept only the hackathon credit.
- Added a hackathon footer to every tab.
- Simplified the UI copy: dropped model / algorithm names and file / command references; renamed the
  "Metrics & Leakage" tab; 1-based note index; 50 sample notes in the browser.
- Fixed the Streamlit `outputs/` path after the app moved to the repo root (the Metrics and Two-Trust
  "Run" buttons read / wrote the wrong location).

## [0.0.1] — 2026-06-20 — first release (Gold-RAP restructure & Hugging Face deploy)
Branches `dev/refactor-cleancode` + `feat/hf-spaces-demo`, merged to `main`.
- Reorganised to a Gold-RAP "analysis as a product" layout: `src/` package, `tests/`, `docs/`,
  `data/` (inputs), `outputs/` (generated artifacts).
- Renamed package `noteguard/` → `src/`; British spelling throughout (`recognisers`).
- Packaged via `pyproject.toml` as the single dependency source (removed `requirements.txt`).
- CI (`.github/workflows/ci.yml`) runs `ruff` + `pytest` on every push/PR; added logging.
- Streamlit demo moved to repo root (`streamlit_app.py`); `run_eval.py` moved under `tests/`.
- Hugging Face Spaces (Docker SDK) deploy: `Dockerfile`, README front-matter, `docs/DEPLOY_HF_SPACES.md`.
- Decluttered Claude tooling (eval skill, post-edit hook, local settings) and the root changelog.

## Post-PR #3 hardening — 2026-06-20
Direct fixes on `main` after the team-briefing work.
- Only **date-of-birth** is masked; visit / admission dates are kept (clinically useful). Human-readable
  redaction labels (e.g. `[NHS number]`).
- Fixed pseudonym mis-mapping caused by spaCy ORGANIZATION over-tagging.
- Removed the roster / gazetteer feature (kept the metric honest — no circularity).
- Consolidated the frontend on Streamlit; removed the Gradio demo.

## PR #3 — Action items 1–4 from the team briefing — 2026-06-20
- Default spaCy model `en_core_web_lg` (best UK-name recall; clinical transformer tested and rejected).
- **Human-in-the-loop review queue**: low-confidence spans are still redacted *and* flagged for IG
  analyst confirmation before SDE-pool admission.
- ORGANIZATION excluded from Presidio's kept entities (it over-tagged labels like "NHS"/"GMC"); NHS
  site names caught by a LOCATION rule instead.
- Overlap-safe, priority-ranked span merge (precise rule entities beat broad NER spans).

## PR #2 — UK entity backport — 2026-06-20
- Added `UK_NINO`, `UK_PASSPORT`, `UK_VEHICLE_REGISTRATION` recognisers and comma-separated
  NHS-number detection, with format-correct fake surrogates for each.

## PR #1 — NoteGuard de-identification gate — 2026-06-20
Consolidated two parallel implementations into one package.
- One pluggable `Detector` interface: `RuleDetector` / `PresidioDetector`, degrading gracefully to
  pure-Python rules when spaCy/Presidio are unavailable.
- Patient-consistent pseudonymisation via a Faker(en_GB) vault + per-patient date-shift; redaction mode.
- Evaluation harness: per-entity precision / recall / F1 **and** a measured residual-leakage rate,
  using the dataset's structured patient/admission tables as free ground truth.
- Two-Trust "sanitise-at-source" demo; Streamlit demo UI.

## Initial NoteGuard implementation
- Pure-Python `noteguard` package: rule recognisers (NHS Modulus-11 + context anchor, postcode, date,
  phone, email), detection layer, redaction/pseudonymisation transforms, evaluation, pipeline.
- Loads the NHSE synthetic clinical notes (3 CSVs joined on `person_id`/`admission_id`).

## Initial commit
- Fork of `NoteGuard/Automatic-PII-preprocessing-tool`.