yumi.h commited on
Commit ·
6f7e511
1
Parent(s): 1ca31db
Add de-identified download, remove FLock.io tech refs, add hackathon footer
Browse files- Try-it tab: download the de-identified output as JSON or CSV — a single sanitised
note, or the full sanitised dataset (only de-identified text is exported).
- Remove FLock.io as a technical reference across the UI and docs; keep the
hackathon credit only.
- Footer on every tab: "Live demo for the FLock Sovereign AI Challenge at the
Encode Vibe Coding Hackathon, hosted by Encode Hub."
- Align the event name to "Encode Vibe Coding Hackathon — FLock Sovereign AI
Challenge (Encode Hub)" across README / CLAUDE / tool_card / report / changelog.
- CHANGELOG: log this live-demo polish.
- CLAUDE.md +1 -1
- README.md +3 -3
- docs/CHANGELOG.md +12 -1
- docs/report.md +2 -2
- docs/tool_card.md +4 -4
- src/trust_demo.py +1 -1
- streamlit_app.py +46 -4
CLAUDE.md
CHANGED
|
@@ -1,7 +1,7 @@
|
|
| 1 |
# NoteGuard — NHS Clinical-Note PII Sanitisation
|
| 2 |
|
| 3 |
Sanitise-at-source: detect + de-identify PII in free-text NHS clinical notes so only de-identified
|
| 4 |
-
data leaves a Trust. Encode
|
| 5 |
|
| 6 |
## Commands
|
| 7 |
```bash
|
|
|
|
| 1 |
# NoteGuard — NHS Clinical-Note PII Sanitisation
|
| 2 |
|
| 3 |
Sanitise-at-source: detect + de-identify PII in free-text NHS clinical notes so only de-identified
|
| 4 |
+
data leaves a Trust. Encode Vibe Coding Hackathon — FLock Sovereign AI Challenge (Encode Hub); fork of `NoteGuard/`.
|
| 5 |
|
| 6 |
## Commands
|
| 7 |
```bash
|
README.md
CHANGED
|
@@ -17,9 +17,9 @@ collaborative / federated training.
|
|
| 17 |
|
| 18 |
> Federated learning lets institutions train without moving data. NoteGuard is the **privacy-preserving
|
| 19 |
> on-ramp** that makes the data safe to train on in the first place — the missing layer in front of an
|
| 20 |
-
> NHS Secure Data Environment / the Federated Data Platform
|
| 21 |
|
| 22 |
-
Encode
|
| 23 |
evaluated on [NHSEDataScience/synthetic_clinical_notes](https://huggingface.co/datasets/NHSEDataScience/synthetic_clinical_notes).
|
| 24 |
|
| 25 |
## What makes this more than "just Presidio"
|
|
@@ -66,7 +66,7 @@ The rules→engine drop is the headline: it shows, with numbers, exactly what th
|
|
| 66 |
│ patient-consistent + date-shift, vault│ (no PHI leaves)
|
| 67 |
└─────────────────────────────────────────────────────────┘
|
| 68 |
same gate runs inside Trust B ──► ┌────────────────────────────┐
|
| 69 |
-
│ shared de-identified pool │ ──► federated AI
|
| 70 |
└────────────────────────────┘
|
| 71 |
```
|
| 72 |
|
|
|
|
| 17 |
|
| 18 |
> Federated learning lets institutions train without moving data. NoteGuard is the **privacy-preserving
|
| 19 |
> on-ramp** that makes the data safe to train on in the first place — the missing layer in front of an
|
| 20 |
+
> NHS Secure Data Environment / the Federated Data Platform.
|
| 21 |
|
| 22 |
+
Encode Vibe Coding Hackathon — *FLock Sovereign AI Challenge* (hosted by Encode Hub). Built on **Microsoft Presidio** + **spaCy**,
|
| 23 |
evaluated on [NHSEDataScience/synthetic_clinical_notes](https://huggingface.co/datasets/NHSEDataScience/synthetic_clinical_notes).
|
| 24 |
|
| 25 |
## What makes this more than "just Presidio"
|
|
|
|
| 66 |
│ patient-consistent + date-shift, vault│ (no PHI leaves)
|
| 67 |
└─────────────────────────────────────────────────────────┘
|
| 68 |
same gate runs inside Trust B ──► ┌────────────────────────────┐
|
| 69 |
+
│ shared de-identified pool │ ──► federated AI training
|
| 70 |
└────────────────────────────┘
|
| 71 |
```
|
| 72 |
|
docs/CHANGELOG.md
CHANGED
|
@@ -1,7 +1,18 @@
|
|
| 1 |
# Changelog
|
| 2 |
|
| 3 |
Big changes since the fork of [`NoteGuard/Automatic-PII-preprocessing-tool`](https://github.com/NoteGuard/Automatic-PII-preprocessing-tool),
|
| 4 |
-
grouped by pull request / milestone (newest first). Encode
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 5 |
|
| 6 |
## [0.0.1] — 2026-06-20 — first release (Gold-RAP restructure & Hugging Face deploy)
|
| 7 |
Branches `dev/refactor-cleancode` + `feat/hf-spaces-demo`, merged to `main`.
|
|
|
|
| 1 |
# Changelog
|
| 2 |
|
| 3 |
Big changes since the fork of [`NoteGuard/Automatic-PII-preprocessing-tool`](https://github.com/NoteGuard/Automatic-PII-preprocessing-tool),
|
| 4 |
+
grouped by pull request / milestone (newest first). Encode Vibe Coding Hackathon — FLock Sovereign AI Challenge (hosted by Encode Hub).
|
| 5 |
+
|
| 6 |
+
## Live-demo polish — 2026-06-21
|
| 7 |
+
Direct UI / docs changes on `main` for the public demo.
|
| 8 |
+
- **Download de-identified output** from the Try-it tab: a single sanitised note *or* the full
|
| 9 |
+
sanitised dataset, as JSON or CSV — only the de-identified text is exported (raw PHI never leaves).
|
| 10 |
+
- Removed FLock.io as a technical reference throughout (UI + docs); kept only the hackathon credit.
|
| 11 |
+
- Added a hackathon footer to every tab.
|
| 12 |
+
- Simplified the UI copy: dropped model / algorithm names and file / command references; renamed the
|
| 13 |
+
"Metrics & Leakage" tab; 1-based note index; 50 sample notes in the browser.
|
| 14 |
+
- Fixed the Streamlit `outputs/` path after the app moved to the repo root (the Metrics and Two-Trust
|
| 15 |
+
"Run" buttons read / wrote the wrong location).
|
| 16 |
|
| 17 |
## [0.0.1] — 2026-06-20 — first release (Gold-RAP restructure & Hugging Face deploy)
|
| 18 |
Branches `dev/refactor-cleancode` + `feat/hf-spaces-demo`, merged to `main`.
|
docs/report.md
CHANGED
|
@@ -20,7 +20,7 @@
|
|
| 20 |
## Tier 2
|
| 21 |
|
| 22 |
### 1. Owner and responsibility
|
| 23 |
-
- **1.1 Organisation:** Encode
|
| 24 |
- **1.2 Team:** Project contributors (see repository history / `docs/CHANGELOG.md`).
|
| 25 |
- **1.3 Senior responsible owner:** None — prototype, not in service. An SRO would be required before deployment.
|
| 26 |
- **1.4 External supplier involvement:** No commercial supplier. Built on open-source components
|
|
@@ -110,4 +110,4 @@ false positive). Recall and leakage are the sound headline metrics.
|
|
| 110 |
|
| 111 |
---
|
| 112 |
|
| 113 |
-
*NoteGuard · Encode
|
|
|
|
| 20 |
## Tier 2
|
| 21 |
|
| 22 |
### 1. Owner and responsibility
|
| 23 |
+
- **1.1 Organisation:** Encode Vibe Coding Hackathon team — FLock Sovereign AI Challenge (fork of `NoteGuard/`).
|
| 24 |
- **1.2 Team:** Project contributors (see repository history / `docs/CHANGELOG.md`).
|
| 25 |
- **1.3 Senior responsible owner:** None — prototype, not in service. An SRO would be required before deployment.
|
| 26 |
- **1.4 External supplier involvement:** No commercial supplier. Built on open-source components
|
|
|
|
| 110 |
|
| 111 |
---
|
| 112 |
|
| 113 |
+
*NoteGuard · Encode Vibe Coding Hackathon — FLock Sovereign AI Challenge · prototype · v0.0.1*
|
docs/tool_card.md
CHANGED
|
@@ -12,7 +12,7 @@
|
|
| 12 |
|---|---|
|
| 13 |
| Description | De-identification gate that detects + removes PII from free-text NHS clinical notes |
|
| 14 |
| Type | Hybrid pipeline — pure-Python rules + Microsoft Presidio (spaCy `en_core_web_lg` NER). **No model is trained**; pre-trained components are composed. |
|
| 15 |
-
| Developer | Encode
|
| 16 |
| Status / version | Prototype · v0.0.1 |
|
| 17 |
| Repository | github.com/chaeyoonyunakim/automatic-pii-preprocessing-tool |
|
| 18 |
|
|
@@ -36,7 +36,7 @@ NoteGuard is a **de-identification gate** for free-text NHS clinical notes. It d
|
|
| 36 |
|---|---|---|
|
| 37 |
| Data Wrangler / IG Analyst | Before releasing notes to research or AI teams | Cannot share raw free-text; must prove zero identifier leakage |
|
| 38 |
| SDE Operator | At the Trust boundary ingestion point | Gate between Trust raw data and the shared pool |
|
| 39 |
-
| Federated AI
|
| 40 |
|
| 41 |
---
|
| 42 |
|
|
@@ -151,11 +151,11 @@ NHS Trust (raw notes)
|
|
| 151 |
│
|
| 152 |
▼ NHS SDE / FDP shared pool
|
| 153 |
│
|
| 154 |
-
▼ Federated AI
|
| 155 |
```
|
| 156 |
|
| 157 |
Same privacy model as OpenSAFELY: *code comes to the data, data never leaves*.
|
| 158 |
|
| 159 |
---
|
| 160 |
|
| 161 |
-
*NoteGuard · Encode
|
|
|
|
| 12 |
|---|---|
|
| 13 |
| Description | De-identification gate that detects + removes PII from free-text NHS clinical notes |
|
| 14 |
| Type | Hybrid pipeline — pure-Python rules + Microsoft Presidio (spaCy `en_core_web_lg` NER). **No model is trained**; pre-trained components are composed. |
|
| 15 |
+
| Developer | Encode Vibe Coding Hackathon team (fork of `NoteGuard/`) |
|
| 16 |
| Status / version | Prototype · v0.0.1 |
|
| 17 |
| Repository | github.com/chaeyoonyunakim/automatic-pii-preprocessing-tool |
|
| 18 |
|
|
|
|
| 36 |
|---|---|---|
|
| 37 |
| Data Wrangler / IG Analyst | Before releasing notes to research or AI teams | Cannot share raw free-text; must prove zero identifier leakage |
|
| 38 |
| SDE Operator | At the Trust boundary ingestion point | Gate between Trust raw data and the shared pool |
|
| 39 |
+
| Federated AI platform | Before each training round | Needs de-identified text; cannot inspect raw Trust data |
|
| 40 |
|
| 41 |
---
|
| 42 |
|
|
|
|
| 151 |
│
|
| 152 |
▼ NHS SDE / FDP shared pool
|
| 153 |
│
|
| 154 |
+
▼ Federated AI
|
| 155 |
```
|
| 156 |
|
| 157 |
Same privacy model as OpenSAFELY: *code comes to the data, data never leaves*.
|
| 158 |
|
| 159 |
---
|
| 160 |
|
| 161 |
+
*NoteGuard · Encode Vibe Coding Hackathon — FLock Sovereign AI Challenge · internal use only*
|
src/trust_demo.py
CHANGED
|
@@ -4,7 +4,7 @@ Each Trust holds its own patients and its own re-identification vault. It
|
|
| 4 |
sanitises its notes LOCALLY and contributes only the
|
| 5 |
de-identified text + a content-free audit manifest to a shared pool. Raw notes and
|
| 6 |
vaults never leave the Trust. This is the sanitise-at-source gate that sits in
|
| 7 |
-
front of a federated SDE
|
| 8 |
|
| 9 |
python -m src.trust_demo # pseudonymise, 300 notes
|
| 10 |
python -m src.trust_demo redaction 600
|
|
|
|
| 4 |
sanitises its notes LOCALLY and contributes only the
|
| 5 |
de-identified text + a content-free audit manifest to a shared pool. Raw notes and
|
| 6 |
vaults never leave the Trust. This is the sanitise-at-source gate that sits in
|
| 7 |
+
front of a federated SDE training round.
|
| 8 |
|
| 9 |
python -m src.trust_demo # pseudonymise, 300 notes
|
| 10 |
python -m src.trust_demo redaction 600
|
streamlit_app.py
CHANGED
|
@@ -13,6 +13,7 @@ import sys
|
|
| 13 |
from collections import Counter
|
| 14 |
from pathlib import Path
|
| 15 |
|
|
|
|
| 16 |
import streamlit as st
|
| 17 |
|
| 18 |
REPO = Path(__file__).resolve().parent
|
|
@@ -105,13 +106,13 @@ with tab_try:
|
|
| 105 |
if source == "Sample note" and NOTES:
|
| 106 |
idx = st.number_input("Note index", 1, len(NOTES), 1, step=1)
|
| 107 |
rec = NOTES[int(idx) - 1]
|
| 108 |
-
text, person_id = rec.text, rec.person_id
|
| 109 |
else:
|
| 110 |
text = st.text_area("Clinical note (messy free-text)", height=200,
|
| 111 |
value="Pt John Smith, NHS no 943 476 5919, DOB 02/03/1981, lives SW1A 1AA. "
|
| 112 |
"Admitted Manchester Royal Infirmary Ward 9. "
|
| 113 |
"Reviewed by Dr Lee, GMC 1234567.")
|
| 114 |
-
person_id = "demo"
|
| 115 |
|
| 116 |
if text.strip():
|
| 117 |
result = Pipeline(detector, PseudonymVault()).sanitise(text, method, person_id)
|
|
@@ -148,6 +149,40 @@ with tab_try:
|
|
| 148 |
else:
|
| 149 |
st.success("All detections auto-confirmed (score ≥ threshold). No human review needed.", icon="✅")
|
| 150 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 151 |
# ---------------------------------------------------------------- Metrics
|
| 152 |
with tab_metrics:
|
| 153 |
st.markdown(
|
|
@@ -248,7 +283,7 @@ NHS Trust (raw notes)
|
|
| 248 |
▼ NHS Secure Data Environment / Federated Data Platform pool
|
| 249 |
│ (same model as OpenSAFELY: code comes to data, data never leaves)
|
| 250 |
│
|
| 251 |
-
▼ Federated AI training
|
| 252 |
each Trust trains locally; only model gradients are shared
|
| 253 |
```
|
| 254 |
""")
|
|
@@ -283,6 +318,13 @@ with tab_trust:
|
|
| 283 |
st.metric("De-identified notes", summary["shared_pool_size"])
|
| 284 |
st.metric("Raw records shared", summary["raw_records_shared"])
|
| 285 |
st.metric("Total residual leaks", summary["total_residual_leaks"])
|
| 286 |
-
st.caption("→ ready for federated AI
|
| 287 |
else:
|
| 288 |
st.info("Click **Run two-Trust demo** above.")
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 13 |
from collections import Counter
|
| 14 |
from pathlib import Path
|
| 15 |
|
| 16 |
+
import pandas as pd
|
| 17 |
import streamlit as st
|
| 18 |
|
| 19 |
REPO = Path(__file__).resolve().parent
|
|
|
|
| 106 |
if source == "Sample note" and NOTES:
|
| 107 |
idx = st.number_input("Note index", 1, len(NOTES), 1, step=1)
|
| 108 |
rec = NOTES[int(idx) - 1]
|
| 109 |
+
text, person_id, note_id = rec.text, rec.person_id, rec.note_id
|
| 110 |
else:
|
| 111 |
text = st.text_area("Clinical note (messy free-text)", height=200,
|
| 112 |
value="Pt John Smith, NHS no 943 476 5919, DOB 02/03/1981, lives SW1A 1AA. "
|
| 113 |
"Admitted Manchester Royal Infirmary Ward 9. "
|
| 114 |
"Reviewed by Dr Lee, GMC 1234567.")
|
| 115 |
+
person_id, note_id = "demo", "pasted"
|
| 116 |
|
| 117 |
if text.strip():
|
| 118 |
result = Pipeline(detector, PseudonymVault()).sanitise(text, method, person_id)
|
|
|
|
| 149 |
else:
|
| 150 |
st.success("All detections auto-confirmed (score ≥ threshold). No human review needed.", icon="✅")
|
| 151 |
|
| 152 |
+
st.markdown("##### 4) Download this de-identified note")
|
| 153 |
+
one = [{"note_id": note_id, "method": method, "sanitised_text": result.sanitised}]
|
| 154 |
+
d1, d2 = st.columns(2)
|
| 155 |
+
d1.download_button("⬇ Download JSON", json.dumps(one, ensure_ascii=False, indent=2),
|
| 156 |
+
file_name="noteguard_note.json", mime="application/json",
|
| 157 |
+
use_container_width=True)
|
| 158 |
+
d2.download_button("⬇ Download CSV", pd.DataFrame(one).to_csv(index=False),
|
| 159 |
+
file_name="noteguard_note.csv", mime="text/csv",
|
| 160 |
+
use_container_width=True)
|
| 161 |
+
|
| 162 |
+
st.divider()
|
| 163 |
+
with st.expander("⬇ Download the full de-identified dataset"):
|
| 164 |
+
st.caption("De-identify a batch of notes and export **only the sanitised text** — "
|
| 165 |
+
"the original PHI never leaves the gate.")
|
| 166 |
+
ca, cb = st.columns([3, 1])
|
| 167 |
+
n_all = ca.slider("Notes to de-identify", 50, 1600, 200, step=50, key="dataset_n")
|
| 168 |
+
if cb.button("Prepare", use_container_width=True):
|
| 169 |
+
with st.spinner(f"De-identifying {n_all} notes…"):
|
| 170 |
+
pipe = Pipeline(detector, PseudonymVault()) # one vault → patient-consistent
|
| 171 |
+
rows = [{"note_id": r.note_id, "method": method,
|
| 172 |
+
"sanitised_text": pipe.sanitise(r.text, method, r.person_id).sanitised}
|
| 173 |
+
for r in load_notes(limit=n_all) if r.text]
|
| 174 |
+
st.session_state["dataset_rows"] = rows
|
| 175 |
+
rows = st.session_state.get("dataset_rows")
|
| 176 |
+
if rows:
|
| 177 |
+
st.success(f"{len(rows)} notes de-identified — ready to download.")
|
| 178 |
+
e1, e2 = st.columns(2)
|
| 179 |
+
e1.download_button("⬇ Download JSON", json.dumps(rows, ensure_ascii=False, indent=2),
|
| 180 |
+
file_name="noteguard_dataset.json", mime="application/json",
|
| 181 |
+
use_container_width=True)
|
| 182 |
+
e2.download_button("⬇ Download CSV", pd.DataFrame(rows).to_csv(index=False),
|
| 183 |
+
file_name="noteguard_dataset.csv", mime="text/csv",
|
| 184 |
+
use_container_width=True)
|
| 185 |
+
|
| 186 |
# ---------------------------------------------------------------- Metrics
|
| 187 |
with tab_metrics:
|
| 188 |
st.markdown(
|
|
|
|
| 283 |
▼ NHS Secure Data Environment / Federated Data Platform pool
|
| 284 |
│ (same model as OpenSAFELY: code comes to data, data never leaves)
|
| 285 |
│
|
| 286 |
+
▼ Federated AI training
|
| 287 |
each Trust trains locally; only model gradients are shared
|
| 288 |
```
|
| 289 |
""")
|
|
|
|
| 318 |
st.metric("De-identified notes", summary["shared_pool_size"])
|
| 319 |
st.metric("Raw records shared", summary["raw_records_shared"])
|
| 320 |
st.metric("Total residual leaks", summary["total_residual_leaks"])
|
| 321 |
+
st.caption("→ ready for federated AI training")
|
| 322 |
else:
|
| 323 |
st.info("Click **Run two-Trust demo** above.")
|
| 324 |
+
|
| 325 |
+
# ---------------------------------------------------------------- Footer (all tabs)
|
| 326 |
+
st.divider()
|
| 327 |
+
st.caption(
|
| 328 |
+
"Live demo for the **FLock Sovereign AI Challenge** at the Encode Vibe Coding Hackathon, "
|
| 329 |
+
"hosted by Encode Hub."
|
| 330 |
+
)
|