yumi.h commited on
Commit
6f7e511
·
1 Parent(s): 1ca31db

Add de-identified download, remove FLock.io tech refs, add hackathon footer

Browse files

- Try-it tab: download the de-identified output as JSON or CSV — a single sanitised
note, or the full sanitised dataset (only de-identified text is exported).
- Remove FLock.io as a technical reference across the UI and docs; keep the
hackathon credit only.
- Footer on every tab: "Live demo for the FLock Sovereign AI Challenge at the
Encode Vibe Coding Hackathon, hosted by Encode Hub."
- Align the event name to "Encode Vibe Coding Hackathon — FLock Sovereign AI
Challenge (Encode Hub)" across README / CLAUDE / tool_card / report / changelog.
- CHANGELOG: log this live-demo polish.

Files changed (7) hide show
  1. CLAUDE.md +1 -1
  2. README.md +3 -3
  3. docs/CHANGELOG.md +12 -1
  4. docs/report.md +2 -2
  5. docs/tool_card.md +4 -4
  6. src/trust_demo.py +1 -1
  7. streamlit_app.py +46 -4
CLAUDE.md CHANGED
@@ -1,7 +1,7 @@
1
  # NoteGuard — NHS Clinical-Note PII Sanitisation
2
 
3
  Sanitise-at-source: detect + de-identify PII in free-text NHS clinical notes so only de-identified
4
- data leaves a Trust. Encode Club "Trusted Data & AI Infrastructure" hackathon; fork of `NoteGuard/`.
5
 
6
  ## Commands
7
  ```bash
 
1
  # NoteGuard — NHS Clinical-Note PII Sanitisation
2
 
3
  Sanitise-at-source: detect + de-identify PII in free-text NHS clinical notes so only de-identified
4
+ data leaves a Trust. Encode Vibe Coding Hackathon FLock Sovereign AI Challenge (Encode Hub); fork of `NoteGuard/`.
5
 
6
  ## Commands
7
  ```bash
README.md CHANGED
@@ -17,9 +17,9 @@ collaborative / federated training.
17
 
18
  > Federated learning lets institutions train without moving data. NoteGuard is the **privacy-preserving
19
  > on-ramp** that makes the data safe to train on in the first place — the missing layer in front of an
20
- > NHS Secure Data Environment / the Federated Data Platform / FLock.io.
21
 
22
- Encode Club hackathon — *Trusted Data & AI Infrastructure*. Built on **Microsoft Presidio** + **spaCy**,
23
  evaluated on [NHSEDataScience/synthetic_clinical_notes](https://huggingface.co/datasets/NHSEDataScience/synthetic_clinical_notes).
24
 
25
  ## What makes this more than "just Presidio"
@@ -66,7 +66,7 @@ The rules→engine drop is the headline: it shows, with numbers, exactly what th
66
  │ patient-consistent + date-shift, vault│ (no PHI leaves)
67
  └─────────────────────────────────────────────────────────┘
68
  same gate runs inside Trust B ──► ┌────────────────────────────┐
69
- │ shared de-identified pool │ ──► federated AI / FLock.io
70
  └────────────────────────────┘
71
  ```
72
 
 
17
 
18
  > Federated learning lets institutions train without moving data. NoteGuard is the **privacy-preserving
19
  > on-ramp** that makes the data safe to train on in the first place — the missing layer in front of an
20
+ > NHS Secure Data Environment / the Federated Data Platform.
21
 
22
+ Encode Vibe Coding Hackathon — *FLock Sovereign AI Challenge* (hosted by Encode Hub). Built on **Microsoft Presidio** + **spaCy**,
23
  evaluated on [NHSEDataScience/synthetic_clinical_notes](https://huggingface.co/datasets/NHSEDataScience/synthetic_clinical_notes).
24
 
25
  ## What makes this more than "just Presidio"
 
66
  │ patient-consistent + date-shift, vault│ (no PHI leaves)
67
  └─────────────────────────────────────────────────────────┘
68
  same gate runs inside Trust B ──► ┌────────────────────────────┐
69
+ │ shared de-identified pool │ ──► federated AI training
70
  └────────────────────────────┘
71
  ```
72
 
docs/CHANGELOG.md CHANGED
@@ -1,7 +1,18 @@
1
  # Changelog
2
 
3
  Big changes since the fork of [`NoteGuard/Automatic-PII-preprocessing-tool`](https://github.com/NoteGuard/Automatic-PII-preprocessing-tool),
4
- grouped by pull request / milestone (newest first). Encode Club "Trusted Data & AI Infrastructure" hackathon.
 
 
 
 
 
 
 
 
 
 
 
5
 
6
  ## [0.0.1] — 2026-06-20 — first release (Gold-RAP restructure & Hugging Face deploy)
7
  Branches `dev/refactor-cleancode` + `feat/hf-spaces-demo`, merged to `main`.
 
1
  # Changelog
2
 
3
  Big changes since the fork of [`NoteGuard/Automatic-PII-preprocessing-tool`](https://github.com/NoteGuard/Automatic-PII-preprocessing-tool),
4
+ grouped by pull request / milestone (newest first). Encode Vibe Coding Hackathon FLock Sovereign AI Challenge (hosted by Encode Hub).
5
+
6
+ ## Live-demo polish — 2026-06-21
7
+ Direct UI / docs changes on `main` for the public demo.
8
+ - **Download de-identified output** from the Try-it tab: a single sanitised note *or* the full
9
+ sanitised dataset, as JSON or CSV — only the de-identified text is exported (raw PHI never leaves).
10
+ - Removed FLock.io as a technical reference throughout (UI + docs); kept only the hackathon credit.
11
+ - Added a hackathon footer to every tab.
12
+ - Simplified the UI copy: dropped model / algorithm names and file / command references; renamed the
13
+ "Metrics & Leakage" tab; 1-based note index; 50 sample notes in the browser.
14
+ - Fixed the Streamlit `outputs/` path after the app moved to the repo root (the Metrics and Two-Trust
15
+ "Run" buttons read / wrote the wrong location).
16
 
17
  ## [0.0.1] — 2026-06-20 — first release (Gold-RAP restructure & Hugging Face deploy)
18
  Branches `dev/refactor-cleancode` + `feat/hf-spaces-demo`, merged to `main`.
docs/report.md CHANGED
@@ -20,7 +20,7 @@
20
  ## Tier 2
21
 
22
  ### 1. Owner and responsibility
23
- - **1.1 Organisation:** Encode Club "Trusted Data & AI Infrastructure" hackathon team (fork of `NoteGuard/`).
24
  - **1.2 Team:** Project contributors (see repository history / `docs/CHANGELOG.md`).
25
  - **1.3 Senior responsible owner:** None — prototype, not in service. An SRO would be required before deployment.
26
  - **1.4 External supplier involvement:** No commercial supplier. Built on open-source components
@@ -110,4 +110,4 @@ false positive). Recall and leakage are the sound headline metrics.
110
 
111
  ---
112
 
113
- *NoteGuard · Encode Club "Trusted Data & AI Infrastructure" hackathon · prototype · v0.0.1*
 
20
  ## Tier 2
21
 
22
  ### 1. Owner and responsibility
23
+ - **1.1 Organisation:** Encode Vibe Coding Hackathon team FLock Sovereign AI Challenge (fork of `NoteGuard/`).
24
  - **1.2 Team:** Project contributors (see repository history / `docs/CHANGELOG.md`).
25
  - **1.3 Senior responsible owner:** None — prototype, not in service. An SRO would be required before deployment.
26
  - **1.4 External supplier involvement:** No commercial supplier. Built on open-source components
 
110
 
111
  ---
112
 
113
+ *NoteGuard · Encode Vibe Coding Hackathon FLock Sovereign AI Challenge · prototype · v0.0.1*
docs/tool_card.md CHANGED
@@ -12,7 +12,7 @@
12
  |---|---|
13
  | Description | De-identification gate that detects + removes PII from free-text NHS clinical notes |
14
  | Type | Hybrid pipeline — pure-Python rules + Microsoft Presidio (spaCy `en_core_web_lg` NER). **No model is trained**; pre-trained components are composed. |
15
- | Developer | Encode Club hackathon team (fork of `NoteGuard/`) |
16
  | Status / version | Prototype · v0.0.1 |
17
  | Repository | github.com/chaeyoonyunakim/automatic-pii-preprocessing-tool |
18
 
@@ -36,7 +36,7 @@ NoteGuard is a **de-identification gate** for free-text NHS clinical notes. It d
36
  |---|---|---|
37
  | Data Wrangler / IG Analyst | Before releasing notes to research or AI teams | Cannot share raw free-text; must prove zero identifier leakage |
38
  | SDE Operator | At the Trust boundary ingestion point | Gate between Trust raw data and the shared pool |
39
- | Federated AI Platform (e.g. FLock.io) | Before each training round | Needs de-identified text; cannot inspect raw Trust data |
40
 
41
  ---
42
 
@@ -151,11 +151,11 @@ NHS Trust (raw notes)
151
 
152
  ▼ NHS SDE / FDP shared pool
153
 
154
- ▼ Federated AI (e.g. FLock.io)
155
  ```
156
 
157
  Same privacy model as OpenSAFELY: *code comes to the data, data never leaves*.
158
 
159
  ---
160
 
161
- *NoteGuard · Encode Club "Trusted Data & AI Infrastructure" hackathon · internal use only*
 
12
  |---|---|
13
  | Description | De-identification gate that detects + removes PII from free-text NHS clinical notes |
14
  | Type | Hybrid pipeline — pure-Python rules + Microsoft Presidio (spaCy `en_core_web_lg` NER). **No model is trained**; pre-trained components are composed. |
15
+ | Developer | Encode Vibe Coding Hackathon team (fork of `NoteGuard/`) |
16
  | Status / version | Prototype · v0.0.1 |
17
  | Repository | github.com/chaeyoonyunakim/automatic-pii-preprocessing-tool |
18
 
 
36
  |---|---|---|
37
  | Data Wrangler / IG Analyst | Before releasing notes to research or AI teams | Cannot share raw free-text; must prove zero identifier leakage |
38
  | SDE Operator | At the Trust boundary ingestion point | Gate between Trust raw data and the shared pool |
39
+ | Federated AI platform | Before each training round | Needs de-identified text; cannot inspect raw Trust data |
40
 
41
  ---
42
 
 
151
 
152
  ▼ NHS SDE / FDP shared pool
153
 
154
+ ▼ Federated AI
155
  ```
156
 
157
  Same privacy model as OpenSAFELY: *code comes to the data, data never leaves*.
158
 
159
  ---
160
 
161
+ *NoteGuard · Encode Vibe Coding Hackathon FLock Sovereign AI Challenge · internal use only*
src/trust_demo.py CHANGED
@@ -4,7 +4,7 @@ Each Trust holds its own patients and its own re-identification vault. It
4
  sanitises its notes LOCALLY and contributes only the
5
  de-identified text + a content-free audit manifest to a shared pool. Raw notes and
6
  vaults never leave the Trust. This is the sanitise-at-source gate that sits in
7
- front of a federated SDE / FLock.io training round.
8
 
9
  python -m src.trust_demo # pseudonymise, 300 notes
10
  python -m src.trust_demo redaction 600
 
4
  sanitises its notes LOCALLY and contributes only the
5
  de-identified text + a content-free audit manifest to a shared pool. Raw notes and
6
  vaults never leave the Trust. This is the sanitise-at-source gate that sits in
7
+ front of a federated SDE training round.
8
 
9
  python -m src.trust_demo # pseudonymise, 300 notes
10
  python -m src.trust_demo redaction 600
streamlit_app.py CHANGED
@@ -13,6 +13,7 @@ import sys
13
  from collections import Counter
14
  from pathlib import Path
15
 
 
16
  import streamlit as st
17
 
18
  REPO = Path(__file__).resolve().parent
@@ -105,13 +106,13 @@ with tab_try:
105
  if source == "Sample note" and NOTES:
106
  idx = st.number_input("Note index", 1, len(NOTES), 1, step=1)
107
  rec = NOTES[int(idx) - 1]
108
- text, person_id = rec.text, rec.person_id
109
  else:
110
  text = st.text_area("Clinical note (messy free-text)", height=200,
111
  value="Pt John Smith, NHS no 943 476 5919, DOB 02/03/1981, lives SW1A 1AA. "
112
  "Admitted Manchester Royal Infirmary Ward 9. "
113
  "Reviewed by Dr Lee, GMC 1234567.")
114
- person_id = "demo"
115
 
116
  if text.strip():
117
  result = Pipeline(detector, PseudonymVault()).sanitise(text, method, person_id)
@@ -148,6 +149,40 @@ with tab_try:
148
  else:
149
  st.success("All detections auto-confirmed (score ≥ threshold). No human review needed.", icon="✅")
150
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
151
  # ---------------------------------------------------------------- Metrics
152
  with tab_metrics:
153
  st.markdown(
@@ -248,7 +283,7 @@ NHS Trust (raw notes)
248
  ▼ NHS Secure Data Environment / Federated Data Platform pool
249
  │ (same model as OpenSAFELY: code comes to data, data never leaves)
250
 
251
- ▼ Federated AI training (e.g. FLock.io round)
252
  each Trust trains locally; only model gradients are shared
253
  ```
254
  """)
@@ -283,6 +318,13 @@ with tab_trust:
283
  st.metric("De-identified notes", summary["shared_pool_size"])
284
  st.metric("Raw records shared", summary["raw_records_shared"])
285
  st.metric("Total residual leaks", summary["total_residual_leaks"])
286
- st.caption("→ ready for federated AI / FLock.io")
287
  else:
288
  st.info("Click **Run two-Trust demo** above.")
 
 
 
 
 
 
 
 
13
  from collections import Counter
14
  from pathlib import Path
15
 
16
+ import pandas as pd
17
  import streamlit as st
18
 
19
  REPO = Path(__file__).resolve().parent
 
106
  if source == "Sample note" and NOTES:
107
  idx = st.number_input("Note index", 1, len(NOTES), 1, step=1)
108
  rec = NOTES[int(idx) - 1]
109
+ text, person_id, note_id = rec.text, rec.person_id, rec.note_id
110
  else:
111
  text = st.text_area("Clinical note (messy free-text)", height=200,
112
  value="Pt John Smith, NHS no 943 476 5919, DOB 02/03/1981, lives SW1A 1AA. "
113
  "Admitted Manchester Royal Infirmary Ward 9. "
114
  "Reviewed by Dr Lee, GMC 1234567.")
115
+ person_id, note_id = "demo", "pasted"
116
 
117
  if text.strip():
118
  result = Pipeline(detector, PseudonymVault()).sanitise(text, method, person_id)
 
149
  else:
150
  st.success("All detections auto-confirmed (score ≥ threshold). No human review needed.", icon="✅")
151
 
152
+ st.markdown("##### 4) Download this de-identified note")
153
+ one = [{"note_id": note_id, "method": method, "sanitised_text": result.sanitised}]
154
+ d1, d2 = st.columns(2)
155
+ d1.download_button("⬇ Download JSON", json.dumps(one, ensure_ascii=False, indent=2),
156
+ file_name="noteguard_note.json", mime="application/json",
157
+ use_container_width=True)
158
+ d2.download_button("⬇ Download CSV", pd.DataFrame(one).to_csv(index=False),
159
+ file_name="noteguard_note.csv", mime="text/csv",
160
+ use_container_width=True)
161
+
162
+ st.divider()
163
+ with st.expander("⬇ Download the full de-identified dataset"):
164
+ st.caption("De-identify a batch of notes and export **only the sanitised text** — "
165
+ "the original PHI never leaves the gate.")
166
+ ca, cb = st.columns([3, 1])
167
+ n_all = ca.slider("Notes to de-identify", 50, 1600, 200, step=50, key="dataset_n")
168
+ if cb.button("Prepare", use_container_width=True):
169
+ with st.spinner(f"De-identifying {n_all} notes…"):
170
+ pipe = Pipeline(detector, PseudonymVault()) # one vault → patient-consistent
171
+ rows = [{"note_id": r.note_id, "method": method,
172
+ "sanitised_text": pipe.sanitise(r.text, method, r.person_id).sanitised}
173
+ for r in load_notes(limit=n_all) if r.text]
174
+ st.session_state["dataset_rows"] = rows
175
+ rows = st.session_state.get("dataset_rows")
176
+ if rows:
177
+ st.success(f"{len(rows)} notes de-identified — ready to download.")
178
+ e1, e2 = st.columns(2)
179
+ e1.download_button("⬇ Download JSON", json.dumps(rows, ensure_ascii=False, indent=2),
180
+ file_name="noteguard_dataset.json", mime="application/json",
181
+ use_container_width=True)
182
+ e2.download_button("⬇ Download CSV", pd.DataFrame(rows).to_csv(index=False),
183
+ file_name="noteguard_dataset.csv", mime="text/csv",
184
+ use_container_width=True)
185
+
186
  # ---------------------------------------------------------------- Metrics
187
  with tab_metrics:
188
  st.markdown(
 
283
  ▼ NHS Secure Data Environment / Federated Data Platform pool
284
  │ (same model as OpenSAFELY: code comes to data, data never leaves)
285
 
286
+ ▼ Federated AI training
287
  each Trust trains locally; only model gradients are shared
288
  ```
289
  """)
 
318
  st.metric("De-identified notes", summary["shared_pool_size"])
319
  st.metric("Raw records shared", summary["raw_records_shared"])
320
  st.metric("Total residual leaks", summary["total_residual_leaks"])
321
+ st.caption("→ ready for federated AI training")
322
  else:
323
  st.info("Click **Run two-Trust demo** above.")
324
+
325
+ # ---------------------------------------------------------------- Footer (all tabs)
326
+ st.divider()
327
+ st.caption(
328
+ "Live demo for the **FLock Sovereign AI Challenge** at the Encode Vibe Coding Hackathon, "
329
+ "hosted by Encode Hub."
330
+ )