README.md CHANGED
@@ -1,14 +1,54 @@
1
  ---
2
- title: Federated Learning Model
3
- emoji: ⚡
4
- colorFrom: pink
5
- colorTo: yellow
6
  sdk: gradio
7
- sdk_version: 6.32.0
8
- python_version: '3.12'
9
  app_file: app.py
10
  pinned: false
11
  license: mit
12
  ---
13
 
14
- Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
+ title: Federated DP Fair Disease Risk
3
+ emoji: 🩺
4
+ colorFrom: blue
5
+ colorTo: green
6
  sdk: gradio
7
+ sdk_version: 5.9.1
8
+ python_version: "3.10"
9
  app_file: app.py
10
  pinned: false
11
  license: mit
12
  ---
13
 
14
+ # Privacy-Preserving Federated Learning for Early Multi-Disease Risk Prediction with Fairness Constraints
15
+
16
+ **Diseases:** diabetes (diabetic retinopathy, retinal fundus images) and heart disease (cardiomegaly, chest X-rays).
17
+ **Domain:** medical image processing. **Cost:** zero (free HF Space CPU + free public datasets).
18
+
19
+ ## What it does
20
+ 1. **Image processing (OpenCV):** CLAHE for X-rays; border crop + Ben-Graham local-contrast normalisation for fundus photos; resize to 224.
21
+ 2. **Frozen MobileNetV2** (ImageNet weights) turns each image into a 1280-d embedding, computed once and cached. No deep-network training is needed.
22
+ 3. **Simulated hospitals** (non-IID Dirichlet split) each train a logistic head on their own embeddings with **DP-SGD** (per-example clipping + Gaussian noise, Poisson sampling).
23
+ 4. **FedAvg** aggregates client updates. Raw images and embeddings never leave a hospital.
24
+ 5. **Fairness controller:** the server watches aggregate recall (TPR) of two patient groups and re-weights examples (public weights, applied after clipping) to shrink the gap (equal opportunity).
25
+ 6. **Privacy accounting:** RDP accountant reports record-level (epsilon, delta=1e-5) per hospital.
26
+ 7. The app compares **Baseline FedAvg vs +DP vs +DP+Fairness** and prints a copy-paste results table.
27
+
28
+ Fairness groups: patient **sex** for chest X-rays if the dataset exposes it; otherwise (and always for fundus) an **image-brightness proxy** for acquisition conditions. The app states which one was used.
29
+
30
+ ## Step 0 (required for every option): build the feature cache once
31
+ Run `python prepare_features.py --task heart --n 150` and `--task diabetes` in free Google Colab (or your laptop).
32
+ This creates `cache/heart.npz` and `cache/diabetes.npz`. With these files the app needs no torch and no dataset download.
33
+
34
+ ## Where to run it (free options, best first for a next-day deadline)
35
+ 1. **Google Colab public link (guaranteed, no account tier needed):** upload this folder, `pip install gradio` and run
36
+ `GRADIO_SHARE=1 python app.py`. Share the printed temporary public URL. Keep the notebook running during the demo.
37
+ 2. **Your laptop (offline fallback):** `pip install gradio numpy pandas scikit-learn matplotlib pillow`, put the cache files in `cache/`, run `python app.py`, open http://127.0.0.1:7860.
38
+ 3. **Hugging Face ZeroGPU Space (permanent link):** per current HF docs, Gradio Spaces need a paid plan EXCEPT up to 2 ZeroGPU Spaces for free personal accounts in good standing (verified email, account older than 30 days).
39
+ New Space -> SDK Gradio -> hardware **ZeroGPU** (if it is not offered, your account is not eligible) -> upload all files including `cache/*.npz`.
40
+ The app never uses the GPU; a no-op `@spaces.GPU` function in `app.py` only satisfies ZeroGPU's startup check.
41
+
42
+ ### Local test (no torch needed)
43
+ `pip install numpy pandas scikit-learn matplotlib` then `python tests/smoke_test.py` (runs the whole federated core on synthetic features).
44
+
45
+ ## Honest limitations (say these before your guide does)
46
+ - Backbone is frozen; only the classifier head is federated. Fine for a prototype, and it is what makes training instant.
47
+ - Clients are simulated on one machine by partitioning one dataset. No real network or secure aggregation.
48
+ - Epsilon covers the DP-SGD updates only; the noisy fairness counts are not included in the budget.
49
+ - Small samples (hundreds of images) => wide confidence intervals. Report multiple seeds.
50
+ - Fundus fairness groups are a proxy, not demographics. Sex is only used where the dataset provides it.
51
+ - Research prototype, not a medical device.
52
+
53
+ ## Files
54
+ `app.py` UI | `fl/fed.py` FedAvg, DP-SGD, fairness | `fl/dp.py` clipping and RDP accountant | `fl/data.py` data and splits | `fl/vision.py` OpenCV and MobileNetV2 | `fl/experiment.py` tables and plots | `prepare_features.py` cache builder | `tests/smoke_test.py` | `docs/`
app.py ADDED
@@ -0,0 +1,125 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Gradio app: federated, differentially-private, fairness-aware disease-risk prediction
2
+ from medical images. Runs on a free CPU Hugging Face Space."""
3
+ import numpy as np
4
+ import gradio as gr
5
+ from PIL import Image
6
+
7
+ from fl import data as D
8
+ from fl import experiment as E
9
+ from fl.fed import Config, proba
10
+
11
+ import os
12
+
13
+ # ZeroGPU Spaces refuse to start unless at least one @spaces.GPU function exists.
14
+ # This tiny no-op satisfies that; the app itself never uses the GPU (so no quota is spent).
15
+ try:
16
+ import spaces
17
+
18
+ @spaces.GPU
19
+ def _gpu_ping():
20
+ return "ok"
21
+ except ImportError: # local machine / Colab / any non-ZeroGPU host
22
+ pass
23
+
24
+ TASK_CHOICES = [(v["title"], k) for k, v in D.TASKS.items()]
25
+ DISCLAIMER = ("Research prototype for a dissertation. Not a medical device; not for diagnosis. "
26
+ "Models use a frozen ImageNet backbone and small public-dataset samples.")
27
+
28
+
29
+ def run(task, n_clients, rounds, local_epochs, noise_mult, clip, alpha, per_class, seed):
30
+ logs = []
31
+
32
+ def log(m):
33
+ logs.append(str(m))
34
+
35
+ def status(extra=""):
36
+ return "```\n" + "\n".join(logs[-14:]) + "\n```\n" + extra
37
+
38
+ try:
39
+ log("Preparing data (first run per disease downloads a small sample and takes a few minutes) ...")
40
+ yield status(), gr.update(), gr.update(), gr.update(), gr.update()
41
+ raw = D.load_raw(task, int(per_class), int(seed), log=log)
42
+ yield status(), gr.update(), gr.update(), gr.update(), gr.update()
43
+
44
+ cfg = Config(n_clients=int(n_clients), rounds=int(rounds), local_epochs=int(local_epochs),
45
+ noise_mult=float(noise_mult), clip=float(clip), alpha=float(alpha), seed=int(seed))
46
+ prepared = D.prepare_split(raw, cfg.n_clients, cfg.alpha, cfg.proj_dim, cfg.seed)
47
+ log(f"Hospitals: sizes={prepared['meta']['client_sizes']} "
48
+ f"positive rates={[round(x, 2) for x in prepared['meta']['client_pos_rate']]}")
49
+ log("Running 3 federated experiments (baseline, +DP, +DP+fairness) ...")
50
+ yield status(), gr.update(), gr.update(), gr.update(), gr.update()
51
+
52
+ results = E.run_modes(prepared, cfg)
53
+ table = E.results_table(results)
54
+ E.STATE[task] = {"w": results[2]["w"], "prep": prepared["prep"], "synthetic": raw["synthetic"]}
55
+ report = E.report_markdown(D.TASKS[task]["title"], prepared, results, table)
56
+ warn = "\n\n**Synthetic fallback data in use - results are NOT real.**" if raw["synthetic"] else ""
57
+ log("Done.")
58
+ yield (status("**Finished.**" + warn), table, E.fig_curves(results),
59
+ E.fig_groups(results, prepared["meta"]["group_name"]), report)
60
+ except Exception as e: # noqa: BLE001
61
+ log(f"ERROR: {type(e).__name__}: {e}")
62
+ yield status(), gr.update(), gr.update(), gr.update(), gr.update()
63
+
64
+
65
+ def predict(task, image):
66
+ st = E.STATE.get(task)
67
+ if st is None:
68
+ return None, "Run the experiment for this disease first (tab 1)."
69
+ if st["synthetic"]:
70
+ return None, "The model was trained on SYNTHETIC data, so it cannot score real images. Fix the data source first."
71
+ if image is None:
72
+ return None, "Upload an image."
73
+ try:
74
+ from fl import vision
75
+ proc = vision.preprocess(task, Image.fromarray(image))
76
+ feats = vision.extract_features([proc])
77
+ z = D.transform_features(st["prep"], feats)
78
+ p = float(proba(st["w"], z)[0])
79
+ except ImportError:
80
+ return None, ("Image scoring needs torch + torchvision, which this deployment does not install "
81
+ "(the experiments in tab 1 do not need them).")
82
+ except Exception as e: # noqa: BLE001
83
+ return None, f"Could not score image: {type(e).__name__}: {e}"
84
+ label = "elevated" if p >= 0.5 else "low"
85
+ return proc, f"**Risk score: {p:.2f}** ({label} risk by this demo model)\n\n_{DISCLAIMER}_"
86
+
87
+
88
+ with gr.Blocks(title="Federated DP Fair Disease Risk", theme=gr.themes.Soft()) as demo:
89
+ gr.Markdown("# Privacy-preserving federated learning for early disease-risk prediction (with fairness)\n"
90
+ "Simulated hospitals train locally with **DP-SGD**, the server aggregates with **FedAvg**, "
91
+ "and a **fairness controller** equalises recall across patient groups. " + DISCLAIMER)
92
+ with gr.Tab("1. Run federated experiment"):
93
+ with gr.Row():
94
+ with gr.Column(scale=1):
95
+ task = gr.Dropdown(TASK_CHOICES, value="heart", label="Disease / image modality")
96
+ n_clients = gr.Slider(2, 6, value=3, step=1, label="Hospitals (clients)")
97
+ rounds = gr.Slider(5, 40, value=15, step=1, label="Federated rounds")
98
+ epochs = gr.Slider(1, 5, value=2, step=1, label="Local epochs per round")
99
+ noise = gr.Slider(0.5, 3.0, value=1.5, step=0.1, label="DP noise multiplier (higher = more private)")
100
+ clip = gr.Slider(0.5, 3.0, value=1.0, step=0.1, label="Gradient clip norm")
101
+ alpha = gr.Slider(0.1, 10.0, value=1.0, step=0.1, label="Non-IID strength (low = very different hospitals)")
102
+ per_class = gr.Slider(50, 300, value=150, step=10, label="Images per class (first run only)")
103
+ seed = gr.Number(value=0, precision=0, label="Seed")
104
+ go = gr.Button("Run experiment", variant="primary")
105
+ with gr.Column(scale=2):
106
+ status_md = gr.Markdown()
107
+ table = gr.Dataframe(label="Results (final round, held-out global test set)")
108
+ curves = gr.Plot(label="Training curves")
109
+ groups = gr.Plot(label="Per-group recall")
110
+ report = gr.Markdown(label="Copy-paste report")
111
+ go.click(run, [task, n_clients, rounds, epochs, noise, clip, alpha, per_class, seed],
112
+ [status_md, table, curves, groups, report])
113
+ with gr.Tab("2. Try an image"):
114
+ gr.Markdown("Score one image with the trained **DP + Fairness** global model "
115
+ "(chest X-ray for heart, retinal fundus photo for diabetes).")
116
+ t2 = gr.Dropdown(TASK_CHOICES, value="heart", label="Disease / image modality")
117
+ img = gr.Image(type="numpy", label="Input image")
118
+ btn = gr.Button("Predict")
119
+ out_img = gr.Image(label="After OpenCV preprocessing (CLAHE / local contrast)")
120
+ out_txt = gr.Markdown()
121
+ btn.click(predict, [t2, img], [out_img, out_txt])
122
+
123
+ if __name__ == "__main__":
124
+ # GRADIO_SHARE=1 creates a temporary public link (useful in Google Colab)
125
+ demo.queue().launch(share=os.environ.get("GRADIO_SHARE") == "1")
cashe/.gitkeep ADDED
File without changes
docs/RESEARCH_PROMPT.md ADDED
@@ -0,0 +1,37 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Research prompt (paste into any deep-research AI: Claude, ChatGPT, Gemini, Perplexity)
2
+
3
+ ```
4
+ ROLE: You are a senior ML researcher writing a literature and implementation brief for a master's
5
+ dissertation. Be precise, cite real papers (author, year, venue, link), and mark anything you cannot verify as "UNVERIFIED".
6
+
7
+ TOPIC: "Privacy-preserving federated learning for early multi-disease risk prediction with fairness
8
+ constraints, in the medical image processing domain."
9
+ SCOPE (fixed): two diseases only: diabetes (diabetic retinopathy from retinal fundus images) and
10
+ heart disease (cardiomegaly from chest X-rays). Binary risk classification.
11
+ CONSTRAINTS: free tools only; CPU-only laptop (i5-6600U, 8 GB RAM); must be deployed on a free
12
+ Hugging Face Space; deadline is days, not weeks; no model training from scratch.
13
+ MY DESIGN: frozen ImageNet MobileNetV2 embeddings, federated training of a small head with FedAvg,
14
+ DP-SGD (per-example clipping + Gaussian noise, RDP accountant), and a server-side fairness controller
15
+ that re-weights examples to equalise group TPR. Clients are simulated by non-IID Dirichlet splits.
16
+
17
+ DELIVER, in this order, each as a short section:
18
+ 1. Problem framing: 5 bullet points a supervisor accepts as "the problem" (privacy, data silos, bias, early detection, image modality).
19
+ 2. Related work table (>= 15 papers): Federated learning in medical imaging; FedAvg (McMahan 2017);
20
+ DP-SGD (Abadi 2016); RDP accountant (Mironov 2017, 2019); DP in federated medical imaging;
21
+ fairness in FL (FairFed, q-FFL, AgnosticFL, FedFair); fairness definitions (Hardt 2016 equalized
22
+ odds / equal opportunity); bias in chest X-ray models (Seyyed-Kalantari 2021) and fundus models.
23
+ Columns: paper | year | method | dataset | key result | how it relates to my design.
24
+ 3. Datasets: for each disease list free, legally usable datasets (name, size, license, URL, label
25
+ schema, whether sex/age metadata exists, Hugging Face dataset id if any). Flag license/attribution duties.
26
+ 4. Methods: exact formulas for (a) DP-SGD step, (b) subsampled-Gaussian RDP and conversion to (eps, delta),
27
+ (c) FedAvg, (d) equal-opportunity and equalized-odds gaps, demographic parity gap, (e) the
28
+ interaction between DP and fairness (DP hurts minority groups: Bagdasaryan 2019). One paragraph on why.
29
+ 5. Evaluation protocol: metrics, number of seeds, baselines (centralized, local-only, FedAvg, +DP,
30
+ +DP+fairness), ablations (epsilon sweep, non-IID sweep, clients sweep), statistical reporting.
31
+ 6. Realistic expected numbers (AUC range) for frozen-backbone linear heads on these tasks, with sources.
32
+ 7. Threats to validity and ethical points (de-identification, consent, bias, not-a-medical-device).
33
+ 8. Viva prep: 12 hard questions a supervisor will ask, each with a 3-sentence answer.
34
+ 9. A reference list in IEEE format with working links.
35
+
36
+ STYLE: terse, tables where useful, no filler, no invented citations.
37
+ ```
docs/VIBE_CODING_GUIDE.md ADDED
@@ -0,0 +1,58 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Vibe-coding guide: take this repo further with ANY coding AI (Claude, Cursor, ChatGPT, Gemini, Copilot)
2
+
3
+ The codebase in this folder already runs. Use the AI to adapt, extend or fix it, not to start over.
4
+
5
+ ## Rules for you
6
+ 1. Give the AI the **whole repo** (zip, or paste files in this order: README.md, fl/fed.py, fl/dp.py, fl/data.py, fl/vision.py, fl/experiment.py, app.py) plus the MASTER PROMPT below.
7
+ 2. One change per message. After each, run `python tests/smoke_test.py`. If it fails, paste the full traceback back.
8
+ 3. Never let the AI "simplify" the DP or fairness code. Those are your dissertation.
9
+ 4. Deploy loop: edit file -> upload to the Space -> check the Space **Logs** tab -> paste errors back to the AI.
10
+
11
+ ## MASTER PROMPT (copy everything in the box)
12
+ ```
13
+ You are a senior ML engineer. I am handing you a working repo (attached) for a dissertation:
14
+ "Privacy-preserving federated learning for early multi-disease risk prediction with fairness
15
+ constraints using image processing". It deploys as a Gradio app on a FREE Hugging Face CPU Space.
16
+
17
+ HARD CONSTRAINTS
18
+ - Free tools only. CPU only. Python 3.10. No network access at runtime except Hugging Face datasets and torchvision weights.
19
+ - The federated core in fl/fed.py, fl/dp.py is pure NumPy and must stay torch-free so tests run anywhere.
20
+ - Do NOT replace DP-SGD with "add noise to final weights". Keep: Poisson sampling, per-example clipping,
21
+ Gaussian noise sigma*C, RDP accounting in fl/dp.py. Public fairness weights are applied AFTER clipping
22
+ and are <= 1 so sensitivity stays C. Do not break this invariant.
23
+ - Backbone = frozen ImageNet MobileNetV2 (1280-d). No deep training. Features are cached in cache/*.npz.
24
+ - Keep every existing file path and function name unless I ask. Keep tests/smoke_test.py passing.
25
+ - Output complete files (not diffs) for every file you change, each in its own code block with the path as a header.
26
+ - If you are unsure about a Hugging Face dataset column name, make the loader auto-detect it and log the columns; never hard-fail silently.
27
+
28
+ ARCHITECTURE (do not change)
29
+ images -> fl/vision.py (OpenCV preprocessing, CLAHE or Ben-Graham, MobileNetV2 embedding, cached)
30
+ -> fl/data.py (standardise, shared random projection to 128-d, non-IID Dirichlet split into hospitals)
31
+ -> fl/fed.py (local DP-SGD, FedAvg, fairness multipliers, metrics)
32
+ -> fl/experiment.py (3 modes: Baseline, +DP, +DP+Fairness; table and plots)
33
+ -> app.py (Gradio: experiment tab, single-image prediction tab)
34
+
35
+ WORKFLOW FOR EACH MY REQUEST
36
+ 1. Restate the change in one line and list the files you will touch.
37
+ 2. Make the change. Keep it minimal.
38
+ 3. Say exactly what command to run to verify, and what output I should see.
39
+ 4. List any new risk to privacy accounting or fairness metrics, or write "none".
40
+
41
+ FIRST TASKS (do them one by one, wait for my "next"):
42
+ A. Read the repo and tell me in 10 lines what it does and any bugs you see. Do not edit yet.
43
+ B. Add a sweep script experiments/sweep.py that runs epsilon in {1,2,4,8,inf} and 5 seeds for both diseases
44
+ and writes results/sweep.csv plus results/sweep.png (mean +- std AUC, TPR gap vs epsilon).
45
+ C. Add a centralized (non-federated) and a local-only baseline to fl/experiment.py and to the results table.
46
+ D. Add per-seed confidence intervals (5 seeds) to the app results table.
47
+ E. Add a "Download results CSV" button to app.py.
48
+ F. Write docs/REPORT.md: abstract, method (with equations), experiments, results table placeholders, limitations.
49
+ ```
50
+
51
+ ## Fast fixes cheat-sheet (tell the AI the symptom)
52
+ | Symptom | Likely cause | Ask the AI to |
53
+ |---|---|---|
54
+ | "no dataset worked for task" in the status box | dataset schema differs | print `first.keys()` and adapt `_heart_label_fn` / `_dr_label_fn` in fl/vision.py |
55
+ | Status says SYNTHETIC | real loading failed (see log lines) | fix loader using the logged columns, or use prepare_features.py in Colab |
56
+ | Space build fails on torch | wheel version | pin torch/torchvision to versions available on download.pytorch.org/whl/cpu |
57
+ | First run times out | too many images | lower "Images per class" to 80 or pre-build cache |
58
+ | Fairness gap not shrinking | groups tiny or noisy | raise `fair_eta`, more rounds, check group sizes in the log |
docs/VIVA_NOTES.md ADDED
@@ -0,0 +1,26 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Viva notes: how to present this without getting caught out
2
+
3
+ ## One-paragraph pitch
4
+ Hospitals cannot pool patient images. We let each hospital train locally, add differential-privacy noise
5
+ to its training updates, and combine updates on a server (FedAvg). Because privacy noise and class
6
+ imbalance can hurt under-represented patient groups, a fairness controller re-weights training so recall
7
+ is equalised across groups. We show the privacy / accuracy / fairness trade-off for diabetic retinopathy
8
+ (fundus) and cardiomegaly (chest X-ray).
9
+
10
+ ## Objectives (cleaned up from your draft)
11
+ 1. Build a federated pipeline for two disease-risk tasks from medical images without sharing raw images.
12
+ 2. Add record-level differential privacy (DP-SGD) with a measured (epsilon, delta).
13
+ 3. Add a group-fairness constraint (equal opportunity) and measure its cost.
14
+ 4. Quantify the trade-off: Baseline vs +DP vs +DP+Fairness, with seeds.
15
+
16
+ Changes vs your draft, and why: "add noise to weights" gives no epsilon, so we use DP-SGD (noise inside training, with accounting); fairness was missing from your objectives although it is in your title; Pneumonia is replaced by the two diseases you chose; "multi-disease" is realised as two independent tasks sharing one pipeline.
17
+
18
+ ## Likely questions
19
+ - **Why frozen MobileNetV2?** Lightweight, CPU-friendly, removes training time; standard transfer learning. Fine-tuning is future work.
20
+ - **Is this real federated learning?** Single-machine simulation with partitioned data, as in most FL research baselines. No network layer or secure aggregation.
21
+ - **What does epsilon mean here?** Record-level (epsilon, delta=1e-5)-DP per hospital over all rounds from the RDP accountant. Lower epsilon = stronger privacy. Values around 8-13 are common in practice; below 1 is very strong.
22
+ - **Why does fairness cost accuracy?** Group ROC curves differ, so equalising TPR at one threshold moves other error rates (see results: FPR gap rises). This is the known impossibility of satisfying all fairness metrics at once.
23
+ - **Why does DP hurt fairness?** Clipping and noise suppress rare/minority signals most (Bagdasaryan et al., 2019).
24
+ - **Where do fairness groups come from?** Sex where the dataset exposes it; otherwise an image-brightness proxy for acquisition conditions. State this plainly.
25
+ - **Is the fairness step private?** It releases noisy aggregate counts; that leakage is small but not included in the reported epsilon. Listed as a limitation.
26
+ - **Small data?** Yes, hundreds of images. Report mean +- std over seeds and avoid strong claims.
fl/__init__.py ADDED
File without changes
fl/data.py ADDED
@@ -0,0 +1,119 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Data layer: cached MobileNetV2 features, synthetic fallback, non-IID hospital split."""
2
+ import os
3
+ import numpy as np
4
+
5
+ CACHE_DIR = os.environ.get(
6
+ "FL_CACHE_DIR", os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "cache"))
7
+ RAW_DIM = 1280
8
+
9
+ TASKS = {
10
+ "heart": {"title": "Heart disease risk (cardiomegaly) - chest X-ray"},
11
+ "diabetes": {"title": "Diabetes risk (diabetic retinopathy) - retinal fundus"},
12
+ }
13
+
14
+
15
+ # ----------------------------------------------------------------- loading
16
+ def load_raw(task, n_per_class=150, seed=0, allow_synthetic=True, log=print):
17
+ path = os.path.join(CACHE_DIR, f"{task}.npz")
18
+ if os.path.exists(path):
19
+ z = np.load(path, allow_pickle=True)
20
+ log(f"Loaded cached features: {path} ({len(z['y'])} images)")
21
+ return {"X": z["X"], "y": z["y"], "g": z["g"], "source": str(z["source"]),
22
+ "group_name": str(z["group_name"]), "synthetic": False}
23
+ try:
24
+ from .vision import build_real_dataset
25
+ raw = build_real_dataset(task, n_per_class, seed, log)
26
+ os.makedirs(CACHE_DIR, exist_ok=True)
27
+ np.savez_compressed(path, X=raw["X"], y=raw["y"], g=raw["g"],
28
+ source=raw["source"], group_name=raw["group_name"])
29
+ log(f"Saved feature cache -> {path}")
30
+ raw["synthetic"] = False
31
+ return raw
32
+ except Exception as e: # noqa: BLE001
33
+ if not allow_synthetic:
34
+ raise
35
+ log(f"[WARNING] real data unavailable ({type(e).__name__}: {e}). "
36
+ f"Falling back to SYNTHETIC features so the pipeline still runs.")
37
+ return make_synthetic(task, seed)
38
+
39
+
40
+ def make_synthetic(task, seed=0, n=800):
41
+ """Synthetic 'MobileNet-like' features. Group 1 has a weaker signal on purpose
42
+ so the fairness machinery has something to fix. NOT real medical data."""
43
+ rng = np.random.default_rng(seed + (0 if task == "heart" else 1000))
44
+ d, k = RAW_DIM, 24 # real CNN embeddings live near a low-dimensional subspace
45
+ y = rng.integers(0, 2, n)
46
+ g = (rng.random(n) < 0.4).astype(int)
47
+ strength = np.where(g == 0, 0.9, 0.45)
48
+ lat = rng.normal(size=(n, k))
49
+ lat[:, :3] += ((2 * y - 1) * strength)[:, None] # disease signal in 3 latent axes
50
+ A = rng.normal(size=(k, d)) / np.sqrt(k)
51
+ X = lat @ A + 0.3 * rng.normal(size=(n, d))
52
+ return {"X": X.astype(np.float32), "y": y, "g": g, "source": "SYNTHETIC (demo only)",
53
+ "group_name": "synthetic group", "synthetic": True}
54
+
55
+
56
+ # ------------------------------------------------------------ transforms
57
+ def transform_features(prep, F):
58
+ """Standardise -> shared random projection -> scale. Same code for training and inference."""
59
+ return ((F - prep["mean"]) / prep["std"]) @ prep["P"] * prep["scale"]
60
+
61
+
62
+ def _try_split(raw, n_clients, alpha, proj_dim, seed, test_frac, val_frac):
63
+ rng = np.random.default_rng(seed)
64
+ X, y, g = raw["X"], raw["y"].astype(int), raw["g"].astype(int)
65
+ strata = y * 2 + g
66
+ test_idx, train_idx = [], []
67
+ for s in np.unique(strata):
68
+ idx = np.where(strata == s)[0]
69
+ rng.shuffle(idx)
70
+ nt = int(round(len(idx) * test_frac)) if len(idx) >= 4 else 0
71
+ test_idx += list(idx[:nt])
72
+ train_idx += list(idx[nt:])
73
+ train_idx, test_idx = np.array(train_idx), np.array(test_idx)
74
+
75
+ mean = X[train_idx].mean(0)
76
+ std = X[train_idx].std(0) + 1e-6
77
+ P = (rng.normal(size=(X.shape[1], proj_dim)) / np.sqrt(X.shape[1])).astype(np.float32)
78
+ prep = {"mean": mean, "std": std, "P": P, "scale": 3.0 / np.sqrt(proj_dim)}
79
+ Z = transform_features(prep, X).astype(np.float64)
80
+
81
+ assign = [[] for _ in range(n_clients)]
82
+ for s in np.unique(strata[train_idx]):
83
+ idx = train_idx[strata[train_idx] == s]
84
+ rng.shuffle(idx)
85
+ props = rng.dirichlet([alpha] * n_clients)
86
+ cuts = (np.cumsum(props) * len(idx)).astype(int)[:-1]
87
+ for c, part in enumerate(np.split(idx, cuts)):
88
+ assign[c] += list(part)
89
+
90
+ clients = []
91
+ for c, idx in enumerate(assign):
92
+ idx = np.array(idx)
93
+ if len(idx) < 30:
94
+ return None
95
+ rng.shuffle(idx)
96
+ nv = max(5, int(len(idx) * val_frac))
97
+ tr, va = idx[nv:], idx[:nv]
98
+ clients.append({"id": c, "X": Z[tr], "y": y[tr], "g": g[tr],
99
+ "Xv": Z[va], "yv": y[va], "gv": g[va]})
100
+ test = {"X": Z[test_idx], "y": y[test_idx], "g": g[test_idx]}
101
+ return {"clients": clients, "test": test, "prep": prep}
102
+
103
+
104
+ def prepare_split(raw, n_clients=3, alpha=1.0, proj_dim=128, seed=0, test_frac=0.25, val_frac=0.2):
105
+ """Split pooled data into a global test set and `n_clients` simulated hospitals
106
+ (Dirichlet label/group skew => non-IID)."""
107
+ for attempt in range(20):
108
+ out = _try_split(raw, n_clients, alpha, proj_dim, seed + attempt, test_frac, val_frac)
109
+ if out is not None:
110
+ break
111
+ else:
112
+ out = _try_split(raw, n_clients, 100.0, proj_dim, seed, test_frac, val_frac)
113
+ if out is None:
114
+ raise ValueError("Not enough samples for that many clients. Lower n_clients or raise samples/class.")
115
+ out["meta"] = {"source": raw["source"], "group_name": raw["group_name"],
116
+ "synthetic": raw["synthetic"], "n_total": int(len(raw["y"])),
117
+ "client_sizes": [len(c["y"]) for c in out["clients"]],
118
+ "client_pos_rate": [float(np.mean(c["y"])) for c in out["clients"]]}
119
+ return out
fl/dp.py ADDED
@@ -0,0 +1,53 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Differential-privacy primitives.
2
+
3
+ * clip_sum: per-example gradient clipping (bounds each patient's influence).
4
+ * epsilon: RDP accountant for the Poisson-subsampled Gaussian mechanism
5
+ (Mironov, Talwar, Zhang 2019, integer orders) with the Balle et al. 2020
6
+ RDP -> (eps, delta) conversion. Same recipe Opacus uses.
7
+ """
8
+ import math
9
+ import numpy as np
10
+
11
+
12
+ def clip_sum(per_example_grads, clip, weights=None):
13
+ """Clip every row to L2 norm <= clip, optionally scale each row by a public weight in [0, 1]
14
+ (applied AFTER clipping so it cannot be normalised away and sensitivity stays <= clip), then sum."""
15
+ norms = np.linalg.norm(per_example_grads, axis=1, keepdims=True)
16
+ scale = np.minimum(1.0, clip / (norms + 1e-12))
17
+ g = per_example_grads * scale
18
+ if weights is not None:
19
+ g = g * weights[:, None]
20
+ return g.sum(axis=0)
21
+
22
+
23
+ def _rdp_sampled_gaussian(q, sigma, alpha):
24
+ """RDP of order `alpha` (integer >= 2) for ONE step of the subsampled Gaussian."""
25
+ if q <= 0:
26
+ return 0.0
27
+ if sigma <= 0:
28
+ return float("inf")
29
+ if q >= 1.0:
30
+ return alpha / (2.0 * sigma ** 2)
31
+ terms = []
32
+ for k in range(alpha + 1):
33
+ log_coef = math.lgamma(alpha + 1) - math.lgamma(k + 1) - math.lgamma(alpha - k + 1)
34
+ terms.append(log_coef + k * math.log(q) + (alpha - k) * math.log1p(-q)
35
+ + (k * k - k) / (2.0 * sigma ** 2))
36
+ m = max(terms)
37
+ log_sum = m + math.log(sum(math.exp(t - m) for t in terms))
38
+ return log_sum / (alpha - 1)
39
+
40
+
41
+ ORDERS = list(range(2, 65)) + [80, 96, 128, 192, 256]
42
+
43
+
44
+ def epsilon(q, sigma, steps, delta=1e-5):
45
+ """(eps, delta)-DP after `steps` compositions of the subsampled Gaussian mechanism."""
46
+ if sigma <= 0:
47
+ return float("inf")
48
+ best = float("inf")
49
+ for a in ORDERS:
50
+ rdp = steps * _rdp_sampled_gaussian(q, sigma, a)
51
+ eps = rdp + math.log((a - 1) / a) - (math.log(delta) + math.log(a)) / (a - 1)
52
+ best = min(best, eps)
53
+ return max(best, 0.0)
fl/experiment.py ADDED
@@ -0,0 +1,86 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Runs the 3-mode comparison, builds the tables / plots / report text."""
2
+ from dataclasses import replace
3
+ import numpy as np
4
+ import pandas as pd
5
+ from matplotlib.figure import Figure
6
+ from .fed import Config, run_federated, proba # noqa: F401
7
+
8
+ STATE = {} # task -> trained model + preprocessing, used by the Predict tab
9
+
10
+ MODES = [
11
+ ("Baseline FedAvg", dict(dp=False, fairness=False)),
12
+ ("FedAvg + DP", dict(dp=True, fairness=False)),
13
+ ("FedAvg + DP + Fairness", dict(dp=True, fairness=True)),
14
+ ]
15
+
16
+
17
+ def run_modes(prepared, cfg):
18
+ out = []
19
+ for name, kw in MODES:
20
+ r = run_federated(prepared, replace(cfg, **kw))
21
+ r["name"] = name
22
+ out.append(r)
23
+ return out
24
+
25
+
26
+ def results_table(results):
27
+ rows = []
28
+ for r in results:
29
+ f = r["final"]
30
+ rows.append({
31
+ "Mode": r["name"],
32
+ "epsilon (worst client)": "inf (no DP)" if not np.isfinite(r["eps"]) else round(r["eps"], 2),
33
+ "Accuracy": round(f["acc"], 3), "AUC": round(f["auc"], 3),
34
+ "TPR gap": round(f["tpr_gap"], 3), "FPR gap": round(f["fpr_gap"], 3),
35
+ "EO gap (max)": round(f["eo_gap"], 3), "Parity gap": round(f["dp_gap"], 3),
36
+ "Worst-group acc": round(f["worst_group_acc"], 3),
37
+ })
38
+ return pd.DataFrame(rows)
39
+
40
+
41
+ def fig_curves(results):
42
+ fig = Figure(figsize=(11, 3.4), dpi=110)
43
+ specs = [("auc", "Test AUC"), ("acc", "Test accuracy"), ("eo_gap", "Equalized-odds gap (lower = fairer)")]
44
+ for i, (key, title) in enumerate(specs):
45
+ ax = fig.add_subplot(1, 3, i + 1)
46
+ for r in results:
47
+ ax.plot([h["round"] for h in r["history"]], [h[key] for h in r["history"]], label=r["name"], lw=1.8)
48
+ ax.set_title(title, fontsize=10)
49
+ ax.set_xlabel("federated round")
50
+ ax.grid(alpha=0.3)
51
+ if i == 0:
52
+ ax.legend(fontsize=7)
53
+ fig.tight_layout()
54
+ return fig
55
+
56
+
57
+ def fig_groups(results, group_name):
58
+ fig = Figure(figsize=(7, 3.4), dpi=110)
59
+ ax = fig.add_subplot(1, 1, 1)
60
+ width = 0.8 / len(results)
61
+ for i, r in enumerate(results):
62
+ t = r["final"]["tpr"]
63
+ ax.bar(np.arange(2) + i * width, t, width, label=r["name"])
64
+ ax.set_xticks(np.arange(2) + width)
65
+ ax.set_xticklabels(["group 0", "group 1"])
66
+ ax.set_ylabel("True-positive rate (recall)")
67
+ ax.set_title(f"Per-group recall - groups: {group_name}", fontsize=10)
68
+ ax.legend(fontsize=7)
69
+ ax.grid(alpha=0.3, axis="y")
70
+ fig.tight_layout()
71
+ return fig
72
+
73
+
74
+ def report_markdown(task_title, prepared, results, table):
75
+ meta = prepared["meta"]
76
+ lines = [f"### {task_title}", "",
77
+ f"- Data: **{meta['source']}**" + (" \n- **WARNING: synthetic demo data, not real patients**" if meta["synthetic"] else ""),
78
+ f"- Simulated hospitals: {len(meta['client_sizes'])} (train sizes {meta['client_sizes']}, "
79
+ f"positive rates {[round(x, 2) for x in meta['client_pos_rate']]})",
80
+ f"- Fairness groups: {meta['group_name']}", "",
81
+ "| " + " | ".join(table.columns) + " |", "|" + "---|" * len(table.columns)]
82
+ for _, row in table.iterrows():
83
+ lines.append("| " + " | ".join(str(v) for v in row.values) + " |")
84
+ lines += ["", "_epsilon is record-level (eps, delta=1e-5)-DP per hospital over all rounds (RDP accountant, "
85
+ "approximate). Research prototype, not a medical device._"]
86
+ return "\n".join(lines)
fl/fed.py ADDED
@@ -0,0 +1,165 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Federated learning core (pure NumPy, runs in seconds on any CPU).
2
+
3
+ Pipeline per round:
4
+ 1. server broadcasts global weights + fairness multipliers
5
+ 2. each client (hospital) trains locally with DP-SGD
6
+ (Poisson sampling, per-example clipping, Gaussian noise)
7
+ 3. server aggregates with FedAvg (weighted by client size)
8
+ 4. server updates fairness multipliers from aggregated group TPR/FPR counts
9
+ """
10
+ from dataclasses import dataclass
11
+ import numpy as np
12
+ from sklearn.metrics import roc_auc_score
13
+ from . import dp as dpm
14
+
15
+
16
+ @dataclass
17
+ class Config:
18
+ n_clients: int = 3
19
+ rounds: int = 20
20
+ local_epochs: int = 2
21
+ batch_size: int = 32
22
+ lr: float = 1.0
23
+ l2: float = 1e-3
24
+ dp: bool = True
25
+ clip: float = 1.0
26
+ noise_mult: float = 1.0
27
+ delta: float = 1e-5
28
+ fairness: bool = True
29
+ fair_eta: float = 2.0
30
+ fair_target: str = "tpr" # "tpr" = equal opportunity, "both" = equalized odds
31
+ count_noise: float = 1.0 # Gaussian noise on the group counts each client releases
32
+ alpha: float = 1.0 # Dirichlet non-IID strength (small = more skew)
33
+ proj_dim: int = 128
34
+ seed: int = 0
35
+
36
+
37
+ # ----------------------------------------------------------------- model
38
+ def sigmoid(z):
39
+ return 1.0 / (1.0 + np.exp(-np.clip(z, -30, 30)))
40
+
41
+
42
+ def proba(w, X):
43
+ return sigmoid(X @ w[:-1] + w[-1])
44
+
45
+
46
+ def per_example_grads(w, X, y, sw):
47
+ r = (proba(w, X) - y) * sw
48
+ return np.concatenate([X * r[:, None], r[:, None]], axis=1)
49
+
50
+
51
+ # -------------------------------------------------------------- fairness
52
+ def group_counts(w, c, noise, rng):
53
+ """Per-group [TP, P, FP, N] on the client's validation split. Only (noisy) aggregate counts leave the client."""
54
+ pred = proba(w, c["Xv"]) >= 0.5
55
+ y, g = c["yv"], c["gv"]
56
+ out = np.zeros((2, 4))
57
+ for gg in (0, 1):
58
+ m = g == gg
59
+ out[gg] = [(pred & (y == 1) & m).sum(), ((y == 1) & m).sum(),
60
+ (pred & (y == 0) & m).sum(), ((y == 0) & m).sum()]
61
+ return out
62
+
63
+
64
+ def update_multipliers(mult, counts, eta, target="tpr"):
65
+ """Multiplicative-weights feedback controller: up-weights the group that is under-served.
66
+ target="tpr": equal opportunity (TPR gap); "both": equalized odds (TPR and FPR gaps)."""
67
+ tp, pos, fp, neg = counts[:, 0], counts[:, 1], counts[:, 2], counts[:, 3]
68
+ if pos.min() < 1 or neg.min() < 1:
69
+ return mult
70
+ tpr, fpr = tp / pos, fp / neg
71
+ d, e = tpr[0] - tpr[1], fpr[0] - fpr[1]
72
+ mp = np.clip(mult["pos"] * np.exp(eta * np.array([-d, d])), 0.1, 10.0)
73
+ mn = mult["neg"]
74
+ if target == "both":
75
+ mn = np.clip(mult["neg"] * np.exp(eta * np.array([e, -e])), 0.1, 10.0)
76
+ return {"pos": mp, "neg": mn}
77
+
78
+
79
+ # --------------------------------------------------------------- clients
80
+ def local_train(w0, c, cfg, mult, rng):
81
+ X, y, g = c["X"], c["y"].astype(float), c["g"]
82
+ n = len(y)
83
+ w = w0.copy()
84
+ q = min(1.0, cfg.batch_size / n)
85
+ steps_per_epoch = max(1, int(round(1.0 / q)))
86
+ steps = cfg.local_epochs * steps_per_epoch
87
+
88
+ sw = np.ones(n) # public, server-provided weights in (0, 1]
89
+ if cfg.fairness:
90
+ top = max(mult["pos"].max(), mult["neg"].max())
91
+ sw = np.where(c["y"] == 1, mult["pos"][g], mult["neg"][g]) / top
92
+
93
+ for _ in range(steps):
94
+ if cfg.dp:
95
+ mask = rng.random(n) < q # Poisson sampling
96
+ if mask.any():
97
+ G = per_example_grads(w, X[mask], y[mask], np.ones(mask.sum()))
98
+ gsum = dpm.clip_sum(G, cfg.clip, sw[mask])
99
+ else:
100
+ gsum = np.zeros_like(w)
101
+ gsum = gsum + rng.normal(0.0, cfg.noise_mult * cfg.clip, size=w.shape)
102
+ grad = gsum / (q * n) # divide by expected lot size
103
+ else:
104
+ b = rng.choice(n, size=min(cfg.batch_size, n), replace=False)
105
+ grad = (per_example_grads(w, X[b], y[b], np.ones(len(b))) * sw[b][:, None]).mean(axis=0)
106
+ grad = grad + cfg.l2 * w
107
+ w = w - cfg.lr * grad
108
+ return w, n, steps, q
109
+
110
+
111
+ def fedavg(ws, ns):
112
+ return np.average(np.stack(ws), axis=0, weights=np.asarray(ns, float))
113
+
114
+
115
+ # ------------------------------------------------------------ evaluation
116
+ def evaluate(w, X, y, g, thr=0.5):
117
+ p = proba(w, X)
118
+ pred = (p >= thr).astype(int)
119
+ out = {"acc": float((pred == y).mean()),
120
+ "auc": float(roc_auc_score(y, p)) if len(np.unique(y)) == 2 else float("nan")}
121
+ tpr, fpr, sel, gacc = [], [], [], []
122
+ for gg in (0, 1):
123
+ m = g == gg
124
+ pos, neg = m & (y == 1), m & (y == 0)
125
+ tpr.append(float(pred[pos].mean()) if pos.any() else float("nan"))
126
+ fpr.append(float(pred[neg].mean()) if neg.any() else float("nan"))
127
+ sel.append(float(pred[m].mean()) if m.any() else float("nan"))
128
+ gacc.append(float((pred[m] == y[m]).mean()) if m.any() else float("nan"))
129
+ out.update(tpr=tpr, fpr=fpr, sel=sel, gacc=gacc)
130
+ out["tpr_gap"] = abs(tpr[0] - tpr[1])
131
+ out["fpr_gap"] = abs(fpr[0] - fpr[1])
132
+ out["eo_gap"] = float(np.nanmax([out["tpr_gap"], out["fpr_gap"]]))
133
+ out["dp_gap"] = abs(sel[0] - sel[1])
134
+ out["worst_group_acc"] = float(np.nanmin(gacc))
135
+ return out
136
+
137
+
138
+ # ------------------------------------------------------------------- run
139
+ def run_federated(prep, cfg):
140
+ rng = np.random.default_rng(cfg.seed)
141
+ clients, test = prep["clients"], prep["test"]
142
+ w = np.zeros(clients[0]["X"].shape[1] + 1)
143
+ mult = {"pos": np.ones(2), "neg": np.ones(2)}
144
+ hist, per_round = [], {}
145
+ for rnd in range(cfg.rounds):
146
+ ws, ns = [], []
147
+ for k, c in enumerate(clients):
148
+ wk, n, steps, q = local_train(w, c, cfg, mult, rng)
149
+ ws.append(wk)
150
+ ns.append(n)
151
+ per_round[k] = (q, steps)
152
+ w = fedavg(ws, ns)
153
+ if cfg.fairness:
154
+ counts = sum(group_counts(w, c, cfg.count_noise, rng) for c in clients)
155
+ mult = update_multipliers(mult, counts, cfg.fair_eta, cfg.fair_target)
156
+ m = evaluate(w, test["X"], test["y"], test["g"])
157
+ m["round"] = rnd + 1
158
+ hist.append(m)
159
+
160
+ if cfg.dp and cfg.noise_mult > 0:
161
+ eps = max(dpm.epsilon(q, cfg.noise_mult, steps * cfg.rounds, cfg.delta)
162
+ for q, steps in per_round.values())
163
+ else:
164
+ eps = float("inf")
165
+ return {"w": w, "history": hist, "final": hist[-1], "eps": eps, "mult": mult, "cfg": cfg}
fl/vision.py ADDED
@@ -0,0 +1,216 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Image-processing layer: OpenCV preprocessing + frozen MobileNetV2 feature extractor
2
+ + streaming loaders for free Hugging Face datasets. torch/datasets are imported lazily."""
3
+ import io
4
+ import os
5
+ import numpy as np
6
+ import cv2
7
+ from PIL import Image
8
+
9
+ IMG = 224
10
+
11
+ # NOTE: dataset ids were verified to exist on the Hub; column names are auto-detected
12
+ # at runtime because dataset cards do not always document them.
13
+ TASK_SPECS = {
14
+ "heart": {"dataset_ids": ["chehablab/NIHChestXR"], "max_scan": 30000,
15
+ "source": "NIH ChestX-ray14 (HF: chehablab/NIHChestXR) - label: Cardiomegaly"},
16
+ "diabetes": {"dataset_ids": ["skip113/retina", "ctmedtech/DDR-dataset"], "max_scan": 10000,
17
+ "source": "Retinal fundus DR grading (HF) - label: DR grade >= 1"},
18
+ }
19
+
20
+
21
+ # ------------------------------------------------------- preprocessing
22
+ def preprocess_cxr(rgb):
23
+ """Chest X-ray: grayscale -> resize -> CLAHE contrast enhancement."""
24
+ gray = cv2.cvtColor(rgb, cv2.COLOR_RGB2GRAY)
25
+ gray = cv2.resize(gray, (IMG, IMG), interpolation=cv2.INTER_AREA)
26
+ g = cv2.createCLAHE(clipLimit=2.0, tileGridSize=(8, 8)).apply(gray)
27
+ return np.stack([g, g, g], -1)
28
+
29
+
30
+ def preprocess_fundus(rgb):
31
+ """Fundus: crop black border -> resize -> Ben Graham local-contrast normalisation."""
32
+ gray = cv2.cvtColor(rgb, cv2.COLOR_RGB2GRAY)
33
+ ys, xs = np.where(gray > 15)
34
+ if len(ys) > 100:
35
+ rgb = rgb[ys.min():ys.max() + 1, xs.min():xs.max() + 1]
36
+ rgb = cv2.resize(rgb, (IMG, IMG), interpolation=cv2.INTER_AREA)
37
+ blur = cv2.GaussianBlur(rgb, (0, 0), IMG / 30)
38
+ return cv2.addWeighted(rgb, 4, blur, -4, 128)
39
+
40
+
41
+ def preprocess(task, pil):
42
+ rgb = np.array(pil.convert("RGB"))
43
+ return preprocess_cxr(rgb) if task == "heart" else preprocess_fundus(rgb)
44
+
45
+
46
+ def brightness(pil):
47
+ """Mean intensity of a downsized image: acquisition-condition proxy for fairness groups."""
48
+ small = np.array(pil.convert("RGB").resize((64, 64)))
49
+ return float(small.mean())
50
+
51
+
52
+ # --------------------------------------------------- feature extractor
53
+ _MODEL = None
54
+
55
+
56
+ def _get_model():
57
+ global _MODEL
58
+ if _MODEL is None:
59
+ import torch
60
+ import torchvision
61
+ torch.set_num_threads(max(1, os.cpu_count() or 1))
62
+ m = torchvision.models.mobilenet_v2(weights=torchvision.models.MobileNet_V2_Weights.IMAGENET1K_V1)
63
+ m.classifier = torch.nn.Identity() # -> 1280-d embedding
64
+ m.eval()
65
+ _MODEL = m
66
+ return _MODEL
67
+
68
+
69
+ def extract_features(imgs, batch=32):
70
+ import torch
71
+ m = _get_model()
72
+ mean = np.array([0.485, 0.456, 0.406], np.float32)
73
+ std = np.array([0.229, 0.224, 0.225], np.float32)
74
+ out = []
75
+ with torch.no_grad():
76
+ for i in range(0, len(imgs), batch):
77
+ b = np.stack(imgs[i:i + batch]).astype(np.float32) / 255.0
78
+ t = torch.from_numpy(((b - mean) / std).transpose(0, 3, 1, 2).copy())
79
+ out.append(m(t).numpy())
80
+ return np.concatenate(out).astype(np.float32)
81
+
82
+
83
+ # ------------------------------------------------- dataset adapters
84
+ def _mentions_cardiomegaly(v):
85
+ if isinstance(v, (list, tuple, np.ndarray)):
86
+ return any(_mentions_cardiomegaly(x) for x in v)
87
+ if isinstance(v, str):
88
+ return "cardiomegaly" in v.lower()
89
+ if isinstance(v, (int, np.integer)):
90
+ return int(v) == 2 # NIH label map: 0 No Finding, 1 Atelectasis, 2 Cardiomegaly
91
+ return False
92
+
93
+
94
+ def _heart_label_fn(first):
95
+ cols = ["labels", "label", "Finding Labels", "finding_labels", "findings", "Labels"]
96
+ col = next((c for c in cols if c in first), None)
97
+ if col is None:
98
+ raise KeyError(f"no label column found; columns = {list(first.keys())}")
99
+ return lambda ex: int(_mentions_cardiomegaly(ex[col]))
100
+
101
+
102
+ def _sex_fn(first):
103
+ cols = ["patient_gender", "Patient Gender", "patient_sex", "gender", "sex", "Sex", "Gender"]
104
+ col = next((c for c in cols if c in first), None)
105
+ if col is None:
106
+ for c, v in first.items():
107
+ if isinstance(v, str) and v.strip().upper() in ("M", "F", "MALE", "FEMALE"):
108
+ col = c
109
+ break
110
+ if col is None:
111
+ return None
112
+
113
+ def fn(ex):
114
+ v = str(ex[col]).strip().upper()
115
+ return 0 if v in ("M", "MALE") else 1 if v in ("F", "FEMALE") else None
116
+ return fn
117
+
118
+
119
+ def _dr_label_fn(first):
120
+ cols = ["label", "labels", "grade", "level", "dr_grade", "diagnosis", "answer", "target", "severity"]
121
+ col = next((c for c in cols if c in first), None)
122
+ if col is None:
123
+ raise KeyError(f"no label column found; columns = {list(first.keys())}")
124
+ qcol = next((c for c in ("question", "prompt", "instruction", "text")
125
+ if c in first and isinstance(first[c], str)), None)
126
+
127
+ def fn(ex):
128
+ if qcol and "retinopathy" not in str(ex[qcol]).lower():
129
+ return None
130
+ v = ex[col]
131
+ if isinstance(v, (list, tuple)) and v:
132
+ v = v[0]
133
+ s = str(v).strip().lower()
134
+ try:
135
+ grade = int(float(s))
136
+ return int(grade >= 1) if 0 <= grade <= 4 else None
137
+ except ValueError:
138
+ if s in ("no dr", "normal", "no_dr", "none"):
139
+ return 0
140
+ if any(t in s for t in ("npdr", "pdr", "mild", "moderate", "severe", "proliferative")):
141
+ return 1
142
+ return None
143
+ return fn
144
+
145
+
146
+ def _get_pil(v):
147
+ if isinstance(v, Image.Image):
148
+ return v
149
+ if isinstance(v, dict) and v.get("bytes"):
150
+ return Image.open(io.BytesIO(v["bytes"]))
151
+ raise ValueError("unsupported image cell")
152
+
153
+
154
+ def _collect(did, task, n_per_class, seed, log):
155
+ from datasets import load_dataset, Image as HFImage
156
+ spec = TASK_SPECS[task]
157
+ log(f"Streaming '{did}' (no full download) ...")
158
+ ds = load_dataset(did, split="train", streaming=True)
159
+ first = next(iter(ds))
160
+ img_col = next((k for k, v in first.items() if isinstance(v, Image.Image)), None)
161
+ if img_col is None:
162
+ raise KeyError(f"no image column; columns = {list(first.keys())}")
163
+ log(f" columns: {list(first.keys())} (image column: '{img_col}')")
164
+ try:
165
+ ds = ds.cast_column(img_col, HFImage(decode=False)) # decode only the rows we keep
166
+ except Exception: # noqa: BLE001
167
+ pass
168
+ ds = ds.shuffle(seed=seed, buffer_size=2000)
169
+
170
+ label_fn = _heart_label_fn(first) if task == "heart" else _dr_label_fn(first)
171
+ group_fn = _sex_fn(first) if task == "heart" else None
172
+
173
+ imgs, ys, gs, bs = [], [], [], []
174
+ counts = {0: 0, 1: 0}
175
+ scanned = 0
176
+ for ex in ds:
177
+ scanned += 1
178
+ if scanned > spec["max_scan"] or all(v >= n_per_class for v in counts.values()):
179
+ break
180
+ y = label_fn(ex)
181
+ if y is None or counts[y] >= n_per_class:
182
+ continue
183
+ pil = _get_pil(ex[img_col])
184
+ imgs.append(preprocess(task, pil))
185
+ bs.append(brightness(pil))
186
+ gs.append(group_fn(ex) if group_fn else None)
187
+ ys.append(y)
188
+ counts[y] += 1
189
+ if len(ys) % 50 == 0:
190
+ log(f" kept {len(ys)} images (neg={counts[0]}, pos={counts[1]}), scanned {scanned}")
191
+ log(f"Scan finished: neg={counts[0]}, pos={counts[1]}, scanned={scanned}")
192
+ if min(counts.values()) < 20:
193
+ raise RuntimeError(f"too few samples per class ({counts}); dataset schema may differ")
194
+
195
+ ys = np.array(ys)
196
+ if group_fn and all(v is not None for v in gs) and len(set(gs)) == 2:
197
+ g, gname = np.array(gs), "sex (0=male, 1=female)"
198
+ else:
199
+ b = np.array(bs)
200
+ g, gname = (b > np.median(b)).astype(int), "image-brightness proxy (acquisition condition)"
201
+ log(f"Fairness groups: {gname}; group sizes = {np.bincount(g).tolist()}")
202
+
203
+ log(f"Extracting MobileNetV2 features for {len(imgs)} images (one-off) ...")
204
+ X = extract_features(imgs)
205
+ return {"X": X, "y": ys, "g": g, "source": f"{did} | {spec['source']}", "group_name": gname}
206
+
207
+
208
+ def build_real_dataset(task, n_per_class, seed, log=print):
209
+ last = None
210
+ for did in TASK_SPECS[task]["dataset_ids"]:
211
+ try:
212
+ return _collect(did, task, n_per_class, seed, log)
213
+ except Exception as e: # noqa: BLE001
214
+ log(f" '{did}' failed: {type(e).__name__}: {e}")
215
+ last = e
216
+ raise RuntimeError(f"no dataset worked for task '{task}': {last}")
prepare_features.py ADDED
@@ -0,0 +1,20 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Build the cached feature files ONCE (optional but recommended), then commit cache/*.npz
2
+ so the Space boots instantly and never depends on the dataset downloads again.
3
+
4
+ pip install torch torchvision datasets opencv-python-headless pillow numpy
5
+ python prepare_features.py --task heart --n 150
6
+ python prepare_features.py --task diabetes --n 150
7
+
8
+ Runs fine on a free Google Colab CPU runtime or on a laptop (~2-5 min per task).
9
+ """
10
+ import argparse
11
+ from fl import data
12
+
13
+ ap = argparse.ArgumentParser()
14
+ ap.add_argument("--task", choices=list(data.TASKS), required=True)
15
+ ap.add_argument("--n", type=int, default=150, help="images per class")
16
+ ap.add_argument("--seed", type=int, default=0)
17
+ a = ap.parse_args()
18
+
19
+ raw = data.load_raw(a.task, a.n, a.seed, allow_synthetic=False)
20
+ print("OK:", a.task, raw["X"].shape, "positives:", int(raw["y"].sum()), "| groups:", raw["group_name"])
requirements.txt ADDED
@@ -0,0 +1,11 @@
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Cached-features mode: the app needs NO torch at runtime (features are built once by prepare_features.py).
2
+ # ZeroGPU Spaces already ship torch, so it is deliberately not pinned here.
3
+ numpy<2
4
+ pandas
5
+ scikit-learn
6
+ matplotlib
7
+ Pillow
8
+ opencv-python-headless==4.10.0.84
9
+ # Optional, only for tab 2 (single-image scoring) on hosts that don't preinstall torch:
10
+ # torch
11
+ # torchvision
tests/smoke_test.py ADDED
@@ -0,0 +1,32 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Torch-free smoke test of the whole federated core on synthetic features.
2
+ Run: python tests/smoke_test.py"""
3
+ import math
4
+ import os
5
+ import sys
6
+
7
+ sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
8
+ from fl import data, dp, experiment as E # noqa: E402
9
+ from fl.fed import Config # noqa: E402
10
+
11
+ # --- DP accountant sanity checks
12
+ assert abs(dp._rdp_sampled_gaussian(1.0, 2.0, 8) - 8 / (2 * 4)) < 1e-9 # q=1 -> plain Gaussian RDP
13
+ assert dp.epsilon(0.05, 1.0, 100) < dp.epsilon(0.05, 1.0, 1000) # more steps -> more eps
14
+ assert dp.epsilon(0.05, 2.0, 500) < dp.epsilon(0.05, 1.0, 500) # more noise -> less eps
15
+ assert dp.epsilon(0.01, 1.0, 500) < dp.epsilon(0.1, 1.0, 500) # less sampling -> less eps
16
+ assert math.isinf(dp.epsilon(0.05, 0.0, 10))
17
+
18
+ # --- full pipeline, 3 modes
19
+ for task in ("heart", "diabetes"):
20
+ raw = data.make_synthetic(task, seed=1)
21
+ cfg = Config(rounds=15)
22
+ prepared = data.prepare_split(raw, cfg.n_clients, cfg.alpha, cfg.proj_dim, cfg.seed)
23
+ res = E.run_modes(prepared, cfg)
24
+ table = E.results_table(res)
25
+ print(f"\n== {task} ==")
26
+ print(table.to_string(index=False))
27
+ assert len(res) == 3 and res[0]["final"]["auc"] > 0.6
28
+ assert math.isfinite(res[1]["eps"]) and math.isfinite(res[2]["eps"]) and math.isinf(res[0]["eps"])
29
+ E.fig_curves(res)
30
+ E.fig_groups(res, "synthetic")
31
+ E.report_markdown("t", prepared, res, table)
32
+ print("\nSMOKE TEST PASSED")