CASE_STUDY.md

# Case Study — NephroScreen

*An end-to-end applied machine-learning project: screening for chronic kidney
disease from routine lab values, built and analysed solo.*

---

## At a glance

| | |
|---|---|
| **Problem** | Binary classification — predict CKD vs. healthy from routine clinical/lab features |
| **Data** | UCI Chronic Kidney Disease dataset — 400 patients, 24 attributes, ~11% cells missing |
| **Best model** | Random Forest — 100% test accuracy, ROC-AUC 1.0000 (read the caveat below) |
| **Stack** | scikit-learn · FastAPI · vanilla JS frontend · Docker/Render |
| **What makes it mature** | Leakage-safe pipeline · SHAP interpretability · calibration · screening-tuned threshold · tests |

## 1. The problem

Chronic kidney disease often progresses silently until it is advanced, yet the
warning signs are already present in routine blood and urine tests. The task:
given a patient's lab panel, flag likely CKD early enough to matter. This is a
classic **screening** problem, which shapes every later decision — especially
how errors are weighted.

## 2. The data — and an honest catch

The UCI CKD dataset is small (400 rows) and the two classes are **highly
separable** on a few biomarkers. Every model I trained scored in the high 90s,
and Random Forest reached **100% on the held-out test set**.

**That number is a red flag, not a trophy.** With an 80-patient test set drawn
from one clean, single-source dataset, near-perfect accuracy reflects the
*separability of this data*, not real-world clinical performance. I treat it that
way throughout — the engineering value here is doing the workflow *correctly*,
not chasing a headline metric.

## 3. Approach

**Leakage-safe preprocessing.** All imputation, encoding, and feature
engineering live in a custom scikit-learn transformer (`CKDPreprocessor`) wrapped
in a `Pipeline`. Every statistic — medians, modes, the scaler, the `age_group`
fill value — is fit on the **training fold only**, so cross-validation and the
live API never see test-set information.

**Domain feature engineering.** Four clinically motivated features, e.g.
`kidney_stress_index = serum_creatinine × blood_urea / hemoglobin` (the CKD triad
in one number). SHAP later confirmed it earns its place in the top five.

**Three models compared** with 10-fold stratified CV:

| Model | Accuracy | Recall | F1 | ROC-AUC | 10-Fold CV |
|---|---|---|---|---|---|
| Logistic Regression | 98.75% | 98% | 98.99% | 1.0000 | 99.38% ± 1.25% |
| **Random Forest** | **100%** | **100%** | **100%** | **1.0000** | 98.75% ± 2.86% |
| Decision Tree | 97.50% | 98% | 98.00% | 0.9893 | 96.56% ± 2.19% |

## 4. Interpretability — what the model actually uses

![SHAP summary](assets/shap_summary.png)

SHAP attributions are clinically coherent and build trust in the model:

- **Low hemoglobin and PCV** push strongly toward CKD (anemia of kidney disease).
- **High serum creatinine** and the engineered **kidney_stress_index** are top drivers.
- **Hypertension and diabetes = "yes"** push toward CKD — the two leading real-world causes.

This is the difference between "the model is accurate" and "I understand *why*
the model is accurate" — the latter is what makes a result defensible.

## 5. Maturity signals

**Calibration.** Do the probabilities mean what they say? The reliability curve
checks whether an "0.8" prediction is CKD ~80% of the time — essential before
anyone acts on a probability.

![Calibration](assets/calibration.png)

**A screening-tuned threshold.** For screening, a *missed* CKD case (false
negative) is far costlier than a false alarm. Instead of the default 0.5, the API
serves a threshold tuned to keep **recall ≥ 0.99**, accepting more false
positives as the safer trade.

![Threshold trade-off](assets/threshold.png)

## 6. From model to product

The trained pipeline is persisted with `joblib` and served by a **FastAPI**
backend; a dependency-free web frontend collects a lab panel (missing fields are
imputed) and shows the probability, a risk band, and which entered values fall
outside normal clinical ranges. Containerised for free-tier deployment.

```
Browser → FastAPI /api/predict → joblib pipeline (CKDPreprocessor → RandomForest)
```

## 7. Limitations & what I'd do next

- **Single small dataset** → results will not transfer to real clinical
  populations without external validation on an independent cohort.
- **No serving-time calibration** yet; probabilities should be calibrated
  (e.g. isotonic) before being shown as clinical likelihoods.
- **Next experiment:** find a **minimal, low-cost feature subset** — how much
  accuracy is lost if the expensive labs are dropped? That's the question with
  real value for screening in resource-limited settings.

---

*Built solo by Sadiq Mansoor. Dataset: UCI ML Repository — Chronic Kidney
Disease (ID 336). Educational project, not a medical device.*