README.md

metadata

title: PneumoOps
emoji: 🫁
colorFrom: blue
colorTo: indigo
sdk: docker
app_port: 7860
pinned: true
license: mit
short_description: MLOps A/B testing & drift monitoring

🫁 PneumoOps

Continuous MLOps Pipeline with A/B Testing & Data Drift Monitoring for 14-Class Thoracic Disease Detection

---

What This Is

PneumoOps is a production-style MLOps system for multi-label chest X-ray classification. It demonstrates real-world deployment challenges:

• A/B Testing — every inference request is randomly routed to either Model A (PyTorch) or Model B (ONNX), letting you measure real-world latency differences between serving backends.
• Data Drift Monitoring — statistical pixel-distribution comparison (KS-test) against the training baseline. When distribution shifts, the system flags DRIFT_DETECTED — the trigger for automated retraining in production.
• Prometheus Observability — request counters, latency histograms, per-disease prediction rates, and drift alert counters are all scraped at /metrics.
• Dockerized Deployment — the entire stack runs in containers, deployable to Hugging Face Spaces via a single git push.

---

Architecture

         Train (ChestMNIST + MobileNetV3-small)
                    │
          ┌─────────┴──────────┐
    Model A (.pth)       Model B (.onnx)
    PyTorch serving      ONNX Runtime serving
          └─────────┬──────────┘
                    │
        ┌───────────┴───────────┐
        │  FastAPI Backend       │
        │  ├─ A/B Router (60/40) │
        │  ├─ Drift Monitor (KS) │
        │  ├─ Prometheus /metrics│
        │  └─ /health /history   │
        └───────────┬───────────┘
                    │
          ┌─────────┴──────────┐
          │   Gradio UI         │
          │  Top-3 predictions  │
          │  Model arm used     │
          │  Latency (ms)       │
          │  Drift alert badge  │
          └─────────────────────┘

---

Dataset & Model

Property	Value
Dataset	ChestMNIST — 14-class multi-label chest X-ray
Classes	Atelectasis, Cardiomegaly, Effusion, Infiltration, Mass, Nodule, Pneumonia, Pneumothorax, Consolidation, Edema, Emphysema, Fibrosis, Pleural Thickening, Hernia
Model A	MobileNetV3-small (PyTorch .pth)
Model B	MobileNetV3-small (ONNX Runtime .onnx)
Training	5 epochs, AdamW, BCEWithLogitsLoss, per-class threshold tuning
Macro AUROC	0.686 (5-epoch, 5k samples — improves with full dataset)

---

Project Structure

pneumo_ops/
├── backend/
│   └── main.py              # FastAPI: A/B routing, drift monitor, Prometheus
├── frontend/
│   └── app.py               # Gradio UI: top-3 chart, drift badge, latency
├── scripts/
│   └── train_chestmnist.py  # Training: ChestMNIST → MobileNetV3 → ONNX export
├── models/
│   └── chestmnist_mobilenetv3/
│       ├── mobilenetv3_chestmnist.pth    # Model A (PyTorch)
│       ├── mobilenetv3_chestmnist.onnx   # Model B (ONNX)
│       ├── training_metrics.json
│       └── baseline_stats.json           # Pixel stats for drift reference
├── model_utils.py           # CalibratedModel + temperature scaling util
├── Dockerfile               # Single-container build
├── docker-compose.yml       # backend + frontend services
├── requirements.txt
└── .github/workflows/
    └── deploy.yml           # CI (lint/import check) + HF Spaces deploy

---

Quick Start

1. Install

git clone https://github.com/Prakhar54-byte/PneumoOps
cd pneumo_ops
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

2. Train the model

# Quick run (5k samples, ~1 min on GPU)
python3 scripts/train_chestmnist.py --epochs 5 --batch-size 32 --max-train-samples 5000

# Full dataset
python3 scripts/train_chestmnist.py --epochs 15 --batch-size 64

Outputs saved to models/chestmnist_mobilenetv3/:

• mobilenetv3_chestmnist.pth — PyTorch checkpoint
• mobilenetv3_chestmnist.onnx — ONNX export
• training_metrics.json — AUROC, AUPRC, F1, thresholds
• baseline_stats.json — pixel reference for drift detection

3. Run the backend

PNEUMOOPS_PROFILE=chestmnist python3 -m uvicorn backend.main:app --port 7860

Key endpoints:

Endpoint	Description
POST /predict	Run inference (A/B routed)
GET /health	System status + model metadata
GET /metrics	Prometheus scrape endpoint
GET /history	Last 20 requests
GET /metrics/class-rates	Per-class prediction rates
GET /metrics/calibration	AUROC / AUPRC / Brier per class

4. Run the UI

BACKEND_PREDICT_URL=http://127.0.0.1:7860/predict python3 frontend/app.py

5. Docker (full stack)

docker compose up --build
# Backend → http://localhost:7860
# Frontend → http://localhost:7861

---

Deployment — Hugging Face Spaces

Manual push

# Add HF remote
git remote add space https://huggingface.co/spaces/Prakhar54-byte/PneumoOps

# Push (Spaces will build the Docker image automatically)
git push space main

Automated (GitHub Actions)

Set these repository secrets on GitHub:

Secret	Description
HF_TOKEN	Hugging Face access token (write permission)
HF_SPACE_REPO	e.g. your-username/pneumoops
HF_MODEL_REPO	(optional) e.g. your-username/pneumoops-models

Every push to main triggers CI checks then deploys to your Space automatically.

---

Real-World MLOps Challenges Addressed

Challenge	Solution
Model degradation over time	Drift Monitor (KS-test on pixel distribution)
Serving latency variance	A/B routing between PyTorch and ONNX, latency tracked per arm
Class imbalance (rare diseases)	Per-class threshold tuning on val set + AUPRC tracking
Missed diagnoses	Per-class recall monitored at /metrics/class-rates
Production observability	Prometheus metrics — latency histograms, per-disease counters, drift alerts
Automated retraining signals	DRIFT_DETECTED flag logged + exposed via Prometheus counter

---

Libraries

PyTorch · ONNX Runtime · FastAPI · Gradio · Docker · Hugging Face Hub/Spaces · scikit-learn · Prometheus · MedMNIST · SciPy

---

License

MIT