--- license: apache-2.0 library_name: transformers pipeline_tag: image-text-to-text tags: - medical - chest-xray - radiology - onnx - clip - blip - multimodal - cpu base_model: - openai/clip-vit-base-patch32 - emilyalsentzer/Bio_ClinicalBERT - Salesforce/blip-image-captioning-base language: - en --- # MedicalAI — Light Weight Chest X-ray analysis that runs on consumer hardware — a laptop CPU, no GPU, no cloud. > **⚠️ Not a medical device.** This is a research and educational project. It is **not** FDA/CE cleared, has not been clinically validated, and must not be used to diagnose, treat, or make any decision about a real patient. Outputs are frequently wrong. See [Limitations](#limitations) — they are substantial and you should read them before using anything here. ## What's in this repo > **The X-ray and the symptoms go into one model, not two.** The fusion model is a single network that consumes the radiograph *and* the symptom text together and emits one set of logits — `fusion_full.onnx` is one graph with three inputs (`pixel_values`, `input_ids`, `attention_mask`). There is no separate image classifier and text classifier whose outputs get merged afterwards; the two modalities are fused inside the model, before the classifier head. The BLIP captioner below is a **separate, optional** model that only writes a text description of the image — it takes no symptom input and plays no part in the diagnosis path. | Component | Path | Size | What it does | |---|---|---|---| | **Fusion model** (ONNX, end-to-end) — *the main model* | `checkpoints/onnx_full/fusion_full.onnx` | 787 MB | X-ray **and** symptom text → diagnosis logits, in one graph. Runs with `onnxruntime` alone — no PyTorch. | | Fusion classifier head (PyTorch) | `checkpoints/fusion_model.pth` | 5.4 MB | Trained classifier head only; needs CLIP + Bio_ClinicalBERT at runtime. | | Fusion classifier head (ONNX) | `checkpoints/onnx/fusion_classifier.onnx` | 4.5 MB | Head-only ONNX; encoders still run in PyTorch. | | **BLIP X-ray captioner** | `blip-xray-finetuned/` | 896 MB | `Salesforce/blip-image-captioning-base` fine-tuned on IU-Xray reports → radiology-style caption. | | Default/demo classifier | `models/default/fusion_classifier.onnx` | 1.3 MB | 15 NIH classes, **random weights**. Ships so the app runs before training. Not predictive. | | Application code | `*.py`, `launch.*`, `config.json` | — | CLI, Gradio web UI, batch predictor, training and ONNX export scripts. | The training dataset is **not** included — see [Data](#data). ## Architecture **Fusion (Symptom Check)** — one multimodal classifier over both inputs. Both encoders feed a single shared head, so the prediction is a joint function of the image and the symptoms; neither modality is scored on its own: ``` image ──► CLIP ViT-B/32 vision tower ──► visual_projection ──► L2-normalize ──┐ ├─► concat ──► MLP classifier ──► logits symptom text ──► Bio_ClinicalBERT ──► mean-pool last_hidden_state ────────────┘ ``` Encoders are **frozen**; only the MLP head is trained. `fusion_full.onnx` bakes the whole graph — encoders included — into one file, which is why it is 787 MB. ONNX signature (opset 14, dynamic batch and sequence length): | | Name | Shape | dtype | |---|---|---|---| | in | `pixel_values` | `[batch, 3, 224, 224]` | float32 | | in | `input_ids` | `[batch, seq_len]` | int64 | | in | `attention_mask` | `[batch, seq_len]` | int64 | | out | `logits` | `[batch, 3018]` | float32 | Preprocess with `CLIPProcessor` (`openai/clip-vit-base-patch32`) for the image and `AutoTokenizer` (`emilyalsentzer/Bio_ClinicalBERT`) for the text. Class names are in `checkpoints/onnx_full/labels.json`, index-aligned to the logits. **Vision (captioning)** — a separate BLIP model, image-only, loadable with `BlipForConditionalGeneration.from_pretrained`. It does not see the symptoms and does not feed the fusion model; it exists to write a human-readable description alongside the diagnosis. ## Usage ### ONNX, no PyTorch ```python import json import numpy as np import onnxruntime as ort from PIL import Image from transformers import CLIPProcessor, AutoTokenizer from huggingface_hub import hf_hub_download repo = "GAD-Research-Lab/MedicalAI-Light-Weight" onnx_path = hf_hub_download(repo, "checkpoints/onnx_full/fusion_full.onnx") labels = json.load(open(hf_hub_download(repo, "checkpoints/onnx_full/labels.json"))) clip = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32") tok = AutoTokenizer.from_pretrained("emilyalsentzer/Bio_ClinicalBERT") image = Image.open("xray.jpg").convert("RGB") pixel_values = clip(images=image, return_tensors="np")["pixel_values"] text = tok("cough and fever", return_tensors="np", padding="max_length", truncation=True, max_length=64) sess = ort.InferenceSession(onnx_path, providers=["CPUExecutionProvider"]) logits = sess.run(["logits"], { "pixel_values": pixel_values.astype(np.float32), "input_ids": text["input_ids"].astype(np.int64), "attention_mask": text["attention_mask"].astype(np.int64), })[0] probs = np.exp(logits - logits.max()) / np.exp(logits - logits.max()).sum() top = probs[0].argmax() print(labels[top], float(probs[0][top])) ``` ### BLIP captioning ```python from transformers import BlipProcessor, BlipForConditionalGeneration from PIL import Image repo = "GAD-Research-Lab/MedicalAI-Light-Weight" processor = BlipProcessor.from_pretrained(repo, subfolder="blip-xray-finetuned") model = BlipForConditionalGeneration.from_pretrained(repo, subfolder="blip-xray-finetuned") inputs = processor(Image.open("xray.jpg").convert("RGB"), return_tensors="pt") print(processor.decode(model.generate(**inputs, max_new_tokens=64)[0], skip_special_tokens=True)) ``` ### Full application ```bash git clone https://huggingface.co/GAD-Research-Lab/MedicalAI-Light-Weight cd MedicalAI-Light-Weight pip install -r requirements.txt gradio python web_ui.py # http://127.0.0.1:7860 ``` Or `launch.ps1` (Windows) / `launch.sh` (Linux/macOS) to set up a venv and start the UI in one step. `python run.py` for the interactive CLI, `python batch_predict.py -o out.csv` for batch. ## Training - **Fusion head** — cross-entropy over the label set, 85/15 random train/val split, AdamW, frozen encoders. `python training.py --mode train --epochs 10 --batch_size 8`. - **BLIP** — fine-tuned on IU-Xray image/report pairs. `python xray_training.py --mode train --epochs 3 --batch_size 4 --max_samples 500`. Sources: IU-Xray (~6,687 rows), NIH Chest X-ray (~3,000 rows), plus 160 synthetic rare-finding rows from `expand_dataset.py` — ~9,847 total. ## Limitations Read these. They are not boilerplate. - **The label space is degenerate.** `labels.json` has **3,018 classes**, and most are not diagnoses — they are raw, deduplicated report strings scraped from IU-Xray, e.g. `"findings: . impression: 1. all lines and tubes in stable , xxxx position..."`. Only a handful (`atelectasis`, `cardiomegaly`, `consolidation`, `edema`, `effusion`, `emphysema`, `fibrosis`, …) are clean condition names. With ~9.8k training rows across 3,018 classes there are roughly **3 examples per class**, and the reported "confidence" is a softmax over that space — it is not calibrated and should not be read as a probability of disease. Treat the classifier as a demonstration of the architecture, not as a working diagnostic. - **No held-out evaluation is published.** Training reports validation *loss* only. There is no accuracy, AUROC, sensitivity/specificity, or per-class breakdown in this repo, and no evaluation on an external site or scanner. Nothing here supports a claim about real-world performance. - **Encoders are frozen and general-purpose.** CLIP ViT-B/32 was pretrained on web images, not radiographs. Only a small MLP adapts it to this domain. - **Narrow data.** Two US datasets, adult chest radiographs, frontal views, English free-text reports. Expect degradation on pediatric films, lateral views, other modalities, other populations, and other equipment. Known demographic and label-noise problems in IU-Xray and NIH ChestX-ray14 are inherited wholesale — NIH labels were themselves NLP-mined from reports and are noisy. - **BLIP captions are fluent, not faithful.** The captioner will produce confident, plausible, well-formed radiology prose for an image it has no ability to read correctly, including for non-X-ray inputs. Fluency here carries no signal about correctness. - **`models/default/` is random weights** by construction, so the app can start before training. Its predictions are noise. - **Automation bias is the main risk.** The most likely harm from this repo is a person believing a confident-looking output. Do not put it in front of patients or in any workflow where a wrong answer reaches one. ## Data The training data is **not redistributed here** — download it yourself with `python training.py --mode prepare-data`. - **IU-Xray** (Indiana University / Open-i, NLM) — de-identified, public. - **NIH ChestX-ray14** (NIH Clinical Center) — de-identified, public. Both are de-identified at source; no PHI is contained in this repo. The BLIP model was fine-tuned on IU-Xray report text and can emit dataset artifacts such as `xxxx` anonymization tokens. Check each dataset's own terms before redistributing derivatives. ## License Apache-2.0 for the code and the trained weights in this repo. Upstream components carry their own licenses — CLIP (MIT), BLIP (BSD-3-Clause), Bio_ClinicalBERT (MIT) — and the source datasets carry their own terms. The license permits use; it does not make the model safe or fit for clinical use. ## Citation ```bibtex @software{medicalai_light_weight, title = {MedicalAI — Light Weight: CPU-friendly chest X-ray analysis}, author = {GAD Research Lab}, year = {2026}, url = {https://huggingface.co/GAD-Research-Lab/MedicalAI-Light-Weight} } ```