| # BiomedCLIP MRI + Clinical Text Classifier | |
| This model fine-tunes [BiomedCLIP (PubMedBERT ViT-B/16)](https://huggingface.co/microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224) for **Alzheimer’s disease classification** from **MRI (3D volumes)** and **synthetic clinical text**. | |
| --- | |
| ## 🧩 Model Details | |
| - **Backbone**: BiomedCLIP (image + text encoders) | |
| - **Input MRI**: 3D NIfTI → reduced to 3 mid-slices (axial, coronal, sagittal) → stacked into RGB | |
| - **Input Text**: Synthetic patient note (tokenized with PubMedBERT) | |
| - **Fusion**: Concatenate image & text embeddings | |
| - **Head**: MLP (Linear → ReLU → Dropout → Linear) → 3-way classification | |
| - **Labels**: | |
| - `CN` – Cognitively Normal | |
| - `MCI` – Mild Cognitive Impairment | |
| - `Dementia` | |
| --- | |
| ## 🚀 Usage | |
| ### Install | |
| ```bash | |
| pip install open_clip_torch nibabel torch torchvision | |
| ##Load Pretrained Model | |
| ``` | |
| import torch | |
| from model import BiomedClipClassifier, predict_from_paths | |
| device = "cuda" if torch.cuda.is_available() else "cpu" | |
| # Load from repo (assuming you pushed pytorch_model.bin + config.json here) | |
| model = BiomedClipClassifier.from_pretrained(".", device=device) | |
| # Example inference | |
| pred, probs = predict_from_paths( | |
| model, | |
| "/path/to/sample_brain.nii.gz", | |
| "Patient shows mild memory impairment and hippocampal atrophy.", | |
| device=device | |
| ) | |
| print("Prediction:", pred) | |
| print("Probabilities:", probs) # [CN, MCI, Dementia] | |
| ``` | |
| ##Run Inference | |
| ``` | |
| python inference.py --weights . --mri /path/to/sample.nii.gz --text "Patient shows memory issues" | |
| ``` | |