Create README.md

README.md

@@ -0,0 +1,116 @@
+This repository contains a fine-tuned version of BiomedCLIP (specifically the PubMedBERT_256-vit_base_patch16_224 variant) using OpenCLIP. The model is trained to recognize and classify various medical images (e.g., chest X-rays, histopathology slides) in a zero-shot manner. It was further adapted on a subset of medical data (e.g., from the WinterSchool/MedificsDataset) to enhance performance on specific image classes.
+
+Model Details
+Architecture: Vision Transformer (ViT-B/16) + PubMedBERT-based text encoder, loaded through open_clip.
+Training Objective: CLIP-style contrastive learning to align medical text prompts with images.
+Fine-Tuned On: Selected medical images and text pairs, including X-rays, histopathology images, etc.
+Intended Use:
+Zero-shot classification of medical images (e.g., “This is a photo of a chest X-ray”).
+Exploratory research or educational demos showcasing multi-modal (image-text) alignment in the medical domain.
+Usage
+Below is a minimal Python snippet using OpenCLIP. Adjust the labels and text prompts as needed:
+
+python
+Copy
+import torch
+import open_clip
+from PIL import Image
+
+# 1) Load the fine-tuned model
+model, preprocess_train, preprocess_val = open_clip.create_model_and_transforms(
+    "hf-hub:your-username/OpenCLIP-BiomedCLIP-Finetuned", 
+    pretrained=None
+)
+tokenizer = open_clip.get_tokenizer("hf-hub:your-username/OpenCLIP-BiomedCLIP-Finetuned")
+
+device = "cuda" if torch.cuda.is_available() else "cpu"
+model.to(device)
+model.eval()
+
+# 2) Example labels
+labels = [
+    "chest X-ray", 
+    "brain MRI",
+    "bone X-ray",
+    "squamous cell carcinoma histopathology",
+    "adenocarcinoma histopathology",
+    "immunohistochemistry histopathology"
+]
+
+# 3) Load and preprocess an image
+image_path = "path/to/your_image.jpg"
+image = Image.open(image_path).convert("RGB")
+image_tensor = preprocess_val(image).unsqueeze(0).to(device)
+
+# 4) Create text prompts & tokenize
+text_prompts = [f"This is a photo of a {label}" for label in labels]
+tokens = tokenizer(text_prompts).to(device)
+
+# 5) Forward pass
+with torch.no_grad():
+    image_features = model.encode_image(image_tensor)
+    text_features = model.encode_text(tokens)
+    logit_scale = model.logit_scale.exp()
+    logits = (logit_scale * image_features @ text_features.t()).softmax(dim=-1)
+
+# 6) Get predictions
+probs = logits[0].cpu().tolist()
+for label, prob in zip(labels, probs):
+    print(f"{label}: {prob:.4f}")
+Example Gradio App
+You can also deploy a simple Gradio demo:
+
+python
+Copy
+import gradio as gr
+import torch
+import open_clip
+from PIL import Image
+
+model, preprocess_train, preprocess_val = open_clip.create_model_and_transforms(
+    "hf-hub:your-username/OpenCLIP-BiomedCLIP-Finetuned", 
+    pretrained=None
+)
+tokenizer = open_clip.get_tokenizer("hf-hub:your-username/OpenCLIP-BiomedCLIP-Finetuned")
+device = "cuda" if torch.cuda.is_available() else "cpu"
+model.to(device)
+model.eval()
+
+labels = ["chest X-ray", "brain MRI", "histopathology", "etc."]
+
+def classify_image(img):
+    if img is None:
+        return {}
+    image_tensor = preprocess_val(img).unsqueeze(0).to(device)
+    prompts = [f"This is a photo of a {label}" for label in labels]
+    tokens = tokenizer(prompts).to(device)
+    with torch.no_grad():
+        image_feats = model.encode_image(image_tensor)
+        text_feats = model.encode_text(tokens)
+        logit_scale = model.logit_scale.exp()
+        logits = (logit_scale * image_feats @ text_feats.T).softmax(dim=-1)
+    probs = logits.squeeze().cpu().numpy().tolist()
+    return {label: float(prob) for label, prob in zip(labels, probs)}
+
+demo = gr.Interface(fn=classify_image, inputs=gr.Image(type="pil"), outputs="label")
+demo.launch()
+Performance
+Accuracy: Varies based on your specific dataset. This model can effectively classify medical images like chest X-rays or histopathology slides, but performance depends heavily on fine-tuning data coverage.
+Potential Limitations:
+Ultrasound, CT, MRI or other modalities might not be recognized if not included in training data.
+The model may incorrectly label images that fall outside its known categories.
+Limitations & Caveats
+Not a Medical Device: This model is not FDA-approved or clinically validated. It’s intended for research and educational purposes only.
+Data Bias: If the training dataset lacked certain pathologies or modalities, the model may systematically misclassify them.
+Security: This model uses standard PyTorch and open_clip. Be mindful of potential vulnerabilities when loading models or code from untrusted sources.
+Privacy: If you use patient data, comply with local regulations (HIPAA, GDPR, etc.).
+Citation & Acknowledgements
+Base Model: BiomedCLIP by Microsoft
+OpenCLIP: GitHub – open_clip
+Fine-tuning dataset: WinterSchool/MedificsDataset
+If you use this model in your research or demos, please cite the above works accordingly.
+
+License
+[Specify your license here—e.g., MIT, Apache 2.0, or a custom license.]
+
+Note: Always include disclaimers that this model is not a substitute for professional medical advice and that it may not generalize to all imaging modalities or patient populations.