|
|
--- |
|
|
tags: |
|
|
- text-to-image |
|
|
- lora |
|
|
- diffusers |
|
|
- template:diffusion-lora |
|
|
widget: |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
- text: >- |
|
|
The CT scan of the brain shows no acute hemorrhage but reveals a small area |
|
|
of hypodensity in the right hemisphere, consistent with an old infarct. No |
|
|
new acute lesions are visible. |
|
|
output: |
|
|
url: images/NEO_049.jpg |
|
|
- text: >- |
|
|
The MRI shows mild edema in the soft tissue around the ankle joint with no |
|
|
significant ligament tears or cartilage damage. |
|
|
output: |
|
|
url: images/NEO_076.jpg |
|
|
- text: >- |
|
|
The chest X-ray shows consolidation in the right lower lobe with air |
|
|
bronchograms, indicative of lobar pneumonia. |
|
|
output: |
|
|
url: images/NEO_017.jpg |
|
|
base_model: stabilityai/stable-diffusion-2-1 |
|
|
instance_prompt: >- |
|
|
medical image generation, text-to-image, synthetic patient, radiology, CT |
|
|
scan, MRI, ultrasound image, chest x-ray, clinical image, radiograph, medical |
|
|
anomaly, patient scan |
|
|
license: mit |
|
|
--- |
|
|
# NeoPatient |
|
|
|
|
|
<Gallery /> |
|
|
|
|
|
|
|
|
## 🧠 NeoPatient - Synthetic Medical Image Generator (LoRA) |
|
|
|
|
|
**NeoPatient** is a Low-Rank Adaptation (LoRA) fine-tuned version of `stabilityai/stable-diffusion-2-1`, purpose-built for generating high-fidelity synthetic patient images. It is trained using the **ROCO (Radiology Objects in Context)** dataset, which contains a large collection of radiology images and captions from biomedical literature. NeoPatient enables privacy-preserving medical AI development through high-quality, domain-specific synthetic image generation. |
|
|
|
|
|
--- |
|
|
|
|
|
### 🔍 Use Case |
|
|
|
|
|
NeoPatient is designed for: |
|
|
|
|
|
* **Medical Data Augmentation**: Supplement training datasets for computer vision models in healthcare. |
|
|
* **Synthetic Dataset Generation**: Produce controlled, diverse imagery representing clinical contexts (e.g., CT scans, X-rays, ultrasound). |
|
|
* **Diagnostic Simulation**: Create visual scenarios for education, assessment, and reinforcement learning environments. |
|
|
* **Privacy-Safe AI**: Enable research and product development without exposure to real patient data. |
|
|
|
|
|
--- |
|
|
|
|
|
### 🏗️ Base Model |
|
|
|
|
|
* **Base**: `stabilityai/stable-diffusion-2-1` |
|
|
* **Fine-Tuning Method**: LoRA |
|
|
* **Fine-Tuning Dataset**: **ROCO** (Radiology Objects in Context), a large-scale medical image-text dataset from PubMed Central. |
|
|
* **LoRA Weights**: `pytorch_lora_weights.safetensors` (6.68 MB) |
|
|
|
|
|
--- |
|
|
|
|
|
### ⚙️ Technical Details |
|
|
|
|
|
* **Training Size**: 40,000 medical images with associated captions |
|
|
* **Image Types**: X-ray, CT, MRI, ultrasound, fluoroscopy, endoscopy, angiography |
|
|
* **Learning Objective**: Align latent representations of medical visual features with text prompts in the medical domain. |
|
|
* **LoRA Config**: Low-rank matrices injected into attention layers for efficient fine-tuning. |
|
|
|
|
|
--- |
|
|
|
|
|
### 📦 How to Use |
|
|
|
|
|
````python |
|
|
from diffusers import StableDiffusionPipeline |
|
|
import torch |
|
|
|
|
|
pipe = StableDiffusionPipeline.from_pretrained("stabilityai/stable-diffusion-2-1", torch_dtype=torch.float16).to("cuda") |
|
|
pipe.load_lora_weights("unicornftk/NeoPatient") |
|
|
pipe(prompt="Abdominal CT scan shows a dilated appendix measuring 9mm in diameter, with surrounding fat stranding indicative of acute appendicitis.").images[0].save("abdominal_ct.png") |
|
|
```` |
|
|
|
|
|
--- |
|
|
|
|
|
### 📜 License |
|
|
|
|
|
**MIT License** — free for research, academic, and commercial use with attribution. |
|
|
|
|
|
|
|
|
|
|
|
### ⚠️ Disclaimer |
|
|
|
|
|
NeoPatient generates **entirely synthetic** medical images. It is not intended for diagnostic use or clinical decision-making. Outputs do **not** represent real patients and should be used **only** for research, development, and educational purposes. |
|
|
|
|
|
|
|
|
--- |
|
|
|
|
|
|
|
|
## Trigger words |
|
|
|
|
|
* You should use `CT scan` to trigger the image generation. |
|
|
|
|
|
* You should use `MRI` to trigger the image generation. |
|
|
|
|
|
* You should use `Ultrasound` to trigger the image generation. |
|
|
|
|
|
* You should use `Chest X-ray` to trigger the image generation. |
|
|
|
|
|
* You should use `Fluoroscopy` to trigger the image generation. |
|
|
|
|
|
* You should use `Endoscopy` to trigger the image generation. |
|
|
|
|
|
* You should use `Angiography` to trigger the image generation. |
|
|
|
|
|
--- |
|
|
|
|
|
## Download model |
|
|
|
|
|
Weights for this model are available in Safetensors format. [Download](/unicornftk/NeoPatient/tree/main) them in the Files & versions tab. |
|
|
|
