Upload 5 files
Browse files- DOCUMENTATION.md +335 -0
- README.md +64 -0
- mediSync.log +80 -0
- requirements.txt +11 -0
- run.py +71 -0
DOCUMENTATION.md
ADDED
|
@@ -0,0 +1,335 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# MediSync: Multi-Modal Medical Analysis System
|
| 2 |
+
|
| 3 |
+
## Comprehensive Technical Documentation
|
| 4 |
+
|
| 5 |
+
### Table of Contents
|
| 6 |
+
1. [Introduction](#introduction)
|
| 7 |
+
2. [System Architecture](#system-architecture)
|
| 8 |
+
3. [Installation](#installation)
|
| 9 |
+
4. [Usage](#usage)
|
| 10 |
+
5. [Core Components](#core-components)
|
| 11 |
+
6. [Model Details](#model-details)
|
| 12 |
+
7. [API Reference](#api-reference)
|
| 13 |
+
8. [Extending the System](#extending-the-system)
|
| 14 |
+
9. [Troubleshooting](#troubleshooting)
|
| 15 |
+
10. [References](#references)
|
| 16 |
+
|
| 17 |
+
---
|
| 18 |
+
|
| 19 |
+
## Introduction
|
| 20 |
+
|
| 21 |
+
MediSync is a multi-modal AI system that combines X-ray image analysis with medical report text processing to provide comprehensive medical insights. By leveraging state-of-the-art deep learning models for both vision and language understanding, MediSync can:
|
| 22 |
+
|
| 23 |
+
- Analyze chest X-ray images to detect abnormalities
|
| 24 |
+
- Extract key clinical information from medical reports
|
| 25 |
+
- Fuse insights from both modalities for enhanced diagnosis support
|
| 26 |
+
- Provide comprehensive visualization of analysis results
|
| 27 |
+
|
| 28 |
+
This AI system demonstrates the power of multi-modal fusion in the healthcare domain, where integrating information from multiple sources can lead to more robust and accurate analyses.
|
| 29 |
+
|
| 30 |
+
## System Architecture
|
| 31 |
+
|
| 32 |
+
MediSync follows a modular architecture with three main components:
|
| 33 |
+
|
| 34 |
+
1. **Image Analysis Module**: Processes X-ray images using pre-trained vision models
|
| 35 |
+
2. **Text Analysis Module**: Analyzes medical reports using NLP models
|
| 36 |
+
3. **Multimodal Fusion Module**: Combines insights from both modalities
|
| 37 |
+
|
| 38 |
+
The system uses the following high-level workflow:
|
| 39 |
+
|
| 40 |
+
```
|
| 41 |
+
┌─────────────────┐
|
| 42 |
+
│ X-ray Image │
|
| 43 |
+
└────────┬────────┘
|
| 44 |
+
│
|
| 45 |
+
▼
|
| 46 |
+
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
|
| 47 |
+
│ Preprocessing │───▶│ Image Analysis │───▶│ │
|
| 48 |
+
└─────────────────┘ └─────────────────┘ │ │
|
| 49 |
+
│ Multimodal │
|
| 50 |
+
┌─────────────────┐ ┌─────────────────┐ │ Fusion │───▶ Results
|
| 51 |
+
│ Medical Report │───▶│ Text Analysis │───▶│ │
|
| 52 |
+
└─────────────────┘ └─────────────────┘ │ │
|
| 53 |
+
└─────────────────┘
|
| 54 |
+
```
|
| 55 |
+
|
| 56 |
+
## Installation
|
| 57 |
+
|
| 58 |
+
### Prerequisites
|
| 59 |
+
- Python 3.8 or higher
|
| 60 |
+
- Pip package manager
|
| 61 |
+
|
| 62 |
+
### Setup Instructions
|
| 63 |
+
|
| 64 |
+
1. Clone the repository:
|
| 65 |
+
```bash
|
| 66 |
+
git clone [repository-url]
|
| 67 |
+
cd mediSync
|
| 68 |
+
```
|
| 69 |
+
|
| 70 |
+
2. Install dependencies:
|
| 71 |
+
```bash
|
| 72 |
+
pip install -r requirements.txt
|
| 73 |
+
```
|
| 74 |
+
|
| 75 |
+
3. Download sample data:
|
| 76 |
+
```bash
|
| 77 |
+
python -m mediSync.utils.download_samples
|
| 78 |
+
```
|
| 79 |
+
|
| 80 |
+
## Usage
|
| 81 |
+
|
| 82 |
+
### Running the Application
|
| 83 |
+
|
| 84 |
+
To launch the MediSync application with the Gradio interface:
|
| 85 |
+
|
| 86 |
+
```bash
|
| 87 |
+
python run.py
|
| 88 |
+
```
|
| 89 |
+
|
| 90 |
+
This will:
|
| 91 |
+
1. Download sample data if not already present
|
| 92 |
+
2. Initialize the application
|
| 93 |
+
3. Launch the Gradio web interface
|
| 94 |
+
|
| 95 |
+
### Web Interface
|
| 96 |
+
|
| 97 |
+
MediSync provides a user-friendly web interface with three main tabs:
|
| 98 |
+
|
| 99 |
+
1. **Multimodal Analysis**: Upload an X-ray image and enter a medical report for combined analysis
|
| 100 |
+
2. **Image Analysis**: Upload an X-ray image for image-only analysis
|
| 101 |
+
3. **Text Analysis**: Enter a medical report for text-only analysis
|
| 102 |
+
|
| 103 |
+
### Command Line Usage
|
| 104 |
+
|
| 105 |
+
You can also use the core components directly from Python:
|
| 106 |
+
|
| 107 |
+
```python
|
| 108 |
+
from mediSync.models import XRayImageAnalyzer, MedicalReportAnalyzer, MultimodalFusion
|
| 109 |
+
|
| 110 |
+
# Initialize models
|
| 111 |
+
fusion_model = MultimodalFusion()
|
| 112 |
+
|
| 113 |
+
# Analyze image and text
|
| 114 |
+
results = fusion_model.analyze("path/to/image.jpg", "Medical report text...")
|
| 115 |
+
|
| 116 |
+
# Get explanation
|
| 117 |
+
explanation = fusion_model.get_explanation(results)
|
| 118 |
+
print(explanation)
|
| 119 |
+
```
|
| 120 |
+
|
| 121 |
+
## Core Components
|
| 122 |
+
|
| 123 |
+
### Image Analysis Module
|
| 124 |
+
|
| 125 |
+
The `XRayImageAnalyzer` class is responsible for analyzing X-ray images:
|
| 126 |
+
|
| 127 |
+
- Uses the DeiT (Data-efficient image Transformers) model fine-tuned on chest X-rays
|
| 128 |
+
- Detects abnormalities and classifies findings
|
| 129 |
+
- Provides confidence scores and primary findings
|
| 130 |
+
|
| 131 |
+
Key methods:
|
| 132 |
+
- `analyze(image_path)`: Analyzes an X-ray image
|
| 133 |
+
- `get_explanation(results)`: Generates a human-readable explanation
|
| 134 |
+
|
| 135 |
+
### Text Analysis Module
|
| 136 |
+
|
| 137 |
+
The `MedicalReportAnalyzer` class processes medical report text:
|
| 138 |
+
|
| 139 |
+
- Extracts medical entities (conditions, treatments, tests)
|
| 140 |
+
- Assesses severity level
|
| 141 |
+
- Extracts key findings
|
| 142 |
+
- Suggests follow-up actions
|
| 143 |
+
|
| 144 |
+
Key methods:
|
| 145 |
+
- `extract_entities(text)`: Extracts medical entities
|
| 146 |
+
- `assess_severity(text)`: Determines severity level
|
| 147 |
+
- `extract_findings(text)`: Extracts key clinical findings
|
| 148 |
+
- `suggest_followup(text, entities, severity)`: Suggests follow-up actions
|
| 149 |
+
- `analyze(text)`: Performs comprehensive analysis
|
| 150 |
+
|
| 151 |
+
### Multimodal Fusion Module
|
| 152 |
+
|
| 153 |
+
The `MultimodalFusion` class combines insights from both modalities:
|
| 154 |
+
|
| 155 |
+
- Calculates agreement between image and text analyses
|
| 156 |
+
- Determines confidence-weighted findings
|
| 157 |
+
- Provides comprehensive severity assessment
|
| 158 |
+
- Merges follow-up recommendations
|
| 159 |
+
|
| 160 |
+
Key methods:
|
| 161 |
+
- `analyze_image(image_path)`: Analyzes image only
|
| 162 |
+
- `analyze_text(text)`: Analyzes text only
|
| 163 |
+
- `analyze(image_path, report_text)`: Performs multimodal analysis
|
| 164 |
+
- `get_explanation(fused_results)`: Generates comprehensive explanation
|
| 165 |
+
|
| 166 |
+
## Model Details
|
| 167 |
+
|
| 168 |
+
### X-ray Analysis Model
|
| 169 |
+
|
| 170 |
+
- **Model**: facebook/deit-base-patch16-224-medical-cxr
|
| 171 |
+
- **Architecture**: Data-efficient image Transformer (DeiT)
|
| 172 |
+
- **Training Data**: Chest X-ray datasets
|
| 173 |
+
- **Input Size**: 224x224 pixels
|
| 174 |
+
- **Output**: Classification probabilities for various conditions
|
| 175 |
+
|
| 176 |
+
### Medical Text Analysis Models
|
| 177 |
+
|
| 178 |
+
- **Entity Recognition Model**: samrawal/bert-base-uncased_medical-ner
|
| 179 |
+
- **Classification Model**: medicalai/ClinicalBERT
|
| 180 |
+
- **Architecture**: BERT-based transformer models
|
| 181 |
+
- **Training Data**: Medical text and reports
|
| 182 |
+
|
| 183 |
+
## API Reference
|
| 184 |
+
|
| 185 |
+
### XRayImageAnalyzer
|
| 186 |
+
|
| 187 |
+
```python
|
| 188 |
+
from mediSync.models import XRayImageAnalyzer
|
| 189 |
+
|
| 190 |
+
# Initialize
|
| 191 |
+
analyzer = XRayImageAnalyzer(model_name="facebook/deit-base-patch16-224-medical-cxr")
|
| 192 |
+
|
| 193 |
+
# Analyze image
|
| 194 |
+
results = analyzer.analyze("path/to/image.jpg")
|
| 195 |
+
|
| 196 |
+
# Get explanation
|
| 197 |
+
explanation = analyzer.get_explanation(results)
|
| 198 |
+
```
|
| 199 |
+
|
| 200 |
+
### MedicalReportAnalyzer
|
| 201 |
+
|
| 202 |
+
```python
|
| 203 |
+
from mediSync.models import MedicalReportAnalyzer
|
| 204 |
+
|
| 205 |
+
# Initialize
|
| 206 |
+
analyzer = MedicalReportAnalyzer()
|
| 207 |
+
|
| 208 |
+
# Analyze report
|
| 209 |
+
results = analyzer.analyze("Medical report text...")
|
| 210 |
+
|
| 211 |
+
# Access specific components
|
| 212 |
+
entities = results["entities"]
|
| 213 |
+
severity = results["severity"]
|
| 214 |
+
findings = results["findings"]
|
| 215 |
+
recommendations = results["followup_recommendations"]
|
| 216 |
+
```
|
| 217 |
+
|
| 218 |
+
### MultimodalFusion
|
| 219 |
+
|
| 220 |
+
```python
|
| 221 |
+
from mediSync.models import MultimodalFusion
|
| 222 |
+
|
| 223 |
+
# Initialize
|
| 224 |
+
fusion = MultimodalFusion()
|
| 225 |
+
|
| 226 |
+
# Multimodal analysis
|
| 227 |
+
results = fusion.analyze("path/to/image.jpg", "Medical report text...")
|
| 228 |
+
|
| 229 |
+
# Get explanation
|
| 230 |
+
explanation = fusion.get_explanation(results)
|
| 231 |
+
```
|
| 232 |
+
|
| 233 |
+
## Extending the System
|
| 234 |
+
|
| 235 |
+
### Adding New Models
|
| 236 |
+
|
| 237 |
+
To add a new image analysis model:
|
| 238 |
+
|
| 239 |
+
1. Create a new class that follows the same interface as `XRayImageAnalyzer`
|
| 240 |
+
2. Update the `MultimodalFusion` class to use your new model
|
| 241 |
+
|
| 242 |
+
```python
|
| 243 |
+
class NewXRayModel:
|
| 244 |
+
def __init__(self, model_name, device=None):
|
| 245 |
+
# Initialize your model
|
| 246 |
+
pass
|
| 247 |
+
|
| 248 |
+
def analyze(self, image_path):
|
| 249 |
+
# Implement analysis logic
|
| 250 |
+
return results
|
| 251 |
+
|
| 252 |
+
def get_explanation(self, results):
|
| 253 |
+
# Generate explanation
|
| 254 |
+
return explanation
|
| 255 |
+
```
|
| 256 |
+
|
| 257 |
+
### Custom Preprocessing
|
| 258 |
+
|
| 259 |
+
You can extend the preprocessing utilities in `utils/preprocessing.py` for custom data preparation:
|
| 260 |
+
|
| 261 |
+
```python
|
| 262 |
+
def my_custom_preprocessor(image_path, **kwargs):
|
| 263 |
+
# Implement custom preprocessing
|
| 264 |
+
return processed_image
|
| 265 |
+
```
|
| 266 |
+
|
| 267 |
+
### Visualization Extensions
|
| 268 |
+
|
| 269 |
+
To add new visualization options, extend the utilities in `utils/visualization.py`:
|
| 270 |
+
|
| 271 |
+
```python
|
| 272 |
+
def my_custom_visualization(results, **kwargs):
|
| 273 |
+
# Create custom visualization
|
| 274 |
+
return figure
|
| 275 |
+
```
|
| 276 |
+
|
| 277 |
+
## Troubleshooting
|
| 278 |
+
|
| 279 |
+
### Common Issues
|
| 280 |
+
|
| 281 |
+
1. **Model Loading Errors**
|
| 282 |
+
- Ensure you have a stable internet connection for downloading models
|
| 283 |
+
- Check that you have sufficient disk space
|
| 284 |
+
- Try specifying a different model checkpoint
|
| 285 |
+
|
| 286 |
+
2. **Image Processing Errors**
|
| 287 |
+
- Ensure images are in a supported format (JPEG, PNG)
|
| 288 |
+
- Check that the image is a valid X-ray image
|
| 289 |
+
- Try preprocessing the image manually using the utility functions
|
| 290 |
+
|
| 291 |
+
3. **Performance Issues**
|
| 292 |
+
- For faster inference, use a GPU if available
|
| 293 |
+
- Reduce image resolution if processing is too slow
|
| 294 |
+
- Use the text-only analysis for quicker results
|
| 295 |
+
|
| 296 |
+
### Logging
|
| 297 |
+
|
| 298 |
+
MediSync uses Python's logging module for debug information:
|
| 299 |
+
|
| 300 |
+
```python
|
| 301 |
+
import logging
|
| 302 |
+
logging.basicConfig(level=logging.DEBUG)
|
| 303 |
+
```
|
| 304 |
+
|
| 305 |
+
Log files are saved to `mediSync.log` in the application directory.
|
| 306 |
+
|
| 307 |
+
## References
|
| 308 |
+
|
| 309 |
+
### Datasets
|
| 310 |
+
|
| 311 |
+
- [MIMIC-CXR](https://physionet.org/content/mimic-cxr/2.0.0/): Large dataset of chest radiographs with reports
|
| 312 |
+
- [ChestX-ray14](https://www.nih.gov/news-events/news-releases/nih-clinical-center-provides-one-largest-publicly-available-chest-x-ray-datasets-scientific-community): NIH dataset of chest X-rays
|
| 313 |
+
|
| 314 |
+
### Papers
|
| 315 |
+
|
| 316 |
+
- He, K., et al. (2020). "Vision Transformers for Medical Image Analysis"
|
| 317 |
+
- Irvin, J., et al. (2019). "CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison"
|
| 318 |
+
- Johnson, A.E.W., et al. (2019). "MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs"
|
| 319 |
+
|
| 320 |
+
### Tools and Libraries
|
| 321 |
+
|
| 322 |
+
- [Hugging Face Transformers](https://huggingface.co/docs/transformers/index)
|
| 323 |
+
- [PyTorch](https://pytorch.org/)
|
| 324 |
+
- [Gradio](https://gradio.app/)
|
| 325 |
+
|
| 326 |
+
---
|
| 327 |
+
|
| 328 |
+
## License
|
| 329 |
+
|
| 330 |
+
This project is licensed under the MIT License - see the LICENSE file for details.
|
| 331 |
+
|
| 332 |
+
## Acknowledgments
|
| 333 |
+
|
| 334 |
+
- The development of MediSync was inspired by recent advances in multi-modal learning in healthcare.
|
| 335 |
+
- Special thanks to the open-source community for providing pre-trained models and tools.
|
README.md
ADDED
|
@@ -0,0 +1,64 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# MediSync: Multi-Modal Medical Analysis System
|
| 2 |
+
|
| 3 |
+
MediSync is an AI-powered healthcare solution that combines X-ray image analysis with patient report text processing to provide comprehensive medical insights.
|
| 4 |
+
|
| 5 |
+
## Features
|
| 6 |
+
|
| 7 |
+
- **X-ray Image Analysis**: Detects abnormalities in chest X-rays using pre-trained vision models from Hugging Face.
|
| 8 |
+
- **Medical Report Processing**: Extracts key information from patient reports using NLP models.
|
| 9 |
+
- **Multi-modal Integration**: Combines insights from both image and text data for more accurate diagnosis suggestions.
|
| 10 |
+
- **User-friendly Interface**: Simple web interface for uploading images and reports.
|
| 11 |
+
|
| 12 |
+
## Project Structure
|
| 13 |
+
|
| 14 |
+
```
|
| 15 |
+
mediSync/
|
| 16 |
+
├── app.py # Main application with Gradio interface
|
| 17 |
+
├── models/
|
| 18 |
+
│ ├── image_analyzer.py # X-ray image analysis module
|
| 19 |
+
│ ├── text_analyzer.py # Medical report text analysis module
|
| 20 |
+
│ └── multimodal_fusion.py # Fusion of image and text insights
|
| 21 |
+
├── utils/
|
| 22 |
+
│ ├── preprocessing.py # Data preprocessing utilities
|
| 23 |
+
│ └── visualization.py # Result visualization utilities
|
| 24 |
+
├── data/
|
| 25 |
+
│ └── sample/ # Sample data for testing
|
| 26 |
+
└── tests/ # Unit tests
|
| 27 |
+
```
|
| 28 |
+
|
| 29 |
+
## Setup Instructions
|
| 30 |
+
|
| 31 |
+
1. Clone this repository:
|
| 32 |
+
```bash
|
| 33 |
+
git clone [repository-url]
|
| 34 |
+
cd MediSync
|
| 35 |
+
```
|
| 36 |
+
|
| 37 |
+
2. Install dependencies:
|
| 38 |
+
```bash
|
| 39 |
+
pip install -r requirements.txt
|
| 40 |
+
```
|
| 41 |
+
|
| 42 |
+
3. Run the application:
|
| 43 |
+
```bash
|
| 44 |
+
python app.py
|
| 45 |
+
```
|
| 46 |
+
|
| 47 |
+
4. Access the web interface at `http://localhost:7860`
|
| 48 |
+
|
| 49 |
+
## Models Used
|
| 50 |
+
|
| 51 |
+
- **X-ray Analysis**: facebook/deit-base-patch16-224-medical-cxr
|
| 52 |
+
- **Medical Text Analysis**: medicalai/ClinicalBERT
|
| 53 |
+
- **Additional Support Models**: Medical question answering and entity recognition models
|
| 54 |
+
|
| 55 |
+
## Use Cases
|
| 56 |
+
|
| 57 |
+
- Preliminary screening of chest X-rays
|
| 58 |
+
- Cross-validation of radiologist reports
|
| 59 |
+
- Educational tool for medical students
|
| 60 |
+
- Research tool for studying correlation between visual findings and written reports
|
| 61 |
+
|
| 62 |
+
## Note
|
| 63 |
+
|
| 64 |
+
This system is designed as a support tool and should not replace professional medical diagnosis. Always consult with healthcare professionals for medical decisions.
|
mediSync.log
ADDED
|
@@ -0,0 +1,80 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
2025-03-10 22:44:41,854 - __main__ - INFO - Starting MediSync setup
|
| 2 |
+
2025-03-10 22:44:41,854 - __main__ - INFO - Checking for sample data
|
| 3 |
+
2025-03-10 22:44:42,626 - __main__ - INFO - No sample images found. Downloading...
|
| 4 |
+
2025-03-10 22:44:42,626 - mediSync.utils.download_samples - INFO - Downloading https://prod-images-static.radiopaedia.org/images/53448173/322830a37f0fa0852773ca2db3e8d8_big_gallery.jpeg to D:\Multi Model AI Problem Solving\mediSync\data\sample\normal_chest_xray.jpg
|
| 5 |
+
2025-03-10 22:44:43,532 - mediSync.utils.download_samples - ERROR - Error downloading https://prod-images-static.radiopaedia.org/images/53448173/322830a37f0fa0852773ca2db3e8d8_big_gallery.jpeg: HTTP Error 404: Not Found
|
| 6 |
+
2025-03-10 22:44:43,532 - mediSync.utils.download_samples - INFO - Downloading https://prod-images-static.radiopaedia.org/images/52465460/e4d8791bd7502ab72af8d9e5c322db_big_gallery.jpg to D:\Multi Model AI Problem Solving\mediSync\data\sample\pneumonia_xray.jpg
|
| 7 |
+
2025-03-10 22:44:43,988 - mediSync.utils.download_samples - ERROR - Error downloading https://prod-images-static.radiopaedia.org/images/52465460/e4d8791bd7502ab72af8d9e5c322db_big_gallery.jpg: HTTP Error 404: Not Found
|
| 8 |
+
2025-03-10 22:44:43,988 - mediSync.utils.download_samples - INFO - Downloading https://prod-images-static.radiopaedia.org/images/556520/cf17c05750adb04b2a6e23afb47c7d_big_gallery.jpg to D:\Multi Model AI Problem Solving\mediSync\data\sample\cardiomegaly_xray.jpg
|
| 9 |
+
2025-03-10 22:44:44,605 - mediSync.utils.download_samples - ERROR - Error downloading https://prod-images-static.radiopaedia.org/images/556520/cf17c05750adb04b2a6e23afb47c7d_big_gallery.jpg: HTTP Error 404: Not Found
|
| 10 |
+
2025-03-10 22:44:44,605 - mediSync.utils.download_samples - INFO - Downloading https://prod-images-static.radiopaedia.org/images/19972291/41eed1a2cdad06d26c3f415a6ed65a_big_gallery.jpeg to D:\Multi Model AI Problem Solving\mediSync\data\sample\nodule_xray.jpg
|
| 11 |
+
2025-03-10 22:44:45,107 - mediSync.utils.download_samples - ERROR - Error downloading https://prod-images-static.radiopaedia.org/images/19972291/41eed1a2cdad06d26c3f415a6ed65a_big_gallery.jpeg: HTTP Error 404: Not Found
|
| 12 |
+
2025-03-10 22:44:45,107 - mediSync.utils.download_samples - INFO - Downloaded 0 out of 4 images
|
| 13 |
+
2025-03-10 22:44:45,107 - mediSync.utils.download_samples - INFO - Created sample info file: D:\Multi Model AI Problem Solving\mediSync\data\sample\sample_info.txt
|
| 14 |
+
2025-03-10 22:44:45,107 - __main__ - INFO - Launching MediSync application
|
| 15 |
+
2025-03-10 22:44:48,918 - httpx - INFO - HTTP Request: GET https://api.gradio.app/gradio-messaging/en "HTTP/1.1 200 OK"
|
| 16 |
+
2025-03-10 22:45:02,402 - mediSync.app - INFO - Initializing MediSync application
|
| 17 |
+
2025-03-10 22:45:03,005 - httpx - INFO - HTTP Request: GET https://api.gradio.app/pkg-version "HTTP/1.1 200 OK"
|
| 18 |
+
2025-03-10 22:45:03,294 - httpx - INFO - HTTP Request: GET http://127.0.0.1:7860/gradio_api/startup-events "HTTP/1.1 200 OK"
|
| 19 |
+
2025-03-10 22:45:03,722 - httpx - INFO - HTTP Request: HEAD http://127.0.0.1:7860/ "HTTP/1.1 200 OK"
|
| 20 |
+
2025-03-10 22:46:47,162 - mediSync.app - INFO - Loading models...
|
| 21 |
+
2025-03-10 22:46:47,162 - models.multimodal_fusion - INFO - Using device: cpu
|
| 22 |
+
2025-03-10 22:46:47,178 - models.image_analyzer - INFO - Using device: cpu
|
| 23 |
+
2025-03-10 22:46:47,391 - models.image_analyzer - ERROR - Failed to load model: facebook/deit-base-patch16-224-medical-cxr is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models'
|
| 24 |
+
If this is a private repository, make sure to pass a token having permission to this repo either by logging in with `huggingface-cli login` or by passing `token=<your_token>`
|
| 25 |
+
2025-03-10 22:46:47,391 - models.multimodal_fusion - ERROR - Failed to initialize image analyzer: facebook/deit-base-patch16-224-medical-cxr is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models'
|
| 26 |
+
If this is a private repository, make sure to pass a token having permission to this repo either by logging in with `huggingface-cli login` or by passing `token=<your_token>`
|
| 27 |
+
2025-03-10 22:46:47,391 - models.text_analyzer - INFO - Using device: cpu
|
| 28 |
+
2025-03-10 22:46:47,490 - models.text_analyzer - ERROR - Failed to load NER model: samrawal/bert-base-uncased_medical-ner is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models'
|
| 29 |
+
If this is a private repository, make sure to pass a token having permission to this repo either by logging in with `huggingface-cli login` or by passing `token=<your_token>`
|
| 30 |
+
2025-03-10 22:46:58,283 - models.text_analyzer - INFO - Successfully loaded classifier model: medicalai/ClinicalBERT
|
| 31 |
+
2025-03-10 22:46:58,283 - models.multimodal_fusion - INFO - Successfully initialized text analyzer
|
| 32 |
+
2025-03-10 22:46:58,283 - mediSync.app - INFO - Models loaded successfully
|
| 33 |
+
2025-03-10 22:46:58,379 - mediSync.app - INFO - Performing multimodal analysis
|
| 34 |
+
2025-03-10 22:46:58,379 - models.multimodal_fusion - WARNING - Image analyzer not available
|
| 35 |
+
2025-03-10 22:46:58,379 - models.text_analyzer - WARNING - NER model not available
|
| 36 |
+
2025-03-10 22:46:58,379 - models.multimodal_fusion - INFO - Agreement score between modalities: 0.25
|
| 37 |
+
2025-03-10 22:47:44,547 - mediSync.app - INFO - Enhancing image: C:\Users\amart\AppData\Local\Temp\tmp19zqepvi\upload.png
|
| 38 |
+
2025-03-10 22:47:55,128 - mediSync.app - INFO - Analyzing medical report text
|
| 39 |
+
2025-03-10 22:47:55,128 - models.text_analyzer - WARNING - NER model not available
|
| 40 |
+
2025-03-10 22:48:20,220 - mediSync.app - INFO - Enhancing image: C:\Users\amart\AppData\Local\Temp\tmpowd6qwty\upload.png
|
| 41 |
+
2025-03-10 22:55:22,019 - mediSync.app - INFO - Performing multimodal analysis
|
| 42 |
+
2025-03-10 22:55:22,019 - models.multimodal_fusion - WARNING - Image analyzer not available
|
| 43 |
+
2025-03-10 22:55:22,025 - models.text_analyzer - WARNING - NER model not available
|
| 44 |
+
2025-03-10 22:55:22,025 - models.multimodal_fusion - INFO - Agreement score between modalities: 0.25
|
| 45 |
+
2025-03-10 22:55:35,606 - mediSync.app - INFO - Enhancing image: C:\Users\amart\AppData\Local\Temp\tmpipzmsxch\upload.png
|
| 46 |
+
2025-03-10 22:55:45,116 - mediSync.app - INFO - Analyzing medical report text
|
| 47 |
+
2025-03-10 22:55:45,116 - models.text_analyzer - WARNING - NER model not available
|
| 48 |
+
2025-03-10 23:01:32,009 - mediSync.app - INFO - Enhancing image: C:\Users\amart\AppData\Local\Temp\tmpo88f1uaw\upload.png
|
| 49 |
+
2025-03-10 23:05:24,892 - mediSync.app - INFO - Enhancing image: C:\Users\amart\AppData\Local\Temp\tmpnckk4w27\upload.png
|
| 50 |
+
2025-03-10 23:05:45,332 - mediSync.app - INFO - Enhancing image: C:\Users\amart\AppData\Local\Temp\tmpxab_m30m\upload.png
|
| 51 |
+
2025-03-10 23:06:01,387 - mediSync.app - INFO - Enhancing image: C:\Users\amart\AppData\Local\Temp\tmpcit78i2e\upload.png
|
| 52 |
+
2025-03-10 23:06:04,157 - mediSync.app - INFO - Enhancing image: C:\Users\amart\AppData\Local\Temp\tmpivli_abx\upload.png
|
| 53 |
+
2025-03-10 23:11:47,429 - mediSync.app - INFO - Enhancing image: C:\Users\amart\AppData\Local\Temp\tmp4cpwvo5h\upload.png
|
| 54 |
+
2025-03-10 23:22:27,118 - mediSync.app - INFO - Performing multimodal analysis
|
| 55 |
+
2025-03-10 23:22:27,118 - models.multimodal_fusion - WARNING - Image analyzer not available
|
| 56 |
+
2025-03-10 23:22:27,118 - models.text_analyzer - WARNING - NER model not available
|
| 57 |
+
2025-03-10 23:22:27,118 - models.multimodal_fusion - INFO - Agreement score between modalities: 0.25
|
| 58 |
+
2025-03-10 23:24:22,378 - mediSync.app - INFO - Performing multimodal analysis
|
| 59 |
+
2025-03-10 23:24:22,378 - models.multimodal_fusion - WARNING - Image analyzer not available
|
| 60 |
+
2025-03-10 23:24:22,378 - models.text_analyzer - WARNING - NER model not available
|
| 61 |
+
2025-03-10 23:24:22,378 - models.multimodal_fusion - INFO - Agreement score between modalities: 0.25
|
| 62 |
+
2025-03-10 23:24:35,252 - mediSync.app - INFO - Enhancing image: C:\Users\amart\AppData\Local\Temp\tmp947ie5e1\upload.png
|
| 63 |
+
2025-03-10 23:24:37,554 - mediSync.app - INFO - Performing multimodal analysis
|
| 64 |
+
2025-03-10 23:24:37,554 - models.multimodal_fusion - WARNING - Image analyzer not available
|
| 65 |
+
2025-03-10 23:24:37,554 - models.text_analyzer - WARNING - NER model not available
|
| 66 |
+
2025-03-10 23:24:37,554 - models.multimodal_fusion - INFO - Agreement score between modalities: 0.25
|
| 67 |
+
2025-03-10 23:24:58,770 - mediSync.app - INFO - Analyzing medical report text
|
| 68 |
+
2025-03-10 23:24:58,770 - models.text_analyzer - WARNING - NER model not available
|
| 69 |
+
2025-03-10 23:25:25,652 - mediSync.app - INFO - Analyzing medical report text
|
| 70 |
+
2025-03-10 23:25:25,652 - models.text_analyzer - WARNING - NER model not available
|
| 71 |
+
2025-03-10 23:27:06,100 - mediSync.app - INFO - Performing multimodal analysis
|
| 72 |
+
2025-03-10 23:27:06,100 - models.multimodal_fusion - WARNING - Image analyzer not available
|
| 73 |
+
2025-03-10 23:27:06,100 - models.text_analyzer - WARNING - NER model not available
|
| 74 |
+
2025-03-10 23:27:06,100 - models.multimodal_fusion - INFO - Agreement score between modalities: 0.25
|
| 75 |
+
2025-03-10 23:27:46,183 - mediSync.app - INFO - Performing multimodal analysis
|
| 76 |
+
2025-03-10 23:27:46,183 - models.multimodal_fusion - WARNING - Image analyzer not available
|
| 77 |
+
2025-03-10 23:27:46,183 - models.text_analyzer - WARNING - NER model not available
|
| 78 |
+
2025-03-10 23:27:46,183 - models.multimodal_fusion - INFO - Agreement score between modalities: 0.25
|
| 79 |
+
2025-03-10 23:28:15,394 - mediSync.app - INFO - Analyzing medical report text
|
| 80 |
+
2025-03-10 23:28:15,394 - models.text_analyzer - WARNING - NER model not available
|
requirements.txt
ADDED
|
@@ -0,0 +1,11 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
torch>=2.0.0
|
| 2 |
+
transformers>=4.30.0
|
| 3 |
+
datasets>=2.12.0
|
| 4 |
+
pillow>=9.0.0
|
| 5 |
+
matplotlib>=3.7.0
|
| 6 |
+
pandas>=2.0.0
|
| 7 |
+
numpy>=1.24.0
|
| 8 |
+
scikit-learn>=1.2.0
|
| 9 |
+
gradio>=4.0.0
|
| 10 |
+
huggingface_hub>=0.16.0
|
| 11 |
+
opencv-python>=4.7.0
|
run.py
ADDED
|
@@ -0,0 +1,71 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
MediSync: Multi-Modal Medical Analysis System - Runner Script
|
| 3 |
+
============================================================
|
| 4 |
+
This script downloads sample data and launches the MediSync application.
|
| 5 |
+
"""
|
| 6 |
+
|
| 7 |
+
import logging
|
| 8 |
+
import sys
|
| 9 |
+
from pathlib import Path
|
| 10 |
+
|
| 11 |
+
# Set up logging
|
| 12 |
+
logging.basicConfig(
|
| 13 |
+
level=logging.INFO,
|
| 14 |
+
format="%(asctime)s - %(name)s - %(levelname)s - %(message)s",
|
| 15 |
+
handlers=[logging.StreamHandler(), logging.FileHandler("mediSync.log")],
|
| 16 |
+
)
|
| 17 |
+
logger = logging.getLogger(__name__)
|
| 18 |
+
|
| 19 |
+
|
| 20 |
+
def main():
|
| 21 |
+
"""
|
| 22 |
+
Main function to set up and run the MediSync application.
|
| 23 |
+
"""
|
| 24 |
+
logger.info("Starting MediSync setup")
|
| 25 |
+
|
| 26 |
+
# Add the current directory to Python path
|
| 27 |
+
current_dir = Path(__file__).resolve().parent
|
| 28 |
+
sys.path.append(str(current_dir))
|
| 29 |
+
|
| 30 |
+
# Download sample data if needed
|
| 31 |
+
try:
|
| 32 |
+
logger.info("Checking for sample data")
|
| 33 |
+
from mediSync.utils.download_samples import (
|
| 34 |
+
create_sample_info_file,
|
| 35 |
+
download_sample_images,
|
| 36 |
+
)
|
| 37 |
+
|
| 38 |
+
# Check if sample directory exists and contains images
|
| 39 |
+
sample_dir = current_dir / "mediSync" / "data" / "sample"
|
| 40 |
+
sample_dir.mkdir(parents=True, exist_ok=True)
|
| 41 |
+
|
| 42 |
+
image_files = list(sample_dir.glob("*.jpg")) + list(sample_dir.glob("*.png"))
|
| 43 |
+
|
| 44 |
+
if not image_files:
|
| 45 |
+
logger.info("No sample images found. Downloading...")
|
| 46 |
+
download_sample_images()
|
| 47 |
+
create_sample_info_file()
|
| 48 |
+
else:
|
| 49 |
+
logger.info(f"Found {len(image_files)} existing sample images")
|
| 50 |
+
|
| 51 |
+
except Exception as e:
|
| 52 |
+
logger.error(f"Error setting up sample data: {e}")
|
| 53 |
+
print(f"Warning: Could not set up sample data: {e}")
|
| 54 |
+
|
| 55 |
+
# Launch the application
|
| 56 |
+
try:
|
| 57 |
+
logger.info("Launching MediSync application")
|
| 58 |
+
from mediSync.app import create_interface
|
| 59 |
+
|
| 60 |
+
create_interface()
|
| 61 |
+
|
| 62 |
+
except Exception as e:
|
| 63 |
+
logger.error(f"Error launching application: {e}")
|
| 64 |
+
print(f"Error: Failed to launch MediSync application: {e}")
|
| 65 |
+
return 1
|
| 66 |
+
|
| 67 |
+
return 0
|
| 68 |
+
|
| 69 |
+
|
| 70 |
+
if __name__ == "__main__":
|
| 71 |
+
sys.exit(main())
|