README.md

---
title: NeuroSAM 3
emoji: 🏥
colorFrom: blue
colorTo: purple
sdk: gradio
sdk_version: 6.1.0
app_file: app.py
pinned: true
license: apache-2.0
---

# NeuroSAM 3: Medical Image Segmentation

A comprehensive medical image segmentation application using SAM 3 (Segment Anything Model 3) for DICOM file analysis with advanced features.

## ✨ Features

### Core Features
- 🧠 **SAM 3 Integration**: Uses the latest Segment Anything Model 3 for medical image segmentation
- 📁 **DICOM Support**: Process CT and MRI DICOM files
- 🖼️ **Image Formats**: Supports DICOM (.dcm), PNG, and JPG files
- 🎯 **Text Prompts**: Describe what you want to segment (e.g., "brain", "tumor", "skull")
- ⚙️ **Windowing Strategies**: Optimized windowing presets for CT images

### Interactive Features
- 📊 **Interactive Slice Viewer**: Scroll through multiple slices with auto-play
- 🔍 **Subject Detection**: Automatically groups images by patient/subject ID
- ▶️ **Auto-play**: Automatically animate through slices
- 📈 **Ground Truth Comparison**: Compare SAM 3 results with expert annotations (Dice score, IoU)

### Enhanced Features (NEW!)
- ✨ **Enhanced Single Image**: Brightness/contrast controls, colormap options, transparency adjustment
- ✨ **Enhanced Batch Processing**: Process multiple images with ZIP download
- 🎯 **Point/Box Prompts**: Interactive point-click and bounding box-based segmentation
- 📊 **ROI Statistics**: Detailed statistics (area, intensity, centroid, bounding box)
- 📥 **NIFTI Export**: Export segmentation masks to NIFTI format for medical imaging software
- 💾 **Annotation Save/Load**: Save and reload annotations for later use
- 🎭 **Multi-Mask Output**: Generate multiple mask candidates with confidence scores

## Usage

### Single Image Processing
1. Upload a DICOM (.dcm), PNG, or JPG file
2. Enter a text prompt describing what to segment
3. Select the imaging modality (CT or MRI)
4. Choose the windowing strategy (for CT images)
5. Click "Segment Structure" to process

### Interactive Slice Viewer
1. Upload multiple slices/images from the same subject
2. Click "🔍 Detect Subjects" to auto-group by patient ID
3. Select a subject from the dropdown
4. Click "Process All Slices"
5. Use the slider, navigation buttons, or ▶️ Auto-play to scroll through slices

### Enhanced Single Image (with Image Adjustments)
1. Upload an image
2. Adjust brightness and contrast sliders
3. Choose colormap and transparency for mask overlay
4. Click "Segment with Progress"
5. Download result using the download button

### Point/Box Prompts
1. Upload an image
2. **Point Prompt**: Enter X, Y coordinates and click "Segment at Point"
3. **Box Prompt**: Enter bounding box coordinates (X1, Y1, X2, Y2) and click "Segment in Box"

### ROI Statistics & Export
1. Process an image using "Process & Calculate Statistics"
2. View detailed ROI statistics (area, intensity, centroid)
3. Export to NIFTI format for use in medical imaging software
4. Save annotation for later use
5. Load previously saved annotations

### Multi-Mask Output
1. Upload an image and enter prompt
2. Choose number of masks to generate (1-5)
3. Click "Generate Multiple Masks"
4. Compare mask candidates with confidence scores

### Compare with Ground Truth
1. Upload a medical image and its ground truth mask
2. Enter segmentation prompt
3. Click "Compare Segmentation" to see metrics (Dice score, IoU)

## Requirements

- Python 3.8+
- PyTorch
- Gradio
- PyDICOM
- Transformers (SAM 3 requires latest version from GitHub)
- nibabel (for NIFTI export)
- scipy (for ROI statistics)

## Installation

To install the required dependencies:

```bash
pip install -r requirements.txt
```

**Note:** SAM 3 is only available in the development version of transformers. The requirements.txt file automatically installs it from the GitHub repository.

## Model

This app uses the SAM 3 model from Facebook/Meta. You need:
- A Hugging Face account
- Access token with read permissions for the SAM 3 model (get it from https://huggingface.co/settings/tokens)
- Set `HF_TOKEN` environment variable in Space settings

## Public Datasets

- UniqueData/dicom-brain-dataset (Hugging Face) - MRI Brain scans
- The Cancer Imaging Archive (TCIA) - Various medical imaging
- Imaging Data Commons (IDC) - Large collection of DICOM files
- Kaggle Brain MRI datasets - PNG/JPG format images

## Tabs Overview

| Tab | Description |
|-----|-------------|
| Single Image | Basic single image segmentation |
| Interactive Slice Viewer | Browse through multiple slices with auto-play |
| Gallery View | View all segmentations in a grid |
| Compare with Ground Truth | Evaluate with Dice/IoU metrics |
| ✨ Enhanced Single Image | Advanced controls with download |
| ✨ Enhanced Batch Processing | Batch processing with ZIP export |
| 🎯 Point/Box Prompts | Interactive region selection |
| 📊 ROI Statistics & Export | Statistics + NIFTI/annotation export |
| 🎭 Multi-Mask Output | Multiple mask candidates |
| 🔬 Automatic Mask Generator | AMG - segment entire image without prompts |
| 🔧 Advanced Transforms | ResizeLongestSide + CLAHE enhancement |
| 🌊 Edge-Based Segmentation | Sobel edge detection (no AI required) |

## SAM-Medical-Imaging Utilities

Inspired by [amine0110/SAM-Medical-Imaging](https://github.com/amine0110/SAM-Medical-Imaging), we've integrated several utilities:

- **Automatic Mask Generator (AMG)**: Automatically segment all regions in an image using multiple prompts
- **ResizeLongestSide Transform**: Optimal image resizing that maintains aspect ratio
- **CLAHE-like Enhancement**: Contrast enhancement for low-visibility images
- **Edge-Based Segmentation**: Classical computer vision approach using Sobel gradients

## License

Apache 2.0