.github/copilot-instructions.md

# Copilot Instructions for seg_app

> **This document is the authoritative design contract for this project.**
> All implementations must conform to it unless explicitly revised.

---

## Project Intent

**seg_app** is an interactive 3D medical image segmentation tool for CT and MR volumes.

- **Target users**: Radiologists
- **Purpose**: Clinical decision-support and research prototyping
- **Deployment**: Hugging Face Spaces (Gradio frontend, PyTorch backend)
- **Scope**: Research/prototype use only — NOT regulatory-approved

### Explicit Non-Goals
- NOT a PACS viewer or DICOM study manager
- NOT a real-time or streaming inference system
- NOT for multi-user hospital deployment or high concurrency
- NOT intended for FDA/CE clinical use

---

## Architecture Overview

```
UI (Gradio) → Inference Orchestrator → Models → Data I/O
                    ↓
            Metrics & Postprocessing
```

### Architectural Invariants (Must Not Be Broken)

- **UI code must never call model code directly** — all inference goes through the orchestrator
- **Model backends must be swappable without UI changes** — UI depends only on `orchestrator.py`
- **Radiologists interact with tasks, not model architectures** — task names are user-facing, model IDs are internal
- **No model weights are hardcoded locally** — all weights load from HF Hub (or user-uploaded paths)
- **Spatial metadata must flow through the entire pipeline** — required for accurate volume calculations

### Project Structure

```
web_app/
├── app.py                    # HF Spaces entry point
├── requirements.txt
├── README.md                 # HF Spaces metadata (YAML frontmatter)
└── seg_app/
    ├── config/
    │   ├── settings.py       # Global settings, HF Hub IDs, defaults
    │   └── tasks.py          # Task definitions → model mappings
    ├── data/
    │   ├── io.py             # NIfTI loading/saving (nibabel)
    │   └── preprocessing.py  # Normalization, resampling, orientation
    ├── inference/
    │   ├── orchestrator.py   # Task → model dispatch, inference pipeline
    │   ├── model_registry.py # Model registration, lazy loading
    │   └── postprocess.py    # Thresholding, connected components
    ├── metrics/
    │   └── segmentation_metrics.py  # Volume (mm³), Dice, surface metrics
    ├── models/
    │   ├── base.py           # Abstract base class for all models
    │   ├── monai_autoseg.py  # MONAI Auto3DSeg wrapper
    │   ├── unet3d.py         # Task-specific 3D U-Net / VNet
    │   └── medical_sam.py    # Medical SAM for interactive refinement
    └── ui/
        ├── gradio_app.py     # Gradio Blocks layout, component wiring
        ├── viewer.py         # Multi-planar renderer (axial/sag/cor)
        └── overlays.py       # Segmentation mask overlay rendering
```

---

## User Workflow

1. User uploads a 3D CT or MR volume (NIfTI format)
2. Volume displayed in multi-planar view (axial scrollable, sagittal, coronal)
3. User selects a segmentation task (e.g., "Liver", "Brain Lesion")
4. Default model runs automatically for that task
5. User may refine with point/bounding-box prompts (optional)
6. Outputs: on-screen overlays, volume metrics (mm³), downloadable mask (future)

---

## Key Module Responsibilities

### `config/tasks.py`
Defines task registry as Python dataclasses or dicts:
```python
TASKS = {
    "liver": TaskConfig(
        display_name="Liver Segmentation",
        model_id="monai-auto3dseg-liver",
        hf_hub_path="your-org/liver-seg-model",
        supports_refinement=True,
    ),
    "brain_lesion": TaskConfig(
        display_name="Brain Lesion (Tumor) Segmentation",
        model_id="unet3d-brain-tumor",
        hf_hub_path="your-org/brain-tumor-model",
        supports_refinement=True,
    ),
}
```

### `inference/orchestrator.py`
Single entry point for inference:
```python
def run_segmentation(
    volume: np.ndarray,
    task_name: str,
    prompts: Optional[Prompts] = None,
    full_reinference: bool = False,  # Default: SAM refinement only
) -> SegmentationResult:
```
- If `prompts` provided and `full_reinference=False`: run SAM refinement on existing mask
- If `full_reinference=True`: run complete pipeline with prompts

### `inference/model_registry.py`
Model loading strategy:
- **Primary**: Lazy-load from Hugging Face Hub on first use
- **Alternative**: Support local file upload for custom weights
```python
def load_model(model_id: str, local_path: Optional[str] = None) -> BaseModel:
```

### `models/base.py`
Abstract interface all models must implement:
```python
class BaseModel(ABC):
    def load(self, weights_source: str) -> None: ...
    def preprocess(self, volume: np.ndarray, config: dict) -> torch.Tensor: ...
    def predict(self, tensor: torch.Tensor, prompts: Optional[Prompts] = None) -> torch.Tensor: ...
    def postprocess(self, tensor: torch.Tensor) -> np.ndarray: ...
```

### `ui/viewer.py`
Multi-planar viewer behavior:
- Renders axial, sagittal, coronal views simultaneously
- **Default**: Center slice on initial load and after segmentation
- **Optional**: Maintain slice position across runs (configurable)
- Uses matplotlib/PIL for pure Python rendering (no JS viewer)

---

## Data Flow

1. **Input**: NIfTI volumes via `data/io.py` (nibabel)
2. **Preprocessing**: Normalization, resampling, RAS orientation in `data/preprocessing.py`
3. **Inference**: Task lookup in `config/tasks.py` → model dispatch via `orchestrator.py`
4. **Postprocessing**: Label cleanup in `inference/postprocess.py`
5. **Metrics**: Volume calculation (mm³) in `metrics/segmentation_metrics.py`
6. **Output**: Overlay rendering via `ui/overlays.py`

---

## Conventions

- **Type hints**: Required for all function signatures
- **Array ordering**: Medical imaging arrays use `(D, H, W)` or `(C, D, H, W)`
- **Configuration**: All settings in `config/`, never hardcoded
- **Model weights**: Lazy-loaded to minimize startup time
- **Dependencies**: MONAI, segment-anything (Medical SAM), Gradio, nibabel, PyTorch

## Environment

- Python environment managed via **Conda** (see `.vscode/settings.json`)
- Deployment target: Hugging Face Spaces (GPU tier required)
- Single-user / low-concurrency research usage