.github/copilot-instructions.md

Copilot Instructions for seg_app

This document is the authoritative design contract for this project. All implementations must conform to it unless explicitly revised.

---

Project Intent

seg_app is an interactive 3D medical image segmentation tool for CT and MR volumes.

• Target users: Radiologists
• Purpose: Clinical decision-support and research prototyping
• Deployment: Hugging Face Spaces (Gradio frontend, PyTorch backend)
• Scope: Research/prototype use only — NOT regulatory-approved

Explicit Non-Goals

• NOT a PACS viewer or DICOM study manager
• NOT a real-time or streaming inference system
• NOT for multi-user hospital deployment or high concurrency
• NOT intended for FDA/CE clinical use

---

Architecture Overview

UI (Gradio) → Inference Orchestrator → Models → Data I/O
                    ↓
            Metrics & Postprocessing

Architectural Invariants (Must Not Be Broken)

• UI code must never call model code directly — all inference goes through the orchestrator
• Model backends must be swappable without UI changes — UI depends only on orchestrator.py
• Radiologists interact with tasks, not model architectures — task names are user-facing, model IDs are internal
• No model weights are hardcoded locally — all weights load from HF Hub (or user-uploaded paths)
• Spatial metadata must flow through the entire pipeline — required for accurate volume calculations

Project Structure

web_app/
├── app.py                    # HF Spaces entry point
├── requirements.txt
├── README.md                 # HF Spaces metadata (YAML frontmatter)
└── seg_app/
    ├── config/
    │   ├── settings.py       # Global settings, HF Hub IDs, defaults
    │   └── tasks.py          # Task definitions → model mappings
    ├── data/
    │   ├── io.py             # NIfTI loading/saving (nibabel)
    │   └── preprocessing.py  # Normalization, resampling, orientation
    ├── inference/
    │   ├── orchestrator.py   # Task → model dispatch, inference pipeline
    │   ├── model_registry.py # Model registration, lazy loading
    │   └── postprocess.py    # Thresholding, connected components
    ├── metrics/
    │   └── segmentation_metrics.py  # Volume (mm³), Dice, surface metrics
    ├── models/
    │   ├── base.py           # Abstract base class for all models
    │   ├── monai_autoseg.py  # MONAI Auto3DSeg wrapper
    │   ├── unet3d.py         # Task-specific 3D U-Net / VNet
    │   └── medical_sam.py    # Medical SAM for interactive refinement
    └── ui/
        ├── gradio_app.py     # Gradio Blocks layout, component wiring
        ├── viewer.py         # Multi-planar renderer (axial/sag/cor)
        └── overlays.py       # Segmentation mask overlay rendering

---

User Workflow

1. User uploads a 3D CT or MR volume (NIfTI format)
2. Volume displayed in multi-planar view (axial scrollable, sagittal, coronal)
3. User selects a segmentation task (e.g., "Liver", "Brain Lesion")
4. Default model runs automatically for that task
5. User may refine with point/bounding-box prompts (optional)
6. Outputs: on-screen overlays, volume metrics (mm³), downloadable mask (future)

---

Key Module Responsibilities

config/tasks.py

Defines task registry as Python dataclasses or dicts:

TASKS = {
    "liver": TaskConfig(
        display_name="Liver Segmentation",
        model_id="monai-auto3dseg-liver",
        hf_hub_path="your-org/liver-seg-model",
        supports_refinement=True,
    ),
    "brain_lesion": TaskConfig(
        display_name="Brain Lesion (Tumor) Segmentation",
        model_id="unet3d-brain-tumor",
        hf_hub_path="your-org/brain-tumor-model",
        supports_refinement=True,
    ),
}

inference/orchestrator.py

Single entry point for inference:

def run_segmentation(
    volume: np.ndarray,
    task_name: str,
    prompts: Optional[Prompts] = None,
    full_reinference: bool = False,  # Default: SAM refinement only
) -> SegmentationResult:

• If prompts provided and full_reinference=False: run SAM refinement on existing mask
• If full_reinference=True: run complete pipeline with prompts

inference/model_registry.py

Model loading strategy:

• Primary: Lazy-load from Hugging Face Hub on first use
• Alternative: Support local file upload for custom weights

def load_model(model_id: str, local_path: Optional[str] = None) -> BaseModel:

models/base.py

Abstract interface all models must implement:

class BaseModel(ABC):
    def load(self, weights_source: str) -> None: ...
    def preprocess(self, volume: np.ndarray, config: dict) -> torch.Tensor: ...
    def predict(self, tensor: torch.Tensor, prompts: Optional[Prompts] = None) -> torch.Tensor: ...
    def postprocess(self, tensor: torch.Tensor) -> np.ndarray: ...

ui/viewer.py

Multi-planar viewer behavior:

• Renders axial, sagittal, coronal views simultaneously
• Default: Center slice on initial load and after segmentation
• Optional: Maintain slice position across runs (configurable)
• Uses matplotlib/PIL for pure Python rendering (no JS viewer)

---

Data Flow

1. Input: NIfTI volumes via data/io.py (nibabel)
2. Preprocessing: Normalization, resampling, RAS orientation in data/preprocessing.py
3. Inference: Task lookup in config/tasks.py → model dispatch via orchestrator.py
4. Postprocessing: Label cleanup in inference/postprocess.py
5. Metrics: Volume calculation (mm³) in metrics/segmentation_metrics.py
6. Output: Overlay rendering via ui/overlays.py

---

Conventions

• Type hints: Required for all function signatures
• Array ordering: Medical imaging arrays use (D, H, W) or (C, D, H, W)
• Configuration: All settings in config/, never hardcoded
• Model weights: Lazy-loaded to minimize startup time
• Dependencies: MONAI, segment-anything (Medical SAM), Gradio, nibabel, PyTorch

Environment

• Python environment managed via Conda (see .vscode/settings.json)
• Deployment target: Hugging Face Spaces (GPU tier required)
• Single-user / low-concurrency research usage