# AI Agent — Copilot Instructions

This is a **RAG + VLM imaging tool recommender** that helps users find the right imaging software for their images and tasks. Users drop an image, describe their task, and get ranked software recommendations with demo links.

## Architecture Overview

The system follows a two-stage pipeline:

1. **Retrieval Stage** (`retriever/`, `api/pipeline.py`): Fast text search using BGE-M3 embeddings + CrossEncoder reranker. No LLM calls. Returns top-K candidates.

2. **Selection Stage** (`generator/`): Single VLM call (OpenAI GPT-4o/mini) that sees the image + candidates + metadata and returns ranked recommendations with accuracy scores.

### Key Components

- **`api/pipeline.RAGImagingPipeline`**: Main orchestrator. Handles file validation, metadata extraction, retrieval, and VLM selection.
- **`retriever/text_embedder.py`**, **`retriever/vector_index.py`**, **`retriever/reranker.py`**, **`retriever/software_doc.py`**: Embedding, FAISS indexing, reranking, and catalog schema for retrieval.
- **`agent/agent.py`**: PydanticAI agent that orchestrates tool search, alternatives, and recommendation assembly.
- **`utils/image_meta.py`**: Robust metadata extraction for DICOM, NIfTI, TIFF stacks with medical imaging focus.
- **`utils/tags.py`**: Control tag parsing/stripping utilities (notably `[EXCLUDE:tool1|tool2]`).

## Data Flow Patterns

### Input Processing
- Files validated via `utils/file_validator.py` (size limits, format checks)
- Images converted to PNG previews for VLM via `utils/previews.py`
- Metadata extracted preserving original format info (critical for format compatibility matching)
- Format tokens added to retrieval query (e.g. `format:DICOM format:NIfTI`)

### Retrieval Query Construction
```python
# Clean task text + format tokens from uploaded files
query = f"{clean_task} format:{ext_tokens}"  # e.g. "segment lungs format:DICOM"
```

### VLM Selection Input
The VLM receives:
- **Text**: User task + candidate table + original file metadata
- **Image**: PNG preview (converted from any format) 
- **Metadata**: Original extension, dimensions, file info (crucial for IO compatibility)

## Critical Patterns

### Error Handling
- **Graceful degradation**: If image conversion fails, continue text-only
- **Robust metadata**: All metadata extraction wrapped in try/catch with sensible defaults
- **File validation**: Early validation prevents downstream errors

### Control Tags System
Users can control behavior via tags in their queries:
- `[EXCLUDE:toolname1|toolname2]` - Exclude specific tools from results

### Conversation Flow
- **Complete**: Normal success with tool recommendations
- **Needs Clarification**: VLM asks followup questions when task is ambiguous
- **Terminal No-Tool**: No suitable tools found with explanation

## Development Workflows

### Running the App
```bash
# Install with pip using pyproject.toml
pip install -e ".[dev]"

# Configure .env with OPENAI_API_KEY and SOFTWARE_CATALOG path
ai_agent chat  # Launches Gradio chat UI
```

### Testing
- Run targeted tests in `tests/` (e.g., retrieval, agent tools, repo info)
- Run with: `pytest tests/`

### Change Documentation
- **`CHANGELOG.md`**: Follow [Keep a Changelog](https://keepachangelog.com/) format
- Use semantic versioning with sections: Added, Changed, Deprecated, Removed, Fixed, Security
- Update CHANGELOG.md for ALL user-facing changes before merging PRs
- Format: `### Added\n- New feature description` under version heading
- Version entries: `## [x.y.z] - YYYY-MM-DD`

### Environment Management
- **uv**: Fast Python package manager used in `tools/image/Dockerfile`
- Creates isolated `.venv` environments for reproducible builds
- Dockerfile uses `uv venv && uv pip install -e .` pattern for container builds

### Logging & Debugging
- Set `LOG_PROMPTS=1` to save VLM prompts + images to `logs/`
- File logs in `logs/app_YYYYMMDD.log` with structured JSON events
- Console/file log levels configurable via `.env`

## Project Conventions

### Schema Patterns
- **Pydantic models** in `generator/schema.py` with robust field validation and aliasing for catalog compatibility
- **Enum-based** conversation states and tool reasons for type safety
- **Field normalization**: Dimensions (2D/3D/4D), modalities (CT/MRI/XR), file formats via validators

### Catalog Integration
- Software catalog in JSONL format following schema.org SoftwareSourceCode structure
- **Runnable examples**: Links to HuggingFace Spaces, notebooks, web demos
- **Supporting data**: Format compatibility info used for matching

### Module Boundaries
- `api/`: Pipeline orchestration, no UI dependencies
- `generator/`: Pure VLM logic, no retrieval dependencies  
- `retriever/`: Pure vector search, no generation dependencies
- `utils/`: Shared utilities, no business logic
- `ui/`: Gradio interface only

### Configuration
- Environment-based config via `.env` (API keys, model names, catalog paths)
- Sensible defaults for all settings
- No hardcoded paths or credentials

## Medical Imaging Context

This tool specializes in medical/scientific imaging:
- **Modalities**: CT, MRI, X-ray, Ultrasound, PET, SPECT, Microscopy
- **Formats**: DICOM, NIfTI, TIFF stacks, standard images
- **Dimensions**: 2D images, 3D volumes, 4D timeseries
- **Tasks**: Segmentation, registration, analysis, visualization

The VLM selection considers format compatibility as a primary factor - tools supporting the user's input format are strongly preferred.

## Security Notes
- Only makes external calls to OpenAI VLM API (with user image preview)
- Never uploads user data to third-party tool demos
- Returns links only; user chooses whether to visit demos
- Prompt logging is optional and local-only