docs/user-guide/chat-interface.md

# Using the Chat Interface

The AI Imaging Agent provides a conversational interface for discovering and using imaging software. This guide explains how to interact with the chat interface effectively.

## Interface Layout

The interface consists of three main areas:

### Left Panel: Chat Conversation
- **Message History**: Your conversation with the agent
- **Rich Media Rendering**: Images, tool cards, and files are displayed inline
- **Input Box**: Type your messages at the bottom
- **File Upload**: Attach files via the paperclip icon or drag-and-drop

### Right Panel: Sidebar
- **Files Tab**: View uploaded files with format information
- **State Tab**: Debug information showing conversation state

### Header
- **Model Selector**: Choose which AI model to use
- **Settings**: Access configuration options

## Basic Workflow

### 1. Upload Files

Upload images or other files in several ways:

- **Drag and Drop**: Drag files directly onto the upload area
- **Click to Browse**: Click the upload area to select files
- **Attach to Message**: Use the paperclip icon in the input box

Files are automatically processed and metadata is extracted.

### 2. Describe Your Task

Use natural language to describe what you want to do:

!!! example "Good Task Descriptions"
    - "I want to segment the lungs from this CT scan"
    - "Help me detect tumors in this MRI"
    - "I need to register these two brain images"
    - "Extract text from this medical report"
    - "Classify the organ shown in this ultrasound"

### 3. Review Recommendations

The agent returns ranked tool recommendations with:

- **Tool Cards**: Each tool is presented in a card format
- **Accuracy Scores**: Confidence levels for each recommendation
- **Explanations**: Why each tool matches your request
- **Metadata**: Technical details about compatibility

### 4. Run Demos (Optional)

The agent may offer to run demos:

```

Agent: Would you like me to run the demo with your image?

```

Respond with affirmative language:
- "yes"
- "sure"
- "ok"
- "please"
- "go ahead"

The agent will execute the tool and show results.

## Multi-Turn Conversations

The agent maintains context across multiple messages:

!!! example "Multi-Turn Example"
    ```

    You: I have a lung CT scan [uploads file]

    

    Agent: I can see you have a DICOM CT image. What would you like to do with it?

    

    You: Segment the airways

    

    Agent: [Provides airway segmentation tool recommendations]

    

    You: What about segmenting the whole lung?

    

    Agent: [Provides lung segmentation tools, remembering you're working with CT]

    

    You: Show me alternatives

    

    Agent: [Provides additional options]

    ```


## Advanced Features

### Excluding Tools

Exclude specific tools using the `[EXCLUDE:...]` tag:

```

Find segmentation tools [EXCLUDE:totalsegmentator|medicalsam]

```

You can exclude multiple tools separated by `|`.

### Requesting Alternatives

Ask the agent to search with different strategies:

```

Can you search for alternatives?



Show me other options



Find different tools for this task

```

The agent can perform up to 3 alternative searches per conversation.

## Understanding Agent Responses

### Recommendation Cards

Each recommendation includes:

#### Header
- **Rank Number**: 1, 2, 3 (1 = best match)
- **Tool Name**: Software/tool identifier
- **Accuracy Score**: 0-100% confidence

#### Body
- **Description**: What the tool does
- **Explanation**: Why it matches your task
- **Demo Link**: Click to visit runnable example

#### Footer Metadata
- **Modalities**: CT, MRI, X-ray, etc.
- **Dimensions**: 2D, 3D, 4D
- **Formats**: Supported file formats (DICOM, NIfTI, etc.)
- **License**: Software license information
- **Tags**: Categorization and keywords

### Execution Traces

When demos run, you'll see execution details:

```

<details>

<summary>Tool Execution Trace</summary>



Image uploaded to Gradio Space

Processing started...

Result: Success

Output saved to: result.png

</details>

```

Click to expand and see full execution logs.

### Clarification Questions

Sometimes the agent needs more information:

```

Agent: I found several segmentation tools. Which organ are you trying to segment?



You: The liver



Agent: [Provides liver-specific segmentation tools]

```

## File Management

### Uploaded Files List

The sidebar shows all uploaded files with:

- **Filename**: Original file name
- **Format**: File type/extension
- **Size**: File size
- **Preview**: Thumbnail (for images)

### Image Previews

Medical images are automatically converted:

- **DICOM**: PNG previews; 3D series use orthogonal composite views (MIPs + central slices) rather than a single slice
- **NIfTI**: PNG previews built from orthogonal composite views of the volume
- **TIFF Stacks**: PNG previews built from orthogonal composite views of the stack
- **Standard 2D Images**: Resized PNG preview of the original image

Previews are used for VLM analysis while preserving original format metadata.

### Removing Files

Click the 'X' button next to a file to remove it from the current session.

## Conversation State

The debug sidebar shows:

### Current State
- **Status**: idle, processing, waiting
- **Conversation Turn**: Current turn number
- **Excluded Tools**: Tools filtered from results

### Preview Images
- Images prepared for VLM analysis
- Format conversions applied

## Tips for Effective Interaction

!!! tip "Be Specific About Requirements"
    Mention specific needs:

    

    - "I need a tool that works with NIfTI files"

    - "Must support 3D volumes"

    - "Looking for open-source options"


!!! tip "Use Conversational Language"
    Natural language works best:

    

    - ✅ "Help me find tool that segments kidneys"

    - ❌ "kidney_segmentation_tool filter:3D"


!!! tip "Iterate Based on Results"
    If initial results aren't perfect, refine:

    

    - "Can you find tools with higher accuracy?"

    - "Show me open-source alternatives"

    - "What about tools that support DICOM?"


!!! tip "Ask Follow-Up Questions"
    The agent maintains context:

    

    - "What about the second recommendation?"

    - "Can you compare these two tools?"

    - "Which one is fastest?"


## Troubleshooting

### No Recommendations

If the agent can't find suitable tools:

- Try rephrasing your query
- Be more specific about the task
- Check that your file uploaded successfully
- Ensure your task matches the catalog domain (imaging/medical)

### Wrong Recommendations

If recommendations don't match:

- Provide more context about your specific needs
- Mention required file format support
- Specify modality or domain
- Use the exclude feature to filter out irrelevant tools

### Demo Execution Fails

If a demo doesn't run:

- Check your internet connection
- Verify the demo link is still active
- Try a different recommended tool
- Check file format compatibility

## Next Steps

- Learn about [Supported File Formats](file-formats.md)
- Understand [How Recommendations Work](recommendations.md)
- Explore [Running Demos](running-demos.md)
- Check out [Advanced Features](advanced-features.md)