Using the Chat Interface
The AI Imaging Agent provides a conversational interface for discovering and using imaging software. This guide explains how to interact with the chat interface effectively.
Interface Layout
The interface consists of three main areas:
Left Panel: Chat Conversation
- Message History: Your conversation with the agent
- Rich Media Rendering: Images, tool cards, and files are displayed inline
- Input Box: Type your messages at the bottom
- File Upload: Attach files via the paperclip icon or drag-and-drop
Right Panel: Sidebar
- Files Tab: View uploaded files with format information
- State Tab: Debug information showing conversation state
Header
- Model Selector: Choose which AI model to use
- Settings: Access configuration options
Basic Workflow
1. Upload Files
Upload images or other files in several ways:
- Drag and Drop: Drag files directly onto the upload area
- Click to Browse: Click the upload area to select files
- Attach to Message: Use the paperclip icon in the input box
Files are automatically processed and metadata is extracted.
2. Describe Your Task
Use natural language to describe what you want to do:
!!! example "Good Task Descriptions" - "I want to segment the lungs from this CT scan" - "Help me detect tumors in this MRI" - "I need to register these two brain images" - "Extract text from this medical report" - "Classify the organ shown in this ultrasound"
3. Review Recommendations
The agent returns ranked tool recommendations with:
- Tool Cards: Each tool is presented in a card format
- Accuracy Scores: Confidence levels for each recommendation
- Explanations: Why each tool matches your request
- Metadata: Technical details about compatibility
4. Run Demos (Optional)
The agent may offer to run demos:
Agent: Would you like me to run the demo with your image?
Respond with affirmative language:
- "yes"
- "sure"
- "ok"
- "please"
- "go ahead"
The agent will execute the tool and show results.
Multi-Turn Conversations
The agent maintains context across multiple messages:
!!! example "Multi-Turn Example" ``` You: I have a lung CT scan [uploads file]
Agent: I can see you have a DICOM CT image. What would you like to do with it?
You: Segment the airways
Agent: [Provides airway segmentation tool recommendations]
You: What about segmenting the whole lung?
Agent: [Provides lung segmentation tools, remembering you're working with CT]
You: Show me alternatives
Agent: [Provides additional options]
```
Advanced Features
Excluding Tools
Exclude specific tools using the [EXCLUDE:...] tag:
Find segmentation tools [EXCLUDE:totalsegmentator|medicalsam]
You can exclude multiple tools separated by |.
Requesting Alternatives
Ask the agent to search with different strategies:
Can you search for alternatives?
Show me other options
Find different tools for this task
The agent can perform up to 3 alternative searches per conversation.
Understanding Agent Responses
Recommendation Cards
Each recommendation includes:
Header
- Rank Number: 1, 2, 3 (1 = best match)
- Tool Name: Software/tool identifier
- Accuracy Score: 0-100% confidence
Body
- Description: What the tool does
- Explanation: Why it matches your task
- Demo Link: Click to visit runnable example
Footer Metadata
- Modalities: CT, MRI, X-ray, etc.
- Dimensions: 2D, 3D, 4D
- Formats: Supported file formats (DICOM, NIfTI, etc.)
- License: Software license information
- Tags: Categorization and keywords
Execution Traces
When demos run, you'll see execution details:
<details>
<summary>Tool Execution Trace</summary>
Image uploaded to Gradio Space
Processing started...
Result: Success
Output saved to: result.png
</details>
Click to expand and see full execution logs.
Clarification Questions
Sometimes the agent needs more information:
Agent: I found several segmentation tools. Which organ are you trying to segment?
You: The liver
Agent: [Provides liver-specific segmentation tools]
File Management
Uploaded Files List
The sidebar shows all uploaded files with:
- Filename: Original file name
- Format: File type/extension
- Size: File size
- Preview: Thumbnail (for images)
Image Previews
Medical images are automatically converted:
- DICOM: PNG previews; 3D series use orthogonal composite views (MIPs + central slices) rather than a single slice
- NIfTI: PNG previews built from orthogonal composite views of the volume
- TIFF Stacks: PNG previews built from orthogonal composite views of the stack
- Standard 2D Images: Resized PNG preview of the original image
Previews are used for VLM analysis while preserving original format metadata.
Removing Files
Click the 'X' button next to a file to remove it from the current session.
Conversation State
The debug sidebar shows:
Current State
- Status: idle, processing, waiting
- Conversation Turn: Current turn number
- Excluded Tools: Tools filtered from results
Preview Images
- Images prepared for VLM analysis
- Format conversions applied
Tips for Effective Interaction
!!! tip "Be Specific About Requirements" Mention specific needs:
- "I need a tool that works with NIfTI files"
- "Must support 3D volumes"
- "Looking for open-source options"
!!! tip "Use Conversational Language" Natural language works best:
- ✅ "Help me find tool that segments kidneys"
- ❌ "kidney_segmentation_tool filter:3D"
!!! tip "Iterate Based on Results" If initial results aren't perfect, refine:
- "Can you find tools with higher accuracy?"
- "Show me open-source alternatives"
- "What about tools that support DICOM?"
!!! tip "Ask Follow-Up Questions" The agent maintains context:
- "What about the second recommendation?"
- "Can you compare these two tools?"
- "Which one is fastest?"
Troubleshooting
No Recommendations
If the agent can't find suitable tools:
- Try rephrasing your query
- Be more specific about the task
- Check that your file uploaded successfully
- Ensure your task matches the catalog domain (imaging/medical)
Wrong Recommendations
If recommendations don't match:
- Provide more context about your specific needs
- Mention required file format support
- Specify modality or domain
- Use the exclude feature to filter out irrelevant tools
Demo Execution Fails
If a demo doesn't run:
- Check your internet connection
- Verify the demo link is still active
- Try a different recommended tool
- Check file format compatibility
Next Steps
- Learn about Supported File Formats
- Understand How Recommendations Work
- Explore Running Demos
- Check out Advanced Features