docs/dev/agents_config-README.md

# Agent Configuration Guide

**Complete guide for adding, removing, and modifying agents/subagents in BirdScope AI**

---

## 📋 Table of Contents

1. [Architecture Overview](#architecture-overview)
2. [Key Files Reference](#key-files-reference)
3. [Adding a New Subagent](#adding-a-new-subagent)
4. [Removing a Subagent](#removing-a-subagent)
5. [Modifying Existing Subagents](#modifying-existing-subagents)
6. [App.py Integration Points](#apppy-integration-points)
7. [Testing Your Changes](#testing-your-changes)

---

## Architecture Overview

BirdScope AI uses a **LangGraph supervisor pattern** with specialized subagents:

```
User Request
    ↓
Supervisor (Router)
    ↓
┌───────────────┬──────────────────┬─────────────────┐
│ image_identifier │ taxonomy_specialist │ (other agents) │
└───────────────┴──────────────────┴─────────────────┘
```

**Key Concepts:**
- **Supervisor**: LLM-based router that delegates tasks to specialists
- **Subagents**: Specialized agents with filtered tool access and focused prompts
- **Modes**: Different agent configurations (e.g., "Supervisor (Multi-Agent)")
- **Tool Filtering**: Each subagent only has access to relevant tools

---

## Key Files Reference

### Core Agent Files

| File | Purpose | What to Change |
|------|---------|----------------|
| `langgraph_agent/subagent_supervisor.py` | Creates supervisor workflow | Add/remove agents from supervisor list |
| `langgraph_agent/subagent_config.py` | Defines subagent configurations | Add/remove/modify subagent definitions and modes |
| `langgraph_agent/subagent_factory.py` | Builds subagent instances | (Usually no changes needed) |
| `langgraph_agent/prompts.py` | System prompts for agents | Add provider-specific prompts |

### UI Integration

| File | Purpose | What to Change |
|------|---------|----------------|
| `app.py` | Gradio UI and agent orchestration | Update mode dropdown, default values, examples |

---

## Adding a New Subagent

### Step 1: Define Subagent Configuration

**File:** `langgraph_agent/subagent_config.py`

Add your subagent to `get_subagent_definitions()`:

```python
@staticmethod
def get_subagent_definitions(provider: str = "openai") -> Dict[str, Dict]:
    return {
        # ... existing agents ...

        "my_new_agent": {
            "name": "My New Specialist",
            "description": "Expert at specific bird-related tasks",
            "tools": [
                "tool_name_1",
                "tool_name_2",
                "tool_name_3"
            ],
            "prompt": get_prompt("my_new_agent", provider) or """You are a My New Specialist.

**Your Role:**
1. Primary responsibility
2. Secondary responsibility
3. When to use specific tools

**Response Style:**
- How to format responses
- What to emphasize

**When to defer:**
- Task type 1 -> other_agent_name
- Task type 2 -> another_agent_name
""",
            "temperature": AgentConfig.OPENAI_TEMPERATURE,
        }
    }
```

### Step 2: Create System Prompts

**File:** `langgraph_agent/prompts.py`

Add prompts for your new agent:

```python
# Default prompt (used by OpenAI/Anthropic)
MY_NEW_AGENT_PROMPT = """Detailed prompt for your agent..."""

# HuggingFace-optimized prompt (more explicit, step-by-step)
MY_NEW_AGENT_PROMPT_HF = """Simplified, step-by-step prompt..."""

# Add to PROMPTS dictionary
PROMPTS = {
    # ... existing prompts ...
    "my_new_agent": {
        "default": MY_NEW_AGENT_PROMPT,
        "huggingface": MY_NEW_AGENT_PROMPT_HF,
    },
}
```

### Step 3: Add to Supervisor Workflow

**File:** `langgraph_agent/subagent_supervisor.py`

```python
async def create_supervisor_workflow(all_tools, llm, provider="openai"):
    # Create existing agents
    image_agent = await SubAgentFactory.create_subagent(
        "image_identifier", all_tools, llm, provider=provider
    )
    # ... other agents ...

    # Add your new agent
    my_new_agent = await SubAgentFactory.create_subagent(
        "my_new_agent", all_tools, llm, provider=provider
    )

    # Add to supervisor list
    workflow = create_supervisor(
        [image_agent, taxonomy_agent, my_new_agent],  # Add here
        model=llm,
        prompt=SubAgentConfig.get_router_prompt(provider=provider)
    )
```

### Step 4: Update Router Prompts

**File:** `langgraph_agent/subagent_config.py`

Update `get_router_prompt()`:

```python
return """You are BirdScope AI Supervisor...

**Your Team:**
- **image_identifier**: Identifies birds from photos...
- **taxonomy_specialist**: Conservation status, families...
- **my_new_agent**: Specific tasks for my new agent  # Add this

**Routing Guidelines:**
1. **Image uploads/URLs** → image_identifier
2. **Conservation queries** → taxonomy_specialist
3. **New task type** → my_new_agent  # Add this
```

Also update `prompts.py` for HuggingFace router:

```python
ROUTER_PROMPT_HF = """...
**Specialists:**
- image_identifier: ...
- taxonomy_specialist: ...
- my_new_agent: New task handling  # Add this

**Routing Rules:**
...
6. "New task keyword" → my_new_agent  # Add this
"""
```

### Step 5: Update Mode Definition

**File:** `langgraph_agent/subagent_config.py`

Update `get_mode_definitions()`:

```python
return {
    "Supervisor (Multi-Agent)": {  # Current mode name
        "description": "Router orchestrates 3 specialized agents",
        "subagents": ["image_identifier", "taxonomy_specialist", "my_new_agent"],  # Add here
        "use_router": True
    },
}
```

### Step 6: Integrate with app.py

See [App.py Integration Points](#apppy-integration-points) below.

---

## Removing a Subagent

**Example: Removing `species_explorer` from the supervisor**

### Step 1: Remove from Supervisor Workflow

**File:** `langgraph_agent/subagent_supervisor.py`

```python
async def create_supervisor_workflow(all_tools, llm, provider="openai"):
    # Remove agent creation
    # species_agent = await SubAgentFactory.create_subagent(...)  # DELETE

    # Remove from supervisor list
    workflow = create_supervisor(
        [image_agent, taxonomy_agent],  # Remove species_agent
        model=llm,
        prompt=SubAgentConfig.get_router_prompt(provider=provider)
    )
```

### Step 2: Update Mode Definition

**File:** `langgraph_agent/subagent_config.py`

```python
return {
    "Supervisor (Multi-Agent)": {  # Update count if needed
        "description": "Router orchestrates 2 specialized agents",
        "subagents": ["image_identifier", "taxonomy_specialist"],  # Remove agent
        "use_router": True
    },
}
```

### Step 3: Update Router Prompts

**File:** `langgraph_agent/subagent_config.py` (default router)

```python
return """You are BirdScope AI Supervisor...

**Your Team:**
- **image_identifier**: ...
- **taxonomy_specialist**: ...
# Remove species_explorer reference

**Routing Guidelines:**
# Remove routing rules for deleted agent
# Reassign its responsibilities to other agents
```

**File:** `langgraph_agent/prompts.py` (HuggingFace router)

```python
ROUTER_PROMPT_HF = """...
**Specialists:**
- image_identifier: ...
- taxonomy_specialist: ...
# Remove deleted agent

**Routing Rules:**
# Remove routing rules
# Reassign to remaining agents
"""
```

### Step 4: Update "When to defer" Sections

**File:** `langgraph_agent/subagent_config.py`

Update remaining subagents' prompts:

```python
"image_identifier": {
    # ...
    "prompt": """...
    **When to defer:**
    - For family/taxonomy queries -> taxonomy_specialist
    # Remove references to deleted agent
    """,
}
```

### Step 5: Update app.py References

See [App.py Integration Points](#apppy-integration-points) below.

---

## Modifying Existing Subagents

### Changing Tool Access

**File:** `langgraph_agent/subagent_config.py`

```python
"image_identifier": {
    "tools": [
        "classify_from_url",
        "classify_from_base64",
        "get_bird_info",
        "new_tool_name"  # Add new tool
    ],
}
```

### Updating Prompts

**File:** `langgraph_agent/subagent_config.py` or `langgraph_agent/prompts.py`

```python
# For inline prompts (in subagent_config.py)
"image_identifier": {
    "prompt": get_prompt("image_identifier", provider) or """Updated prompt..."""
}

# For dedicated prompts (in prompts.py)
IMAGE_IDENTIFIER_PROMPT = """Updated comprehensive prompt..."""
```

### Changing Temperature

**File:** `langgraph_agent/subagent_config.py`

```python
"species_explorer": {
    "temperature": 0.2,  # More creative (was 0.1)
}
```

---

## App.py Integration Points

**When you change agent modes, you MUST update these sections in app.py:**

### 1. Mode Dropdown Choices

**Location:** `app.py` ~line 1486-1491

```python
agent_mode = gr.Dropdown(
    choices=[
        "Supervisor (Multi-Agent)"  # Update mode name here
    ],
    value="Supervisor (Multi-Agent)",  # Update default here
    show_label=False,
    container=False
)
```

### 2. Initial Session Status HTML

**Location:** `app.py` ~line 1560

```python
session_status = gr.HTML(
    value=create_config_html(
        provider_choice="OpenAI",
        agent_mode_choice="Supervisor (Multi-Agent)",  # Update here
        hf_key_input="",
        openai_key_input="",
        anthropic_key_input=""
    )
)
```

### 3. Health Check Config HTML

**Location:** `app.py` ~line 1654

```python
config_html = create_config_html(
    provider_choice=provider_str,
    agent_mode_choice="Supervisor (Multi-Agent)",  # Update here
    hf_key_input=hf_key_value,
    openai_key_input=openai_key_input,
    anthropic_key_input=anthropic_key_input
)
```

### 4. Example Loading Logic Comments

**Location:** `app.py` ~line 1033

```python
else:  # Supervisor (Multi-Agent)  # Update comment
    samples = [[text] for text in MULTI_AGENT_TEXT_EXAMPLES]
```

### 5. (Optional) Add Mode-Specific Examples

**Location:** `app.py` ~line 30-40 (add new example list)

```python
# Text-only examples for Specialized Subagents mode
MULTI_AGENT_TEXT_EXAMPLES = [
    "Tell me about Northern Cardinals - show me images and audio",
    "What birds are in the Cardinalidae family?",
    "Show me species with endangered status",
    "Find me audio recordings for Snow Goose",
    "Get me bird call samples for any two species"
]

# Add examples for your new agent mode
MY_NEW_AGENT_EXAMPLES = [
    "Example query 1 for new mode",
    "Example query 2 for new mode",
    "Example query 3 for new mode"
]
```

**Location:** `app.py` ~line 1027-1041 (update conditional logic)

The function includes a **placeholder for future modes**. Uncomment and customize:

```python
def update_text_examples_for_mode(mode):
    """Return appropriate text example dataset based on agent mode."""
    print(f"[DEBUG] Updating text examples for mode: {mode}")

    # Placeholder for future mode-specific examples
    if mode == "My New Agent Mode":  # UNCOMMENT and update mode name
        samples = [[text] for text in MY_NEW_AGENT_EXAMPLES]
        print(f"[DEBUG] New mode text samples: {len(samples)} examples")
    # elif mode == "Another Mode":  # Add more modes as needed
    #     samples = [[text] for text in ANOTHER_MODE_EXAMPLES]
    else:  # Default: Specialized Subagents

    # Default: Supervisor (Multi-Agent)
    samples = [[text] for text in MULTI_AGENT_TEXT_EXAMPLES]
    print(f"[DEBUG] Multi-agent text samples: {len(samples)} examples")

    return gr.Dataset(samples=samples)
```

**Why keep the conditional?** Even with only one mode, we maintain the placeholder structure to make it easy to add new modes later without refactoring the entire function.

---

## Testing Your Changes

### 1. Local Testing

```bash
# Run the app locally
python app.py
# or
gradio app.py
```

### 2. Check for Errors

**Common errors to watch for:**

```
Unknown mode: Old Mode Name. Available: ['Supervisor (Multi-Agent)']
```
→ **Fix:** Update app.py mode references

```
ValueError: Unknown subagent: species_explorer
```
→ **Fix:** Remove references to deleted subagent in supervisor or mode definitions

### 3. Test Agent Routing

Try queries that should route to different agents:

```python
# Test image_identifier routing
"What bird is this? [upload image]"

# Test taxonomy_specialist routing
"Show me endangered bird families"

# Test your new agent
"Query specific to new agent capability"
```

### 4. Check Tool Access

Verify agents only use their assigned tools:

```bash
# In terminal, watch for:
[SUBAGENT]: Creating Image Identification Specialist
  • Tools: classify_from_url, classify_from_base64, get_bird_info, get_bird_images
```

### 5. Verify Provider-Specific Prompts

Test with different LLM providers:

```python
# OpenAI should use default prompts
# HuggingFace should use _HF prompts
```

---

## Quick Reference Checklist

**Adding a new subagent:**
- [ ] Define in `subagent_config.py` → `get_subagent_definitions()`
- [ ] Create prompts in `prompts.py` (default + HF versions)
- [ ] Add to PROMPTS dictionary
- [ ] Create agent in `subagent_supervisor.py`
- [ ] Add to supervisor list
- [ ] Update router prompts (default + HF)
- [ ] Update mode definition
- [ ] Update app.py mode references (5 locations)
- [ ] Test locally

**Removing a subagent:**
- [ ] Remove from `subagent_supervisor.py` workflow
- [ ] Update mode definition count and list
- [ ] Update router prompts (remove references)
- [ ] Update "When to defer" in remaining agents
- [ ] Update app.py mode references (5 locations)
- [ ] Test locally

**Modifying a subagent:**
- [ ] Update tools list in `subagent_config.py`
- [ ] Update prompts if needed
- [ ] Update router if responsibilities changed
- [ ] Test locally

---

## Troubleshooting

### Error: "Unknown mode"

**Cause:** Mode name mismatch between `subagent_config.py` and `app.py`

**Fix:** Search for all occurrences in app.py and update:
```bash
grep -n "Old Mode Name" app.py
```

### Error: "Unknown subagent"

**Cause:** Subagent referenced in supervisor but not defined in config

**Fix:** Either define the subagent or remove references

### Agent Not Using Expected Tools

**Cause:** Tool name mismatch or tool not available

**Fix:** Check MCP server is providing the tool:
```python
print([tool.name for tool in all_tools])
```

---

## Best Practices

1. **Always update both default and HuggingFace prompts** - HF models need more explicit instructions
2. **Keep tool lists minimal** - Only give agents tools they truly need
3. **Update router prompts** - Supervisor needs to know when to use your agent
4. **Test routing logic** - Verify supervisor correctly delegates tasks
5. **Document agent responsibilities** - Clear "Your Role" section in prompts
6. **Use provider-specific prompts** - Optimize for OpenAI vs Anthropic vs HuggingFace
7. **Keep "When to defer" up to date** - Agents should know their boundaries

---

## Example: Recent Changes

**Evolution of the supervisor mode:**

1. **Removed `species_explorer`** - Streamlined from 3 to 2 specialists
2. **Added `generalist` (audio finder)** - Back to 3 specialists with audio capabilities
3. **Renamed mode** - "Specialized Subagents (3 Specialists)" → "Supervisor (Multi-Agent)"

**Files changed:**
1. `subagent_supervisor.py` - Added/removed agents from supervisor list
2. `subagent_config.py` - Updated mode definition and router prompts
3. `prompts.py` - Updated HuggingFace router prompt, removed pirate personality
4. `subagent_config.py` - Updated "When to defer" sections
5. `app.py` - Updated all 5 mode references throughout

**Current architecture:** Single unified "Supervisor (Multi-Agent)" mode with 3 specialists (image_identifier, taxonomy_specialist, generalist)

---

**Questions?** Check the LangGraph documentation: https://langchain-ai.github.io/langgraph/