docs/bugs/archive/P0_ADVANCED_MODE_TIMEOUT_NO_SYNTHESIS.md

# P0 - Advanced Mode Timeout Yields False "Synthesizing" Message

**Status:** RESOLVED
**Priority:** P0 (Blocker for Advanced/Magentic mode)
**Found:** 2025-11-30 (Manual Testing)
**Resolved:** 2025-11-30
**Component:** `src/orchestrators/advanced.py`

## Resolution Summary

The issue where Advanced Mode timeouts produced a fake synthesis message has been fully resolved.
We implemented a robust fallback mechanism that synthesizes a report from collected evidence upon timeout.

### Fix Details

1.  **Implemented `ResearchMemory.get_context_summary()`**:
    -   Added missing method to `src/services/research_memory.py`.
    -   Generates a structured summary of hypotheses and top 20 evidence items.
    -   Enables the ReportAgent to function even without a formal handoff from JudgeAgent.

2.  **Fixed Factory Configuration**:
    -   Updated `src/orchestrators/factory.py` to use `settings.advanced_max_rounds` (default 5).
    -   Previously used global `max_iterations` (default 10), causing workflows to run 2x longer than intended and hitting timeouts.

3.  **Implemented Timeout Synthesis Logic**:
    -   Updated `src/orchestrators/advanced.py` to catch `TimeoutError`.
    -   Now retrieves `get_context_summary()` from memory.
    -   Directly invokes `ReportAgent` to generate a final report from available evidence.
    -   Yields the actual report content instead of a static placeholder message.

### Verification

-   **Unit Tests**: `tests/unit/orchestrators/test_advanced_timeout.py` verifies:
    -   Timeout triggers synthesis (mocked ReportAgent is called).
    -   Factory correctly sets `max_rounds=5`.
-   **Manual Verification**:
    -   Confirmed logic flow via TDD.
    -   SearchAgent verbosity mitigated by reduced round count (5 rounds = ~20KB context vs 40KB+).

---

## Symptom (Archive)

When using Advanced mode (Magentic/Multi-Agent) with an OpenAI API key, the workflow:

1. Starts correctly ("Starting research (Advanced mode)")
2. Shows "Multi-agent reasoning in progress (10 rounds max)"
3. Streams SearchAgent results successfully
4. Shows "Round 1/10" progress
5. Then hangs for ~5 minutes (timeout period)
6. Finally shows: **"Research timed out. Synthesizing available evidence..."**
7. **BUT NO SYNTHESIS OCCURS** - the output ends there

User sees massive streaming output from SearchAgent but NO final research report.

## Observed Output

```text
🚀 **STARTED**: Starting research (Advanced mode): Clinical trials for PDE5 inhibitors alternatives?
⏳ **THINKING**: Multi-agent reasoning in progress (10 rounds max)...
🧠 **JUDGING**: Manager (user_task): Research sexual health and wellness interventions...
📡 **STREAMING**: [MASSIVE SearchAgent output - 10KB+ of clinical trial data]
⏱️ **PROGRESS**: Round 1/10 (~6m 45s remaining)
📚 **SEARCH_COMPLETE**: searcher: Below is a structured evidence dataset...

Research timed out. Synthesizing available evidence...
[END - Nothing more happens]
```

## Root Cause Analysis

### Bug Location: `src/orchestrators/advanced.py:254-261`

```python
except TimeoutError:
    logger.warning("Workflow timed out", iterations=iteration)
    yield AgentEvent(
        type="complete",
        message="Research timed out. Synthesizing available evidence...",  # <-- LIE
        data={"reason": "timeout", "iterations": iteration},
        iteration=iteration,
    )
```

**The message is a lie.** It says "Synthesizing available evidence..." but:
1. No synthesis code is called
2. The `MagenticState` (containing gathered evidence) is never accessed
3. The `ReportAgent` is never invoked
4. User just sees the raw streaming output

### Secondary Issue: Workflow Never Progresses Past Round 1

The SearchAgent produces a MASSIVE response (10KB+) in Round 1, but the workflow appears to stall and never delegate to:
- HypothesisAgent
- JudgeAgent
- ReportAgent

This suggests the Manager agent may be:
1. Overwhelmed by the verbose SearchAgent output
2. Stuck in a decision loop
3. Not receiving proper signals to delegate to next agent

### Configuration Issue: Wrong `max_rounds` Used

**File:** `src/orchestrators/factory.py:93-97`

```python
return orchestrator_cls(
    max_rounds=effective_config.max_iterations,  # <-- Uses max_iterations (10)
    api_key=api_key,
    domain=domain,
)
```

The factory passes `max_iterations` (10) instead of using `settings.advanced_max_rounds` (5).
This means timeout is more likely since workflows run longer.

## Impact

- **User Experience:** After waiting 5+ minutes, users get NO useful output
- **Demo Killer:** Advanced mode is effectively broken for external users
- **Misleading UX:** Message claims synthesis is happening when it's not

## Proposed Fix

### Fix 1: Implement Actual Timeout Synthesis

**File:** `src/orchestrators/advanced.py`

```python
except TimeoutError:
    logger.warning("Workflow timed out", iterations=iteration)

    # ACTUALLY synthesize from gathered evidence
    try:
        from src.agents.state import get_magentic_state
        from src.agents.magentic_agents import create_report_agent

        state = get_magentic_state()
        memory: ResearchMemory = state.memory

        # Get evidence summary from memory
        evidence_summary = await memory.get_context_summary()

        # Create and invoke ReportAgent for synthesis
        report_agent = create_report_agent(self._chat_client, domain=self.domain)
        synthesis_result = await report_agent.invoke(
            f"Synthesize research report from this evidence:\n{evidence_summary}"
        )

        yield AgentEvent(
            type="complete",
            message=synthesis_result,
            data={"reason": "timeout_synthesis", "iterations": iteration},
            iteration=iteration,
        )
    except Exception as synth_error:
        logger.error("Timeout synthesis failed", error=str(synth_error))
        yield AgentEvent(
            type="complete",
            message=(
                f"Research timed out after {iteration} rounds. "
                f"Evidence gathered but synthesis failed: {synth_error}"
            ),
            data={"reason": "timeout_synthesis_failed", "iterations": iteration},
            iteration=iteration,
        )
```

### Fix 2: Address SearchAgent Verbosity

The SearchAgent is producing large outputs (~4KB per search, accumulating to 40KB+ over 10 rounds), which overwhelms the Manager's context window.
Consider:
1. Limiting SearchAgent output length further (currently 300 chars/result)
2. Summarizing results before returning to Manager
3. Using structured output format instead of prose

### Fix 3: Use Correct max_rounds

**File:** `src/orchestrators/factory.py`

```python
# Use advanced-specific setting, not max_iterations
return orchestrator_cls(
    max_rounds=settings.advanced_max_rounds,  # 5 by default
    api_key=api_key,
    domain=domain,
)
```

### Fix 4: Implement `get_context_summary` in ResearchMemory

**File:** `src/services/research_memory.py`

The `ResearchMemory` class is missing the `get_context_summary` method required by Fix 1.

```python
    async def get_context_summary(self) -> str:
        """Generate a summary of all collected evidence for the final report."""
        if not self.evidence_ids:
            return "No evidence collected."
        
        summary = [f"Research Query: {self.query}\n"]
        
        # Add Hypotheses
        if self.hypotheses:
            summary.append("## Hypotheses")
            for h in self.hypotheses:
                summary.append(f"- {h.drug} -> {h.target}: {h.effect} (Conf: {h.confidence})")
            summary.append("")

        # Add Top Evidence (limit to avoid token overflow)
        # We use get_all_evidence() but might need to summarize if too large
        evidence = self.get_all_evidence()
        summary.append(f"## Evidence ({len(evidence)} items)")
        
        # Group by source for cleaner summary
        for i, ev in enumerate(evidence[:20], 1):  # Limit to top 20 items
            summary.append(f"{i}. {ev.citation.title} ({ev.citation.date})")
            summary.append(f"   {ev.content[:200]}...") # Brief snippet
        
        return "\n".join(summary)
```

## Call Stack Trace

```
app.py:research_agent()
  → configure_orchestrator(mode="advanced")
    → factory.py:create_orchestrator()
      → AdvancedOrchestrator(max_rounds=10)  # Should be 5

  → orchestrator.run(query)
    → advanced.py:run()
      → init_magentic_state(query)
      → workflow = _build_workflow()  # MagenticBuilder
      → async for event in workflow.run_stream(task):
          # SearchAgent runs (accumulates 4KB+ per round)
          # Manager receives, but never delegates further
          # TimeoutError after 300 seconds
      → except TimeoutError:
          → yield AgentEvent(message="Synthesizing...")  # LIE - no synthesis
```

## Files to Modify

| File | Change |
|------|--------|
| `src/orchestrators/advanced.py:254-261` | Implement actual synthesis on timeout |
| `src/orchestrators/factory.py:93-97` | Use `settings.advanced_max_rounds` |
| `src/services/research_memory.py` | Implement `get_context_summary()` method |
| `src/agents/magentic_agents.py` | Consider limiting SearchAgent output |

## Test Plan

### Unit Tests

```python
# tests/unit/orchestrators/test_advanced_timeout.py

@pytest.mark.asyncio
async def test_timeout_synthesizes_evidence():
    """Timeout should produce synthesis, not empty message."""
    orchestrator = AdvancedOrchestrator(
        max_rounds=1,
        timeout_seconds=0.1,  # Force immediate timeout
        api_key="sk-test",
    )

    events = [e async for e in orchestrator.run("test query")]
    complete_event = [e for e in events if e.type == "complete"][-1]

    # Should contain synthesis, not just "timed out"
    assert "Research timed out" not in complete_event.message or \
           len(complete_event.message) > 100  # Actual content present

@pytest.mark.asyncio
async def test_factory_uses_advanced_max_rounds():
    """Factory should use settings.advanced_max_rounds for advanced mode."""
    orchestrator = create_orchestrator(
        mode="advanced",
        api_key="sk-test",
    )
    assert orchestrator._max_rounds == settings.advanced_max_rounds
```

### Manual Verification

1. Set `OPENAI_API_KEY` and run app
2. Select "Advanced" mode
3. Submit: "Clinical trials for PDE5 inhibitors alternatives?"
4. Wait for completion or timeout
5. **Verify:** Final output contains synthesized report (not just "timed out" message)

## Related Issues

- This may be related to the SearchAgent being too verbose
- The Magentic pattern expects agents to produce concise outputs
- Microsoft Agent Framework's Manager may struggle with 10KB+ messages

## Priority Justification

**P0 because:**
1. Advanced mode is a major selling point (multi-agent, deep research)
2. Users with paid API keys expect it to work
3. The current behavior is deceptive (claims synthesis, delivers nothing)
4. Demo credibility is destroyed when users wait 5min for nothing