docs/bugs/P0_ADVANCED_MODE_TIMEOUT_NO_SYNTHESIS.md

P0 - Advanced Mode Timeout Yields False "Synthesizing" Message

Status: RESOLVED Priority: P0 (Blocker for Advanced/Magentic mode) Found: 2025-11-30 (Manual Testing) Resolved: 2025-11-30 Component: src/orchestrators/advanced.py

Resolution Summary

The issue where Advanced Mode timeouts produced a fake synthesis message has been fully resolved. We implemented a robust fallback mechanism that synthesizes a report from collected evidence upon timeout.

Fix Details

1. Implemented ResearchMemory.get_context_summary():

   • Added missing method to src/services/research_memory.py.
   • Generates a structured summary of hypotheses and top 20 evidence items.
   • Enables the ReportAgent to function even without a formal handoff from JudgeAgent.

2. Fixed Factory Configuration:

   • Updated src/orchestrators/factory.py to use settings.advanced_max_rounds (default 5).
   • Previously used global max_iterations (default 10), causing workflows to run 2x longer than intended and hitting timeouts.

3. Implemented Timeout Synthesis Logic:

   • Updated src/orchestrators/advanced.py to catch TimeoutError.
   • Now retrieves get_context_summary() from memory.
   • Directly invokes ReportAgent to generate a final report from available evidence.
   • Yields the actual report content instead of a static placeholder message.

Verification

• Unit Tests: tests/unit/orchestrators/test_advanced_timeout.py verifies:
  • Timeout triggers synthesis (mocked ReportAgent is called).
  • Factory correctly sets max_rounds=5.
• Manual Verification:
  • Confirmed logic flow via TDD.
  • SearchAgent verbosity mitigated by reduced round count (5 rounds = ~20KB context vs 40KB+).

---

Symptom (Archive)

When using Advanced mode (Magentic/Multi-Agent) with an OpenAI API key, the workflow:

1. Starts correctly ("Starting research (Advanced mode)")
2. Shows "Multi-agent reasoning in progress (10 rounds max)"
3. Streams SearchAgent results successfully
4. Shows "Round 1/10" progress
5. Then hangs for ~5 minutes (timeout period)
6. Finally shows: "Research timed out. Synthesizing available evidence..."
7. BUT NO SYNTHESIS OCCURS - the output ends there

User sees massive streaming output from SearchAgent but NO final research report.

Observed Output

🚀 **STARTED**: Starting research (Advanced mode): Clinical trials for PDE5 inhibitors alternatives?
⏳ **THINKING**: Multi-agent reasoning in progress (10 rounds max)...
🧠 **JUDGING**: Manager (user_task): Research sexual health and wellness interventions...
📡 **STREAMING**: [MASSIVE SearchAgent output - 10KB+ of clinical trial data]
⏱️ **PROGRESS**: Round 1/10 (~6m 45s remaining)
📚 **SEARCH_COMPLETE**: searcher: Below is a structured evidence dataset...

Research timed out. Synthesizing available evidence...
[END - Nothing more happens]

Root Cause Analysis

Bug Location: src/orchestrators/advanced.py:254-261

except TimeoutError:
    logger.warning("Workflow timed out", iterations=iteration)
    yield AgentEvent(
        type="complete",
        message="Research timed out. Synthesizing available evidence...",  # <-- LIE
        data={"reason": "timeout", "iterations": iteration},
        iteration=iteration,
    )

The message is a lie. It says "Synthesizing available evidence..." but:

1. No synthesis code is called
2. The MagenticState (containing gathered evidence) is never accessed
3. The ReportAgent is never invoked
4. User just sees the raw streaming output

Secondary Issue: Workflow Never Progresses Past Round 1

The SearchAgent produces a MASSIVE response (10KB+) in Round 1, but the workflow appears to stall and never delegate to:

• HypothesisAgent
• JudgeAgent
• ReportAgent

This suggests the Manager agent may be:

1. Overwhelmed by the verbose SearchAgent output
2. Stuck in a decision loop
3. Not receiving proper signals to delegate to next agent

Configuration Issue: Wrong max_rounds Used

File: src/orchestrators/factory.py:93-97

return orchestrator_cls(
    max_rounds=effective_config.max_iterations,  # <-- Uses max_iterations (10)
    api_key=api_key,
    domain=domain,
)

The factory passes max_iterations (10) instead of using settings.advanced_max_rounds (5). This means timeout is more likely since workflows run longer.

Impact

• User Experience: After waiting 5+ minutes, users get NO useful output
• Demo Killer: Advanced mode is effectively broken for external users
• Misleading UX: Message claims synthesis is happening when it's not

Proposed Fix

Fix 1: Implement Actual Timeout Synthesis

File: src/orchestrators/advanced.py

except TimeoutError:
    logger.warning("Workflow timed out", iterations=iteration)

    # ACTUALLY synthesize from gathered evidence
    try:
        from src.agents.state import get_magentic_state
        from src.agents.magentic_agents import create_report_agent

        state = get_magentic_state()
        memory: ResearchMemory = state.memory

        # Get evidence summary from memory
        evidence_summary = await memory.get_context_summary()

        # Create and invoke ReportAgent for synthesis
        report_agent = create_report_agent(self._chat_client, domain=self.domain)
        synthesis_result = await report_agent.invoke(
            f"Synthesize research report from this evidence:\n{evidence_summary}"
        )

        yield AgentEvent(
            type="complete",
            message=synthesis_result,
            data={"reason": "timeout_synthesis", "iterations": iteration},
            iteration=iteration,
        )
    except Exception as synth_error:
        logger.error("Timeout synthesis failed", error=str(synth_error))
        yield AgentEvent(
            type="complete",
            message=(
                f"Research timed out after {iteration} rounds. "
                f"Evidence gathered but synthesis failed: {synth_error}"
            ),
            data={"reason": "timeout_synthesis_failed", "iterations": iteration},
            iteration=iteration,
        )

Fix 2: Address SearchAgent Verbosity

The SearchAgent is producing large outputs (~4KB per search, accumulating to 40KB+ over 10 rounds), which overwhelms the Manager's context window. Consider:

1. Limiting SearchAgent output length further (currently 300 chars/result)
2. Summarizing results before returning to Manager
3. Using structured output format instead of prose

Fix 3: Use Correct max_rounds

File: src/orchestrators/factory.py

# Use advanced-specific setting, not max_iterations
return orchestrator_cls(
    max_rounds=settings.advanced_max_rounds,  # 5 by default
    api_key=api_key,
    domain=domain,
)

Fix 4: Implement get_context_summary in ResearchMemory

File: src/services/research_memory.py

The ResearchMemory class is missing the get_context_summary method required by Fix 1.

    async def get_context_summary(self) -> str:
        """Generate a summary of all collected evidence for the final report."""
        if not self.evidence_ids:
            return "No evidence collected."
        
        summary = [f"Research Query: {self.query}\n"]
        
        # Add Hypotheses
        if self.hypotheses:
            summary.append("## Hypotheses")
            for h in self.hypotheses:
                summary.append(f"- {h.drug} -> {h.target}: {h.effect} (Conf: {h.confidence})")
            summary.append("")

        # Add Top Evidence (limit to avoid token overflow)
        # We use get_all_evidence() but might need to summarize if too large
        evidence = self.get_all_evidence()
        summary.append(f"## Evidence ({len(evidence)} items)")
        
        # Group by source for cleaner summary
        for i, ev in enumerate(evidence[:20], 1):  # Limit to top 20 items
            summary.append(f"{i}. {ev.citation.title} ({ev.citation.date})")
            summary.append(f"   {ev.content[:200]}...") # Brief snippet
        
        return "\n".join(summary)

Call Stack Trace

app.py:research_agent()
  → configure_orchestrator(mode="advanced")
    → factory.py:create_orchestrator()
      → AdvancedOrchestrator(max_rounds=10)  # Should be 5

  → orchestrator.run(query)
    → advanced.py:run()
      → init_magentic_state(query)
      → workflow = _build_workflow()  # MagenticBuilder
      → async for event in workflow.run_stream(task):
          # SearchAgent runs (accumulates 4KB+ per round)
          # Manager receives, but never delegates further
          # TimeoutError after 300 seconds
      → except TimeoutError:
          → yield AgentEvent(message="Synthesizing...")  # LIE - no synthesis

Files to Modify

File	Change
src/orchestrators/advanced.py:254-261	Implement actual synthesis on timeout
src/orchestrators/factory.py:93-97	Use settings.advanced_max_rounds
src/services/research_memory.py	Implement get_context_summary() method
src/agents/magentic_agents.py	Consider limiting SearchAgent output

Test Plan

Unit Tests

# tests/unit/orchestrators/test_advanced_timeout.py

@pytest.mark.asyncio
async def test_timeout_synthesizes_evidence():
    """Timeout should produce synthesis, not empty message."""
    orchestrator = AdvancedOrchestrator(
        max_rounds=1,
        timeout_seconds=0.1,  # Force immediate timeout
        api_key="sk-test",
    )

    events = [e async for e in orchestrator.run("test query")]
    complete_event = [e for e in events if e.type == "complete"][-1]

    # Should contain synthesis, not just "timed out"
    assert "Research timed out" not in complete_event.message or \
           len(complete_event.message) > 100  # Actual content present

@pytest.mark.asyncio
async def test_factory_uses_advanced_max_rounds():
    """Factory should use settings.advanced_max_rounds for advanced mode."""
    orchestrator = create_orchestrator(
        mode="advanced",
        api_key="sk-test",
    )
    assert orchestrator._max_rounds == settings.advanced_max_rounds

Manual Verification

1. Set OPENAI_API_KEY and run app
2. Select "Advanced" mode
3. Submit: "Clinical trials for PDE5 inhibitors alternatives?"
4. Wait for completion or timeout
5. Verify: Final output contains synthesized report (not just "timed out" message)

Related Issues

• This may be related to the SearchAgent being too verbose
• The Magentic pattern expects agents to produce concise outputs
• Microsoft Agent Framework's Manager may struggle with 10KB+ messages

Priority Justification

P0 because:

1. Advanced mode is a major selling point (multi-agent, deep research)
2. Users with paid API keys expect it to work
3. The current behavior is deceptive (claims synthesis, delivers nothing)
4. Demo credibility is destroyed when users wait 5min for nothing