Upload folder using huggingface_hub

ARCHITECTURE.md

@@ -0,0 +1,346 @@
+# Deep Research System - Architecture Documentation
+
+## Overview
+This is an agentic research system built using OpenAI Agents SDK that performs deep research on user queries with quality evaluation and automatic improvement.
+
+## Agentic Framework: OpenAI Agents SDK
+
+The system uses **OpenAI Agents SDK** which provides:
+- **Agent**: An AI agent with specific instructions and capabilities
+- **Runner**: Executes agents and manages their execution
+- **Tools**: Agents can be converted to tools for use by other agents
+- **Structured Outputs**: Using Pydantic models for type-safe outputs
+- **Tracing**: Built-in tracing for debugging and monitoring
+
+## System Architecture
+
+```
+┌─────────────────────────────────────────────────────────────────┐
+│                        Gradio UI Layer                           │
+│  (deep_research.py) - User Interface                            │
+└────────────────────────────┬────────────────────────────────────┘
+                              │
+                              ▼
+┌─────────────────────────────────────────────────────────────────┐
+│                    Research Manager                              │
+│              (research_manager.py)                               │
+│  Orchestrates the entire research workflow                      │
+└────────────────────────────┬────────────────────────────────────┘
+                              │
+        ┌─────────────────────┼─────────────────────┐
+        │                     │                     │
+        ▼                     ▼                     ▼
+┌──────────────┐    ┌──────────────┐    ┌──────────────┐
+│ Clarifier    │    │ Planner      │    │ Search       │
+│ Agent        │    │ Agent        │    │ Agent        │
+└──────────────┘    └──────────────┘    └──────────────┘
+        │                     │                     │
+        │                     ▼                     │
+        │            ┌──────────────┐              │
+        │            │ Writer       │              │
+        │            │ Agent        │              │
+        │            └──────────────┘              │
+        │                     │                     │
+        │                     ▼                     │
+        │            ┌──────────────┐              │
+        │            │ Evaluator   │              │
+        │            │ Agent        │              │
+        │            └──────────────┘              │
+        │                     │                     │
+        │                     ▼                     │
+        │            ┌──────────────┐              │
+        │            │ Email        │              │
+        │            │ Agent        │              │
+        │            └──────────────┘              │
+        │                     │                     │
+        └─────────────────────┴─────────────────────┘
+                              │
+                              ▼
+                    ┌─────────────────┐
+                    │  Final Report   │
+                    │  (Markdown)     │
+                    └─────────────────┘
+```
+
+## Agent Details
+
+### 1. Clarifier Agent (`clarifier_agent.py`)
+**Purpose**: Generates 3 clarifying questions based on user query
+
+**Input**: User's research query
+**Output**: `Clarification` (Pydantic model with 3 questions)
+
+**Example**:
+- Query: "Latest AI Agent frameworks"
+- Output: ["What time period are you interested in?", "Are you looking for commercial or open-source?", "What specific use cases?"]
+
+---
+
+### 2. Planner Agent (`planner_agent.py`)
+**Purpose**: Creates a search plan based on query + clarification answers
+
+**Input**: 
+- Original query
+- 3 question-answer pairs (structured as `ResearchContext`)
+
+**Output**: `WebSearchPlan` (Pydantic model with list of search items)
+
+**Schema Used**:
+```python
+ResearchContext:
+  - original_query: str
+  - clarification_qa: List[QuestionAnswerPair]
+    - question: str
+    - answer: str
+
+WebSearchPlan:
+  - searches: List[WebSearchItem]
+    - query: str (search term)
+    - reason: str (why this search is important)
+```
+
+**Example**:
+- Query: "Latest AI Agent frameworks in 2025"
+- Answers: ["2025", "Both", "Enterprise automation"]
+- Output: 2 search terms with reasons
+
+---
+
+### 3. Search Agent (`search_agent.py`)
+**Purpose**: Performs web searches for each search term
+
+**Input**: Search term + reason
+**Output**: Search results (text)
+
+**Execution**: Runs in parallel for all searches
+
+---
+
+### 4. Writer Agent (`writer_agent.py`)
+**Purpose**: Writes comprehensive research report
+
+**Input**: 
+- Original query
+- Search results
+- Clarification Q&A (optional)
+- Evaluation feedback (optional, for regeneration)
+
+**Output**: `ReportData` (Pydantic model)
+```python
+ReportData:
+  - short_summary: str
+  - markdown_report: str (full report)
+  - follow_up_questions: List[str]
+```
+
+---
+
+### 5. Evaluator Agent (`evaluator_agent.py`)
+**Purpose**: Evaluates report quality and provides scores
+
+**Input**: 
+- Generated report
+- Original query
+- Clarification Q&A
+
+**Output**: `ReportEvaluation` (Pydantic model)
+```python
+ReportEvaluation:
+  - relevance_score: CriterionScore (1-5)
+  - clarification_usage_score: CriterionScore (1-5)
+  - formatting_score: CriterionScore (1-5)
+  - completeness_score: CriterionScore (1-5)
+  - overall_quality_score: CriterionScore (1-5)
+  - average_score: float (average of all scores)
+  - feedback: str (improvement suggestions)
+```
+
+**Evaluation Criteria**:
+1. **Relevance**: How well report addresses original query
+2. **Use of Clarifications**: Incorporation of user's clarification answers
+3. **Formatting & Clarity**: Structure and readability
+4. **Completeness**: Comprehensive coverage
+5. **Overall Quality**: General assessment
+
+---
+
+### 6. Email Agent (`email_agent.py`)
+**Purpose**: Sends the final report via email
+
+**Input**: Markdown report
+**Output**: Email sent confirmation
+
+**Technology**: Uses Resend API (SendGrid free tier was retired)
+
+---
+
+## Complete Workflow
+
+```
+┌─────────────────────────────────────────────────────────────────┐
+│                    WORKFLOW DIAGRAM                              │
+└─────────────────────────────────────────────────────────────────┘
+
+1. USER INPUT
+   │
+   ▼
+2. CLARIFIER AGENT
+   │  Generates 3 clarifying questions
+   │
+   ▼
+3. USER ANSWERS
+   │  User provides answers to questions
+   │
+   ▼
+4. PLANNER AGENT
+   │  Creates search plan using:
+   │  - Original query
+   │  - Question-Answer pairs (ResearchContext)
+   │
+   ▼
+5. PARALLEL SEARCHES
+   │  Search Agent runs for each search term
+   │  (Executed concurrently)
+   │
+   ▼
+6. WRITER AGENT
+   │  Generates comprehensive report using:
+   │  - Original query
+   │  - Search results
+   │  - Clarification Q&A
+   │
+   ▼
+7. EVALUATOR AGENT ⭐
+   │  Evaluates report quality:
+   │  - Scores 5 criteria (1-5 each)
+   │  - Calculates average score
+   │  - Provides feedback
+   │
+   ▼
+8. QUALITY CHECK
+   │
+   ├─→ Score >= 3.0? ──YES──→ 9. EMAIL AGENT
+   │                          Send report
+   │
+   └─→ Score < 3.0? ──NO──→ REGENERATION LOOP
+       │                    (Max 2 attempts)
+       │
+       ├─→ Attempt 1: Regenerate with feedback
+       │   └─→ Re-evaluate
+       │
+       ├─→ Attempt 2: Regenerate with feedback
+       │   └─→ Re-evaluate
+       │
+       └─→ Final: Send best report (even if < 3.0)
+           └─→ 9. EMAIL AGENT
+```
+
+## Data Flow
+
+```
+User Query (string)
+    │
+    ├─→ Clarifier Agent
+    │   └─→ Clarification (questions: List[str])
+    │
+    └─→ User Answers (List[str])
+        │
+        ├─→ ResearchContext (Pydantic)
+        │   ├─→ original_query: str
+        │   └─→ clarification_qa: List[QuestionAnswerPair]
+        │
+        ├─→ Planner Agent
+        │   └─→ WebSearchPlan
+        │       └─→ searches: List[WebSearchItem]
+        │
+        ├─→ Search Agent (parallel)
+        │   └─→ Search Results: List[str]
+        │
+        ├─→ Writer Agent
+        │   └─→ ReportData
+        │       ├─→ short_summary: str
+        │       ├─→ markdown_report: str
+        │       └─→ follow_up_questions: List[str]
+        │
+        ├��→ Evaluator Agent
+        │   └─→ ReportEvaluation
+        │       ├─→ Individual scores (5 criteria)
+        │       ├─→ average_score: float
+        │       └─→ feedback: str
+        │
+        └─→ Email Agent
+            └─→ Email sent (via Resend API)
+```
+
+## Key Features
+
+### 1. Structured Data with Pydantic
+All agents use Pydantic models for type-safe, validated outputs:
+- Ensures correct data structure
+- Provides clear schemas
+- Enables automatic validation
+
+### 2. Agent Orchestration
+The `ResearchManager` coordinates all agents:
+- Sequential execution where needed
+- Parallel execution for searches
+- Conditional logic for regeneration
+
+### 3. Quality Assurance Loop
+- Automatic evaluation after report generation
+- Regeneration if score < 3.0/5.0
+- Up to 2 regeneration attempts
+- Feedback-driven improvements
+
+### 4. Context Preservation
+- Original query maintained throughout
+- Clarification Q&A passed to planner and writer
+- Evaluation feedback used for regeneration
+
+## Technology Stack
+
+- **Framework**: OpenAI Agents SDK (`openai-agents`)
+- **UI**: Gradio (Python web interface)
+- **Email**: Resend API
+- **Type Safety**: Pydantic models
+- **Async**: Python asyncio for concurrent operations
+- **Tracing**: OpenAI platform tracing for debugging
+
+## File Structure
+
+```
+deep_research/
+├── deep_research.py          # Gradio UI
+├── research_manager.py        # Main orchestrator
+├── clarifier_agent.py         # Question generation
+├── planner_agent.py           # Search planning
+├── search_agent.py            # Web search execution
+├── writer_agent.py            # Report writing
+├── evaluator_agent.py         # Quality evaluation ⭐
+├── email_agent.py             # Email sending
+└── ARCHITECTURE.md           # This file
+```
+
+## Execution Flow Example
+
+1. **User**: "Latest AI Agent frameworks in 2025"
+2. **Clarifier**: Generates 3 questions
+3. **User**: Answers questions
+4. **Planner**: Creates 2 search terms based on query + answers
+5. **Searches**: Run in parallel, collect results
+6. **Writer**: Generates report incorporating clarifications
+7. **Evaluator**: Scores report (e.g., 2.8/5.0)
+8. **System**: Detects score < 3.0, regenerates with feedback
+9. **Evaluator**: Re-scores (e.g., 3.5/5.0)
+10. **System**: Approves, sends email
+11. **User**: Receives high-quality report
+
+## Benefits of This Architecture
+
+1. **Modularity**: Each agent has a single responsibility
+2. **Reusability**: Agents can be used as tools by other agents
+3. **Quality Control**: Automatic evaluation and improvement
+4. **User Experience**: Clarifying questions improve relevance
+5. **Transparency**: Users see scores and regeneration attempts
+6. **Scalability**: Easy to add new agents or modify workflow
+

README.md

@@ -1,12 +1,49 @@
 ---
-title: Deep Research
-emoji: 🚀
-colorFrom: gray
-colorTo: indigo
-sdk: gradio
-sdk_version: 6.2.0
+title: deep-research
 app_file: app.py
-pinned: false
+sdk: gradio
+sdk_version: 5.33.1
 ---
 
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+# Deep Research Multi-Agent System
+
+An intelligent research system that uses multiple AI agents to perform comprehensive research on any topic. The system includes:
+
+- **Clarifier Agent**: Generates 3 clarifying questions to better understand your research needs
+- **Planner Agent**: Creates an optimized search plan based on your query and clarifications
+- **Search Agent**: Performs web searches in parallel
+- **Writer Agent**: Synthesizes information into a comprehensive report
+- **Evaluator Agent**: Evaluates report quality and automatically improves it if needed
+- **Email Agent**: Sends the final report via email
+
+## Features
+
+- 🤔 **Clarifying Questions**: Get more targeted research by answering 3 clarifying questions
+- 🔍 **Intelligent Search Planning**: Search terms are optimized based on your clarifications
+- 📊 **Quality Evaluation**: Reports are automatically evaluated on 5 criteria (relevance, clarity, completeness, etc.)
+- 🔄 **Auto-Improvement**: Reports scoring below 3.0/5.0 are automatically regenerated with feedback
+- 📧 **Email Delivery**: Receive your research report via email
+
+## How to Use
+
+1. Enter your research query
+2. Answer the 3 clarifying questions that appear
+3. Wait for the system to:
+   - Plan searches
+   - Perform web searches
+   - Write the report
+   - Evaluate quality
+   - Improve if needed
+4. View your comprehensive research report
+
+## Environment Variables
+
+The following secrets need to be configured in Hugging Face Spaces:
+
+- `OPENAI_API_KEY`: Your OpenAI API key (required)
+- `RESEND_API_KEY`: Your Resend API key (for email functionality)
+
+## Architecture
+
+This system uses the OpenAI Agents SDK with a multi-agent architecture. See `ARCHITECTURE.md` for detailed documentation.
+

app.py

@@ -0,0 +1,117 @@
+import gradio as gr
+from dotenv import load_dotenv
+from research_manager import ResearchManager
+from clarifier_agent import clarifier_agent
+from agents import Runner
+
+load_dotenv(override=True)
+
+
+async def generate_questions(query: str):
+    """Automatically generate and display clarifying questions for the query"""
+    # Show loading message first
+    yield "Analyzing your query and generating clarifying questions...", []
+    
+    # Generate questions
+    result = await Runner.run(clarifier_agent, f"User query: {query}")
+    questions = result.final_output.questions
+    
+    # Format questions for display
+    questions_text = "**Please answer these clarifying questions:**\n\n"
+    for i, q in enumerate(questions, 1):
+        questions_text += f"{i}. {q}\n\n"
+    
+    # Yield the formatted questions and the questions list for state
+    yield questions_text, questions
+
+
+async def run_research(query: str, answer1: str, answer2: str, answer3: str, questions: list):
+    """Run the research process with clarification answers"""
+    manager = ResearchManager()
+    answers = [answer1, answer2, answer3]
+    
+    async for chunk in manager.run(query, questions, answers):
+        yield chunk
+
+
+with gr.Blocks(theme=gr.themes.Default(primary_hue="sky")) as ui:
+    gr.Markdown("# Deep Research")
+    
+    # State to store generated questions
+    questions_state = gr.State(value=[])
+    
+    query_textbox = gr.Textbox(
+        label="What topic would you like to research?",
+        placeholder="e.g., Latest AI Agent frameworks in 2025"
+    )
+    
+    submit_btn = gr.Button("Start Research", variant="primary")
+    
+    questions_output = gr.Markdown(
+        label="Clarifying Questions", 
+        value="*Enter your research query above and click 'Start Research' to generate clarifying questions.*",
+        visible=True
+    )
+    
+    with gr.Row():
+        answer1 = gr.Textbox(label="Answer 1", placeholder="Enter your answer to question 1", visible=False)
+        answer2 = gr.Textbox(label="Answer 2", placeholder="Enter your answer to question 2", visible=False)
+        answer3 = gr.Textbox(label="Answer 3", placeholder="Enter your answer to question 3", visible=False)
+    
+    continue_btn = gr.Button("Continue Research", variant="primary", visible=False)
+    report = gr.Markdown(label="Research Report")
+    
+    def show_questions_ui():
+        """Show the questions and answer fields"""
+        return (
+            gr.update(visible=True),  # questions_output
+            gr.update(visible=True),  # answer1
+            gr.update(visible=True),  # answer2
+            gr.update(visible=True),  # answer3
+            gr.update(visible=True),  # continue_btn
+            gr.update(visible=False)  # submit_btn
+        )
+    
+    def hide_questions_ui():
+        """Hide the questions and answer fields after research starts"""
+        return (
+            gr.update(visible=False),  # questions_output
+            gr.update(visible=False),  # answer1
+            gr.update(visible=False),  # answer2
+            gr.update(visible=False),  # answer3
+            gr.update(visible=False),  # continue_btn
+            gr.update(visible=True)    # submit_btn
+        )
+    
+    # When user submits query, automatically generate questions
+    submit_btn.click(
+        fn=generate_questions,
+        inputs=query_textbox,
+        outputs=[questions_output, questions_state]
+    ).then(
+        fn=show_questions_ui,
+        outputs=[questions_output, answer1, answer2, answer3, continue_btn, submit_btn]
+    )
+    
+    query_textbox.submit(
+        fn=generate_questions,
+        inputs=query_textbox,
+        outputs=[questions_output, questions_state]
+    ).then(
+        fn=show_questions_ui,
+        outputs=[questions_output, answer1, answer2, answer3, continue_btn, submit_btn]
+    )
+    
+    # Continue research with answers
+    continue_btn.click(
+        fn=run_research,
+        inputs=[query_textbox, answer1, answer2, answer3, questions_state],
+        outputs=report
+    ).then(
+        fn=hide_questions_ui,
+        outputs=[questions_output, answer1, answer2, answer3, continue_btn, submit_btn]
+    )
+
+if __name__ == "__main__":
+    ui.launch()
+

clarifier_agent.py

@@ -0,0 +1,43 @@
+from pydantic import BaseModel, Field
+from agents import Agent
+from typing import List
+
+import os
+from dotenv import load_dotenv
+
+load_dotenv(override=True)
+
+openai_api_key = os.getenv('OPENAI_API_KEY')
+
+INSTRUCTIONS = """You are a helpful research assistant. 
+Given a user's research query, you come up with 3 clarifying questions to better understand what they need.
+
+Rules:
+    - Keep the questions short (one sentence each question)
+    - Do not answer the questions yourself
+    - Ask exactly 3 questions
+    - Every question must end with a '?'
+    - Focus on understanding: scope, depth, timeframe, specific aspects, or context
+    - Make questions relevant to the research query"""
+
+
+class Clarification(BaseModel):
+    questions: List[str] = Field(
+        description="A list of exactly 3 clarifying questions",
+        min_items=3,
+        max_items=3
+    )
+
+
+clarifier_agent = Agent(
+    name='ClarifyAgent',
+    instructions=INSTRUCTIONS,
+    model='gpt-4o-mini',
+    output_type=Clarification
+)
+
+clarify_tool = clarifier_agent.as_tool(
+    tool_name='clarify',
+    tool_description='Ask 3 clarifying questions based on the user query'
+)
+

deep_research.py

@@ -0,0 +1,116 @@
+import gradio as gr
+from dotenv import load_dotenv
+from research_manager import ResearchManager
+from clarifier_agent import clarifier_agent
+from agents import Runner
+
+load_dotenv(override=True)
+
+
+async def generate_questions(query: str):
+    """Automatically generate and display clarifying questions for the query"""
+    # Show loading message first
+    yield "Analyzing your query and generating clarifying questions...", []
+    
+    # Generate questions
+    result = await Runner.run(clarifier_agent, f"User query: {query}")
+    questions = result.final_output.questions
+    
+    # Format questions for display
+    questions_text = "**Please answer these clarifying questions:**\n\n"
+    for i, q in enumerate(questions, 1):
+        questions_text += f"{i}. {q}\n\n"
+    
+    # Yield the formatted questions and the questions list for state
+    yield questions_text, questions
+
+
+async def run_research(query: str, answer1: str, answer2: str, answer3: str, questions: list):
+    """Run the research process with clarification answers"""
+    manager = ResearchManager()
+    answers = [answer1, answer2, answer3]
+    
+    async for chunk in manager.run(query, questions, answers):
+        yield chunk
+
+
+with gr.Blocks(theme=gr.themes.Default(primary_hue="sky")) as ui:
+    gr.Markdown("# Deep Research")
+    
+    # State to store generated questions
+    questions_state = gr.State(value=[])
+    
+    query_textbox = gr.Textbox(
+        label="What topic would you like to research?",
+        placeholder="e.g., Latest AI Agent frameworks in 2025"
+    )
+    
+    submit_btn = gr.Button("Start Research", variant="primary")
+    
+    questions_output = gr.Markdown(
+        label="Clarifying Questions", 
+        value="*Enter your research query above and click 'Start Research' to generate clarifying questions.*",
+        visible=True
+    )
+    
+    with gr.Row():
+        answer1 = gr.Textbox(label="Answer 1", placeholder="Enter your answer to question 1", visible=False)
+        answer2 = gr.Textbox(label="Answer 2", placeholder="Enter your answer to question 2", visible=False)
+        answer3 = gr.Textbox(label="Answer 3", placeholder="Enter your answer to question 3", visible=False)
+    
+    continue_btn = gr.Button("Continue Research", variant="primary", visible=False)
+    report = gr.Markdown(label="Research Report")
+    
+    def show_questions_ui():
+        """Show the questions and answer fields"""
+        return (
+            gr.update(visible=True),  # questions_output
+            gr.update(visible=True),  # answer1
+            gr.update(visible=True),  # answer2
+            gr.update(visible=True),  # answer3
+            gr.update(visible=True),  # continue_btn
+            gr.update(visible=False)  # submit_btn
+        )
+    
+    def hide_questions_ui():
+        """Hide the questions and answer fields after research starts"""
+        return (
+            gr.update(visible=False),  # questions_output
+            gr.update(visible=False),  # answer1
+            gr.update(visible=False),  # answer2
+            gr.update(visible=False),  # answer3
+            gr.update(visible=False),  # continue_btn
+            gr.update(visible=True)    # submit_btn
+        )
+    
+    # When user submits query, automatically generate questions
+    submit_btn.click(
+        fn=generate_questions,
+        inputs=query_textbox,
+        outputs=[questions_output, questions_state]
+    ).then(
+        fn=show_questions_ui,
+        outputs=[questions_output, answer1, answer2, answer3, continue_btn, submit_btn]
+    )
+    
+    query_textbox.submit(
+        fn=generate_questions,
+        inputs=query_textbox,
+        outputs=[questions_output, questions_state]
+    ).then(
+        fn=show_questions_ui,
+        outputs=[questions_output, answer1, answer2, answer3, continue_btn, submit_btn]
+    )
+    
+    # Continue research with answers
+    continue_btn.click(
+        fn=run_research,
+        inputs=[query_textbox, answer1, answer2, answer3, questions_state],
+        outputs=report
+    ).then(
+        fn=hide_questions_ui,
+        outputs=[questions_output, answer1, answer2, answer3, continue_btn, submit_btn]
+    )
+
+ui.launch(inbrowser=True)
+

email_agent.py

@@ -0,0 +1,48 @@
+import os
+from typing import Dict
+
+import resend
+from agents import Agent, function_tool
+
+# Note: SendGrid free plans were retired (May 2025). Using Resend instead.
+# Resend free tier: 3,000 emails/month
+# Get API key from: https://resend.com/api-keys
+# Add to .env: RESEND_API_KEY=re_xxxxx
+
+
+@function_tool
+def send_email(subject: str, html_body: str) -> Dict[str, str]:
+    """Send an email with the given subject and HTML body using Resend"""
+    api_key = os.environ.get('RESEND_API_KEY')
+    if not api_key:
+        return {"status": "error", "message": "RESEND_API_KEY not found in environment variables"}
+    
+    resend.api_key = api_key
+    
+    from_email = "onboarding@resend.dev"  # Change to your verified sender
+    to_email = "anmolkumarimusician@gmail.com"  # Change to your recipient
+    
+    try:
+        r = resend.Emails.send({
+            "from": from_email,
+            "to": to_email,
+            "subject": subject,
+            "html": html_body
+        })
+        print(f"Email sent successfully! Email ID: {r.get('id', 'N/A')}")
+        return {"status": "success", "id": r.get('id')}
+    except Exception as e:
+        print(f"Failed to send email: {str(e)}")
+        return {"status": "error", "message": f"Failed to send email: {str(e)}"}
+
+
+INSTRUCTIONS = """You are able to send a nicely formatted HTML email based on a detailed report.
+You will be provided with a detailed report. You should use your tool to send one email, providing the 
+report converted into clean, well presented HTML with an appropriate subject line."""
+
+email_agent = Agent(
+    name="Email agent",
+    instructions=INSTRUCTIONS,
+    tools=[send_email],
+    model="gpt-4o-mini",
+)

evaluator_agent.py

@@ -0,0 +1,49 @@
+from pydantic import BaseModel, Field
+from agents import Agent
+from typing import List
+
+INSTRUCTIONS = """You are an expert evaluator for research reports. Your task is to evaluate a research report 
+based on the following criteria:
+
+1. **Relevance**: How well does the report address the original query?
+2. **Use of Clarifications**: How well does the report incorporate the clarification answers provided by the user?
+3. **Formatting and Clarity**: Is the report well-structured, clearly formatted, and easy to read?
+4. **Completeness**: Does the report provide comprehensive information on the topic?
+5. **Overall Quality**: Overall assessment of the report's quality and usefulness
+
+For each criterion, provide a score from 1 to 5 (where 1 is poor and 5 is excellent) and a brief justification.
+Calculate an overall average score by taking the mean of all 5 criterion scores.
+
+In your feedback, be specific about:
+- What areas need improvement
+- How to better incorporate the clarification answers
+- Formatting and structure suggestions
+- Any missing information or gaps
+
+Be thorough but fair in your evaluation. Consider that the report should directly address the user's original query 
+and incorporate insights from their clarification answers."""
+
+
+class CriterionScore(BaseModel):
+    criterion: str = Field(description="The name of the evaluation criterion")
+    score: int = Field(description="Score from 1 to 5", ge=1, le=5)
+    justification: str = Field(description="Brief explanation for the score")
+
+
+class ReportEvaluation(BaseModel):
+    relevance_score: CriterionScore = Field(description="Score for relevance to original query")
+    clarification_usage_score: CriterionScore = Field(description="Score for use of clarification answers")
+    formatting_score: CriterionScore = Field(description="Score for formatting and clarity")
+    completeness_score: CriterionScore = Field(description="Score for completeness of information")
+    overall_quality_score: CriterionScore = Field(description="Overall quality assessment")
+    average_score: float = Field(description="Average of all scores (out of 5)")
+    feedback: str = Field(description="Overall feedback and suggestions for improvement")
+
+
+evaluator_agent = Agent(
+    name="EvaluatorAgent",
+    instructions=INSTRUCTIONS,
+    model="gpt-4o-mini",
+    output_type=ReportEvaluation,
+)
+

planner_agent.py

@@ -0,0 +1,50 @@
+from pydantic import BaseModel, Field
+from agents import Agent
+from typing import List
+
+HOW_MANY_SEARCHES = 2
+
+
+class QuestionAnswerPair(BaseModel):
+    question: str = Field(description="The clarifying question that was asked")
+    answer: str = Field(description="The user's answer to the clarifying question")
+
+
+class ResearchContext(BaseModel):
+    original_query: str = Field(description="The original research query from the user")
+    clarification_qa: List[QuestionAnswerPair] = Field(
+        description="List of question-answer pairs from the clarification process",
+        min_items=3,
+        max_items=3
+    )
+
+
+class WebSearchItem(BaseModel):
+    reason: str = Field(description="Your reasoning for why this search is important to the query.")
+    query: str = Field(description="The search term to use for the web search.")
+
+
+class WebSearchPlan(BaseModel):
+    searches: List[WebSearchItem] = Field(description="A list of web searches to perform to best answer the query.")
+    
+INSTRUCTIONS = f"""You are a helpful research assistant. Given a research context that includes:
+1. The original user query
+2. Three clarifying questions and their answers
+
+Your task is to come up with {HOW_MANY_SEARCHES} web search terms that will best answer the user's query, 
+taking into account both the original query AND the clarification answers provided.
+
+The clarification answers provide important context about:
+- What specific aspects the user is interested in
+- The scope and depth they want
+- Timeframe or other constraints
+- Specific focus areas
+
+Use this information to create more targeted and relevant search terms. Output exactly {HOW_MANY_SEARCHES} search terms."""
+
+planner_agent = Agent(
+    name="PlannerAgent",
+    instructions=INSTRUCTIONS,
+    model="gpt-4o-mini",
+    output_type=WebSearchPlan,
+)

requirements.txt

@@ -0,0 +1,9 @@
+gradio>=5.22.0
+openai>=1.68.2
+openai-agents>=0.0.15
+python-dotenv>=1.0.1
+resend>=1.0.0
+pydantic>=2.0.0
+typing-extensions>=4.0.0
+httpx>=0.28.1
+

research_manager.py

@@ -0,0 +1,248 @@
+from agents import Runner, trace, gen_trace_id
+from search_agent import search_agent
+from planner_agent import planner_agent, WebSearchItem, WebSearchPlan, ResearchContext, QuestionAnswerPair
+from writer_agent import writer_agent, ReportData
+from email_agent import email_agent
+from clarifier_agent import clarifier_agent
+from evaluator_agent import evaluator_agent, ReportEvaluation
+from typing import Optional, List
+import asyncio
+
+class ResearchManager:
+
+    async def run(self, query: str, clarification_questions: Optional[List[str]] = None, clarification_answers: Optional[List[str]] = None):
+        """ Run the deep research process, yielding the status updates and the final report"""
+        trace_id = gen_trace_id()
+        with trace("Research trace", trace_id=trace_id):
+            print(f"View trace: https://platform.openai.com/traces/trace?trace_id={trace_id}")
+            yield f"View trace: https://platform.openai.com/traces/trace?trace_id={trace_id}"
+            
+            # Step 1: Get clarifying questions and answers
+            if clarification_questions is None or clarification_answers is None:
+                yield "Generating clarifying questions..."
+                clarification_result = await Runner.run(clarifier_agent, f"User query: {query}")
+                questions = clarification_result.final_output.questions
+                questions_text = "**Please answer these clarifying questions:**\n\n"
+                for i, q in enumerate(questions, 1):
+                    questions_text += f"{i}. {q}\n\n"
+                yield questions_text
+                return  # Stop here to wait for user answers
+            
+            # Step 2: Continue with structured research context
+            print("Starting research with clarifications...")
+            yield "Using your clarifications to refine the research...\n\n"
+            
+            # Step 3: Plan searches using structured context
+            search_plan = await self.plan_searches(query, clarification_questions, clarification_answers)
+            yield "Searches planned, starting to search..."     
+            
+            # Step 4: Perform searches
+            search_results = await self.perform_searches(search_plan)
+            yield "Searches complete, writing report..."
+            
+            # Step 5: Write report
+            report = await self.write_report(query, search_results, clarification_questions, clarification_answers)
+            yield "Report written, evaluating quality..."
+            
+            # Step 6: Evaluate report
+            evaluation = await self.evaluate_report(report, query, clarification_questions, clarification_answers)
+            
+            # Format detailed evaluation results
+            eval_details = self._format_evaluation(evaluation, attempt=1)
+            yield eval_details
+            
+            # Step 7: If score is low, regenerate report with feedback
+            max_attempts = 2
+            attempt = 1
+            regeneration_attempted = False
+            
+            while evaluation.average_score < 3.0 and attempt < max_attempts:
+                regeneration_attempted = True
+                attempt += 1
+                yield f"\n⚠️ **Report score ({evaluation.average_score:.2f}/5.0) is below threshold (3.0)**\n**Regenerating with improvements (Attempt {attempt})...**\n"
+                
+                report = await self.write_report(
+                    query, 
+                    search_results, 
+                    clarification_questions, 
+                    clarification_answers,
+                    feedback=evaluation.feedback
+                )
+                evaluation = await self.evaluate_report(report, query, clarification_questions, clarification_answers)
+                
+                # Format evaluation results for regenerated report
+                eval_details = self._format_evaluation(evaluation, attempt=attempt, is_regeneration=True)
+                yield eval_details
+            
+            # Final status
+            if regeneration_attempted:
+                if evaluation.average_score < 3.0:
+                    yield f"\n⚠️ **Final Status**: Score {evaluation.average_score:.2f}/5.0 (below threshold, max attempts reached)"
+                else:
+                    yield f"\n✅ **Final Status**: Report improved! Final score: {evaluation.average_score:.2f}/5.0"
+            else:
+                yield f"\n✅ **Final Status**: Report quality approved! Score: {evaluation.average_score:.2f}/5.0"
+            
+            # Step 8: Send email
+            yield "Sending email..."
+            await self.send_email(report)
+            yield "Email sent, research complete"
+            yield report.markdown_report
+
+    async def plan_searches(self, query: str, questions: List[str], answers: List[str]) -> WebSearchPlan:
+        """ Plan the searches to perform using the structured research context """
+        print("Planning searches...")
+        
+        # Create structured research context with question-answer pairs
+        qa_pairs = [
+            QuestionAnswerPair(question=q, answer=a)
+            for q, a in zip(questions, answers)
+        ]
+        
+        research_context = ResearchContext(
+            original_query=query,
+            clarification_qa=qa_pairs
+        )
+        
+        # Format the context for the planner agent
+        context_text = f"""Original Query: {research_context.original_query}
+
+Clarification Questions and Answers:
+"""
+        for i, qa in enumerate(research_context.clarification_qa, 1):
+            context_text += f"{i}. Question: {qa.question}\n   Answer: {qa.answer}\n\n"
+        
+        result = await Runner.run(
+            planner_agent,
+            context_text,
+        )
+        print(f"Will perform {len(result.final_output.searches)} searches")
+        return result.final_output_as(WebSearchPlan)
+
+    async def perform_searches(self, search_plan: WebSearchPlan) -> List[str]:
+        """ Perform the searches to perform for the query """
+        print("Searching...")
+        num_completed = 0
+        tasks = [asyncio.create_task(self.search(item)) for item in search_plan.searches]
+        results = []
+        for task in asyncio.as_completed(tasks):
+            result = await task
+            if result is not None:
+                results.append(result)
+            num_completed += 1
+            print(f"Searching... {num_completed}/{len(tasks)} completed")
+        print("Finished searching")
+        return results
+
+    async def search(self, item: WebSearchItem) -> Optional[str]:
+        """ Perform a search for the query """
+        input = f"Search term: {item.query}\nReason for searching: {item.reason}"
+        try:
+            result = await Runner.run(
+                search_agent,
+                input,
+            )
+            return str(result.final_output)
+        except Exception:
+            return None
+    
+    async def write_report(
+        self, 
+        query: str, 
+        search_results: List[str], 
+        clarification_questions: Optional[List[str]] = None,
+        clarification_answers: Optional[List[str]] = None,
+        feedback: Optional[str] = None
+    ) -> ReportData:
+        """ Write the report for the query """
+        print("Thinking about report...")
+        
+        # Build input with query, search results, and clarifications
+        input_parts = [f"Original query: {query}"]
+        input_parts.append(f"Summarized search results: {search_results}")
+        
+        if clarification_questions and clarification_answers:
+            input_parts.append("\nClarification Questions and Answers:")
+            for i, (q, a) in enumerate(zip(clarification_questions, clarification_answers), 1):
+                input_parts.append(f"{i}. Question: {q}\n   Answer: {a}")
+        
+        if feedback:
+            input_parts.append(f"\n\nIMPORTANT: Previous evaluation feedback for improvement:\n{feedback}\n\nPlease address these issues in your report.")
+        
+        input_text = "\n".join(input_parts)
+        
+        result = await Runner.run(
+            writer_agent,
+            input_text,
+        )
+
+        print("Finished writing report")
+        return result.final_output_as(ReportData)
+    
+    async def evaluate_report(
+        self, 
+        report: ReportData, 
+        query: str, 
+        clarification_questions: Optional[List[str]], 
+        clarification_answers: Optional[List[str]]
+    ) -> ReportEvaluation:
+        """ Evaluate the quality of the report """
+        print("Evaluating report...")
+        
+        # Build evaluation context
+        eval_context = f"""Original Query: {query}
+
+Clarification Questions and Answers:
+"""
+        if clarification_questions and clarification_answers:
+            for i, (q, a) in enumerate(zip(clarification_questions, clarification_answers), 1):
+                eval_context += f"{i}. Question: {q}\n   Answer: {a}\n\n"
+        
+        eval_context += f"""
+Report to Evaluate:
+{report.markdown_report}
+
+Please evaluate this report based on:
+1. How well it addresses the original query
+2. How well it incorporates the clarification answers
+3. Formatting and clarity
+4. Completeness
+5. Overall quality
+"""
+        
+        result = await Runner.run(
+            evaluator_agent,
+            eval_context,
+        )
+        
+        evaluation = result.final_output_as(ReportEvaluation)
+        print(f"Evaluation complete. Average score: {evaluation.average_score:.2f}/5.0")
+        return evaluation
+    
+    async def send_email(self, report: ReportData) -> None:
+        print("Writing email...")
+        result = await Runner.run(
+            email_agent,
+            report.markdown_report,
+        )
+        print("Email sent")
+        return report
+    
+    def _format_evaluation(self, evaluation: ReportEvaluation, attempt: int = 1, is_regeneration: bool = False) -> str:
+        """Format evaluation results for display"""
+        header = "🔄 **Regeneration Attempt " + str(attempt) + "**\n\n" if is_regeneration else "📊 **Evaluation Results**\n\n"
+        
+        details = f"""{header}
+**Overall Score: {evaluation.average_score:.2f}/5.0**
+
+**Detailed Scores:**
+- **Relevance**: {evaluation.relevance_score.score}/5 - {evaluation.relevance_score.justification}
+- **Use of Clarifications**: {evaluation.clarification_usage_score.score}/5 - {evaluation.clarification_usage_score.justification}
+- **Formatting & Clarity**: {evaluation.formatting_score.score}/5 - {evaluation.formatting_score.justification}
+- **Completeness**: {evaluation.completeness_score.score}/5 - {evaluation.completeness_score.justification}
+- **Overall Quality**: {evaluation.overall_quality_score.score}/5 - {evaluation.overall_quality_score.justification}
+
+**Feedback:**
+{evaluation.feedback}
+"""
+        return details

search_agent.py

@@ -0,0 +1,17 @@
+from agents import Agent, WebSearchTool, ModelSettings
+
+INSTRUCTIONS = (
+    "You are a research assistant. Given a search term, you search the web for that term and "
+    "produce a concise summary of the results. The summary must 2-3 paragraphs and less than 300 "
+    "words. Capture the main points. Write succintly, no need to have complete sentences or good "
+    "grammar. This will be consumed by someone synthesizing a report, so its vital you capture the "
+    "essence and ignore any fluff. Do not include any additional commentary other than the summary itself."
+)
+
+search_agent = Agent(
+    name="Search agent",
+    instructions=INSTRUCTIONS,
+    tools=[WebSearchTool(search_context_size="low")],
+    model="gpt-4o-mini",
+    model_settings=ModelSettings(tool_choice="required"),
+)

writer_agent.py

@@ -0,0 +1,30 @@
+from pydantic import BaseModel, Field
+from agents import Agent
+from typing import List
+
+INSTRUCTIONS = (
+    "You are a senior researcher tasked with writing a cohesive report for a research query. "
+    "You will be provided with the original query, and some initial research done by a research assistant.\n"
+    "You should first come up with an outline for the report that describes the structure and "
+    "flow of the report. Then, generate the report and return that as your final output.\n"
+    "The final output should be in markdown format, and it should be lengthy and detailed. Aim "
+    "for 5-10 pages of content, at least 1000 words.\n"
+    "If clarification questions and answers are provided, make sure to incorporate those insights into your report.\n"
+    "If feedback from evaluation is provided, address those specific points to improve the report."
+)
+
+
+class ReportData(BaseModel):
+    short_summary: str = Field(description="A short 2-3 sentence summary of the findings.")
+
+    markdown_report: str = Field(description="The final report")
+
+    follow_up_questions: List[str] = Field(description="Suggested topics to research further")
+
+
+writer_agent = Agent(
+    name="WriterAgent",
+    instructions=INSTRUCTIONS,
+    model="gpt-4o-mini",
+    output_type=ReportData,
+)