docs/design_notes.md

📈 Detailed Working of the Web Research Agent

The Web Research Agent follows a modular, multi-step pipeline that transforms a user query into a reliable, summarized report. It leverages Google Gemini for natural language processing, Google Custom Search for information retrieval, and BeautifulSoup for scraping.

---

🔹 Step-by-Step Flow

1. User Input

• User enters a natural language research query in the Streamlit interface.
• Example: "India US trade deal 2025"

2. Query Analyzer (query_analyzer.py)

• Gemini 2.5 Pro model is used to extract structured metadata from the input query.
• Returns:
  • Intent (e.g., news, opinion, analysis)
  • Keywords (extracted for search)
  • Information Types (e.g., statistics, policy summaries)
  • Time Range (e.g., "last year")

3. Google Search Tool (search_tool.py)

• Uses the GOOGLE_CSE_API_KEY and GOOGLE_CSE_CX to perform a search via Google Custom Search API.
• Pulls top n results (default = 15) based on relevance to query keywords.
• Returns list of dictionaries with:
  • Title
  • URL
  • Snippet

4. Web Scraper Tool (scraper_tool.py)

• Visits each URL and extracts readable <p> tags using BeautifulSoup.
• Clips long text to 5000 characters for optimal LLM processing.
• Returns:
  • Page content
  • URL

5. Content Analyzer (content_analyzer.py)

• Uses Gemini 2.5 to summarize scraped content.
• Adds metadata such as:
  • Summary (in bullet points)
  • Content Type (e.g., "news report")
  • Relevance Rating (high, medium, low)

6. Synthesizer (synthesizer.py)

• Receives all article summaries and the original query.
• Synthesizes content using Gemini with the following logic:
  • Group similar insights across sources.
  • Highlight contradictions.
  • End with a unified "Final Takeaway".
• Returns the final report in Markdown format.

7. Streamlit Output (app.py)

• Displays the following to the user:
  • Sidebar: Query analysis & article summaries.
  • Main view: Top links and final synthesized report.

---

🛠️ Tech Stack & Tools

Component	Technology/Library
UI	Streamlit
LLM	Google Gemini 2.5 Pro
Search API	Google Custom Search (CSE)
Scraper	BeautifulSoup, Requests
Config Management	Python-dotenv

---

🚀 Example End-to-End Flow

Query: "Electric vehicle subsidies in Europe 2024"

Result:

• 15 relevant articles scraped
• 12 summaries processed
• Synthesized final markdown with policy trends, contradictions in subsidy effectiveness, and a closing insight.