ARCHITECTURE.md

## 🛠️ Technical Architecture

- **Frontend**: Gradio with modern, responsive UI
- **Backend**: Modal for serverless, scalable processing
- **Search Engine**: Tavily API for intelligent web discovery
- **AI Processing**: OpenAI GPT-4 with structured JSON outputs
- **Web Scraping**: BeautifulSoup with intelligent content extraction
- **Text Analysis**: TextStat for readability and linguistic metrics

## 🎨 Why SitegeistAI?

SitegeistAI stands out as the "Swiss Army knife" for web analysis by combining multiple analysis layers into a single, powerful platform. Unlike basic web scrapers or simple content analyzers, it provides:

- **Contextual Intelligence**: Understands content within its broader context
- **Scalable Processing**: Handles everything from single URLs to large-scale analysis
- **Marketing Focus**: Built specifically for marketing and content intelligence needs
- **AI-Native Design**: Leverages cutting-edge AI for deeper insights
- **Integration Ready**: MCP server capabilities for seamless tool integration

Whether you're conducting competitive research, planning content strategies, or analyzing market trends, SitegeistAI transforms raw web data into actionable intelligence that drives better decision-making.

## 🏗️ Architecture Overview

```mermaid
graph TB
    subgraph "Frontend Layer"
        UI["🎨 Gradio Interface"]
        TAB1["📋 Specific URLs Tab"]
        TAB2["🔍 Discovery Tab"]
    end
    
    subgraph "Application Layer"
        APP["🚀 SitegeistAI App<br/>(app.py)"]
        FUNC1["analyze_specific_urls()"]
        FUNC2["discover_and_analyze_web()"]
    end
    
    subgraph "Modal Cloud Infrastructure"
        MODAL["☁️ Modal App<br/>(sitegeist_core)"]
        SEARCH["🔍 tavily_search_engine()"]
        DEEP["🔬 deep_analyze_url()"]
        SWARM["🧠 swarm_analyze_urls()"]
        LLM["🤖 query_llm()"]
    end
    
    subgraph "External APIs"
        TAVILY["🌐 Tavily Search API"]
        OPENAI["🧠 OpenAI GPT-4 API"]
    end
    
    subgraph "Analysis Components"
        SCRAPE["📄 Web Scraping<br/>(BeautifulSoup)"]
        TEXT["📊 Text Analysis<br/>(TextStat)"]
        SEO["🔍 SEO Metrics"]
        SENTIMENT["😊 Sentiment Analysis"]
    end
    
    subgraph "Output Layer"
        JSON["📋 JSON Results"]
        INSIGHTS["💡 Marketing Insights"]
        THEMES["🎯 Content Themes"]
    end
    
    UI --> TAB1
    UI --> TAB2
    TAB1 --> FUNC1
    TAB2 --> FUNC2
    
    FUNC1 --> SWARM
    FUNC2 --> SEARCH
    FUNC2 --> SWARM
    
    SEARCH --> TAVILY
    SWARM --> DEEP
    DEEP --> SCRAPE
    DEEP --> TEXT
    DEEP --> SEO
    DEEP --> LLM
    SWARM --> LLM
    LLM --> OPENAI
    
    SCRAPE --> SENTIMENT
    TEXT --> JSON
    SEO --> JSON
    SENTIMENT --> JSON
    LLM --> INSIGHTS
    SWARM --> THEMES
    
    JSON --> UI
    INSIGHTS --> UI
    THEMES --> UI
    
    style UI fill:#e1f5fe
    style MODAL fill:#f3e5f5
    style TAVILY fill:#fff3e0
    style OPENAI fill:#e8f5e8
```

### 🔄 Data Flow

1. **User Input**: Users interact with the Gradio interface, choosing between specific URL analysis or web discovery
2. **Processing Layer**: Application functions coordinate between frontend and backend services
3. **Modal Infrastructure**: Cloud-based functions handle scalable processing and API orchestration
4. **External Integrations**: Tavily provides web search capabilities while OpenAI delivers AI-powered analysis
5. **Analysis Pipeline**: Multiple analysis components work in parallel to extract insights
6. **Results Aggregation**: Individual analyses are synthesized into comprehensive reports and themes
7. **Output Delivery**: Structured results are returned to users through the intuitive interface

## 🛠️ Development & Deployment

### **Package Structure**

SitegeistAI follows Modal's best practices for multi-file projects with a proper Python package structure:

```
SitegeistAI/
├── sitegeist_core/              # Core package
│   ├── __init__.py             # Package initialization & exports
│   ├── analysis_components.py  # Modular analysis classes
│   └── modal_functions.py      # Modal functions & app definition
├── app.py                      # Gradio frontend application
├── deploy.py                   # Deployment configuration
├── test_package_components.py  # Package test suite
└── requirements.txt            # Dependencies
```

### **Deployment Commands**

```bash
# Test the package locally
python test_package_components.py

# Deploy to Modal (recommended method)
modal deploy -m sitegeist_core.modal_functions

# Alternative deployment method
modal deploy deploy.py

# Run tests on Modal
modal run -m sitegeist_core.modal_functions

# Run Gradio frontend locally
python app.py
```

### **Development Workflow**

1. **Local Development**: Test individual components with `test_package_components.py`
2. **Package Testing**: Verify imports and functionality work correctly
3. **Modal Deployment**: Deploy using the module mode (`-m`) for proper package handling
4. **Integration Testing**: Run Modal functions remotely to test full pipeline