# Deep Analysis API Documentation ## Overview The Deep Analysis system provides advanced multi-agent analytical capabilities that automatically generate comprehensive reports based on user goals. The system uses DSPy (Declarative Self-improving Language Programs) to orchestrate multiple AI agents and create detailed analytical insights. ## Key Features - **Multi-Agent Analysis**: Orchestrates multiple specialized agents (preprocessing, statistical analysis, machine learning, visualization) - **Template Integration**: Uses the user's active templates/agents for analysis - **Streaming Progress**: Real-time progress updates during analysis execution - **Report Persistence**: Stores complete analysis reports in database with metadata - **HTML Export**: Generates downloadable HTML reports with visualizations - **Credit Tracking**: Monitors token usage, costs, and credits consumed ## Template Integration The deep analysis system integrates with the user's active templates through the agent system: 1. **Agent Selection**: Uses agents from the user's active template preferences (configured via `/templates` endpoints) 2. **Default Agents**: Falls back to system default agents if user hasn't configured preferences: - `preprocessing` (both individual and planner variants) - `statistical_analytics` (both individual and planner variants) - `sk_learn` (both individual and planner variants) - `data_viz` (both individual and planner variants) 3. **Template Limits**: Respects the 10-template limit for planner performance optimization 4. **Dynamic Planning**: The planner automatically selects the most appropriate agents based on the analysis goal and available templates ## Analysis Flow The deep analysis process follows these steps: 1. **Question Generation** (20% progress): Generates 5 targeted analytical questions based on the user's goal 2. **Planning** (40% progress): Creates an optimized execution plan using available agents 3. **Agent Execution** (60% progress): Executes analysis using user's active templates 4. **Code Synthesis** (80% progress): Combines and optimizes code from all agents 5. **Code Execution** (85% progress): Runs the synthesized analysis code 6. **Synthesis** (90% progress): Synthesizes results into coherent insights 7. **Conclusion** (100% progress): Generates final conclusions and recommendations --- ## API Endpoints ### Create Deep Analysis Report **POST** `/deep_analysis/reports` Creates a new deep analysis report in the database. **Request Body:** ```json { "report_uuid": "string", "user_id": 123, "goal": "Analyze customer churn patterns", "status": "completed", "deep_questions": "1. What factors...\n2. How does...", "deep_plan": "{\n \"@preprocessing\": {\n \"create\": [...],\n \"use\": [...],\n \"instruction\": \"...\"\n }\n}", "summaries": ["Agent summary 1", "Agent summary 2"], "analysis_code": "import pandas as pd\n# Analysis code...", "plotly_figures": [{"data": [...], "layout": {...}}], "synthesis": ["Synthesis result 1"], "final_conclusion": "## Conclusion\nThe analysis reveals...", "html_report": "...", "report_summary": "Brief summary of findings", "progress_percentage": 100, "duration_seconds": 120, "credits_consumed": 5, "error_message": null, "model_provider": "anthropic", "model_name": "claude-sonnet-4-20250514", "total_tokens_used": 15000, "estimated_cost": 0.25, "steps_completed": ["questions", "planning", "execution", "synthesis", "conclusion"] } ``` **Response:** ```json { "report_id": 1, "report_uuid": "uuid-string", "user_id": 123, "goal": "Analyze customer churn patterns", "status": "completed", "start_time": "2024-01-01T12:00:00Z", "end_time": "2024-01-01T12:02:00Z", "duration_seconds": 120, "report_summary": "Brief summary of findings", "created_at": "2024-01-01T12:02:00Z", "updated_at": "2024-01-01T12:02:00Z" } ``` ### Get Deep Analysis Reports **GET** `/deep_analysis/reports` Retrieves a list of deep analysis reports with optional filtering. **Query Parameters:** - `user_id` (optional): Filter by user ID - `limit` (optional): Number of reports to return (1-100, default: 10) - `offset` (optional): Number of reports to skip (default: 0) - `status` (optional): Filter by status ("pending", "running", "completed", "failed") **Response:** ```json [ { "report_id": 1, "report_uuid": "uuid-string", "user_id": 123, "goal": "Analyze customer churn patterns", "status": "completed", "start_time": "2024-01-01T12:00:00Z", "end_time": "2024-01-01T12:02:00Z", "duration_seconds": 120, "report_summary": "Brief summary of findings", "created_at": "2024-01-01T12:02:00Z", "updated_at": "2024-01-01T12:02:00Z" } ] ``` ### Get User Historical Reports **GET** `/deep_analysis/reports/user_historical` Retrieves all historical deep analysis reports for a specific user. **Query Parameters:** - `user_id`: User ID (required) - `limit` (optional): Number of reports to return (1-100, default: 50) ### Get Report by ID **GET** `/deep_analysis/reports/{report_id}` Retrieves a complete deep analysis report by ID. **Query Parameters:** - `user_id` (optional): Ensures report belongs to specified user **Response:** ```json { "report_id": 1, "report_uuid": "uuid-string", "user_id": 123, "goal": "Analyze customer churn patterns", "status": "completed", "start_time": "2024-01-01T12:00:00Z", "end_time": "2024-01-01T12:02:00Z", "duration_seconds": 120, "deep_questions": "1. What factors contribute to churn?\n2. How does churn vary by segment?", "deep_plan": "{\n \"@preprocessing\": {...},\n \"@statistical_analytics\": {...}\n}", "summaries": ["Agent performed data cleaning...", "Statistical analysis revealed..."], "analysis_code": "import pandas as pd\n# Complete analysis code", "plotly_figures": [{"data": [...], "layout": {...}}], "synthesis": ["The analysis shows clear patterns..."], "final_conclusion": "## Conclusion\nCustomer churn is primarily driven by...", "html_report": "...", "report_summary": "Analysis of customer churn patterns reveals...", "progress_percentage": 100, "credits_consumed": 5, "error_message": null, "model_provider": "anthropic", "model_name": "claude-sonnet-4-20250514", "total_tokens_used": 15000, "estimated_cost": 0.25, "steps_completed": ["questions", "planning", "execution", "synthesis", "conclusion"], "created_at": "2024-01-01T12:02:00Z", "updated_at": "2024-01-01T12:02:00Z" } ``` ### Get Report by UUID **GET** `/deep_analysis/reports/uuid/{report_uuid}` Retrieves a complete deep analysis report by UUID. Same response format as get by ID. ### Delete Report **DELETE** `/deep_analysis/reports/{report_id}` Deletes a deep analysis report. **Query Parameters:** - `user_id` (optional): Ensures report belongs to specified user **Response:** ```json { "message": "Report 1 deleted successfully" } ``` ### Update Report Status **PUT** `/deep_analysis/reports/{report_id}/status` Updates the status of a deep analysis report. **Request Body:** ```json { "status": "completed" } ``` **Valid Status Values:** - `pending`: Analysis queued but not started - `running`: Analysis in progress - `completed`: Analysis finished successfully - `failed`: Analysis encountered errors ### Get HTML Report **GET** `/deep_analysis/reports/uuid/{report_uuid}/html` Retrieves only the HTML report content for a specific analysis. **Query Parameters:** - `user_id` (optional): Ensures report belongs to specified user **Response:** ```json { "html_report": "...", "filename": "deep_analysis_report_20240101_120200.html" } ``` ### Download HTML Report **POST** `/deep_analysis/download_from_db/{report_uuid}` Downloads the HTML report as a file attachment. **Query Parameters:** - `user_id` (optional): Ensures report belongs to specified user **Response:** - Content-Type: `text/html; charset=utf-8` - Content-Disposition: `attachment; filename="deep_analysis_report_TIMESTAMP.html"` --- ## Deep Analysis Module Architecture ### DSPy Signatures The system uses several DSPy signatures for different analysis phases: #### 1. `deep_questions` Generates 5 targeted analytical questions based on the user's goal and dataset structure. #### 2. `deep_planner` Creates an optimized execution plan using the user's active templates/agents. The planner: - Verifies feasibility using available datasets and agent descriptions - Batches similar questions per agent call for efficiency - Reuses outputs across questions to minimize agent calls - Defines clear variable flow and dependencies between agents #### 3. `deep_code_synthesizer` Combines and optimizes code from multiple agents: - Fixes errors and inconsistencies between agent outputs - Ensures proper data flow and type handling - Converts all visualizations to Plotly format - Adds comprehensive error handling and validation #### 4. `deep_synthesizer` Synthesizes analysis results into coherent insights and findings. #### 5. `final_conclusion` Generates final conclusions and strategic recommendations based on all analysis results. ### Streaming Analysis The `execute_deep_analysis_streaming` method provides real-time progress updates: ```python async for update in deep_analysis.execute_deep_analysis_streaming(goal, dataset_info, session_df): if update["step"] == "questions": # Handle questions generation progress elif update["step"] == "planning": # Handle planning progress elif update["step"] == "agent_execution": # Handle agent execution progress # ... handle other steps ``` ### Integration with User Templates The deep analysis system integrates with user templates in several ways: 1. **Agent Discovery**: Retrieves user's active template preferences from the database 2. **Dynamic Planning**: The planner uses available agents to create optimal execution plans 3. **Template Validation**: Ensures all referenced agents exist in the user's active templates 4. **Fallback Handling**: Uses default agents if user preferences are incomplete 5. **Performance Optimization**: Respects template limits for efficient execution ### Error Handling The system includes comprehensive error handling: - **Code Execution Errors**: Automatically attempts to fix and retry failed code - **Template Missing**: Falls back to default agents if user templates are unavailable - **Timeout Protection**: Includes timeouts for long-running operations - **Memory Management**: Handles large datasets and visualization efficiently - **Unicode Handling**: Cleans problematic characters that might cause encoding issues ### Visualization Integration All visualizations are standardized to Plotly format: - Consistent styling and color schemes - Interactive features (zoom, pan, hover) - Accessibility compliance (colorblind-friendly palettes) - Export capabilities for reports - Responsive design for different screen sizes --- ## Frontend Integration The deep analysis system includes React components for: - **DeepAnalysisSidebar**: Main interface for starting and managing analyses - **NewAnalysisForm**: Form for initiating new deep analyses - **CurrentAnalysisView**: Real-time progress tracking during analysis - **HistoryView**: Browse and access historical analysis reports - **AnalysisStep**: Individual step progress visualization The frontend integrates with the streaming API to provide real-time feedback and uses the user's active template configuration for personalized analysis capabilities. ## Credit and Cost Tracking The system tracks detailed usage metrics: - **Credits Consumed**: Number of credits deducted from user account - **Token Usage**: Total tokens used across all model calls - **Estimated Cost**: Dollar cost estimate based on model pricing - **Model Information**: Provider and model name used for analysis - **Execution Time**: Duration of analysis for performance monitoring This information helps users understand resource consumption and optimize their analysis strategies.