# Auto-Analyst Backend Documentation This directory contains comprehensive documentation for the Auto-Analyst backend - a sophisticated multi-agent AI platform for data analysis built with FastAPI, DSPy, and modern Python technologies. ## ๐Ÿ“ Documentation Structure ### **๐Ÿ—๏ธ Architecture** (`/architecture/`) - **[System Architecture](./architecture/architecture.md)** - Comprehensive overview of backend system design, components, and data flow patterns ### **๐Ÿš€ Development** (`/development/`) - **[Development Workflow](./development/development_workflow.md)** - Complete development guide with patterns, best practices, and code organization principles ### **๐Ÿ”ง System** (`/system/`) - **[Database Schema](./system/database-schema.md)** - Complete database schema with all tables, relationships, and performance optimization - **[Shared DataFrame System](./system/shared_dataframe.md)** - Inter-agent data sharing and session management ### **๐ŸŒ API** (`/api/`) - **[API Endpoints Overview](./api/README.md)** - Main API reference hub - **[Route Documentation](./api/routes/)** - Detailed endpoint documentation: - **[Core Routes](./api/routes/session.md)** - File uploads, sessions, authentication - **[Chat Routes](./api/routes/chats.md)** - Chat and messaging endpoints - **[Code Routes](./api/routes/code.md)** - Code execution and processing - **[Analytics Routes](./api/routes/analytics.md)** - Usage analytics and monitoring - **[Deep Analysis Routes](./api/routes/deep_analysis.md)** - Multi-agent analysis system - **[Template Routes](./api/routes/templates.md)** - Agent template management - **[Feedback Routes](./api/routes/feedback.md)** - User feedback and rating system ### **๐Ÿ› Troubleshooting** (`/troubleshooting/`) - **[Troubleshooting Guide](./troubleshooting/troubleshooting.md)** - Common issues, debugging tools, and solutions ## ๐ŸŽฏ Backend Overview ### **Tech Stack** - **FastAPI** - Modern async Python web framework - **DSPy** - AI agent orchestration and LLM integration - **SQLAlchemy** - Database ORM with PostgreSQL/SQLite support - **Plotly** - Interactive data visualizations - **Pandas/NumPy** - Data manipulation and analysis - **Scikit-learn** - Machine learning models - **Statsmodels** - Statistical analysis ### **Core Features** - **Multi-Agent System** - 4+ specialized AI agents for different analysis tasks - **Template System** - User-customizable agent configurations - **Deep Analysis** - Multi-step analytical workflows with streaming progress - **Session Management** - Stateful user sessions with shared data context - **Code Execution** - Safe Python code execution environment - **Real-time Streaming** - WebSocket support for live analysis updates ### **Agent Types** 1. **Data Preprocessing Agent** - Data cleaning and preparation 2. **Statistical Analytics Agent** - Statistical analysis using statsmodels 3. **Machine Learning Agent** - ML modeling with scikit-learn 4. **Data Visualization Agent** - Interactive charts with Plotly 5. **Feature Engineering Agent** (Premium) - Advanced feature creation 6. **Polars Agent** (Premium) - High-performance data processing ## ๐Ÿš€ Quick Start Guide ### **1. Environment Setup** ```bash # Navigate to backend directory cd Auto-Analyst-CS/auto-analyst-backend # Create virtual environment python -m venv venv source venv/bin/activate # Linux/Mac venv\Scripts\activate # Windows # Install dependencies pip install -r requirements.txt ``` ### **2. Environment Configuration** Create `.env` file with required variables: ```env # Database Configuration DATABASE_URL=sqlite:///./chat_database.db # AI Model Configuration OPENAI_API_KEY=your-openai-api-key MODEL_PROVIDER=openai # openai, anthropic, groq, gemini MODEL_NAME=gpt-4o-mini TEMPERATURE=0.7 MAX_TOKENS=6000 # Optional: Additional AI Providers ANTHROPIC_API_KEY=your-anthropic-key GROQ_API_KEY=your-groq-key GEMINI_API_KEY=your-gemini-key # Security ADMIN_API_KEY=your-admin-key # Application Settings ENVIRONMENT=development FRONTEND_URL=http://localhost:3000/ ``` ### **3. Database Initialization** ```bash # Initialize database and default agents python -c " from src.db.init_db import init_db init_db() print('โœ… Database and agents initialized successfully') " ``` ### **4. Start Development Server** ```bash # Start the FastAPI server python -m app # Or with uvicorn for more control uvicorn app:app --reload --host 0.0.0.0 --port 8000 ``` ### **5. Verify Installation** - **API Documentation**: `http://localhost:8000/docs` - **Health Check**: `http://localhost:8000/health` ## ๐Ÿ”ง Development Workflow ### **Adding New Agents** 1. **Define Agent Signature** in `src/agents/agents.py` 2. **Add Configuration** to `agents_config.json` 3. **Register Agent** in loading system 4. **Test Integration** with multi-agent pipeline ### **Adding New API Endpoints** 1. **Create Route File** in `src/routes/` 2. **Define Pydantic Models** for request/response 3. **Implement Endpoints** with proper error handling 4. **Register Router** in `app.py` 5. **Update Documentation** ### **Database Changes** 1. **Modify Models** in `src/db/schemas/models.py` 2. **Create Migration**: `alembic revision --autogenerate -m "description"` 3. **Apply Migration**: `alembic upgrade head` 4. **Update Documentation** ## ๐Ÿ“Š System Architecture ### **Request Processing Flow** ``` HTTP Request โ†’ FastAPI Router โ†’ Route Handler โ†’ Business Logic โ†’ Database/Agent System โ†’ AI Model โ†’ Response Processing โ†’ JSON Response ``` ### **Agent Execution Flow** ``` User Query โ†’ Session Manager โ†’ Agent Selection โ†’ Context Preparation โ†’ DSPy Chain โ†’ AI Model โ†’ Code Generation โ†’ Execution โ†’ Response Formatting ``` ### **Deep Analysis Workflow** ``` Goal Input โ†’ Question Generation โ†’ Planning โ†’ Multi-Agent Execution โ†’ Code Synthesis โ†’ Result Compilation โ†’ HTML Report Generation ``` ## ๐Ÿงช Testing & Validation ### **API Testing** ```bash # Interactive documentation open http://localhost:8000/docs # cURL examples curl -X GET "http://localhost:8000/health" curl -X POST "http://localhost:8000/chat/preprocessing_agent" \ -H "Content-Type: application/json" \ -d '{"query": "Clean this dataset", "session_id": "test"}' ``` ### **Agent Testing** ```python # Test individual agents from src.agents.agents import preprocessing_agent import dspy # Configure DSPy lm = dspy.LM('openai/gpt-4o-mini', api_key='your-key') dspy.configure(lm=lm) # Test agent agent = dspy.ChainOfThought(preprocessing_agent) result = agent(goal='clean data', dataset='test dataset') print(result) ``` ## ๐Ÿ”’ Security & Production ### **Security Features** - **Session-based authentication** with secure session management - **API key protection** for admin endpoints - **Input validation** using Pydantic models - **Error handling** with proper HTTP status codes - **CORS configuration** for frontend integration ### **Production Considerations** - **PostgreSQL database** for production deployment - **Environment variable management** for secrets - **Logging configuration** for monitoring - **Rate limiting** for API protection - **Performance optimization** for large datasets ## ๐Ÿ“ˆ Monitoring & Analytics The backend includes comprehensive analytics for: - **Usage tracking** - API endpoint usage and performance - **Model usage** - AI model consumption and costs - **User analytics** - User behavior and engagement - **Error monitoring** - System health and error tracking - **Performance metrics** - Response times and throughput ## ๐Ÿค Contributing 1. **Follow coding standards** defined in development workflow 2. **Add comprehensive tests** for new features 3. **Update documentation** for all changes 4. **Use proper error handling** patterns 5. **Submit detailed pull requests** with clear descriptions --- ## ๐Ÿ“– Detailed Documentation For specific implementation details, refer to the organized documentation in each subdirectory: - **[Getting Started Guide](./getting_started.md)** - Complete setup walkthrough - **[Architecture Documentation](./architecture/)** - System design and components - **[Development Guides](./development/)** - Workflow and best practices - **[API Reference](./api/)** - Complete endpoint documentation - **[System Documentation](./system/)** - Database and core systems - **[Troubleshooting](./troubleshooting/)** - Debugging and solutions --- **Need help?** Check the troubleshooting guide or refer to the comprehensive documentation in each section.