docs/setup/SETUP.md

# Atlas Setup Guide

## Overview
Atlas is an enhanced chat API service that provides intelligent question-answering capabilities with web search augmentation and comprehensive analytics. It uses Google's Gemini model, combines multiple search engines for comprehensive results, and includes a full analytics dashboard with MongoDB integration for session and message tracking.

## Virtual Environment Setup

The project has been set up with a Python virtual environment using the specifications from the Dockerfile:

- **Python Version**: 3.13.5 (newer than the 3.9 specified in Dockerfile)
- **Virtual Environment**: `atlas_env`
- **All dependencies**: Successfully installed

## Environment Variables

Create a `.env` file in the project root with the following variables:

```bash
# Required: Google API Key for Gemini model
GOOGLE_API_KEY=your_google_api_key_here

# Optional: Brave Search API Key (falls back to DuckDuckGo if not provided)
BRAVE_API_KEY=your_brave_api_key_here

# Required: MongoDB Configuration for Analytics
MONGODB_URL=mongodb+srv://username:password@cluster.mongodb.net/?retryWrites=true&w=majority&appName=Atlas
MONGODB_DATABASE=Atlas

# Application Settings (optional - defaults are used if not set)
PORT=7860
HOST=0.0.0.0
```

### Getting API Keys and Database Setup

1. **Google API Key**: 
   - Go to [Google AI Studio](https://makersuite.google.com/app/apikey)
   - Create a new API key
   - Add it to your `.env` file

2. **Brave Search API Key** (Optional):
   - Go to [Brave Search API](https://api.search.brave.com/)
   - Sign up and get your API key
   - Add it to your `.env` file

3. **MongoDB Atlas Setup** (Required for Analytics):
   - Go to [MongoDB Atlas](https://www.mongodb.com/atlas)
   - Create a free account and cluster
   - Create a database user with read/write permissions
   - Get your connection string and add it to your `.env` file
   - The analytics system requires MongoDB for session and message tracking

## Running the Application

### Option 1: Using the startup script
```bash
./start.sh
```

### Option 2: Manual startup
```bash
# Activate virtual environment
source atlas_env/bin/activate

# Run the application
python app.py
```

### Option 3: Using uvicorn directly
```bash
# Activate virtual environment
source atlas_env/bin/activate

# Run with uvicorn
uvicorn app:app --host 0.0.0.0 --port 7860
```

## Testing the Setup

### MongoDB Connection Test
Verify your MongoDB connection is working:

```bash
python test_mongo_connection.py
```

This will test:
- MongoDB connection with both sync and async drivers
- Database accessibility
- Environment variable configuration

### Quick Setup Verification
You can also verify the setup by starting the server and checking the health endpoint:

```bash
# Start the server
./start.sh

# In another terminal, test the health endpoint
curl http://localhost:7860/
```

## API Endpoints

Once running, the application provides these endpoints:

### Core Functionality
- **`/`** - Health check and status
- **`/chat`** - Main chat endpoint with search augmentation
- **`/search`** - Direct search functionality
- **`/docs`** - Interactive API documentation (Swagger UI)

### Analytics & Cache Management
- **`/analytics/stats`** - JSON API with analytics statistics
- **`/analytics/dashboard`** - Interactive HTML dashboard with charts
- **`/analytics/export`** - Export analytics data (JSON/CSV format)
- **`/analytics/cache`** - Cache performance metrics and statistics
- **`/analytics/cache/clear`** - Cache management and maintenance
- **`/analytics/users`** - User statistics and anonymous vs authenticated metrics
- **`/analytics/user/{user_id}`** - Individual user analytics and insights
- **`/analytics/comparison`** - Detailed authenticated vs anonymous comparison

### Example Usage

#### Anonymous Mode (No Authentication Required)
```bash
# Health check
curl http://localhost:7860/

# Simple anonymous chat request
curl -X POST http://localhost:7860/chat \
  -H "Content-Type: application/json" \
  -d '{"prompt": "What is artificial intelligence?", "use_search": true}'

# Anonymous request without search
curl -X POST http://localhost:7860/chat \
  -H "Content-Type: application/json" \
  -d '{"prompt": "What is 2+2?", "use_search": false}'

# Anonymous request with search optimization control
curl -X POST http://localhost:7860/chat \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "What are the latest AI developments?",
    "search_decision_mode": "aggressive",
    "force_search": true
  }'

# Anonymous request with conversation history
curl -X POST http://localhost:7860/chat \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "Can you elaborate on that?",
    "use_search": false,
    "history": [
      {"role": "user", "content": "What is machine learning?"},
      {"role": "assistant", "content": "Machine learning is a subset of AI..."}
    ]
  }'
```

#### Authenticated Mode (With User Tracking)
```bash
# Authenticated chat request
curl -X POST http://localhost:7860/chat \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "What is my chat history?",
    "user_id": "test-user-123",
    "use_search": true
  }'

# Authenticated request with session continuity
curl -X POST http://localhost:7860/chat \
  -H "Content-Type: application/json" \
  -H "X-Session-ID: session-uuid-here" \
  -d '{
    "prompt": "Continue our previous conversation",
    "user_id": "test-user-123",
    "use_search": false
  }'
```

#### Analytics & Monitoring
```bash
# View analytics (includes anonymous vs authenticated breakdown)
curl http://localhost:7860/analytics/stats

# Export analytics data
curl "http://localhost:7860/analytics/export?format=json&days=7"

# View cache performance metrics
curl http://localhost:7860/analytics/cache

# Clear expired cache entries
curl -X POST http://localhost:7860/analytics/cache/clear?cache_type=expired

# View user statistics breakdown
curl http://localhost:7860/analytics/users

# View specific user analytics  
curl http://localhost:7860/analytics/user/user123

# Access interactive dashboard in browser
open http://localhost:7860/analytics/dashboard
```

## Features

### 🤖 AI-Powered Chat
- Uses Google's Gemini 1.5 Flash model
- Configurable parameters (temperature, max tokens)
- Intelligent responses based on web search results
- Session-based conversation tracking

### 🔍 Advanced Web Search & Optimization
- **Dual Search Engine Strategy**: Brave Search + DuckDuckGo
- **Resilient Fallback**: Automatic fallback if one engine fails
- **Smart Query Extraction**: NLP-powered search term extraction using spaCy and RAKE
- **Deduplication**: Removes duplicate results across engines
- **🧠 Intelligent Search Optimization**: AI-powered search decision engine
- **⚡ Context-Aware Flow**: Cache-first for new conversations, smart decisions for follow-ups
- **🗄️ ChromaDB Vector Caching**: Semantic similarity matching with persistent storage
- **📊 Search Analytics**: Comprehensive search decision and performance tracking

### 🧠 NLP-Powered Processing
- Named Entity Recognition
- Dependency parsing for question focus
- RAKE keyword extraction
- Text preprocessing and lemmatization

### 📊 Comprehensive Analytics
- **Real-time Session Tracking**: Monitor user sessions and activity
- **Message Analytics**: Track response times, search usage, and success rates
- **Interactive Dashboard**: Beautiful HTML dashboard with charts and metrics
- **Data Export**: Export analytics data in JSON or CSV format
- **MongoDB Integration**: Persistent storage for all analytics data
- **Performance Monitoring**: Response time percentiles and error tracking

## Troubleshooting

### Common Issues

1. **Import Errors**: Make sure you're in the virtual environment
   ```bash
   source atlas_env/bin/activate
   ```

2. **API Key Errors**: Check your `.env` file and ensure API keys are set correctly

3. **MongoDB Connection Issues**: 
   - Verify your `MONGODB_URL` is correct in `.env`
   - Check your MongoDB Atlas cluster is running
   - Ensure your IP address is whitelisted in MongoDB Atlas
   - Test connection with: `python test_mongo_connection.py`

4. **spaCy Model Issues**: The model should be automatically downloaded, but you can manually download it:
   ```bash
   python -m spacy download en_core_web_sm
   ```

5. **NLTK Data Issues**: NLTK data is automatically downloaded on first run

6. **Analytics Not Working**: 
   - Check MongoDB connection
   - Verify environment variables are loaded
   - Restart the server after updating `.env`

### Port Conflicts

If port 7860 is already in use, you can change it in the `.env` file or run with a different port:

```bash
uvicorn app:app --host 0.0.0.0 --port 8000
```

## Development

### Adding New Dependencies

1. Add to `requirements.txt`
2. Install in virtual environment:
   ```bash
   source atlas_env/bin/activate
   pip install -r requirements.txt
   ```

### Testing Changes

- Use `python test_mongo_connection.py` to verify MongoDB connectivity
- Check the health endpoint at `http://localhost:7860/` after starting the server
- Monitor the analytics dashboard at `http://localhost:7860/analytics/dashboard`

### Current Dependencies

The project includes these key packages:
- `fastapi` - Web framework
- `motor` - Async MongoDB driver
- `google-generativeai` - Google Gemini API
- `spacy` - NLP processing
- `nltk` - Natural language toolkit
- `duckduckgo-search` - Web search
- `httpx` - HTTP client for Brave Search

## Production Deployment

For production deployment, consider:
- Using the provided Dockerfile
- Setting up proper environment variables (especially secure MongoDB credentials)
- Configuring reverse proxy (nginx)
- Setting up monitoring and logging
- Using a process manager (systemd, supervisor)
- Implementing proper MongoDB security (authentication, network restrictions)
- Setting up MongoDB backups for analytics data
- Configuring CORS properly for your domain

## Support

If you encounter issues:
1. Run `python test_mongo_connection.py` to test MongoDB connectivity
2. Check the server logs for error messages
3. Verify all environment variables are set correctly in `.env`
4. Ensure you're using the virtual environment
5. Test the health endpoint: `curl http://localhost:7860/`
6. Check the analytics dashboard for system status
7. Verify your MongoDB Atlas cluster is running and accessible

## Analytics System

The analytics system provides comprehensive insights into your chat application usage:

### Features
- **Session Tracking**: Each user interaction creates a session with unique ID
- **Message Analytics**: Response times, search usage, success rates
- **Real-time Dashboard**: Interactive charts and statistics
- **Data Export**: Download analytics data for external analysis
- **Performance Monitoring**: Track system performance and errors

### Accessing Analytics
- **Dashboard**: `http://localhost:7860/analytics/dashboard`
- **API**: `http://localhost:7860/analytics/stats`
- **Export**: `http://localhost:7860/analytics/export?format=json&days=7`

### Data Collected
- Session information (start time, duration, message count)
- Message metrics (prompt length, response time, search usage)
- Performance data (response time percentiles, error rates)
- Search analytics (engine usage, result counts)