docs/DOCKER_SETUP.md

# Docker Setup Guide

Complete guide for containerizing and deploying the Real-Time Misinformation Heatmap using Docker.

## Overview

This project includes comprehensive Docker support with:
- **Multi-stage Dockerfile** for optimized production builds
- **Docker Compose** configurations for development and production
- **Management scripts** for easy deployment and maintenance
- **Monitoring stack** with Prometheus and Grafana
- **SSL/TLS support** for production deployments

## Quick Start

### Development Environment

1. **Copy environment template**
   ```bash
   cp .env.example .env
   # Edit .env with your configuration
   ```

2. **Start development environment**
   ```bash
   # Unix/Linux/macOS
   ./scripts/docker-dev.sh up
   
   # Windows PowerShell
   docker-compose up -d
   ```

3. **Access services**
   - Frontend: http://localhost:3000
   - API: http://localhost:8000
   - API Docs: http://localhost:8000/docs
   - Pub/Sub Emulator: http://localhost:8085

### Production Environment

1. **Create production configuration**
   ```bash
   cp .env.example .env.production
   # Edit .env.production with production values
   ```

2. **Deploy to production**
   ```bash
   # Unix/Linux/macOS
   ./scripts/docker-prod.sh deploy
   
   # Windows PowerShell
   docker-compose -f docker-compose.prod.yml up -d
   ```

## Architecture

### Multi-Stage Dockerfile

```
┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   Base Stage    │───▶│ Dependencies    │───▶│  Application    │
│                 │    │                 │    │                 │
│ • Python 3.9    │    │ • Pip packages  │    │ • App code      │
│ • System deps   │    │ • ML models     │    │ • Configuration │
│ • Security      │    │ • Cloud SDKs    │    │ • Static files  │
└─────────────────┘    └─────────────────┘    └─────────────────┘
                                                         │
                       ┌─────────────────┐    ┌─────────────────┐
                       │   Development   │    │   Production    │
                       │                 │    │                 │
                       │ • Debug tools   │    │ • Gunicorn      │
                       │ • Hot reload    │    │ • Optimized     │
                       │ • Test deps     │    │ • Security      │
                       └─────────────────┘    └─────────────────┘
```

### Service Architecture

```
┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│     Nginx       │    │   Application   │    │     Redis       │
│                 │    │                 │    │                 │
│ • SSL/TLS       │───▶│ • FastAPI       │───▶│ • Caching       │
│ • Load Balance  │    │ • Data Sources  │    │ • Session Store │
│ • Static Files  │    │ • NLP Pipeline  │    │ • Rate Limiting │
└─────────────────┘    └─────────────────┘    └─────────────────┘
         │                       │                       │
         ▼                       ▼                       ▼
┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   Monitoring    │    │  Pub/Sub        │    │   BigQuery      │
│                 │    │                 │    │                 │
│ • Prometheus    │    │ • Event Queue   │    │ • Data Storage  │
│ • Grafana       │    │ • Processing    │    │ • Analytics     │
│ • Alerting      │    │ • Scaling       │    │ • Reporting     │
└─────────────────┘    └─────────────────┘    └─────────────────┘
```

## Configuration

### Environment Variables

#### Core Application
```bash
# Application mode
MODE=local|cloud
ENVIRONMENT=development|staging|production
LOG_LEVEL=DEBUG|INFO|WARNING|ERROR

# API Configuration
API_PORT=8000
API_HOST=0.0.0.0
CORS_ORIGINS=["http://localhost:3000"]

# Security
API_KEY_ENABLED=true
API_KEYS=["key1","key2"]
```

#### Google Cloud
```bash
# Project Configuration
GOOGLE_CLOUD_PROJECT=your-project-id
GOOGLE_APPLICATION_CREDENTIALS=/app/credentials.json

# BigQuery
BIGQUERY_DATASET=misinformation_heatmap
BIGQUERY_LOCATION=US

# Pub/Sub
PUBSUB_EVENTS_RAW_TOPIC=events-raw
PUBSUB_EVENTS_PROCESSED_TOPIC=events-processed
```

#### Performance & Caching
```bash
# Caching
CACHE_TYPE=redis
REDIS_URL=redis://redis:6379/0

# Rate Limiting
RATE_LIMIT_ENABLED=true
RATE_LIMIT_REQUESTS_PER_MINUTE=100

# Monitoring
ENABLE_METRICS=true
ENABLE_TRACING=true
```

### Docker Compose Profiles

#### Development Profile
```yaml
# Start development services
COMPOSE_PROFILES=local,development
docker-compose up -d
```

#### Production Profile
```yaml
# Start production services with monitoring
COMPOSE_PROFILES=production,monitoring
docker-compose -f docker-compose.prod.yml up -d
```

#### Testing Profile
```yaml
# Run tests
COMPOSE_PROFILES=testing
docker-compose up test-runner
```

## Management Scripts

### Development Script (`scripts/docker-dev.sh`)

```bash
# Build development images
./scripts/docker-dev.sh build

# Start development environment
./scripts/docker-dev.sh up

# Show logs
./scripts/docker-dev.sh logs [service]

# Open shell in container
./scripts/docker-dev.sh shell

# Run tests
./scripts/docker-dev.sh test

# Clean up resources
./scripts/docker-dev.sh clean
```

### Production Script (`scripts/docker-prod.sh`)

```bash
# Build production images
./scripts/docker-prod.sh build

# Deploy to production
./scripts/docker-prod.sh deploy

# Update deployment
./scripts/docker-prod.sh update

# Rollback deployment
./scripts/docker-prod.sh rollback

# Scale services
./scripts/docker-prod.sh scale app=3

# Create backup
./scripts/docker-prod.sh backup

# Restore from backup
./scripts/docker-prod.sh restore backups/backup_20241126_143022
```

## Production Deployment

### Prerequisites

1. **SSL Certificates**
   ```bash
   mkdir -p ssl
   # Place your SSL certificate files:
   # ssl/cert.pem
   # ssl/key.pem
   ```

2. **Google Cloud Credentials**
   ```bash
   # Place service account key file:
   # credentials.json
   ```

3. **Production Configuration**
   ```bash
   # Create .env.production with production values
   cp .env.example .env.production
   ```

### Deployment Steps

1. **Build Production Images**
   ```bash
   ./scripts/docker-prod.sh build
   ```

2. **Deploy Services**
   ```bash
   ./scripts/docker-prod.sh deploy
   ```

3. **Verify Deployment**
   ```bash
   ./scripts/docker-prod.sh health
   ```

4. **Monitor Services**
   ```bash
   ./scripts/docker-prod.sh status
   ```

### Production Services

#### Main Application
- **Container**: `misinformation-heatmap-prod`
- **Port**: 8080 (internal)
- **Health Check**: `/health` endpoint
- **Resources**: 2 CPU, 4GB RAM

#### Nginx Reverse Proxy
- **Container**: `misinformation-heatmap-nginx`
- **Ports**: 80 (HTTP), 443 (HTTPS)
- **Features**: SSL termination, static file serving, load balancing
- **Resources**: 0.5 CPU, 512MB RAM

#### Redis Cache
- **Container**: `misinformation-heatmap-redis-prod`
- **Port**: 6379 (internal)
- **Persistence**: Enabled with snapshots
- **Resources**: 1 CPU, 1.5GB RAM

#### Monitoring Stack
- **Prometheus**: Port 9090, metrics collection
- **Grafana**: Port 3001, dashboards and visualization
- **Resources**: 1.5 CPU, 3GB RAM total

## Monitoring & Observability

### Metrics Collection

The application exposes metrics at `/metrics` endpoint:
- **Request metrics**: Rate, latency, errors
- **Business metrics**: Events processed, sources active
- **System metrics**: CPU, memory, disk usage
- **Custom metrics**: Data quality, processing pipeline health

### Dashboards

#### System Overview Dashboard
- Service health and uptime
- Request rate and error rate
- Response time percentiles
- Resource utilization

#### Business Metrics Dashboard
- Events processed per hour
- Data source health
- Misinformation detection rates
- Geographic distribution

#### Performance Dashboard
- API endpoint performance
- Database query performance
- Cache hit rates
- Processing pipeline throughput

### Alerting Rules

#### Critical Alerts
- Service down for >1 minute
- Error rate >5% for >5 minutes
- Response time >2 seconds for >5 minutes

#### Warning Alerts
- High memory usage >80% for >10 minutes
- High CPU usage >80% for >10 minutes
- Cache hit rate <50% for >15 minutes

## Backup & Recovery

### Automated Backups

```bash
# Create daily backup (add to cron)
0 2 * * * /path/to/scripts/docker-prod.sh backup
```

### Backup Contents
- **Application Data**: Event data, configurations
- **Redis Data**: Cache and session data
- **Grafana Data**: Dashboards and settings
- **Logs**: Application and access logs

### Recovery Procedures

1. **List Available Backups**
   ```bash
   ls -la backups/
   ```

2. **Restore from Backup**
   ```bash
   ./scripts/docker-prod.sh restore backups/backup_20241126_143022
   ```

3. **Verify Recovery**
   ```bash
   ./scripts/docker-prod.sh health
   ```

## Security

### Container Security
- **Non-root user**: Application runs as `appuser`
- **Minimal base image**: Python slim image
- **Security updates**: Regular base image updates
- **Secrets management**: Environment variables and mounted files

### Network Security
- **Internal networks**: Services communicate on private networks
- **SSL/TLS**: HTTPS enforced in production
- **Rate limiting**: API and nginx level protection
- **Security headers**: Comprehensive HTTP security headers

### Access Control
- **API authentication**: Key-based authentication
- **Service isolation**: Each service runs in isolated container
- **Resource limits**: CPU and memory limits enforced
- **Health checks**: Automated health monitoring

## Troubleshooting

### Common Issues

#### Container Won't Start
```bash
# Check logs
docker-compose logs service-name

# Check resource usage
docker stats

# Verify configuration
docker-compose config
```

#### High Memory Usage
```bash
# Check memory usage by container
docker stats --format "table {{.Container}}\t{{.MemUsage}}\t{{.MemPerc}}"

# Scale down if needed
./scripts/docker-prod.sh scale app=1
```

#### SSL Certificate Issues
```bash
# Verify certificate files
ls -la ssl/
openssl x509 -in ssl/cert.pem -text -noout

# Check nginx configuration
docker-compose exec nginx nginx -t
```

#### Database Connection Issues
```bash
# Check BigQuery connectivity
docker-compose exec app python -c "from backend.database import Database; db = Database(); print('Connected')"

# Check Redis connectivity
docker-compose exec redis redis-cli ping
```

### Performance Tuning

#### Application Performance
- **Worker processes**: Adjust Gunicorn workers based on CPU cores
- **Connection pooling**: Optimize database connections
- **Caching**: Tune Redis cache settings
- **Resource limits**: Adjust container resource limits

#### Database Performance
- **Query optimization**: Monitor slow queries
- **Indexing**: Ensure proper BigQuery table partitioning
- **Connection limits**: Optimize connection pool sizes

#### Network Performance
- **Nginx tuning**: Optimize worker connections and buffers
- **Compression**: Enable gzip for static assets
- **CDN**: Consider CloudFlare or similar for static assets

## Best Practices

### Development
- Use volume mounts for hot reloading
- Run tests in isolated containers
- Use development-specific environment variables
- Enable debug logging and profiling

### Production
- Use multi-stage builds for smaller images
- Implement health checks for all services
- Use resource limits and requests
- Enable monitoring and alerting
- Implement proper backup strategies
- Use secrets management for sensitive data

### Security
- Regularly update base images
- Scan images for vulnerabilities
- Use non-root users in containers
- Implement network segmentation
- Enable audit logging
- Use strong authentication and authorization

This Docker setup provides a robust, scalable, and production-ready deployment solution for the Real-Time Misinformation Heatmap system.