docs/DEPLOYMENT.md

# Deployment Guide

## Production Checklist

- [ ] All environment variables configured
- [ ] Database migrations run
- [ ] Redis cache configured
- [ ] API keys secured (use secrets management)
- [ ] CORS configured properly
- [ ] Rate limiting enabled
- [ ] Logging configured
- [ ] SSL/TLS certificates installed
- [ ] Health checks passing
- [ ] Load testing completed

## Deployment Options

## 1. Docker Compose (Recommended for Small Deployments)

```bash
# Build and start all services
docker-compose up -d

# View logs
docker-compose logs -f

# Stop services
docker-compose down
```

## 2. Kubernetes

### Prerequisites
- Kubernetes cluster (1.24+)
- kubectl configured
- Container registry access

### Deploy

```bash
# Create namespace
kubectl create namespace multi-agent

# Create secrets for API keys
kubectl create secret generic api-keys \
  --from-literal=openai-api-key=sk-... \
  --from-literal=database-url=postgresql://... \
  -n multi-agent

# Apply deployment
kubectl apply -f k8s/ -n multi-agent

# Check status
kubectl get pods -n multi-agent
kubectl logs -f deployment/backend -n multi-agent
```

## 3. Cloud Platforms

### AWS (ECS/Fargate)

1. Push image to ECR
2. Create ECS task definition
3. Create ECS service
4. Configure RDS for database
5. Configure ElastiCache for Redis

### Google Cloud (Cloud Run)

```bash
# Build and push
gcloud builds submit --tag gcr.io/PROJECT/multi-agent

# Deploy
gcloud run deploy multi-agent \
  --image gcr.io/PROJECT/multi-agent \
  --platform managed \
  --region us-central1
```

### Azure (App Service)

```bash
# Create resource group
az group create -n multi-agent-rg -l eastus

# Create app service plan
az appservice plan create -n multi-agent-plan \
  -g multi-agent-rg --sku B2 --is-linux

# Deploy container
az webapp create -n multi-agent -g multi-agent-rg \
  -p multi-agent-plan --deployment-container-image-name-user-provided
```

### Heroku

```bash
# Login
heroku login

# Create app
heroku create multi-agent

# Set environment variables
heroku config:set OPENAI_API_KEY=sk-...
heroku config:set DATABASE_URL=postgresql://...

# Deploy
git push heroku main

# View logs
heroku logs --tail
```

## Environment Variables

For production, secure these variables:

```bash
# Critical
OPENAI_API_KEY=sk-your-production-key
DATABASE_URL=postgresql://user:pass@host/db
REDIS_URL=redis://host:6379/0

# Security
ENVIRONMENT=production
CORS_ORIGINS=https://yourdomain.com
API_KEY_SECRET=your-secret-key

# Logging
LOG_LEVEL=INFO
SENTRY_DSN=https://...  # Optional: error tracking

# Performance
WORKER_PROCESSES=4
MAX_CONNECTIONS=100
```

## Database Migrations

```bash
# Using Alembic (if configured)
alembic upgrade head

# Or with SQLAlchemy directly
python -c "from backend.core.config import init_db; init_db()"
```

## Monitoring & Logging

### Application Logs

Use structured logging to centralize logs:

```bash
# Docker
docker-compose logs -f backend

# Kubernetes
kubectl logs -f deployment/backend

# Cloud services
# Check respective platform dashboards
```

### Metrics to Monitor

- Response time (p50, p95, p99)
- Error rate
- Task completion rate
- API usage (requests/minute)
- Memory/CPU usage
- Database connection pool health
- Queue depth (if using task queue)

### Recommended Tools

- **Monitoring**: Prometheus + Grafana
- **Logging**: ELK Stack, Datadog, or cloud provider
- **Error Tracking**: Sentry
- **APM**: New Relic, DataDog

## Scaling

### Horizontal Scaling

```bash
# Docker Compose
docker-compose up -d --scale backend=3

# Kubernetes
kubectl scale deployment backend --replicas=3
```

### Performance Optimization

1. **Cache frequently accessed data**
   ```python
   # In backend code
   @cache.cached(timeout=300)
   def expensive_operation():
       pass
   ```

2. **Use connection pooling**
   - Redis already handles this
   - Database: SQLAlchemy creates connection pool automatically

3. **Optimize database queries**
   - Add indexes to frequently queried columns
   - Use query profiling tools

## Security

### API Security

1. **Enable HTTPS/TLS** (required for production)
2. **Add API authentication**
   ```python
   from fastapi import Security, HTTPBearer
   security = HTTPBearer()
   ```

3. **Implement rate limiting**
   ```python
   from slowapi import Limiter
   limiter = Limiter(key_func=get_remote_address)
   ```

4. **Add CORS restrictions**
   ```python
   CORSMiddleware(
       allow_origins=["https://yourdomain.com"],
       allow_credentials=True,
   )
   ```

### Data Security

- Encrypt sensitive data at rest
- Use environment variables for secrets (never commit .env)
- Regular security audits
- Keep dependencies updated

## Backup & Recovery

```bash
# Backup PostgreSQL
pg_dump multi_agent > backup.sql

# Restore
psql multi_agent < backup.sql

# Backup Redis
redis-cli BGSAVE
# RDB file: /var/lib/redis/dump.rdb

# Docker volume backup
docker-compose exec postgres pg_dump multi_agent > backup.sql
```

## Troubleshooting

### Service Won't Start

```bash
# Check logs
docker-compose logs backend

# Verify environment
docker-compose config

# Reset containers
docker-compose down -v
docker-compose up --build
```

### High Memory Usage

- Check for memory leaks in agent code
- Reduce worker processes
- Increase available memory
- Monitor with: `docker stats`

### Slow Queries

```bash
# Enable query logging
# In .env: SQLALCHEMY_ECHO=true

# Analyze slow queries
# Enable PostgreSQL slow query log
```

## Rollback Procedure

```bash
# Docker
docker-compose down
docker-compose pull  # Get previous version
docker-compose up

# Kubernetes
kubectl rollout history deployment/backend
kubectl rollout undo deployment/backend --to-revision=1

# Heroku
heroku releases
heroku rollback v123
```

## Support & Documentation

- GitHub Issues: Report bugs
- Documentation: `./docs/`
- API Docs: http://yourapp.com/docs