Spaces:

Ndg07
/

heatmap

Running

App Files Files Community

heatmap / docs /DOCKER_SETUP.md

Ndg07

Initial commit: Enhanced Fake News Detection System

3f0cc2f 9 months ago

preview code

Raw

History Blame Contribute Delete

13 kB

Docker Setup Guide

Complete guide for containerizing and deploying the Real-Time Misinformation Heatmap using Docker.

Overview

This project includes comprehensive Docker support with:

Multi-stage Dockerfile for optimized production builds
Docker Compose configurations for development and production
Management scripts for easy deployment and maintenance
Monitoring stack with Prometheus and Grafana
SSL/TLS support for production deployments

Quick Start

Development Environment

Copy environment template

cp .env.example .env
# Edit .env with your configuration

Start development environment

# Unix/Linux/macOS
./scripts/docker-dev.sh up

# Windows PowerShell
docker-compose up -d

Access services
- Frontend: http://localhost:3000
- API: http://localhost:8000
- API Docs: http://localhost:8000/docs
- Pub/Sub Emulator: http://localhost:8085

Production Environment

Create production configuration

cp .env.example .env.production
# Edit .env.production with production values

Deploy to production

# Unix/Linux/macOS
./scripts/docker-prod.sh deploy

# Windows PowerShell
docker-compose -f docker-compose.prod.yml up -d

Architecture

Multi-Stage Dockerfile

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   Base Stage    │───▶│ Dependencies    │───▶│  Application    │
│                 │    │                 │    │                 │
│ • Python 3.9    │    │ • Pip packages  │    │ • App code      │
│ • System deps   │    │ • ML models     │    │ • Configuration │
│ • Security      │    │ • Cloud SDKs    │    │ • Static files  │
└─────────────────┘    └─────────────────┘    └─────────────────┘
                                                         │
                       ┌─────────────────┐    ┌─────────────────┐
                       │   Development   │    │   Production    │
                       │                 │    │                 │
                       │ • Debug tools   │    │ • Gunicorn      │
                       │ • Hot reload    │    │ • Optimized     │
                       │ • Test deps     │    │ • Security      │
                       └─────────────────┘    └─────────────────┘

Service Architecture

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│     Nginx       │    │   Application   │    │     Redis       │
│                 │    │                 │    │                 │
│ • SSL/TLS       │───▶│ • FastAPI       │───▶│ • Caching       │
│ • Load Balance  │    │ • Data Sources  │    │ • Session Store │
│ • Static Files  │    │ • NLP Pipeline  │    │ • Rate Limiting │
└─────────────────┘    └─────────────────┘    └─────────────────┘
         │                       │                       │
         ▼                       ▼                       ▼
┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   Monitoring    │    │  Pub/Sub        │    │   BigQuery      │
│                 │    │                 │    │                 │
│ • Prometheus    │    │ • Event Queue   │    │ • Data Storage  │
│ • Grafana       │    │ • Processing    │    │ • Analytics     │
│ • Alerting      │    │ • Scaling       │    │ • Reporting     │
└─────────────────┘    └─────────────────┘    └─────────────────┘

Configuration

Environment Variables

Core Application

# Application mode
MODE=local|cloud
ENVIRONMENT=development|staging|production
LOG_LEVEL=DEBUG|INFO|WARNING|ERROR

# API Configuration
API_PORT=8000
API_HOST=0.0.0.0
CORS_ORIGINS=["http://localhost:3000"]

# Security
API_KEY_ENABLED=true
API_KEYS=["key1","key2"]

Google Cloud

# Project Configuration
GOOGLE_CLOUD_PROJECT=your-project-id
GOOGLE_APPLICATION_CREDENTIALS=/app/credentials.json

# BigQuery
BIGQUERY_DATASET=misinformation_heatmap
BIGQUERY_LOCATION=US

# Pub/Sub
PUBSUB_EVENTS_RAW_TOPIC=events-raw
PUBSUB_EVENTS_PROCESSED_TOPIC=events-processed

Performance & Caching

# Caching
CACHE_TYPE=redis
REDIS_URL=redis://redis:6379/0

# Rate Limiting
RATE_LIMIT_ENABLED=true
RATE_LIMIT_REQUESTS_PER_MINUTE=100

# Monitoring
ENABLE_METRICS=true
ENABLE_TRACING=true

Docker Compose Profiles

Development Profile

# Start development services
COMPOSE_PROFILES=local,development
docker-compose up -d

Production Profile

# Start production services with monitoring
COMPOSE_PROFILES=production,monitoring
docker-compose -f docker-compose.prod.yml up -d

Testing Profile

# Run tests
COMPOSE_PROFILES=testing
docker-compose up test-runner

Management Scripts

Development Script (`scripts/docker-dev.sh`)

# Build development images
./scripts/docker-dev.sh build

# Start development environment
./scripts/docker-dev.sh up

# Show logs
./scripts/docker-dev.sh logs [service]

# Open shell in container
./scripts/docker-dev.sh shell

# Run tests
./scripts/docker-dev.sh test

# Clean up resources
./scripts/docker-dev.sh clean

Production Script (`scripts/docker-prod.sh`)

# Build production images
./scripts/docker-prod.sh build

# Deploy to production
./scripts/docker-prod.sh deploy

# Update deployment
./scripts/docker-prod.sh update

# Rollback deployment
./scripts/docker-prod.sh rollback

# Scale services
./scripts/docker-prod.sh scale app=3

# Create backup
./scripts/docker-prod.sh backup

# Restore from backup
./scripts/docker-prod.sh restore backups/backup_20241126_143022

Production Deployment

Prerequisites

SSL Certificates

mkdir -p ssl
# Place your SSL certificate files:
# ssl/cert.pem
# ssl/key.pem

Google Cloud Credentials

# Place service account key file:
# credentials.json

Production Configuration

# Create .env.production with production values
cp .env.example .env.production

Deployment Steps

Build Production Images
```
./scripts/docker-prod.sh build
```
Deploy Services
```
./scripts/docker-prod.sh deploy
```
Verify Deployment
```
./scripts/docker-prod.sh health
```
Monitor Services
```
./scripts/docker-prod.sh status
```

Production Services

Main Application

Container: misinformation-heatmap-prod
Port: 8080 (internal)
Health Check: /health endpoint
Resources: 2 CPU, 4GB RAM

Nginx Reverse Proxy

Container: misinformation-heatmap-nginx
Ports: 80 (HTTP), 443 (HTTPS)
Features: SSL termination, static file serving, load balancing
Resources: 0.5 CPU, 512MB RAM

Redis Cache

Container: misinformation-heatmap-redis-prod
Port: 6379 (internal)
Persistence: Enabled with snapshots
Resources: 1 CPU, 1.5GB RAM

Monitoring Stack

Prometheus: Port 9090, metrics collection
Grafana: Port 3001, dashboards and visualization
Resources: 1.5 CPU, 3GB RAM total

Monitoring & Observability

Metrics Collection

The application exposes metrics at /metrics endpoint:

Request metrics: Rate, latency, errors
Business metrics: Events processed, sources active
System metrics: CPU, memory, disk usage
Custom metrics: Data quality, processing pipeline health

Dashboards

System Overview Dashboard

Service health and uptime
Request rate and error rate
Response time percentiles
Resource utilization

Business Metrics Dashboard

Events processed per hour
Data source health
Misinformation detection rates
Geographic distribution

Performance Dashboard

API endpoint performance
Database query performance
Cache hit rates
Processing pipeline throughput

Alerting Rules

Critical Alerts

Service down for >1 minute
Error rate >5% for >5 minutes
Response time >2 seconds for >5 minutes

Warning Alerts

High memory usage >80% for >10 minutes
High CPU usage >80% for >10 minutes
Cache hit rate <50% for >15 minutes

Backup & Recovery

Automated Backups

# Create daily backup (add to cron)
0 2 * * * /path/to/scripts/docker-prod.sh backup

Backup Contents

Application Data: Event data, configurations
Redis Data: Cache and session data
Grafana Data: Dashboards and settings
Logs: Application and access logs

Recovery Procedures

List Available Backups
```
ls -la backups/
```

Restore from Backup

./scripts/docker-prod.sh restore backups/backup_20241126_143022

Verify Recovery
```
./scripts/docker-prod.sh health
```

Security

Container Security

Non-root user: Application runs as appuser
Minimal base image: Python slim image
Security updates: Regular base image updates
Secrets management: Environment variables and mounted files

Network Security

Internal networks: Services communicate on private networks
SSL/TLS: HTTPS enforced in production
Rate limiting: API and nginx level protection
Security headers: Comprehensive HTTP security headers

Access Control

API authentication: Key-based authentication
Service isolation: Each service runs in isolated container
Resource limits: CPU and memory limits enforced
Health checks: Automated health monitoring

Troubleshooting

Common Issues

Container Won't Start

# Check logs
docker-compose logs service-name

# Check resource usage
docker stats

# Verify configuration
docker-compose config

High Memory Usage

# Check memory usage by container
docker stats --format "table {{.Container}}\t{{.MemUsage}}\t{{.MemPerc}}"

# Scale down if needed
./scripts/docker-prod.sh scale app=1

SSL Certificate Issues

# Verify certificate files
ls -la ssl/
openssl x509 -in ssl/cert.pem -text -noout

# Check nginx configuration
docker-compose exec nginx nginx -t

Database Connection Issues

# Check BigQuery connectivity
docker-compose exec app python -c "from backend.database import Database; db = Database(); print('Connected')"

# Check Redis connectivity
docker-compose exec redis redis-cli ping

Performance Tuning

Application Performance

Worker processes: Adjust Gunicorn workers based on CPU cores
Connection pooling: Optimize database connections
Caching: Tune Redis cache settings
Resource limits: Adjust container resource limits

Database Performance

Query optimization: Monitor slow queries
Indexing: Ensure proper BigQuery table partitioning
Connection limits: Optimize connection pool sizes

Network Performance

Nginx tuning: Optimize worker connections and buffers
Compression: Enable gzip for static assets
CDN: Consider CloudFlare or similar for static assets

Best Practices

Development

Use volume mounts for hot reloading
Run tests in isolated containers
Use development-specific environment variables
Enable debug logging and profiling

Production

Use multi-stage builds for smaller images
Implement health checks for all services
Use resource limits and requests
Enable monitoring and alerting
Implement proper backup strategies
Use secrets management for sensitive data

Security

Regularly update base images
Scan images for vulnerabilities
Use non-root users in containers
Implement network segmentation
Enable audit logging
Use strong authentication and authorization

This Docker setup provides a robust, scalable, and production-ready deployment solution for the Real-Time Misinformation Heatmap system.

Docker Setup Guide

Overview

Quick Start

Development Environment

Production Environment

Architecture

Multi-Stage Dockerfile

Service Architecture

Configuration

Environment Variables

Core Application

Google Cloud

Performance & Caching

Docker Compose Profiles

Development Profile

Production Profile

Testing Profile

Management Scripts

Development Script (scripts/docker-dev.sh)

Production Script (scripts/docker-prod.sh)

Production Deployment

Prerequisites

Deployment Steps

Production Services

Main Application

Nginx Reverse Proxy

Redis Cache

Monitoring Stack

Monitoring & Observability

Metrics Collection

Dashboards

System Overview Dashboard

Business Metrics Dashboard

Performance Dashboard

Alerting Rules

Critical Alerts

Warning Alerts

Backup & Recovery

Automated Backups

Backup Contents

Recovery Procedures

Security

Container Security

Network Security

Access Control

Troubleshooting

Common Issues

Container Won't Start

High Memory Usage

SSL Certificate Issues

Database Connection Issues

Performance Tuning

Application Performance

Database Performance

Network Performance

Best Practices

Development

Production

Security

Development Script (`scripts/docker-dev.sh`)

Production Script (`scripts/docker-prod.sh`)