docs/DEPLOYMENT.md

Face Intel — Deployment Guide

This guide covers local development setup, Docker packaging, environment configuration, database setup, storage layout, reverse proxy integration, and load balancer health probes.

No Dockerfile exists yet in the repository. This document describes the recommended Docker layout — implementing it is tracked as a follow-up task (see §13).

---

Table of Contents

1. Prerequisites
2. Local Development Setup
3. Running the Server
4. Environment Variable Configuration
5. Database Setup
6. Storage Directory Layout
7. Docker Deployment
8. Kubernetes Deployment
9. Reverse Proxy Considerations
10. Health Check Endpoints for Load Balancers
11. TLS / HTTPS
12. Operational Runbook
13. Follow-ups

---

1. Prerequisites

Runtime

• Python 3.11+ (3.12 recommended).
• pip and venv.
• OpenCV system dependencies (Linux): libgl1, libglib2.0-0. On Debian/Ubuntu: apt install -y libgl1 libglib2.0-0.
• Chrome/Chromium (only if you enable selenium_scraper or google_lens): google-chrome or chromium-browser.

Python dependencies

The full list is in requirements.txt. Key packages:

Package	Used by
fastapi, uvicorn, pydantic, pydantic-settings	API layer
opencv-python, Pillow, numpy	Image processing
face-recognition, dlib	Recognition (optional)
mtcnn, tensorflow	Detection (optional)
beautifulsoup4, lxml, requests	Scraping
selenium, webdriver-manager	JS-rendered scraping
loguru	Structured logging
pytest, pytest-asyncio	Test runner

Hardware

• CPU-only: Haar + DNN + image_analysis + forensics work well. DNN inference: ~30-80 ms per image.
• GPU (optional): DNN auto-detects CUDA. InsightFace and DeepFace benefit substantially. Requires opencv-python-headless built with CUDA or the onnxruntime-gpu package.
• RAM: 1 GB minimum for dev. 4 GB recommended for production with MTCNN/InsightFace enabled.
• Disk: ~50 MB for the DNN model, ~550 MB for InsightFace buffalo_l pack, ~1 GB for data/ over a year of jobs.

---

2. Local Development Setup

Step 1 — Clone and create a virtualenv

git clone <repo-url> face-intel
cd face-intel
python3.11 -m venv .venv
source .venv/bin/activate

Step 2 — Install dependencies

pip install --upgrade pip
pip install -r requirements.txt

dlib compilation: dlib==19.24.2 requires CMake and a C++ compiler. On Debian/Ubuntu: apt install -y build-essential cmake. Compilation takes ~5 minutes. See docs/TROUBLESHOOTING.md if it fails.

Step 3 — Configure

cp .env.example .env
# Edit .env to enable/disable providers

At minimum, review:

• FI_ENABLE_* flags (which providers to load).
• FI_SERPAPI_KEY, FI_BING_API_KEY, FI_TINEYE_PUBLIC_KEY, FI_TINEYE_PRIVATE_KEY (only if you enable the paid providers).
• FI_LOG_LEVEL, FI_DEBUG for verbosity.

See docs/CONFIGURATION.md for the full table.

Step 4 — Verify installation

# Verify imports don't have circular deps
python scripts/check_imports.py

# Verify settings load
python -c "from config.settings import settings; print(settings.model_dump_json(indent=2))"

# Run the test suite (should be 145 passing)
python -m pytest tests/ -v

---

3. Running the Server

Option A — Direct

python app.py

Reads config/settings.py::settings and runs uvicorn with reload if FI_DEBUG=true.

Option B — uvicorn directly

uvicorn app:app --host 0.0.0.0 --port 8000 --reload

Useful for development — --reload watches for file changes.

Option C — production-style

uvicorn app:app --host 0.0.0.0 --port 8000 \
  --workers 4 --no-access-log --log-level info

• --workers 4: run 4 worker processes (each with its own in-memory cache — see §9).
• --no-access-log: silence per-request access logs (useful when structured logging is enabled).

Verifying it's up

curl http://localhost:8000/health
# → {"status":"ok"}

curl http://localhost:8000/providers | jq '.providers | length'
# → 24

curl http://localhost:8000/docs  # Swagger UI in browser

---

4. Environment Variable Configuration

All settings use the FI_ prefix and are loaded by pydantic-settings from (in priority order):

1. Real environment variables.
2. .env file at the project root.
3. Field defaults in config/settings.py.

Quick reference

See docs/CONFIGURATION.md §2 for the full table of every setting, its env var name, type, and default.

Common patterns

Single-process dev:

export FI_DEBUG=true
export FI_LOG_LEVEL=DEBUG
python app.py

Multi-process prod (systemd unit):

# /etc/systemd/system/face-intel.service
[Unit]
Description=Face Intel
After=network.target

[Service]
Type=exec
User=face-intel
WorkingDirectory=/opt/face-intel
EnvironmentFile=/etc/face-intel/env
ExecStart=/opt/face-intel/.venv/bin/uvicorn app:app \
    --host 0.0.0.0 --port 8000 --workers 4 --no-access-log
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target

/etc/face-intel/env:

FI_ENVIRONMENT=production
FI_DEBUG=false
FI_LOG_LEVEL=INFO
FI_LOG_JSON=true
FI_DB_PATH=/var/lib/face-intel/face_intel.db
FI_AUDIT_LOG_PATH=/var/log/face-intel/audit.log
# ... etc

Loading env vars from a secrets manager

For Kubernetes, AWS, or HashiCorp Vault, write the secrets to env vars in the container entrypoint before running uvicorn. The app reads them at startup — once running, it doesn't poll for changes.

---

5. Database Setup

Engine

SQLite via the stdlib sqlite3 module. No external database server required. See storage/database.py.

Schema

Auto-created on first run:

CREATE TABLE jobs (
    id           TEXT PRIMARY KEY,
    kind         TEXT NOT NULL,
    status       TEXT NOT NULL,
    created_at   TEXT NOT NULL,
    started_at   TEXT,
    completed_at TEXT,
    request      TEXT NOT NULL,    -- JSON blob
    image_hash   TEXT,
    error        TEXT
);

CREATE TABLE job_results (
    job_id     TEXT PRIMARY KEY,
    status     TEXT NOT NULL,
    report     TEXT,               -- JSON blob (UnifiedFaceReport)
    error      TEXT,
    elapsed_ms REAL,
    created_at TEXT NOT NULL,
    FOREIGN KEY (job_id) REFERENCES jobs(id)
);

CREATE INDEX idx_jobs_status ON jobs(status);
CREATE INDEX idx_jobs_created ON jobs(created_at);

Path configuration

• Default: data/face_intel.db (relative to project root).
• Override via FI_DB_PATH=/absolute/path/to/face_intel.db.
• For tests: FI_DB_PATH=:memory: (in-memory, no disk file).

Thread safety

Database uses sqlite3.connect(check_same_thread=False) plus an internal threading.Lock around every operation. Safe to share across threads in one process. Not safe across multiple worker processes — each worker gets its own connection (and its own in-memory state).

Concurrency under multiple workers

If you run uvicorn with --workers 4, you'll have 4 independent SQLite connections to the same file. SQLite handles this via file locking — but write throughput drops sharply under contention. For high-throughput multi-worker deployments:

1. Use a separate PostgreSQL/MySQL backend (would require a new Database implementation — not currently supported), OR
2. Run a single worker and rely on async concurrency (orchestrator_max_concurrency), OR
3. Shard jobs across multiple Face Intel instances, each with its own SQLite DB.

Job retention

Set FI_JOB_RETENTION_DAYS=7 (default) to keep jobs for one week. Run Database.cleanup_old_jobs(retention_days) from a cron job:

# scripts/cleanup_jobs.py
from config.settings import settings
from storage.database import Database

db = Database(path=settings.db_path)
deleted = db.cleanup_old_jobs(settings.job_retention_days)
print(f"Deleted {deleted} old jobs")

Cron entry:

0 3 * * * /opt/face-intel/.venv/bin/python /opt/face-intel/scripts/cleanup_jobs.py

Backup

SQLite files can be backed up live using the sqlite3 CLI:

sqlite3 /var/lib/face-intel/face_intel.db ".backup /backup/face_intel-$(date +%F).db"

Or use the Online Backup API via Python:

import sqlite3
src = sqlite3.connect("/var/lib/face-intel/face_intel.db")
dst = sqlite3.connect("/backup/face_intel.db")
src.backup(dst)

---

6. Storage Directory Layout

config/settings.py auto-creates these directories at import time:

data/
├── face_intel.db          # SQLite database (jobs + results)
├── audit.log              # JSONL audit log
├── models/                # Auto-downloaded model files
│   ├── deploy.prototxt    #   DNN Caffe SSD prototxt (~28 KB)
│   └── res10_300x300_ssd_iter_140000.caffemodel  # (~10.7 MB)
├── gallery/               # Known-faces reference store
│   ├── manifest.json      #   {person_name: [embedding_filenames]}
│   ├── alice_0.npy        #   Per-person face embeddings
│   └── bob_0.npy
├── uploads/               # Source images saved by ArtifactStore
└── generated/             # Annotated images, montages

Volume mounting in containers

For Docker/Kubernetes, mount data/ as a persistent volume:

volumes:
  - face-intel-data:/app/data

For multi-replica deployments, data/gallery/ must be on a shared filesystem (NFS, EFS, S3-FUSE) so all replicas see the same known faces. Otherwise each replica has its own gallery.

Model files

The DNN provider auto-downloads its model files on first use via urllib.request.urlretrieve to data/models/. If your deployment has no internet access:

1. Pre-download the files on a build machine:

   mkdir -p data/models
   curl -L -o data/models/deploy.prototxt \
     https://raw.githubusercontent.com/opencv/opencv_3rdparty/dnn_samples_face_detector_20170830/deploy.prototxt
   curl -L -o data/models/res10_300x300_ssd_iter_140000.caffemodel \
     https://raw.githubusercontent.com/opencv/opencv_3rdparty/dnn_samples_face_detector_20170830/res10_300x300_ssd_iter_140000.caffemodel
   

2. Bake them into your Docker image or volume.

Audit log

utils.audit.audit_log() appends one JSON line per sensitive operation (gallery mutations, reverse image searches). Rotate with logrotate:

# /etc/logrotate.d/face-intel
/var/log/face-intel/audit.log {
    daily
    rotate 30
    compress
    missingok
    notifempty
    copytruncate
}

---

7. Docker Deployment

No Dockerfile is currently in the repo. The snippet below is the recommended baseline. Track follow-up work in §13.

Dockerfile (recommended)

FROM python:3.11-slim AS base

# OpenCV system deps + Chrome (for Selenium) + build tools for dlib
RUN apt-get update && apt-get install -y --no-install-recommends \
      libgl1 libglib2.0-0 \
      build-essential cmake \
      chromium \
    && rm -rf /var/lib/apt/lists/*

WORKDIR /app

# Install Python deps first (better layer caching)
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Copy app code
COPY . .

# Create runtime dirs
RUN mkdir -p /app/data/models /app/data/gallery \
             /app/data/uploads /app/data/generated

ENV FI_ENVIRONMENT=production \
    FI_DEBUG=false \
    FI_LOG_JSON=true \
    FI_DB_PATH=/app/data/face_intel.db \
    FI_AUDIT_LOG_PATH=/app/data/audit.log \
    FI_SELENIUM_HEADLESS=true

EXPOSE 8000

# Single worker — multi-worker requires shared storage / external DB
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000", \
     "--no-access-log", "--log-level", "info"]

Build & run

docker build -t face-intel:latest .
docker run -d --name face-intel \
  -p 8000:8000 \
  -v face-intel-data:/app/data \
  --env-file .env \
  face-intel:latest

.dockerignore

.venv/
__pycache__/
*.pyc
.pytest_cache/
data/
.env
*.log
.git/

Multi-stage build for smaller image

If image size matters, split into a builder stage that compiles dlib and a slim runtime stage:

FROM python:3.11-slim AS builder
RUN apt-get update && apt-get install -y build-essential cmake libgl1 libglib2.0-0
WORKDIR /app
COPY requirements.txt .
RUN pip install --user -r requirements.txt

FROM python:3.11-slim AS runtime
RUN apt-get update && apt-get install -y --no-install-recommends \
      libgl1 libglib2.0-0 chromium && rm -rf /var/lib/apt/lists/*
WORKDIR /app
COPY --from=builder /root/.local /root/.local
COPY . .
ENV PATH=/root/.local/bin:$PATH
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]

Final image: ~600 MB (vs ~1.5 GB without multi-stage).

Image variants

Variant	Tag pattern	Use case
CPU-only	face-intel:latest	Default. Works everywhere.
CPU-only, no Selenium	face-intel:cpu-slim	Smaller (~400 MB). No Chrome.
GPU	face-intel:gpu-cuda12	Built on nvidia/cuda:12.x-runtime. For InsightFace/DeepFace acceleration.

---

8. Kubernetes Deployment

Deployment + Service

apiVersion: apps/v1
kind: Deployment
metadata:
  name: face-intel
  labels:
    app: face-intel
spec:
  replicas: 2
  selector:
    matchLabels:
      app: face-intel
  template:
    metadata:
      labels:
        app: face-intel
    spec:
      containers:
        - name: face-intel
          image: face-intel:latest
          ports:
            - containerPort: 8000
          envFrom:
            - configMapRef:
                name: face-intel-config
            - secretRef:
                name: face-intel-secrets
          readinessProbe:
            httpGet:
              path: /health/ready
              port: 8000
            initialDelaySeconds: 5
            periodSeconds: 10
          livenessProbe:
            httpGet:
              path: /health/live
              port: 8000
            initialDelaySeconds: 30
            periodSeconds: 30
          resources:
            requests:
              cpu: "500m"
              memory: "1Gi"
            limits:
              cpu: "2000m"
              memory: "4Gi"
          volumeMounts:
            - name: data
              mountPath: /app/data
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: face-intel-pvc
---
apiVersion: v1
kind: Service
metadata:
  name: face-intel
spec:
  selector:
    app: face-intel
  ports:
    - port: 80
      targetPort: 8000
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: face-intel-pvc
spec:
  accessModes: ["ReadWriteOnce"]
  resources:
    requests:
      storage: 10Gi

ConfigMap + Secret

apiVersion: v1
kind: ConfigMap
metadata:
  name: face-intel-config
data:
  FI_ENVIRONMENT: "production"
  FI_DEBUG: "false"
  FI_LOG_LEVEL: "INFO"
  FI_LOG_JSON: "true"
  FI_RATE_LIMIT_PER_MINUTE: "120"
  FI_CACHE_ENABLED: "true"
  FI_CACHE_TTL_SECONDS: "86400"
  FI_DB_PATH: "/app/data/face_intel.db"
  FI_AUDIT_LOG_PATH: "/app/data/audit.log"
  FI_JOB_RETENTION_DAYS: "30"
  # Provider enable flags...
  FI_ENABLE_HAAR: "true"
  FI_ENABLE_DNN: "true"
  # ... etc
---
apiVersion: v1
kind: Secret
metadata:
  name: face-intel-secrets
type: Opaque
stringData:
  FI_SERPAPI_KEY: "..."
  FI_BING_API_KEY: "..."
  FI_TINEYE_PUBLIC_KEY: "..."
  FI_TINEYE_PRIVATE_KEY: "..."

Multi-replica gallery sharing

If you run >1 replica, data/gallery/ must be on a shared filesystem. Options:

1. EFS / NFS — mount an EFS access point at /app/data/gallery.
2. S3 + s3fs-fuse — mount an S3 bucket.
3. Move the gallery to an external store (Postgres, Redis) — requires extending ReferenceStore.

Horizontal Pod Autoscaler

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: face-intel-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: face-intel
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70

---

9. Reverse Proxy Considerations

Nginx

upstream face_intel {
    server 127.0.0.1:8000;
    # For multi-worker:
    # server 127.0.0.1:8001;
    # server 127.0.0.1:8002;
}

server {
    listen 443 ssl http2;
    server_name face-intel.example.com;

    ssl_certificate     /etc/ssl/face-intel.crt;
    ssl_certificate_key /etc/ssl/face-intel.key;

    client_max_body_size 25M;   # match FI_MAX_REQUEST_BODY_BYTES

    location / {
        proxy_pass http://face_intel;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        # Pass through request-id for tracing
        proxy_set_header X-Request-ID $request_id;

        # Long-running jobs (full_pipeline can take minutes)
        proxy_read_timeout 600s;
        proxy_send_timeout 600s;
    }

    # Health checks — bypass rate limiter implicitly (already bypassed in code)
    location /health {
        proxy_pass http://face_intel;
        access_log off;
    }
}

server {
    listen 80;
    server_name face-intel.example.com;
    return 301 https://$server_name$request_uri;
}

Caddy

face-intel.example.com {
    reverse_proxy 127.0.0.1:8000 {
        header_up X-Request-ID {http.request.uuid}
    }
    request_body {
        max_size 25MB
    }
}

Authentication at the proxy

Put authentication in front of the proxy — Face Intel has none by default. Common patterns:

• OAuth2 Proxy — Google/GitHub login, hands a session cookie.
• Cloudflare Access — Zero-trust identity provider.
• AWS API Gateway with a Lambda authorizer.
• mTLS at the Nginx layer (ssl_client_certificate).

Cache coordination

Each uvicorn worker process has its own in-memory Cache. There is no shared cache across workers. This means:

• Cache hit ratio is lower with more workers (each caches independently).
• For high-throughput deployments, replace storage/cache.py with a Redis-backed implementation that all workers share.

The Cache class has a clean interface (get, set, invalidate, clear, stats) — a drop-in Redis replacement is straightforward.

WebSockets / Server-Sent Events

Currently Face Intel uses plain HTTP. If you add streaming endpoints (SSE for job progress, WebSocket for live updates), make sure your reverse proxy supports them — Nginx does by default, but some load balancers (ALB) need explicit configuration.

---

10. Health Check Endpoints for Load Balancers

Endpoint	Purpose	Recommended for
GET /health	Liveness — always returns 200 {"status":"ok"}.	AWS ALB ping path, HAProxy option httpchk.
GET /health/live	Same as /health, separate path.	Kubernetes livenessProbe.
GET /health/ready	Readiness — currently always ready.	Kubernetes readinessProbe.
GET /health/providers	Per-provider health + circuit state.	Dashboards, alerting.

AWS ALB

• Health check path: /health
• Healthy threshold: 2
• Unhealthy threshold: 3
• Timeout: 5 s
• Interval: 10 s

HAProxy

backend face_intel
    option httpchk GET /health
    http-check expect status 200
    server app1 127.0.0.1:8000 check
    server app2 127.0.0.1:8001 check

Kubernetes probes

readinessProbe:
  httpGet:
    path: /health/ready
    port: 8000
  initialDelaySeconds: 5
  periodSeconds: 10
  failureThreshold: 3
livenessProbe:
  httpGet:
    path: /health/live
    port: 8000
  initialDelaySeconds: 30   # let models load
  periodSeconds: 30
  failureThreshold: 3

Alerting on circuit state

Poll /health/providers and alert when any provider's circuit_open == true for more than 5 minutes:

curl -s http://localhost:8000/health/providers | \
  jq -e '.providers[] | select(.circuit_open == true)' && \
  trigger_alert "Face Intel circuit breaker open"

---

11. TLS / HTTPS

Option A — terminate at the reverse proxy (recommended)

Run Face Intel on plain HTTP behind Nginx/Caddy/ALB that terminates TLS. The app doesn't know about TLS.

Option B — terminate at uvicorn

uvicorn app:app --host 0.0.0.0 --port 8443 \
  --ssl-keyfile /etc/ssl/face-intel.key \
  --ssl-certfile /etc/ssl/face-intel.crt

Useful for direct exposure without a proxy (e.g. internal microservice mesh).

HSTS

If you terminate at the proxy, add HSTS:

add_header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload" always;

---

12. Operational Runbook

Starting up

# 1. Verify env file is present
test -f .env || (echo "Missing .env"; exit 1)

# 2. Verify deps are installed
python -c "import fastapi, cv2, pydantic_settings; print('OK')"

# 3. Verify DB path is writable
python -c "import os; os.makedirs('$(dirname $(grep FI_DB_PATH .env | cut -d= -f2))', exist_ok=True); print('OK')"

# 4. Start
uvicorn app:app --host 0.0.0.0 --port 8000

Shutting down

Send SIGTERM (Ctrl-C or kill <pid>). The lifespan handler closes the SQLite connection cleanly:

# api/main.py
async def lifespan(app):
    ...
    yield
    container.database.close()

Upgrading

1. Pull new code.
2. pip install -r requirements.txt (in case new deps were added).
3. Run tests: python -m pytest tests/ -v.
4. Restart the server.

SQLite schema migrations are forward-compatible (new columns / tables use CREATE TABLE IF NOT EXISTS). No explicit migration tool is currently bundled.

Rotating API keys

1. Update FI_SERPAPI_KEY (etc.) in your secrets manager / .env.
2. Restart the server.
3. Verify with curl /providers/serpapi | jq .available.

Clearing the cache

# API
curl -X DELETE http://localhost:8000/cache

# Or direct on the host (no-op for in-memory cache, but useful
# if you've moved to a Redis-backed implementation)

Cleaning old jobs

python scripts/cleanup_jobs.py
# Or run via cron — see §5

Inspecting the gallery

curl http://localhost:8000/faces/gallery | jq .

Viewing audit log

tail -f /var/log/face-intel/audit.log | jq .

---

13. Follow-ups

The architecture is production-grade but the following ops infrastructure is not yet in the repo:

Item	Priority	Notes
Dockerfile	high	See §7 for the recommended baseline.
docker-compose.yml	medium	For local multi-service dev (Face Intel + Redis + mock reverse-search).
Redis-backed Cache	medium	Drop-in replacement for in-memory Cache; needed for >1 worker.
Prometheus metrics endpoint	medium	Expose /metrics with prometheus_client. Currently only JSON /stats.
Database migration tool	low	Alembic — currently schemas use CREATE TABLE IF NOT EXISTS.
Reference gallery REST API	medium	POST /faces/gallery/{name} to add embeddings. Currently only add_known_person() on the service.
Background job cleanup	medium	asyncio.create_task in lifespan to run cleanup_old_jobs daily.
Multi-process gallery sharing	high	Move ReferenceStore to Postgres/Redis for >1 replica.

---

See Also

• docs/CONFIGURATION.md — every FI_* env var.
• docs/API_REFERENCE.md — health endpoints.
• docs/TROUBLESHOOTING.md — startup failures, model download issues, dlib compilation.
• docs/ARCHITECTURE.md — composition root, lifespan, DI container.