docs/DEPLOYMENT.md

# Face Intel — Deployment Guide

This guide covers local development setup, Docker packaging,
environment configuration, database setup, storage layout, reverse
proxy integration, and load balancer health probes.

> **No Dockerfile exists yet** in the repository. This document
> describes the recommended Docker layout — implementing it is
> tracked as a follow-up task (see [§13](#13-follow-ups)).

---

## Table of Contents

1. [Prerequisites](#1-prerequisites)
2. [Local Development Setup](#2-local-development-setup)
3. [Running the Server](#3-running-the-server)
4. [Environment Variable Configuration](#4-environment-variable-configuration)
5. [Database Setup](#5-database-setup)
6. [Storage Directory Layout](#6-storage-directory-layout)
7. [Docker Deployment](#7-docker-deployment)
8. [Kubernetes Deployment](#8-kubernetes-deployment)
9. [Reverse Proxy Considerations](#9-reverse-proxy-considerations)
10. [Health Check Endpoints for Load Balancers](#10-health-check-endpoints-for-load-balancers)
11. [TLS / HTTPS](#11-tls--https)
12. [Operational Runbook](#12-operational-runbook)
13. [Follow-ups](#13-follow-ups)

---

## 1. Prerequisites

### Runtime

- **Python 3.11+** (3.12 recommended).
- **pip** and **venv**.
- **OpenCV system dependencies** (Linux): `libgl1`, `libglib2.0-0`.
  On Debian/Ubuntu: `apt install -y libgl1 libglib2.0-0`.
- **Chrome/Chromium** (only if you enable `selenium_scraper` or
  `google_lens`): `google-chrome` or `chromium-browser`.

### Python dependencies

The full list is in [`requirements.txt`](../requirements.txt). Key
packages:

| Package | Used by |
|---|---|
| `fastapi`, `uvicorn`, `pydantic`, `pydantic-settings` | API layer |
| `opencv-python`, `Pillow`, `numpy` | Image processing |
| `face-recognition`, `dlib` | Recognition (optional) |
| `mtcnn`, `tensorflow` | Detection (optional) |
| `beautifulsoup4`, `lxml`, `requests` | Scraping |
| `selenium`, `webdriver-manager` | JS-rendered scraping |
| `loguru` | Structured logging |
| `pytest`, `pytest-asyncio` | Test runner |

### Hardware

- **CPU-only:** Haar + DNN + image_analysis + forensics work well.
  DNN inference: ~30-80 ms per image.
- **GPU (optional):** DNN auto-detects CUDA. InsightFace and
  DeepFace benefit substantially. Requires `opencv-python-headless`
  built with CUDA or the `onnxruntime-gpu` package.
- **RAM:** 1 GB minimum for dev. 4 GB recommended for production with
  MTCNN/InsightFace enabled.
- **Disk:** ~50 MB for the DNN model, ~550 MB for InsightFace
  `buffalo_l` pack, ~1 GB for `data/` over a year of jobs.

---

## 2. Local Development Setup

### Step 1 — Clone and create a virtualenv

```bash
git clone <repo-url> face-intel
cd face-intel
python3.11 -m venv .venv
source .venv/bin/activate
```

### Step 2 — Install dependencies

```bash
pip install --upgrade pip
pip install -r requirements.txt
```

> **dlib compilation:** `dlib==19.24.2` requires CMake and a C++
> compiler. On Debian/Ubuntu: `apt install -y build-essential cmake`.
> Compilation takes ~5 minutes. See
> [`docs/TROUBLESHOOTING.md`](TROUBLESHOOTING.md) if it fails.

### Step 3 — Configure

```bash
cp .env.example .env
# Edit .env to enable/disable providers
```

At minimum, review:

- `FI_ENABLE_*` flags (which providers to load).
- `FI_SERPAPI_KEY`, `FI_BING_API_KEY`, `FI_TINEYE_PUBLIC_KEY`,
  `FI_TINEYE_PRIVATE_KEY` (only if you enable the paid providers).
- `FI_LOG_LEVEL`, `FI_DEBUG` for verbosity.

See [`docs/CONFIGURATION.md`](CONFIGURATION.md) for the full table.

### Step 4 — Verify installation

```bash
# Verify imports don't have circular deps
python scripts/check_imports.py

# Verify settings load
python -c "from config.settings import settings; print(settings.model_dump_json(indent=2))"

# Run the test suite (should be 145 passing)
python -m pytest tests/ -v
```

---

## 3. Running the Server

### Option A — Direct

```bash
python app.py
```

Reads `config/settings.py::settings` and runs uvicorn with reload if
`FI_DEBUG=true`.

### Option B — uvicorn directly

```bash
uvicorn app:app --host 0.0.0.0 --port 8000 --reload
```

Useful for development — `--reload` watches for file changes.

### Option C — production-style

```bash
uvicorn app:app --host 0.0.0.0 --port 8000 \
  --workers 4 --no-access-log --log-level info
```

- `--workers 4`: run 4 worker processes (each with its own in-memory
  cache — see [§9](#9-reverse-proxy-considerations)).
- `--no-access-log`: silence per-request access logs (useful when
  structured logging is enabled).

### Verifying it's up

```bash
curl http://localhost:8000/health
# → {"status":"ok"}

curl http://localhost:8000/providers | jq '.providers | length'
# → 24

curl http://localhost:8000/docs  # Swagger UI in browser
```

---

## 4. Environment Variable Configuration

All settings use the `FI_` prefix and are loaded by
`pydantic-settings` from (in priority order):

1. Real environment variables.
2. `.env` file at the project root.
3. Field defaults in `config/settings.py`.

### Quick reference

See [`docs/CONFIGURATION.md` §2](CONFIGURATION.md#2-quick-reference-all-settings)
for the full table of every setting, its env var name, type, and
default.

### Common patterns

**Single-process dev:**

```bash
export FI_DEBUG=true
export FI_LOG_LEVEL=DEBUG
python app.py
```

**Multi-process prod (systemd unit):**

```ini
# /etc/systemd/system/face-intel.service
[Unit]
Description=Face Intel
After=network.target

[Service]
Type=exec
User=face-intel
WorkingDirectory=/opt/face-intel
EnvironmentFile=/etc/face-intel/env
ExecStart=/opt/face-intel/.venv/bin/uvicorn app:app \
    --host 0.0.0.0 --port 8000 --workers 4 --no-access-log
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target
```

`/etc/face-intel/env`:

```env
FI_ENVIRONMENT=production
FI_DEBUG=false
FI_LOG_LEVEL=INFO
FI_LOG_JSON=true
FI_DB_PATH=/var/lib/face-intel/face_intel.db
FI_AUDIT_LOG_PATH=/var/log/face-intel/audit.log
# ... etc
```

### Loading env vars from a secrets manager

For Kubernetes, AWS, or HashiCorp Vault, write the secrets to env
vars in the container entrypoint before running uvicorn. The app
reads them at startup — once running, it doesn't poll for changes.

---

## 5. Database Setup

### Engine

SQLite via the stdlib `sqlite3` module. No external database server
required. See [`storage/database.py`](../storage/database.py).

### Schema

Auto-created on first run:

```sql
CREATE TABLE jobs (
    id           TEXT PRIMARY KEY,
    kind         TEXT NOT NULL,
    status       TEXT NOT NULL,
    created_at   TEXT NOT NULL,
    started_at   TEXT,
    completed_at TEXT,
    request      TEXT NOT NULL,    -- JSON blob
    image_hash   TEXT,
    error        TEXT
);

CREATE TABLE job_results (
    job_id     TEXT PRIMARY KEY,
    status     TEXT NOT NULL,
    report     TEXT,               -- JSON blob (UnifiedFaceReport)
    error      TEXT,
    elapsed_ms REAL,
    created_at TEXT NOT NULL,
    FOREIGN KEY (job_id) REFERENCES jobs(id)
);

CREATE INDEX idx_jobs_status ON jobs(status);
CREATE INDEX idx_jobs_created ON jobs(created_at);
```

### Path configuration

- Default: `data/face_intel.db` (relative to project root).
- Override via `FI_DB_PATH=/absolute/path/to/face_intel.db`.
- For tests: `FI_DB_PATH=:memory:` (in-memory, no disk file).

### Thread safety

`Database` uses `sqlite3.connect(check_same_thread=False)` plus an
internal `threading.Lock` around every operation. Safe to share
across threads in one process. **Not** safe across multiple worker
processes — each worker gets its own connection (and its own
in-memory state).

### Concurrency under multiple workers

If you run uvicorn with `--workers 4`, you'll have 4 independent
SQLite connections to the same file. SQLite handles this via file
locking — but write throughput drops sharply under contention. For
high-throughput multi-worker deployments:

1. Use a separate PostgreSQL/MySQL backend (would require a new
   `Database` implementation — not currently supported), OR
2. Run a single worker and rely on async concurrency
   (`orchestrator_max_concurrency`), OR
3. Shard jobs across multiple Face Intel instances, each with its
   own SQLite DB.

### Job retention

Set `FI_JOB_RETENTION_DAYS=7` (default) to keep jobs for one week.
Run `Database.cleanup_old_jobs(retention_days)` from a cron job:

```python
# scripts/cleanup_jobs.py
from config.settings import settings
from storage.database import Database

db = Database(path=settings.db_path)
deleted = db.cleanup_old_jobs(settings.job_retention_days)
print(f"Deleted {deleted} old jobs")
```

Cron entry:

```cron
0 3 * * * /opt/face-intel/.venv/bin/python /opt/face-intel/scripts/cleanup_jobs.py
```

### Backup

SQLite files can be backed up live using the `sqlite3` CLI:

```bash
sqlite3 /var/lib/face-intel/face_intel.db ".backup /backup/face_intel-$(date +%F).db"
```

Or use the Online Backup API via Python:

```python
import sqlite3
src = sqlite3.connect("/var/lib/face-intel/face_intel.db")
dst = sqlite3.connect("/backup/face_intel.db")
src.backup(dst)
```

---

## 6. Storage Directory Layout

`config/settings.py` auto-creates these directories at import time:

```
data/
├── face_intel.db          # SQLite database (jobs + results)
├── audit.log              # JSONL audit log
├── models/                # Auto-downloaded model files
│   ├── deploy.prototxt    #   DNN Caffe SSD prototxt (~28 KB)
│   └── res10_300x300_ssd_iter_140000.caffemodel  # (~10.7 MB)
├── gallery/               # Known-faces reference store
│   ├── manifest.json      #   {person_name: [embedding_filenames]}
│   ├── alice_0.npy        #   Per-person face embeddings
│   └── bob_0.npy
├── uploads/               # Source images saved by ArtifactStore
└── generated/             # Annotated images, montages
```

### Volume mounting in containers

For Docker/Kubernetes, mount `data/` as a persistent volume:

```yaml
volumes:
  - face-intel-data:/app/data
```

For multi-replica deployments, `data/gallery/` must be on a **shared
filesystem** (NFS, EFS, S3-FUSE) so all replicas see the same known
faces. Otherwise each replica has its own gallery.

### Model files

The DNN provider auto-downloads its model files on first use via
`urllib.request.urlretrieve` to `data/models/`. If your deployment
has no internet access:

1. Pre-download the files on a build machine:

   ```bash
   mkdir -p data/models
   curl -L -o data/models/deploy.prototxt \
     https://raw.githubusercontent.com/opencv/opencv_3rdparty/dnn_samples_face_detector_20170830/deploy.prototxt
   curl -L -o data/models/res10_300x300_ssd_iter_140000.caffemodel \
     https://raw.githubusercontent.com/opencv/opencv_3rdparty/dnn_samples_face_detector_20170830/res10_300x300_ssd_iter_140000.caffemodel
   ```

2. Bake them into your Docker image or volume.

### Audit log

`utils.audit.audit_log()` appends one JSON line per sensitive
operation (gallery mutations, reverse image searches). Rotate with
`logrotate`:

```
# /etc/logrotate.d/face-intel
/var/log/face-intel/audit.log {
    daily
    rotate 30
    compress
    missingok
    notifempty
    copytruncate
}
```

---

## 7. Docker Deployment

> **No Dockerfile is currently in the repo.** The snippet below is
> the recommended baseline. Track follow-up work in
> [§13](#13-follow-ups).

### `Dockerfile` (recommended)

```dockerfile
FROM python:3.11-slim AS base

# OpenCV system deps + Chrome (for Selenium) + build tools for dlib
RUN apt-get update && apt-get install -y --no-install-recommends \
      libgl1 libglib2.0-0 \
      build-essential cmake \
      chromium \
    && rm -rf /var/lib/apt/lists/*

WORKDIR /app

# Install Python deps first (better layer caching)
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Copy app code
COPY . .

# Create runtime dirs
RUN mkdir -p /app/data/models /app/data/gallery \
             /app/data/uploads /app/data/generated

ENV FI_ENVIRONMENT=production \
    FI_DEBUG=false \
    FI_LOG_JSON=true \
    FI_DB_PATH=/app/data/face_intel.db \
    FI_AUDIT_LOG_PATH=/app/data/audit.log \
    FI_SELENIUM_HEADLESS=true

EXPOSE 8000

# Single worker — multi-worker requires shared storage / external DB
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000", \
     "--no-access-log", "--log-level", "info"]
```

### Build & run

```bash
docker build -t face-intel:latest .
docker run -d --name face-intel \
  -p 8000:8000 \
  -v face-intel-data:/app/data \
  --env-file .env \
  face-intel:latest
```

### `.dockerignore`

```
.venv/
__pycache__/
*.pyc
.pytest_cache/
data/
.env
*.log
.git/
```

### Multi-stage build for smaller image

If image size matters, split into a `builder` stage that compiles
`dlib` and a slim `runtime` stage:

```dockerfile
FROM python:3.11-slim AS builder
RUN apt-get update && apt-get install -y build-essential cmake libgl1 libglib2.0-0
WORKDIR /app
COPY requirements.txt .
RUN pip install --user -r requirements.txt

FROM python:3.11-slim AS runtime
RUN apt-get update && apt-get install -y --no-install-recommends \
      libgl1 libglib2.0-0 chromium && rm -rf /var/lib/apt/lists/*
WORKDIR /app
COPY --from=builder /root/.local /root/.local
COPY . .
ENV PATH=/root/.local/bin:$PATH
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]
```

Final image: ~600 MB (vs ~1.5 GB without multi-stage).

### Image variants

| Variant | Tag pattern | Use case |
|---|---|---|
| CPU-only | `face-intel:latest` | Default. Works everywhere. |
| CPU-only, no Selenium | `face-intel:cpu-slim` | Smaller (~400 MB). No Chrome. |
| GPU | `face-intel:gpu-cuda12` | Built on `nvidia/cuda:12.x-runtime`. For InsightFace/DeepFace acceleration. |

---

## 8. Kubernetes Deployment

### Deployment + Service

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: face-intel
  labels:
    app: face-intel
spec:
  replicas: 2
  selector:
    matchLabels:
      app: face-intel
  template:
    metadata:
      labels:
        app: face-intel
    spec:
      containers:
        - name: face-intel
          image: face-intel:latest
          ports:
            - containerPort: 8000
          envFrom:
            - configMapRef:
                name: face-intel-config
            - secretRef:
                name: face-intel-secrets
          readinessProbe:
            httpGet:
              path: /health/ready
              port: 8000
            initialDelaySeconds: 5
            periodSeconds: 10
          livenessProbe:
            httpGet:
              path: /health/live
              port: 8000
            initialDelaySeconds: 30
            periodSeconds: 30
          resources:
            requests:
              cpu: "500m"
              memory: "1Gi"
            limits:
              cpu: "2000m"
              memory: "4Gi"
          volumeMounts:
            - name: data
              mountPath: /app/data
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: face-intel-pvc
---
apiVersion: v1
kind: Service
metadata:
  name: face-intel
spec:
  selector:
    app: face-intel
  ports:
    - port: 80
      targetPort: 8000
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: face-intel-pvc
spec:
  accessModes: ["ReadWriteOnce"]
  resources:
    requests:
      storage: 10Gi
```

### ConfigMap + Secret

```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: face-intel-config
data:
  FI_ENVIRONMENT: "production"
  FI_DEBUG: "false"
  FI_LOG_LEVEL: "INFO"
  FI_LOG_JSON: "true"
  FI_RATE_LIMIT_PER_MINUTE: "120"
  FI_CACHE_ENABLED: "true"
  FI_CACHE_TTL_SECONDS: "86400"
  FI_DB_PATH: "/app/data/face_intel.db"
  FI_AUDIT_LOG_PATH: "/app/data/audit.log"
  FI_JOB_RETENTION_DAYS: "30"
  # Provider enable flags...
  FI_ENABLE_HAAR: "true"
  FI_ENABLE_DNN: "true"
  # ... etc
---
apiVersion: v1
kind: Secret
metadata:
  name: face-intel-secrets
type: Opaque
stringData:
  FI_SERPAPI_KEY: "..."
  FI_BING_API_KEY: "..."
  FI_TINEYE_PUBLIC_KEY: "..."
  FI_TINEYE_PRIVATE_KEY: "..."
```

### Multi-replica gallery sharing

If you run >1 replica, `data/gallery/` must be on a shared
filesystem. Options:

1. **EFS / NFS** — mount an EFS access point at `/app/data/gallery`.
2. **S3 + s3fs-fuse** — mount an S3 bucket.
3. **Move the gallery to an external store** (Postgres, Redis) —
   requires extending `ReferenceStore`.

### Horizontal Pod Autoscaler

```yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: face-intel-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: face-intel
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
```

---

## 9. Reverse Proxy Considerations

### Nginx

```nginx
upstream face_intel {
    server 127.0.0.1:8000;
    # For multi-worker:
    # server 127.0.0.1:8001;
    # server 127.0.0.1:8002;
}

server {
    listen 443 ssl http2;
    server_name face-intel.example.com;

    ssl_certificate     /etc/ssl/face-intel.crt;
    ssl_certificate_key /etc/ssl/face-intel.key;

    client_max_body_size 25M;   # match FI_MAX_REQUEST_BODY_BYTES

    location / {
        proxy_pass http://face_intel;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        # Pass through request-id for tracing
        proxy_set_header X-Request-ID $request_id;

        # Long-running jobs (full_pipeline can take minutes)
        proxy_read_timeout 600s;
        proxy_send_timeout 600s;
    }

    # Health checks — bypass rate limiter implicitly (already bypassed in code)
    location /health {
        proxy_pass http://face_intel;
        access_log off;
    }
}

server {
    listen 80;
    server_name face-intel.example.com;
    return 301 https://$server_name$request_uri;
}
```

### Caddy

```caddyfile
face-intel.example.com {
    reverse_proxy 127.0.0.1:8000 {
        header_up X-Request-ID {http.request.uuid}
    }
    request_body {
        max_size 25MB
    }
}
```

### Authentication at the proxy

Put authentication in front of the proxy — Face Intel has none by
default. Common patterns:

- **OAuth2 Proxy** — Google/GitHub login, hands a session cookie.
- **Cloudflare Access** — Zero-trust identity provider.
- **AWS API Gateway** with a Lambda authorizer.
- **mTLS** at the Nginx layer (`ssl_client_certificate`).

### Cache coordination

Each uvicorn worker process has its own in-memory `Cache`. There is
**no shared cache** across workers. This means:

- Cache hit ratio is lower with more workers (each caches
  independently).
- For high-throughput deployments, replace
  [`storage/cache.py`](../storage/cache.py) with a Redis-backed
  implementation that all workers share.

The `Cache` class has a clean interface (`get`, `set`, `invalidate`,
`clear`, `stats`) — a drop-in Redis replacement is straightforward.

### WebSockets / Server-Sent Events

Currently Face Intel uses plain HTTP. If you add streaming endpoints
(SSE for job progress, WebSocket for live updates), make sure your
reverse proxy supports them — Nginx does by default, but some load
balancers (ALB) need explicit configuration.

---

## 10. Health Check Endpoints for Load Balancers

| Endpoint | Purpose | Recommended for |
|---|---|---|
| `GET /health` | Liveness — always returns `200 {"status":"ok"}`. | AWS ALB ping path, HAProxy `option httpchk`. |
| `GET /health/live` | Same as `/health`, separate path. | Kubernetes `livenessProbe`. |
| `GET /health/ready` | Readiness — currently always ready. | Kubernetes `readinessProbe`. |
| `GET /health/providers` | Per-provider health + circuit state. | Dashboards, alerting. |

### AWS ALB

- **Health check path:** `/health`
- **Healthy threshold:** 2
- **Unhealthy threshold:** 3
- **Timeout:** 5 s
- **Interval:** 10 s

### HAProxy

```
backend face_intel
    option httpchk GET /health
    http-check expect status 200
    server app1 127.0.0.1:8000 check
    server app2 127.0.0.1:8001 check
```

### Kubernetes probes

```yaml
readinessProbe:
  httpGet:
    path: /health/ready
    port: 8000
  initialDelaySeconds: 5
  periodSeconds: 10
  failureThreshold: 3
livenessProbe:
  httpGet:
    path: /health/live
    port: 8000
  initialDelaySeconds: 30   # let models load
  periodSeconds: 30
  failureThreshold: 3
```

### Alerting on circuit state

Poll `/health/providers` and alert when any provider's
`circuit_open == true` for more than 5 minutes:

```bash
curl -s http://localhost:8000/health/providers | \
  jq -e '.providers[] | select(.circuit_open == true)' && \
  trigger_alert "Face Intel circuit breaker open"
```

---

## 11. TLS / HTTPS

### Option A — terminate at the reverse proxy (recommended)

Run Face Intel on plain HTTP behind Nginx/Caddy/ALB that terminates
TLS. The app doesn't know about TLS.

### Option B — terminate at uvicorn

```bash
uvicorn app:app --host 0.0.0.0 --port 8443 \
  --ssl-keyfile /etc/ssl/face-intel.key \
  --ssl-certfile /etc/ssl/face-intel.crt
```

Useful for direct exposure without a proxy (e.g. internal
microservice mesh).

### HSTS

If you terminate at the proxy, add HSTS:

```nginx
add_header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload" always;
```

---

## 12. Operational Runbook

### Starting up

```bash
# 1. Verify env file is present
test -f .env || (echo "Missing .env"; exit 1)

# 2. Verify deps are installed
python -c "import fastapi, cv2, pydantic_settings; print('OK')"

# 3. Verify DB path is writable
python -c "import os; os.makedirs('$(dirname $(grep FI_DB_PATH .env | cut -d= -f2))', exist_ok=True); print('OK')"

# 4. Start
uvicorn app:app --host 0.0.0.0 --port 8000
```

### Shutting down

Send `SIGTERM` (Ctrl-C or `kill <pid>`). The lifespan handler closes
the SQLite connection cleanly:

```python
# api/main.py
async def lifespan(app):
    ...
    yield
    container.database.close()
```

### Upgrading

1. Pull new code.
2. `pip install -r requirements.txt` (in case new deps were added).
3. Run tests: `python -m pytest tests/ -v`.
4. Restart the server.

SQLite schema migrations are forward-compatible (new columns /
tables use `CREATE TABLE IF NOT EXISTS`). No explicit migration tool
is currently bundled.

### Rotating API keys

1. Update `FI_SERPAPI_KEY` (etc.) in your secrets manager / `.env`.
2. Restart the server.
3. Verify with `curl /providers/serpapi | jq .available`.

### Clearing the cache

```bash
# API
curl -X DELETE http://localhost:8000/cache

# Or direct on the host (no-op for in-memory cache, but useful
# if you've moved to a Redis-backed implementation)
```

### Cleaning old jobs

```bash
python scripts/cleanup_jobs.py
# Or run via cron — see §5
```

### Inspecting the gallery

```bash
curl http://localhost:8000/faces/gallery | jq .
```

### Viewing audit log

```bash
tail -f /var/log/face-intel/audit.log | jq .
```

---

## 13. Follow-ups

The architecture is production-grade but the following ops
infrastructure is not yet in the repo:

| Item | Priority | Notes |
|---|---|---|
| `Dockerfile` | high | See [§7](#7-docker-deployment) for the recommended baseline. |
| `docker-compose.yml` | medium | For local multi-service dev (Face Intel + Redis + mock reverse-search). |
| Redis-backed `Cache` | medium | Drop-in replacement for in-memory `Cache`; needed for >1 worker. |
| Prometheus metrics endpoint | medium | Expose `/metrics` with `prometheus_client`. Currently only JSON `/stats`. |
| Database migration tool | low | Alembic — currently schemas use `CREATE TABLE IF NOT EXISTS`. |
| Reference gallery REST API | medium | `POST /faces/gallery/{name}` to add embeddings. Currently only `add_known_person()` on the service. |
| Background job cleanup | medium | `asyncio.create_task` in lifespan to run `cleanup_old_jobs` daily. |
| Multi-process gallery sharing | high | Move `ReferenceStore` to Postgres/Redis for >1 replica. |

---

## See Also

- [`docs/CONFIGURATION.md`](CONFIGURATION.md) — every `FI_*` env var.
- [`docs/API_REFERENCE.md`](API_REFERENCE.md) — health endpoints.
- [`docs/TROUBLESHOOTING.md`](TROUBLESHOOTING.md) — startup failures,
  model download issues, dlib compilation.
- [`docs/ARCHITECTURE.md`](ARCHITECTURE.md) — composition root,
  lifespan, DI container.