docs/TROUBLESHOOTING.md

# Face Intel — Troubleshooting Guide

This guide covers common errors, their causes, and how to fix them.
Organized by symptom — find your error message or behavior below.

> **First steps for any issue:**
> 1. Check `/health/providers` — are circuits open? Are providers
>    `not_configured`?
> 2. Check `/stats` — are failures climbing? Are cache hits 0?
> 3. Check the structured logs — every line carries `eid`, `pid`,
>    `retry`, `status` for tracing.
> 4. Check `/providers` — the `errors` field shows manifest
>    discovery failures.

---

## Table of Contents

1. [Startup Failures](#1-startup-failures)
2. [Provider Not Available / `not_configured`](#2-provider-not-available--not_configured)
3. [Circuit Breaker Open](#3-circuit-breaker-open)
4. [Model Download Failures](#4-model-download-failures)
5. [dlib Compilation Issues](#5-dlib-compilation-issues)
6. [Selenium / Chrome Issues](#6-selenium--chrome-issues)
7. [Cache Issues](#7-cache-issues)
8. [Database Issues](#8-database-issues)
9. [Performance Tuning](#9-performance-tuning)
10. [API / HTTP Errors](#10-api--http-errors)
11. [Image Validation Errors](#11-image-validation-errors)
12. [Logging Issues](#12-logging-issues)
13. [Recovery Procedures](#13-recovery-procedures)

---

## 1. Startup Failures

### Symptom: `ModuleNotFoundError: No module named 'cv2'`

**Cause:** OpenCV not installed.

**Fix:**

```bash
pip install opencv-python==4.9.0.80
```

If you see `libGL.so.1: cannot open shared object file` on Linux:

```bash
# Debian/Ubuntu
sudo apt install -y libgl1 libglib2.0-0

# Or install the headless variant (no GUI deps)
pip uninstall opencv-python
pip install opencv-python-headless==4.9.0.80
```

### Symptom: `ModuleNotFoundError: No module named 'pydantic_settings'`

**Cause:** Wrong pydantic version. Face Intel requires pydantic 2.x
with the separate `pydantic-settings` package.

**Fix:**

```bash
pip install pydantic==2.6.1 pydantic-settings==2.2.1
```

### Symptom: `ImportError: cannot import name 'X' from 'Y'` during `build_container()`

**Cause:** Circular import or a layer violation. The refactor
enforces strict one-way dependency direction.

**Fix:**

1. Run `python scripts/check_imports.py` to detect cycles.
2. Check the layer the failing module is in — it must only import
   from layers below it (see [`docs/ARCHITECTURE.md` §1](ARCHITECTURE.md)).
3. If you recently added a provider, ensure it imports only from
   `models/`, `utils/`, `config/`, `pipeline/`, `providers/base.py`.

### Symptom: App boots but `/providers` returns an empty list

**Cause:** All providers are disabled in settings, OR none of them
match the manifest.

**Fix:**

1. Check `.env`:

   ```bash
   grep FI_ENABLE .env
   ```

2. Check the loaded settings:

   ```bash
   python -c "from config.settings import settings; print([k for k,v in settings.model_dump().items() if k.startswith('enable_') and v])"
   ```

3. Check `/providers` for `errors`:

   ```bash
   curl -s http://localhost:8000/providers | jq '.errors'
   ```

---

## 2. Provider Not Available / `not_configured`

A provider shows up in `/providers` with `status: "not_configured"`
and `available: false`.

### Cause 1: Optional Python dependency missing

**Symptom:**

```json
{
  "name": "insightface",
  "status": "not_configured",
  "available": false
}
```

`/providers` `errors` field:

```json
{ "insightface": "missing dependency: No module named 'insightface'" }
```

**Fix:**

```bash
pip install insightface onnxruntime
# Also set in .env:
# FI_ENABLE_INSIGHTFACE=true
```

Restart and verify:

```bash
curl -s http://localhost:8000/providers/insightface | jq '.available'
# → true
```

### Cause 2: API key not set

**Symptom:** Provider enabled in config but `is_available()` returns
`False` because the key is empty.

```bash
curl -s http://localhost:8000/providers/serpapi | jq '.'
# → {"name": "serpapi", "status": "not_configured", "available": false, ...}
```

**Fix:**

1. Edit `.env`:

   ```env
   FI_ENABLE_SERPAPI=true
   FI_SERPAPI_KEY=your_actual_key_here
   ```

2. Restart.

3. Verify:

   ```bash
   curl -s http://localhost:8000/providers/serpapi | jq '.available'
   # → true
   ```

See [`docs/CONFIGURATION.md` §17](CONFIGURATION.md#17-how-to-configure-api-keys)
for the full key-setup workflow for each paid provider.

### Cause 3: Provider's `is_available()` raised an exception

**Symptom:** Provider shows `not_configured` but no manifest error is
recorded. The registry's `info()` method catches the exception and
returns `available=False`.

**Fix:** Add logging to your provider's `is_available()`:

```python
def is_available(self) -> bool:
    try:
        return self._net is not None
    except Exception as e:
        logger.warning(f"is_available() raised: {e}")
        return False
```

### Cause 4: Provider class name doesn't match manifest

**Symptom:** Manifest entry says `class_name="MyProvider"` but the
file declares `class MyProviderV2`. The registry will fail to
instantiate.

**Fix:** Match the class name exactly. See
[`docs/PROVIDERS.md` §7](PROVIDERS.md#7-manifest-entry-format).

### Cause 5: Provider constructor raised

**Symptom:** Manifest error like `"init error: RuntimeError(...)"`.

**Fix:** Check the constructor. Provider constructors must not raise
for missing optional deps — they should catch and store the error,
then return `False` from `is_available()`:

```python
def __init__(self, settings):
    super().__init__(settings=settings)
    self._init_error = None
    try:
        self._model = load_model()
    except Exception as e:
        self._init_error = str(e)
```

See the `dnn` provider
([`providers/detection/dnn.py`](../providers/detection/dnn.py))
for the canonical pattern.

---

## 3. Circuit Breaker Open

### Symptom

A provider that was previously working is now skipped — `providers_invoked`
no longer includes it, and `/health/providers` shows `circuit_open: true`:

```json
{
  "name": "serpapi",
  "healthy": false,
  "consecutive_failures": 5,
  "circuit_open": true,
  "avg_latency_ms": 0.0
}
```

### How the circuit breaker works

See [`docs/CONFIGURATION.md` §11](CONFIGURATION.md#11-health--circuit-breaker).
Briefly:

1. Provider fails → `consecutive_failures += 1`.
2. When `consecutive_failures >= circuit_breaker_failure_threshold`
   (default 5), the circuit **opens**.
3. After `circuit_breaker_recovery_seconds` (default 120s), the
   circuit transitions to **half-open**: the next call is allowed.
4. Success → circuit closes. Failure → circuit re-opens.

### Cause 1: Genuine provider failure

**Diagnosis:**

```bash
curl -s http://localhost:8000/stats | \
  jq '.counters | to_entries | map(select(.key | startswith("failures.")))'
```

If `failures.<provider>` is climbing, the provider is genuinely
failing. Check its logs:

```bash
# Tail logs filtered to the provider
tail -f /var/log/face-intel/app.log | grep "pid=<provider_name>"
```

Common root causes:

- API key expired / revoked.
- Rate limited by upstream (429).
- Network partition.
- Upstream service is down.

**Fix:** Resolve the underlying issue. The circuit will close
automatically on the next successful call after the recovery window.

### Cause 2: Misconfigured retry policy

If `retry_max_attempts` is too low and the provider has transient
failures, you can hit the failure threshold quickly.

**Fix:** Tune retry:

```env
FI_RETRY_MAX_ATTEMPTS=5
FI_RETRY_INITIAL_BACKOFF_SECONDS=1.0
FI_RETRY_MAX_BACKOFF_SECONDS=30.0
FI_CIRCUIT_BREAKER_FAILURE_THRESHOLD=10
FI_CIRCUIT_BREAKER_RECOVERY_SECONDS=300
```

### Cause 3: Provider timeout too aggressive

If `orchestrator_timeout_seconds=10` but the provider takes 15s on
average, every call times out → circuit opens.

**Fix:**

```env
FI_ORCHESTRATOR_TIMEOUT_SECONDS=60
```

### Manual reset

There's no API endpoint to manually close a circuit. Two options:

1. **Wait** — the circuit auto-transitions to half-open after
   `circuit_breaker_recovery_seconds`.
2. **Restart** — restart the process. The metrics are in-memory only;
   they reset on restart (the circuit starts closed).

For a programmatic reset (e.g. for testing), use the metrics facade
directly:

```python
container.metrics.health.reset()
```

### Verifying recovery

```bash
# Trigger a single call to the provider
curl -X POST http://localhost:8000/faces/detect \
  -H "Content-Type: application/json" \
  -d '{"image_url": "https://example.com/face.jpg", "providers": ["<name>"]}'

# Check circuit state
curl -s http://localhost:8000/health/providers | \
  jq '.providers[] | select(.name == "<name>") | .circuit_open'
```

---

## 4. Model Download Failures

### Affected providers

- `dnn` — downloads `deploy.prototxt` (~28 KB) and
  `res10_300x300_ssd_iter_140000.caffemodel` (~10.7 MB) on first
  use to `data/models/`.
- `insightface` — downloads model packs (~100-550 MB) on first use.
- `deepface` — downloads backend weights on first use.

### Symptom: `urllib.error.URLError: <urlopen error ...>`

**Cause:** No internet access from the host, or GitHub raw content
is blocked.

**Fix — pre-download the DNN model files:**

```bash
mkdir -p data/models
curl -L -o data/models/deploy.prototxt \
  https://raw.githubusercontent.com/opencv/opencv_3rdparty/dnn_samples_face_detector_20170830/deploy.prototxt
curl -L -o data/models/res10_300x300_ssd_iter_140000.caffemodel \
  https://raw.githubusercontent.com/opencv/opencv_3rdparty/dnn_samples_face_detector_20170830/res10_300x300_ssd_iter_140000.caffemodel

# Verify checksums (sha256)
sha256sum data/models/*
```

Then restart. The provider will see the files already exist and skip
the download.

### Symptom: Model file is corrupt (cv2.dnn.readNetFromCaffe fails)

**Cause:** Partial download (e.g. network interrupted).

**Fix:**

```bash
rm data/models/deploy.prototxt data/models/res10_300x300_ssd_iter_140000.caffemodel
# Restart — provider will re-download
```

### Symptom: Downloads work but model fails to load

```python
cv2.error: OpenCV(4.9.0) /io/opencv/modules/dnn/src/caffe/caffe_importer.cpp...
```

**Cause:** Corrupt file or OpenCV version mismatch.

**Fix:**

1. Re-download (see above).
2. Verify OpenCV version: `python -c "import cv2; print(cv2.__version__)"`.
   Must be `4.9.0.80` per `requirements.txt`.
3. Try `opencv-python-headless` if GUI deps cause issues.

### Symptom: InsightFace model pack download is very slow

**Cause:** InsightFace downloads from AWS S3 in the US-east region.

**Fix:** Pre-download on a build machine and bake into your Docker
image:

```dockerfile
RUN python -c "from insightface.app import FaceAnalysis; FaceAnalysis(name='buffalo_l').prepare(ctx_id=-1)"
```

This caches the model in `~/.insightface/models/`. Copy that
directory into the production image.

### Air-gapped deployment

For hosts with zero internet access:

1. On a build machine with internet, run the app once to trigger
   all model downloads.
2. Tar up `data/models/` and `~/.insightface/` (or wherever the
   optional libs cache models).
3. Copy to the production host at the same paths.
4. Restart — providers will find the models locally.

---

## 5. dlib Compilation Issues

`dlib==19.24.2` does not ship prebuilt wheels for all platforms —
on Linux you'll often need to compile from source.

### Symptom: `error: command 'gcc' failed` during `pip install dlib`

**Cause:** Missing build tools.

**Fix (Debian/Ubuntu):**

```bash
sudo apt install -y build-essential cmake python3-dev
pip install dlib==19.24.2
```

**Fix (Alpine):**

```bash
apk add --no-cache build-base cmake linux-headers
pip install dlib==19.24.2
```

**Fix (macOS):**

```bash
xcode-select --install
brew install cmake
pip install dlib==19.24.2
```

### Symptom: Compilation hangs or runs out of memory

**Cause:** dlib compiles with all cores by default; on small VMs
this OOMs.

**Fix:** Limit parallelism:

```bash
pip install --no-build-isolation dlib==19.24.2 \
  --config-settings cmake.define.BUILD_SHARED_LIBS=OFF \
  -j 2
```

Or use a machine with ≥4 GB RAM for the build step.

### Symptom: `ModuleNotFoundError: No module named 'dlib'` after install

**Cause:** Wrong Python interpreter — dlib installed in a different
venv than the one running Face Intel.

**Fix:**

```bash
which python    # should be your venv python
python -m pip install dlib==19.24.2
python -c "import dlib; print(dlib.__version__)"
```

### Symptom: `face_recognition` provider crashes with `Illegal instruction`

**Cause:** dlib compiled with AVX instructions but CPU doesn't
support them (common on older Xeons / VPS hosts).

**Fix:** Recompile dlib without AVX:

```bash
pip uninstall dlib
git clone https://github.com/davisking/dlib.git
cd dlib
mkdir build && cd build
cmake .. -DUSE_AVX_INSTRUCTIONS=OFF -DCMAKE_BUILD_TYPE=Release
make -j 4
cd ..
python setup.py install
```

### Alternative: skip dlib entirely

If you don't need recognition:

```env
FI_ENABLE_FACE_RECOGNITION=false
```

Then `pip uninstall dlib face-recognition` to remove the broken
install. The other 23 providers continue to work.

---

## 6. Selenium / Chrome Issues

Affects: `selenium` (scraper), `google_lens` (reverse search).

### Symptom: `selenium.common.exceptions.WebDriverException: Message: unknown error: cannot find Chrome binary`

**Cause:** Chrome/Chromium not installed.

**Fix (Debian/Ubuntu):**

```bash
# Stable Chrome
wget -q https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
sudo apt install -y ./google-chrome-stable_current_amd64.deb

# Or Chromium
sudo apt install -y chromium-browser
```

**Fix (macOS):**

```bash
brew install --cask google-chrome
```

**Fix (Docker):** Add to Dockerfile:

```dockerfile
RUN apt-get update && apt-get install -y chromium
ENV CHROME_BIN=/usr/bin/chromium
```

### Symptom: `selenium.common.exceptions.SessionNotCreatedException: Message: session not created: Chrome version must be between ...`

**Cause:** `webdriver-manager` downloaded a ChromeDriver version
that doesn't match your installed Chrome.

**Fix:**

```bash
google-chrome --version  # note the major version
# Then ensure chromedriver matches
pip install --upgrade webdriver-manager
```

Or use Chrome for Testing with pinned versions:

```bash
# Install Chrome for Testing 121
# https://googlechromelabs.github.io/chrome-for-testing/
```

### Symptom: Selenium tests hang indefinitely

**Cause:** `selenium_implicit_wait` is too high, or Chrome is
waiting for a never-resolving resource.

**Fix:**

1. Lower implicit wait:

   ```env
   FI_SELENIUM_IMPLICIT_WAIT=5
   ```

2. Reduce scroll iterations:

   ```env
   FI_SELENIUM_SCROLL_ITERATIONS=2
   ```

3. Set a global scrape timeout:

   ```env
   FI_SCRAPE_TIMEOUT=15
   ```

4. For debugging, run non-headless to see what's happening:

   ```env
   FI_SELENIUM_HEADLESS=false
   ```

### Symptom: Headless Chrome returns empty pages

**Cause:** Some sites detect headless Chrome and serve different
content. Or JavaScript hasn't finished rendering.

**Fix:**

1. Increase scroll iterations:

   ```env
   FI_SELENIUM_SCROLL_ITERATIONS=10
   ```

2. Use a more realistic User-Agent:

   ```env
   FI_USER_AGENT="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36"
   ```

3. Fall back to the `beautifulsoup` scraper (static HTML) for sites
   that don't require JS:

   ```env
   FI_ENABLE_SELENIUM_SCRAPER=false
   FI_ENABLE_BEAUTIFULSOUP_SCRAPER=true
   ```

### Symptom: Chrome crashes with `--no-sandbox` warning in containers

**Cause:** Running Chrome as root in a container requires
`--no-sandbox`. Selenium providers should set this automatically;
if not, set in your Chrome options.

**Fix:** Make sure your selenium provider uses:

```python
from selenium.webdriver.chrome.options import Options
opts = Options()
opts.add_argument("--no-sandbox")
opts.add_argument("--disable-dev-shm-usage")  # critical in containers
opts.add_argument("--headless=new")
```

The `--disable-dev-shm-usage` flag tells Chrome to use `/tmp`
instead of `/dev/shm` (which is too small in default containers).

---

## 7. Cache Issues

### Symptom: Cache hit ratio is always 0

**Cause:** Cache is disabled, or every request uses a different
image (cache key is `provider:image_hash`).

**Diagnosis:**

```bash
curl -s http://localhost:8000/cache | jq
```

```json
{
  "entries": 0,
  "max_entries": 1000,
  "ttl_seconds": 3600,
  "hits": 0,
  "misses": 24,
  "hit_ratio": 0.0,
  "evictions": 0
}
```

**Fix:**

1. Ensure cache is enabled:

   ```env
   FI_CACHE_ENABLED=true
   ```

2. Verify you're hitting the same image. The cache key is
   `f"{provider_name}:{image_hash}"` where `image_hash` is the
   SHA-256 of the preprocessed image. Slight image differences (URL
   vs base64, re-encoded JPEGs) produce different hashes.

3. If you're load-testing with random images, expect 0% hit ratio.

### Symptom: Cache evictions are high

**Cause:** `cache_max_entries` is too small for the working set.

**Diagnosis:**

```bash
curl -s http://localhost:8000/cache | jq '.evictions'
```

**Fix:**

```env
FI_CACHE_MAX_ENTRIES=10000
```

### Symptom: Same image returns different results across requests

**Cause:** Cache was cleared, or the TTL expired.

**Diagnosis:**

1. Check `GET /cache` for `cleared` events — but actually the cache
   doesn't expose clear count via API. Check the structured logs
   for `cache clear` events.
2. Check `cache_ttl_seconds` — if it's very short, entries expire
   between requests.

**Fix:**

```env
FI_CACHE_TTL_SECONDS=86400  # 24 hours
```

### Symptom: Stale data — provider updated but cache still serves old result

**Cause:** Cache TTL hasn't expired.

**Fix:**

1. Clear the cache:

   ```bash
   curl -X DELETE http://localhost:8000/cache
   ```

2. Or invalidate a specific key (not exposed via API — would need a
   new endpoint, see [`docs/API_REFERENCE.md` §9](API_REFERENCE.md#9-cache-endpoints)).

3. For permanent fix, lower the TTL or implement a webhook that
   clears the cache when the upstream model updates.

### Symptom: Memory usage grows unboundedly

**Cause:** Each cache entry stores a full `ProviderResult` including
the `raw` field, which can be large for image-analysis providers.

**Fix:**

1. Lower `cache_max_entries`:

   ```env
   FI_CACHE_MAX_ENTRIES=100
   ```

2. For multi-worker deployments, replace the in-memory cache with
   Redis (see [`docs/DEPLOYMENT.md` §9](DEPLOYMENT.md#9-reverse-proxy-considerations)).

3. If you can afford recomputation, disable the cache entirely:

   ```env
   FI_CACHE_ENABLED=false
   ```

---

## 8. Database Issues

### Symptom: `sqlite3.OperationalError: database is locked`

**Cause:** SQLite writer contention — multiple workers writing
concurrently.

**Fix:**

1. Run a single uvicorn worker (no `--workers N`).
2. Or shard: different DB paths per worker (each worker serves a
   subset of requests).
3. Or migrate to Postgres (would require a new `Database`
   implementation — not currently supported).

### Symptom: `sqlite3.OperationalError: unable to open database file`

**Cause:** The `FI_DB_PATH` directory doesn't exist or isn't
writable.

**Fix:**

```bash
# Check the path
python -c "from config.settings import settings; print(settings.db_path)"

# Ensure the directory exists
mkdir -p $(dirname $(python -c "from config.settings import settings; print(settings.db_path)"))

# Check permissions
ls -la $(dirname $(python -c "from config.settings import settings; print(settings.db_path)"))
```

### Symptom: Jobs disappear from `/jobs` after a restart

**Cause:** Using `:memory:` DB by accident.

**Fix:** Check your `.env`:

```bash
grep FI_DB_PATH .env
# Should NOT be:
# FI_DB_PATH=:memory:
```

### Symptom: `/jobs/{id}` returns 404 even though the job ran

**Cause:** The job was created but persistence failed silently (the
service catches the exception and continues).

**Diagnosis:** Check the structured logs for `eid=<your job id>`.

**Fix:** If the DB write is failing, you'll see the error in logs.
Common causes: disk full, DB locked (see above), schema mismatch.

### Symptom: DB file is huge

**Cause:** Old jobs and results accumulated.

**Fix:**

1. Run cleanup:

   ```bash
   python -c "
   from config.settings import settings
   from storage.database import Database
   db = Database(path=settings.db_path)
   n = db.cleanup_old_jobs(settings.job_retention_days)
   print(f'Deleted {n} old jobs')
   "
   ```

2. Compact the file:

   ```bash
   sqlite3 /var/lib/face-intel/face_intel.db "VACUUM;"
   ```

3. Schedule cleanup as a cron job — see
   [`docs/DEPLOYMENT.md` §5](DEPLOYMENT.md#5-database-setup).

---

## 9. Performance Tuning

### Symptom: Detection jobs take >500 ms

**Diagnosis:** Check `/stats` for per-provider latency:

```bash
curl -s http://localhost:8000/stats | \
  jq '.providers[] | {name, avg_latency_ms, p95_latency_ms}'
```

**Causes & fixes:**

| Cause | Fix |
|---|---|
| `dnn` running on CPU | Add CUDA-enabled OpenCV (`opencv-python` with CUDA build), or use `haar` for fast pre-filtering. |
| `mtcnn` running on CPU | Switch to `dnn` for speed (MTCNN is more accurate but slower). |
| Image is huge (e.g. 4K) | Pre-resize before upload, or lower `pipeline.ImagePreprocessor.max_dim` (currently hardcoded to 1024). |
| Many providers invoked | Use the `providers` whitelist to invoke only what you need. |
| Orchestrator concurrency too low | Raise `FI_ORCHESTRATOR_MAX_CONCURRENCY=16`. |

### Symptom: Full-pipeline jobs take >30 s

**Cause:** Six concurrent sub-jobs (detection, recognition, image
analysis, metadata, forensics, search) all running serially within
`asyncio.gather`.

**Fix:**

1. Disable providers you don't need. If you're not doing reverse
   search, disable all `SCRAPING` and `REVERSE_SEARCH` providers:

   ```env
   FI_ENABLE_BEAUTIFULSOUP_SCRAPER=false
   FI_ENABLE_SELENIUM_SCRAPER=false
   FI_ENABLE_DUCKDUCKGO_SCRAPER=false
   FI_ENABLE_GOOGLE_LENS=false
   ```

2. Increase orchestrator concurrency:

   ```env
   FI_ORCHESTRATOR_MAX_CONCURRENCY=32
   ```

3. Increase job timeout if needed:

   ```env
   FI_JOB_TIMEOUT_SECONDS=600
   ```

### Symptom: High memory usage

**Causes & fixes:**

| Cause | Fix |
|---|---|
| Many face crops in memory | Lower `pipeline.ImagePreprocessor.max_dim`. |
| Cache storing large raw responses | Lower `FI_CACHE_MAX_ENTRIES`. |
| InsightFace / DeepFace loaded models | Disable if not needed. |
| Selenium Chrome processes | Disable Selenium if not needed. |
| Memory leak in custom provider | Profile with `tracemalloc`. |

### Symptom: High CPU usage at idle

**Cause:** Background polling. Currently Face Intel has no
background tasks, so idle CPU should be ~0%. If you see sustained
CPU:

1. Check for runaway Chrome processes (`ps aux | grep chromium`).
2. Check for stuck asyncio tasks (look in the structured logs for
   `eid` values that never reach `success` or `failed`).
3. Profile with `py-spy`:

   ```bash
   py-spy top --pid <face-intel-pid>
   ```

### Symptom: Network-bound providers are slow

**Causes & fixes:**

| Cause | Fix |
|---|---|
| High latency to upstream API | Use a CDN or proxy closer to the upstream. |
| Rate-limited by upstream | Lower `reverse_search_max_results` to reduce call size. |
| Connection pool exhausted | The shared session uses 10 connections per host. For higher concurrency, modify `utils/http.py::make_session()` to bump `pool_maxsize`. |
| DNS lookups slow | Configure a local DNS cache (`systemd-resolved`, `dnsmasq`). |

### Recommended production settings

```env
FI_ORCHESTRATOR_MAX_CONCURRENCY=16
FI_ORCHESTRATOR_TIMEOUT_SECONDS=60
FI_RETRY_MAX_ATTEMPTS=3
FI_CACHE_ENABLED=true
FI_CACHE_TTL_SECONDS=86400
FI_CACHE_MAX_ENTRIES=10000
FI_CIRCUIT_BREAKER_FAILURE_THRESHOLD=5
FI_CIRCUIT_BREAKER_RECOVERY_SECONDS=120
```

---

## 10. API / HTTP Errors

### `429 Too Many Requests`

**Cause:** Per-IP rate limit exceeded (default 30/min).

**Fix:**

1. Raise the limit:

   ```env
   FI_RATE_LIMIT_PER_MINUTE=120
   ```

2. Or exclude specific IPs (would require a code change to
   `RateLimitMiddleware`).

3. For multi-process deployments, the in-memory limiter is per
   process — each worker allows `rate_limit_per_minute`. Replace
   with Redis-backed limiter for accurate cross-process limiting.

### `413 Payload Too Large`

**Cause:** Request body exceeded `FI_MAX_REQUEST_BODY_BYTES`
(default 25 MB).

**Fix:**

1. Compress the image before upload.
2. Or raise the limit:

   ```env
   FI_MAX_REQUEST_BODY_BYTES=52428800  # 50 MB
   ```

3. Also raise `FI_MAX_IMAGE_BYTES` if the decoded image is being
   rejected (default 20 MB):

   ```env
   FI_MAX_IMAGE_BYTES=41943040  # 40 MB
   ```

### `422 Unprocessable Entity`

**Cause:** Pydantic validation failed on the request body.

**Fix:** Check the response body for the validation error details:

```bash
curl -X POST http://localhost:8000/faces/detect \
  -H "Content-Type: application/json" \
  -d '{"bad": "request"}' -i
```

The response will include the exact field that failed:

```json
{
  "detail": [
    {
      "loc": ["body", "image_url"],
      "msg": "field required",
      "type": "value_error.missing"
    }
  ]
}
```

### `500 Internal Server Error`

**Cause:** Unhandled exception. `GlobalExceptionMiddleware` catches
it and returns:

```json
{
  "success": false,
  "error": "<exception message>",
  "error_type": "<ExceptionClassName>",
  "request_id": "abc123def456"
}
```

**Fix:**

1. Note the `request_id`.
2. Find the matching log entries:

   ```bash
   grep "eid=abc123def456" /var/log/face-intel/app.log
   ```

3. The stack trace will be in the log thanks to `logger.exception()`.

### `GET /jobs/{id}` returns 404 even though `POST /jobs` returned a job_id

**Cause:** Job ID mismatch, or the DB write failed silently.

**Fix:**

1. Verify the job ID you're passing (copy-paste from the POST
   response).
2. Check the structured logs for `eid=<job_id>` to see if the job
   actually persisted.

---

## 11. Image Validation Errors

The `InputValidator` (see [`pipeline/validation.py`](../pipeline/validation.py))
returns `ValidationError` for these cases:

### `"No image input provided."`

**Cause:** Neither `image_url` nor `image_base64` was set on the
request.

**Fix:** Pass at least one:

```json
{"image_url": "https://example.com/photo.jpg"}
```

### `"Unsupported URL scheme: ftp"`

**Cause:** Only `http` and `https` schemes are allowed.

**Fix:** Use an HTTPS URL.

### `"URL missing host."`

**Cause:** Malformed URL like `http:///path`.

**Fix:** Use a well-formed URL.

### `"Localhost URLs not permitted."`

**Cause:** URL host is `localhost`, `127.0.0.1`, `0.0.0.0`, or
`::1`. This is a defense against SSRF attacks.

**Fix:** Use a publicly resolvable URL. For testing locally, host
the image with a tool like `ngrok` or use base64 encoding.

### `"Invalid base64: ..."`

**Cause:** Base64 string couldn't be decoded.

**Fix:** Verify the base64 is valid:

```bash
echo "your_base64_string" | base64 -d | file -
# Should print "JPEG image data, ..."
```

If you're including the `data:image/jpeg;base64,` prefix, that's
fine — the validator strips it.

### `"Decoded image exceeds 20971520 bytes"`

**Cause:** Image is larger than `FI_MAX_IMAGE_BYTES` (default 20 MB).

**Fix:** Compress or resize the image, or raise the limit.

### `"Unrecognized image format (magic bytes mismatch)."`

**Cause:** The bytes don't match any recognized image signature
(JPEG, PNG, GIF, WEBP, BMP, TIFF).

**Fix:** Verify the file is actually an image:

```bash
file your_image.jpg
# Should print "JPEG image data, ..."
```

Common cause: the URL returned an HTML error page instead of an
image. Check the URL in a browser first.

---

## 12. Logging Issues

### Symptom: Logs are too verbose

**Fix:**

```env
FI_LOG_LEVEL=WARNING
```

Levels: `TRACE` < `DEBUG` < `INFO` < `WARNING` < `ERROR` <
`CRITICAL`. Most production setups use `INFO`.

### Symptom: Logs are too quiet (can't see provider invocations)

**Fix:**

```env
FI_LOG_LEVEL=DEBUG
```

Note: DEBUG will include every cache hit/miss and orchestrator
decision. Use `TRACE` only for active debugging.

### Symptom: JSON logs have escaped quotes

**Cause:** The JSON format string in `utils/logging.py` uses
single-quote wrapping around double-quoted JSON. This is correct
loguru behavior — the output is valid JSON, just visually escaped
when viewed in some terminals.

**Fix:** Pipe through `jq`:

```bash
tail -f /var/log/face-intel/app.log | jq .
```

### Symptom: `eid` and `pid` fields are always `-`

**Cause:** The log was produced outside an `execution_context()`
block. This is normal for startup/shutdown logs. If you see it for
provider invocation logs, the context isn't being propagated —
check that your provider is invoked through the orchestrator (which
sets the context) rather than directly.

### Symptom: Logs from `loguru` aren't picked up by systemd journal

**Cause:** loguru writes to `sys.stderr` by default. systemd
captures stderr, but only if the unit doesn't redirect it.

**Fix:** Either:

1. Let systemd capture stderr (default behavior — check with
   `journalctl -u face-intel -f`).
2. Or have loguru write to a file directly by extending
   `setup_logging()`:

   ```python
   logger.add("/var/log/face-intel/app.log", format=fmt, level=settings.log_level)
   ```

---

## 13. Recovery Procedures

### Blackout recovery: all providers failing

If every provider circuit is open:

1. Check `/health/providers`:

   ```bash
   curl -s http://localhost:8000/health/providers | \
     jq '.providers[] | select(.circuit_open == true) | .name'
   ```

2. Identify the common cause (network down? disk full? CPU
   pegged?).
3. Fix the underlying issue.
4. Restart the app (resets all circuits).

### Cache corruption

If the cache is returning bad data:

```bash
curl -X DELETE http://localhost:8000/cache
# Returns: {"cleared": <N>}
```

This clears all entries. The next request to each provider will
re-populate the cache.

### Database corruption

If SQLite reports corruption:

1. Stop the app.
2. Backup the corrupt file (for forensics):

   ```bash
   cp /var/lib/face-intel/face_intel.db /tmp/face_intel.corrupt.db
   ```

3. Try to recover:

   ```bash
   sqlite3 /var/lib/face-intel/face_intel.db ".recover" > /tmp/recovered.sql
   sqlite3 /var/lib/face-intel/face_intel.new.db < /tmp/recovered.sql
   mv /var/lib/face-intel/face_intel.new.db /var/lib/face-intel/face_intel.db
   ```

4. If recovery fails, delete the DB and restart (you lose job
   history but the app re-creates an empty schema):

   ```bash
   rm /var/lib/face-intel/face_intel.db
   systemctl restart face-intel
   ```

### Gallery corruption

If `data/gallery/manifest.json` is corrupt:

1. The `ReferenceStore` constructor catches the JSON decode error
   and starts fresh with an empty gallery. You'll see a warning
   log:

   ```
   Gallery manifest corrupted, starting fresh
   ```

2. To restore, re-add known persons via the service:

   ```python
   from api.container import build_container
   from config.settings import settings
   container = build_container(settings)
   # Add each person back...
   container.recognition_service.add_known_person("alice", embedding_bytes)
   ```

3. If you have backups of the `.npy` files, restore them to
   `data/gallery/` and rebuild `manifest.json` manually.

### Provider stuck in `not_configured` after a fix

If you've installed the missing dep or set the API key but the
provider still shows `not_configured`:

1. **Restart the app.** The registry only runs `discover()` at
   startup — it doesn't re-import providers later.
2. Verify the env var is loaded:

   ```bash
   python -c "from config.settings import settings; print(settings.enable_insightface, settings.insightface_model_pack)"
   ```

3. Check `/providers` `errors` for that provider.

---

## Diagnostic Cheat Sheet

| Symptom | First command to run |
|---|---|
| "Provider X not working" | `curl -s http://localhost:8000/providers/X \| jq` |
| "Job failed" | `curl -s http://localhost:8000/jobs/<id>/result \| jq` |
| "All providers failing" | `curl -s http://localhost:8000/health/providers \| jq '.providers[] \| select(.circuit_open)'` |
| "Slow responses" | `curl -s http://localhost:8000/stats \| jq '.providers[] \| {name, p95_latency_ms}'` |
| "Cache not working" | `curl -s http://localhost:8000/cache \| jq` |
| "Memory growing" | `ps -o pid,rss,vsz,cmd -p <face-intel-pid>` |
| "Disk filling" | `du -sh data/*` |
| "Mystery error" | `grep "eid=<request_id>" /var/log/face-intel/app.log` |

---

## See Also

- [`docs/CONFIGURATION.md`](CONFIGURATION.md) — full settings table
  for tuning circuit breaker, cache, retry, timeouts.
- [`docs/PROVIDERS.md`](PROVIDERS.md) — provider `is_available()`
  patterns and error handling.
- [`docs/DEPLOYMENT.md`](DEPLOYMENT.md) — operational runbook,
  backup procedures.
- [`docs/API_REFERENCE.md` §5](API_REFERENCE.md#5-error-response-format)
  — error response envelope.
- [`docs/ARCHITECTURE.md` §4.3](ARCHITECTURE.md) — circuit breaker
  lifecycle diagram.