docs/TROUBLESHOOTING.md

Face Intel — Troubleshooting Guide

This guide covers common errors, their causes, and how to fix them. Organized by symptom — find your error message or behavior below.

First steps for any issue:

1. Check /health/providers — are circuits open? Are providers not_configured?
2. Check /stats — are failures climbing? Are cache hits 0?
3. Check the structured logs — every line carries eid, pid, retry, status for tracing.
4. Check /providers — the errors field shows manifest discovery failures.

---

Table of Contents

1. Startup Failures
2. Provider Not Available / not_configured
3. Circuit Breaker Open
4. Model Download Failures
5. dlib Compilation Issues
6. Selenium / Chrome Issues
7. Cache Issues
8. Database Issues
9. Performance Tuning
10. API / HTTP Errors
11. Image Validation Errors
12. Logging Issues
13. Recovery Procedures

---

1. Startup Failures

Symptom: ModuleNotFoundError: No module named 'cv2'

Cause: OpenCV not installed.

Fix:

pip install opencv-python==4.9.0.80

If you see libGL.so.1: cannot open shared object file on Linux:

# Debian/Ubuntu
sudo apt install -y libgl1 libglib2.0-0

# Or install the headless variant (no GUI deps)
pip uninstall opencv-python
pip install opencv-python-headless==4.9.0.80

Symptom: ModuleNotFoundError: No module named 'pydantic_settings'

Cause: Wrong pydantic version. Face Intel requires pydantic 2.x with the separate pydantic-settings package.

Fix:

pip install pydantic==2.6.1 pydantic-settings==2.2.1

Symptom: ImportError: cannot import name 'X' from 'Y' during build_container()

Cause: Circular import or a layer violation. The refactor enforces strict one-way dependency direction.

Fix:

1. Run python scripts/check_imports.py to detect cycles.
2. Check the layer the failing module is in — it must only import from layers below it (see docs/ARCHITECTURE.md §1).
3. If you recently added a provider, ensure it imports only from models/, utils/, config/, pipeline/, providers/base.py.

Symptom: App boots but /providers returns an empty list

Cause: All providers are disabled in settings, OR none of them match the manifest.

Fix:

1. Check .env:

   grep FI_ENABLE .env
   

2. Check the loaded settings:

   python -c "from config.settings import settings; print([k for k,v in settings.model_dump().items() if k.startswith('enable_') and v])"
   

3. Check /providers for errors:

   curl -s http://localhost:8000/providers | jq '.errors'
   

---

2. Provider Not Available / not_configured

A provider shows up in /providers with status: "not_configured" and available: false.

Cause 1: Optional Python dependency missing

Symptom:

{
  "name": "insightface",
  "status": "not_configured",
  "available": false
}

/providers errors field:

{ "insightface": "missing dependency: No module named 'insightface'" }

Fix:

pip install insightface onnxruntime
# Also set in .env:
# FI_ENABLE_INSIGHTFACE=true

Restart and verify:

curl -s http://localhost:8000/providers/insightface | jq '.available'
# → true

Cause 2: API key not set

Symptom: Provider enabled in config but is_available() returns False because the key is empty.

curl -s http://localhost:8000/providers/serpapi | jq '.'
# → {"name": "serpapi", "status": "not_configured", "available": false, ...}

Fix:

1. Edit .env:

   FI_ENABLE_SERPAPI=true
   FI_SERPAPI_KEY=your_actual_key_here
   

2. Restart.

3. Verify:

   curl -s http://localhost:8000/providers/serpapi | jq '.available'
   # → true
   

See docs/CONFIGURATION.md §17 for the full key-setup workflow for each paid provider.

Cause 3: Provider's is_available() raised an exception

Symptom: Provider shows not_configured but no manifest error is recorded. The registry's info() method catches the exception and returns available=False.

Fix: Add logging to your provider's is_available():

def is_available(self) -> bool:
    try:
        return self._net is not None
    except Exception as e:
        logger.warning(f"is_available() raised: {e}")
        return False

Cause 4: Provider class name doesn't match manifest

Symptom: Manifest entry says class_name="MyProvider" but the file declares class MyProviderV2. The registry will fail to instantiate.

Fix: Match the class name exactly. See docs/PROVIDERS.md §7.

Cause 5: Provider constructor raised

Symptom: Manifest error like "init error: RuntimeError(...)".

Fix: Check the constructor. Provider constructors must not raise for missing optional deps — they should catch and store the error, then return False from is_available():

def __init__(self, settings):
    super().__init__(settings=settings)
    self._init_error = None
    try:
        self._model = load_model()
    except Exception as e:
        self._init_error = str(e)

See the dnn provider (providers/detection/dnn.py) for the canonical pattern.

---

3. Circuit Breaker Open

Symptom

A provider that was previously working is now skipped — providers_invoked no longer includes it, and /health/providers shows circuit_open: true:

{
  "name": "serpapi",
  "healthy": false,
  "consecutive_failures": 5,
  "circuit_open": true,
  "avg_latency_ms": 0.0
}

How the circuit breaker works

See docs/CONFIGURATION.md §11. Briefly:

1. Provider fails → consecutive_failures += 1.
2. When consecutive_failures >= circuit_breaker_failure_threshold (default 5), the circuit opens.
3. After circuit_breaker_recovery_seconds (default 120s), the circuit transitions to half-open: the next call is allowed.
4. Success → circuit closes. Failure → circuit re-opens.

Cause 1: Genuine provider failure

Diagnosis:

curl -s http://localhost:8000/stats | \
  jq '.counters | to_entries | map(select(.key | startswith("failures.")))'

If failures.<provider> is climbing, the provider is genuinely failing. Check its logs:

# Tail logs filtered to the provider
tail -f /var/log/face-intel/app.log | grep "pid=<provider_name>"

Common root causes:

• API key expired / revoked.
• Rate limited by upstream (429).
• Network partition.
• Upstream service is down.

Fix: Resolve the underlying issue. The circuit will close automatically on the next successful call after the recovery window.

Cause 2: Misconfigured retry policy

If retry_max_attempts is too low and the provider has transient failures, you can hit the failure threshold quickly.

Fix: Tune retry:

FI_RETRY_MAX_ATTEMPTS=5
FI_RETRY_INITIAL_BACKOFF_SECONDS=1.0
FI_RETRY_MAX_BACKOFF_SECONDS=30.0
FI_CIRCUIT_BREAKER_FAILURE_THRESHOLD=10
FI_CIRCUIT_BREAKER_RECOVERY_SECONDS=300

Cause 3: Provider timeout too aggressive

If orchestrator_timeout_seconds=10 but the provider takes 15s on average, every call times out → circuit opens.

Fix:

FI_ORCHESTRATOR_TIMEOUT_SECONDS=60

Manual reset

There's no API endpoint to manually close a circuit. Two options:

1. Wait — the circuit auto-transitions to half-open after circuit_breaker_recovery_seconds.
2. Restart — restart the process. The metrics are in-memory only; they reset on restart (the circuit starts closed).

For a programmatic reset (e.g. for testing), use the metrics facade directly:

container.metrics.health.reset()

Verifying recovery

# Trigger a single call to the provider
curl -X POST http://localhost:8000/faces/detect \
  -H "Content-Type: application/json" \
  -d '{"image_url": "https://example.com/face.jpg", "providers": ["<name>"]}'

# Check circuit state
curl -s http://localhost:8000/health/providers | \
  jq '.providers[] | select(.name == "<name>") | .circuit_open'

---

4. Model Download Failures

Affected providers

• dnn — downloads deploy.prototxt (28 KB) and res10_300x300_ssd_iter_140000.caffemodel (10.7 MB) on first use to data/models/.
• insightface — downloads model packs (~100-550 MB) on first use.
• deepface — downloads backend weights on first use.

Symptom: urllib.error.URLError: <urlopen error ...>

Cause: No internet access from the host, or GitHub raw content is blocked.

Fix — pre-download the DNN model files:

mkdir -p data/models
curl -L -o data/models/deploy.prototxt \
  https://raw.githubusercontent.com/opencv/opencv_3rdparty/dnn_samples_face_detector_20170830/deploy.prototxt
curl -L -o data/models/res10_300x300_ssd_iter_140000.caffemodel \
  https://raw.githubusercontent.com/opencv/opencv_3rdparty/dnn_samples_face_detector_20170830/res10_300x300_ssd_iter_140000.caffemodel

# Verify checksums (sha256)
sha256sum data/models/*

Then restart. The provider will see the files already exist and skip the download.

Symptom: Model file is corrupt (cv2.dnn.readNetFromCaffe fails)

Cause: Partial download (e.g. network interrupted).

Fix:

rm data/models/deploy.prototxt data/models/res10_300x300_ssd_iter_140000.caffemodel
# Restart — provider will re-download

Symptom: Downloads work but model fails to load

cv2.error: OpenCV(4.9.0) /io/opencv/modules/dnn/src/caffe/caffe_importer.cpp...

Cause: Corrupt file or OpenCV version mismatch.

Fix:

1. Re-download (see above).
2. Verify OpenCV version: python -c "import cv2; print(cv2.__version__)". Must be 4.9.0.80 per requirements.txt.
3. Try opencv-python-headless if GUI deps cause issues.

Symptom: InsightFace model pack download is very slow

Cause: InsightFace downloads from AWS S3 in the US-east region.

Fix: Pre-download on a build machine and bake into your Docker image:

RUN python -c "from insightface.app import FaceAnalysis; FaceAnalysis(name='buffalo_l').prepare(ctx_id=-1)"

This caches the model in ~/.insightface/models/. Copy that directory into the production image.

Air-gapped deployment

For hosts with zero internet access:

1. On a build machine with internet, run the app once to trigger all model downloads.
2. Tar up data/models/ and ~/.insightface/ (or wherever the optional libs cache models).
3. Copy to the production host at the same paths.
4. Restart — providers will find the models locally.

---

5. dlib Compilation Issues

dlib==19.24.2 does not ship prebuilt wheels for all platforms — on Linux you'll often need to compile from source.

Symptom: error: command 'gcc' failed during pip install dlib

Cause: Missing build tools.

Fix (Debian/Ubuntu):

sudo apt install -y build-essential cmake python3-dev
pip install dlib==19.24.2

Fix (Alpine):

apk add --no-cache build-base cmake linux-headers
pip install dlib==19.24.2

Fix (macOS):

xcode-select --install
brew install cmake
pip install dlib==19.24.2

Symptom: Compilation hangs or runs out of memory

Cause: dlib compiles with all cores by default; on small VMs this OOMs.

Fix: Limit parallelism:

pip install --no-build-isolation dlib==19.24.2 \
  --config-settings cmake.define.BUILD_SHARED_LIBS=OFF \
  -j 2

Or use a machine with ≥4 GB RAM for the build step.

Symptom: ModuleNotFoundError: No module named 'dlib' after install

Cause: Wrong Python interpreter — dlib installed in a different venv than the one running Face Intel.

Fix:

which python    # should be your venv python
python -m pip install dlib==19.24.2
python -c "import dlib; print(dlib.__version__)"

Symptom: face_recognition provider crashes with Illegal instruction

Cause: dlib compiled with AVX instructions but CPU doesn't support them (common on older Xeons / VPS hosts).

Fix: Recompile dlib without AVX:

pip uninstall dlib
git clone https://github.com/davisking/dlib.git
cd dlib
mkdir build && cd build
cmake .. -DUSE_AVX_INSTRUCTIONS=OFF -DCMAKE_BUILD_TYPE=Release
make -j 4
cd ..
python setup.py install

Alternative: skip dlib entirely

If you don't need recognition:

FI_ENABLE_FACE_RECOGNITION=false

Then pip uninstall dlib face-recognition to remove the broken install. The other 23 providers continue to work.

---

6. Selenium / Chrome Issues

Affects: selenium (scraper), google_lens (reverse search).

Symptom: selenium.common.exceptions.WebDriverException: Message: unknown error: cannot find Chrome binary

Cause: Chrome/Chromium not installed.

Fix (Debian/Ubuntu):

# Stable Chrome
wget -q https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
sudo apt install -y ./google-chrome-stable_current_amd64.deb

# Or Chromium
sudo apt install -y chromium-browser

Fix (macOS):

brew install --cask google-chrome

Fix (Docker): Add to Dockerfile:

RUN apt-get update && apt-get install -y chromium
ENV CHROME_BIN=/usr/bin/chromium

Symptom: selenium.common.exceptions.SessionNotCreatedException: Message: session not created: Chrome version must be between ...

Cause: webdriver-manager downloaded a ChromeDriver version that doesn't match your installed Chrome.

Fix:

google-chrome --version  # note the major version
# Then ensure chromedriver matches
pip install --upgrade webdriver-manager

Or use Chrome for Testing with pinned versions:

# Install Chrome for Testing 121
# https://googlechromelabs.github.io/chrome-for-testing/

Symptom: Selenium tests hang indefinitely

Cause: selenium_implicit_wait is too high, or Chrome is waiting for a never-resolving resource.

Fix:

1. Lower implicit wait:

   FI_SELENIUM_IMPLICIT_WAIT=5
   

2. Reduce scroll iterations:

   FI_SELENIUM_SCROLL_ITERATIONS=2
   

3. Set a global scrape timeout:

   FI_SCRAPE_TIMEOUT=15
   

4. For debugging, run non-headless to see what's happening:

   FI_SELENIUM_HEADLESS=false
   

Symptom: Headless Chrome returns empty pages

Cause: Some sites detect headless Chrome and serve different content. Or JavaScript hasn't finished rendering.

Fix:

1. Increase scroll iterations:

   FI_SELENIUM_SCROLL_ITERATIONS=10
   

2. Use a more realistic User-Agent:

   FI_USER_AGENT="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36"
   

3. Fall back to the beautifulsoup scraper (static HTML) for sites that don't require JS:

   FI_ENABLE_SELENIUM_SCRAPER=false
   FI_ENABLE_BEAUTIFULSOUP_SCRAPER=true
   

Symptom: Chrome crashes with --no-sandbox warning in containers

Cause: Running Chrome as root in a container requires --no-sandbox. Selenium providers should set this automatically; if not, set in your Chrome options.

Fix: Make sure your selenium provider uses:

from selenium.webdriver.chrome.options import Options
opts = Options()
opts.add_argument("--no-sandbox")
opts.add_argument("--disable-dev-shm-usage")  # critical in containers
opts.add_argument("--headless=new")

The --disable-dev-shm-usage flag tells Chrome to use /tmp instead of /dev/shm (which is too small in default containers).

---

7. Cache Issues

Symptom: Cache hit ratio is always 0

Cause: Cache is disabled, or every request uses a different image (cache key is provider:image_hash).

Diagnosis:

curl -s http://localhost:8000/cache | jq

{
  "entries": 0,
  "max_entries": 1000,
  "ttl_seconds": 3600,
  "hits": 0,
  "misses": 24,
  "hit_ratio": 0.0,
  "evictions": 0
}

Fix:

1. Ensure cache is enabled:

   FI_CACHE_ENABLED=true
   

2. Verify you're hitting the same image. The cache key is f"{provider_name}:{image_hash}" where image_hash is the SHA-256 of the preprocessed image. Slight image differences (URL vs base64, re-encoded JPEGs) produce different hashes.

3. If you're load-testing with random images, expect 0% hit ratio.

Symptom: Cache evictions are high

Cause: cache_max_entries is too small for the working set.

Diagnosis:

curl -s http://localhost:8000/cache | jq '.evictions'

Fix:

FI_CACHE_MAX_ENTRIES=10000

Symptom: Same image returns different results across requests

Cause: Cache was cleared, or the TTL expired.

Diagnosis:

1. Check GET /cache for cleared events — but actually the cache doesn't expose clear count via API. Check the structured logs for cache clear events.
2. Check cache_ttl_seconds — if it's very short, entries expire between requests.

Fix:

FI_CACHE_TTL_SECONDS=86400  # 24 hours

Symptom: Stale data — provider updated but cache still serves old result

Cause: Cache TTL hasn't expired.

Fix:

1. Clear the cache:

   curl -X DELETE http://localhost:8000/cache
   

2. Or invalidate a specific key (not exposed via API — would need a new endpoint, see docs/API_REFERENCE.md §9).

3. For permanent fix, lower the TTL or implement a webhook that clears the cache when the upstream model updates.

Symptom: Memory usage grows unboundedly

Cause: Each cache entry stores a full ProviderResult including the raw field, which can be large for image-analysis providers.

Fix:

1. Lower cache_max_entries:

   FI_CACHE_MAX_ENTRIES=100
   

2. For multi-worker deployments, replace the in-memory cache with Redis (see docs/DEPLOYMENT.md §9).

3. If you can afford recomputation, disable the cache entirely:

   FI_CACHE_ENABLED=false
   

---

8. Database Issues

Symptom: sqlite3.OperationalError: database is locked

Cause: SQLite writer contention — multiple workers writing concurrently.

Fix:

1. Run a single uvicorn worker (no --workers N).
2. Or shard: different DB paths per worker (each worker serves a subset of requests).
3. Or migrate to Postgres (would require a new Database implementation — not currently supported).

Symptom: sqlite3.OperationalError: unable to open database file

Cause: The FI_DB_PATH directory doesn't exist or isn't writable.

Fix:

# Check the path
python -c "from config.settings import settings; print(settings.db_path)"

# Ensure the directory exists
mkdir -p $(dirname $(python -c "from config.settings import settings; print(settings.db_path)"))

# Check permissions
ls -la $(dirname $(python -c "from config.settings import settings; print(settings.db_path)"))

Symptom: Jobs disappear from /jobs after a restart

Cause: Using :memory: DB by accident.

Fix: Check your .env:

grep FI_DB_PATH .env
# Should NOT be:
# FI_DB_PATH=:memory:

Symptom: /jobs/{id} returns 404 even though the job ran

Cause: The job was created but persistence failed silently (the service catches the exception and continues).

Diagnosis: Check the structured logs for eid=<your job id>.

Fix: If the DB write is failing, you'll see the error in logs. Common causes: disk full, DB locked (see above), schema mismatch.

Symptom: DB file is huge

Cause: Old jobs and results accumulated.

Fix:

1. Run cleanup:

   python -c "
   from config.settings import settings
   from storage.database import Database
   db = Database(path=settings.db_path)
   n = db.cleanup_old_jobs(settings.job_retention_days)
   print(f'Deleted {n} old jobs')
   "
   

2. Compact the file:

   sqlite3 /var/lib/face-intel/face_intel.db "VACUUM;"
   

3. Schedule cleanup as a cron job — see docs/DEPLOYMENT.md §5.

---

9. Performance Tuning

Symptom: Detection jobs take >500 ms

Diagnosis: Check /stats for per-provider latency:

curl -s http://localhost:8000/stats | \
  jq '.providers[] | {name, avg_latency_ms, p95_latency_ms}'

Causes & fixes:

Cause	Fix
dnn running on CPU	Add CUDA-enabled OpenCV (opencv-python with CUDA build), or use haar for fast pre-filtering.
mtcnn running on CPU	Switch to dnn for speed (MTCNN is more accurate but slower).
Image is huge (e.g. 4K)	Pre-resize before upload, or lower pipeline.ImagePreprocessor.max_dim (currently hardcoded to 1024).
Many providers invoked	Use the providers whitelist to invoke only what you need.
Orchestrator concurrency too low	Raise FI_ORCHESTRATOR_MAX_CONCURRENCY=16.

Symptom: Full-pipeline jobs take >30 s

Cause: Six concurrent sub-jobs (detection, recognition, image analysis, metadata, forensics, search) all running serially within asyncio.gather.

Fix:

1. Disable providers you don't need. If you're not doing reverse search, disable all SCRAPING and REVERSE_SEARCH providers:

   FI_ENABLE_BEAUTIFULSOUP_SCRAPER=false
   FI_ENABLE_SELENIUM_SCRAPER=false
   FI_ENABLE_DUCKDUCKGO_SCRAPER=false
   FI_ENABLE_GOOGLE_LENS=false
   

2. Increase orchestrator concurrency:

   FI_ORCHESTRATOR_MAX_CONCURRENCY=32
   

3. Increase job timeout if needed:

   FI_JOB_TIMEOUT_SECONDS=600
   

Symptom: High memory usage

Causes & fixes:

Cause	Fix
Many face crops in memory	Lower pipeline.ImagePreprocessor.max_dim.
Cache storing large raw responses	Lower FI_CACHE_MAX_ENTRIES.
InsightFace / DeepFace loaded models	Disable if not needed.
Selenium Chrome processes	Disable Selenium if not needed.
Memory leak in custom provider	Profile with tracemalloc.

Symptom: High CPU usage at idle

Cause: Background polling. Currently Face Intel has no background tasks, so idle CPU should be ~0%. If you see sustained CPU:

1. Check for runaway Chrome processes (ps aux | grep chromium).

2. Check for stuck asyncio tasks (look in the structured logs for eid values that never reach success or failed).

3. Profile with py-spy:

   py-spy top --pid <face-intel-pid>
   

Symptom: Network-bound providers are slow

Causes & fixes:

Cause	Fix
High latency to upstream API	Use a CDN or proxy closer to the upstream.
Rate-limited by upstream	Lower reverse_search_max_results to reduce call size.
Connection pool exhausted	The shared session uses 10 connections per host. For higher concurrency, modify utils/http.py::make_session() to bump pool_maxsize.
DNS lookups slow	Configure a local DNS cache (systemd-resolved, dnsmasq).

Recommended production settings

FI_ORCHESTRATOR_MAX_CONCURRENCY=16
FI_ORCHESTRATOR_TIMEOUT_SECONDS=60
FI_RETRY_MAX_ATTEMPTS=3
FI_CACHE_ENABLED=true
FI_CACHE_TTL_SECONDS=86400
FI_CACHE_MAX_ENTRIES=10000
FI_CIRCUIT_BREAKER_FAILURE_THRESHOLD=5
FI_CIRCUIT_BREAKER_RECOVERY_SECONDS=120

---

10. API / HTTP Errors

429 Too Many Requests

Cause: Per-IP rate limit exceeded (default 30/min).

Fix:

1. Raise the limit:

   FI_RATE_LIMIT_PER_MINUTE=120
   

2. Or exclude specific IPs (would require a code change to RateLimitMiddleware).

3. For multi-process deployments, the in-memory limiter is per process — each worker allows rate_limit_per_minute. Replace with Redis-backed limiter for accurate cross-process limiting.

413 Payload Too Large

Cause: Request body exceeded FI_MAX_REQUEST_BODY_BYTES (default 25 MB).

Fix:

1. Compress the image before upload.

2. Or raise the limit:

   FI_MAX_REQUEST_BODY_BYTES=52428800  # 50 MB
   

3. Also raise FI_MAX_IMAGE_BYTES if the decoded image is being rejected (default 20 MB):

   FI_MAX_IMAGE_BYTES=41943040  # 40 MB
   

422 Unprocessable Entity

Cause: Pydantic validation failed on the request body.

Fix: Check the response body for the validation error details:

curl -X POST http://localhost:8000/faces/detect \
  -H "Content-Type: application/json" \
  -d '{"bad": "request"}' -i

The response will include the exact field that failed:

{
  "detail": [
    {
      "loc": ["body", "image_url"],
      "msg": "field required",
      "type": "value_error.missing"
    }
  ]
}

500 Internal Server Error

Cause: Unhandled exception. GlobalExceptionMiddleware catches it and returns:

{
  "success": false,
  "error": "<exception message>",
  "error_type": "<ExceptionClassName>",
  "request_id": "abc123def456"
}

Fix:

1. Note the request_id.

2. Find the matching log entries:

   grep "eid=abc123def456" /var/log/face-intel/app.log
   

3. The stack trace will be in the log thanks to logger.exception().

GET /jobs/{id} returns 404 even though POST /jobs returned a job_id

Cause: Job ID mismatch, or the DB write failed silently.

Fix:

1. Verify the job ID you're passing (copy-paste from the POST response).
2. Check the structured logs for eid=<job_id> to see if the job actually persisted.

---

11. Image Validation Errors

The InputValidator (see pipeline/validation.py) returns ValidationError for these cases:

"No image input provided."

Cause: Neither image_url nor image_base64 was set on the request.

Fix: Pass at least one:

{"image_url": "https://example.com/photo.jpg"}

"Unsupported URL scheme: ftp"

Cause: Only http and https schemes are allowed.

Fix: Use an HTTPS URL.

"URL missing host."

Cause: Malformed URL like http:///path.

Fix: Use a well-formed URL.

"Localhost URLs not permitted."

Cause: URL host is localhost, 127.0.0.1, 0.0.0.0, or ::1. This is a defense against SSRF attacks.

Fix: Use a publicly resolvable URL. For testing locally, host the image with a tool like ngrok or use base64 encoding.

"Invalid base64: ..."

Cause: Base64 string couldn't be decoded.

Fix: Verify the base64 is valid:

echo "your_base64_string" | base64 -d | file -
# Should print "JPEG image data, ..."

If you're including the data:image/jpeg;base64, prefix, that's fine — the validator strips it.

"Decoded image exceeds 20971520 bytes"

Cause: Image is larger than FI_MAX_IMAGE_BYTES (default 20 MB).

Fix: Compress or resize the image, or raise the limit.

"Unrecognized image format (magic bytes mismatch)."

Cause: The bytes don't match any recognized image signature (JPEG, PNG, GIF, WEBP, BMP, TIFF).

Fix: Verify the file is actually an image:

file your_image.jpg
# Should print "JPEG image data, ..."

Common cause: the URL returned an HTML error page instead of an image. Check the URL in a browser first.

---

12. Logging Issues

Symptom: Logs are too verbose

Fix:

FI_LOG_LEVEL=WARNING

Levels: TRACE < DEBUG < INFO < WARNING < ERROR < CRITICAL. Most production setups use INFO.

Symptom: Logs are too quiet (can't see provider invocations)

Fix:

FI_LOG_LEVEL=DEBUG

Note: DEBUG will include every cache hit/miss and orchestrator decision. Use TRACE only for active debugging.

Symptom: JSON logs have escaped quotes

Cause: The JSON format string in utils/logging.py uses single-quote wrapping around double-quoted JSON. This is correct loguru behavior — the output is valid JSON, just visually escaped when viewed in some terminals.

Fix: Pipe through jq:

tail -f /var/log/face-intel/app.log | jq .

Symptom: eid and pid fields are always -

Cause: The log was produced outside an execution_context() block. This is normal for startup/shutdown logs. If you see it for provider invocation logs, the context isn't being propagated — check that your provider is invoked through the orchestrator (which sets the context) rather than directly.

Symptom: Logs from loguru aren't picked up by systemd journal

Cause: loguru writes to sys.stderr by default. systemd captures stderr, but only if the unit doesn't redirect it.

Fix: Either:

1. Let systemd capture stderr (default behavior — check with journalctl -u face-intel -f).

2. Or have loguru write to a file directly by extending setup_logging():

   logger.add("/var/log/face-intel/app.log", format=fmt, level=settings.log_level)
   

---

13. Recovery Procedures

Blackout recovery: all providers failing

If every provider circuit is open:

1. Check /health/providers:

   curl -s http://localhost:8000/health/providers | \
     jq '.providers[] | select(.circuit_open == true) | .name'
   

2. Identify the common cause (network down? disk full? CPU pegged?).

3. Fix the underlying issue.

4. Restart the app (resets all circuits).

Cache corruption

If the cache is returning bad data:

curl -X DELETE http://localhost:8000/cache
# Returns: {"cleared": <N>}

This clears all entries. The next request to each provider will re-populate the cache.

Database corruption

If SQLite reports corruption:

1. Stop the app.

2. Backup the corrupt file (for forensics):

   cp /var/lib/face-intel/face_intel.db /tmp/face_intel.corrupt.db
   

3. Try to recover:

   sqlite3 /var/lib/face-intel/face_intel.db ".recover" > /tmp/recovered.sql
   sqlite3 /var/lib/face-intel/face_intel.new.db < /tmp/recovered.sql
   mv /var/lib/face-intel/face_intel.new.db /var/lib/face-intel/face_intel.db
   

4. If recovery fails, delete the DB and restart (you lose job history but the app re-creates an empty schema):

   rm /var/lib/face-intel/face_intel.db
   systemctl restart face-intel
   

Gallery corruption

If data/gallery/manifest.json is corrupt:

1. The ReferenceStore constructor catches the JSON decode error and starts fresh with an empty gallery. You'll see a warning log:

   Gallery manifest corrupted, starting fresh
   

2. To restore, re-add known persons via the service:

   from api.container import build_container
   from config.settings import settings
   container = build_container(settings)
   # Add each person back...
   container.recognition_service.add_known_person("alice", embedding_bytes)
   

3. If you have backups of the .npy files, restore them to data/gallery/ and rebuild manifest.json manually.

Provider stuck in not_configured after a fix

If you've installed the missing dep or set the API key but the provider still shows not_configured:

1. Restart the app. The registry only runs discover() at startup — it doesn't re-import providers later.

2. Verify the env var is loaded:

   python -c "from config.settings import settings; print(settings.enable_insightface, settings.insightface_model_pack)"
   

3. Check /providers errors for that provider.

---

Diagnostic Cheat Sheet

Symptom	First command to run
"Provider X not working"	curl -s http://localhost:8000/providers/X | jq
"Job failed"	curl -s http://localhost:8000/jobs/<id>/result | jq
"All providers failing"	curl -s http://localhost:8000/health/providers | jq '.providers[] | select(.circuit_open)'
"Slow responses"	curl -s http://localhost:8000/stats | jq '.providers[] | {name, p95_latency_ms}'
"Cache not working"	curl -s http://localhost:8000/cache | jq
"Memory growing"	ps -o pid,rss,vsz,cmd -p <face-intel-pid>
"Disk filling"	du -sh data/*
"Mystery error"	grep "eid=<request_id>" /var/log/face-intel/app.log

---

See Also

• docs/CONFIGURATION.md — full settings table for tuning circuit breaker, cache, retry, timeouts.
• docs/PROVIDERS.md — provider is_available() patterns and error handling.
• docs/DEPLOYMENT.md — operational runbook, backup procedures.
• docs/API_REFERENCE.md §5 — error response envelope.
• docs/ARCHITECTURE.md §4.3 — circuit breaker lifecycle diagram.