docs/reference/config-deploy.md

Config & deploy

Every runtime knob the Inspector reads, plus how the image is built and shipped to the HF Spaces. Deployment tunables that aren't env-driven live in inspector/config.py (one constant per line, each commented — read it directly when changing a threshold). This doc covers the env surface + build/deploy/provisioning.

Runtime profiles

Same code, profile selected by env presence:

	Local dev (python inspector/app.py)	Deployed (HF Space, gunicorn)
Identity	INSPECTOR_DEV_MODE=1 auto-set when OAuth unconfigured → synthetic per-role user, OAuth bypassed	HF OAuth → signed inspector_session cookie
DB	inspector.db at INSPECTOR_DB_PATH (defaults to a tmp path)	same; canonical bucket artefact is <mount>/db/inspector.db
Bucket	FUSE auto-mount via hf-mount (unless opted out) or hffs fall-back	NFS mount provided by the Space runtime at INSPECTOR_BUCKET_MOUNT
Port	:5000 (Flask dev server)	:7860 (gunicorn, EXPOSE 7860)
Workers	Flask dev server	gunicorn-gthread -w 1 --threads 16 (single-worker invariant)

INSPECTOR_BEHIND_PROXY=1 is the deployed-mode signal: it enables ProxyFix and skips the local auto-mount (the Space already provides the mount). Set in the image.

Spaces

Env	Space repo	Bucket (INSPECTOR_BUCKET_REPO)
dev	hetchyy/quranic-inspector-dev	hetchyy/quranic-inspector-bucket-dev
prod	hetchyy/quranic-universal-audio	hetchyy/quranic-inspector-bucket

The image bakes the dev bucket as default (INSPECTOR_BUCKET_REPO=hetchyy/quranic-inspector-bucket-dev); the prod Space overrides it via a Space variable. Both ship the Docker SDK and bind 7860.

Env vars — what the code actually reads

Storage / bucket / DB

Var	Default	Read by	Purpose
INSPECTOR_BACKEND	bucket	services/storage/hf_bucket.py, auto_mount.py	bucket (HF storage) or filesystem (tests + the Tier-0 fixtures/offline path seeded by scripts/devenv/seed_fixtures.py).
INSPECTOR_BUCKET_REPO	…-bucket-dev	auto_mount.py, hf_bucket.py	HF bucket repo id. Resolved by resolve_bucket_repo(): defaults to the dev bucket for every non-deployed process and refuses prod unless INSPECTOR_ALLOW_PROD_BUCKET=1.
INSPECTOR_ALLOW_PROD_BUCKET	unset	hf_bucket.py, auto_mount.py	1 lets a local/non-deployed process use the prod bucket (otherwise ProdBucketRefused). The deployed prod Space is exempt (it runs behind the proxy).
INSPECTOR_DB_SYNC	unset (1)	app.py, services/db/sync.py	0 disarms per-commit bucket DB write-back at boot. Boot still pulls the DB; DB commits stay local and never upload. Stops only the DB sync — pair with INSPECTOR_READ_ONLY to also stop direct content writes.
INSPECTOR_READ_ONLY	unset	services/storage/hf_bucket.py	1 wraps the storage backend in ReadOnlyBackend, which refuses every mutating op (write_*/append_jsonl/copy/move/delete) with StorageReadOnly. Covers the writes INSPECTOR_DB_SYNC misses — segment save (detailed.json/edit_history.jsonl), intake manifests, job records — since they all go straight through the backend, not the DB sync. launch.py --mode prod sets both, so a local process on the prod bucket cannot mutate it by any path.
INSPECTOR_AUDIO_FROM_BUCKET	1	services/audio/audio_fetch.py	0 makes read_prefetched_audio_* short-circuit so audio_source.resolve skips the bucket (mount + hffs full-MP3 read) and streams from the CDN — faster locally. launch.py sets 0 for dev/prod (override with --bucket-audio), 1 for fixtures (offline → local audio, no CDN). Deployed leaves it on (mount).
INSPECTOR_PEAKS_FROM_BUCKET	1	services/audio/audio_fetch.py	0 makes read_prefetched_peaks short-circuit to None so every chapter misses and the FE falls through to per-segment ffmpeg compute. launch.py --ffmpeg-peaks sets 0.
INSPECTOR_BUCKET_MOUNT	unset local; /data/inspector-bucket (image)	app.py, health.py, hf_bucket.py, auto_mount.py	Mount path. Presence ⇒ "deployed mode" for /healthz. When unset locally the backend falls back to per-call hffs reads (~50–500× slower).
INSPECTOR_FILESYSTEM_ROOT	unset	hf_bucket.py	Required when INSPECTOR_BACKEND=filesystem. seed_fixtures.py sets it to inspector/.fixtures.
INSPECTOR_AUTO_MOUNT	1	auto_mount.py	0 opts out of local FUSE auto-mount.
INSPECTOR_DB_PATH	<tmpdir>/inspector.db	services/db/connection.py	On-disk SQLite path.
INSPECTOR_HF_TOKEN / HF_TOKEN	unset	hf_bucket.py, auto_mount.py	Token for bucket read/write (needs write scope). INSPECTOR_HF_TOKEN wins; HF_TOKEN is the fall-back + what hf-mount reads.

Data (bundled reference)

Var	Default	Purpose
INSPECTOR_DATA_DIR	/app/data (image) / <repo>/data (local)	Root for bundled static data (surah_info + linguistic JSONs).
INSPECTOR_QUA_DATA_PATH	mirrors INSPECTOR_DATA_DIR	Base for qpc_hafs.json, digital_khatt_v2_script.json, phoneme_sub_costs.json.

Server / proxy / process

Var	Default	Read by	Purpose
INSPECTOR_HOST	0.0.0.0	config.py	Bind host.
INSPECTOR_BEHIND_PROXY	unset; 1 (image)	app.py, auth.py, auto_mount.py	Deployed signal — enables ProxyFix, skips auto-mount.
INSPECTOR_PUBLIC_URL	unset	routes/auth/auth.py	Override the public https base used to build the OAuth callback.
INSPECTOR_COMMIT_SHA	unknown	health.py	Reported by /healthz so deploys are identifiable.
GUNICORN_WORKERS / WEB_CONCURRENCY / GUNICORN_CMD_ARGS	1	app.py::_assert_single_worker	Any multi-worker signal aborts boot. The single-worker invariant is load-bearing (single writer, in-process caches, per-slug locks).
FLASK_ENV	unset	config.py / app.py	development → Flask debug + reloader (local only).

Background loops

Var	Default	Purpose
INSPECTOR_AUTO_DETECT	off; 1 (image)	Background loop that scans the bucket for newly-aligned reciters and folds them into state.
INSPECTOR_AUTO_DETECT_INTERVAL_S	(see app.py)	Tick interval for the auto-detect loop.
INSPECTOR_AUTOMATIONS	off; 1 (image)	Release-automation reconciler daemon (per-automation enable lives in the owner's config blob; the daemon no-ops while all are off). See automation.md.
INSPECTOR_AUTOMATIONS_INTERVAL_S	60	Tick cadence for the automation reconciler.
INSPECTOR_PUBLIC_BASE_URL	empty	Public https root the daemon threads into job completion webhooks (no request.url_root in a thread). Required for automated GH cuts — the cut job is webhook-only. Set as a Space variable per environment.

Auth / identity / secrets

Var	Source	Purpose
OAUTH_CLIENT_ID / OAUTH_CLIENT_SECRET / OPENID_PROVIDER_URL	HF-injected (hf_oauth: true)	OAuth client + OIDC discovery. Auto-managed by HF; never persisted.
OAUTH_SCOPES	openid profile	OAuth scopes requested.
INSPECTOR_SESSION_SECRET	Space secret (≥32 hex)	Signs the inspector_session cookie. services/auth/secrets_guard.py::get_session_secret rejects unset/short values. Rotating logs everyone out.
INSPECTOR_DEV_MODE	1 auto-set locally when OAuth unconfigured	Bypasses OAuth; mints a synthetic per-role user.
INSPECTOR_DEV_<ROLE>_HF_ID / INSPECTOR_DEV_<ROLE>_LOGIN	unset	Override the synthetic dev identity per role (OWNER/MAINTAINER/CONTRIBUTOR); default dev-<role>. Set to a real HF id+login when local dev points at the prod bucket so audit attribution is correct.
INSPECTOR_GITHUB_DISPATCH_TOKEN	Space secret	Fine-grained PAT (actions:write on the project repo). Fires repository_dispatch after publish. secrets_guard.get_dispatch_token raises if unset or still a PLACEHOLDER_ value.
INSPECTOR_WEBHOOK_SECRET	Space secret (optional)	Shared secret for the timestamps-job completion webhook (POST /api/webhooks/ts-job-complete). When set, launch() threads it (+ the job's callback URL) to the HF job so it auto-publishes the reciter on success; the route validates it with hmac.compare_digest. Unset → webhook disabled (503) and release falls back to the drawer-poll path. Read by routes/webhooks/ts_jobs.py + services/admin/timestamps_jobs.py::launch.
INSPECTOR_WEBHOOK_URL	set on the job, not the Space	The Inspector's public callback URL. The launch route derives it from request.url_root (deployed ProxyFix → https://…hf.space/) and passes it to the job as env — operators don't set it on the Space.
BREVO_API_KEY	Space secret (optional)	Brevo transactional REST API key for the email-notifications sender (services/email/). HF Spaces block outbound SMTP on every port, so delivery is over HTTPS (POST /v3/smtp/email), not smtplib. Unset → no email is sent; the sender logs the rendered message instead (dev exercises the flow without secrets).
INSPECTOR_EMAIL_FROM_ADDRESS / GMAIL	Space secret	From address for outgoing email — MUST be a Brevo-verified sender. INSPECTOR_EMAIL_FROM_ADDRESS wins; falls back to GMAIL (the already-verified single-sender account).
INSPECTOR_EMAIL_SITE_URL / INSPECTOR_EMAIL_GH_RELEASES_URL / INSPECTOR_EMAIL_FROM_NAME	defaults in config.py	Email link targets + From display name. Functional links (manage/unsubscribe) use INSPECTOR_PUBLIC_BASE_URL (localhost in dev).

The error strings in secrets_guard.py still reference a scripts.inspector_v2_seed.setup_space provisioning module that no longer exists — secrets are now set by hand in the Space secrets panel. Treat that hint as stale.

Quran Foundation (see architecture.md → QF integration)

Var	Read by	Purpose
QF_PREPROD_CLIENT_ID / QF_PREPROD_CLIENT_SECRET	services/quran_foundation/config.py	User-API (bookmarks/collections) pre-prod OAuth client. client_secret_basic only.
QF_OAUTH_REDIRECT_URI	same	Registered callback; default http://localhost:5001/api/qf/callback. Must match the QF client exactly.
QF_CONTENT_CLIENT_ID / QF_CONTENT_CLIENT_SECRET	same	Content-API server-to-server (client_credentials) client. Production endpoints.

Local QF creds live in inspector/.env (gitignored), loaded by app.py.

Misc tunables (env-overridable constants)

Var	Default	Purpose
INSPECTOR_MFA_SPACE_URL	https://hetchyy-quran-phoneme-mfa-dev.hf.space	MFA Space the cross-verse "Auto Split" calls.
MISSED_BASMALA_FLAG_MIN_DELETED	10	Min pipeline-stripped basmalas before flagging non-stripped chapters as "missed Basmala".

Image build — inspector/Dockerfile

Three-stage build, context is the repo root (so qua_shared/ ships alongside inspector/):

1. frontend-build (node:20-alpine) — npm ci + npm run build → dist/.
2. ffmpeg-build (alpine:3.23) — compiles a minimal static ffmpeg/ffprobe (mp3 decode + pcm_s16le/wav for peaks + libmp3lame/mp3 muxer for the segment-clip route). libmp3lame 3.100 is built from source --enable-static because Alpine ships no static lame archive — without it clip.py returns 200/0-bytes and playback hangs. http/https protocols enabled so ffmpeg can decode remote chapters via HTTP Range.
3. runtime (python:3.11-alpine) — installs inspector/requirements.txt, purges pip/setuptools/wheel, selective COPY (app.py/config.py/constants.py, adapters/ domain/ routes/ services/ utils/, scripts/__init__.py + qua_shared/ + qua_jobs/, .github/config/, LICENSE, the 4 bundled data JSONs, and frontend/dist from stage 1). qua_jobs/ + .github/config/ + LICENSE are read by the HF-Job entrypoints (config_loader, cut_release asset upload). Runs as non-root uid/gid 1000. EXPOSE 7860. Adding a COPY here needs no staging edit — the deploy stages the whole tracked tree (see Deploy).

CMD: gunicorn -k gthread -w 1 --threads 16 --max-requests 5000 --max-requests-jitter 500 --timeout 60 --graceful-timeout 30 --bind 0.0.0.0:7860 --access-logfile - --error-logfile - --chdir /app/inspector app:app. -w 1 is load-bearing (asserted at import). --threads 16 sizes the I/O-bound pool. --max-requests recycles the worker to bound slow leaks. The access log (one line per request → stdout) is de-noised by app._AccessLogFilter, attached to the gunicorn.access + werkzeug loggers: it drops static-asset (/assets/*), /healthz, and 304 lines while always keeping 4xx/5xx and real API/page hits.

docker-compose.yml (repo inspector/) is the local reviewer entry (docker compose up → :5000, bind-mounts data/).

Deploy — inspector-deploy.yml

Push to dev or main (paths under inspector/**, qua_shared/**, scripts/deploy/upload_inspector.py, the bundled data JSONs, .dockerignore) → workflow runs inspector-checks.yml (reused), then deploy runs scripts/deploy/upload_inspector.py <env>. Branch→env: dev→dev Space, main→prod Space; workflow_dispatch can force dev/prod. The Space then rebuilds the Docker image from the uploaded files.

Fast-path dev dispatch. A manual workflow_dispatch that resolves to the dev Space (env=dev, or env=auto off any non-main branch) skips the checks job entirely for a quick deploy — the dev Space is a verification target, not prod. Every push and any prod-targeting dispatch still runs the full check suite. The deploy gate accepts a skipped checks (success or skipped), so it never confuses "intentionally skipped" with "failed".

Staging = the whole git-tracked tree. upload_inspector._stage() copies every git-tracked file verbatim (then overlays the root Dockerfile from inspector/Dockerfile + a frontmatter README.md), and upload_folder(delete_patterns="*") mirrors it to the Space. .dockerignore is the single source of truth that prunes the build context — there is no hand-curated allowlist to drift from the Dockerfile's COPY set (that drift took prod down once: a COPY LICENSE with no matching staging entry → BUILD_ERROR "/LICENSE: not found"). Consequence: the Space build context equals the context: . that docker-publish.yml builds on every push, so its green image build is a faithful "the Space will compile" signal. dist/ is built inside the image (stage 1), so the deploy does not pre-build the frontend.

Caveat — async build, no CI feedback. upload_inspector.py only uploads + restart_space(factory_reboot=True) and returns; the HF docker rebuild is asynchronous. A green inspector-deploy run does not mean the Space built. After a deploy, poll the runtime stage (…/api/spaces/<id>/runtime → RUNNING_BUILDING → RUNNING_APP_STARTING → RUNNING; build logs at …/logs/build).

Opt-in boot check (prod-only, default off). scripts/deploy/smoke_boot.py builds the image and boots it offline (filesystem backend + public seed_fixtures, no secrets, no bucket), polling /healthz for 200 + state_loaded + db.open (offline it's 200/status: degraded since no bucket is mounted — so it does not assert status == ok). Two entry points: scripts/deploy/upload_inspector.py --verify-boot (gates a local/manual deploy on a passing boot), and a boot-smoke CI job that runs only on a manual workflow_dispatch with verify_boot=true targeting prod. The deploy job's if is always() && (needs.checks.result=='success' || =='skipped') && needs.boot-smoke.result != 'failure' && != 'cancelled' so a skipped boot-smoke (every normal push) or a skipped checks (fast-path dev dispatch) still deploys, while a failing one blocks the deploy. Automatic deploys stay lean (no image build on the deploy path; docker-publish.yml builds the identical context in parallel as a non-blocking red signal).

The canonical dev/prod Spaces have no auto-provisioning script — their secrets/variables (INSPECTOR_SESSION_SECRET, INSPECTOR_GITHUB_DISPATCH_TOKEN, QF creds, INSPECTOR_BUCKET_REPO override, bucket attachment) are configured by hand in the Space settings panels. OAuth vars are auto-injected by HF via hf_oauth: true in the Space README frontmatter.

Contributor (personal) Spaces are fully scripted — no manual Space-settings clicks. scripts/devenv/bootstrap_dev_env.py <name> provisions a private bucket + Space under the contributor's own account, attaches the bucket as a Space volume at /data/inspector-bucket (HfApi.set_space_volumes), auto-generates INSPECTOR_SESSION_SECRET, sets HF_TOKEN + the INSPECTOR_BUCKET_REPO variable, and (with --deploy) pushes code via scripts/deploy/deploy_space.py <space-id> (the committed, parameterized cousin of scripts/deploy/upload_inspector.py). The only manual prerequisite is an HF token with write scope; OAuth vars are auto-injected via hf_oauth: true. See inspector/README.md for the three-tier dev workflow.

Type-check gate (pyright)

The entire Python backend (inspector/, qua_jobs/, qua_shared/) is type-checked in basic mode as a blocking CI gate (type-check job in inspector-checks.yml). It runs via npx -y pyright --level error inspector qua_jobs qua_shared so no Python type-checker dependency is added to the image — node 24 is already present. The extraPaths in pyrightconfig.json make the flat-layout imports resolve cleanly across the three packages.

Touching any file under qua_shared/schemas/ also requires regenerating the codegen'd FE types and committing the result — see schema-codegen-check in inspector-checks.yml and docs/reference/schemas.md.

Secret rotation

• INSPECTOR_HF_TOKEN — new token → update Space secret → restart. Revoke the old after verifying.
• INSPECTOR_SESSION_SECRET — replace the Space secret → restart. All in-flight cookies invalidate (everyone logged out).
• INSPECTOR_GITHUB_DISPATCH_TOKEN — mint new PAT → update secret → restart.

Health

GET /healthz (routes/auth/health.py) returns the resolved config + substrate status: status, mode (deployed iff INSPECTOR_BUCKET_MOUNT set), bucket_mounted (checks <mount>/db/inspector.db exists), state_loaded, reciters_count, oauth_configured, auto_detect_loop, commit, and a db block (open, schema_version, last_bucket_upload_ts, bucket_lag_seconds, last_error). Returns 503 in deployed mode when degraded so probes fail loud; always 200 in local mode (no mount to check). GET /livez is the always-200 liveness probe.

GET /healthz?deep=1 adds an opt-in sample_validation block — a bounded external-bucket drift probe (services/storage/bucket_audit.py::sample_validation): it round-trips the DB catalog through repo_catalog.snapshot() and audits a small reciter-folder sample (default 3, spread across the sorted slug list) against the qua_shared/schemas definitions. Shape: {ok, catalog_ok, sampled, n_reciters, errors}. The default probe never walks the bucket (latency unchanged); a deep probe that finds drift flips status to degraded (503 in deployed mode) so external-file drift surfaces at health-check time instead of mid-request.