docs/reference/architecture.md

Inspector backend architecture — START HERE

The Inspector is a single-worker Flask app serving a Svelte 5 SPA. It manages the lifecycle of Quran-recitation forced-alignment deliveries (catalogue → align → review → release → publish) and lets editors trim/split/merge/re-reference word-level segments. The deployed app is a Docker HF Space; its source of truth is a container-local SQLite DB (inspector.db) full-file synced to an HF bucket after every committed write. Per-reciter content (detailed.json, segments.json, timestamps/<ch>.json, peaks, audio) stays JSON/binary on the bucket, read on demand.

Layering

HTTP ─▶ routes/<domain>/*          thin Flask blueprints: parse → service → jsonify
                │                   (primary Flask layer; only services/ exception is auth/auth.py)
                ▼
        services/<domain>/*        business logic, Flask-free, returns plain dicts
                │                   db/ storage/ audio/ auth/ state/ segments/
                │                   validation/ activity/ reference/ quran_foundation/
                ▼
   domain/  adapters/  utils/      pure model · JSON↔domain conversion · helpers

   services/storage/cache.py       single owner of every mutable cache var
   config.py · constants.py        env-overridable tunables · domain literals

Invariants:

• Routes are the Flask layer. Services accept params, return dicts, raise typed errors; app.py maps those errors to the {error: str} JSON envelope. The sole services/ exception is services/auth/auth.py — authlib's flask_client OAuth integration (_CacheStateOAuth/_CacheStateIntegration subclass Flask integration) cannot be abstracted away. The services-flask-free CI guard enforces this with auth.py as the only allow-listed file.
• Single worker. app.py::_assert_single_worker refuses to boot under any multi-worker signal. State store, per-slug locks, role cache, OAuth-state cookie, and all in-memory caches assume one process.
• SQLite is the source of truth for the 7 former bucket JSON stores + audit. Reads go through services/db/repo_*; writes commit in services/db/connection.py::transaction() then sync via services/db/sync.py.
• Bucket-first reads for reciter content go through services/storage/data_dir.py + storage_paths.py, never raw Path.read_text().
• services/__init__.py re-exports every submodule at the legacy flat name (from services.cache import … still resolves to services/storage/cache.py) via sys.modules aliasing. New modules drop into a subpackage + one re-export line.

Services subpackage map

Subpackage	Role	Key files
db/	SQLite substrate — source of truth for state/catalog/access/claims/requests/activity + transitions (audit). WAL single-writer, full-file bucket sync.	connection.py (txn primitives, transaction(), get_writer, current_db_seq), migrate.py (PRAGMA user_version runner over migrations/NNNN_*.sql), sync.py (bucket pull/push + CAS guard), repo_state.py, repo_catalog.py, repo_access.py, repo_claims.py, repo_requests.py, repo_activity.py, repo_transitions.py, _serde.py, errors.py
storage/	Bucket/filesystem backend, per-reciter content IO, the cache registry, local auto-mount, external-file validation.	cache.py (all cache vars + invalidation), data_dir.py (slug→bytes/dict under reciters/<slug>/), storage_paths.py (pure on-bucket path strings), data_loader.py (cached loaders for segs/verses/probe/reference), hf_bucket.py (BucketBackend/FilesystemBackend), auto_mount.py (local hf-mount FUSE), bucket_audit.py (Flask-free per-reciters/<slug>/ artefact validator against qua_shared/schemas; backs the per-slug CLI scripts/bucket/audit_bucket_reciter.py, the whole-bucket gate scripts/diagnostics/validate_bucket.py, and /healthz?deep=1 via sample_validation)
audio/	Everything bytes-on-disk → <audio>: source resolution, fetch, peaks, manifest.	audio_source.py (resolve bucket/CDN + VBR), audio_fetch.py (bucket audio + peaks reads), audio_meta.py (per-chapter sidecar), peaks.py (ffmpeg compute), peaks_slim.py (canonical schema-v3 packed format), peaks_history.py (per-op JSONL)
auth/	HF OAuth + signed identity cookie, roles, pure authorization predicates.	auth.py (OAuth + inspector_session cookie + dev-mode), access.py (roles facade over DB), permissions.py (pure predicates + normalize_reason), predicates.py (/api/reciter-task can-edit predicates), hf_users.py (login→hf_user_id proxy), secrets_guard.py
state/	Reciter lifecycle state machine, catalog, requests, audit facade.	state.py (transition(slug, event, actor), _HANDLERS, typed errors), catalog.py (snapshot() → ReciterCatalog, mutations), pending_requests.py, request_archive.py, audit.py (legacy append() facade → repo_transitions)
segments/	Save / undo / read queries for the Segments editor, plus auto-detect + auto-split.	save.py (write-through, history append, rebuild segments.json), undo.py (batch reversal + snapshot verify), segments_query.py (read-only data), auto_detect.py (reconciler: reciters/ dir → alignment_completed), auto_split.py (precomputed cursor lookup), qalqala.py (persisted qalqala_letter)
validation/	Registry-backed segment validation + chapter counts (Flask-free).	registry.py (IssueRegistry, category sets — keep in lockstep with FE registry.ts), classifier.py, snapshot_classifier.py, detail.py, _missing.py, _structural.py; package __init__.py exposes validate_reciter_segments
activity/	Audit-event classification + public activity feed + history queries + stats. The admin notifications rail was retired — admin awareness lives in the Admin dashboard tabs now.	activity_classification.py (event → public/hidden), activity_state.py (global-tombstone facade), public_activity.py, history_query.py (edit-history read + split-group/resolved-by-edit indexes), stats.py, search_normalize.py (mirrors FE normalizeArabic)
reference/	Static Quran reference payloads + timestamps server + public dashboard mapper.	quran_refs.py (DK words + verse word counts, one immutable asset), timestamps.py (manifest + per-chapter shard server, gzip LRU), public_state.py (ReciterRow → six-bucket public taxonomy, strips identity)
quran_foundation/	Quran.Foundation API integration (OAuth user APIs + content APIs + bookmarks).	oauth.py, content.py, bookmarks.py, reciter_map.py, session.py (signed qf_session cookie), config.py — see Quran Foundation integration

Routes registry

Blueprints register in routes/__init__.py::register_blueprints. Subpackage __init__.py files are empty; each module defines its own Blueprint.

URL prefix	Blueprint	Module	Purpose
/healthz, /readyz	health	routes/auth/health.py	Liveness/readiness; reports SQLite + bucket-mount status (503 when degraded)
/api (/api/auth/*)	auth	routes/auth/auth.py	HF OAuth login/callback/logout; sets inspector_session identity cookie
/api/dev	dev	routes/auth/dev.py	Dev-only: flip inspector_dev_role cookie (404 when dev mode off)
/api	claims	routes/claims/claims.py	Claim/release/mark/unmark + /api/reciter-task/<slug> (via state.transition)
/api	requests	routes/claims/requests.py	Request flow — user submit + admin review/reject/undiscard
/api/public	public	routes/public/public.py	Public read-only dashboard data (identity-redacted)
/api/static	static_data	routes/public/static.py	Static projections (catalog.json, quran-refs.json, …) for the FE
/api/admin/access	access_admin	routes/admin/access.py	Role grant/revoke/update (mirrors services/access.py)
/api/admin	admin_actions	routes/admin/actions.py	Admin override transitions (force-release, reassign, force-set-state, send-back)
/api/public/activity	public_activity_admin	routes/admin/activity.py	Owner-only DELETE — global tombstone for a public-feed card
/api/admin	admin_reconcile	routes/admin/reconcile.py	Manual trigger for the auto-detect reconciler
/api/ts	ts	routes/timestamps/timestamps.py	Timestamps tab: /manifest, /shard/<reciter>/<chapter>, /config
/api/seg	seg_data	routes/segments/data.py	Segments tab read-only data (/all, refs, …)
/api/seg	seg_edit	routes/segments/edit.py	Segments save/undo (@require_same_origin → @require_edit_lock)
/api/seg	seg_val	routes/segments/validation.py	Validate, stats, edit-history
/api/seg	peaks	routes/segments/peaks.py	/peaks, /segment-peaks, /history-peaks
/api/seg	audio_proxy	routes/audio/proxy.py	/audio-proxy/<reciter> — bucket-resident audio; CDN stream-through fallback (same-origin 200/206 + ACAO, not a 302)
/api/audio	audio_meta	routes/audio/metadata.py	Audio tab metadata
/api/seg	segment_clip	routes/audio/clip.py	/segment-clip — ffmpeg MP3 window clip (VBR-safe seek)
/api/qf	qf_auth	routes/qf_auth.py	Quran.Foundation OAuth2 (pre-prod user APIs)
/api/qf/content	qf_content	routes/qf_content.py	QF Content-API read proxy (word-by-word translations)
/api/bookmarks	bookmarks	routes/bookmarks.py	QF bookmarks proxy ([] for anonymous)
/ + static	(Flask static)	app.py	SPA shell frontend/dist/index.html; /api/surah-info cross-tab route

_admin_helpers.py (package root, not a blueprint) is shared by admin/ + claims/ routes.

domain / adapters / utils

inspector/domain/ — pure model, no Flask, no IO:

File	Role
segment.py	Frozen Segment dataclass — one aligned segment as stored in detailed.json
command.py	SegmentPatch — change set produced by a command; undo applies its inverse
identity.py	Deterministic segment_uid (uuid5 over NAMESPACE_INSPECTOR); must match the TS impl

inspector/adapters/ — JSON ↔ domain conversion:

File	Role
detailed_json.py	detailed.json entries list ↔ segment dicts with backfilled UIDs (load_entries)
segments_json.py	Rebuild verse-aggregated segments.json from detailed entries
save_payload.py	Incoming save payload → canonical segment dicts (make_seg)

inspector/utils/ — pure utilities + cross-cutting decorators:

File	Role
decorators.py	require_same_origin (CSRF), require_edit_lock (signed-in + active claim + editable, admin_bypass)
json_response.py	orjson_response — fast JSON for big-payload routes
arabic_text.py	Strip Quranic decoration, last-letter extraction
references.py	Reference-string parsing + segment classification helpers
repetitions.py	Repetition-segment helpers over wrap_word_ranges
formatting.py	Display formatting
io.py	Atomic writes + safe filenames
uuid7.py	UUIDv7 (time-ordered) generator

app.py boot sequence

Runs at module import (so python3 inspector/app.py and gunicorn inspector.app:app follow the same path):

1. Insert repo root on sys.path (resolves qua_shared.*).
2. _load_dotenv_for_local_dev() — hydrate env from <repo>/.env then inspector/.env (process env wins).
3. Auto-enable INSPECTOR_DEV_MODE=1 when not behind proxy and not under pytest.
4. auto_mount() — local hf-mount FUSE of the bucket (silent degrade to API path).
5. _configure_logging() — plain single-line formatter; silence httpx/urllib3/hf noise.
6. _assert_single_worker() — abort on any -w >1 / WEB_CONCURRENCY / GUNICORN_* signal.
7. Build Flask(static_folder=frontend/dist); wrap with Compress (gzip/brotli).
8. ProxyFix if INSPECTOR_BEHIND_PROXY=1; set session cookie SameSite/Secure accordingly.
9. app.secret_key = get_session_secret() (warns, doesn't crash, if missing).
10. auth_service.init_oauth(app) — register HF OAuth provider with Authlib.
11. register_blueprints(app) — all route blueprints.
12. _boot_substrate() (skipped under pytest): sync.pull() bucket DB → local → init_db() (open writer + run migrations) → auto_detect.hydrate_initial_seen() boot-scan inside deferred_sync() (one coalesced upload). Then the opt-in auto-detect loop (INSPECTOR_AUTO_DETECT=1). Deployed-mode init failure leaves app importable with db_open:false; local failure raises.
13. Register error handlers (HTTPException, UnknownReciter, InvalidTransition, NotAuthorizedForTransition, catch-all) → {error: …} envelope.
14. Register / (SPA shell) + /api/surah-info.
15. if __name__ == "__main__": parse --port, run dev server (debug/reloader only under FLASK_ENV=development).

No eager warms of reciter data — timestamps + per-reciter caches are lazy on first request.

Caching

All mutable cache vars live in services/storage/cache.py (services.cache); no global for caches elsewhere. Classes: _SingletonCache (one nullable value), _KeyedCache (LRU dict, ceiling _KEYED_CACHE_LRU_MAX=20).

• Eager (singletons, immutable post-boot): word counts, single-word verses, QPC/DK data, surah-info-lite — filled on first use, never invalidated.
• Lazy per-reciter (LRU-bounded): segments, seg meta/verses, probe-v2, auto-split, pipeline-meta, history batches, split-group index, validate/stats results, audio manifest.
• db_seq-keyed: public_reciters and catalog_snapshot cache on the monotonic current_db_seq() — any committed DB write bumps the seq and transparently invalidates them (no per-mutation hook).
• Peaks: _PEAKS_RESPONSE_CACHE (global LRU, 50 entries, serialized JSON bytes), thread-safe with explicit locks.

Invalidation hook: invalidate_seg_caches(reciter) runs on every save/undo. The surgical variant pop_seg_caches_affected_by_segment_edit preserves append-in-place caches (history batches, split-group index) and immutable ones (pipeline-meta). The chapter-peaks LRU (_PEAKS_RESPONSE_CACHE) is deliberately NOT evicted on save — peaks track immutable audio bytes, not segment edits; it sheds only via its own 50-entry LRU and an explicit pop_reciter_peaks_response_cache wherever a future path rewrites bucket peaks. Add a new cache here with its invalidation tied to the mutation that dirties it.

Quran Foundation integration

services/quran_foundation/ integrates the Quran.Foundation (QF) APIs. No standalone doc — this is the reference.

Backend-proxy pattern: per-user OAuth2 tokens stay server-side in a signed qf_session cookie; the browser never sees them. Outbound QF calls inject x-auth-token + x-client-id. Two-environment split (never mix tokens): User APIs (bookmarks) use the pre-prod client + apis-prelive base; Content APIs use the prod client (client_credentials, HTTP Basic). All modules Flask-free; only routes/qf_* + routes/bookmarks.py import Flask.

File	Role
oauth.py	authorization_code + PKCE helpers (pre-prod); token endpoint requires client_secret_basic
content.py	Content API (client_credentials) — full-surah audio URLs, word-by-word translations
bookmarks.py	Bookmarks proxy → QF /auth/v1/bookmarks + dev in-memory store; normalizes to {surah, ayah, key}
reciter_map.py	(reciter_id, style) → QF chapter-reciter id for verified reciters
session.py	Signed qf_session cookie holding the per-user token server-side (itsdangerous)
config.py	Env-driven endpoints + credentials (pre-prod OAuth issuer, user-API base, auth method)

Routes: routes/qf_auth.py (/api/qf — OAuth2 login/callback + dev-stub login), routes/qf_content.py (/api/qf/content — read-only word-by-word for the Timestamps Analysis view), routes/bookmarks.py (/api/bookmarks — returns [] for anonymous so the FE silently stays app-local). QF caches (content token, chapter URLs, token cooldown, WBW translations) live in services/storage/cache.py.

Reference index — which sibling doc per task

Doc	When to read
database.md	Working on the SQLite substrate — schema, migrations, repos, WAL/sync, CAS
state-machine.md	Lifecycle states, events, transition() handlers, admin overrides
auth-permissions.md	OAuth, identity cookie, roles, predicates, edit-lock/CSRF gating
catalog.md	Reciter taxonomy, ReciterCatalog, audio URL templates, accept flow
segments-editor.md	Save/undo flow, segment domain model, adapters, auto-split
validation.md	Validation registry, classifier categories, accordion lockstep with FE
frontend.md	Svelte 5 SPA — stores, tabs (dashboard/timestamps/segments), editGate
accordion-guides.md	Per-category guide copy for the validation accordion
config-deploy.md	Env vars, runtime profiles (dev/dev-Space/prod), HF Space deploy
data-migrations.md	Schema/data migration procedures, backfill scripts, drift checks