docs/reference/catalog.md

Catalog Reference

Storage: the catalog is SQLite (inspector.db, tables read/written via inspector/services/db/repo_catalog.py), uploaded whole-file to the bucket after each committed write (services/db/sync.py). It was catalog/reciter_catalog.json pre-migration; that JSON backup (storage_paths.catalog_path()) is now dead — no app reads it (the DB is the catalog source of truth; the whole-bucket validator deliberately skips it) and is slated for removal. Audio metadata sidecars stay JSON on the bucket at catalog/audio_manifest/<slug>.json (storage_paths.audio_manifest_path), read via services/audio/audio_meta.py.

Layers, slug convention, schema, audio-metadata split, naming guide, and add/probe/maintain procedures.

1. Where things live

Concern	Location	Writer
vocab / reciters / deliveries / aliases / derived	SQLite tables (see §4 col map)	repo_catalog.py (via services/state/catalog.py)
Read model (ReciterCatalog)	repo_catalog.snapshot() → cached on db_seq by services/state/catalog.py::snapshot()	—
Pydantic shapes (runtime authority)	qua_shared/schemas/bucket/catalog.py	—
Per-delivery audio sidecar	catalog/audio_manifest/<slug>.json (bucket JSON)	offline probe scripts/audio/probe_audio_meta.py
Public read endpoint	/api/static/catalog.json (routes/public/static.py) — serializes snapshot()	—

repo_catalog.snapshot() reassembles the full ReciterCatalog so the public JSON stays byte-identical to the pre-migration file (migration parity gate diffs model_dump(by_alias=True)). Sidecars are 5–20 KB each for by_surah (114 chapters), larger for by_ayah (6236 ayahs).

2. The three layers

Layer	Identity question	Where it lives
Reciter	Who is the human?	reciters table row, keyed by reciter_id
Recording	Which artistic act? (implicit — group-by, not stored)	computed over deliveries
Delivery	Which file did we actually fetch?	deliveries table row, keyed by slug

A "recording" — same reciter performing the same style+riwayah+year — is not a stored entity. It's a group-by:

recording = (reciter_id, riwayah, style, recording_year, variant_label)

Sibling deliveries of the same recording (e.g. mp3quran + qul both serving Mishary's 2008 Hafs Murattal) are discovered with GROUP BY over deliveries. No recording_id middle layer — denormalizing year/variant across CDN-rows is cheap and removes a foreign key.

Source × Channel are orthogonal

• source = the website we found the audio on (mp3quran.net, everyayah.com, qul.tarteel.ai, surahquran.com, quranicaudio.com, tvquran.com).
• channel = the host actually serving the bytes (server*.mp3quran.net, download.quranicaudio.com, audio-cdn.tarteel.ai, download.tvquran.com, everyayah.com, *.archive.org).

Relationship is M:N. One source can route through multiple channels (qul → both tarteel and quranicaudio); one channel can serve multiple sources (quranicaudio channel serves both quranicaudio and qul; archive_org serves surah-quran).

The (source, channel) pair on each delivery row is authoritative — vocab stores neither direction of the mapping. The materialized rollup derived.source_channels[] caches it (refreshed by repo_catalog.refresh_derived() on every delivery add).

CDN is the primary dedup axis. Same (reciter, riwayah, style, channel) is kept as separate deliveries when bitrate, sample rate, channels, or duration differ — different masters. Within-tuple disambiguators on the slug: _<bitrate>k when bitrates differ, _v2 otherwise (§3).

Download-only sources (YouTube / yt-dlp)

Some audio isn't a CDN of direct .mp3 URLs — it's a playlist on YouTube (or any yt-dlp-supported host: SoundCloud, archive.org, Spreaker). These map onto the same model:

• channel = youtube (the host). Surfaces in the public dashboard filters like any channel. Added on first ingest via vocab_additions (idempotent) — not a migration, since channel vocab is data, not schema; the canonical definition lives in BUILTIN_CHANNELS (.local/extraction/intake/channel_match.py).
• source = the uploader (per-uploader provenance, e.g. a specific YouTube channel). Not surfaced in public filters (there is no source filter axis — only channel); supplied per-request at ingest (--source + --proposed).
• source_url = the originating playlist URL, stored on the delivery row (§4); the reciter-detail modal renders the channel cell as a hyperlink to it.

Download-only deliveries are bucket-served only: their per-chapter audio/<ch>.mp3 is created during extraction (YouTube serves opus/m4a, never mp3 — see §5). The watch-page URL in the sidecar is provenance, not a streamable fallback — the audio-proxy never streams it; playback always uses the persisted bucket mp3.

3. Slug convention

<reciter_id>[_<riwayah_short>][_<style_short>][_<year>]_<channel_short>[_<disambiguator>]

Fixed ordering (left to right): reciter, riwayah, style, year, channel, disambiguator.

Regex: SLUG_RE = ^[a-z][a-z0-9_]{1,79}$ (qua_shared/schemas/config/state.py). ASCII lowercase, single underscores. No code parses the slug — opaque human-readable ID. reciter_id and delivery slug both use this regex (ReciterEntry, Delivery validators). Source slug is the exception: SOURCE_SLUG_RE = ^[a-z][a-z0-9_-]{1,79}$ allows hyphens (surah-quran).

Suffix rules:

Component	Omit when	Source
riwayah_short	hafs (Hafs an Asim, the default)	riwayahs.short
style_short	murattal (the default)	styles.short
year	not known — never invent	row's recording_year
channel_short	never — mandatory	channels.short
disambiguator	only when same (reciter, riwayah, style, year, channel) has >1 delivery	derived at seed

Disambiguator forms:

• _<bitrate>k — colliding deliveries differ by bitrate, e.g. mahmoud_khalil_al_husary_qdc_64k vs ..._qdc_128k.
• _v2, _v3, … — fallback when bitrates match (different masters, same encoding); extend the chain when 3+ collide.
• _byayah — appended after the channel for a by_ayah delivery: ahmad_al_nufais_tarteel_byayah. Category marker, not a within-tuple disambiguator; stays last so left-to-right ordering holds for everything before it.

Worked examples:

Source manifest	Slug
mp3quran/abdulbasit_abdulsamad.json	abdulbasit_abdulsamad_mp3quran
qul/abdulbasit_abdulsamad.json (→ tarteel)	abdulbasit_abdulsamad_tarteel
qul/abdulbasit_abdulsamad_qdc.json	abdulbasit_abdulsamad_qdc
qul/abdulbasit_abdulsamad_mujawwad_qdc.json	abdulbasit_abdulsamad_mujawwad_qdc
mp3quran/abdelaziz_sheim_warsh.json	abdelaziz_sheim_warsh_mp3quran
surah-quran/abbadi_houssem_eddine.json (→ archive.org)	abbadi_houssem_eddine_archive
by_ayah/qul/ahmad_al_nufais.json (tarteel, by_surah sibling exists)	ahmad_al_nufais_tarteel_byayah

Renames are free for a bare catalog row — URLs are preserved per-delivery in the sidecar; nothing about the audio depends on the slug. But once a lifecycle state row exists (delivery_states/transitions, reciters/<slug>/, the audio manifest sidecar all key on the slug), a rename becomes a real cross-table migration. This is why slug minting for admin-dashboard intake contributions is deferred to the offline ingest pipeline (services/admin/intake.py accept records only the owner's canonical reciter_id): the slug's mandatory channel_short suffix needs the channel, which is only known after the audio is fetched + probed. Minting the slug at ingest — when source/channel/bitrate are all known — gets it right on the first try rather than guessing at accept and migrating later.

4. Schema

Authority: the pydantic models at qua_shared/schemas/bucket/catalog.py are the runtime authority (cross-consumer: Inspector, training pipeline, dataset builder, GH Actions). FE types are codegen'd from them — never hand-edit inspector/frontend/src/lib/types/generated/schemas.ts. The tables below map pydantic field ↔ SQLite column.

Null convention: any <type> | null field accepts null = "missing / not yet identified". Never "" or "unknown" sentinels. Applies to name_ar, country, recording_context, recording_year, variant_label, and all audio-metadata fields.

DDL: inspector/services/db/migrations/0001_init.sql. Datetimes are TEXT ISO-8601 UTC; JSON-valued columns (audio_categories, host_patterns, derived) are TEXT (orjson). FKs enforced by SQLite; uniqueness + cross-table FK integrity also re-validated by ReciterCatalog's _check_unique_keys model validator on snapshot.

Vocab — riwayahs / styles (_ShortNamed)

Column	Type	Notes
slug	TEXT PK	e.g. hafs_an_asim. Immutable.
short	TEXT	Used in delivery slugs, e.g. hafs.
name	TEXT	Display, e.g. Hafs A'n Assem.

styles values: murattal / mujawwad / muallim / children_repeat / hadr. Legacy taraweeh style was migrated to style: "murattal" + recording_context: "taraweeh" — the taraweeh style slug is gone from vocab. hadr = fast-paced, less ornamented than murattal.

Vocab — sources (Source)

Column	Type	Notes
slug	TEXT PK	mp3quran, qul, surah-quran (hyphen allowed in source slug only).
name	TEXT	Display.
url	TEXT | null	Website root.
audio_categories	TEXT (JSON string[])	Subset of ["by_surah", "by_ayah"] (AudioCategory enum).

Vocab — channels (Channel)

Column	Type	Notes
slug	TEXT PK	mp3quran, quranicaudio, archive_org.
short	TEXT	Used in delivery slugs, e.g. qdc for quranicaudio, archive for archive_org.
name	TEXT	Display.
host_patterns	TEXT (JSON string[])	Globs matched against URL hostnames at ingestion.

Vocab — recording_contexts (RecordingContext)

Column	Type	Notes
slug	TEXT PK	Literal["studio","broadcast","prayer","taraweeh","mixed"].
name	TEXT	Display.

Orthogonal to style (pace/ornamentation). Semantics: studio = deliberate studio recording; broadcast = radio (iza'a)/TV; prayer = live non-Taraweeh prayer (Fajr, Qiyam, …); taraweeh = Ramadan night prayers (split from prayer for volume + annual cadence); mixed = single delivery spanning multiple contexts. Delivery's recording_context is nullable — null = not yet classified (default for seed rows). No "unknown" sentinel.

reciters (ReciterEntry)

Column	Type	Notes
reciter_id	TEXT PK	Slug-form (SLUG_RE), e.g. mahmoud_khalil_al_husary. Immutable.
name_en	TEXT (≥1)	Canonical display. §6 naming guide.
name_ar	TEXT | null	Arabic; null if unknown.
country	TEXT | null	ISO-2 or null.
notes	TEXT | null	Free-form, ≤500 chars.

Editable columns: _RECITER_WRITABLE = (name_en, name_ar, country, notes) (repo_catalog.py). reciter_id is immutable.

deliveries (Delivery)

Column	Type	Notes
slug	TEXT PK	Per §3.
reciter_id	TEXT FK→reciters
riwayah	TEXT FK→riwayahs.slug
style	TEXT FK→styles.slug	Pace/ornamentation only.
source	TEXT FK→sources.slug
channel	TEXT FK→channels.slug
source_url	TEXT | null	Originating playlist/set URL for download-only sources (YouTube playlist, SoundCloud set). null for CDN deliveries — their per-chapter URLs live in the sidecar. Surfaced on PublicDelivery; the reciter-detail channel cell hyperlinks to it.
recording_context	TEXT | null FK→recording_contexts.slug	null when unclassified. Orthogonal to style.
recording_year	INTEGER | null	4-digit Hijri or CE; null when unknown.
variant_label	TEXT | null	Free token (madinah, studio_2008); rare.
audio_category	TEXT	by_surah | by_ayah (AudioCategory).
chapter_count	INTEGER (≥0)	URL count. Full mushaf = 114 (by_surah) or 6236 (by_ayah); partial coverage lower.
codec	TEXT	mp3.
container	TEXT	mp3.
sample_rate_hz	INTEGER | null	Probed; null for unprobed by_ayah.
channels	INTEGER | null	1 (mono) or 2 (stereo); ge=1, le=2.
bitrate_mode	TEXT	BitrateMode enum (§5). Default unknown.
bitrate_kbps_nominal	INTEGER | null	CBR exact / VBR avg / ABR target. null when bitrate_mode == "mixed" (model validator enforces).
total_duration_sec	INTEGER | null	Whole-mushaf seconds.
added_at	TEXT (datetime)	ISO-8601 UTC.
added_by_hf_id	TEXT	HF user id of the maintainer who added the row.

Editable columns: _DELIVERY_WRITABLE (repo_catalog.py) covers riwayah/style/recording_context/recording_year/variant_label/source/channel/source_url/audio_category/chapter_count/codec/container/sample_rate_hz/channels/bitrate_mode/bitrate_kbps_nominal/total_duration_sec. The service (services/state/catalog.py::edit_delivery) exposes a narrower public surface: only riwayah/style/recording_context/recording_year. slug/reciter_id immutable. Checksum does not live on the row (sidecar _meta.checksum only).

audio_manifest/<slug>.json (bucket sidecar — AudioManifestSidecar)

{
  "schema_version": 1,
  "slug": "<delivery slug>",
  "_meta": {                                 // SidecarMeta (alias "_meta")
    "checksum": "sha256:<hex>",              // sole storage; row carries no copy
    "source_meta_reciter": "<original>",     // migration/audit
    "source_manifest_path": "data/...",      // original ingestion path
    "chapter_count": 114,
    "category": "by_surah"
  },
  "chapters": {                              // dict[str, ChapterEntry]
    "1": { "url": "https://...", "size_bytes": 803456, "duration_sec": 50, "bitrate_kbps": 128, "bitrate_mode": "cbr", "max_linear_seek_err_ms": 26 }
  }
}

Chapter keys: "1"–"114" (by_surah) or "<surah>:<ayah>" (by_ayah). Per-chapter metric fields nullable until probed. ChapterEntry fields: url (required), size_bytes, duration_sec, bitrate_kbps, bitrate_mode (cbr/vbr per chapter), and max_linear_seek_err_ms (probe verdict evidence).

AudioManifestSidecar (qua_shared/schemas/bucket/catalog.py) is a bucket artefact with pure extra="forbid": any unknown/legacy sidecar field raises ValidationError on parse (writer-drift signal), never silently stripped — the same external-file strictness the bucket-validation harness surfaces.

Checksum (_meta.checksum): sha256(normalized_urls_sorted.joined_by_newline) at build time. Normalization: lowercase hostname, strip trailing slashes, drop fragment; query order + path case preserved (CDN-sensitive). Lives only in the sidecar; re-probe jobs compute + compare.

audio_meta.py reads sidecars via get_backend().read_json(audio_manifest_path(slug)) and caches each into LRU slots (_audio_manifest doc + _audio_manifest_url_index URL→chapter inverse). Powers VBR routing (is_vbr), audio-proxy short-circuit (chapter_for_url), the /peaks route's URL enumeration (chapter_urls), and FE warmup hints (vbr_chapters_for_reciter, chapter_bitrate_kbps_for_reciter).

catalog_aliases (Alias)

Empty in seed. Reserved for slug/reciter_id renames. Columns: kind (slug|reciter_id), old, new, ts. Insert via repo_catalog.insert_alias.

catalog_meta + derived (Derived)

Single row id=1: schema_version (2), generated_at, derived (TEXT JSON). derived.source_channels[] = M:N rollup {source, channel, delivery_count} per observed pair, persisted verbatim (not reconstructed) for parity; recomputed by repo_catalog.refresh_derived() whenever the delivery set changes (deterministic order (source, channel)).

5. Audio metadata: row vs sidecar

Field	Uniform across chapters?	Lives in
codec, container, sample_rate_hz, channels, bitrate_mode	yes (encoder settings)	deliveries row
bitrate_kbps_nominal (CBR exact; VBR avg)	yes (label)	row
total_duration_sec (whole mushaf)	yes (aggregate)	row
bitrate_kbps (measured per chapter)	no (varies for VBR)	sidecar
duration_sec / size_bytes / url (per chapter)	no	sidecar

CBR vs VBR

Row-level bitrate_mode (BitrateMode enum) derived from per-chapter probe data:

Enum collapsed to four values by migration 0014 (legacy mostly_cbr/mostly_vbr/abr folded into mixed):

Value	Meaning
cbr	All probed chapters CBR. bitrate_kbps_nominal = exact.
vbr	All probed VBR. bitrate_kbps_nominal = target / observed average.
mixed	Chapters disagree (any CBR+VBR within the delivery, or differing CBR rates). Use sidecar for per-chapter truth. bitrate_kbps_nominal is null (model validator enforces).
unknown	No chapters probed yet (default for by_ayah).

Per-chapter ChapterEntry.bitrate_mode (sidecar) is cbr/vbr only — the mixed collapse is a row-level rollup over chapters that individually are one or the other.

Detection — whole-file linear byte→time seek error (qua_shared/mp3_frames.py::classify_bitrate_mode, mirrored inline in probe_audio_meta.py): walk every audio frame, measure how far a browser's linear time→byte seek would land from each frame's true time; ≤ 200 ms ⇒ cbr, else vbr. This is the only reliable signal — it is exactly what the playback transport needs to know (can the browser seek this natively?). The two shortcuts this replaced both produced systematic mislabels: mutagen's header bitrate_mode calls every Xing-tagged file VBR (incl. CBR audio our pipeline tagged Xing) and len(set(bitrates))==1 / head-only uniformity is fooled by a stray frame or by VBR whose variation starts past the head (whole tvquran/archive reciters slipped through as CBR and stalled on seek). Because the metric needs every frame, source probing (probe_audio_meta.classify, mp3_probe.probe_source(allow_full=True)) downloads the full file. Per-chapter results land in the sidecar, rolled up to the row at build.

Download-only sources create their own encode. Unlike CDN audio (publisher mp3 bytes preserved verbatim, with only a Xing seek header injected via -c:a copy), a YouTube/yt-dlp source is opus/m4a — so extraction produces the canonical mp3 once: 192 kbps CBR, 44.1 kHz, mono, cover-art stripped (segments/audio_io.py::_download_via_ytdlp). Because the watch URL can't be HTTP-frame-probed, the row + sidecar audio fields come from a post-align reprobe of the produced files (ingest_intake.py::reprobe_persisted_audio). Forced-CBR ⇒ these deliveries never hit the VBR playback path.

Style vs recording_context

• style = pace/ornamentation: murattal / mujawwad / muallim / children_repeat / hadr.
• recording_context = why the recording exists: studio / broadcast / prayer / taraweeh / mixed; null when not identified.

Two independent axes — "what kind of recitation, and where from?" The taraweeh style slug is removed from vocab; new taraweeh entries pick style: "murattal" + recording_context: "taraweeh".

6. Naming style guide

New name_en entries must follow these (applied at seed).

Axis	Rule
Al- prefix	Al-<Surname> (capital A, hyphen). Never AlSurname, al-surname, Al Surname.
Egyptian El-	Preserve when source-attested (Ahmed El-Shafei, Hamza El-Far). Regional signal.
Sun-letter (Az-, Ath-, As-)	Preserve where attested. Don't retroactively introduce — Al-Sudais stays.
Abdul- / Abdel-	Glued: Abdulrahman, Abdelaziz. Never spaced/hyphenated.
First-name canonical spellings	Mohammed (not Muhammad/Mohamed/Mohammad), Ahmed (not Ahmad/Ahmet), Yusuf (not Yousef/Youssef/Yousif), Mahmoud (not Mahmood). Also Khalid←Khaled, Mansour←Mansoor, Mishary←Mishari/Meshari, Adel←Adil, Khalil←Khaleel, Yasser←Yassir, Mustafa←Mostafa, Hisham←Hesham, Saeed←Sayeed, Tawfiq←Tawfeeq. Maintainer override: canonical normalization is deliberate (a prior report recommended against it to preserve regional signal — overridden).
bin patronymic	Lowercase between names: Ahmed Talib bin Humaid.
Honorifics	Strip (Sheikh, Sheik, Shaik, Qari, Hafiz). One seed exception (Ustaz Zamri).
Numbers in names	Only Madinah/Makkah Taraweeh series years (1437H).
Apostrophes	Straight ASCII ' (U+0027).
Compound-word casing	Lowercase after first letter (Abdulmajid, never AbdulMajid).
Slug mirrors name	Al-X → al_x (e.g. mahmoud_khalil_al_husary).

name_ar is preserved as-source — no Arabic normalization beyond emptying a literal "unknown" to null.

7. Seed origin (historical)

The seed catalog (864 deliveries, 422 reciter clusters, 0 slug collisions) was built one-time by a dedup + probe pipeline whose scratch artifacts lived in .local/dedup/ (gitignored): dual-agent name clustering, same-channel ffprobe duplicate detection, naming-consistency + canonical-spelling normalization passes, then a bulk 256 KB-range audio probe and a build_catalog.py assembly to reciter_catalog.json + sidecars. That JSON was later migrated into SQLite (scripts/migrations/migrate_json_to_sqlite.py). The .local/dedup/ artifacts are historical scratch — not load-bearing. Seed rows carry added_at = 2026-05-12T00:00:00Z, added_by_hf_id = system_seed.

8. Mutation surface

All catalog writes go through inspector/services/state/catalog.py (sole writer). Each mutation requires _require_maintainer(actor) (MAINTAINER or OWNER → else NotAuthorizedForCatalog), runs inside db.sync.durable_transaction() (atomic with its audit row; nesting-safe for the accept flow), and appends a catalog.* audit event.

Service fn	Repo fn	Audit event	Notes
add_reciter	repo_catalog.add_reciter	catalog.added (kind=reciter)	Duplicate → InvalidCatalogChange.
edit_reciter	repo_catalog.edit_reciter	catalog.edited (kind=reciter)	Patch computed before txn; no-change → no I/O. reciter_id immutable.
add_delivery	repo_catalog.add_delivery	catalog.added (kind=delivery)	Refreshes derived. FK/Duplicate → InvalidCatalogChange.
edit_delivery	repo_catalog.edit_delivery	catalog.edited (kind=delivery)	Public surface = riwayah/style/recording_context/recording_year only. FK violation → InvalidCatalogChange.
add_audio_source	repo_catalog.add_source	catalog.audio_source_added	New vocab source.

Audit patch shape {field: {from, to}} is computed in the service; the repo persists only. Reads: snapshot() (cached on db_seq), find_delivery, find_reciter, display_name.

Public-projection edits stamp release staleness. edit_delivery / edit_reciter carry PUBLIC_DELIVERY_FIELDS / PUBLIC_RECITER_FIELDS — the catalog fields that surface in the HF mushafs catalog + GH release catalog.json. When a changed field is in that set, the service stamps the affected published hf/gh rows catalog_edit-stale (repo_releases.stamp_stale / stamp_stale_for_reciter, reciter edits fanning out to every delivery) inside the same txn — so the Releases tab surfaces the drift and offers the cheap catalog refresh. Admin-only fields (notes / variant_label / source) never stamp. → dataset-and-releases.md.

9. Adding a single reciter / delivery

1. Check for collision. find_reciter(reciter_id) and scan name_en/name_ar variants. If the reciter exists, you're adding a delivery — skip to step 3.
2. Add the reciter via catalog.add_reciter(actor=…, reciter_id=…, name_en=…, …). Pick reciter_id per SLUG_RE + §6.
3. Decide the delivery shape — riwayah, style, recording_year (if known), source (must exist in sources; add_audio_source first if not), channel (must exist in channels; §10 to add).
4. Compute the slug (§3); apply disambiguator rules on collision.
5. Probe at least chapter 1 (scripts/audio/probe_audio_meta.py) → bitrate_mode, bitrate_kbps_nominal, sample_rate_hz, channels. codec/container = mp3.
6. Add the delivery via catalog.add_delivery(actor=…, delivery=Delivery(...)) (constructs + validates the pydantic model; FK + duplicate enforced at insert; derived auto-refreshed; row + audit committed durably to the bucket DB).
7. Generate the sidecar at catalog/audio_manifest/<slug>.json (URL map; per-chapter metrics may be null until a later probe). Checksum → _meta.checksum, never the row.

There is no separate validate-then-rebuild step: the SQLite FKs + Delivery/ReciterCatalog validators enforce integrity at write time, and sync uploads the DB after the committed txn.

10. Adding a source / channel

New source (vocab.sources): catalog.add_audio_source(actor=…, source=Source(slug=…, name=…, url=…, audio_categories=[…])). Hyphens permitted in source slug only.

New channel (vocab.channels) — needed when bytes come from an unregistered host. No service endpoint today; insert via repo_catalog.load_vocab / a migration:

1. Identify the host pattern — most specific glob without false positives (download.example.com, not *.example.com).
2. Slug: lowercase ASCII + underscores, 2–40 chars, <domain_root> or <host>_<role> (archive_org).
3. short: ≤10 chars, mnemonic, used in delivery slugs.
4. Verify the probe's host→channel mapping picks up the new pattern.

Channel host migration (CDN relocates, e.g. download.tvquran.com → cdn.tvquran.net): add the new pattern to channels.host_patterns alongside the old (keep old for historical sidecars); re-scrape affected manifests (row identity stable, sidecar URLs change → _meta.checksum changes, invalidating downstream caches); re-probe; audit-log the date + host shift + delivery count.

Registering a download-only (yt-dlp) source

YouTube, SoundCloud, archive.org, Spreaker, Bandcamp, … all flow through the offline intake's enumerate + download path (ingest_intake.py → intake/playlist.py + segments/audio_io.py). Registering one is data, not code:

1. Add a Channel to BUILTIN_CHANNELS (.local/extraction/intake/channel_match.py) with its host_patterns and gh_release_eligible=False (not a public CDN — its source URLs can't go in a public GH release; the bucket mp3 is the distributed artifact). The driver detects it out of the box and auto-adds it to the catalog via vocab_additions on first ingest (idempotent).
2. The per-uploader source is supplied per-request: ingest_intake.py --source <uploader_slug> --proposed <vocab.json> (a Source with name/url/audio_categories).

No new download/encode/probe code is needed — yt-dlp handles enumeration + fetch, and the canonical encode (§5) + post-align reprobe are source-agnostic. Full workflow: the segments-extraction skill's references/playlist_intake.md.

11. Removing a delivery / reciter

Maintainer-only, manual; mandatory audit. No automation / row-level archive flag yet.

Single delivery: delete the deliveries row + the sidecar; refresh_derived(); if the reciter has no remaining deliveries optionally remove the reciters row (orphans are harmless); audit the reason + actor.

Entire reciter: remove all deliveries + sidecars, then the reciters row; optionally record a retirement alias; same audit requirement.

Never repurpose a reciter_id or slug — once retired it stays burned, repurposing breaks URL stability for any external system that cached the old mapping.

12. Probing

Job	Script	Scope
Single-reciter VBR/CBR + bitrate	scripts/audio/probe_audio_meta.py --reciter <slug>	one manifest
Bulk row fields	scripts/audio/probe_audio_meta.py --all	all tracked reciters
Per-ayah probing	not run — by_ayah deferred until a consumer needs per-ayah metrics	—

How: 256 KB HTTP Range fetch per URL (no full downloads). mutagen reads Xing/Info/VBRI/LAME (authoritative bitrate mode + duration); frame-by-frame scan as fallback. Content-Range response header gives total size free. URL-keyed probe cache short-circuits reruns. Concurrency capped (≤10 for *.archive.org politeness). Re-probe compares _meta.checksum; unchanged → skip.

When: new delivery (probe ≥chapter 1 for row fields); new source (bulk by_surah); URL-list change (= checksum change → re-probe that sidecar); periodic maintenance pass for silent format/URL drift.

13. Known limitations

• by_ayah deliveries are unprobed — row audio fields null, URL-only sidecars. Bounded re-probe work, deferred until a consumer needs per-ayah metrics.
• Per-chapter duration_sec is null when no Xing TOC is in the first 256 KB. Full-file probing deferred.
• surah-quran source slug carries a hyphen — tolerated in sources.slug only; never in reciter_id, delivery slug, or channel slug.
• No aliases yet — catalog_aliases empty; renames are manual.
• No recording_year in the seed — field exists; populating it is per-reciter research, deferred.
• recording_context null for ~all seed rows except migrated taraweeh. Backfill is a domain-research task, deferred.