billingsmoore Claude Sonnet 5 commited on
Commit
fc33d54
Β·
1 Parent(s): 07e532f

Add docs/ covering configuration, storage, deployment, and operations

Browse files

Mirrors TranslationUI's docs structure. Documents the shared-droplet
architecture specifically (same Caddy/oauth2-proxy as TranslationUI,
cookie-domain widened to cover both apps' subdomains) since that's the
part most likely to confuse someone debugging this app in isolation.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

README.md CHANGED
@@ -73,3 +73,7 @@ the July Launch content pipeline (e.g. `Dharmakaya` β†’ `Darmakaya`, `Phagmo`
73
  | `ELEVENLABS_API_KEY` | Yes | ElevenLabs synthesis calls. |
74
  | `HF_TOKEN` | Yes, to save/remove glossary entries | Write access to the `tts-glossary` dataset repo. Reads fall back to a local cache (or built-in defaults) without it. |
75
  | `GLOSSARY_DATASET_REPO` | No | Overrides the glossary dataset repo id. Defaults to `GarchenArchive/tts-glossary`. |
 
 
 
 
 
73
  | `ELEVENLABS_API_KEY` | Yes | ElevenLabs synthesis calls. |
74
  | `HF_TOKEN` | Yes, to save/remove glossary entries | Write access to the `tts-glossary` dataset repo. Reads fall back to a local cache (or built-in defaults) without it. |
75
  | `GLOSSARY_DATASET_REPO` | No | Overrides the glossary dataset repo id. Defaults to `GarchenArchive/tts-glossary`. |
76
+
77
+ ## Docs
78
+
79
+ This app also deploys to a DigitalOcean droplet (shared with TranslationUI), independent of the HF Space. See [docs/](docs/) for configuration, storage design, deployment, and operations.
docs/README.md ADDED
@@ -0,0 +1,15 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # ArchiveAI-TTS docs
2
+
3
+ ArchiveAI-TTS is a Gradio UI around the ElevenLabs API for interpreter audio synthesis β€” no local ML model, purely CPU. Deployed to two independent targets from the same `main` branch:
4
+
5
+ - **HuggingFace Space** (`GarchenArchive/ArchiveAI-TTS`) β€” the original deployment, still live.
6
+ - **DigitalOcean droplet** β€” shares the same droplet as TranslationUI (same Caddy instance, same Google-auth gate), on its own container/port/subdomain.
7
+
8
+ Start here, then go deeper:
9
+
10
+ - **[configuration.md](configuration.md)** β€” every environment variable the app reads.
11
+ - **[storage.md](storage.md)** β€” how projects (including generated audio) and the glossary are stored, and why there are two backends.
12
+ - **[deployment.md](deployment.md)** β€” how each target is built and deployed, and what "sharing a droplet with TranslationUI" actually means.
13
+ - **[operations.md](operations.md)** β€” runbook and known gaps.
14
+
15
+ Root-level [README.md](../README.md) covers the app's actual features (Turn Review, Quick Synthesize).
docs/configuration.md ADDED
@@ -0,0 +1,26 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Configuration
2
+
3
+ ## App-level environment variables
4
+
5
+ | Variable | Required | Purpose |
6
+ |---|---|---|
7
+ | `ELEVENLABS_API_KEY` | Yes | Actual TTS synthesis β€” this app has no local model, everything goes through ElevenLabs. |
8
+ | `GEMINI_API_KEY` | Yes (for AI-assisted turn generation) | Used by the turn-generation feature (`app.py`); raises a `gr.Error` if unset when that feature is used. |
9
+ | `HF_TOKEN` | Only on the HF Space, and on DO (for the shared glossary β€” see below) | Selects the HF-dataset backend for project storage when `TTS_DATA_DIR` is unset. Independently, also used by `glossary.py` to sync pronunciation entries regardless of which project-storage backend is active. |
10
+ | `TTS_DATA_DIR` | Only on DO | Root directory for local project/defaults storage. Set to `/data` on the droplet (the mounted volume subdirectory). Unset locally defaults to `data/tts_projects/` (repo-relative, git-ignored). |
11
+ | `TTS_PROJECTS_DATASET_REPO` | No (has a default) | Overrides which HF dataset the HF backend uses for projects. Default: `GarchenArchive/tts-turn-projects`. |
12
+ | `GLOSSARY_DATASET_REPO` | No (has a default) | Overrides which HF dataset the glossary syncs to. Default: `GarchenArchive/tts-glossary`. |
13
+
14
+ ## Infra-level secrets (DO deployment, not read by the app itself)
15
+
16
+ Shared with TranslationUI's droplet β€” see [TranslationUI's configuration.md](../../TranslationUI/docs/configuration.md) for the Google OAuth / Caddy / oauth2-proxy secrets, which are identical here since it's the same auth layer.
17
+
18
+ | Secret | Where it lives | Purpose |
19
+ |---|---|---|
20
+ | `DO_CI_TOKEN` | GitHub Actions secret (this repo) | Registry-only DO API token β€” same credential value as TranslationUI's, added separately here since it's a different GitHub repo. |
21
+ | `DO_SSH_PRIVATE_KEY` | GitHub Actions secret (this repo) | Same droplet SSH key as TranslationUI's. |
22
+ | `DO_DROPLET_HOST` | GitHub Actions variable (not secret) | The shared droplet's IP. |
23
+
24
+ ## Which backend runs where
25
+
26
+ Same pattern as TranslationUI: `_use_local()` in `project_store.py` picks the local-file backend whenever `TTS_DATA_DIR` is set, the HF-dataset backend otherwise (if `HF_TOKEN` is set), local-file as the final fallback. See [storage.md](storage.md).
docs/deployment.md ADDED
@@ -0,0 +1,41 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Deployment
2
+
3
+ Two independent targets, both built from `main`. Same pattern as TranslationUI: separate GitHub Actions workflows, neither edits the other.
4
+
5
+ | | HF Space | DO droplet |
6
+ |---|---|---|
7
+ | Workflow | `.github/workflows/deploy.yml` | `.github/workflows/deploy-do.yml` |
8
+ | Triggers | `workflow_dispatch` + nightly (00:00 UTC, skipped if unchanged) | Same |
9
+ | Target | `huggingface.co/spaces/GarchenArchive/ArchiveAI-TTS` | The droplet shared with TranslationUI, via SSH |
10
+ | Image registry | N/A | DO Container Registry, repo `garchen-archive/archiveai-tts` |
11
+ | Public URL | The Space's own URL | `https://archiveai-tts.64-225-26-230.nip.io` |
12
+ | Access control | HF's private-Space account whitelist | Google OAuth via oauth2-proxy, shared with TranslationUI |
13
+
14
+ ## Sharing a droplet with TranslationUI
15
+
16
+ Both apps are CPU-only and lightweight enough to coexist on one droplet:
17
+
18
+ - **Same droplet, separate containers**: `translationui-pilot` on internal port `7860`, `archiveai-tts-pilot` on `7861`, both bound to `127.0.0.1` only.
19
+ - **Same Caddy instance**, routing by hostname β€” a second site block was added to the existing `/etc/caddy/Caddyfile` for `archiveai-tts.64-225-26-230.nip.io`, reverse-proxying to `127.0.0.1:7861`.
20
+ - **Same oauth2-proxy, same Google OAuth app, same allowlist** β€” one login covers both apps. This required widening oauth2-proxy's cookie scope: `OAUTH2_PROXY_COOKIE_DOMAINS` and `OAUTH2_PROXY_WHITELIST_DOMAINS` were both set to `.64-225-26-230.nip.io` so the session cookie set on one subdomain is valid on the other. (nip.io is on the Public Suffix List at the `nip.io` level, not per-IP, so `.64-225-26-230.nip.io` is a valid single-site cookie scope β€” it doesn't leak across other unrelated nip.io users.)
21
+ - **Same Volume, separate subdirectory**: TranslationUI's SQLite file lives at the volume root; TTS's project files live under `tts/` on the same volume, mounted into its container at `/data` (a different container path than TranslationUI's, but the same physical disk).
22
+ - **Same DO Project** ("HF Migration"), same isolation-token approach as the rest of this migration.
23
+
24
+ ## CI/CD (`deploy-do.yml`)
25
+
26
+ Same shape as TranslationUI's: build β†’ push to `registry.digitalocean.com/garchen-archive/archiveai-tts` β†’ SSH in β†’ pull, stop/remove the old container, start the new one with:
27
+
28
+ - `-p 127.0.0.1:7861:7860`
29
+ - `-v /mnt/hfmigration_translationui_data/tts:/data`
30
+ - `-e TTS_DATA_DIR=/data`
31
+ - `--env-file /root/archiveai-tts.env` (holds `ELEVENLABS_API_KEY`, `GEMINI_API_KEY`, `HF_TOKEN` β€” never passed through CI logs or command args)
32
+
33
+ Reuses the same `DO_CI_TOKEN`/`DO_SSH_PRIVATE_KEY` credential *values* as TranslationUI, added as this repo's own GitHub Actions secrets (a registry-scoped token isn't tied to a specific app, but GitHub secrets are per-repo, so it has to be added twice).
34
+
35
+ ## Manual deploy / debugging
36
+
37
+ ```bash
38
+ ssh -i <hf-migration-do private key> root@<droplet-ip>
39
+ docker ps --filter name=archiveai-tts-pilot
40
+ docker logs archiveai-tts-pilot --tail 50
41
+ ```
docs/operations.md ADDED
@@ -0,0 +1,41 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Operations
2
+
3
+ ## Redeploying
4
+
5
+ - **HF Space**: run the `Deploy to HF Space` workflow (`workflow_dispatch`), or wait for the nightly run.
6
+ - **DO droplet**: run the `Deploy to DO` workflow (`workflow_dispatch`), or wait for the nightly run.
7
+
8
+ Both skip themselves on the nightly schedule if `main` hasn't changed since the last deploy.
9
+
10
+ ## Checking on the DO droplet
11
+
12
+ ```bash
13
+ ssh -i <hf-migration-do private key> root@<droplet-ip>
14
+
15
+ docker ps --filter name=archiveai-tts-pilot
16
+ docker logs archiveai-tts-pilot --tail 50
17
+
18
+ # Shared with TranslationUI β€” check these if auth or routing seems broken,
19
+ # not just this app's own container:
20
+ systemctl status caddy
21
+ docker logs oauth2-proxy --tail 50
22
+ ```
23
+
24
+ ## Rotating secrets
25
+
26
+ | Secret | How |
27
+ |---|---|
28
+ | `ELEVENLABS_API_KEY` / `GEMINI_API_KEY` | Edit `/root/archiveai-tts.env` on the droplet, then restart the container (or redeploy). |
29
+ | `HF_TOKEN` | Same file β€” remember this one also gates glossary sync, not just an unused leftover. |
30
+ | `DO_CI_TOKEN`, `DO_SSH_PRIVATE_KEY`, Google OAuth credentials | Shared with TranslationUI β€” see that repo's `docs/operations.md`. Rotating them there also affects this app, since it's the same droplet/auth layer. |
31
+
32
+ ## Managing the oauth2-proxy allowlist
33
+
34
+ Shared with TranslationUI β€” same file (`/root/oauth2-proxy/authenticated-emails.txt` on the droplet), same restart command (`docker restart oauth2-proxy`). Adding someone there grants access to both apps at once.
35
+
36
+ ## Known gaps
37
+
38
+ - **No automated backup of local project data.** TranslationUI's nightly-cron-to-HF-dataset pattern is the template to reuse here; not yet done.
39
+ - **No staging slot, single droplet, no HA** β€” same acknowledged tradeoffs as TranslationUI, for the same reason (expected usage is a handful of internal users).
40
+ - **No real domain** β€” same `nip.io` placeholder as TranslationUI.
41
+ - **Not yet burned in.** Just stood up; hasn't run in parallel with the HF Space long enough to build confidence before any cutover conversation.
docs/storage.md ADDED
@@ -0,0 +1,39 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Storage: two backends for projects, one shared store for the glossary
2
+
3
+ ## Turn Review projects (`tts/core/project_store.py`)
4
+
5
+ Two backends, auto-selected by `_use_local()`:
6
+
7
+ - **Local files** β€” used whenever `TTS_DATA_DIR` is set (the DO droplet). Same relative layout as the HF backend, rooted at `TTS_DATA_DIR`:
8
+ ```
9
+ {root}/index.json
10
+ {root}/projects/{id}/project.json
11
+ {root}/projects/{id}/turns.json
12
+ {root}/projects/{id}/captions.*
13
+ {root}/projects/{id}/audio/turn_{NN}.mp3
14
+ {root}/defaults.json
15
+ ```
16
+ Plain files, not a database β€” most of what's stored here is generated audio (mp3s), which fits a filesystem far more naturally than database rows. This is a deliberate difference from TranslationUI's SQLite approach, made because this store's data is mostly binary media, not small JSON records.
17
+ - **HF dataset** β€” used when only `HF_TOKEN` is set and no `TTS_DATA_DIR` is given (the HF Space, which has no persistent disk of its own).
18
+
19
+ **Why two backends exist at all**: same reason as TranslationUI β€” both deploy targets build from the same `main` branch, so a local-only rewrite would have silently broken the HF Space's persistence on its next deploy.
20
+
21
+ A project save overwrites the previous save for that `id` β€” it's a single in-progress document, not versioned history. `turns_json_bytes`/`captions_bytes` are only written on the initial save (right after upload); routine autosaves don't rewrite them. A process-level lock (`_SAVE_LOCK`) serializes the index read-modify-write in both backends, since autosaves fire frequently and several sessions can share one process.
22
+
23
+ Every function fails open (`None` / `[]` / `False`, logged warning) if its backend can't be reached.
24
+
25
+ ## Pronunciation glossary (`tts/core/glossary.py`) β€” intentionally shared, not dual-backend
26
+
27
+ Unlike project storage, the glossary was **not** rewritten. It already had the right shape: a local cache file (at `/data/glossary_cache.json` if `/data` exists, else a repo-relative path) plus a best-effort sync to a shared HF dataset (`GarchenArchive/tts-glossary`) β€” reads prefer the Hub, fall back to the local cache, then to hardcoded seed defaults if neither is available.
28
+
29
+ **Decision: DO and the HF Space share this same glossary dataset.** Low-risk (it's just pronunciation substitutions, not user data) and keeps both deployments producing identical pronunciations. This means the DO container needs `HF_TOKEN` set even though it also has `TTS_DATA_DIR` set for local project storage β€” the two env vars serve different, independent purposes here (`TTS_DATA_DIR` controls where *projects* live; `HF_TOKEN` controls whether the *glossary* syncs anywhere beyond its local cache).
30
+
31
+ Mounting the shared Volume at `/data` in the container gives the glossary a persistent cache location for free, with zero code changes β€” it was already written to look for `/data` if present.
32
+
33
+ ## Migrating data between backends
34
+
35
+ No standing migration tooling in the repo β€” the one-time HF-to-local migration (3 projects, all their audio) was done with a throwaway script, not committed. If this needs to happen again, `project_store.save_project()` / `save_turn_audio()` are the safest way to write into the local backend from a migration script, since they guarantee the on-disk layout matches what the running app writes.
36
+
37
+ ## Known gap
38
+
39
+ No automated backup of the local project data yet β€” see [operations.md](operations.md#known-gaps). TranslationUI's nightly HF-dataset backup pattern (see its own docs) is the natural template to reuse here, adapted for a directory tree instead of a single file.