Spaces:
Sleeping
Sleeping
Commit Β·
fc33d54
1
Parent(s): 07e532f
Add docs/ covering configuration, storage, deployment, and operations
Browse filesMirrors TranslationUI's docs structure. Documents the shared-droplet
architecture specifically (same Caddy/oauth2-proxy as TranslationUI,
cookie-domain widened to cover both apps' subdomains) since that's the
part most likely to confuse someone debugging this app in isolation.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- README.md +4 -0
- docs/README.md +15 -0
- docs/configuration.md +26 -0
- docs/deployment.md +41 -0
- docs/operations.md +41 -0
- docs/storage.md +39 -0
README.md
CHANGED
|
@@ -73,3 +73,7 @@ the July Launch content pipeline (e.g. `Dharmakaya` β `Darmakaya`, `Phagmo`
|
|
| 73 |
| `ELEVENLABS_API_KEY` | Yes | ElevenLabs synthesis calls. |
|
| 74 |
| `HF_TOKEN` | Yes, to save/remove glossary entries | Write access to the `tts-glossary` dataset repo. Reads fall back to a local cache (or built-in defaults) without it. |
|
| 75 |
| `GLOSSARY_DATASET_REPO` | No | Overrides the glossary dataset repo id. Defaults to `GarchenArchive/tts-glossary`. |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 73 |
| `ELEVENLABS_API_KEY` | Yes | ElevenLabs synthesis calls. |
|
| 74 |
| `HF_TOKEN` | Yes, to save/remove glossary entries | Write access to the `tts-glossary` dataset repo. Reads fall back to a local cache (or built-in defaults) without it. |
|
| 75 |
| `GLOSSARY_DATASET_REPO` | No | Overrides the glossary dataset repo id. Defaults to `GarchenArchive/tts-glossary`. |
|
| 76 |
+
|
| 77 |
+
## Docs
|
| 78 |
+
|
| 79 |
+
This app also deploys to a DigitalOcean droplet (shared with TranslationUI), independent of the HF Space. See [docs/](docs/) for configuration, storage design, deployment, and operations.
|
docs/README.md
ADDED
|
@@ -0,0 +1,15 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# ArchiveAI-TTS docs
|
| 2 |
+
|
| 3 |
+
ArchiveAI-TTS is a Gradio UI around the ElevenLabs API for interpreter audio synthesis β no local ML model, purely CPU. Deployed to two independent targets from the same `main` branch:
|
| 4 |
+
|
| 5 |
+
- **HuggingFace Space** (`GarchenArchive/ArchiveAI-TTS`) β the original deployment, still live.
|
| 6 |
+
- **DigitalOcean droplet** β shares the same droplet as TranslationUI (same Caddy instance, same Google-auth gate), on its own container/port/subdomain.
|
| 7 |
+
|
| 8 |
+
Start here, then go deeper:
|
| 9 |
+
|
| 10 |
+
- **[configuration.md](configuration.md)** β every environment variable the app reads.
|
| 11 |
+
- **[storage.md](storage.md)** β how projects (including generated audio) and the glossary are stored, and why there are two backends.
|
| 12 |
+
- **[deployment.md](deployment.md)** β how each target is built and deployed, and what "sharing a droplet with TranslationUI" actually means.
|
| 13 |
+
- **[operations.md](operations.md)** β runbook and known gaps.
|
| 14 |
+
|
| 15 |
+
Root-level [README.md](../README.md) covers the app's actual features (Turn Review, Quick Synthesize).
|
docs/configuration.md
ADDED
|
@@ -0,0 +1,26 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Configuration
|
| 2 |
+
|
| 3 |
+
## App-level environment variables
|
| 4 |
+
|
| 5 |
+
| Variable | Required | Purpose |
|
| 6 |
+
|---|---|---|
|
| 7 |
+
| `ELEVENLABS_API_KEY` | Yes | Actual TTS synthesis β this app has no local model, everything goes through ElevenLabs. |
|
| 8 |
+
| `GEMINI_API_KEY` | Yes (for AI-assisted turn generation) | Used by the turn-generation feature (`app.py`); raises a `gr.Error` if unset when that feature is used. |
|
| 9 |
+
| `HF_TOKEN` | Only on the HF Space, and on DO (for the shared glossary β see below) | Selects the HF-dataset backend for project storage when `TTS_DATA_DIR` is unset. Independently, also used by `glossary.py` to sync pronunciation entries regardless of which project-storage backend is active. |
|
| 10 |
+
| `TTS_DATA_DIR` | Only on DO | Root directory for local project/defaults storage. Set to `/data` on the droplet (the mounted volume subdirectory). Unset locally defaults to `data/tts_projects/` (repo-relative, git-ignored). |
|
| 11 |
+
| `TTS_PROJECTS_DATASET_REPO` | No (has a default) | Overrides which HF dataset the HF backend uses for projects. Default: `GarchenArchive/tts-turn-projects`. |
|
| 12 |
+
| `GLOSSARY_DATASET_REPO` | No (has a default) | Overrides which HF dataset the glossary syncs to. Default: `GarchenArchive/tts-glossary`. |
|
| 13 |
+
|
| 14 |
+
## Infra-level secrets (DO deployment, not read by the app itself)
|
| 15 |
+
|
| 16 |
+
Shared with TranslationUI's droplet β see [TranslationUI's configuration.md](../../TranslationUI/docs/configuration.md) for the Google OAuth / Caddy / oauth2-proxy secrets, which are identical here since it's the same auth layer.
|
| 17 |
+
|
| 18 |
+
| Secret | Where it lives | Purpose |
|
| 19 |
+
|---|---|---|
|
| 20 |
+
| `DO_CI_TOKEN` | GitHub Actions secret (this repo) | Registry-only DO API token β same credential value as TranslationUI's, added separately here since it's a different GitHub repo. |
|
| 21 |
+
| `DO_SSH_PRIVATE_KEY` | GitHub Actions secret (this repo) | Same droplet SSH key as TranslationUI's. |
|
| 22 |
+
| `DO_DROPLET_HOST` | GitHub Actions variable (not secret) | The shared droplet's IP. |
|
| 23 |
+
|
| 24 |
+
## Which backend runs where
|
| 25 |
+
|
| 26 |
+
Same pattern as TranslationUI: `_use_local()` in `project_store.py` picks the local-file backend whenever `TTS_DATA_DIR` is set, the HF-dataset backend otherwise (if `HF_TOKEN` is set), local-file as the final fallback. See [storage.md](storage.md).
|
docs/deployment.md
ADDED
|
@@ -0,0 +1,41 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Deployment
|
| 2 |
+
|
| 3 |
+
Two independent targets, both built from `main`. Same pattern as TranslationUI: separate GitHub Actions workflows, neither edits the other.
|
| 4 |
+
|
| 5 |
+
| | HF Space | DO droplet |
|
| 6 |
+
|---|---|---|
|
| 7 |
+
| Workflow | `.github/workflows/deploy.yml` | `.github/workflows/deploy-do.yml` |
|
| 8 |
+
| Triggers | `workflow_dispatch` + nightly (00:00 UTC, skipped if unchanged) | Same |
|
| 9 |
+
| Target | `huggingface.co/spaces/GarchenArchive/ArchiveAI-TTS` | The droplet shared with TranslationUI, via SSH |
|
| 10 |
+
| Image registry | N/A | DO Container Registry, repo `garchen-archive/archiveai-tts` |
|
| 11 |
+
| Public URL | The Space's own URL | `https://archiveai-tts.64-225-26-230.nip.io` |
|
| 12 |
+
| Access control | HF's private-Space account whitelist | Google OAuth via oauth2-proxy, shared with TranslationUI |
|
| 13 |
+
|
| 14 |
+
## Sharing a droplet with TranslationUI
|
| 15 |
+
|
| 16 |
+
Both apps are CPU-only and lightweight enough to coexist on one droplet:
|
| 17 |
+
|
| 18 |
+
- **Same droplet, separate containers**: `translationui-pilot` on internal port `7860`, `archiveai-tts-pilot` on `7861`, both bound to `127.0.0.1` only.
|
| 19 |
+
- **Same Caddy instance**, routing by hostname β a second site block was added to the existing `/etc/caddy/Caddyfile` for `archiveai-tts.64-225-26-230.nip.io`, reverse-proxying to `127.0.0.1:7861`.
|
| 20 |
+
- **Same oauth2-proxy, same Google OAuth app, same allowlist** β one login covers both apps. This required widening oauth2-proxy's cookie scope: `OAUTH2_PROXY_COOKIE_DOMAINS` and `OAUTH2_PROXY_WHITELIST_DOMAINS` were both set to `.64-225-26-230.nip.io` so the session cookie set on one subdomain is valid on the other. (nip.io is on the Public Suffix List at the `nip.io` level, not per-IP, so `.64-225-26-230.nip.io` is a valid single-site cookie scope β it doesn't leak across other unrelated nip.io users.)
|
| 21 |
+
- **Same Volume, separate subdirectory**: TranslationUI's SQLite file lives at the volume root; TTS's project files live under `tts/` on the same volume, mounted into its container at `/data` (a different container path than TranslationUI's, but the same physical disk).
|
| 22 |
+
- **Same DO Project** ("HF Migration"), same isolation-token approach as the rest of this migration.
|
| 23 |
+
|
| 24 |
+
## CI/CD (`deploy-do.yml`)
|
| 25 |
+
|
| 26 |
+
Same shape as TranslationUI's: build β push to `registry.digitalocean.com/garchen-archive/archiveai-tts` β SSH in β pull, stop/remove the old container, start the new one with:
|
| 27 |
+
|
| 28 |
+
- `-p 127.0.0.1:7861:7860`
|
| 29 |
+
- `-v /mnt/hfmigration_translationui_data/tts:/data`
|
| 30 |
+
- `-e TTS_DATA_DIR=/data`
|
| 31 |
+
- `--env-file /root/archiveai-tts.env` (holds `ELEVENLABS_API_KEY`, `GEMINI_API_KEY`, `HF_TOKEN` β never passed through CI logs or command args)
|
| 32 |
+
|
| 33 |
+
Reuses the same `DO_CI_TOKEN`/`DO_SSH_PRIVATE_KEY` credential *values* as TranslationUI, added as this repo's own GitHub Actions secrets (a registry-scoped token isn't tied to a specific app, but GitHub secrets are per-repo, so it has to be added twice).
|
| 34 |
+
|
| 35 |
+
## Manual deploy / debugging
|
| 36 |
+
|
| 37 |
+
```bash
|
| 38 |
+
ssh -i <hf-migration-do private key> root@<droplet-ip>
|
| 39 |
+
docker ps --filter name=archiveai-tts-pilot
|
| 40 |
+
docker logs archiveai-tts-pilot --tail 50
|
| 41 |
+
```
|
docs/operations.md
ADDED
|
@@ -0,0 +1,41 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Operations
|
| 2 |
+
|
| 3 |
+
## Redeploying
|
| 4 |
+
|
| 5 |
+
- **HF Space**: run the `Deploy to HF Space` workflow (`workflow_dispatch`), or wait for the nightly run.
|
| 6 |
+
- **DO droplet**: run the `Deploy to DO` workflow (`workflow_dispatch`), or wait for the nightly run.
|
| 7 |
+
|
| 8 |
+
Both skip themselves on the nightly schedule if `main` hasn't changed since the last deploy.
|
| 9 |
+
|
| 10 |
+
## Checking on the DO droplet
|
| 11 |
+
|
| 12 |
+
```bash
|
| 13 |
+
ssh -i <hf-migration-do private key> root@<droplet-ip>
|
| 14 |
+
|
| 15 |
+
docker ps --filter name=archiveai-tts-pilot
|
| 16 |
+
docker logs archiveai-tts-pilot --tail 50
|
| 17 |
+
|
| 18 |
+
# Shared with TranslationUI β check these if auth or routing seems broken,
|
| 19 |
+
# not just this app's own container:
|
| 20 |
+
systemctl status caddy
|
| 21 |
+
docker logs oauth2-proxy --tail 50
|
| 22 |
+
```
|
| 23 |
+
|
| 24 |
+
## Rotating secrets
|
| 25 |
+
|
| 26 |
+
| Secret | How |
|
| 27 |
+
|---|---|
|
| 28 |
+
| `ELEVENLABS_API_KEY` / `GEMINI_API_KEY` | Edit `/root/archiveai-tts.env` on the droplet, then restart the container (or redeploy). |
|
| 29 |
+
| `HF_TOKEN` | Same file β remember this one also gates glossary sync, not just an unused leftover. |
|
| 30 |
+
| `DO_CI_TOKEN`, `DO_SSH_PRIVATE_KEY`, Google OAuth credentials | Shared with TranslationUI β see that repo's `docs/operations.md`. Rotating them there also affects this app, since it's the same droplet/auth layer. |
|
| 31 |
+
|
| 32 |
+
## Managing the oauth2-proxy allowlist
|
| 33 |
+
|
| 34 |
+
Shared with TranslationUI β same file (`/root/oauth2-proxy/authenticated-emails.txt` on the droplet), same restart command (`docker restart oauth2-proxy`). Adding someone there grants access to both apps at once.
|
| 35 |
+
|
| 36 |
+
## Known gaps
|
| 37 |
+
|
| 38 |
+
- **No automated backup of local project data.** TranslationUI's nightly-cron-to-HF-dataset pattern is the template to reuse here; not yet done.
|
| 39 |
+
- **No staging slot, single droplet, no HA** β same acknowledged tradeoffs as TranslationUI, for the same reason (expected usage is a handful of internal users).
|
| 40 |
+
- **No real domain** β same `nip.io` placeholder as TranslationUI.
|
| 41 |
+
- **Not yet burned in.** Just stood up; hasn't run in parallel with the HF Space long enough to build confidence before any cutover conversation.
|
docs/storage.md
ADDED
|
@@ -0,0 +1,39 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Storage: two backends for projects, one shared store for the glossary
|
| 2 |
+
|
| 3 |
+
## Turn Review projects (`tts/core/project_store.py`)
|
| 4 |
+
|
| 5 |
+
Two backends, auto-selected by `_use_local()`:
|
| 6 |
+
|
| 7 |
+
- **Local files** β used whenever `TTS_DATA_DIR` is set (the DO droplet). Same relative layout as the HF backend, rooted at `TTS_DATA_DIR`:
|
| 8 |
+
```
|
| 9 |
+
{root}/index.json
|
| 10 |
+
{root}/projects/{id}/project.json
|
| 11 |
+
{root}/projects/{id}/turns.json
|
| 12 |
+
{root}/projects/{id}/captions.*
|
| 13 |
+
{root}/projects/{id}/audio/turn_{NN}.mp3
|
| 14 |
+
{root}/defaults.json
|
| 15 |
+
```
|
| 16 |
+
Plain files, not a database β most of what's stored here is generated audio (mp3s), which fits a filesystem far more naturally than database rows. This is a deliberate difference from TranslationUI's SQLite approach, made because this store's data is mostly binary media, not small JSON records.
|
| 17 |
+
- **HF dataset** β used when only `HF_TOKEN` is set and no `TTS_DATA_DIR` is given (the HF Space, which has no persistent disk of its own).
|
| 18 |
+
|
| 19 |
+
**Why two backends exist at all**: same reason as TranslationUI β both deploy targets build from the same `main` branch, so a local-only rewrite would have silently broken the HF Space's persistence on its next deploy.
|
| 20 |
+
|
| 21 |
+
A project save overwrites the previous save for that `id` β it's a single in-progress document, not versioned history. `turns_json_bytes`/`captions_bytes` are only written on the initial save (right after upload); routine autosaves don't rewrite them. A process-level lock (`_SAVE_LOCK`) serializes the index read-modify-write in both backends, since autosaves fire frequently and several sessions can share one process.
|
| 22 |
+
|
| 23 |
+
Every function fails open (`None` / `[]` / `False`, logged warning) if its backend can't be reached.
|
| 24 |
+
|
| 25 |
+
## Pronunciation glossary (`tts/core/glossary.py`) β intentionally shared, not dual-backend
|
| 26 |
+
|
| 27 |
+
Unlike project storage, the glossary was **not** rewritten. It already had the right shape: a local cache file (at `/data/glossary_cache.json` if `/data` exists, else a repo-relative path) plus a best-effort sync to a shared HF dataset (`GarchenArchive/tts-glossary`) β reads prefer the Hub, fall back to the local cache, then to hardcoded seed defaults if neither is available.
|
| 28 |
+
|
| 29 |
+
**Decision: DO and the HF Space share this same glossary dataset.** Low-risk (it's just pronunciation substitutions, not user data) and keeps both deployments producing identical pronunciations. This means the DO container needs `HF_TOKEN` set even though it also has `TTS_DATA_DIR` set for local project storage β the two env vars serve different, independent purposes here (`TTS_DATA_DIR` controls where *projects* live; `HF_TOKEN` controls whether the *glossary* syncs anywhere beyond its local cache).
|
| 30 |
+
|
| 31 |
+
Mounting the shared Volume at `/data` in the container gives the glossary a persistent cache location for free, with zero code changes β it was already written to look for `/data` if present.
|
| 32 |
+
|
| 33 |
+
## Migrating data between backends
|
| 34 |
+
|
| 35 |
+
No standing migration tooling in the repo β the one-time HF-to-local migration (3 projects, all their audio) was done with a throwaway script, not committed. If this needs to happen again, `project_store.save_project()` / `save_turn_audio()` are the safest way to write into the local backend from a migration script, since they guarantee the on-disk layout matches what the running app writes.
|
| 36 |
+
|
| 37 |
+
## Known gap
|
| 38 |
+
|
| 39 |
+
No automated backup of the local project data yet β see [operations.md](operations.md#known-gaps). TranslationUI's nightly HF-dataset backup pattern (see its own docs) is the natural template to reuse here, adapted for a directory tree instead of a single file.
|