CLAUDE.md

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

What this is

A self-hosted FastAPI wrapper around the k2-fsa/OmniVoice TTS model, packaged as a HuggingFace Space (Docker SDK, T4 GPU). The model is loaded in-process at startup — there is no proxying to the public Gradio Space. The entire service is one file: app.py.

Primary use case is Urdu / Punjabi synthesis (plus 600+ languages OmniVoice supports), consumed by a downstream voice-agent client that retries 502/503/504 with backoff to absorb cold starts.

Commands

# Run locally (downloads ~2 GB model to HF_HOME on first request)
pip install -r requirements.txt          # torch is NOT here — see below
uvicorn app:app --host 0.0.0.0 --port 7860

# torch must be installed separately to match upstream's CUDA build:
pip install torch==2.8.0 torchaudio==2.8.0 --extra-index-url https://download.pytorch.org/whl/cu128

# Build the Space image
docker build -t omnivoice-tts .

# Smoke-test endpoints
curl localhost:7860/health
curl -X POST localhost:7860/tts -H 'Content-Type: application/json' \
     -d '{"text":"آپ کا شکریہ","language":"Urdu","gender":"Female"}' --output out.wav

There is no test suite, linter config, or build step beyond the Docker image.

Architecture

Everything lives in app.py. Request flow:

1. Model load (module top-level) — OmniVoice.from_pretrained() runs once at import time. SAMPLE_RATE is read from the loaded model. This means any import of app.py triggers the full model download/load; there is no lazy init.
2. TTSRequest (pydantic) — the shared request schema for /tts, /tts/stream, and /ws/tts. /tts/clone uses multipart form fields instead and constructs a TTSRequest internally.
3. Voice-design attribute layer — gender/age/pitch/style/accent/dialect are normalized then composed into a single OmniVoice instruct string:
   • _normalize_label() maps friendly aliases ("female", "young", "low") → canonical bilingual labels ("Female / 女").
   • _attr_part() then picks the English half of each label — except dialects, where it picks the Chinese half (is_dialect=True). This mirrors the upstream omnivoice/cli/demo.py convention; preserve it.
   • A caller-supplied instruct string bypasses this whole layer (raw override).
4. _synthesize() — the single inference path for all endpoints. Returns int16 PCM (1-D ndarray). Voice cloning is the same path with voice_clone_prompt added: either a prebuilt prompt passed in (registry reuse) or one built inline via model.create_voice_clone_prompt() from a ref_audio_path (ad-hoc /tts/clone).
   • Voice registry (clone once, reuse many): POST /voices runs create_voice_clone_prompt() a single time and caches it in _VOICE_CACHE (a thread-safe OrderedDict LRU, lock = _VOICE_LOCK, cap = OMNIVOICE_MAX_VOICES, default 50) under a client-chosen name. Any TTSRequest with voice_id set then reuses the cached prompt across /tts, /tts/stream, and /ws/tts — skipping all reference re-encoding. Unknown voice_id → 404. The cache is in-process/in-memory only (lost on restart). _resolve_voice() does the lookup-or-404 for HTTP endpoints; the WebSocket handler checks inline and replies with an error frame instead.
5. WAV framing — _wav_header() builds a 44-byte int16-mono header; /tts wraps the full buffer, /tts/stream and /ws/tts emit the header then chunk raw PCM (~0.2 s HTTP frames, ~0.5 s WS frames).

Inference is always wrapped in asyncio.to_thread(_synthesize, ...) so the event loop stays responsive — keep new synthesis calls off the main thread the same way.

Conventions and gotchas

• The attribute vocabulary sets (GENDERS, AGES, PITCHES, STYLES, ACCENTS, DIALECTS) must stay in sync with the upstream Gradio demo's "English / 中文" labels. The / (GET root) descriptor advertises these to clients, so changes here are part of the public API.
• Dependency pins are load-bearing. transformers==5.3 is required — older versions break the higgs_audio_v2_tokenizer that omnivoice imports. torch==2.8.0 matches the upstream Space. The Dockerfile installs torch before requirements.txt so other deps don't drag in a different CUDA variant. Don't unpin these casually.
• Config is env-var driven (read at module load): OMNIVOICE_MODEL, OMNIVOICE_DEVICE (auto CPU fallback), OMNIVOICE_LOAD_ASR (0 skips ~1 GB whisper download — but then cloning/registration must supply ref_text), OMNIVOICE_MAX_VOICES (registry LRU cap, default 50), LOG_LEVEL, PORT (7860, the HF Spaces standard).
• HF Spaces persistence: caches go to /data/.cache/... (set via HF_HOME/TORCH_HOME in the Dockerfile); /data is the writeable persistent mount.
• Generation params carry validated bounds: nfe_steps 4–64, guidance 0–4. Higher nfe_steps = better quality, slower. Lowering nfe_steps (e.g. 16) is the main lever for faster responses / better RTF.
• /tts/stream and /ws/tts are pseudo-streaming. The upstream omnivoice library has no incremental decoding — generate() returns the complete audio. Both endpoints call _synthesize fully (blocking until the whole utterance is generated) and then chunk the finished PCM buffer. So they reduce transmission framing latency but do not lower time-to-first-audio or improve RTF. Don't market them as real-time streaming.

Deploying

This repo is the Space. Push to the HF Space remote (huggingface.co/spaces/ebitlogix/omnivoice-tts) and set hardware to t4-small+ in Settings → Hardware. README front-matter (sdk: docker, suggested_hardware: t4-small) configures the Space.