---
license: apache-2.0
tags:
  - voice
  - stt
  - tts
  - sglang
  - vllm
  - arabic
  - blackwell
  - vast
---

# Dual-5090 voice stack (STT + LLM + TTS)

Bootstrap scripts that bring up a known-good Arabic voice stack on **2× RTX 5090**
(Vast.ai / similar). Tuned for Blackwell (sm_120): no FP8 KV / ngram on the LLM,
Moss TTS forced to SDPA, CUDA graphs off, HF cache on `/dev/shm` when disk is tight.

| Service | Port | GPU | Model | Runtime |
|---------|------|-----|-------|---------|
| LLM | `8000` | 0 (shared w/ STT) | `ibm-granite/granite-4.1-8b` | SGLang |
| STT | `8002` | 0 (shared w/ LLM) | `CohereLabs/cohere-transcribe-arabic-07-2026` | vLLM 0.19 |
| TTS | `8003` | 1 (dedicated) | `Rabe3/Moss-Saudi-3` | sglang-omni |

## One-time setup on a new machine

```bash
# 1) HF token (write/read for this private repo; also needed for gated STT)
export HF_TOKEN=hf_...

# 2) Clone this private repo (replace USER with your HF username)
git clone https://USER:$HF_TOKEN@huggingface.co/USER/vast-voice-stack ~/agent-deploy
cd ~/agent-deploy
chmod +x *.sh

# 3) Disk: need ~40G+ free for two CUDA envs, OR leave HF cache on /dev/shm
#    (default in common.sh). Prefer a host volume / larger disk when possible.

# 4) Launch (sequential install — do not parallelize first boot)
./deploy_all.sh
```

Upload from a machine that already has the scripts:

```bash
export HF_TOKEN=hf_...   # token for the target account
./push_to_hf.sh          # creates private USER/vast-voice-stack and uploads
```
Health checks: `curl -sf localhost:8000/health` / `:8002` / `:8003`.

Restart clean:

```bash
./deploy_all.sh restart-clean
```

## Load test (Locust)

```bash
./run_locust_ui.sh --local-only   # or without flag to expose via Vast portal port 10100
```

Custom metrics: `llm_ttfb`, `tts_ttfa`, `stt_ttfb`, `pipe_e2e`. On this layout TTS is
the concurrency ceiling (~5–6 RPS full-WAV), not the LLM.

## What these scripts already fix

- **Disk**: sequential env install, `PIP_NO_CACHE_DIR`, shared `~/sglang-env` for LLM+TTS, HF cache → `/dev/shm/hf_home` when overlay is ~32G.
- **sglang-omni**: not on PyPI — clone + `uv pip install -e` with protobuf override.
- **STT**: transformers from git for `cohere_asr`; prefer local HF snapshot if present.
- **GPU1 budget**: TTS dedicated at `MEM_FRACTION=0.72` (colocated Moss OOM’d at 0.85).
- **GPU0 budget**: STT `MEM_UTIL=0.18` + LLM `MEM_FRACTION=0.68`.
- **LLM on Blackwell**: bf16 KV, no ngram speculative decoding (FP8/ngram caused device asserts).
- **Moss on Blackwell**: patch FA→SDPA in vocoder + audio tokenizer; `cuda_graph: false`.

Load test:

```bash
./run_loadtest.sh          # TTS + pipeline concurrency ramp (stream+pcm)
./run_loadtest.sh pipe     # pipeline only
./run_locust_ui.sh         # interactive UI
```
## Env knobs

| Variable | Default | Meaning |
|----------|---------|---------|
| `HF_TOKEN` | — | Required for gated STT download |
| `MODEL` | per-script | Override model id or local path |
| `PORT` | 8000/8002/8003 | Bind port |
| `MEM_FRACTION` / `MEM_UTIL` | see scripts | GPU memory share |
| `LOG_DIR` | `~/agent-logs` | Server logs |

## Layout

```
common.sh                      # toolchain, HF home, venvs, omni + Blackwell patches
start_stt.sh / start_llm.sh / start_tts.sh
deploy_all.sh
moss_tts_local_blackwell.yaml  # Moss pipeline config for 5090
locustfile.py / run_locust_ui.sh
```

## Notes

- Models are **not** vendored here (~30G). Scripts pull them via Hugging Face Hub.
- `/dev/shm` HF cache is **lost on reboot** — re-download or point `HF_HOME` at a persistent volume.
- TTS returns full WAV (not streaming); latency grows under load even when LLM TTFT stays low.