---
title: Open-Source Voice Agent
emoji: 🎤
colorFrom: blue
colorTo: indigo
sdk: docker
app_port: 7860
pinned: false
license: apache-2.0
---

# 🎤 Open-Source Voice Agent

End-to-end English voice agent built entirely from open-source HuggingFace
models with a **streaming overlap pipeline** — TTS synthesis for sentence N
starts the moment sentence N is detected in the LLM token stream, while the
model continues generating sentences N+1, N+2 … in parallel.

---

## Models

| Stage | Model | Params | Device |
|-------|-------|--------|--------|
| VAD   | [silero-vad](https://github.com/snakers4/silero-vad) | 2 MB | CPU |
| STT   | [openai/whisper-base.en](https://huggingface.co/openai/whisper-base.en) | 74 M | GPU / CPU |
| LLM   | [HuggingFaceTB/SmolLM2-1.7B-Instruct](https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B-Instruct) | 1.7 B | GPU / CPU |
| TTS   | [facebook/mms-tts-eng](https://huggingface.co/facebook/mms-tts-eng) | ~430 M | CPU |

> **Lighter CPU-only alternative**: swap STT → `whisper-tiny.en` (39 M) and
> LLM → `SmolLM2-360M-Instruct` (360 M) in `models.py`.

---

## Architecture

```
Browser mic  (16 kHz PCM)
     │
     ▼  WebSocket binary frames
Silero VAD  ──────────────────────► discard noise/silence
     │ speech segment detected
     ▼
Whisper base.en  ──────────────────► transcript text
     │
     ▼
SmolLM2-1.7B  (TextIteratorStreamer)
     │  token stream
     ▼
SentenceBuffer  ──► complete sentence
     │                    │
     │          MMS-TTS-eng  (CPU executor)   ← overlap: LLM still generating!
     │                    │
     │          PCM bytes ──► ws.send_bytes()
     │                    │
     ◄────────────────────┘  repeat until stream ends
     │
     ▼
Browser  AudioQueue  ──► AudioContext playback
```

**Streaming overlap benefit**: while the browser plays sentence 1, the server
is already synthesising sentence 2. This removes the full TTS latency from the
inter-sentence gap, giving noticeably more natural turn-taking.

---

## Project structure

```
sts-pipeline/
├── app.py                      FastAPI server + WebSocket handler
├── models.py                   All model loading & inference (STT/LLM/TTS/VAD)
├── pipeline.py                 Streaming overlap pipeline
├── sentence_buffer.py          Token-stream → sentence boundary detector
├── requirements.txt
├── Dockerfile
└── static/
    ├── index.html              Browser UI
    ├── app.js                  WebSocket client + audio queue
    └── audio-capture-worklet.js  Mic capture @ 16 kHz (AudioWorklet)
```

---

## Local development

```bash
# 1. Clone
git clone https://huggingface.co/spaces/<your-user>/voice-agent
cd voice-agent

# 2. Install (GPU)
pip install -r requirements.txt

# 2b. Install (CPU-only)
pip install torch==2.5.1+cpu torchaudio==2.5.1+cpu \
    --index-url https://download.pytorch.org/whl/cpu
pip install -r requirements.txt

# 3. Run
python app.py        # or: uvicorn app:app --port 7860
# Open http://localhost:7860
```

Models are downloaded automatically on first run (~3 GB total) and cached in
`~/.cache/huggingface/hub`.

---

## HuggingFace Spaces deployment

1. Create a new Space → **Docker** SDK.
2. Push this repo as-is.
3. The Space will build the Docker image and serve on port 7860.
4. For GPU hardware: remove the CPU-only torch lines in `Dockerfile` and
   uncomment the plain `pip install -r requirements.txt`.

---

## WebSocket protocol reference

| Direction | Frame type | Payload |
|-----------|-----------|---------|
| Client → Server | binary | PCM int16, mono, 16 kHz, 640 B chunks |
| Client → Server | text | `{"type":"start"}` or `{"type":"stop"}` |
| Server → Client | binary | PCM int16, mono, 16 kHz (TTS audio) |
| Server → Client | text | `{"type":"transcript","text":"..."}` |
| Server → Client | text | `{"type":"agent_start"}` |
| Server → Client | text | `{"type":"agent_done","text":"...","latency_ms":000}` |
| Server → Client | text | `{"type":"error","message":"..."}` |

---

## License

Apache 2.0 — all component models carry their own licenses; see their
respective HuggingFace model cards for terms of use.