README.md

metadata

title: Open-Source Voice Agent
emoji: 🎤
colorFrom: blue
colorTo: indigo
sdk: docker
app_port: 7860
pinned: false
license: apache-2.0

🎤 Open-Source Voice Agent

End-to-end English voice agent built entirely from open-source HuggingFace models with a streaming overlap pipeline — TTS synthesis for sentence N starts the moment sentence N is detected in the LLM token stream, while the model continues generating sentences N+1, N+2 … in parallel.

---

Models

Stage	Model	Params	Device
VAD	silero-vad	2 MB	CPU
STT	openai/whisper-base.en	74 M	GPU / CPU
LLM	HuggingFaceTB/SmolLM2-1.7B-Instruct	1.7 B	GPU / CPU
TTS	facebook/mms-tts-eng	~430 M	CPU

Lighter CPU-only alternative: swap STT → whisper-tiny.en (39 M) and LLM → SmolLM2-360M-Instruct (360 M) in models.py.

---

Architecture

Browser mic  (16 kHz PCM)
     │
     ▼  WebSocket binary frames
Silero VAD  ──────────────────────► discard noise/silence
     │ speech segment detected
     ▼
Whisper base.en  ──────────────────► transcript text
     │
     ▼
SmolLM2-1.7B  (TextIteratorStreamer)
     │  token stream
     ▼
SentenceBuffer  ──► complete sentence
     │                    │
     │          MMS-TTS-eng  (CPU executor)   ← overlap: LLM still generating!
     │                    │
     │          PCM bytes ──► ws.send_bytes()
     │                    │
     ◄────────────────────┘  repeat until stream ends
     │
     ▼
Browser  AudioQueue  ──► AudioContext playback

Streaming overlap benefit: while the browser plays sentence 1, the server is already synthesising sentence 2. This removes the full TTS latency from the inter-sentence gap, giving noticeably more natural turn-taking.

---

Project structure

sts-pipeline/
├── app.py                      FastAPI server + WebSocket handler
├── models.py                   All model loading & inference (STT/LLM/TTS/VAD)
├── pipeline.py                 Streaming overlap pipeline
├── sentence_buffer.py          Token-stream → sentence boundary detector
├── requirements.txt
├── Dockerfile
└── static/
    ├── index.html              Browser UI
    ├── app.js                  WebSocket client + audio queue
    └── audio-capture-worklet.js  Mic capture @ 16 kHz (AudioWorklet)

---

Local development

# 1. Clone
git clone https://huggingface.co/spaces/<your-user>/voice-agent
cd voice-agent

# 2. Install (GPU)
pip install -r requirements.txt

# 2b. Install (CPU-only)
pip install torch==2.5.1+cpu torchaudio==2.5.1+cpu \
    --index-url https://download.pytorch.org/whl/cpu
pip install -r requirements.txt

# 3. Run
python app.py        # or: uvicorn app:app --port 7860
# Open http://localhost:7860

Models are downloaded automatically on first run (3 GB total) and cached in `/.cache/huggingface/hub`.

---

HuggingFace Spaces deployment

1. Create a new Space → Docker SDK.
2. Push this repo as-is.
3. The Space will build the Docker image and serve on port 7860.
4. For GPU hardware: remove the CPU-only torch lines in Dockerfile and uncomment the plain pip install -r requirements.txt.

---

WebSocket protocol reference

Direction	Frame type	Payload
Client → Server	binary	PCM int16, mono, 16 kHz, 640 B chunks
Client → Server	text	{"type":"start"} or {"type":"stop"}
Server → Client	binary	PCM int16, mono, 16 kHz (TTS audio)
Server → Client	text	{"type":"transcript","text":"..."}
Server → Client	text	{"type":"agent_start"}
Server → Client	text	{"type":"agent_done","text":"...","latency_ms":000}
Server → Client	text	{"type":"error","message":"..."}

---

License

Apache 2.0 — all component models carry their own licenses; see their respective HuggingFace model cards for terms of use.