README.md

metadata

title: Voice AI Demo
emoji: 🎙️
colorFrom: indigo
colorTo: blue
sdk: docker
app_port: 8765
pinned: false

Voice AI Demo — Real-Time Speech Assistant

A real-time voice AI assistant built on Alibaba Cloud Model Studio (Bailian). It supports the full pipeline of speech input, LLM reasoning, and speech playback. All three services (ASR, LLM, TTS) are freely configurable via the web UI or config file, making it easy to adapt to different customer deployments.

---

Features

• Real-time voice interaction — Tap the mic to record. The system automatically runs speech recognition, LLM inference, and text-to-speech in sequence.
• Text input — Type a message directly and chat with the LLM.
• Multi-language switching — Ships with English and Yoruba presets. Add more languages in config.json.
• Flexible configuration — ASR, LLM, and TTS endpoints, API keys, and models can all be changed from the Settings panel or config file.
• DashScope auto-detection — When a Bailian MaaS URL is detected, the app automatically switches to the DashScope native API format.
• OpenAI-compatible — Also works with any OpenAI-compatible ASR / LLM / TTS endpoint.

Architecture

User Speech → ASR (qwen3-asr-flash) → Text → LLM (qwen-plus) → Reply → TTS (qwen3-tts-flash) → Audio Playback

Service	Bailian Model	Notes
ASR	qwen3-asr-flash	Speech recognition with auto language detection
LLM	qwen-plus	Large language model with streaming output
TTS	qwen3-tts-flash	Text-to-speech with multiple voice options

---

Quick Start

Prerequisites

• Python 3.10+
• A modern browser (Chrome, Edge, Safari, or Firefox)

Step 1: Install dependencies

cd voice-ai-demo
python3 -m venv venv
source venv/bin/activate        # On Windows: venv\Scripts\activate
pip install -r requirements.txt

Step 2: Configure your model services

Edit config.json and fill in your Bailian MaaS instance details:

{
  "asr": {
    "mode": "api",
    "base_url": "https://<your-instance>.maas.aliyuncs.com/compatible-mode/v1",
    "api_key": "<your-api-key>",
    "model": "qwen3-asr-flash"
  },
  "llm": {
    "base_url": "https://<your-instance>.maas.aliyuncs.com/compatible-mode/v1",
    "api_key": "<your-api-key>",
    "model": "qwen-plus"
  },
  "tts": {
    "base_url": "https://<your-instance>.maas.aliyuncs.com/compatible-mode/v1",
    "api_key": "<your-api-key>",
    "model": "qwen3-tts-flash",
    "voice": "Cherry"
  }
}

Tip: ASR, LLM, and TTS can each point to a different MaaS instance and API key. Just fill in the corresponding fields independently.

Step 3: Run

python3 app.py

The server starts on http://localhost:8765 by default. Open this URL in your browser to begin.

---

Deploying to Hugging Face Spaces (Docker)

This app is a FastAPI server with its own frontend, so it runs on a Space using the Docker SDK (not the Gradio SDK). The included Dockerfile and the YAML front matter at the top of this README are all that's needed.

Step 1: Create the Space

Create a new Space, choose Docker → Blank, then push these files (or upload via the web UI):

app.py  config.json  requirements.txt  Dockerfile  README.md  .gitignore

The sdk: docker and app_port: 8765 lines in this README's front matter tell the Space how to build and route to the app.

Step 2: Add your keys as Secrets — do not commit them

In Settings → Variables and secrets, add your credentials as Secrets. The app reads these at startup and they always override config.json, so real keys never touch the repo.

Simplest setup — one Bailian/DashScope key for everything (two secrets total):

Secret	Value
DASHSCOPE_API_KEY	Your Model Studio key (sk-...) — applied to ASR + LLM + TTS
DASHSCOPE_BASE_URL	Your region's endpoint (see below) — applied to all three services

Pick the base URL for your region:

• International / outside mainland China: https://dashscope-intl.aliyuncs.com/compatible-mode/v1
• Mainland China (Beijing): https://dashscope.aliyuncs.com/compatible-mode/v1

The app auto-detects these DashScope endpoints and uses the native multimodal format for ASR (qwen3-asr-flash) and TTS (qwen3-tts-flash), while the LLM (qwen-plus) uses the OpenAI-compatible chat endpoint.

Per-service override (only if services differ): set any of ASR_BASE_URL / LLM_BASE_URL / TTS_BASE_URL and ASR_API_KEY / LLM_API_KEY / TTS_API_KEY to point a single service somewhere else. A per-service value wins over the shared DASHSCOPE_* value. Optional model/voice overrides: ASR_MODEL, LLM_MODEL, TTS_MODEL, TTS_VOICE.

Per-language ASR (speech-to-text): a language can route its voice input to a different ASR engine — useful because Qwen's qwen3-asr-flash doesn't cover Yoruba, while Whisper does. Set asr_base_url / asr_api_key / asr_model on the language (or via env ASR_<LANGID>_BASE_URL etc.), and it uses that OpenAI-compatible /audio/transcriptions endpoint with the language's asr_lang hint. Yoruba ships pointed at a Whisper model (whisper-large-v3, hint yo); set ASR_YORUBA_BASE_URL to any OpenAI-compatible Whisper endpoint (e.g. https://api.groq.com/openai/v1 or https://api.openai.com/v1) plus ASR_YORUBA_API_KEY. A language with its own asr_base_url never inherits the global Qwen ASR endpoint.

Per-language TTS backend: each language can use its own TTS engine, so e.g. English speaks via Qwen while Yoruba speaks via a self-hosted model. A language uses its own endpoint when it sets tts_format / tts_base_url / tts_api_key (in config.json or via env). Supported tts_format values: dashscope, openai, or custom. The custom format POSTs {text, speed} to the URL verbatim (so an API Gateway invoke URL like .../prod/tts is used exactly as given), sends both Authorization: Bearer and x-api-key when a key is set, and accepts either raw audio (audio/*) or JSON carrying base64 audio (key audio / audio_base64 / data / wav / audio_content). Per-language env vars follow the pattern TTS_<LANGID>_BASE_URL, TTS_<LANGID>_API_KEY, TTS_<LANGID>_FORMAT, TTS_<LANGID>_MODEL, TTS_<LANGID>_VOICE — e.g. TTS_YORUBA_BASE_URL. A custom-format language never inherits the global Qwen URL/key, so a missing value fails safely instead of sending text to the wrong engine.

Leave the placeholder values in config.json as they are — they're scrubbed automatically at load, and the Secrets fill in the real values.

Step 3: Open the Space

Once the build finishes, the app is live. Microphone capture works out of the box because Spaces are served over HTTPS (browsers require HTTPS or localhost for mic access).

Note on the in-app Settings panel: A Space's filesystem is ephemeral, so any keys you type into the Settings panel are session-only and reset on restart/rebuild. Secrets set via env vars are the durable source of truth and are re-applied on every reload. For persistence across restarts beyond env vars, attach Persistent Storage to the Space.

---

Usage

Voice chat — Click the microphone button at the bottom to start recording. Click again to stop. The system handles speech recognition, LLM inference, and speech synthesis automatically.

Text chat — Type a message in the input box at the bottom, then press Enter or click the send button.

Switch language — Click the language buttons in the top bar. Chat history is cleared when switching languages.

Change settings — Click the gear icon in the top-right corner to open the Settings panel. You can modify the ASR / LLM / TTS endpoint, API key, model, and voice directly from the UI. Changes take effect immediately after clicking Save.

---

Configuration Reference

Full field reference for config.json:

Field	Description	Example
asr.mode	ASR mode: api (remote) or local (on-device Whisper)	"api"
asr.base_url	ASR service base URL	Bailian MaaS URL
asr.api_key	ASR API key	sk-xxx
asr.model	ASR model name	"qwen3-asr-flash"
llm.base_url	LLM service base URL	Bailian MaaS URL
llm.api_key	LLM API key	sk-xxx
llm.model	LLM model name	"qwen-plus"
llm.max_tokens	Max output tokens for LLM	512
llm.temperature	LLM sampling temperature	0.7
tts.base_url	TTS service base URL	Bailian MaaS URL
tts.api_key	TTS API key	sk-xxx
tts.model	TTS model name	"qwen3-tts-flash"
tts.voice	TTS voice name	"Cherry"
tts.speed	TTS playback speed multiplier	1.0
languages	Language presets (add or remove freely)	See file
app.host	Server listen address	"0.0.0.0"
app.port	Server listen port	8765

Adding languages — Add a new entry to the languages array in config.json. Each language can have its own asr_lang (ASR language hint), system_prompt (LLM system prompt), and tts_model (TTS model override).

---

DashScope Auto-Detection

When the base URL contains maas.aliyuncs.com, the app automatically switches to the DashScope native API format:

• ASR calls /api/v1/services/aigc/multimodal-generation/generation using the qwen3-asr-flash model, with audio transmitted as a base64 data URI.
• TTS calls the same endpoint using the qwen3-tts-flash model and automatically downloads the audio from the returned temporary URL.

For non-Bailian URLs, the app uses standard OpenAI-compatible endpoints (/audio/transcriptions, /chat/completions, /audio/speech).

---

FAQ

Q: The microphone button does not respond?

Make sure your browser has granted microphone permission. In Chrome, check the lock icon next to the address bar. Microphone access also requires HTTPS or localhost.

Q: Speech recognition returns empty results?

This may be caused by excessive background noise or a recording that is too short. Ensure recordings are at least 1 second long and use the app in a relatively quiet environment.

Q: No audio from TTS?

Check that the voice field in config.json matches a voice supported by your TTS model. The Bailian qwen3-tts-flash model supports voices such as Cherry, Serena, and Ethan.

Q: How do I switch models?

Change the Model field directly in the Settings panel from the web UI, or edit the corresponding field in config.json and restart the server.

---

Project Structure

voice-ai-demo/
├── app.py              # Main application (FastAPI backend + embedded HTML frontend)
├── config.json         # Configuration (endpoints, API keys, languages, etc.)
├── requirements.txt    # Python dependencies
├── Dockerfile          # Container build for Hugging Face Spaces (Docker SDK)
├── .gitignore          # Excludes venv, caches, local .env
└── README.md           # This document (front matter configures the Space)