README.md

metadata

title: Hugging Face Voice - Inkling
emoji: 🎙️
colorFrom: indigo
colorTo: purple
sdk: docker
app_port: 7860
pinned: false
short_description: Talk to Inkling with faster-qwen3tts voice replies
hf_oauth: true

Realtime Voice Demo — Inkling

Browser voice-chat UI for the huggingface/speech-to-speech backend, speaking the OpenAI Realtime GA protocol over WebSocket (the deployed Space path) or WebRTC (env-pinned direct deployments only).

How it works

1. App POSTs <lb_url>/session (empty JSON body).
2. The LB picks a ready compute (round-robin) and returns:

   {
     "session_id": "...",
     "websocket_url": "wss://<compute>/v1/realtime",
     "connect_url": "wss://<compute>/v1/realtime?session_token=<JWT>",
     "session_token": "<JWT>",
     "pending_timeout_s": 60
   }
   

3. App opens a WebSocket directly on connect_url (no rewrite to https://; unlike the WebRTC client which POSTs an SDP offer).
4. Server pushes session.created on connect. Client replies with session.update (OpenAI Realtime GA schema: session.audio.input, session.audio.output, session.output_modalities).
5. Client streams mic audio as PCM16 16 kHz mono base64 chunks (input_audio_buffer.append, one frame every ~40 ms).
6. Server pushes response.output_audio.delta (PCM16 24 kHz mono base64) and transcript deltas.

The backend exposes one concurrent session per compute (same as WebRTC mode); the LB pins the session via a signed session_token.

WebRTC transport

With SPEECH_TO_SPEECH_URL set, Settings → Transport offers WebRTC as an alternative to WebSocket. The browser sends its SDP offer through the same-origin /api/calls proxy; negotiated audio and the OpenAI Realtime event data channel then flow directly between browser and backend.

The proxy forwards only to the env-pinned URL, never to a client-supplied target. This is why user-typed URLs and load-balancer mode remain WebSocket-only. Set RTC_ICE_SERVERS on this app and SPEECH_TO_SPEECH_ICE_SERVERS on the backend when remote connections need STUN/TURN.

The WebSocket-only noise gate is hidden during WebRTC calls. Camera snapshots are also re-encoded to fit the data-channel message limit.

Backend requirement

The backend must be installed with the webrtc extra to serve POST /v1/realtime/calls; otherwise WebRTC responds with 501. The WebSocket route remains /v1/realtime.

Smoke-test from the shell:

LB="https://kaa1l6rplzb1gg3y.us-east-1.aws.endpoints.huggingface.cloud"
curl -X POST "$LB/session" -H "Content-Type: application/json" -d '{}'
# -> { "connect_url": "wss://<compute>/v1/realtime?session_token=..." }
# Feed connect_url into a wscat / websocat and you should get a
# session.created event back immediately.

Tools

The assistant can call two tools mid-conversation (toggle them from the Tools button, top-right):

• Web search — Google results via Serper.dev, proxied server-side so the key never reaches the browser. Set SERPER_API_KEY as a Space secret. Without it, the tool is disabled unless the user pastes their own key in the Tools panel.
• Camera — while enabled, a live self-view shows bottom-left; when the model calls the tool, the current frame is sent to the vision-language model so it can see what you're showing it.

Connecting to a backend

Three modes, picked by env (/api/config tells the client which one is active):

• SPEECH_TO_SPEECH_URL env — highest priority. The browser connects directly to this realtime WebSocket URL; it's shown read-only in Settings. Setting it disables the load-balancer logic entirely (no /api/session proxy, no queue, no metering, no sign-in). Unlike the LB address it is not a secret. This is the only mode in which WebRTC can be selected.
• LOAD_BALANCER_URL env — the original flow: the browser POSTs the same-origin /api/session proxy, the server forwards to the LB, and the browser dials the per-session compute URL the LB hands back. The LB address never reaches the browser; the Settings URL field is hidden. When a visitor signs in with Hugging Face, the proxy also forwards their OAuth access token to the LB through X-Reachy-Mini-Authorization, matching the Reachy Mini client. Anonymous requests carry no token, and the token is never returned to browser JavaScript. Set REQUIRE_LOGIN=true to reject anonymous session allocation and prompt visitors to sign in before starting.
• Neither — Settings → Speech-to-speech server URL: paste a full connect_url (wss://host/v1/realtime?...) or a bare host like localhost:8080 (the app adds /v1/realtime), and the browser connects to it directly.

SPEECH_TO_SPEECH_URL	LOAD_BALANCER_URL	SPACE_ID	Connection	URL field	Transport	Metering
✅	any	any	direct → pinned URL	visible, locked	WS or WebRTC	off
–	✅	✅	LB proxy	hidden	WS only	on
–	✅	–	LB proxy	hidden	WS only	off
–	–	any	direct → user URL	editable	WS only	off

Settings → Restart reconnects with the current voice, instructions and URL.

Usage limits

Conversation time is metered per UTC day by sign-in tier (see limiter.py / auth.py), but only on the deployed Space — metering turns on only when BOTH LOAD_BALANCER_URL and SPACE_ID (injected automatically by the HF Space runtime) are present. Running locally — even with LOAD_BALANCER_URL exported — leaves the app unmetered. Tunable via env:

Env	Default	What
LIMIT_ANON_SEC	300	Daily seconds for anonymous visitors (5 min)
LIMIT_FREE_SEC	600	Daily seconds for signed-in non-PRO users (10 min)
REQUIRE_LOGIN	unset	Set to true to require HF sign-in before allocating or claiming a session
UNLIMITED_ORGS	(adds to defaults)	Extra HF org names whose members get unlimited usage, like PRO
USAGE_HASH_SECRET	(random)	HMAC secret for hashing identity keys + signing the anon cookie

PRO members are always unlimited. Members of cerebras, HuggingFaceM4, smolagents, and pollen-robotics are unlimited out of the box (shown as "Team", not "PRO"); set UNLIMITED_ORGS=my-team to add more. Matched case-insensitively against the user's organisations from HF OAuth.

Run locally

The app is now a small FastAPI server (it serves the front-end and the search proxy from one container).

pip install -r requirements.txt
export SERPER_API_KEY=...          # optional; web search is disabled without it
export SPEECH_TO_SPEECH_URL=...    # optional; pin a direct s2s server URL (overrides the LB)
export LOAD_BALANCER_URL=...       # optional; session-proxy flow (set a URL in Settings otherwise)
uvicorn server:app --reload --port 7860
# or, matching production: docker build -t s2s . && docker run -p 7860:7860 -e SERPER_API_KEY=... -e LOAD_BALANCER_URL=... s2s

Then open http://localhost:7860/, click the orb, allow the mic, talk.

Browsers require HTTPS or localhost for getUserMedia() (mic + camera). 127.0.0.1 and localhost both work; plain http://192.168.x.y does NOT.

Settings (stored in localStorage)

Key	What
Speech-to-speech server URL	Direct realtime WebSocket URL (hidden/locked when deployment-owned)
Transport	WebSocket (default) or WebRTC; selectable only with an env-pinned URL
Microphone	Input device for capture. Applies on the next conversation / Restart.
Speakers	Output device for assistant audio. Chrome/Edge can switch live; other browsers keep the system default.
Voice	Qwen3-TTS speaker name (Aiden, Ryan, Dylan, Eric, Ono_Anna, Serena, Sohee, Uncle_Fu, Vivian)
Instructions	System prompt sent in session.update once the connection opens

LocalStorage keys are namespaced s2s.ws.*, plus s2s.transport and s2s.audio.inputId / s2s.audio.outputId.

Files

File	Role
index.html	Single page, orb + settings modal (identical UI to the WebRTC app)
main.js	State machine, settings, tools, camera, noise-gate UI wiring
ui/chat.js	ChatView: history panel, ephemeral bubbles, transcript/tool streaming
ui/account.js	Account: HF login chip + popover, daily-limit modal
ui/dom.js	Shared helpers: $, escHtml, truncateError, DEBUG
auth.py	HF OAuth + per-request identity (tier, hashed keys)
limiter.py	SQLite per-day talk-time budget (chunked server-clock reservation)
ws/s2s-ws-client.js	WebSocket handshake + OpenAI Realtime GA protocol
rtc/s2s-rtc-client.js	WebRTC sibling: SDP via /api/calls, events over data channel, track audio
ws/codec.js	base64 <-> PCM helpers + transcript extraction (pure)
ws/orb-visualizer.js	OrbVisualiser: FFT bands -> orb CSS custom properties
worklets/mic-capture.js	AudioWorklet: 48 kHz Float32 -> 16 kHz Int16 PCM, posts ~40 ms chunks
worklets/audio-playback.js	AudioWorklet: 24 kHz Float32 ring buffer -> 48 kHz, linear interp, fade in/out
style.css	Orb animations, layout, dark theme (verbatim from the WebRTC app)

Audio pipeline notes

• Input: getUserMedia({ echoCancellation, noiseSuppression, autoGainControl }) feeds the mic-capture worklet at the AudioContext rate. The worklet resamples to 16 kHz (boxcar lowpass + decimation on the 48 -> 16 fast path, linear interpolation fallback for odd rates) and packs Int16 LE.
• Output: response.output_audio.delta decodes to Int16 -> Float32 and is posted to the audio-playback worklet. The worklet maintains a per-context ring buffer, linearly interpolates 24 -> 48, and applies short 32-frame fades on entry/exit to suppress clicks.
• Barge-in: when the server VAD detects user speech mid-response (input_audio_buffer.speech_started while ai-speaking), the client posts { kind: "clear" } to the playback worklet to wipe the queue immediately. The server itself cancels the in-flight response.

Credits

• Backend: huggingface/speech-to-speech
• UI verbatim from amir-tfrere/minimal-conversation-app-s2s-backend (Pollen Robotics × Hugging Face)