docs/realtime_voice.md

Realtime Voice Loop

The realtime path runs the Gradio app as an ASGI app:

• /app serves the UI.
• / redirects to /app.
• /realtime/voice is the browser microphone WebSocket.

The browser sends 16 kHz mono PCM16 chunks, uses local VAD to commit a speech turn after silence, and plays returned audio chunks through an ordered queue.

Local Fixture Smoke

This does not require Modal or external model credentials. It starts the actual ASGI app on a temporary localhost port, opens a real WebSocket, sends synthetic PCM, and verifies transcript, text, and audio events.

.\.runtime-venv\Scripts\python.exe scripts\realtime_ws_smoke.py

Start The App

$env:TIME_MACHINE_ADAPTER_PROFILE="modal"
$env:GRADIO_SERVER_NAME="127.0.0.1"
$env:GRADIO_SERVER_PORT="7860"
.\scripts\run_realtime_app.ps1

Open http://127.0.0.1:7860/app, launch an encounter, then use the Live voice control in the cockpit.

Start Modal Audio

The original command still works if .venv is healthy:

$env:MODAL_PROFILE="manikandanj"
$env:PYTHONUTF8="1"
$env:PYTHONIOENCODING="utf-8"
$env:TTY_COMPATIBLE="0"
.\.venv\Scripts\modal.exe serve scripts\modal_audio.py

In this workspace .venv\Scripts\modal.exe points at a missing Python install. Use this wrapper to prefer the working .runtime-venv Modal CLI:

.\scripts\run_modal_audio_runtime.ps1

When modal serve prints endpoint URLs, set:

$env:TIME_MACHINE_MODAL_STT_URL="https://..."
$env:TIME_MACHINE_MODAL_TTS_URL="https://..."
$env:TIME_MACHINE_LLM_API_KEY="..."
# or $env:TOGETHER_API_KEY="..."

Then start the app with TIME_MACHINE_ADAPTER_PROFILE=modal.

Current Cascade

browser mic/VAD
  -> /realtime/voice WebSocket
  -> STT turn commit
  -> streaming LLM text
  -> sentence-level TTS work queue
  -> ordered browser audio playback

The current Modal STT endpoint still transcribes per committed speech turn. The browser no longer requires clicking record for each turn, and LLM/TTS output is streamed through the app as soon as usable segments are available.