VOICE_MODE_PLAN.md

# Voice Mode Plan — make the bot hear like OpenWebUI's browser voice mode

Status: PLAN (nothing here is implemented yet except where marked DONE)
Repo: `T:\reachy-mini\HF_Repos\Reachy_OpenWebUI_v2` → Space `Jacid23/Reachy_OpenWebUI`
Bots: `.203` reachy-wireless (desk), `.204` reachy-mini (Pi 5)
Rule: NO remote daemon/app restarts — the user reboots the bot himself.
After any app update from the dashboard: check camera frames; if dead run
`/venvs/apps_venv/bin/pip install -U reachy-mini==1.9.0` (user or assistant-with-permission) — see memory notes.

---

## 1. The riddle, answered

Browser voice mode, Conduit, and the bot all use the SAME OpenWebUI backend:
batch Whisper STT (`/api/v1/audio/transcriptions`), streaming chat, sentence-level
TTS (`/api/v1/audio/speech`). **None of them are realtime models.** They are all
cascades. The browser and Conduit feel realtime because of three things the bot
lacked — all fixable:

1. **A working VAD with proper hysteresis.** Conduit uses Silero VAD (the `vad`
   Dart package) with a dual-threshold envelope. The bot's Silero ONNX port was
   broken from day one (missing the v5 64-sample context — returned prob ~0.001
   on real speech) so it silently ran on a crude RMS energy gate. FIXED in
   v0.6.5.7 (2026-07-19), verified on-bot: speech now scores mean 0.86.
2. **A clean audio front-end.** Phone/browser mics get OS-level AEC, noise
   suppression, and AGC for free. The bot's mic array is BETTER hardware — an
   XMOS-class DSP with on-chip AEC, beamforming, and AGC — but it is barely
   configured (only `PP_AGCGAIN=5.0` is applied).
3. **Tuned endpointing + latency masking.** Conduit ships proven constants;
   the bot's were guesses made to compensate for the broken VAD.

Conclusion: there is no architectural reason the bot can't feel like browser
voice mode. Torch is NOT required — the fixed ONNX VAD is now provably correct.

## 2. Evidence collected (2026-07-19)

### Conduit's proven VAD envelope
From `T:\reachy-mini\openwebui_related\conduit\lib\features\chat\services\voice_input_service.dart`:

| Constant | Value | Meaning |
|---|---|---|
| sample rate / frame | 16000 / 512 | same as bot |
| positiveSpeechThreshold | **0.6** | prob to ENTER speech |
| negativeSpeechThreshold | **0.35** | prob to STAY in speech (hysteresis!) |
| preSpeechPadFrames | 16 (~512 ms) | pre-roll kept before trigger |
| minSpeechFrames | 8 (~256 ms) | discard shorter blips |
| endSpeechPadFrames | 6 (~192 ms) | audio kept after end |
| redemptionFrames | 4..max (~128 ms+) | silence tolerated before ending |

Controller: `.../voice_mode/chat_voice_mode_controller.dart` (1650 lines) —
pauses mic during assistant speech (no barge-in), batch STT, sentence TTS.

### Bot hardware audio DSP (huge, untapped)
Daemon exposes runtime audio-DSP control:
- `POST http://<bot>:8000/api/audio/config/apply` (ApplyAudioConfigRequest)
- `GET  http://<bot>:8000/api/audio/config/parameter/{name}`
Parameter families found in
`/venvs/mini_daemon/.../reachy_mini/media/audio_control_utils.py`:
`AEC_*` (echo canceller: AECSILENCELEVEL, FILTER_LENGTH, PATHCHANGE, …),
`AEC_FIXEDBEAMS*` (beamforming: azimuth/elevation/gating), `PP_AGCGAIN`, HPF etc.
Currently only `PP_AGCGAIN=5.0` is applied at app start.

### Current bot pipeline state (post today's fixes)
- Silero ONNX VAD fixed (context window) — Space v0.6.5.7.
- Single VAD threshold now 0.3, RMS fallback raised to 0.10 (was falsely
  triggering at 0.014), mic capture `Headset,0` back at 37/60 (62%).
- Echo handling = VAD fully suppressed while TTS plays
  (`_suppress_vad_until` / `_active_pipeline_count` in
  `src/Reachy_OpenWebUI/sub_apps/conversation_app/local/handler.py` step 5)
  → no barge-in, and trailing suppression windows eat the user's next utterance.
- STT observed slow: 4563 ms for 9 s audio on the GB10 (needs investigation —
  browser voice hits the same endpoint; short utterances + warm model are fast).
- Known infra gotchas: SDK-downgrade-on-update, media-stream wedge on remote
  restart (hence the no-restart rule), pipewire masked on both bots.

## 3. Gap analysis

| Piece | Browser/Conduit | Bot today | Gap |
|---|---|---|---|
| VAD model | Silero, working | Silero ONNX, working since v0.6.5.7 | none |
| VAD envelope | dual threshold + pads | single threshold + chunk counts | Phase 1 |
| Mic front-end | OS AEC/NS/AGC | XMOS AEC/beamform barely configured | Phase 2 |
| Echo/barge-in | mic paused during TTS | VAD hard-suppressed during TTS | Phase 3 (can EXCEED them) |
| STT | same backend | same backend, seen slow | Phase 4 |
| Latency masking | call UI feedback | none | Phase 4 |
| Transport | none (local mic) | daemon media stream (fragile) | out of scope; reboot rule |

## 4. The plan

### Phase 1 — Adopt Conduit's VAD envelope — **DONE, shipped v0.6.5.8**
Implemented 2026-07-19: dual-threshold hysteresis in vad.py (enter=threshold,
stay=threshold-0.25 floor 0.15; RMS gate diagnostic-only when model loaded);
defaults now threshold 0.6, onset 3, silence-end 16 (~512ms), min-speech 8.
On-bot verified: speech 127/149 chunks / 3 segments; 4s noise 0 triggers.
NOTE: user's persisted vad_threshold=0.3 overrides the new default — after
updating the app, POST /vad {"vad_threshold": 0.6} (or set in settings UI).

### (original Phase 1 text follows)
File: `src/Reachy_OpenWebUI/sub_apps/conversation_app/vad.py` + `local/handler.py`.
1. Add dual-threshold hysteresis to `SileroVAD.is_speech`: enter at
   `threshold_enter` (default 0.6), remain while prob ≥ `threshold_stay`
   (default 0.35). Keep RMS fallback only as a diagnostic (log when it WOULD
   have fired; do not let it trigger).
2. Map envelope to Conduit values in handler chunking (512 @16 kHz frames):
   pre-roll (`lookback_buffer`) ≥ 16 frames, min speech 8 frames, end pad 6,
   silence-end (redemption) configurable 4–30 frames (expose in settings as
   today's `vad_*_chunks`).
3. Keep everything settable via the existing `/vad` endpoint + settings UI;
   change only the defaults.
4. Bump version, push. User updates bots (then SDK-check ritual).
Acceptance: VAD probe shows silence ≤0.05 prob, speech ≥0.6; no clipped first
syllables; utterance ends within ~0.5 s of stopping; zero triggers from room
noise at normal mic gain (62%).

### Phase 2 — Wake the XMOS front-end — **CORE DONE, shipped v0.6.5.9**
2026-07-19: root cause was the app's own inherited startup config
(`audio/startup_config.py`): MIN_NS/NN 0.8 (suppressor passing 80% of noise)
+ AGC max gain 10x. Now MIN_NS/NN 0.15, AGC gain/max 4.0 — user-verified live:
noise floor collapsed to faint fan hum. HPF already at mode 2 (max).
Remaining Phase 2 items: beamforming (AEC_FIXEDBEAMS*) experiments, verify
persistence after user's reboot+update, STT latency (11s for 9s clip on GB10 —
CPU whisper; fix is OpenWebUI-side GPU STT server or Deepgram).
User's hand-tuned analog mixer sweet spot (card 0, alsamixer, 2026-07-19):
`Headset,0` ≈ −8 dB, `Headset,1` ≈ −22 dB. Persist with `sudo alsactl store`.
Caution: the app's mic-volume slider overwrites Headset,0.

### (original Phase 2 text follows)
Read current values first with `GET /api/audio/config/parameter/{name}`.
1. Inventory: dump all `AEC_*` and `PP_*` current values into a file for
   baseline (script it; read-only).
2. Experiment matrix (apply one at a time via `POST /api/audio/config/apply`,
   verify with the VAD probe logs + recorded STT quality):
   - `AEC_HPFONOFF` on (kill low-frequency rumble/fan)
   - `PP_AGCGAIN` sweep (current 5.0; try 2–8) with capture fixed at 62%
   - `AEC_FIXEDBEAMSONOFF` + azimuth toward the user's usual position
   - AEC on with far-end reference (see Phase 3)
3. Whatever wins: persist by having the app apply it at startup (the app
   already applies `PP_AGCGAIN` — extend that config block; find it via
   `grep -rn PP_AGCGAIN` in the app/daemon startup path) or via daemon config.
Caution: apply-parameter is live but survives ↔ reboot semantics unknown —
verify persistence after the user's next reboot before relying on it.
Acceptance: speech probs at conversation distance ≥0.8; noise floor rms < 0.01
at 62% gain; STT word error noticeably down (subjective A/B is fine).

### Phase 3 — Barge-in — **FIRST CUT SHIPPED v0.6.6.0 (untested on hardware)**
2026-07-19: handler no longer discards mic audio during playback. VAD keeps
running on the live stream; ~320ms sustained speech (BARGE_IN_CHUNKS=10 at
0.6 enter threshold) calls _interrupt_current_response and listening resumes
instantly (lookback keeps the interjection's start). Env knobs:
REACHY_BARGE_IN=0 reverts to half-duplex; REACHY_BARGE_IN_CHUNKS tunes
sensitivity. RISK: if the XMOS AEC leaks our own voice, the bot may
interrupt itself — raise chunks or disable, then investigate AEC far-end
(AEC_AECCONVERGED probe while TTS plays).
Test protocol after update+reboot: (1) normal turn works; (2) talk over the
bot mid-reply → it stops and handles the interjection; (3) stay silent
through a long reply → it must NOT self-interrupt.

### (original Phase 3 text follows)
Today the app deafens itself while speaking (handler step 5). With the XMOS
AEC cancelling the bot's own speaker from the mic signal, we can listen while
talking — which neither browser voice mode nor Conduit does.
1. Verify AEC has a far-end reference on this hardware (speaker loopback):
   check `AEC_NUM_FARENDS`, `AEC_FAR_MIC_INDEX`, `AEC_AECCONVERGED` while TTS
   plays (read-only probes while user runs a conversation).
2. If converged AEC is real: replace the hard `_suppress_vad_until` window
   with "VAD active during playback, but require prob ≥ enter-threshold for N
   consecutive frames (e.g. 10) to interrupt"; on trigger, call the existing
   interrupt path (`_interrupt_current_response`) then treat as new utterance.
3. If AEC is not usable: fall back to half-duplex but shorten the trailing
   suppression (currently eats speech after TTS ends) to ≤300 ms.
Acceptance: user can talk over the bot and it stops and listens (like
ChatGPT realtime); no self-triggering from its own voice.

### Phase 4 — Latency: measure, then mask
CORRECTED 2026-07-19: NO network problem — the running apps on both bots
already use the direct LAN route http://172.30.30.15:3001 (9.9ms), set via
the app UI (UI value overrides the .env OPENWEBUI_URL fallback; do not
diagnose from the .env file — ask the running app: GET :7860/status →
openwebui_url). The .env fallbacks were aligned to :3001 anyway. Other
routes that exist: openweb.sunrisecablema.com → 172.30.30.200 (LAN reverse
proxy) and gb10.sunrisecablema.com (Cloudflare-proxied, WAN) — for external
clients, not the bots. Network is NOT a latency lever; whisper speed is.
1. Investigate the 4.5 s STT: time `POST /api/v1/audio/transcriptions` from
   the bot with a 3 s wav (script exists in session history). If slow on GB10,
   check OpenWebUI's STT engine setting (faster-whisper model size / GPU use)
   — same win applies to every client.
2. Use the ported LatencyTracker (`REACHY_LATENCY_DETAIL=1`) to get per-stage
   numbers for 10 turns; attack the biggest stage only.
3. Masking: on `Speech detected` end (STT upload start), fire an instant
   "listening/thinking" cue — antenna twitch or attitude accent move via the
   existing reactions/moves layer. Cheap and transforms perceived latency.
4. Optional eager-STT: at speech-END, we already upload immediately; consider
   ALSO uploading a partial at 2 s into long utterances so Whisper is warm
   (needs care with OpenWebUI; low priority).

### Phase 5 (optional, the real realtime) — streaming ASR transplant
Port lyon_chatbox cascade's streaming ASR provider layer
(`T:\reachy-mini\HF_Repos\lyon_chatbox\src\lyon_chatbox\cascade\asr\` —
`base_streaming.py`, `deepgram.py`) into conversation_app as an alternative
STT path: audio streams continuously, Deepgram does server-side endpointing
and partials, local VAD leaves the critical path entirely. Partials also feed
the transcript-reactions layer for mid-sentence robot reactions.
Cost: Deepgram API key + cloud dependency for STT only (LLM/TTS stay local).
This is the only phase that changes the architecture; Phases 1–4 likely make
it unnecessary.

## 5. Order of work & effort
1. Phase 1 — one sitting, app-only, ships via Space update.
2. Phase 2 — experiments on `.203` with user present (mic A/B), an afternoon.
3. Phase 4.1/4.3 — quick wins alongside Phase 2.
4. Phase 3 — after Phase 2 confirms AEC; the flagship feature.
5. Phase 5 — only if still unsatisfied.

## 5b. Torch on the bots (if ever needed — user-verified command)
Not required by any current phase (ONNX VAD works). If a phase needs torch
(torch-native Silero, GLiNER entity reactions, torchaudio):
`uv pip install torch==2.5.1 torchvision==0.20.1 torchaudio==2.5.1 --index-url https://download.pytorch.org/whl/cpu`
(run inside /venvs/apps_venv; CPU wheels, aarch64-compatible.)

## 6. Hard-won constraints (do not relearn these)
- Do NOT restart daemon/app remotely; wedges media streams. User reboots.
- Dashboard app updates may downgrade `reachy-mini` to 1.8.0 → camera dies →
  pip-upgrade to 1.9.0 + user restarts.
- pipewire is masked on both bots — leave it masked.
- `_HTML_TAG_RE` in handler preserves `<|...|>` TTS tags (Higgs) — keep it.
- Mic capture control is `Headset,0` on card 0 (0–60 scale); the app's
  mic_volume writes it. 37 (62%) is the sane baseline.
- Journald is volatile on the bots — capture evidence before reboots.