feat(voice): KI-057 — noise-robust Live VAD + flush-on-stop; KI-058 — scope lib/ ignore to venvs

KI-057 — Live mode VAD hardening (useLiveConversation.ts)
═══════════════════════════════════════════════════════════════

User report: "Background noise continues to play a big issue, even random
noise without people speaking or just ambient noise keeps the live listening
on, and it keeps processing on something and not moving on. Also, if an
entire series of things have been said and I turn off the live chat, should
that not auto submit?"

Root causes:
- Static rmsThreshold: 18 triggered on HVAC / fan / traffic noise.
- Once triggered, silenceEndFrames: 40 never accumulated because ambient
kept the meter above threshold → segment never closed.
- tearDown() silently dropped speechBufferRef when Live was toggled off
mid-utterance.

Six layered fixes (all overridable via opts; tuned defaults shipped):
A. Adaptive noise floor — EMA of ambient avg while idle. Effective
threshold = max(rmsThreshold, noise_floor * 2 + 4). HVAC keeps the bar high.
B. Voice-band spectral gate — require ≥35% of FFT energy in bins 2-22
(~190-2150 Hz at 48 kHz). Broadband noise fails this even when loud.
C. speechStartFrames raised 1 → 3 (~48 ms). Single clicks/clacks don't
trigger. KI-044's preroll buffer (300 ms lookback) still catches the
first phoneme.
D. maxUtteranceMs: 18 s hard cap. Force-close stuck segments where
ambient noise prevented silence-end from firing.
E. Post-utterance cooldown (700 ms). Prevents bot TTS attack transient
from re-triggering capture through mic loopback.
F. Flush-on-teardown. Toggling Live OFF while a capture is in progress
now encodes + fires onUtterance once (fire-and-forget) before
clearing buffers — so the user's words still get submitted.

Verification:
- npx tsc --noEmit → 0 errors
- npm run build (Next.js prod) → ✓ Compiled successfully

KI-058 — Scope lib/ gitignore to virtualenv layouts only
═══════════════════════════════════════════════════════════════

Bare `lib/` + `lib64/` (intended for `venv/lib/`) globbed against
`frontend/src/lib/` and made `git add` refuse to add new files there —
even though existing files were already tracked and committing normally.
This caused friction during this very session (KI-057 file edit hit the
"paths are ignored" wall on staging). Tightened to anchored patterns:
`/lib/`, `/lib64/`, `/.venv/lib/`, `/venv/lib/`. Verified
`git check-ignore -v frontend/src/lib/useLiveConversation.ts` now exits 1.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

.gitignore

@@ -18,8 +18,13 @@ dist/
 downloads/
 eggs/
 .eggs/
-lib/
-lib64/
+# KI-058 (2026-05-15) — scope these to virtualenv layouts only. A bare
+# `lib/` matched `frontend/src/lib/` and made `git add` refuse new files
+# in that path even though existing files were already tracked.
+/lib/
+/lib64/
+/.venv/lib/
+/venv/lib/
 parts/
 sdist/
 var/

frontend/src/lib/useLiveConversation.ts

@@ -4,11 +4,46 @@
  * useLiveConversation — full-duplex voice mode with barge-in.
  *
  * KI-044 (2026-05-14) — PCM pre-roll via AudioWorklet.
+ * KI-057 (2026-05-15) — Noise-robust VAD + flush-on-stop.
+ *
+ * Why KI-057 was needed
  * --------------------------------------------------------------------
- * Previous implementation started a MediaRecorder ONLY when VAD declared
- * "speech started" — by which point ~50-80 ms of the first word was
- * already past the mic. Users reported the bot only hearing "ello" / "i
- * am" rather than "hello" / "hi i am".
+ * Real user feedback after KI-044 shipped: "Background noise continues
+ * to play a big issue, even random noise without people speaking or
+ * just ambient noise keeps the live listening on, and it keeps
+ * processing on something and not moving on. Also, if an entire series
+ * of things have been said and I turn off the live chat, should that
+ * not auto submit?"
+ *
+ * Two failure modes:
+ *   1. Static `rmsThreshold: 18` triggered on HVAC / fan / traffic.
+ *      Once triggered, `silenceEndFrames: 40` (~640 ms of silence)
+ *      never accumulated because ambient noise kept the meter above
+ *      threshold — segment never closed, "Hearing you…" stuck on.
+ *   2. Toggling Live OFF mid-utterance ran tearDown() which silently
+ *      dropped `speechBufferRef` — user's words were lost.
+ *
+ * Fixes layered in (defaults — overridable via opts):
+ *   A. Adaptive noise floor. While not recording, EMA the ambient
+ *      energy. Effective threshold = max(noise_floor * 2 + 4,
+ *      cfg.rmsThreshold). HVAC keeps the bar high.
+ *   B. Voice-band spectral gate. Require ≥35% of FFT energy to live
+ *      in bins 2-22 (~190-2150 Hz at 48 kHz) — the voiced-speech band.
+ *      Broadband noise fails this even when loud.
+ *   C. `speechStartFrames: 3` (~48 ms) so a single click/clack
+ *      doesn't open a segment.
+ *   D. Hard cap: `maxUtteranceMs: 18 s`. If a segment runs that long
+ *      without silence-end firing, force-close it. Prevents the
+ *      "noise pinned the meter open forever" state.
+ *   E. Post-utterance cooldown (700 ms). After we close + dispatch a
+ *      segment, suppress new triggers — even with echoCancellation,
+ *      the bot's TTS attack transient sometimes bleeds in.
+ *   F. Flush-on-teardown. If the user toggles Live OFF while a
+ *      capture is in progress and the duration meets minUtteranceMs,
+ *      encode + fire `onUtterance` once before tearing down — so
+ *      whatever they were saying gets submitted.
+ *
+ * KI-044 still applies — see below.
  *
  * Current implementation:
  *   - Single getUserMedia stream + AudioContext stay open while Live is on.
@@ -50,17 +85,32 @@ export type LiveConversationState = {
 };
 
 const DEFAULTS = {
-  // KI-041 — sensitivity bumped to catch normal conversational volume.
+  // KI-041/057 — minimum bar. Effective threshold is max(this, adaptive
+  // noise_floor * 2 + 4). With KI-044's preroll buffer we can afford to
+  // require more frames before declaring speech.
   rmsThreshold: 18,
-  // KI-043/044 — fire on first loud frame (~16 ms) so the preroll buffer
-  // snapshot captures as much pre-trigger audio as possible.
-  speechStartFrames: 1,
+  // KI-057 — 3 consecutive loud frames (~48 ms). Single clicks/clacks
+  // don't open a segment. KI-044's preroll buffer still captures the
+  // first phoneme since we look back 300 ms.
+  speechStartFrames: 3,
   silenceEndFrames: 40, // ~640 ms of silence to declare utterance end
   minUtteranceMs: 400,
   // KI-044 — How much pre-trigger PCM we keep in the rolling buffer.
   // 300 ms is generous; covers the ~80 ms VAD latency + ~100 ms of
   // user onset before the first detectable frame, with margin.
   prerollMs: 300,
+  // KI-057 — hard cap. If silence-end never fires (e.g. continuous
+  // ambient noise pinned the meter open), force-close the segment.
+  maxUtteranceMs: 18000,
+  // KI-057 — suppress new triggers for this long after we close a
+  // segment. Avoids bot's TTS attack transient bleeding through even
+  // with echoCancellation on.
+  postUtteranceCooldownMs: 700,
+  // KI-057 — minimum fraction of total FFT energy that must sit in the
+  // voice band (bins 2-22 ≈ 190-2150 Hz at 48 kHz). Broadband HVAC /
+  // fan / traffic noise typically scores 0.20-0.30; voiced speech
+  // typically scores 0.40-0.70.
+  voiceBandMinProp: 0.35,
 };
 
 // AudioWorklet processor source — inlined as a Blob URL so we don't need
@@ -136,6 +186,10 @@ export function useLiveConversation(opts: LiveConversationOptions): LiveConversa
   const rafIdRef = useRef<number | null>(null);
   const inflightAbortRef = useRef<AbortController | null>(null);
   const recStartTsRef = useRef<number>(0);
+  // KI-057 — adaptive noise floor (EMA of ambient avg while idle).
+  const noiseFloorRef = useRef<number>(0);
+  // KI-057 — gates "did the bot just stop talking?" cooldown.
+  const lastUtteranceEndedAtRef = useRef<number>(0);
 
   const onUtteranceRef = useRef(opts.onUtterance);
   const onSpeechStartRef = useRef(opts.onSpeechStart);
@@ -152,6 +206,9 @@ export function useLiveConversation(opts: LiveConversationOptions): LiveConversa
     silenceEndFrames: opts.silenceEndFrames ?? DEFAULTS.silenceEndFrames,
     minUtteranceMs: DEFAULTS.minUtteranceMs,
     prerollMs: DEFAULTS.prerollMs,
+    maxUtteranceMs: DEFAULTS.maxUtteranceMs,
+    postUtteranceCooldownMs: DEFAULTS.postUtteranceCooldownMs,
+    voiceBandMinProp: DEFAULTS.voiceBandMinProp,
   };
 
   const interruptBotAudio = useCallback(() => {
@@ -178,10 +235,12 @@ export function useLiveConversation(opts: LiveConversationOptions): LiveConversa
   }, []);
 
   // KI-044 — close speech capture: encode WAV, run guards, fire onUtterance.
+  // KI-057 — also anchors the post-utterance cooldown.
   const endSpeechCapture = useCallback(async () => {
     if (!recordingRef.current) return;
     recordingRef.current = false;
     setRecording(false);
+    lastUtteranceEndedAtRef.current = Date.now();
     const durationMs = Date.now() - (recStartTsRef.current || Date.now());
     const chunks = speechBufferRef.current;
     speechBufferRef.current = [];
@@ -225,20 +284,49 @@ export function useLiveConversation(opts: LiveConversationOptions): LiveConversa
   }, [cfg.minUtteranceMs]);
 
   // VAD loop — runs while `live` is true.
+  // KI-057 — adaptive threshold + voice-band gate + max-utterance cap.
   const tickVAD = useCallback(() => {
     if (!analyserRef.current) return;
     const a = analyserRef.current;
     const buf = new Uint8Array(a.frequencyBinCount);
     let loud = 0;
     let quiet = 0;
+    // Voice band: bins 2-22 at fftSize=512 cover ~190-2150 Hz at 48 kHz —
+    // where voiced speech lives. Capped at bin count for safety.
+    const voiceBandStart = 2;
+    const voiceBandEnd = Math.min(22, buf.length - 1);
+
     const loop = () => {
       if (!analyserRef.current) return;
       a.getByteFrequencyData(buf);
+
       let sum = 0;
-      for (let i = 0; i < buf.length; i++) sum += buf[i];
+      let voiceSum = 0;
+      for (let i = 0; i < buf.length; i++) {
+        sum += buf[i];
+        if (i >= voiceBandStart && i <= voiceBandEnd) voiceSum += buf[i];
+      }
       const avg = sum / buf.length;
-
-      if (avg > cfg.rmsThreshold) {
+      const voiceProp = sum > 0 ? voiceSum / sum : 0;
+
+      // KI-057 — adaptive threshold. Floor at cfg.rmsThreshold so
+      // genuinely quiet rooms don't open the gate too low.
+      const effectiveThreshold = Math.max(
+        cfg.rmsThreshold,
+        noiseFloorRef.current * 2.0 + 4,
+      );
+
+      // KI-057 — suppress new triggers right after we closed a segment
+      // (bot's TTS onset can bleed in via the mic loopback).
+      const cooldownActive =
+        Date.now() - lastUtteranceEndedAtRef.current < cfg.postUtteranceCooldownMs;
+
+      const speechLike =
+        avg > effectiveThreshold &&
+        voiceProp >= cfg.voiceBandMinProp &&
+        !cooldownActive;
+
+      if (speechLike) {
         loud++;
         quiet = 0;
         if (loud === cfg.speechStartFrames && !recordingRef.current) {
@@ -253,11 +341,32 @@ export function useLiveConversation(opts: LiveConversationOptions): LiveConversa
       } else {
         quiet++;
         loud = 0;
+        // KI-057 — only learn the noise floor while idle, so ongoing
+        // speech doesn't poison the EMA.
+        if (!recordingRef.current) {
+          noiseFloorRef.current =
+            noiseFloorRef.current === 0
+              ? avg
+              : noiseFloorRef.current * 0.95 + avg * 0.05;
+        }
         if (quiet === cfg.silenceEndFrames && recordingRef.current) {
           void endSpeechCapture();
         }
       }
 
+      // KI-057 — max-utterance cap. If recording has run too long
+      // without silence-end firing, force-close it. Prevents the
+      // "noise pinned the meter open" stuck state.
+      if (
+        recordingRef.current &&
+        recStartTsRef.current > 0 &&
+        Date.now() - recStartTsRef.current > cfg.maxUtteranceMs
+      ) {
+        // eslint-disable-next-line no-console
+        console.debug("[live-mode] force-closing at max-utterance cap");
+        void endSpeechCapture();
+      }
+
       rafIdRef.current = requestAnimationFrame(loop);
     };
     rafIdRef.current = requestAnimationFrame(loop);
@@ -265,6 +374,9 @@ export function useLiveConversation(opts: LiveConversationOptions): LiveConversa
     cfg.rmsThreshold,
     cfg.silenceEndFrames,
     cfg.speechStartFrames,
+    cfg.maxUtteranceMs,
+    cfg.postUtteranceCooldownMs,
+    cfg.voiceBandMinProp,
     interruptBotAudio,
     beginSpeechCapture,
     endSpeechCapture,
@@ -278,10 +390,44 @@ export function useLiveConversation(opts: LiveConversationOptions): LiveConversa
         cancelAnimationFrame(rafIdRef.current);
         rafIdRef.current = null;
       }
-      // Drop any pending capture without firing onUtterance
+      // KI-057 — flush a mid-utterance capture before dropping refs.
+      // If the user toggled Live OFF while speaking, encode + fire
+      // onUtterance once (fire-and-forget) so their words still land.
+      if (recordingRef.current && speechBufferRef.current.length > 0) {
+        const durationMs = Date.now() - (recStartTsRef.current || Date.now());
+        if (durationMs >= DEFAULTS.minUtteranceMs) {
+          let total = 0;
+          for (const c of speechBufferRef.current) total += c.length;
+          const merged = new Float32Array(total);
+          let off = 0;
+          for (const c of speechBufferRef.current) {
+            merged.set(c, off);
+            off += c.length;
+          }
+          const wav = encodeWAV(merged, sampleRateRef.current);
+          if (wav.size >= 3000) {
+            const handler = onUtteranceRef.current;
+            const abort = new AbortController();
+            // Fire-and-forget. The page handler is independent of Live
+            // being on, so the response will still render in the chat
+            // pane after teardown completes.
+            try {
+              handler(wav, abort).catch((e) => {
+                const name = (e as { name?: string })?.name;
+                if (name !== "AbortError") {
+                  // eslint-disable-next-line no-console
+                  console.error("[live-mode] flush-on-stop failed:", e);
+                }
+              });
+            } catch {}
+          }
+        }
+      }
       recordingRef.current = false;
       speechBufferRef.current = [];
       prerollRef.current = [];
+      noiseFloorRef.current = 0;
+      recStartTsRef.current = 0;
       if (workletRef.current) {
         try { workletRef.current.disconnect(); } catch {}
         workletRef.current = null;