rohitsar567 Claude Opus 4.7 (1M context) commited on
Commit
cdf5ea5
·
1 Parent(s): 64f55e3

fix(voice): KI-113 + KI-114 + KI-115 — reject background noise, distant speech, mid-thought pauses

Browse files

User reported Live mode triggering on background noise + cutting off
mid-thought when pausing to think. Tightened all four VAD knobs:

KI-113 (background noise rejection):
- rmsThreshold 18 → 26 (~44% higher floor)
- speechStartFrames 3 → 5 (~80ms sustained, no single-spike opens)
- voiceBandMinProp 0.35 → 0.50 (FFT energy must be clearly voiced;
HVAC/traffic scores 0.20-0.30, speech scores 0.40-0.70)

KI-114 (proximity gate / distant speech rejection):
- adaptive multiplier 2.0+4 → 2.5+6
- Distant speech rides just above ambient (high frequencies
attenuate with distance). Close-in speech is 10-20× ambient so
it still clears the wider gap; distant conversation gets rejected.

KI-115 (pause tolerance / don't submit mid-thought):
- silenceEndFrames 120 → 180 (~2s → ~3s)
- Tradeoff: +1s tail latency before bot responds. Accepted to kill
the "submit when I pause to think" UX bug.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

frontend/src/lib/useLiveConversation.ts CHANGED
@@ -87,22 +87,26 @@ export type LiveConversationState = {
87
  };
88
 
89
  const DEFAULTS = {
90
- // KI-041/057minimum bar. Effective threshold is max(this, adaptive
91
- // noise_floor * 2 + 4). With KI-044's preroll buffer we can afford to
92
- // require more frames before declaring speech.
93
- rmsThreshold: 18,
94
- // KI-057 — 3 consecutive loud frames (~48 ms). Single clicks/clacks
95
- // don't open a segment. KI-044's preroll buffer still captures the
96
- // first phoneme since we look back 300 ms.
97
- speechStartFrames: 3,
98
- // KI-060/064 (2026-05-15) — silence-end window tuning.
99
  // v1 (KI-057): 40 (~640 ms) — too tight; users said pause→submit.
100
  // v2 (KI-060): 90 (~1.5 s) — still cut "Hi, I'm looking to buy a
101
  // new insurance ..." before "policy".
102
  // v3 (KI-064): 120 (~2 s) — covers a normal thinking pause between
103
- // phrases. Trade: ~0.5 s extra tail latency before the bot
104
- // responds, accepted by the user.
105
- silenceEndFrames: 120,
 
 
 
 
106
  minUtteranceMs: 400,
107
  // KI-044 — How much pre-trigger PCM we keep in the rolling buffer.
108
  // 300 ms is generous; covers the ~80 ms VAD latency + ~100 ms of
@@ -115,11 +119,12 @@ const DEFAULTS = {
115
  // segment. Avoids bot's TTS attack transient bleeding through even
116
  // with echoCancellation on.
117
  postUtteranceCooldownMs: 700,
118
- // KI-057 — minimum fraction of total FFT energy that must sit in the
119
- // voice band (bins 2-22 ≈ 190-2150 Hz at 48 kHz). Broadband HVAC /
120
- // fan / traffic noise typically scores 0.20-0.30; voiced speech
121
- // typically scores 0.40-0.70.
122
- voiceBandMinProp: 0.35,
 
123
  };
124
 
125
  // AudioWorklet processor source — inlined as a Blob URL so we don't need
@@ -318,11 +323,15 @@ export function useLiveConversation(opts: LiveConversationOptions): LiveConversa
318
  const avg = sum / buf.length;
319
  const voiceProp = sum > 0 ? voiceSum / sum : 0;
320
 
321
- // KI-057 — adaptive threshold. Floor at cfg.rmsThreshold so
322
- // genuinely quiet rooms don't open the gate too low.
 
 
 
 
323
  const effectiveThreshold = Math.max(
324
  cfg.rmsThreshold,
325
- noiseFloorRef.current * 2.0 + 4,
326
  );
327
 
328
  // KI-057 — suppress new triggers right after we closed a segment
 
87
  };
88
 
89
  const DEFAULTS = {
90
+ // KI-113 (2026-05-15) raised from 18 26 to reject ambient background
91
+ // noise (HVAC, traffic, distant chatter). Effective threshold is
92
+ // max(this, adaptive noise_floor * 2.5 + 6) — see KI-114 in the VAD loop.
93
+ rmsThreshold: 26,
94
+ // KI-113raised 3 5 (~80 ms sustained). Single clicks / cutlery /
95
+ // typing transients no longer flip the gate. Preroll buffer (KI-044)
96
+ // still captures the first phoneme via the 300 ms look-back.
97
+ speechStartFrames: 5,
98
+ // KI-060/064/115 (2026-05-15) — silence-end window tuning.
99
  // v1 (KI-057): 40 (~640 ms) — too tight; users said pause→submit.
100
  // v2 (KI-060): 90 (~1.5 s) — still cut "Hi, I'm looking to buy a
101
  // new insurance ..." before "policy".
102
  // v3 (KI-064): 120 (~2 s) — covers a normal thinking pause between
103
+ // phrases.
104
+ // v4 (KI-115): 180 (~3 s) — user reported the 2s window still cut
105
+ // mid-thought pauses ("um", "let me think", etc.). 3 s is the
106
+ // pause length where most speakers genuinely consider the
107
+ // utterance complete. Trade: +1 s tail latency before bot
108
+ // responds, accepted to kill the "submit on pause" UX bug.
109
+ silenceEndFrames: 180,
110
  minUtteranceMs: 400,
111
  // KI-044 — How much pre-trigger PCM we keep in the rolling buffer.
112
  // 300 ms is generous; covers the ~80 ms VAD latency + ~100 ms of
 
119
  // segment. Avoids bot's TTS attack transient bleeding through even
120
  // with echoCancellation on.
121
  postUtteranceCooldownMs: 700,
122
+ // KI-113raised 0.35 → 0.50 minimum fraction of total FFT energy
123
+ // that must sit in the voice band (bins 2-22 ≈ 190-2150 Hz at 48 kHz).
124
+ // Broadband HVAC / fan / traffic noise typically scores 0.20-0.30;
125
+ // voiced speech typically scores 0.40-0.70. Lifting to 0.50 puts the
126
+ // gate squarely above noise and just below the bottom of speech.
127
+ voiceBandMinProp: 0.50,
128
  };
129
 
130
  // AudioWorklet processor source — inlined as a Blob URL so we don't need
 
323
  const avg = sum / buf.length;
324
  const voiceProp = sum > 0 ? voiceSum / sum : 0;
325
 
326
+ // KI-057/114 — adaptive threshold. Floor at cfg.rmsThreshold so
327
+ // genuinely quiet rooms don't open the gate too low. KI-114 raised
328
+ // the multiplier 2.0→2.5 and the offset 4→6 so DISTANT speech
329
+ // (which sits just above ambient because high frequencies attenuate
330
+ // with distance) is rejected. Close-in speech still clears the gate
331
+ // because the speaker's directly-radiated energy is ~10-20× ambient.
332
  const effectiveThreshold = Math.max(
333
  cfg.rmsThreshold,
334
+ noiseFloorRef.current * 2.5 + 6,
335
  );
336
 
337
  // KI-057 — suppress new triggers right after we closed a segment