rohitsar567 Claude Opus 4.7 (1M context) commited on
Commit
748ce54
·
1 Parent(s): 4449d44

fix(voice): KI-041/042 — VAD sensitivity + Live OFF by default + dots on first turn

Browse files

KI-041 — VAD threshold tuned for real-world barge-in. User reported
"speaking over the bot doesn't interrupt" despite KI-030's in-DOM audio
fix. Root cause was VAD insensitivity: rmsThreshold 28 (0-255 byte
scale) was too high for typical conversational volume — VAD didn't
fire, so interruptBotAudio() never ran. Lowered to 18 (catches normal
speech, still rejects breathing / keyboard / room hum) and
speechStartFrames from 5 → 3 (~48ms vs ~80ms; cuts barge-in latency).

KI-042 — Live is OFF by default. User feedback: always-on mic was
"a huge problem — user can't control when to speak, can't control
background noise." Initial state is now grey pill "Voice off — click
to enable always-on". Preference still persists in
localStorage("insurance_live_pref") so returning users who DID opt in
see Live on. Only "on" persists explicitly; default and "off" both
render grey.

Updated copy:
• Pill: "Voice off — click to enable always-on" (was "click to turn
on, or use 🎤")
• Empty-state hero: "You can type below, click 🎤 Push-to-talk for
one voice turn, or turn on Voice for always-on listening with
barge-in" (replaces "Voice is on" copy that suggested always-on
was the default)

KI-038 follow-up — waiting dots edge case. ThinkingDots were only
rendered INSIDE the `messages.length > 0` branch, so on the very first
turn (user just spoke, no message bubble yet) the indicator was
hidden. Now also rendered inside the empty-state branch so users see
"Hearing you…" / "Thinking…" even before their first message lands.

PTT-suspends-Live behavior (KI-026 + KI-027 + KI-029 — already
shipped) is unchanged: clicking the 🎤 button while Voice is on
temporarily flips live.live=false (releases the persistent mic),
records VAD-bounded, submits, then setLive(true) to resume.
PTT-while-Live can never produce duplicate transcripts.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

frontend/src/app/page.tsx CHANGED
@@ -184,18 +184,20 @@ export default function Page() {
184
  // Hands-free continuous loop: when toggled ON, immediately open mic. When
185
  // toggled OFF, close any in-progress recording. The send() function takes
186
  // care of re-opening the mic after each assistant TTS finishes.
187
- // KI-028 (2026-05-14) — Live can now be explicitly turned off via the
188
- // status pill (click the green dot red dot → can click again to bring
189
- // it back, OR just keep using 🎤 push-to-talk). userPrefersLive is the
190
- // user's *intent*, persisted across reloads. live.live is the *actual*
191
- // mic state which PTT temporarily flips to false during a recording
 
192
  // even while userPrefersLive stays true (so Live resumes after PTT).
193
- const [userPrefersLive, setUserPrefersLive] = useState(true);
194
- // Load persisted preference on mount.
 
195
  useEffect(() => {
196
  if (typeof window === "undefined") return;
197
  const pref = localStorage.getItem("insurance_live_pref");
198
- if (pref === "off") setUserPrefersLive(false);
199
  }, []);
200
  // Persist + sync to the live hook whenever the user toggles preference.
201
  useEffect(() => {
@@ -688,7 +690,16 @@ export default function Page() {
688
  : "max-w-6xl w-full mx-auto"
689
  }`}>
690
  {messages.length === 0 ? (
691
- <EmptyState onSuggest={(q) => send(q)} coverage={coverage} t={t} />
 
 
 
 
 
 
 
 
 
692
  ) : (
693
  <>
694
  {/* KI-020 / KI-039 / KI-040 — single Clear-chat control. One
@@ -797,20 +808,20 @@ export default function Page() {
797
  className={`flex items-center gap-1.5 px-2 py-0.5 rounded-full border text-xs font-medium transition cursor-pointer ${
798
  userPrefersLive
799
  ? "border-emerald-300 bg-emerald-50 text-emerald-700 hover:bg-emerald-100 dark:border-emerald-700 dark:bg-emerald-900/30 dark:text-emerald-300"
800
- : "border-rose-300 bg-rose-50 text-rose-700 hover:bg-rose-100 dark:border-rose-700 dark:bg-rose-900/30 dark:text-rose-300"
801
  }`}
802
  title={userPrefersLive
803
- ? "Click to turn voice OFF (you can still use 🎤 push-to-talk)"
804
- : "Click to turn voice back ON (always-on listening + barge-in)"}
805
  >
806
  <span className={`inline-block w-2 h-2 rounded-full ${
807
  userPrefersLive
808
  ? (live.recording ? "bg-red-500 animate-pulse" : "bg-emerald-500 animate-pulse")
809
- : "bg-rose-500"
810
  }`} />
811
  {userPrefersLive
812
  ? (live.recording ? "Listening…" : "Voice on — just speak")
813
- : "Voice off — click to turn on, or use 🎤"}
814
  </button>
815
  ) : (
816
  <span className="text-rose-500 text-xs" title="Allow mic in your browser site settings, or use the 🎤 push-to-talk button">
@@ -1343,10 +1354,12 @@ function EmptyState({ onSuggest, coverage, t }: { onSuggest: (q: string) => void
1343
  <div className="text-xs font-semibold text-[var(--primary)] mb-1">{t("welcome.trust_title")}</div>
1344
  <p className="text-xs text-[var(--muted-foreground)] leading-snug">{t("welcome.trust_body")}</p>
1345
  </div>
1346
- {/* KI-027make it obvious that voice is always-on. */}
1347
  <div className="flex items-center gap-2 max-w-xl mb-6 text-sm text-[var(--muted-foreground)]">
1348
- <span className="inline-block w-2.5 h-2.5 rounded-full bg-emerald-500 animate-pulse" />
1349
- <span><strong className="text-[var(--foreground)]">Voice is on.</strong> Just start speaking — I&apos;m listening. Speak over me to interrupt at any time. Prefer to type? Use the box below.</span>
 
 
1350
  </div>
1351
  <div className="grid grid-cols-1 sm:grid-cols-2 gap-2 w-full max-w-2xl">
1352
  {suggested.map((key, i) => {
 
184
  // Hands-free continuous loop: when toggled ON, immediately open mic. When
185
  // toggled OFF, close any in-progress recording. The send() function takes
186
  // care of re-opening the mic after each assistant TTS finishes.
187
+ // KI-028 + KI-042 (2026-05-14) — Live is now OFF by default on first
188
+ // visit. User feedback: always-on mic was uncontrollable in noisy
189
+ // environments and surprising for first-timers. The user must click the
190
+ // status pill (grey green) to opt in; preference persists across
191
+ // reloads. userPrefersLive = user's *intent*, persisted; live.live =
192
+ // actual mic state, PTT temporarily flips it to false during a recording
193
  // even while userPrefersLive stays true (so Live resumes after PTT).
194
+ const [userPrefersLive, setUserPrefersLive] = useState(false);
195
+ // Load persisted preference on mount. Only "on" persists; default and
196
+ // explicit "off" both render grey.
197
  useEffect(() => {
198
  if (typeof window === "undefined") return;
199
  const pref = localStorage.getItem("insurance_live_pref");
200
+ if (pref === "on") setUserPrefersLive(true);
201
  }, []);
202
  // Persist + sync to the live hook whenever the user toggles preference.
203
  useEffect(() => {
 
690
  : "max-w-6xl w-full mx-auto"
691
  }`}>
692
  {messages.length === 0 ? (
693
+ <>
694
+ <EmptyState onSuggest={(q) => send(q)} coverage={coverage} t={t} />
695
+ {/* KI-038 — dots visible even on the very first turn (no messages
696
+ yet but the bot is hearing you / thinking) */}
697
+ {(busy || voicePhase) && (
698
+ <div className="mt-4">
699
+ <ThinkingDots phase={voicePhase} />
700
+ </div>
701
+ )}
702
+ </>
703
  ) : (
704
  <>
705
  {/* KI-020 / KI-039 / KI-040 — single Clear-chat control. One
 
808
  className={`flex items-center gap-1.5 px-2 py-0.5 rounded-full border text-xs font-medium transition cursor-pointer ${
809
  userPrefersLive
810
  ? "border-emerald-300 bg-emerald-50 text-emerald-700 hover:bg-emerald-100 dark:border-emerald-700 dark:bg-emerald-900/30 dark:text-emerald-300"
811
+ : "border-[var(--border)] text-[var(--muted-foreground)] hover:border-[var(--primary)] hover:text-[var(--primary)]"
812
  }`}
813
  title={userPrefersLive
814
+ ? "Voice on bot is always listening, you can speak over it. Click to turn off."
815
+ : "Click to turn on always-on listening (you can also use 🎤 push-to-talk for one turn)"}
816
  >
817
  <span className={`inline-block w-2 h-2 rounded-full ${
818
  userPrefersLive
819
  ? (live.recording ? "bg-red-500 animate-pulse" : "bg-emerald-500 animate-pulse")
820
+ : "bg-gray-400"
821
  }`} />
822
  {userPrefersLive
823
  ? (live.recording ? "Listening…" : "Voice on — just speak")
824
+ : "Voice off — click to enable always-on"}
825
  </button>
826
  ) : (
827
  <span className="text-rose-500 text-xs" title="Allow mic in your browser site settings, or use the 🎤 push-to-talk button">
 
1354
  <div className="text-xs font-semibold text-[var(--primary)] mb-1">{t("welcome.trust_title")}</div>
1355
  <p className="text-xs text-[var(--muted-foreground)] leading-snug">{t("welcome.trust_body")}</p>
1356
  </div>
1357
+ {/* KI-042voice is OFF by default; user opts in via the pill. */}
1358
  <div className="flex items-center gap-2 max-w-xl mb-6 text-sm text-[var(--muted-foreground)]">
1359
+ <span className="inline-block w-2.5 h-2.5 rounded-full bg-gray-400" />
1360
+ <span>
1361
+ You can <strong className="text-[var(--foreground)]">type</strong> below, click <strong className="text-[var(--foreground)]">🎤 Push-to-talk</strong> for one voice turn, or turn on <strong className="text-[var(--foreground)]">Voice</strong> (the grey pill at the bottom) for always-on listening with barge-in.
1362
+ </span>
1363
  </div>
1364
  <div className="grid grid-cols-1 sm:grid-cols-2 gap-2 w-full max-w-2xl">
1365
  {suggested.map((key, i) => {
frontend/src/lib/useLiveConversation.ts CHANGED
@@ -49,8 +49,16 @@ export type LiveConversationState = {
49
  };
50
 
51
  const DEFAULTS = {
52
- rmsThreshold: 28, // 0-255 byte FFT magnitude floor; tune in browser
53
- speechStartFrames: 5, // ~80 ms of consistent loudness to declare speech
 
 
 
 
 
 
 
 
54
  silenceEndFrames: 40, // ~640 ms of silence to declare utterance end
55
  };
56
 
 
49
  };
50
 
51
  const DEFAULTS = {
52
+ // KI-041 (2026-05-14) sensitivity bumped. Previous threshold 28 was high
53
+ // enough that normal speaking volume in a moderately-quiet room didn't
54
+ // trigger barge-in detection, so users reported "speaking over the bot
55
+ // doesn't interrupt it". 18 catches typical conversational volume reliably
56
+ // while still rejecting room hum / breathing / keyboard clatter.
57
+ rmsThreshold: 18,
58
+ // ~48ms of speech to fire. Previous 5 frames (~80ms) added perceptible
59
+ // latency on barge-in; 3 frames keeps false-positive immunity but cuts
60
+ // the response time by ~32ms.
61
+ speechStartFrames: 3,
62
  silenceEndFrames: 40, // ~640 ms of silence to declare utterance end
63
  };
64