Spaces:
Running on Zero
Running on Zero
Commit ·
f705b22
1
Parent(s): f937854
Stream guarded stable audio prefixes
Browse files- README.md +21 -12
- live_app.py +337 -111
README.md
CHANGED
|
@@ -28,11 +28,11 @@ RoleForge is a private feasibility lab for directing a fictional voice NPC. This
|
|
| 28 |
|
| 29 |
## Current gate
|
| 30 |
|
| 31 |
-
This phase tests a
|
| 32 |
|
| 33 |
1. Wear headphones and press **Start 120-second live session**; approve microphone access when prompted.
|
| 34 |
2. Wait for **Listening**, then speak naturally without pressing Stop.
|
| 35 |
-
3. A short pause ends the visitor turn;
|
| 36 |
4. Speaking clearly during playback interrupts it and begins a new visitor turn.
|
| 37 |
|
| 38 |
The app retains LFM's multimodal conversation state for up to six turns inside one bounded 120-second listening
|
|
@@ -43,9 +43,11 @@ This is interruptible turn-taking rather than mathematically simultaneous model
|
|
| 43 |
speech can cancel an active reply before the new utterance is processed. A browser AudioWorklet captures continuous
|
| 44 |
raw PCM and a same-origin WebSocket carries input and output packets. Gradio still owns controls, diagnostics, and
|
| 45 |
the bounded ZeroGPU call, but it no longer owns the microphone or audio playback. This avoids both the native
|
| 46 |
-
Record/Waiting state machine and HLS buffering.
|
| 47 |
-
|
| 48 |
-
|
|
|
|
|
|
|
| 49 |
and immediate playback flush on barge-in. Playback epochs reset at reply boundaries so gaps between separate turns
|
| 50 |
are not counted as underruns. Normal session completion stops microphone capture but lets already scheduled reply
|
| 51 |
audio drain before closing WebAudio; manual stop and barge-in remain immediate. Microphone worklet packets are 4096 samples to reduce queue
|
|
@@ -54,20 +56,27 @@ cancellations.
|
|
| 54 |
|
| 55 |
The system turn is exactly LiquidAI's required interleaved-mode instruction: `Respond with interleaved text and
|
| 56 |
audio.` A short, public user-turn instruction pins the assistant's name to Lyra without assigning or inferring any
|
| 57 |
-
visitor identity.
|
|
|
|
|
|
|
| 58 |
Generated text is labeled `Model response text` until a physical test proves it matches the
|
| 59 |
spoken assistant response. Audio-code handling follows LiquidAI's reference implementation: retain the complete code
|
| 60 |
sequence and remove only a confirmed terminal all-2048 end-of-audio frame. The obsolete incremental Mimi decoder and
|
| 61 |
its GPU warm-up are removed. A 600 ms pause closes the visitor turn. Model output runs until its audio EOS or the
|
| 62 |
-
bounded 768-token generation cap so decoded text cannot outrun audible speech on longer replies
|
| 63 |
-
|
| 64 |
-
|
| 65 |
-
|
| 66 |
|
| 67 |
-
On every turn, live generation rejects false assistant identities
|
|
|
|
| 68 |
generic assistant openings and openings that closely match any earlier session response. Two bounded regeneration
|
| 69 |
attempts use progressively broader text sampling; a remaining false-identity output fails closed rather than becoming
|
| 70 |
-
audible.
|
|
|
|
|
|
|
|
|
|
|
|
|
| 71 |
changing the official system prompt or audio-code path.
|
| 72 |
|
| 73 |
## Privacy and safety
|
|
|
|
| 28 |
|
| 29 |
## Current gate
|
| 30 |
|
| 31 |
+
This phase tests a guarded low-latency free-Space approximation to live conversation:
|
| 32 |
|
| 33 |
1. Wear headphones and press **Start 120-second live session**; approve microphone access when prompted.
|
| 34 |
2. Wait for **Listening**, then speak naturally without pressing Stop.
|
| 35 |
+
3. A short pause ends the visitor turn; playback begins once the opening guard and first stable audio prefix are ready.
|
| 36 |
4. Speaking clearly during playback interrupts it and begins a new visitor turn.
|
| 37 |
|
| 38 |
The app retains LFM's multimodal conversation state for up to six turns inside one bounded 120-second listening
|
|
|
|
| 43 |
speech can cancel an active reply before the new utterance is processed. A browser AudioWorklet captures continuous
|
| 44 |
raw PCM and a same-origin WebSocket carries input and output packets. Gradio still owns controls, diagnostics, and
|
| 45 |
the bounded ZeroGPU call, but it no longer owns the microphone or audio playback. This avoids both the native
|
| 46 |
+
Record/Waiting state machine and HLS buffering. During generation, the server decodes growing audio-code prefixes,
|
| 47 |
+
holds back one unstable 80 ms frame, and emits only samples proven stable by the preceding physical A/B test. At EOS it
|
| 48 |
+
decodes the clean whole response once and emits only the not-yet-played reference tail. A prefix-decode exception falls
|
| 49 |
+
back automatically to the clean whole-response path. PCM is divided into packets of at most 160 ms and the browser
|
| 50 |
+
schedules them sample-exactly through WebAudio with an adaptive 180 ms lead, a bounded server buffer,
|
| 51 |
and immediate playback flush on barge-in. Playback epochs reset at reply boundaries so gaps between separate turns
|
| 52 |
are not counted as underruns. Normal session completion stops microphone capture but lets already scheduled reply
|
| 53 |
audio drain before closing WebAudio; manual stop and barge-in remain immediate. Microphone worklet packets are 4096 samples to reduce queue
|
|
|
|
| 56 |
|
| 57 |
The system turn is exactly LiquidAI's required interleaved-mode instruction: `Respond with interleaved text and
|
| 58 |
audio.` A short, public user-turn instruction pins the assistant's name to Lyra without assigning or inferring any
|
| 59 |
+
fixed visitor identity. It states that personal names heard in visitor audio belong to the speaker, forbids Lyra from
|
| 60 |
+
adopting them, and asks for direct replies without opening fillers such as Ah, Um, or Hmm. Protected facts, trust scores,
|
| 61 |
+
and private director instructions remain outside model context.
|
| 62 |
Generated text is labeled `Model response text` until a physical test proves it matches the
|
| 63 |
spoken assistant response. Audio-code handling follows LiquidAI's reference implementation: retain the complete code
|
| 64 |
sequence and remove only a confirmed terminal all-2048 end-of-audio frame. The obsolete incremental Mimi decoder and
|
| 65 |
its GPU warm-up are removed. A 600 ms pause closes the visitor turn. Model output runs until its audio EOS or the
|
| 66 |
+
bounded 768-token generation cap so decoded text cannot outrun audible speech on longer replies. Diagnostics report
|
| 67 |
+
token/EOS handling, microphone signal quality/drops, playback underruns, startup/generation/retry/decode timing,
|
| 68 |
+
prefix-stream first-audio time, checkpoints, fallback/late-guard status, emitted duration, overhead, and live prefix/reference
|
| 69 |
+
waveform equivalence.
|
| 70 |
|
| 71 |
+
On every turn, live generation rejects false assistant identities—including attempts to “go by” a visitor's name—and
|
| 72 |
+
narrated openings before playback. The first attempt also rejects filler openings. After the first turn it rejects
|
| 73 |
generic assistant openings and openings that closely match any earlier session response. Two bounded regeneration
|
| 74 |
attempts use progressively broader text sampling; a remaining false-identity output fails closed rather than becoming
|
| 75 |
+
audible. Prefix playback waits for a complete opening sentence or a 96-character guard window. A protected marker,
|
| 76 |
+
false identity, or narrated role error discovered after streaming begins immediately aborts and flushes that reply.
|
| 77 |
+
The completed clean decode is compared with the emitted prefix; a waveform mismatch disables prefix streaming for
|
| 78 |
+
the remainder of that session and returns subsequent turns to whole-response playback.
|
| 79 |
+
Diagnostics distinguish identity, filler, generic-opening, history-repeat, and narration regenerations without
|
| 80 |
changing the official system prompt or audio-code path.
|
| 81 |
|
| 82 |
## Privacy and safety
|
live_app.py
CHANGED
|
@@ -47,14 +47,17 @@ SPEECH_RMS = 0.012
|
|
| 47 |
BARGE_IN_RMS = 0.035
|
| 48 |
BARGE_IN_CONFIRM_SECONDS = 0.22
|
| 49 |
BARGE_IN_GRACE_SECONDS = 0.3
|
| 50 |
-
|
| 51 |
-
|
| 52 |
-
|
| 53 |
-
STREAM_DIAGNOSTIC_LATEST_START_SECONDS = 165.0
|
| 54 |
TRANSPORT_VERSION = "webaudio-websocket-pcm-v1"
|
| 55 |
INTERLEAVED_SYSTEM_PROMPT = "Respond with interleaved text and audio."
|
| 56 |
ASSISTANT_NAME = "Lyra"
|
| 57 |
-
ASSISTANT_IDENTITY_CONTEXT =
|
|
|
|
|
|
|
|
|
|
|
|
|
| 58 |
AUDIO_EOS_TOKEN = 2048
|
| 59 |
|
| 60 |
DIRECTOR_CUES = {
|
|
@@ -203,12 +206,20 @@ def generic_assistant_opening(value):
|
|
| 203 |
|
| 204 |
|
| 205 |
def false_assistant_identity(value):
|
| 206 |
-
opening = " ".join(normalized_words(value).split()[:
|
| 207 |
if not opening:
|
| 208 |
return False
|
| 209 |
-
|
|
|
|
| 210 |
return True
|
| 211 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 212 |
return True
|
| 213 |
wrong_identity_markers = (
|
| 214 |
"im qwen",
|
|
@@ -226,6 +237,18 @@ def false_assistant_identity(value):
|
|
| 226 |
return "lyra" not in opening and any(marker in opening for marker in wrong_identity_markers)
|
| 227 |
|
| 228 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 229 |
@dataclass
|
| 230 |
class TurnDetector:
|
| 231 |
speaking: bool = False
|
|
@@ -287,11 +310,11 @@ def prefix_stream_diagnostic(processor, audio_codes, reference_pcm):
|
|
| 287 |
decode_seconds = 0.0
|
| 288 |
|
| 289 |
for checkpoint in range(
|
| 290 |
-
|
| 291 |
total_frames,
|
| 292 |
-
|
| 293 |
):
|
| 294 |
-
if len(checkpoints) >=
|
| 295 |
break
|
| 296 |
started = time.perf_counter()
|
| 297 |
prefix_pcm = (
|
|
@@ -305,7 +328,7 @@ def prefix_stream_diagnostic(processor, audio_codes, reference_pcm):
|
|
| 305 |
decode_seconds += time.perf_counter() - started
|
| 306 |
stable_samples = min(
|
| 307 |
prefix_pcm.size,
|
| 308 |
-
max(0, checkpoint -
|
| 309 |
reference.size,
|
| 310 |
)
|
| 311 |
if stable_samples > emitted_samples:
|
|
@@ -355,6 +378,38 @@ def prefix_stream_diagnostic(processor, audio_codes, reference_pcm):
|
|
| 355 |
}
|
| 356 |
|
| 357 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 358 |
_model_cache = {}
|
| 359 |
_manager = multiprocessing.Manager()
|
| 360 |
_live_sessions = _manager.dict()
|
|
@@ -466,6 +521,7 @@ def session_metrics(phase, started, turns, counters, first_reply, interruptions,
|
|
| 466 |
"duplicate_regenerations": playback.get("history_repeat_regenerations", 0),
|
| 467 |
"generic_opening_regenerations": playback.get("generic_opening_regenerations", 0),
|
| 468 |
"identity_regenerations": playback.get("identity_regenerations", 0),
|
|
|
|
| 469 |
"total_regenerations": playback.get("regenerations", 0),
|
| 470 |
"narration_regenerations": playback.get("style_regenerations", 0),
|
| 471 |
"unsafe_outputs_suppressed": playback.get("unsafe_suppressed", 0),
|
|
@@ -549,9 +605,23 @@ def session_metrics(phase, started, turns, counters, first_reply, interruptions,
|
|
| 549 |
"stream_diagnostic_max_abs_error": None
|
| 550 |
if playback.get("stream_diagnostic_max_abs_error") is None
|
| 551 |
else round(playback["stream_diagnostic_max_abs_error"], 6),
|
| 552 |
-
"stream_diagnostic_audible":
|
| 553 |
-
"
|
| 554 |
-
"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 555 |
"reply_packets_emitted": playback["packets"],
|
| 556 |
"reply_audio_emitted_seconds": round(playback["samples"] / OUTPUT_SAMPLE_RATE, 3),
|
| 557 |
"reply_packet_target_seconds": round(OUTPUT_PACKET_SAMPLES / OUTPUT_SAMPLE_RATE, 3),
|
|
@@ -562,9 +632,9 @@ def session_metrics(phase, started, turns, counters, first_reply, interruptions,
|
|
| 562 |
"transport": TRANSPORT_VERSION,
|
| 563 |
"model_contract": "official exact interleaved system prompt",
|
| 564 |
"assistant_identity": ASSISTANT_NAME,
|
| 565 |
-
"identity_grounding": "public
|
| 566 |
-
"audio_delivery": "
|
| 567 |
-
"mode": "interruptible
|
| 568 |
},
|
| 569 |
indent=2,
|
| 570 |
)
|
|
@@ -611,6 +681,7 @@ def run_live_session(scene, transport_id):
|
|
| 611 |
"history_repeat_regenerations": 0,
|
| 612 |
"generic_opening_regenerations": 0,
|
| 613 |
"identity_regenerations": 0,
|
|
|
|
| 614 |
"style_regenerations": 0,
|
| 615 |
"style_suppressed": 0,
|
| 616 |
"identity_suppressed": 0,
|
|
@@ -639,7 +710,7 @@ def run_live_session(scene, transport_id):
|
|
| 639 |
"last_false_identity": None,
|
| 640 |
"last_history_repeat_score": None,
|
| 641 |
"legacy_filter_would_drop": None,
|
| 642 |
-
"stream_diagnostic_status": "
|
| 643 |
"stream_diagnostic_checkpoints": 0,
|
| 644 |
"stream_diagnostic_compared_seconds": None,
|
| 645 |
"stream_diagnostic_decode_seconds": None,
|
|
@@ -647,9 +718,15 @@ def run_live_session(scene, transport_id):
|
|
| 647 |
"stream_diagnostic_rms_ratio": None,
|
| 648 |
"stream_diagnostic_snr_db": None,
|
| 649 |
"stream_diagnostic_max_abs_error": None,
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 650 |
}
|
| 651 |
-
|
| 652 |
-
stream_diagnostic_reference = None
|
| 653 |
|
| 654 |
def emit_audio_packet(packet, response_started):
|
| 655 |
nonlocal first_reply
|
|
@@ -663,6 +740,42 @@ def run_live_session(scene, transport_id):
|
|
| 663 |
first_reply = time.perf_counter() - response_started
|
| 664 |
return True
|
| 665 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 666 |
ready_event.set()
|
| 667 |
queue_control(output_queue, {"type": "phase", "phase": "listening"})
|
| 668 |
live_started = time.perf_counter()
|
|
@@ -723,6 +836,18 @@ def run_live_session(scene, transport_id):
|
|
| 723 |
yield "**Lyra is thinking.** You can speak again to cancel this reply.", gr.skip(), gr.skip()
|
| 724 |
|
| 725 |
response_started = time.perf_counter()
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 726 |
interrupted = False
|
| 727 |
history_texts = [
|
| 728 |
entry["response"]
|
|
@@ -735,6 +860,8 @@ def run_live_session(scene, transport_id):
|
|
| 735 |
legacy_filter_would_drop = 0
|
| 736 |
attempts_used = 0
|
| 737 |
retry_generation_seconds = 0.0
|
|
|
|
|
|
|
| 738 |
for attempt in range(MAX_GENERATION_ATTEMPTS):
|
| 739 |
attempts_used = attempt + 1
|
| 740 |
text_tokens = []
|
|
@@ -746,6 +873,17 @@ def run_live_session(scene, transport_id):
|
|
| 746 |
attempt_started = time.perf_counter()
|
| 747 |
attempt_first_token_seconds = None
|
| 748 |
attempt_first_audio_token_seconds = None
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 749 |
|
| 750 |
with torch.inference_mode():
|
| 751 |
generation = model.generate_interleaved(
|
|
@@ -793,11 +931,28 @@ def run_live_session(scene, transport_id):
|
|
| 793 |
text_tokens.append(token)
|
| 794 |
modalities.append(LFMModality.TEXT)
|
| 795 |
current_text = processor.text.decode(torch.cat(text_tokens)).removesuffix("<|text_end|>").strip()
|
| 796 |
-
|
| 797 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 798 |
opening_guard_reason = "identity"
|
| 799 |
generation.close()
|
| 800 |
break
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 801 |
if guard_active and generic_assistant_opening(current_text):
|
| 802 |
opening_guard_reason = "generic"
|
| 803 |
generation.close()
|
|
@@ -817,8 +972,83 @@ def run_live_session(scene, transport_id):
|
|
| 817 |
modalities.append(LFMModality.AUDIO_OUT)
|
| 818 |
if torch.any(token == AUDIO_EOS_TOKEN).item():
|
| 819 |
legacy_filter_would_drop += 1
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 820 |
|
| 821 |
-
|
|
|
|
|
|
|
| 822 |
playback["generation_seconds_last_attempt"] = attempt_seconds
|
| 823 |
playback["first_token_seconds_last_attempt"] = attempt_first_token_seconds
|
| 824 |
playback["first_audio_token_seconds_last_attempt"] = attempt_first_audio_token_seconds
|
|
@@ -832,8 +1062,12 @@ def run_live_session(scene, transport_id):
|
|
| 832 |
playback["regenerations"] += 1
|
| 833 |
if opening_guard_reason == "identity":
|
| 834 |
playback["identity_regenerations"] += 1
|
|
|
|
|
|
|
| 835 |
elif opening_guard_reason == "generic":
|
| 836 |
playback["generic_opening_regenerations"] += 1
|
|
|
|
|
|
|
| 837 |
else:
|
| 838 |
playback["history_repeat_regenerations"] += 1
|
| 839 |
print(f"RoleForge rejected opening reason={opening_guard_reason}; regenerating.")
|
|
@@ -843,7 +1077,11 @@ def run_live_session(scene, transport_id):
|
|
| 843 |
if text_tokens
|
| 844 |
else ""
|
| 845 |
)
|
| 846 |
-
if
|
|
|
|
|
|
|
|
|
|
|
|
|
| 847 |
retry_generation_seconds += attempt_seconds
|
| 848 |
playback["regenerations"] += 1
|
| 849 |
playback["style_regenerations"] += 1
|
|
@@ -851,7 +1089,10 @@ def run_live_session(scene, transport_id):
|
|
| 851 |
continue
|
| 852 |
break
|
| 853 |
|
| 854 |
-
turn_generation_seconds =
|
|
|
|
|
|
|
|
|
|
| 855 |
playback["generation_seconds_last_turn"] = turn_generation_seconds
|
| 856 |
playback["retry_generation_seconds_last_turn"] = retry_generation_seconds
|
| 857 |
model_seconds += turn_generation_seconds
|
|
@@ -879,6 +1120,10 @@ def run_live_session(scene, transport_id):
|
|
| 879 |
identity_violation = false_assistant_identity(response_text)
|
| 880 |
style_violation = dialogue_style_violation(response_text)
|
| 881 |
reference_pcm = np.empty(0, dtype=np.float32)
|
|
|
|
|
|
|
|
|
|
|
|
|
| 882 |
if (
|
| 883 |
not interrupted
|
| 884 |
and playable_audio_tokens
|
|
@@ -894,9 +1139,30 @@ def run_live_session(scene, transport_id):
|
|
| 894 |
turn_decode_seconds = time.perf_counter() - decode_started
|
| 895 |
playback["reference_decode_seconds"] += turn_decode_seconds
|
| 896 |
playback["reference_decode_seconds_last_turn"] = turn_decode_seconds
|
| 897 |
-
if
|
| 898 |
-
|
| 899 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 900 |
|
| 901 |
if leaks:
|
| 902 |
playback["unsafe_suppressed"] += 1
|
|
@@ -912,42 +1178,43 @@ def run_live_session(scene, transport_id):
|
|
| 912 |
response_text = "[narrated model output suppressed]"
|
| 913 |
|
| 914 |
if reference_pcm.size and not interrupted:
|
| 915 |
-
|
| 916 |
-
|
| 917 |
-
|
| 918 |
-
|
| 919 |
-
|
| 920 |
-
|
| 921 |
-
|
| 922 |
-
|
| 923 |
-
pending_rms = (
|
| 924 |
-
float(np.sqrt(np.mean(pending_audio * pending_audio))) if pending_audio.size else 0.0
|
| 925 |
)
|
| 926 |
-
|
| 927 |
-
|
| 928 |
-
|
| 929 |
-
|
| 930 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 931 |
)
|
| 932 |
-
|
| 933 |
-
|
| 934 |
-
|
| 935 |
-
|
| 936 |
-
|
| 937 |
-
|
| 938 |
-
|
| 939 |
-
|
| 940 |
-
|
| 941 |
-
|
| 942 |
-
|
| 943 |
-
|
| 944 |
-
|
| 945 |
-
|
| 946 |
-
|
| 947 |
-
|
| 948 |
-
if not emit_audio_packet(packet, response_started):
|
| 949 |
-
interrupted = True
|
| 950 |
-
break
|
| 951 |
|
| 952 |
repeat_score = history_repetition_score(response_text, history_texts)
|
| 953 |
repeated = repeat_score >= 0.86
|
|
@@ -977,48 +1244,6 @@ def run_live_session(scene, transport_id):
|
|
| 977 |
"repeated": repeated,
|
| 978 |
}
|
| 979 |
)
|
| 980 |
-
if stream_diagnostic_codes is not None and playback["stream_diagnostic_status"] == "pending":
|
| 981 |
-
if time.perf_counter() - started > STREAM_DIAGNOSTIC_LATEST_START_SECONDS:
|
| 982 |
-
playback["stream_diagnostic_status"] = "skipped_deadline_guard"
|
| 983 |
-
else:
|
| 984 |
-
queue_control(output_queue, {"type": "phase", "phase": "calibrating"})
|
| 985 |
-
yield (
|
| 986 |
-
"**Lyra audio calibration.** Comparing inaudible stable-prefix audio with the clean reply…",
|
| 987 |
-
render_transcript(entries),
|
| 988 |
-
session_metrics(
|
| 989 |
-
"diagnostics",
|
| 990 |
-
started,
|
| 991 |
-
turns,
|
| 992 |
-
counters,
|
| 993 |
-
first_reply,
|
| 994 |
-
interruptions,
|
| 995 |
-
model_seconds,
|
| 996 |
-
playback,
|
| 997 |
-
repeats,
|
| 998 |
-
),
|
| 999 |
-
)
|
| 1000 |
-
try:
|
| 1001 |
-
with torch.inference_mode():
|
| 1002 |
-
diagnostic = prefix_stream_diagnostic(
|
| 1003 |
-
processor,
|
| 1004 |
-
stream_diagnostic_codes.to(DEVICE),
|
| 1005 |
-
stream_diagnostic_reference,
|
| 1006 |
-
)
|
| 1007 |
-
playback["stream_diagnostic_status"] = (
|
| 1008 |
-
"completed" if diagnostic["attempted"] else "skipped_insufficient_checkpoints"
|
| 1009 |
-
)
|
| 1010 |
-
playback["stream_diagnostic_checkpoints"] = len(diagnostic["checkpoints"])
|
| 1011 |
-
playback["stream_diagnostic_compared_seconds"] = diagnostic["compared_seconds"]
|
| 1012 |
-
playback["stream_diagnostic_decode_seconds"] = diagnostic["decode_seconds"]
|
| 1013 |
-
playback["stream_diagnostic_correlation"] = diagnostic["correlation"]
|
| 1014 |
-
playback["stream_diagnostic_rms_ratio"] = diagnostic["rms_ratio"]
|
| 1015 |
-
playback["stream_diagnostic_snr_db"] = diagnostic["snr_db"]
|
| 1016 |
-
playback["stream_diagnostic_max_abs_error"] = diagnostic["max_abs_error"]
|
| 1017 |
-
except Exception as exc:
|
| 1018 |
-
playback["stream_diagnostic_status"] = f"failed_{type(exc).__name__}"
|
| 1019 |
-
print(f"RoleForge stable-prefix diagnostic failed: {type(exc).__name__}")
|
| 1020 |
-
stream_diagnostic_codes = None
|
| 1021 |
-
stream_diagnostic_reference = None
|
| 1022 |
if turns >= MAX_TURNS:
|
| 1023 |
break
|
| 1024 |
queue_control(output_queue, {"type": "phase", "phase": "listening"})
|
|
@@ -1031,8 +1256,8 @@ def run_live_session(scene, transport_id):
|
|
| 1031 |
ready_event.clear()
|
| 1032 |
queue_control(output_queue, {"type": "complete"})
|
| 1033 |
stop_event.set()
|
| 1034 |
-
if playback["stream_diagnostic_status"] == "
|
| 1035 |
-
playback["stream_diagnostic_status"] = "
|
| 1036 |
final_status = "Session complete. Start a new live session to continue." if turns else "Session ended before a turn completed."
|
| 1037 |
yield (
|
| 1038 |
final_status,
|
|
@@ -1242,15 +1467,16 @@ with gr.Blocks(title="RoleForge Live Voice NPC", theme=gr.themes.Soft(), css=CSS
|
|
| 1242 |
gr.HTML(
|
| 1243 |
"""
|
| 1244 |
<div class="hero">
|
| 1245 |
-
<div class="phase">
|
| 1246 |
<h1>🎭 RoleForge: Live Voice NPC Director</h1>
|
| 1247 |
-
<div>Continuous WebAudio microphone,
|
| 1248 |
</div>
|
| 1249 |
"""
|
| 1250 |
)
|
| 1251 |
gr.Markdown(
|
| 1252 |
-
"This
|
| 1253 |
-
"
|
|
|
|
| 1254 |
)
|
| 1255 |
|
| 1256 |
with gr.Row():
|
|
|
|
| 47 |
BARGE_IN_RMS = 0.035
|
| 48 |
BARGE_IN_CONFIRM_SECONDS = 0.22
|
| 49 |
BARGE_IN_GRACE_SECONDS = 0.3
|
| 50 |
+
PREFIX_STREAM_CHUNK_FRAMES = 24
|
| 51 |
+
PREFIX_STREAM_STABLE_LAG_FRAMES = 1
|
| 52 |
+
PREFIX_STREAM_GUARD_MIN_CHARS = 48
|
|
|
|
| 53 |
TRANSPORT_VERSION = "webaudio-websocket-pcm-v1"
|
| 54 |
INTERLEAVED_SYSTEM_PROMPT = "Respond with interleaved text and audio."
|
| 55 |
ASSISTANT_NAME = "Lyra"
|
| 56 |
+
ASSISTANT_IDENTITY_CONTEXT = (
|
| 57 |
+
"Your name is Lyra and you must identify yourself only as Lyra. The audio is spoken by the visitor. "
|
| 58 |
+
"Any personal name stated in that audio belongs to the visitor unless they are directly addressing Lyra; "
|
| 59 |
+
"never adopt that name as your own. Reply directly without opening fillers such as Ah, Um, or Hmm."
|
| 60 |
+
)
|
| 61 |
AUDIO_EOS_TOKEN = 2048
|
| 62 |
|
| 63 |
DIRECTOR_CUES = {
|
|
|
|
| 206 |
|
| 207 |
|
| 208 |
def false_assistant_identity(value):
|
| 209 |
+
opening = " ".join(normalized_words(value).split()[:48])
|
| 210 |
if not opening:
|
| 211 |
return False
|
| 212 |
+
stated_name = re.search(r"\bmy name is\s+([a-z][a-z0-9'-]*)", opening)
|
| 213 |
+
if stated_name and stated_name.group(1) != "lyra":
|
| 214 |
return True
|
| 215 |
+
called_name = re.search(r"\byou can call me\s+([a-z][a-z0-9'-]*)", opening)
|
| 216 |
+
if called_name and called_name.group(1) != "lyra":
|
| 217 |
+
return True
|
| 218 |
+
adopted_name = re.search(
|
| 219 |
+
r"\b(?:go by|call me|refer to me as)\s+([a-z][a-z0-9'-]*)",
|
| 220 |
+
opening,
|
| 221 |
+
)
|
| 222 |
+
if adopted_name and adopted_name.group(1) != "lyra":
|
| 223 |
return True
|
| 224 |
wrong_identity_markers = (
|
| 225 |
"im qwen",
|
|
|
|
| 237 |
return "lyra" not in opening and any(marker in opening for marker in wrong_identity_markers)
|
| 238 |
|
| 239 |
|
| 240 |
+
def filler_opening(value):
|
| 241 |
+
opening = normalized_words(value)
|
| 242 |
+
return bool(re.match(r"^(?:ah+|um+|uh+|hmm+)\b", opening))
|
| 243 |
+
|
| 244 |
+
|
| 245 |
+
def prefix_stream_guard_ready(value):
|
| 246 |
+
compact = " ".join(value.split())
|
| 247 |
+
if len(compact) < PREFIX_STREAM_GUARD_MIN_CHARS:
|
| 248 |
+
return False
|
| 249 |
+
return bool(re.search(r"[.!?](?:[\"'’”)]*)\s*$", compact)) or len(compact) >= 96
|
| 250 |
+
|
| 251 |
+
|
| 252 |
@dataclass
|
| 253 |
class TurnDetector:
|
| 254 |
speaking: bool = False
|
|
|
|
| 310 |
decode_seconds = 0.0
|
| 311 |
|
| 312 |
for checkpoint in range(
|
| 313 |
+
PREFIX_STREAM_CHUNK_FRAMES,
|
| 314 |
total_frames,
|
| 315 |
+
PREFIX_STREAM_CHUNK_FRAMES,
|
| 316 |
):
|
| 317 |
+
if len(checkpoints) >= 4:
|
| 318 |
break
|
| 319 |
started = time.perf_counter()
|
| 320 |
prefix_pcm = (
|
|
|
|
| 328 |
decode_seconds += time.perf_counter() - started
|
| 329 |
stable_samples = min(
|
| 330 |
prefix_pcm.size,
|
| 331 |
+
max(0, checkpoint - PREFIX_STREAM_STABLE_LAG_FRAMES) * AUDIO_FRAME_SAMPLES,
|
| 332 |
reference.size,
|
| 333 |
)
|
| 334 |
if stable_samples > emitted_samples:
|
|
|
|
| 378 |
}
|
| 379 |
|
| 380 |
|
| 381 |
+
def compare_prefix_waveform(candidate_pcm, reference_pcm):
|
| 382 |
+
candidate = np.asarray(candidate_pcm, dtype=np.float32).reshape(-1)
|
| 383 |
+
reference = np.asarray(reference_pcm, dtype=np.float32).reshape(-1)
|
| 384 |
+
compared_samples = min(candidate.size, reference.size)
|
| 385 |
+
if not compared_samples:
|
| 386 |
+
return {
|
| 387 |
+
"compared_seconds": 0.0,
|
| 388 |
+
"correlation": None,
|
| 389 |
+
"rms_ratio": None,
|
| 390 |
+
"snr_db": None,
|
| 391 |
+
"max_abs_error": None,
|
| 392 |
+
}
|
| 393 |
+
candidate = candidate[:compared_samples].astype(np.float64)
|
| 394 |
+
reference = reference[:compared_samples].astype(np.float64)
|
| 395 |
+
error = candidate - reference
|
| 396 |
+
reference_rms = float(np.sqrt(np.mean(reference * reference)))
|
| 397 |
+
candidate_rms = float(np.sqrt(np.mean(candidate * candidate)))
|
| 398 |
+
error_rms = float(np.sqrt(np.mean(error * error)))
|
| 399 |
+
correlation = (
|
| 400 |
+
float(np.corrcoef(candidate, reference)[0, 1])
|
| 401 |
+
if np.std(candidate) > 1e-12 and np.std(reference) > 1e-12
|
| 402 |
+
else (1.0 if np.allclose(candidate, reference) else 0.0)
|
| 403 |
+
)
|
| 404 |
+
return {
|
| 405 |
+
"compared_seconds": compared_samples / OUTPUT_SAMPLE_RATE,
|
| 406 |
+
"correlation": correlation,
|
| 407 |
+
"rms_ratio": candidate_rms / max(reference_rms, 1e-12),
|
| 408 |
+
"snr_db": 20.0 * np.log10(reference_rms / max(error_rms, 1e-12)),
|
| 409 |
+
"max_abs_error": float(np.max(np.abs(error))),
|
| 410 |
+
}
|
| 411 |
+
|
| 412 |
+
|
| 413 |
_model_cache = {}
|
| 414 |
_manager = multiprocessing.Manager()
|
| 415 |
_live_sessions = _manager.dict()
|
|
|
|
| 521 |
"duplicate_regenerations": playback.get("history_repeat_regenerations", 0),
|
| 522 |
"generic_opening_regenerations": playback.get("generic_opening_regenerations", 0),
|
| 523 |
"identity_regenerations": playback.get("identity_regenerations", 0),
|
| 524 |
+
"filler_opening_regenerations": playback.get("filler_regenerations", 0),
|
| 525 |
"total_regenerations": playback.get("regenerations", 0),
|
| 526 |
"narration_regenerations": playback.get("style_regenerations", 0),
|
| 527 |
"unsafe_outputs_suppressed": playback.get("unsafe_suppressed", 0),
|
|
|
|
| 605 |
"stream_diagnostic_max_abs_error": None
|
| 606 |
if playback.get("stream_diagnostic_max_abs_error") is None
|
| 607 |
else round(playback["stream_diagnostic_max_abs_error"], 6),
|
| 608 |
+
"stream_diagnostic_audible": playback.get("stream_diagnostic_status")
|
| 609 |
+
== "completed_audible_prefix",
|
| 610 |
+
"stream_diagnostic_chunk_frames": PREFIX_STREAM_CHUNK_FRAMES,
|
| 611 |
+
"stream_diagnostic_stable_lag_frames": PREFIX_STREAM_STABLE_LAG_FRAMES,
|
| 612 |
+
"prefix_stream_status": playback.get("prefix_stream_status"),
|
| 613 |
+
"prefix_stream_fallbacks": playback.get("prefix_stream_fallbacks", 0),
|
| 614 |
+
"prefix_stream_late_guard_aborts": playback.get("prefix_stream_late_guard_aborts", 0),
|
| 615 |
+
"prefix_stream_first_audio_seconds": None
|
| 616 |
+
if playback.get("prefix_stream_first_audio_seconds") is None
|
| 617 |
+
else round(playback["prefix_stream_first_audio_seconds"], 3),
|
| 618 |
+
"prefix_stream_emitted_seconds": round(
|
| 619 |
+
playback.get("prefix_stream_emitted_seconds", 0.0), 3
|
| 620 |
+
),
|
| 621 |
+
"prefix_stream_overhead_seconds_last_turn": round(
|
| 622 |
+
playback.get("prefix_stream_overhead_seconds_last_turn", 0.0), 3
|
| 623 |
+
),
|
| 624 |
+
"prefix_stream_session_enabled": playback.get("prefix_stream_session_enabled", True),
|
| 625 |
"reply_packets_emitted": playback["packets"],
|
| 626 |
"reply_audio_emitted_seconds": round(playback["samples"] / OUTPUT_SAMPLE_RATE, 3),
|
| 627 |
"reply_packet_target_seconds": round(OUTPUT_PACKET_SAMPLES / OUTPUT_SAMPLE_RATE, 3),
|
|
|
|
| 632 |
"transport": TRANSPORT_VERSION,
|
| 633 |
"model_contract": "official exact interleaved system prompt",
|
| 634 |
"assistant_identity": ASSISTANT_NAME,
|
| 635 |
+
"identity_grounding": "Lyra-only public role boundary attached to each visitor turn",
|
| 636 |
+
"audio_delivery": "guarded held-tail stable-prefix stream with clean reference tail",
|
| 637 |
+
"mode": "interruptible low-latency LFM2.5-Audio",
|
| 638 |
},
|
| 639 |
indent=2,
|
| 640 |
)
|
|
|
|
| 681 |
"history_repeat_regenerations": 0,
|
| 682 |
"generic_opening_regenerations": 0,
|
| 683 |
"identity_regenerations": 0,
|
| 684 |
+
"filler_regenerations": 0,
|
| 685 |
"style_regenerations": 0,
|
| 686 |
"style_suppressed": 0,
|
| 687 |
"identity_suppressed": 0,
|
|
|
|
| 710 |
"last_false_identity": None,
|
| 711 |
"last_history_repeat_score": None,
|
| 712 |
"legacy_filter_would_drop": None,
|
| 713 |
+
"stream_diagnostic_status": "pending_audible_prefix",
|
| 714 |
"stream_diagnostic_checkpoints": 0,
|
| 715 |
"stream_diagnostic_compared_seconds": None,
|
| 716 |
"stream_diagnostic_decode_seconds": None,
|
|
|
|
| 718 |
"stream_diagnostic_rms_ratio": None,
|
| 719 |
"stream_diagnostic_snr_db": None,
|
| 720 |
"stream_diagnostic_max_abs_error": None,
|
| 721 |
+
"prefix_stream_status": "not_started",
|
| 722 |
+
"prefix_stream_fallbacks": 0,
|
| 723 |
+
"prefix_stream_late_guard_aborts": 0,
|
| 724 |
+
"prefix_stream_first_audio_seconds": None,
|
| 725 |
+
"prefix_stream_emitted_seconds": 0.0,
|
| 726 |
+
"prefix_stream_overhead_seconds_last_turn": 0.0,
|
| 727 |
+
"prefix_stream_session_enabled": True,
|
| 728 |
}
|
| 729 |
+
prefix_stream_enabled = True
|
|
|
|
| 730 |
|
| 731 |
def emit_audio_packet(packet, response_started):
|
| 732 |
nonlocal first_reply
|
|
|
|
| 740 |
first_reply = time.perf_counter() - response_started
|
| 741 |
return True
|
| 742 |
|
| 743 |
+
def collect_stream_interrupt(playback_started):
|
| 744 |
+
nonlocal utterance_rate
|
| 745 |
+
for pending_rate, pending_audio in drain_queue(input_queue):
|
| 746 |
+
pending_audio = np.asarray(pending_audio, dtype=np.float32).reshape(-1)
|
| 747 |
+
pending_duration = pending_audio.size / max(float(pending_rate), 1.0)
|
| 748 |
+
pending_rms = (
|
| 749 |
+
float(np.sqrt(np.mean(pending_audio * pending_audio))) if pending_audio.size else 0.0
|
| 750 |
+
)
|
| 751 |
+
pending_events = detector.push(
|
| 752 |
+
pending_rms,
|
| 753 |
+
pending_duration,
|
| 754 |
+
assistant_speaking=True,
|
| 755 |
+
assistant_elapsed=time.perf_counter() - playback_started,
|
| 756 |
+
)
|
| 757 |
+
if "speech_started" in pending_events:
|
| 758 |
+
utterance.extend(pre_roll)
|
| 759 |
+
pre_roll.clear()
|
| 760 |
+
utterance_rate = pending_rate
|
| 761 |
+
if detector.speaking or "turn_ready" in pending_events:
|
| 762 |
+
if utterance_rate == pending_rate:
|
| 763 |
+
utterance.append(pending_audio)
|
| 764 |
+
else:
|
| 765 |
+
pre_roll.append(pending_audio)
|
| 766 |
+
if "interrupt" in pending_events:
|
| 767 |
+
return True
|
| 768 |
+
return False
|
| 769 |
+
|
| 770 |
+
def emit_pcm(pcm, response_started, playback_started):
|
| 771 |
+
pcm16 = (np.clip(np.asarray(pcm, dtype=np.float32), -1.0, 1.0) * 32767.0).astype(np.int16)
|
| 772 |
+
for offset in range(0, pcm16.size, OUTPUT_PACKET_SAMPLES):
|
| 773 |
+
if collect_stream_interrupt(playback_started):
|
| 774 |
+
return False
|
| 775 |
+
if not emit_audio_packet(pcm16[offset : offset + OUTPUT_PACKET_SAMPLES], response_started):
|
| 776 |
+
return False
|
| 777 |
+
return True
|
| 778 |
+
|
| 779 |
ready_event.set()
|
| 780 |
queue_control(output_queue, {"type": "phase", "phase": "listening"})
|
| 781 |
live_started = time.perf_counter()
|
|
|
|
| 836 |
yield "**Lyra is thinking.** You can speak again to cancel this reply.", gr.skip(), gr.skip()
|
| 837 |
|
| 838 |
response_started = time.perf_counter()
|
| 839 |
+
playback["prefix_stream_status"] = "not_started"
|
| 840 |
+
playback["prefix_stream_first_audio_seconds"] = None
|
| 841 |
+
playback["prefix_stream_emitted_seconds"] = 0.0
|
| 842 |
+
playback["prefix_stream_overhead_seconds_last_turn"] = 0.0
|
| 843 |
+
playback["stream_diagnostic_status"] = "pending_audible_prefix"
|
| 844 |
+
playback["stream_diagnostic_checkpoints"] = 0
|
| 845 |
+
playback["stream_diagnostic_compared_seconds"] = None
|
| 846 |
+
playback["stream_diagnostic_decode_seconds"] = None
|
| 847 |
+
playback["stream_diagnostic_correlation"] = None
|
| 848 |
+
playback["stream_diagnostic_rms_ratio"] = None
|
| 849 |
+
playback["stream_diagnostic_snr_db"] = None
|
| 850 |
+
playback["stream_diagnostic_max_abs_error"] = None
|
| 851 |
interrupted = False
|
| 852 |
history_texts = [
|
| 853 |
entry["response"]
|
|
|
|
| 860 |
legacy_filter_would_drop = 0
|
| 861 |
attempts_used = 0
|
| 862 |
retry_generation_seconds = 0.0
|
| 863 |
+
stream_overhead_turn = 0.0
|
| 864 |
+
stream_guard_abort = False
|
| 865 |
for attempt in range(MAX_GENERATION_ATTEMPTS):
|
| 866 |
attempts_used = attempt + 1
|
| 867 |
text_tokens = []
|
|
|
|
| 873 |
attempt_started = time.perf_counter()
|
| 874 |
attempt_first_token_seconds = None
|
| 875 |
attempt_first_audio_token_seconds = None
|
| 876 |
+
attempt_stream_overhead = 0.0
|
| 877 |
+
current_text = ""
|
| 878 |
+
stream_emitted_samples = 0
|
| 879 |
+
stream_candidate_parts = []
|
| 880 |
+
stream_checkpoints = 0
|
| 881 |
+
stream_last_checkpoint_frames = 0
|
| 882 |
+
stream_decode_seconds = 0.0
|
| 883 |
+
stream_started = False
|
| 884 |
+
stream_playback_started = None
|
| 885 |
+
stream_decode_failed = False
|
| 886 |
+
stream_guard_abort = False
|
| 887 |
|
| 888 |
with torch.inference_mode():
|
| 889 |
generation = model.generate_interleaved(
|
|
|
|
| 931 |
text_tokens.append(token)
|
| 932 |
modalities.append(LFMModality.TEXT)
|
| 933 |
current_text = processor.text.decode(torch.cat(text_tokens)).removesuffix("<|text_end|>").strip()
|
| 934 |
+
identity_error = false_assistant_identity(current_text)
|
| 935 |
+
style_error = dialogue_style_violation(current_text)
|
| 936 |
+
leak_error = bool(leak_markers(current_text))
|
| 937 |
+
if stream_started and (identity_error or style_error or leak_error):
|
| 938 |
+
stream_guard_abort = True
|
| 939 |
+
playback["prefix_stream_late_guard_aborts"] += 1
|
| 940 |
+
generation.close()
|
| 941 |
+
flush_output(output_queue)
|
| 942 |
+
break
|
| 943 |
+
if not stream_started and attempt < MAX_GENERATION_ATTEMPTS - 1:
|
| 944 |
+
if identity_error:
|
| 945 |
opening_guard_reason = "identity"
|
| 946 |
generation.close()
|
| 947 |
break
|
| 948 |
+
if attempt == 0 and filler_opening(current_text):
|
| 949 |
+
opening_guard_reason = "filler"
|
| 950 |
+
generation.close()
|
| 951 |
+
break
|
| 952 |
+
if style_error:
|
| 953 |
+
opening_guard_reason = "style"
|
| 954 |
+
generation.close()
|
| 955 |
+
break
|
| 956 |
if guard_active and generic_assistant_opening(current_text):
|
| 957 |
opening_guard_reason = "generic"
|
| 958 |
generation.close()
|
|
|
|
| 972 |
modalities.append(LFMModality.AUDIO_OUT)
|
| 973 |
if torch.any(token == AUDIO_EOS_TOKEN).item():
|
| 974 |
legacy_filter_would_drop += 1
|
| 975 |
+
should_decode_prefix = (
|
| 976 |
+
prefix_stream_enabled
|
| 977 |
+
and not is_audio_eos(token)
|
| 978 |
+
and not stream_decode_failed
|
| 979 |
+
and len(audio_tokens) >= PREFIX_STREAM_CHUNK_FRAMES
|
| 980 |
+
and (
|
| 981 |
+
not stream_started
|
| 982 |
+
or len(audio_tokens) - stream_last_checkpoint_frames
|
| 983 |
+
>= PREFIX_STREAM_CHUNK_FRAMES
|
| 984 |
+
)
|
| 985 |
+
and prefix_stream_guard_ready(current_text)
|
| 986 |
+
and not false_assistant_identity(current_text)
|
| 987 |
+
and not dialogue_style_violation(current_text)
|
| 988 |
+
and not leak_markers(current_text)
|
| 989 |
+
and (attempt > 0 or not filler_opening(current_text))
|
| 990 |
+
)
|
| 991 |
+
if should_decode_prefix:
|
| 992 |
+
prefix_started = time.perf_counter()
|
| 993 |
+
stable_delta = None
|
| 994 |
+
stable_samples = stream_emitted_samples
|
| 995 |
+
try:
|
| 996 |
+
prefix_codes = torch.stack(audio_tokens, dim=1).unsqueeze(0)
|
| 997 |
+
prefix_pcm = np.clip(
|
| 998 |
+
processor.decode(prefix_codes)[0]
|
| 999 |
+
.detach()
|
| 1000 |
+
.float()
|
| 1001 |
+
.cpu()
|
| 1002 |
+
.numpy(),
|
| 1003 |
+
-1.0,
|
| 1004 |
+
1.0,
|
| 1005 |
+
).reshape(-1)
|
| 1006 |
+
stable_samples = min(
|
| 1007 |
+
prefix_pcm.size,
|
| 1008 |
+
max(0, len(audio_tokens) - PREFIX_STREAM_STABLE_LAG_FRAMES)
|
| 1009 |
+
* AUDIO_FRAME_SAMPLES,
|
| 1010 |
+
)
|
| 1011 |
+
if stable_samples > stream_emitted_samples:
|
| 1012 |
+
stable_delta = prefix_pcm[stream_emitted_samples:stable_samples].copy()
|
| 1013 |
+
except Exception as exc:
|
| 1014 |
+
stream_decode_failed = True
|
| 1015 |
+
playback["prefix_stream_fallbacks"] += 1
|
| 1016 |
+
print(f"RoleForge prefix decode fell back: {type(exc).__name__}")
|
| 1017 |
+
finally:
|
| 1018 |
+
decode_elapsed = time.perf_counter() - prefix_started
|
| 1019 |
+
stream_decode_seconds += decode_elapsed
|
| 1020 |
+
attempt_stream_overhead += decode_elapsed
|
| 1021 |
+
if stable_delta is not None:
|
| 1022 |
+
if not stream_started:
|
| 1023 |
+
stream_started = True
|
| 1024 |
+
stream_playback_started = time.perf_counter()
|
| 1025 |
+
playback["prefix_stream_first_audio_seconds"] = (
|
| 1026 |
+
stream_playback_started - response_started
|
| 1027 |
+
)
|
| 1028 |
+
queue_control(output_queue, {"type": "phase", "phase": "speaking"})
|
| 1029 |
+
emission_started = time.perf_counter()
|
| 1030 |
+
emitted = emit_pcm(
|
| 1031 |
+
stable_delta,
|
| 1032 |
+
response_started,
|
| 1033 |
+
stream_playback_started,
|
| 1034 |
+
)
|
| 1035 |
+
attempt_stream_overhead += time.perf_counter() - emission_started
|
| 1036 |
+
if not emitted:
|
| 1037 |
+
interrupted = True
|
| 1038 |
+
interruptions += 1
|
| 1039 |
+
generation.close()
|
| 1040 |
+
flush_output(output_queue)
|
| 1041 |
+
break
|
| 1042 |
+
stream_candidate_parts.append(stable_delta)
|
| 1043 |
+
stream_emitted_samples = stable_samples
|
| 1044 |
+
stream_checkpoints += 1
|
| 1045 |
+
stream_last_checkpoint_frames = len(audio_tokens)
|
| 1046 |
+
if interrupted or stream_guard_abort:
|
| 1047 |
+
break
|
| 1048 |
|
| 1049 |
+
attempt_wall_seconds = time.perf_counter() - attempt_started
|
| 1050 |
+
stream_overhead_turn += attempt_stream_overhead
|
| 1051 |
+
attempt_seconds = max(0.0, attempt_wall_seconds - attempt_stream_overhead)
|
| 1052 |
playback["generation_seconds_last_attempt"] = attempt_seconds
|
| 1053 |
playback["first_token_seconds_last_attempt"] = attempt_first_token_seconds
|
| 1054 |
playback["first_audio_token_seconds_last_attempt"] = attempt_first_audio_token_seconds
|
|
|
|
| 1062 |
playback["regenerations"] += 1
|
| 1063 |
if opening_guard_reason == "identity":
|
| 1064 |
playback["identity_regenerations"] += 1
|
| 1065 |
+
elif opening_guard_reason == "filler":
|
| 1066 |
+
playback["filler_regenerations"] += 1
|
| 1067 |
elif opening_guard_reason == "generic":
|
| 1068 |
playback["generic_opening_regenerations"] += 1
|
| 1069 |
+
elif opening_guard_reason == "style":
|
| 1070 |
+
playback["style_regenerations"] += 1
|
| 1071 |
else:
|
| 1072 |
playback["history_repeat_regenerations"] += 1
|
| 1073 |
print(f"RoleForge rejected opening reason={opening_guard_reason}; regenerating.")
|
|
|
|
| 1077 |
if text_tokens
|
| 1078 |
else ""
|
| 1079 |
)
|
| 1080 |
+
if (
|
| 1081 |
+
dialogue_style_violation(candidate_text)
|
| 1082 |
+
and attempt < MAX_GENERATION_ATTEMPTS - 1
|
| 1083 |
+
and not stream_started
|
| 1084 |
+
):
|
| 1085 |
retry_generation_seconds += attempt_seconds
|
| 1086 |
playback["regenerations"] += 1
|
| 1087 |
playback["style_regenerations"] += 1
|
|
|
|
| 1089 |
continue
|
| 1090 |
break
|
| 1091 |
|
| 1092 |
+
turn_generation_seconds = max(
|
| 1093 |
+
0.0,
|
| 1094 |
+
time.perf_counter() - response_started - stream_overhead_turn,
|
| 1095 |
+
)
|
| 1096 |
playback["generation_seconds_last_turn"] = turn_generation_seconds
|
| 1097 |
playback["retry_generation_seconds_last_turn"] = retry_generation_seconds
|
| 1098 |
model_seconds += turn_generation_seconds
|
|
|
|
| 1120 |
identity_violation = false_assistant_identity(response_text)
|
| 1121 |
style_violation = dialogue_style_violation(response_text)
|
| 1122 |
reference_pcm = np.empty(0, dtype=np.float32)
|
| 1123 |
+
playback["prefix_stream_overhead_seconds_last_turn"] = stream_overhead_turn
|
| 1124 |
+
playback["stream_diagnostic_checkpoints"] = stream_checkpoints
|
| 1125 |
+
playback["stream_diagnostic_decode_seconds"] = stream_decode_seconds
|
| 1126 |
+
playback["prefix_stream_emitted_seconds"] = stream_emitted_samples / OUTPUT_SAMPLE_RATE
|
| 1127 |
if (
|
| 1128 |
not interrupted
|
| 1129 |
and playable_audio_tokens
|
|
|
|
| 1139 |
turn_decode_seconds = time.perf_counter() - decode_started
|
| 1140 |
playback["reference_decode_seconds"] += turn_decode_seconds
|
| 1141 |
playback["reference_decode_seconds_last_turn"] = turn_decode_seconds
|
| 1142 |
+
if stream_candidate_parts:
|
| 1143 |
+
streamed_candidate = np.concatenate(stream_candidate_parts)
|
| 1144 |
+
comparison = compare_prefix_waveform(streamed_candidate, reference_pcm)
|
| 1145 |
+
playback["stream_diagnostic_status"] = "completed_audible_prefix"
|
| 1146 |
+
playback["stream_diagnostic_compared_seconds"] = comparison["compared_seconds"]
|
| 1147 |
+
playback["stream_diagnostic_correlation"] = comparison["correlation"]
|
| 1148 |
+
playback["stream_diagnostic_rms_ratio"] = comparison["rms_ratio"]
|
| 1149 |
+
playback["stream_diagnostic_snr_db"] = comparison["snr_db"]
|
| 1150 |
+
playback["stream_diagnostic_max_abs_error"] = comparison["max_abs_error"]
|
| 1151 |
+
waveform_safe = (
|
| 1152 |
+
comparison["correlation"] is not None
|
| 1153 |
+
and comparison["correlation"] >= 0.9999
|
| 1154 |
+
and comparison["rms_ratio"] is not None
|
| 1155 |
+
and 0.999 <= comparison["rms_ratio"] <= 1.001
|
| 1156 |
+
and comparison["snr_db"] is not None
|
| 1157 |
+
and comparison["snr_db"] >= 60.0
|
| 1158 |
+
and comparison["max_abs_error"] is not None
|
| 1159 |
+
and comparison["max_abs_error"] <= 0.001
|
| 1160 |
+
)
|
| 1161 |
+
if not waveform_safe:
|
| 1162 |
+
prefix_stream_enabled = False
|
| 1163 |
+
playback["prefix_stream_session_enabled"] = False
|
| 1164 |
+
playback["prefix_stream_fallbacks"] += 1
|
| 1165 |
+
playback["stream_diagnostic_status"] = "disabled_after_waveform_mismatch"
|
| 1166 |
|
| 1167 |
if leaks:
|
| 1168 |
playback["unsafe_suppressed"] += 1
|
|
|
|
| 1178 |
response_text = "[narrated model output suppressed]"
|
| 1179 |
|
| 1180 |
if reference_pcm.size and not interrupted:
|
| 1181 |
+
if stream_started:
|
| 1182 |
+
playback["prefix_stream_status"] = (
|
| 1183 |
+
"disabled_after_waveform_mismatch"
|
| 1184 |
+
if not prefix_stream_enabled
|
| 1185 |
+
else (
|
| 1186 |
+
"completed_with_reference_tail_after_decode_fallback"
|
| 1187 |
+
if stream_decode_failed
|
| 1188 |
+
else "completed"
|
|
|
|
|
|
|
| 1189 |
)
|
| 1190 |
+
)
|
| 1191 |
+
tail_pcm = reference_pcm[stream_emitted_samples:]
|
| 1192 |
+
playback_started = stream_playback_started
|
| 1193 |
+
else:
|
| 1194 |
+
playback["prefix_stream_status"] = (
|
| 1195 |
+
"fallback_whole_response_stream_disabled"
|
| 1196 |
+
if not prefix_stream_enabled
|
| 1197 |
+
else (
|
| 1198 |
+
"fallback_whole_response_decode_error"
|
| 1199 |
+
if stream_decode_failed
|
| 1200 |
+
else "fallback_whole_response_guard_or_short_reply"
|
| 1201 |
)
|
| 1202 |
+
)
|
| 1203 |
+
playback["stream_diagnostic_status"] = playback["prefix_stream_status"]
|
| 1204 |
+
tail_pcm = reference_pcm
|
| 1205 |
+
playback_started = time.perf_counter()
|
| 1206 |
+
queue_control(output_queue, {"type": "phase", "phase": "speaking"})
|
| 1207 |
+
yield "**Model is speaking.** Speak clearly to interrupt.", gr.skip(), gr.skip()
|
| 1208 |
+
if tail_pcm.size and not emit_pcm(
|
| 1209 |
+
tail_pcm,
|
| 1210 |
+
response_started,
|
| 1211 |
+
playback_started,
|
| 1212 |
+
):
|
| 1213 |
+
interrupted = True
|
| 1214 |
+
interruptions += 1
|
| 1215 |
+
flush_output(output_queue)
|
| 1216 |
+
elif stream_guard_abort:
|
| 1217 |
+
playback["prefix_stream_status"] = "aborted_late_guard"
|
|
|
|
|
|
|
|
|
|
| 1218 |
|
| 1219 |
repeat_score = history_repetition_score(response_text, history_texts)
|
| 1220 |
repeated = repeat_score >= 0.86
|
|
|
|
| 1244 |
"repeated": repeated,
|
| 1245 |
}
|
| 1246 |
)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1247 |
if turns >= MAX_TURNS:
|
| 1248 |
break
|
| 1249 |
queue_control(output_queue, {"type": "phase", "phase": "listening"})
|
|
|
|
| 1256 |
ready_event.clear()
|
| 1257 |
queue_control(output_queue, {"type": "complete"})
|
| 1258 |
stop_event.set()
|
| 1259 |
+
if playback["stream_diagnostic_status"] == "pending_audible_prefix":
|
| 1260 |
+
playback["stream_diagnostic_status"] = "no_eligible_response"
|
| 1261 |
final_status = "Session complete. Start a new live session to continue." if turns else "Session ended before a turn completed."
|
| 1262 |
yield (
|
| 1263 |
final_status,
|
|
|
|
| 1467 |
gr.HTML(
|
| 1468 |
"""
|
| 1469 |
<div class="hero">
|
| 1470 |
+
<div class="phase">LOW-LATENCY LIVE LFM2.5-AUDIO</div>
|
| 1471 |
<h1>🎭 RoleForge: Live Voice NPC Director</h1>
|
| 1472 |
+
<div>Continuous WebAudio microphone, stable-prefix PCM streaming, bounded memory, and barge-in.</div>
|
| 1473 |
</div>
|
| 1474 |
"""
|
| 1475 |
)
|
| 1476 |
gr.Markdown(
|
| 1477 |
+
"This build begins playback from guarded, proven-stable LFM prefix decodes and finishes with the clean reference tail. "
|
| 1478 |
+
"It automatically falls back to whole-response playback if prefix decoding fails. The microphone remains open and playback "
|
| 1479 |
+
"can be interrupted. **Wear headphones** to prevent echo-triggered interruption."
|
| 1480 |
)
|
| 1481 |
|
| 1482 |
with gr.Row():
|