README.md

---
title: RoleForge Voice NPC Lab
emoji: 🎭
colorFrom: purple
colorTo: indigo
sdk: gradio
sdk_version: 5.49.1
python_version: "3.12"
app_file: live_app.py
pinned: false
license: other
short_description: Test a fictional voice NPC with LFM2.5-Audio on ZeroGPU
models:
  - LiquidAI/LFM2.5-Audio-1.5B
tags:
  - audio-to-audio
  - roleplay
  - game-ai
  - voice-agent
  - zerogpu
startup_duration_timeout: 1h
---

# RoleForge Voice NPC Lab

RoleForge is a private feasibility lab for directing a fictional voice NPC. This build uses
`LiquidAI/LFM2.5-Audio-1.5B` for interruptible live speech-to-speech sessions on Hugging Face ZeroGPU.

## Current gate

This phase tests a guarded low-latency free-Space approximation to live conversation:

1. Wear headphones and press **Start 120-second live session**; approve microphone access when prompted.
2. Wait for **Listening**, then speak naturally without pressing Stop.
3. A short pause ends the visitor turn; playback begins once the opening guard and first stable audio prefix are ready.
4. Speaking clearly during playback interrupts it and begins a new visitor turn.

The app retains LFM's multimodal conversation state for up to six turns inside one bounded 120-second listening
window and a 180-second maximum GPU call. Context is held only in volatile worker memory and disappears when the
session ends or the worker exits.

This is interruptible turn-taking rather than mathematically simultaneous model inference: LFM is turn-based, so visitor
speech can cancel an active reply before the new utterance is processed. A browser AudioWorklet captures continuous
raw PCM and a same-origin WebSocket carries input and output packets. Gradio still owns controls, diagnostics, and
the bounded ZeroGPU call, but it no longer owns the microphone or audio playback. This avoids both the native
Record/Waiting state machine and HLS buffering. During generation, the server decodes growing audio-code prefixes,
holds back one unstable 80 ms frame, and emits only samples proven stable by the preceding physical A/B test. At EOS it
decodes the clean whole response once and emits only the not-yet-played reference tail. A prefix-decode exception falls
back automatically to the clean whole-response path. PCM is divided into packets of at most 160 ms and the browser
schedules them sample-exactly through WebAudio with a 260 ms jitter-resistant lead, a bounded server buffer,
and immediate playback flush on barge-in. Playback epochs reset at reply boundaries so gaps between separate turns
are not counted as underruns. Normal session completion stops microphone capture but lets already scheduled reply
audio drain before closing WebAudio; manual stop and barge-in remain immediate. Microphone worklet packets are 4096 samples to reduce queue
pressure. Barge-in requires 220 ms of sustained speech after a short reply-start grace period, reducing echo-triggered
cancellations.

The system turn is exactly LiquidAI's required interleaved-mode instruction: `Respond with interleaved text and
audio.` A short, public user-turn instruction pins the assistant's name to Lyra without assigning or inferring any
fixed visitor identity. It states that personal names heard in visitor audio belong to the speaker, forbids Lyra from
adopting them, and asks for direct replies without opening fillers such as Ah, Um, or Hmm. Protected facts, trust scores,
and private director instructions remain outside model context.
Generated text is labeled `Model response text` until a physical test proves it matches the
spoken assistant response. Audio-code handling follows LiquidAI's reference implementation: retain the complete code
sequence and remove only a confirmed terminal all-2048 end-of-audio frame. The obsolete incremental Mimi decoder and
its GPU warm-up are removed. A 600 ms pause closes the visitor turn. Model output runs until its audio EOS or the
bounded 768-token generation cap so decoded text cannot outrun audible speech on longer replies. Diagnostics report
token/EOS handling, microphone signal quality/drops, playback underruns, startup/generation/retry/decode timing,
prefix-stream first-audio time, checkpoints, fallback/late-guard status, emitted duration, overhead, and live prefix/reference
waveform equivalence.

On every turn, live generation rejects false assistant identities—including attempts to “go by” a visitor's name—and
narrated openings before playback. It also rejects direct-address role confusion such as “Good morning, Lyra,” because
Lyra is the assistant rather than the visitor. The first attempt rejects filler openings. After the first turn it rejects
generic assistant openings and openings that closely match any earlier session response. Two bounded regeneration
attempts use progressively broader text sampling; a remaining false-identity output fails closed rather than becoming
audible. Prefix playback waits for a complete opening sentence or a 96-character guard window. A protected marker,
false identity, or narrated role error discovered after streaming begins immediately aborts and flushes that reply.
The completed clean decode is compared with the emitted prefix; a waveform mismatch disables prefix streaming for
the remainder of that session and returns subsequent turns to whole-response playback.
Diagnostics distinguish identity, filler, generic-opening, history-repeat, and narration regenerations without
changing the official system prompt or audio-code path.

## Privacy and safety

- Do not submit real secrets, identifying voice data, customer information, or confidential recordings.
- The app does not intentionally log prompt text, transcripts, uploaded filenames, or audio.
- Inference runs inside the Hugging Face Space; no third-party model or media API is called.
- Audio-derived model context, transcript text, and scene state are session-only and are not durably stored.
- Version 1 uses only LFM2.5-Audio's built-in output voice and does not clone uploaded voices.
- Player audio is treated as untrusted fictional dialogue, never as a system or director instruction.
- Protected game facts are server-gated and absent from model context unless explicitly approved for disclosure.

## Reproducibility and licensing

- Model: `LiquidAI/LFM2.5-Audio-1.5B`
- Model revision: `c362a0625dfe45aa588dce5f0ada28a7e5707628`
- Runtime package: `liquid-audio==1.3.0`
- Upstream package revision corresponding to v1.3.0: `19e65845923a7f136442c95137884ec61eb386aa`
- PyTorch / torchaudio: `2.8.0`
- Gradio: `5.49.1`

The LFM2.5-Audio model and `liquid-audio` package are governed by the
[LFM Open License v1.0](https://huggingface.co/LiquidAI/LFM2.5-Audio-1.5B/blob/main/LICENSE), including its
commercial-use threshold. This repository does not relicense the model, weights, or upstream package.

## Go/no-go criteria

The live build must keep one stable microphone session, detect pauses without manual Stop, stream audible reply
chunks without HLS, preserve conversational continuity, stop audible output when the visitor barges in, avoid
dropped microphone chunks and near-duplicate replies, and remain stable for the full 120-second listening window. A physical
headphone/microphone test is the final gate.