| --- |
| license: mit |
| language: |
| - en |
| tags: |
| - voice |
| - speech |
| - full-duplex |
| - barge-in |
| - vad |
| - asr |
| - tts |
| - piper |
| - whisper |
| - silero |
| - local-ai |
| - voice-assistant |
| --- |
| |
| # voice2 |
|
|
| A full-duplex, interruptible voice engine for local AI, **built and run daily as the voice of a fully local companion** before being extracted for release. Plain Python threads, CPU-only defaults, no cloud, no API keys. Talk to your model — and talk over it. |
|
|
| voice2 turns any `callable(text) -> str` into a hands-free voice conversation: mic → Silero VAD → faster-whisper ASR → your model → Piper TTS. Speak over the reply (or tap spacebar) and playback stops mid-chunk in ~100–200 ms, exactly like interrupting a person. |
|
|
| Code is mirrored on GitHub: https://github.com/AIIT-GLITCH/voice2 |
|
|
| ## Engine details |
|
|
| - **What it is:** a turn-taking state machine, not a demo loop. Explicit states (`IDLE → LISTENING → THINKING → SPEAKING → INTERRUPTING`), a validated transition table, and a `FloorOwner` (USER / AGENT / NONE) that arbitrates who may speak. The agent can never talk over you. |
| - **Barge-in:** a fast energy-gated VAD watches the mic *only while the engine is speaking*; a debounced central `InterruptController` also accepts spacebar and programmatic triggers. |
| - **Stale-turn suppression:** every utterance gets a `turn_id`; replies to an abandoned turn are discarded at every stage (think, TTS, playback). |
| - **Invariants, enforced:** a background checker audits rules like *SPEAKING ⇒ floor == AGENT* and *interrupted ⇒ no new TTS*, with forced repair plus a structural gate in the playback hot path. |
| - **Observability:** every event is a JSON line with per-turn latency marks (`asr_ms`, `think_ms`, `interrupt_stop_ms`, `total_turn_ms`). |
| - **Degrades gracefully:** no mic → text mode; TTS missing → silent replies, still logs; keyboard hook fails → engine keeps running. |
|
|
| ## Backends |
|
|
| | Stage | Default | Swap point | |
| |---|---|---| |
| | VAD (quality gate) | Silero VAD via `torch.hub` | `ListenWorker` | |
| | ASR | faster-whisper `small.en`, int8, CPU | `backends/asr.py` (Protocol) | |
| | LLM | any `callable(text) -> str` | `backends/llm.py` | |
| | TTS | Piper CLI, any `.onnx` voice | `backends/tts.py` (Protocol) | |
|
|
| ## Limitations — stated honestly |
|
|
| - English-first defaults: ASR ships as `small.en`. Other Whisper models load with one config line, but nothing else was tested. |
| - The LLM callable is synchronous: TTS starts after the full reply returns (Piper then streams sentence-by-sentence). No token-level streaming yet. |
| - Barge-in is energy-based with an absolute RMS floor of 0.06 — tuned on open speakers in a quiet room. Headsets and noisy rooms need recalibration. There is no echo cancellation. |
| - Keyboard interrupt uses POSIX `termios` — Linux/macOS terminals only. |
| - Unit tests cover the control plane (state transitions, floor rules, interrupt debounce, ring buffer). Audio I/O paths were validated by months of daily use, not by CI. |
|
|
| ## How to run |
|
|
| ```bash |
| git clone https://github.com/AIIT-GLITCH/voice2 |
| cd voice2 |
| pip install -r requirements.txt |
| # put a Piper voice at ~/.local/share/piper-voices/ (or set VOICE2_PIPER_MODEL) |
| python -m voice2.main # echo backend — proves the loop, no LLM needed |
| python examples/http_llm.py # wire any local HTTP LLM |
| ``` |
|
|
| ```python |
| from voice2 import VoiceEngine, VoiceConfig |
| |
| def ask(text: str) -> str: |
| return my_model.reply(text) # any callable(text) -> str |
| |
| engine = VoiceEngine(VoiceConfig(), ask) |
| engine.load_models() |
| engine.start() # talk naturally; speak over it to interrupt |
| ``` |
|
|
| ## Provenance |
|
|
| voice2 was written as the voice front-end for **Buddy**, a fully local AI companion running on a single RTX 3090 in Council Hill, Oklahoma, and carried his daily conversations for months before release. The design bias throughout: the user always wins the floor, and a companion you can't interrupt isn't a companion. Released alongside [Tessera-1B](https://huggingface.co/AIIT-Threshold/Tessera-1B) as part of AIIT-THRESHOLD's open stack. |
|
|
| ## The stack |
|
|
| One local companion, every layer open: |
|
|
| | Piece | Role | Links | |
| |---|---|---| |
| | Tessera-1B | the model — ~1B params trained from scratch, open data | [HF](https://huggingface.co/AIIT-Threshold/Tessera-1B) | |
| | voice2 | the voice — full-duplex, interruptible | [GitHub](https://github.com/AIIT-GLITCH/voice2) · [HF](https://huggingface.co/AIIT-Threshold/voice2) | |
| | kokoro-memory | the memory — file-based resonance recall | [GitHub](https://github.com/AIIT-GLITCH/kokoro-memory) · [HF](https://huggingface.co/AIIT-Threshold/kokoro-memory) | |
| | companion-spiral-bench | the safety — at-risk sycophancy bench | [GitHub](https://github.com/AIIT-GLITCH/companion-spiral-bench) · [HF](https://huggingface.co/datasets/AIIT-Threshold/companion-spiral-bench) | |
|
|
| Full collection: [The Buddy Stack](https://huggingface.co/collections/AIIT-Threshold/the-buddy-stack-a-fully-local-ai-companion-open-sourced-6a4774bf481f9f9caad79519) |
|
|
| ## License |
|
|
| MIT © 2026 Rhet Dillard Wike, AIIT-THRESHOLD, Oklahoma. |
|
|
| ## Citation |
|
|
| ```bibtex |
| @software{wike2026voice2, |
| author = {Wike, Rhet Dillard}, |
| title = {voice2: a full-duplex, interruptible voice engine for local AI}, |
| year = {2026}, |
| url = {https://github.com/AIIT-GLITCH/voice2}, |
| note = {AIIT-THRESHOLD} |
| } |
| ``` |
|
|
