--- license: cc-by-4.0 language: - en base_model: kyutai/moshiko-pytorch-bf16 datasets: - ai4bharat/Svarah tags: - speech-to-speech - full-duplex - spoken-dialogue - conversational-ai - voice-agent - voice-assistant - real-time - indian-english - indian-accent - india - customer-support - call-center - barge-in - moshi - mimi - lora - audio pipeline_tag: audio-to-audio --- # Roxi-Duplex: a voice agent you can interrupt, in Indian English Most voice agents take turns: you speak, you wait, they speak. Roxi-Duplex does not wait. It is a **full-duplex speech-to-speech** model that listens while it talks, so callers can interrupt it mid-sentence, murmur "haan, okay" while it speaks, and get a response in a natural **Indian-English** voice that opens with "Welcome to Voz Vox. My name is Roxi. How may I help you today?" To our knowledge this is one of the first openly released Indian-English adaptations of a full-duplex speech-to-speech model. It is a compact LoRA adapter (370 MB) for Kyutai's Moshi 7B, so you get frontier duplex behavior plus an Indian support-agent persona without downloading a new foundation model. Paper: "Roxi-Duplex: Low-Resource Indian-English Adaptation of a Full-Duplex Speech-to-Speech Model with Synthetic Two-Channel Training Data" (https://doi.org/10.5281/zenodo.21445239). See the Citation section below. ## Why you might want this - **True barge-in.** Duplex is architectural, not a VAD hack: the model tracks both audio streams on one timeline, so interruptions and backchannels work the way they do between humans. - **Indian-English out of the box.** Accent and delivery learned from Indian speakers, for the hundreds of millions of users that US-accented agents serve poorly. - **Support-agent behavior built in.** Greetings, bookings, order status, complaints, payments. Prompt it with a caller and it answers like a call-center agent, not a chatbot. - **Real time on one GPU.** Moshi streams at a real-time factor of about 0.34 on an A100 40 GB, with about 200 ms theoretical latency. The adapter adds nothing to inference cost. - **A reproducible recipe.** Open Indian full-duplex dialogue data does not exist, so we synthesized it. The full pipeline is documented below and cheap to rerun (training takes 17 minutes on one A100). ## Quick facts | Field | Value | |---|---| | Base model | kyutai/moshiko-pytorch-bf16 (7B, CC-BY 4.0) | | Audio codec | Mimi (streaming, 12.5 Hz frames, 24 kHz) | | Method | LoRA rank 64, scaling 2.0, 1500 steps, via Kyutai moshi-finetune | | Adapter size | 370 MB (safetensors) | | Training data | 150 synthetic two-channel support conversations, about 94 minutes | | Assistant channel | Roxi TTS (Indian-English, 1.7B MOSS-TTS-Local fine-tune) speaking scripted support dialogues | | User channel | Real Indian-English speakers from ai4bharat/Svarah (117 speakers, CC-BY 4.0) | | Training cost | About 17 minutes on one rented A100 40 GB | ## How the data was made There is no open Indian-English full-duplex dialogue corpus, so we built one: 1. Scripted VozVox support dialogues (greetings, bookings, complaints, payments) with Indian names, cities, and numbers written as words. 2. Assistant turns rendered with an Indian-English TTS voice (Roxi), silence-trimmed and time-stretched 1.25x with WSOLA for a natural pace. WSOLA matters: phase-vocoder stretching made the voice sound robotic. 3. User turns taken from real Svarah recordings across India. 4. Both sides placed on a shared stereo timeline with turn gaps, backchannels, and overlaps, plus word-level alignments for Moshi's inner-monologue text stream. ## Usage Requires a Linux GPU with the `moshi` package (Triton is needed for the real-time compiled path, so native Windows is not supported for real-time use). ```bash pip install moshi ``` ```python import torch from huggingface_hub import hf_hub_download from moshi.models.loaders import CheckpointInfo, get_lora_moshi from moshi.models import LMGen adapter = hf_hub_download("IOTEverythin/roxi-duplex", "lora.safetensors") info = CheckpointInfo.from_hf_repo("kyutai/moshiko-pytorch-bf16") mimi = info.get_mimi(device="cuda") lm = info.get_moshi(device="cuda", dtype=torch.bfloat16) lm = get_lora_moshi(lm, adapter, 64, 2.0, dtype=torch.bfloat16, device="cuda", fuse_lora=True) lm_gen = LMGen(lm, temp=0.7, temp_text=0.7) # Stream user audio through mimi.encode and lm_gen.step exactly as with base Moshi. ``` Two tips from our experiments: - Prompt the model with real user audio. Feeding only silence makes any Moshi-family model produce unfocused speech. - If you fine-tune further, stay light. Around rank 64 and 1500 steps was the sweet spot; heavier adapters (rank 96, 3000 steps) kept the accent but degraded intelligibility. ## Limitations - Proof-of-concept scale: 150 synthetic conversations from four scripted scenario templates. Coverage outside customer-support topics is limited. - The dialogue structure is synthetic; real-call turn-taking dynamics may differ. - English only (Indian-English accent); no Hindi code-switching yet. - Inherits all Moshi limitations and requires a GPU for real-time use. ## License and attribution Released under **CC-BY 4.0**, matching the base model. This work builds on: - **Moshi and Mimi** by Kyutai (kyutai/moshiko-pytorch-bf16, CC-BY 4.0). Defossez et al., "Moshi: a speech-text foundation model for real-time dialogue". - **Svarah** by AI4Bharat (ai4bharat/Svarah, CC-BY 4.0), used for the user audio channel. - The assistant voice derives from our Roxi TTS, fine-tuned on the IIT-Madras Indic TTS English set. Required notice: COPYRIGHT 2016 TTS Consortium, TDIL, Meity, represented by Hema A. Murthy and S. Umesh, Department of Computer Science and Engineering and Electrical Engineering, IIT Madras. ALL RIGHTS RESERVED. - Trained with Kyutai's moshi-finetune. See also our Indian-English TTS models: IOTEverythin/roxi-tts-pro (1.7B premium) and IOTEverythin/roxi-tts-v3.1 (0.1B real-time). ## Citation If you use this model, the data recipe, or the RoxiDuplex-Eval benchmark, please cite: ```bibtex @misc{roxiduplex2026, title = {Roxi-Duplex: Low-Resource Indian-English Adaptation of a Full-Duplex Speech-to-Speech Model with Synthetic Two-Channel Training Data}, author = {A, Joshua Nishanth Tarun and A, Joel Ajitesh Varun}, year = {2026}, month = {july}, publisher = {Zenodo}, doi = {10.5281/zenodo.21445239}, url = {https://doi.org/10.5281/zenodo.21445239}, note = {Preprint} } ``` ## Responsible use This model speaks with a synthetic voice derived from consented and licensed datasets. Do not use it to impersonate real people or for fraud, social engineering, or deception. Disclose AI-generated audio where required by law or policy. Provided as is, without warranty.