Audio-to-Audio
Moshi
English
speech-to-speech
full-duplex
spoken-dialogue
conversational-ai
voice-agent
voice-assistant
real-time
indian-english
indian-accent
india
customer-support
call-center
barge-in
mimi
lora
audio
Instructions to use IOTEverythin/roxi-duplex with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Moshi
How to use IOTEverythin/roxi-duplex with Moshi:
# pip install moshi # Run the interactive web server python -m moshi.server --hf-repo "IOTEverythin/roxi-duplex" # Then open https://localhost:8998 in your browser
# pip install moshi import torch from moshi.models import loaders # Load checkpoint info from HuggingFace checkpoint = loaders.CheckpointInfo.from_hf_repo("IOTEverythin/roxi-duplex") # Load the Mimi audio codec mimi = checkpoint.get_mimi(device="cuda") mimi.set_num_codebooks(8) # Encode audio (24kHz, mono) wav = torch.randn(1, 1, 24000 * 10) # [batch, channels, samples] with torch.no_grad(): codes = mimi.encode(wav.cuda()) decoded = mimi.decode(codes) - Notebooks
- Google Colab
- Kaggle
sharper card: hook, why-you-want-this, discoverability tags
Browse files
README.md
CHANGED
|
@@ -9,36 +9,50 @@ tags:
|
|
| 9 |
- speech-to-speech
|
| 10 |
- full-duplex
|
| 11 |
- spoken-dialogue
|
|
|
|
| 12 |
- voice-agent
|
|
|
|
|
|
|
| 13 |
- indian-english
|
| 14 |
- indian-accent
|
|
|
|
| 15 |
- customer-support
|
|
|
|
|
|
|
| 16 |
- moshi
|
|
|
|
| 17 |
- lora
|
| 18 |
- audio
|
| 19 |
pipeline_tag: audio-to-audio
|
| 20 |
---
|
| 21 |
|
| 22 |
-
# Roxi-Duplex:
|
| 23 |
-
|
| 24 |
-
|
| 25 |
-
full-duplex speech-to-speech model
|
| 26 |
-
|
| 27 |
-
|
| 28 |
-
|
| 29 |
-
|
| 30 |
-
|
| 31 |
-
|
| 32 |
-
|
| 33 |
-
|
| 34 |
-
|
| 35 |
-
|
| 36 |
-
|
| 37 |
-
|
| 38 |
-
|
| 39 |
-
|
| 40 |
-
-
|
| 41 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 42 |
|
| 43 |
## Quick facts
|
| 44 |
|
|
@@ -50,18 +64,19 @@ released Indian-English adaptations of a full-duplex speech-to-speech model.
|
|
| 50 |
| Adapter size | 370 MB (safetensors) |
|
| 51 |
| Training data | 150 synthetic two-channel support conversations, about 94 minutes |
|
| 52 |
| Assistant channel | Roxi TTS (Indian-English, 1.7B MOSS-TTS-Local fine-tune) speaking scripted support dialogues |
|
| 53 |
-
| User channel | Real Indian-English speakers from ai4bharat/Svarah (CC-BY 4.0) |
|
| 54 |
-
|
|
| 55 |
|
| 56 |
## How the data was made
|
| 57 |
|
| 58 |
-
|
| 59 |
|
| 60 |
1. Scripted VozVox support dialogues (greetings, bookings, complaints, payments) with Indian
|
| 61 |
names, cities, and numbers written as words.
|
| 62 |
2. Assistant turns rendered with an Indian-English TTS voice (Roxi), silence-trimmed and
|
| 63 |
-
time-stretched 1.25x with WSOLA for a natural pace.
|
| 64 |
-
|
|
|
|
| 65 |
4. Both sides placed on a shared stereo timeline with turn gaps, backchannels, and overlaps,
|
| 66 |
plus word-level alignments for Moshi's inner-monologue text stream.
|
| 67 |
|
|
@@ -89,8 +104,12 @@ lm_gen = LMGen(lm, temp=0.7, temp_text=0.7)
|
|
| 89 |
# Stream user audio through mimi.encode and lm_gen.step exactly as with base Moshi.
|
| 90 |
```
|
| 91 |
|
| 92 |
-
|
| 93 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 94 |
|
| 95 |
## Limitations
|
| 96 |
|
|
@@ -99,8 +118,6 @@ model produce unfocused speech.
|
|
| 99 |
- The dialogue structure is synthetic; real-call turn-taking dynamics may differ.
|
| 100 |
- English only (Indian-English accent); no Hindi code-switching yet.
|
| 101 |
- Inherits all Moshi limitations and requires a GPU for real-time use.
|
| 102 |
-
- Do not fine-tune this recipe heavier than about rank 64 / 1500 steps; heavier adapters
|
| 103 |
-
degraded speech intelligibility in our experiments.
|
| 104 |
|
| 105 |
## License and attribution
|
| 106 |
|
|
@@ -115,6 +132,9 @@ Released under **CC-BY 4.0**, matching the base model. This work builds on:
|
|
| 115 |
Engineering, IIT Madras. ALL RIGHTS RESERVED.
|
| 116 |
- Trained with Kyutai's moshi-finetune.
|
| 117 |
|
|
|
|
|
|
|
|
|
|
| 118 |
## Responsible use
|
| 119 |
|
| 120 |
This model speaks with a synthetic voice derived from consented and licensed datasets. Do not
|
|
|
|
| 9 |
- speech-to-speech
|
| 10 |
- full-duplex
|
| 11 |
- spoken-dialogue
|
| 12 |
+
- conversational-ai
|
| 13 |
- voice-agent
|
| 14 |
+
- voice-assistant
|
| 15 |
+
- real-time
|
| 16 |
- indian-english
|
| 17 |
- indian-accent
|
| 18 |
+
- india
|
| 19 |
- customer-support
|
| 20 |
+
- call-center
|
| 21 |
+
- barge-in
|
| 22 |
- moshi
|
| 23 |
+
- mimi
|
| 24 |
- lora
|
| 25 |
- audio
|
| 26 |
pipeline_tag: audio-to-audio
|
| 27 |
---
|
| 28 |
|
| 29 |
+
# Roxi-Duplex: a voice agent you can interrupt, in Indian English
|
| 30 |
+
|
| 31 |
+
Most voice agents take turns: you speak, you wait, they speak. Roxi-Duplex does not wait.
|
| 32 |
+
It is a **full-duplex speech-to-speech** model that listens while it talks, so callers can
|
| 33 |
+
interrupt it mid-sentence, murmur "haan, okay" while it speaks, and get a response in a
|
| 34 |
+
natural **Indian-English** voice that opens with "Welcome to Voz Vox. My name is Roxi.
|
| 35 |
+
How may I help you today?"
|
| 36 |
+
|
| 37 |
+
To our knowledge this is one of the first openly released Indian-English adaptations of a
|
| 38 |
+
full-duplex speech-to-speech model. It is a compact LoRA adapter (370 MB) for Kyutai's
|
| 39 |
+
Moshi 7B, so you get frontier duplex behavior plus an Indian support-agent persona without
|
| 40 |
+
downloading a new foundation model.
|
| 41 |
+
|
| 42 |
+
## Why you might want this
|
| 43 |
+
|
| 44 |
+
- **True barge-in.** Duplex is architectural, not a VAD hack: the model tracks both audio
|
| 45 |
+
streams on one timeline, so interruptions and backchannels work the way they do between
|
| 46 |
+
humans.
|
| 47 |
+
- **Indian-English out of the box.** Accent and delivery learned from Indian speakers, for
|
| 48 |
+
the hundreds of millions of users that US-accented agents serve poorly.
|
| 49 |
+
- **Support-agent behavior built in.** Greetings, bookings, order status, complaints,
|
| 50 |
+
payments. Prompt it with a caller and it answers like a call-center agent, not a chatbot.
|
| 51 |
+
- **Real time on one GPU.** Moshi streams at a real-time factor of about 0.34 on an A100
|
| 52 |
+
40 GB, with about 200 ms theoretical latency. The adapter adds nothing to inference cost.
|
| 53 |
+
- **A reproducible recipe.** Open Indian full-duplex dialogue data does not exist, so we
|
| 54 |
+
synthesized it. The full pipeline is documented below and cheap to rerun (training takes
|
| 55 |
+
17 minutes on one A100).
|
| 56 |
|
| 57 |
## Quick facts
|
| 58 |
|
|
|
|
| 64 |
| Adapter size | 370 MB (safetensors) |
|
| 65 |
| Training data | 150 synthetic two-channel support conversations, about 94 minutes |
|
| 66 |
| Assistant channel | Roxi TTS (Indian-English, 1.7B MOSS-TTS-Local fine-tune) speaking scripted support dialogues |
|
| 67 |
+
| User channel | Real Indian-English speakers from ai4bharat/Svarah (117 speakers, CC-BY 4.0) |
|
| 68 |
+
| Training cost | About 17 minutes on one rented A100 40 GB |
|
| 69 |
|
| 70 |
## How the data was made
|
| 71 |
|
| 72 |
+
There is no open Indian-English full-duplex dialogue corpus, so we built one:
|
| 73 |
|
| 74 |
1. Scripted VozVox support dialogues (greetings, bookings, complaints, payments) with Indian
|
| 75 |
names, cities, and numbers written as words.
|
| 76 |
2. Assistant turns rendered with an Indian-English TTS voice (Roxi), silence-trimmed and
|
| 77 |
+
time-stretched 1.25x with WSOLA for a natural pace. WSOLA matters: phase-vocoder
|
| 78 |
+
stretching made the voice sound robotic.
|
| 79 |
+
3. User turns taken from real Svarah recordings across India.
|
| 80 |
4. Both sides placed on a shared stereo timeline with turn gaps, backchannels, and overlaps,
|
| 81 |
plus word-level alignments for Moshi's inner-monologue text stream.
|
| 82 |
|
|
|
|
| 104 |
# Stream user audio through mimi.encode and lm_gen.step exactly as with base Moshi.
|
| 105 |
```
|
| 106 |
|
| 107 |
+
Two tips from our experiments:
|
| 108 |
+
|
| 109 |
+
- Prompt the model with real user audio. Feeding only silence makes any Moshi-family model
|
| 110 |
+
produce unfocused speech.
|
| 111 |
+
- If you fine-tune further, stay light. Around rank 64 and 1500 steps was the sweet spot;
|
| 112 |
+
heavier adapters (rank 96, 3000 steps) kept the accent but degraded intelligibility.
|
| 113 |
|
| 114 |
## Limitations
|
| 115 |
|
|
|
|
| 118 |
- The dialogue structure is synthetic; real-call turn-taking dynamics may differ.
|
| 119 |
- English only (Indian-English accent); no Hindi code-switching yet.
|
| 120 |
- Inherits all Moshi limitations and requires a GPU for real-time use.
|
|
|
|
|
|
|
| 121 |
|
| 122 |
## License and attribution
|
| 123 |
|
|
|
|
| 132 |
Engineering, IIT Madras. ALL RIGHTS RESERVED.
|
| 133 |
- Trained with Kyutai's moshi-finetune.
|
| 134 |
|
| 135 |
+
See also our Indian-English TTS models: IOTEverythin/roxi-tts-pro (1.7B premium) and
|
| 136 |
+
IOTEverythin/roxi-tts-v3.1 (0.1B real-time).
|
| 137 |
+
|
| 138 |
## Responsible use
|
| 139 |
|
| 140 |
This model speaks with a synthetic voice derived from consented and licensed datasets. Do not
|