Joshuant commited on
Commit
194be9c
·
verified ·
1 Parent(s): f8827fd

sharper card: hook, why-you-want-this, discoverability tags

Browse files
Files changed (1) hide show
  1. README.md +49 -29
README.md CHANGED
@@ -9,36 +9,50 @@ tags:
9
  - speech-to-speech
10
  - full-duplex
11
  - spoken-dialogue
 
12
  - voice-agent
 
 
13
  - indian-english
14
  - indian-accent
 
15
  - customer-support
 
 
16
  - moshi
 
17
  - lora
18
  - audio
19
  pipeline_tag: audio-to-audio
20
  ---
21
 
22
- # Roxi-Duplex: full-duplex Indian-English speech-to-speech voice agent
23
-
24
- Roxi-Duplex is a LoRA adapter for Kyutai's **Moshi** (kyutai/moshiko-pytorch-bf16), a 7B
25
- full-duplex speech-to-speech model. The adapter shifts Moshi toward an **Indian-English
26
- customer-support agent**: it listens and speaks on a shared timeline (true full-duplex, so it
27
- supports interruptions and backchannels), responds in an Indian-English voice, and behaves like
28
- a support agent. Prompted with a caller's audio, it produces replies such as "Welcome to Voz
29
- Vox. My name is Roxi. How may I help you today?"
30
-
31
- Built for the VozVox voice-agent platform. To our knowledge this is one of the first openly
32
- released Indian-English adaptations of a full-duplex speech-to-speech model.
33
-
34
- ## What it does
35
-
36
- - Full-duplex spoken dialogue: models two audio streams (its own voice and the user's) on one
37
- timeline, inherited from Moshi, so barge-in and backchannels are native rather than bolted on.
38
- - Indian-English accent and delivery, learned from Indian speakers.
39
- - Customer-support behavior: greetings, bookings, order status, complaints, payments.
40
- - Streams in real time on a single A100-class GPU (Moshi runs at real-time factor about 0.34
41
- on an A100 40 GB with the compiled path).
 
 
 
 
 
 
 
42
 
43
  ## Quick facts
44
 
@@ -50,18 +64,19 @@ released Indian-English adaptations of a full-duplex speech-to-speech model.
50
  | Adapter size | 370 MB (safetensors) |
51
  | Training data | 150 synthetic two-channel support conversations, about 94 minutes |
52
  | Assistant channel | Roxi TTS (Indian-English, 1.7B MOSS-TTS-Local fine-tune) speaking scripted support dialogues |
53
- | User channel | Real Indian-English speakers from ai4bharat/Svarah (CC-BY 4.0) |
54
- | Hardware used | One rented A100 40 GB; training takes about 17 minutes |
55
 
56
  ## How the data was made
57
 
58
- Open Indian-English full-duplex dialogue data does not exist, so it was synthesized:
59
 
60
  1. Scripted VozVox support dialogues (greetings, bookings, complaints, payments) with Indian
61
  names, cities, and numbers written as words.
62
  2. Assistant turns rendered with an Indian-English TTS voice (Roxi), silence-trimmed and
63
- time-stretched 1.25x with WSOLA for a natural pace.
64
- 3. User turns taken from real Svarah recordings (117 speakers across India).
 
65
  4. Both sides placed on a shared stereo timeline with turn gaps, backchannels, and overlaps,
66
  plus word-level alignments for Moshi's inner-monologue text stream.
67
 
@@ -89,8 +104,12 @@ lm_gen = LMGen(lm, temp=0.7, temp_text=0.7)
89
  # Stream user audio through mimi.encode and lm_gen.step exactly as with base Moshi.
90
  ```
91
 
92
- Important: prompt the model with real user audio. Feeding only silence makes any Moshi-family
93
- model produce unfocused speech.
 
 
 
 
94
 
95
  ## Limitations
96
 
@@ -99,8 +118,6 @@ model produce unfocused speech.
99
  - The dialogue structure is synthetic; real-call turn-taking dynamics may differ.
100
  - English only (Indian-English accent); no Hindi code-switching yet.
101
  - Inherits all Moshi limitations and requires a GPU for real-time use.
102
- - Do not fine-tune this recipe heavier than about rank 64 / 1500 steps; heavier adapters
103
- degraded speech intelligibility in our experiments.
104
 
105
  ## License and attribution
106
 
@@ -115,6 +132,9 @@ Released under **CC-BY 4.0**, matching the base model. This work builds on:
115
  Engineering, IIT Madras. ALL RIGHTS RESERVED.
116
  - Trained with Kyutai's moshi-finetune.
117
 
 
 
 
118
  ## Responsible use
119
 
120
  This model speaks with a synthetic voice derived from consented and licensed datasets. Do not
 
9
  - speech-to-speech
10
  - full-duplex
11
  - spoken-dialogue
12
+ - conversational-ai
13
  - voice-agent
14
+ - voice-assistant
15
+ - real-time
16
  - indian-english
17
  - indian-accent
18
+ - india
19
  - customer-support
20
+ - call-center
21
+ - barge-in
22
  - moshi
23
+ - mimi
24
  - lora
25
  - audio
26
  pipeline_tag: audio-to-audio
27
  ---
28
 
29
+ # Roxi-Duplex: a voice agent you can interrupt, in Indian English
30
+
31
+ Most voice agents take turns: you speak, you wait, they speak. Roxi-Duplex does not wait.
32
+ It is a **full-duplex speech-to-speech** model that listens while it talks, so callers can
33
+ interrupt it mid-sentence, murmur "haan, okay" while it speaks, and get a response in a
34
+ natural **Indian-English** voice that opens with "Welcome to Voz Vox. My name is Roxi.
35
+ How may I help you today?"
36
+
37
+ To our knowledge this is one of the first openly released Indian-English adaptations of a
38
+ full-duplex speech-to-speech model. It is a compact LoRA adapter (370 MB) for Kyutai's
39
+ Moshi 7B, so you get frontier duplex behavior plus an Indian support-agent persona without
40
+ downloading a new foundation model.
41
+
42
+ ## Why you might want this
43
+
44
+ - **True barge-in.** Duplex is architectural, not a VAD hack: the model tracks both audio
45
+ streams on one timeline, so interruptions and backchannels work the way they do between
46
+ humans.
47
+ - **Indian-English out of the box.** Accent and delivery learned from Indian speakers, for
48
+ the hundreds of millions of users that US-accented agents serve poorly.
49
+ - **Support-agent behavior built in.** Greetings, bookings, order status, complaints,
50
+ payments. Prompt it with a caller and it answers like a call-center agent, not a chatbot.
51
+ - **Real time on one GPU.** Moshi streams at a real-time factor of about 0.34 on an A100
52
+ 40 GB, with about 200 ms theoretical latency. The adapter adds nothing to inference cost.
53
+ - **A reproducible recipe.** Open Indian full-duplex dialogue data does not exist, so we
54
+ synthesized it. The full pipeline is documented below and cheap to rerun (training takes
55
+ 17 minutes on one A100).
56
 
57
  ## Quick facts
58
 
 
64
  | Adapter size | 370 MB (safetensors) |
65
  | Training data | 150 synthetic two-channel support conversations, about 94 minutes |
66
  | Assistant channel | Roxi TTS (Indian-English, 1.7B MOSS-TTS-Local fine-tune) speaking scripted support dialogues |
67
+ | User channel | Real Indian-English speakers from ai4bharat/Svarah (117 speakers, CC-BY 4.0) |
68
+ | Training cost | About 17 minutes on one rented A100 40 GB |
69
 
70
  ## How the data was made
71
 
72
+ There is no open Indian-English full-duplex dialogue corpus, so we built one:
73
 
74
  1. Scripted VozVox support dialogues (greetings, bookings, complaints, payments) with Indian
75
  names, cities, and numbers written as words.
76
  2. Assistant turns rendered with an Indian-English TTS voice (Roxi), silence-trimmed and
77
+ time-stretched 1.25x with WSOLA for a natural pace. WSOLA matters: phase-vocoder
78
+ stretching made the voice sound robotic.
79
+ 3. User turns taken from real Svarah recordings across India.
80
  4. Both sides placed on a shared stereo timeline with turn gaps, backchannels, and overlaps,
81
  plus word-level alignments for Moshi's inner-monologue text stream.
82
 
 
104
  # Stream user audio through mimi.encode and lm_gen.step exactly as with base Moshi.
105
  ```
106
 
107
+ Two tips from our experiments:
108
+
109
+ - Prompt the model with real user audio. Feeding only silence makes any Moshi-family model
110
+ produce unfocused speech.
111
+ - If you fine-tune further, stay light. Around rank 64 and 1500 steps was the sweet spot;
112
+ heavier adapters (rank 96, 3000 steps) kept the accent but degraded intelligibility.
113
 
114
  ## Limitations
115
 
 
118
  - The dialogue structure is synthetic; real-call turn-taking dynamics may differ.
119
  - English only (Indian-English accent); no Hindi code-switching yet.
120
  - Inherits all Moshi limitations and requires a GPU for real-time use.
 
 
121
 
122
  ## License and attribution
123
 
 
132
  Engineering, IIT Madras. ALL RIGHTS RESERVED.
133
  - Trained with Kyutai's moshi-finetune.
134
 
135
+ See also our Indian-English TTS models: IOTEverythin/roxi-tts-pro (1.7B premium) and
136
+ IOTEverythin/roxi-tts-v3.1 (0.1B real-time).
137
+
138
  ## Responsible use
139
 
140
  This model speaks with a synthetic voice derived from consented and licensed datasets. Do not