Instructions to use Chechen123/Duopra with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use Chechen123/Duopra with Transformers:
# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("Chechen123/Duopra", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
Chiyu Chen1,2* · Pengzhou Cheng3* · Yuezhao Ma1 · Xiaocheng Li1 · Can Liu1 · Haonan Hu2 · Yao Yao1 · Yifei Yang1
Yijie Lu1 · Taolei Wang1 · Yansi Li1 · Muhuan Wu4 · Zhuosheng Zhang1† · Gongshen Liu1† · Chenglu Sun2‡
1Shanghai Jiao Tong University · 2Tencent · 3Shanghai University · 4University of Bristol
*Equal contribution · †Corresponding authors · ‡Project lead
English · 简体中文
Overview · Demo · How it works · Results · Model · Quick start · Tools · Citation
3:06, in Chinese with English at the end · also on the project page
News
- 2026-11-16 The demo video is out: three minutes in the Duopra app — a tool call while it talks, the backend model called as one more tool, and interruptions it keeps up with.
- 2026-11-15 The project page is live, with both recorded sessions replayed second by second, the full results and every number. Code and model weights are out; the preprint follows soon.
Overview
Full-duplex models already know when to listen and when to speak. Duopra also knows when to act: it calls tools itself, mid-sentence, through actions that are never voiced.
Every second, a single policy reads the new audio and any tool results that have come back, and emits three things at once: whether to listen or to speak, the words to voice, and an action that is never spoken. A backend model, if you connect one, is simply one more tool.
|
Listens while it speaks Every second it decides whether to listen or to speak. It takes the turn when you finish, holds back in a mid-sentence pause and gives way when you cut in. |
Acts mid-sentence, silently Tool calls go out on a private channel while Duopra is still talking. The call itself is never spoken — only its result is. |
Keeps up with corrections Change a value halfway through a request and only the final value reaches the tool: |
|
Several steps in one request Two changes asked for in one hesitant breath become two calls, and each is reported only after its result comes back. |
Calls a backend like any tool Connect a backend model and it becomes one more tool, |
Bring your own tools A tool is a JSON schema — a name, a description and its parameters — supplied at run time. Your calendar, your database, your home: describe it, and Duopra can call it. How to add one → |
Demo
Session 01 · a correction. The user asks for 100 euros in US dollars, then corrects it to 150 before finishing. Duopra waits for the turn to end, starts answering and — while it is still speaking — sends get_exchange_rate(amount: 150, from_currency: "EUR", to_currency: "USD"). It reports 165 once the result is back. The abandoned 100 never reaches the tool.
Session 02 · two steps. Two autopay changes asked for in one hesitant breath become two modify_autopay calls; the second goes out while Duopra is still reading the result of the first.
▶ Listen to both sessions, second by second, on the project page
Two selected sessions from the evaluation, not a success rate. User audio is from Full-Duplex-Bench v3; the tools are benchmark mocks.
How it works
One policy decides what to say and what to do. Every second, Duopra reads the new audio and any tool results, then emits
- — turn state: listen or speak
- — words, voiced by the talker
- — action: a tool call, never voiced
One causal policy, conditioned on the audio so far, the observations injected so far and its own past outputs. Conversation and tool use are decided together, not handed between components.
- The backend is one more tool. A background agent is exposed as
query(task), whose argument is a self-contained description of the user's task. Calling a tool directly or delegating is the same kind of decision, made by the same policy — there is no external dispatcher. - Results land on unit boundaries. A runtime executes each call asynchronously and injects the result at the first one-second boundary after it returns. The model keeps listening and speaking while it waits, and reports a value only once it has arrived.
Figure 1. Frontend responsibility in an evolving FDB v3 request (ecommerce_19; schematic). The user corrects both the product (laptop to tablet) and the quantity (two to one); add_to_cart depends on the product ID returned by search_products. (a) When domain schemas are available only to the backend, the frontend may need successive hand-offs to convey the evolving request, making it harder to preserve corrections and tool dependencies. (b) With schemas visible, Duopra can issue the corrected domain-tool calls directly without a hand-off, while query(task) remains available. Crossed-out values and curved arrows denote superseded requests. Table 18(a) of the paper reports an observed delegated success on this request.
Trained in three stages
One stage for each problem: the form of an action, the decision to act, and a voice that drifts when the thinker changes.
| Stage | What it trains | Outcome |
|---|---|---|
| 1 · Protocol SFT the form of an action |
Supervised fine-tuning of the thinker on synthetic sessions that demonstrate the protocol: call tools by schema, wait for the result before reporting it, ask only for missing values. | Pass@1 26.0% — calls are well formed, but 55% of sessions still end without one. |
| 2 · Full-duplex RL the decision to act |
Dr. GRPO on a thinker LoRA, eight rollouts per task, in a sandbox that plays the same scripted user to every rollout. Time-anchored deadlines and outcome tiers reward calls that are timely, complete and grounded. | Pass@1 26.0 → 47.7% — sessions without a call fall from 55% to 19%. |
| 3 · Projector-only repair a voice that stays intelligible |
Post-training shifts the thinker's hidden states, which condition the talker. Only the projector between them is retrained, on cached features; text, control tokens and calls are untouched. | Chinese character error rate 6.01 → 2.58% with the thinker held fixed; English WER 9.74 → 7.66% (221-utterance projector-selection set). |
Figure 2 · the three training stages
Figure 2. Duopra's three training stages on Realtime-Venus-Audio (ht: thinker hidden state; P: projector; ct: talker conditioning). Stage 1 trains the thinker on unit-level sessions and yields Duopra-SFT. Stage 2 trains a thinker LoRA with Dr. GRPO in the sandbox; Stage 2a yields Duopra-RL1 and Stage 2b yields Duopra-RL. Stage 3 freezes the Duopra-RL thinker and trains only its projector, producing the final Duopra model.
Results
Pass@1 on Full-Duplex-Bench v3: 49.3% with a backend, 47.7% calling tools itself. A session passes only when every executed call matches the gold call: right tool, every argument, no extra call. Published results are shown next to Duopra as reference points, not as a common-protocol ranking.
Left, commercial systems as reported; right, open-source systems and Duopra; one scale for both. Full-Duplex-Bench v3: 100 real, disfluent English recordings and 12 simulated APIs. Duopra runs in our harness — three seeds, 300 sessions per configuration, judged by Qwen3.8-27B; ± is the standard deviation across seeds. Hatched bars are published results, measured with each source's own harness and judge (mostly GPT-4o) — reference points, not a common-protocol ranking: MOSS-Live reports a tool-call success rate, DuplexAgent-RSI is measured on VoiceChat with exact-string argument matching, and Realtime-Venus-Audio offline has the tool definitions in the prompt. * Commercial system.
Every metric for the systems in the chart
| System | Tool | Arg. | Resp. | Pass@1 |
|---|---|---|---|---|
| Commercial systems*, as reported | ||||
| GPT-Realtime | 87.6 | 68.0 | 79.2 | 60.0 |
| Gemini Live 3.1 | 81.7 | 58.8 | 71.8 | 54.0 |
| MOSS-Live | n/a | n/a | n/a | 51.0 |
| Gemini Live 2.5 | 78.6 | 59.3 | 55.4 | 49.0 |
| Cascaded | 80.3 | 56.2 | 60.0 | 45.0 |
| Grok | 79.7 | 54.2 | 61.7 | 43.0 |
| Ultravox v0.7 | 79.4 | 51.3 | 51.0 | 41.0 |
| Open-source systems, as reported | ||||
| Hu et al., Qwen3-235B backend | 71.7 | 55.2 | 67.0 | 48.0 |
| DuplexAgent-RSI | 97.4 | 57.0 | n/a | 47.0 |
| Realtime-Venus-Audio, offline | 82.0 | 52.2 | n/a | 42.0 |
| Gander, GPT-5.6 | 75.9 | 50.3 | 49.0 | 40.0 |
| NemotronLabs VoiceChat | 82.5 | 42.2 | n/a | 33.0 |
| Duopra, our harness | ||||
| Duopra, calling tools itself | 76.7 | 56.2 | 66.3 | 47.7 ± 3.5 |
| Duopra, Qwen3-8B backend | 81.8 | 57.0 | 70.6 | 44.7 ± 5.7 |
| Duopra, DeepSeek backend | 71.8 | 55.0 | 69.0 | 49.3 ± 3.5 |
Tool: tool-name F1, which involves no judge (Hu et al. report tool-selection accuracy). Arg.: argument accuracy. Resp.: response quality. n/a: not reported. Bold: best within each group. DuplexAgent-RSI reaches 49.0 on Qwen-Realtime. Calling tools itself is the paper's Direct setting; a named backend is its Backend setting, in which only query(task) is visible. Further open-source systems that we ran in our harness are in the paper's Tables 2 and 11.
Training cuts failures to act
The outcome of each of the 300 sessions, Duopra models calling tools themselves.
| Run | Passed | Wrong argument | Missing call | Extra call | No call |
|---|---|---|---|---|---|
| Duopra-SFT · after Stage 1 | 78 | 17 | 38 | 1 | 166 |
| Duopra-RL1 · after Stage 2a | 80 | 17 | 72 | 0 | 131 |
| Duopra · final model | 143 | 58 | 40 | 3 | 56 |
Multi-call requests become solvable: Duopra passes 28 of 102 sessions that need two or three calls, against 2 after SFT. The remaining headroom is in arguments — wrong arguments (19.3% of sessions) are now as common as sessions without a call (18.7%).
Learning to act, while taking turns better
| Metric | Duopra |
|---|---|
| Takes the turn after the user finishes ↑ FDB v1.0 · base model 59.1 | 75.6 |
| Takes the turn during a mid-sentence pause ↓ FDB v1.0 · base model 18.1 | 5.4 |
| Responds after an interruption ↑ FDB v1.0 · base model 79.5 | 91.3 |
| Takes the turn at all ↑ FDB v3, calling tools itself | 93.0 |
| VoiceBench, mean of nine categories ↑ | 55.9 |
| Factual-answer latency ↓ domain tools and backend both visible, Qwen3-8B | 10.42 s |
Three seeds, in our streaming pipeline; the base model is Realtime-Venus-Audio, which Duopra is built on. Pass@1 is read from the call log rather than the transcript; judged metrics use Qwen3.8-27B, tool-name F1 involves no judge. Factual-answer latency is measured until the factual answer, including runtime and tool execution.
Model
| Base model | Realtime-Venus-Audio, on the MiniCPM-o 4.5 architecture |
| Components | Whisper-medium audio encoder · Qwen3-8B thinker · talker with a streaming flow-matching speech decoder |
| Training | Stage 1 protocol SFT · Stage 2 full-duplex RL (Dr. GRPO on a thinker LoRA, merged into the weights) · Stage 3 projector-only repair |
| Parameters | 9.4B; the thinker's layers and the projector are stored in FP32, the rest in BF16 |
| Input | Streaming audio in one-second units, and each tool result once it returns |
| Output | Every second: listen or speak, the words to voice, and an action that is never voiced — a tool call |
| Languages | English, Chinese |
| Runtime | The Duopra app (GitHub) on an NVIDIA GPU with BF16 support; verified on an NVIDIA A800 |
Quick start
These weights run in the Duopra app, which streams the audio in one-second units, executes the tool calls and returns their results. The app is on GitHub.
1. Get the app and the weights
git clone https://github.com/cyChen2003/duopra.git && cd duopra
pip install -U huggingface_hub
hf auth login # the repository is gated: accept its conditions on this page first
hf download Chechen123/Duopra --local-dir weights/duopra # the merged model
2. Run the voice service (GPU)
Linux x86_64 with Conda, an NVIDIA GPU with BF16 support and a driver compatible with CUDA 12.8.
export DUPLEX_MODEL_DIR=$PWD/weights/duopra
export DUPLEX_GPU_ID=0
python scripts/inventory-model.py --model "$DUPLEX_MODEL_DIR" --output private/model-inventory.json
bash scripts/start.sh --check # runtime and model, without starting
bash scripts/start.sh
Open http://localhost:23479, allow the microphone and start talking — or upload an audio file. On a remote server: ssh -N -L 23479:127.0.0.1:23479 user@server.
3. Connect a backend model (optional)
python -m pip install --upgrade --target "${DUPLEX_HARNESS_DEPS:-$CONDA_PREFIX/harness-deps}" \
-r backend/harness/requirements.txt
In the app, open 后台助手 / Background assistant, enter the base URL, model name and API key — DeepSeek is preset, and OpenAI-compatible endpoints work too — save, and run a manual test. Duopra now sees the backend as one more tool, query(task). Calendar and tasks live in the personal workspace. The container image already includes these dependencies.
The repository README also covers the CPU-only interface preview, container deployment and the checks.
Tools
The app ships 68 tools in seven domains. Each is a JSON schema — name, description, parameters, returns — in backend/tools/registry.json, and the interface labels where every result comes from. You can add your own.
| Domain | Tools | Live data | Local workspace | Sandbox fixture |
|---|---|---|---|---|
| Personal | 17 | – | 10 | 7 |
| Information | 11 | 5 | – | 6 |
| Travel | 8 | 2 | – | 6 |
| Daily life | 8 | – | – | 8 |
| Culture | 8 | – | – | 8 |
| Environment | 8 | – | – | 8 |
| Facilities | 8 | – | – | 8 |
| All | 68 | 7 | 10 | 51 |
- Live data — weather, daylight, time zones, reference exchange rates, stock quotes, market indices and currency trends, from public sources.
- Local workspace — calendar and tasks in SQLite on your machine; the model can use them only after you allow personal tools for the session.
- Sandbox fixture — fixed demonstration results, marked as such in the interface: they check the call, not the world.
One entry of the registry
{
"name": "weather.day",
"description": "查询指定城市某天的气温与降水",
"read_only": true,
"parameters": {
"type": "object",
"properties": { "city": { "type": "string" }, "date": { "type": "string" } },
"required": ["city", "date"],
"additionalProperties": false
},
"returns": {
"type": "object",
"properties": { "temperature_c": { "type": "integer" }, "rain_probability": { "type": "number" } },
"required": ["temperature_c", "rain_probability"]
}
}
Add your own tools
Beyond the 68, any tool you describe is one Duopra can find and call — in conversation, and from the backend model too. A tool is one JSON file in backend/tools/custom/ (or the directory named by DUPLEX_CUSTOM_TOOLS): the registry's schema format, plus the endpoint that does the work.
1 · Describe it in backend/tools/custom/home.lights.json:
{
"name": "home.lights",
"description": "Turn the lights in one room of the home on or off",
"parameters": {
"type": "object",
"properties": {
"room": { "type": "string", "description": "The room, such as kitchen or bedroom" },
"on": { "type": "boolean" }
},
"required": ["room", "on"],
"additionalProperties": false
},
"returns": {
"type": "object",
"properties": { "room": { "type": "string" }, "on": { "type": "boolean" } },
"required": ["room", "on"]
},
"endpoint": "http://127.0.0.1:8765/lights",
"read_only": false
}
2 · Serve it. Duopra sends each call's arguments to the endpoint as one JSON object in a POST request — {"room": "kitchen", "on": true} — and the endpoint replies with one JSON object that matches returns. Any language will do; example_endpoint.py is one in thirty lines of the Python standard library.
3 · Restart Duopra. The tool is listed as Custom. Ask for it, and Duopra finds its schema and calls it mid-conversation, like any other tool.
- A tool with
"read_only": falsechanges something, so it runs only after you allow personal tools for the session. - Without an
endpoint, anexample.resultanswers as a marked fixture — a way to try a schema before the service exists. - Definitions are checked when the service starts: a reused name or a schema keyword the validator cannot enforce is refused, with the file named. Every result is checked against
returns. - A container image includes the directory as it was when the image was built; point endpoints at an address the container can reach. All fields:
backend/tools/custom/README.md.
Citation
Abstract
Full-duplex interaction models exhibit strong conversational agency, enabling natural, low-latency turn-taking, backchannels, and interruptions. However, successful task completion further requires task agency that timely tool invocation and argument revision following user corrections. Frontend–backend agents separate initiation from execution: the frontend decides whether and what to request, while the backend determines how. Execution can be offloaded, but unsent requests cannot be executed, and incomplete ones can misdirect the backend. We investigate this issue on FD-Bench-v3, most failures of Realtime-Venus-Audio + Harness in our streaming setting occur before a domain tool executes.
We introduce Duopra, a full-duplex interaction model that learns task execution alongside conversational control. Duopra treats the backend as a tool, accepting a self-contained natural-language request. Direct calls and backend delegation share an action interface and requirements for faithful initiation and correct execution. Its three-stage training combines protocol SFT, full-duplex RL rewarding timely and complete execution, and projector-only repair of talker conditioning after thinker updates. With a fixed backend, Pass@1 rises from 4.7% to 44.7%; Duopra achieves 47.7% under Direct (26.0% after SFT) and 49.3% under Backend with DeepSeek. It improves three FDB v1.0 takeover rates over the base model. These results show that task agency can coexist with key conversational agency.
@misc{duopra2026,
title = {Duopra: From Full-Duplex Conversation to Full-Duplex Agency},
author = {Chen, Chiyu and Cheng, Pengzhou and Ma, Yuezhao and Li, Xiaocheng
and Liu, Can and Hu, Haonan and Yao, Yao and Yang, Yifei
and Lu, Yijie and Wang, Taolei and Li, Yansi and Wu, Muhuan
and Zhang, Zhuosheng and Liu, Gongshen and Sun, Chenglu},
year = {2026},
note = {Preprint forthcoming},
url = {https://cychen2003.github.io/duopra/}
}
License
This repository is released under the MIT license. Duopra is fine-tuned from Realtime-Venus-Audio, which is derived from MiniCPM-o 4.5; both are released under the Apache License 2.0, and their attribution is kept in NOTICE. The app's licences, and how it handles credentials and personal data, are described in the GitHub repository's NOTICE and SECURITY.md.
Duopra · Listen. Speak. Act.
Project page · Code · 简体中文 · Back to top
- Downloads last month
- 2
Model tree for Chechen123/Duopra
Base model
inclusionAI/Realtime-Venus