--- language: - en pipeline_tag: audio-classification tags: - turn-detection - end-of-turn - eou - endpointing - turn-taking - interruption-detection - barge-in - backchannel-detection - full-duplex - voice-activity-detection - conversational-ai - voice-agents - real-time - low-latency - streaming - audio - speech metrics: - recall - false-positive-rate - latency extra_gated_prompt: "Access to Echo Omni Duplex is provided through Zero Runtime. Visit https://zeroruntime.ai/?utm_source=huggingface&utm_medium=referral&utm_campaign=huggingface to get started." ---
![Echo Omni Duplex. 0.988 interruption recall, 0.965 end-of-turn recall, 2 channels, 590 ms interruption latency.](banner.svg)

Echo Omni Duplex listens to both sides of the conversation at once.
It knows who holds the floor, continuously, not once per turn.

Get Access   Documentation

🎧 Built for agents that listen while they speak, the full-duplex member of the Echo family.

A voice agent has to make one decision over and over: **speak, or stay quiet.** Most turn detectors answer it by waiting for silence, and a 500 ms gap looks identical whether someone finished their sentence, paused to think, or took a breath. Agents built on timers talk over people or leave them hanging. Echo Omni Duplex answers it differently. It listens to **both channels continuously**, the user and the agent, and keeps a running estimate of who holds the conversational floor. Because it never stops listening to either side, it can tell a real barge-in from an "mm-hm", and a finished sentence from a breath, while the audio is still arriving. --- ## 🎯 What it does Four things can happen around the moment a speaker stops or starts. Only two of them are events. ![Four situations: end of turn and interruption fire an event; mid-turn pause and backchannel do not.](states.svg) | Situation | What it means | |:--|:--| | **End of turn** | The speaker has given up the floor | | **Interruption** | The listener is taking the floor mid-turn | | **Mid-turn pause** | The speaker stopped, but has not finished | | **Backchannel** | "uh-huh", "right", "okay". Acknowledgement, not a bid for the floor | The last two are where most systems fail. A mid-turn pause looks like an ending, and a backchannel looks like a barge-in, so detectors fire on both. Echo Omni Duplex is built around telling them apart. --- ## 🔌 Input and output ![A conversation timeline on two channels: the user pauses mid-sentence and nothing fires; the turn really ends and an end-of-turn fires; the user backchannels twice while the agent is speaking and nothing fires; the user starts speaking while the agent is still mid-sentence and an interruption fires. A floor strip underneath shows which side holds the conversational floor at every moment.](how-it-works.svg) **Input**: two continuous audio streams: | Channel | Carries | |:--|:--| | **Channel 1** | The user's microphone | | **Channel 2** | The agent's own output audio | **Output**: emitted continuously as the conversation runs: | Field | Description | |:--|:--| | **Floor** | A running estimate of which side currently holds the conversational floor | | **Event** | A committed decision at the moment it is made: **end-of-turn**, **interruption**, or **backchannel**, an acknowledgement the agent should not stop for | | **Confidence** | The floor probability the decision was made on, reported with each event | Both channels are consumed as a live stream. There is no per-turn segmentation step and no transcript: the model sees the conversation as it happens. It runs in a live call, and how that is verified is set out under **Causality** below. > **Why channel 2 matters.** The strongest available cue for *"has this person finished?"* lives on the **other** side of the conversation, not the speaker's own. In a voice agent that other side is the agent, and its audio is known exactly. Feeding it in is the difference between guessing and knowing. --- ## 🧱 How it is built ![Two channels of audio are encoded separately by a frozen front end, combined by a causal model, turned into a running estimate of who is about to be speaking, and committed by a deterministic policy into end-of-turn, interruption and backchannel decisions.](architecture.svg) Echo Omni Duplex is a single neural network rather than a pipeline of separate models, and it works in two stages. **First**, each side of the conversation is encoded on its own by a pretrained front end that stays **frozen** during training, so the user's audio and the agent's audio are never summed into one signal. **Second**, a **causal** model combines the two encoded streams, reading only the recent past and never anything ahead of the current moment. Both channels pass through the same weights, because turn-taking is symmetric: whichever side is speaking, the question being asked is the same one. Because it only ever reads a bounded window of recent audio, the cost per frame stays constant however long the call runs. It works at **50 Hz** and, rather than classifying the present, predicts **who is about to be speaking**, which is what lets it commit before a silence has gone on long enough to be conclusive on its own. A deterministic commit policy sits on top and turns that continuous estimate into end-of-turn, interruption and backchannel decisions. It is separate from the network and tunable without retraining. It is **audio only**: no transcript, no speech recognition, no language model. Small enough to run far faster than real time on a single GPU, alongside speech-to-text, an LLM and text-to-speech. --- ## 🎓 Training and validation Echo Omni Duplex was trained on an internally curated corpus of **two-channel, full-duplex English conversations**. The corpus composition and the hours of data used are internal and not disclosed. The model was validated on a held-out split of that same corpus, and its operating point was then selected on the public TurnBench **dev** set using the official scorer. --- ## 🌍 Supported languages | Code | Language | |:----:|:---------| | `en` | English | --- ## 📊 Performance Measured on the **public development split** of TurnBench, the benchmark described below, using its own scorer. | Task | Recall ↑ | False-positive rate ↓ | Latency p50 ↓ | |:--|--:|--:|--:| | **End-of-turn** | **0.965** | **0.103** | 775 ms | | **Interruption** | **0.988** | **0.098** | 590 ms | **Latency here is not compute time.** It is the benchmark's own measure: how long after the real event the model committed to its decision. Most of that window is evidence the model is still waiting for, because deciding sooner means deciding on less. Every system on the benchmark pays it. The false-positive rates are the numbers users feel. On end-of-turn, a false positive is the agent claiming the floor during a mid-turn pause, cutting somebody off mid-sentence. On interruption, it is the agent stopping because it mistook a backchannel, a cough or its own echo for a barge-in. Echo Omni Duplex has also been evaluated on our own internal test set. Those results are not published here yet. > Results are measured on the benchmark described below. Performance may vary depending on language, deployment configuration, user behaviour and application requirements. --- ## 🏆 TurnBench [TurnBench](https://turnbench.sesame.com/) is Sesame AI Labs' public benchmark for conversational turn-taking: dual-channel conversation scored on recall, false-positive rate and latency. It scores **committed decisions**, not probability curves, which is what a deployed agent actually has to produce. At every moment a system either takes the floor or yields, and that is what gets measured. Its authors summarise the state of the field this way: > *"No system is fast, selective, and high-recall at the same time."* That is the trade-off Echo Omni Duplex was built against. On the interruption track it is **higher recall, lower false-positive rate, and roughly 300 milliseconds sooner** than the strongest published reference system, all three at once, scored the same way on the same data. It is not uniformly ahead. At least one published system commits interruptions faster still, at materially lower recall. The trade-off has moved; it has not disappeared. --- ## 🎚️ How the operating point was chosen The model outputs a continuous estimate of who holds the floor. A policy turns that into discrete events, using thresholds and delays that have to be fixed before submitting. How they were picked: * **On the dev split only.** Chosen against the official scorer on the public TurnBench dev set. The test split was never scored. * **Highest recall inside a 0.10 false-positive rate on dev.** That budget is the one the benchmark's own published baselines were selected under, and it is well inside the 0.15 test ceiling a submission has to clear, so the choice is deliberately conservative. * **One setting for both files.** The same policy produced the dev and test predictions. Nothing was tuned per split. * **We measured how much of the dev result was luck.** Split the 38 dev conversations in half at random, pick the setting on one half, score it on the other, and repeat. The median drop in recall between the half it was chosen on and the half it was not was **0.0052**, so the setting is not specific to the conversations that produced it. --- ## ⏱️ Causality The output at any moment uses audio up to that moment and nothing after it. **Timestamps.** Each one is a commit time: the moment the system decided, not the moment the event happened. We then report it 0.5 s later still, so a reported time is never earlier than the point at which the decision was made. The latency figures above are measured from those reported times, so they include that 0.5 s. **By construction.** The model reads only a bounded window of the recent past, the voice-activity gate is causal, and the input gain uses only audio already heard. **By measurement.** Construction arguments are easy to get wrong, so we tested it directly with a truncation check. * **What was tested.** The public **TurnBench dev split**, 38 two-channel recordings of real human conversation. Each was cut at random points, and the whole system was re-run from scratch on the shortened audio, then compared against the run on the full recording. * **What the result means.** If a system were secretly using audio from later in the file, removing that audio would change its earlier decisions. It does not: **330 of 336** event lists came back exactly as before, so the decisions stand on what had already been heard. | | | what it means | |---|---|---| | Random cuts | **84**, across 18 conversations | how many times the pipeline was re-run on shortened audio | | Events compared | **19,828** | every event early enough to be checked, summed over all cuts | | Event lists identical | **330 of 336** | each cut produces 4 lists, one per speaker per task; 330 came back exactly the same | | Voice activity before each cut | **identical** | zero frames differed, so nothing ahead of the model was reading the future | --- ## 🔬 Beyond the benchmark Echo Omni Duplex runs end to end in a live full-duplex voice agent, alongside real speech-to-text, a real LLM and real text-to-speech. Its streaming path reproduces the offline model output exactly: compared frame by frame against the reference implementation on real dual-channel audio, the floor estimate agrees to `0.00e+00`, and the same deterministic commit policy runs on top of it. --- ## 🎛️ The Echo family Echo Omni Duplex is the full-duplex member. The other three decide one turn at a time, which suits a half-duplex pipeline; Duplex decides continuously, which is what an agent needs when it is listening and speaking at once. | Model | Modality | Decides | Best for | |:--|:--|:--|:--| | `echo-small` | Semantic | Per turn | The lowest latency. The default when responsiveness matters most | | `echo-large` | Semantic | Per turn | Higher accuracy, when it matters more than raw speed | | `echo-omni` | Audio + semantic | Per turn | The widest coverage, with acoustic understanding on top of the semantics | | **`echo-omni-duplex`** | **Two-channel audio** | **Continuously** | **Full-duplex agents that listen while they speak** | --- ## 🚀 Get access Echo Omni Duplex is served for you. There is nothing to download, host, or keep running. Plug it straight into your voice pipeline alongside your existing STT, LLM and TTS. ```python # Set ZERORUNTIME_AUTH_TOKEN in your environment. from zeroruntime.inference import TurnDetector # Full duplex: both channels streamed continuously, events pushed back as they commit turn_detector = TurnDetector(model="echo-omni-duplex") ```
Get Access   Documentation
---
![Echo Omni Duplex](wordmark.svg) Echo Omni Duplex by Zero Runtime