README.md

---
language:
- en
license: openmdw-1.1
license_link: LICENSE.md
pipeline_tag: automatic-speech-recognition
base_model:
- nvidia/nemotron-3.5-asr-streaming-0.6b
tags:
- decision-model
- automatic-speech-recognition
- asr
- feature-extraction
- speech-to-speech
- duplex
- full-duplex
- turn-taking
- endpoint-detection
- barge-in
- backchannel
- streaming
- audio
- gru
- pytorch
- safetensors
---

<div align="center">
  <img src="assets/logo.jpeg" alt="NineNineSix" width="460" />
  <h1>TYGER</h1>
  <p>
    <a href="https://discord.gg/NzP3rjB4SB"><img src="https://img.shields.io/badge/Discord-Join_the_community-5865F2" alt="Discord" /></a>
    <a href="https://www.nineninesix.ai/"><img src="https://img.shields.io/badge/Website-NineNineSix-111827" alt="Website" /></a>
    <a href="https://github.com/nineninesix-ai/tyger-s2s"><img src="https://img.shields.io/badge/Voice_agent-tyger--s2s-181717" alt="Voice agent: tyger-s2s" /></a>
    <a href="LICENSE.md"><img src="https://img.shields.io/badge/License-OpenMDW_1.1-blue" alt="License: OpenMDW-1.1" /></a>
  </p>
</div>

**TYGER** - An open-weight speech decision model for real-time, full-duplex voice agents. It knows when to respond, keep listening, yield, and backchannel.


## Example

<audio controls src="https://cdn-uploads.huggingface.co/production/uploads/64fab67bd268b2f1ad8a826b/MG5P0_PpMeOp8OKydH2TC.wav"></audio>

## Use it

[**tyger-s2s**](https://github.com/nineninesix-ai/tyger-s2s) is our open-source
voice agent built on this model (Apache-2.0). You talk to it in the browser, as
on a phone call:

- this model is its ears: it transcribes you, and knows when you've finished,
  when you cut in, and when to say "mm-hm";
- LLM writes the replies, TTS speaks them in a cloned voice;
- it comes as a [Docker image](https://github.com/nineninesix-ai/tyger-s2s#docker),
  and can be the voice backend of
  [your own web app](https://github.com/nineninesix-ai/tyger-s2s#api-mode-tyger-behind-your-own-app).

To run the model in your own code, start from tyger-s2s's runtime
([`app/plugins/ears/tyger/`](https://github.com/nineninesix-ai/tyger-s2s/tree/main/app/plugins/ears/tyger)).

## The model

| | |
| --- | --- |
| Speech recognition | [NVIDIA Nemotron 3.5 ASR Streaming 0.6B](https://huggingface.co/nvidia/nemotron-3.5-asr-streaming-0.6b), unchanged, with its decoder and tokenizer |
| Turn-taker | a causal GRU: 3 layers, width 768 |
| Parameters | 652M: 638M Nemotron + 14M GRU, in FP32 (2.61 GB) |
| Input | two aligned mono 16 kHz streams: the user's mic and the agent's playback |
| Features | Nemotron's layer 15 (1,024 dims), one frame per 80 ms, no lookahead |
| Streaming | 320 ms hops; Nemotron re-encodes 8s window per stream at each hop |
| Output, per frame | the turn state (listen, hold, speak, backchannel) and hazard, barge-in, backchannel and endpoint scores |
| State per conversation | 9 KiB of GRU state, besides the audio windows |
| Language | English |

This repo holds weights and configs, no inference code.

## Events

| Event | When | The app | Default threshold |
| --- | --- | --- | ---: |
| Endpoint | the user has finished | answers | 0.91 |
| Barge-in | the user cuts in while the agent talks | stops the reply at once | 0.50 |
| Backchannel | a "mm-hm" fits while the user talks | plays one; the user keeps the turn | 0.40 |

Guards: 6 warm-up frames, at least 350 ms of speech above -50 dB, 1.2 s between
backchannels, and an endpoint after 1.2 s of silence even if the score stays
low. The state input is LISTEN while the agent listens and SPEAK while it
talks, never the model's own guess. The barge-in and backchannel defaults are
looser than the evaluation's (0.93 and 0.80) and have no measurements of their
own: tune them in your app (tyger-s2s's page has sliders for that).

## Training

The GRU, its input layers and its heads were trained from scratch on
[Open Yap 1K v1.0](https://theagenticdatacompany.com/open-yap-1k) (1,395
training, 107 development and 100 test conversations), with Nemotron frozen.
The labels come automatically from when each side speaks, not from human
annotators: the turn state, an upcoming speaking onset (hazard), when to yield
to an interruption, when a backchannel fits, and whether a pause ends the turn.
One training seed.

## Evaluation

Offline scores for the released weights, at endpoint 0.91, barge-in 0.93 and
backchannel 0.80, set on development data and frozen before test scoring.
[evaluation/README.md](evaluation/README.md) has the definitions and caveats,
and [metrics.json](evaluation/metrics.json) every number.

![Endpoint latency, early cut-ins and misses across three evaluation sets](assets/endpoint-evaluation.png)

**Endpoints,**:

| Set | Turns | Latency, median / mean | Early cut-ins | No endpoint within 1.2 s | Pauses with a false endpoint |
| --- | ---: | ---: | ---: | ---: | ---: |
| Open Yap 1K test | 1,000 | 252.5 / 465.0 ms | 29.90% | 7.40% | 415 of 1,000 |
| Full-Duplex-Bench v1.0, CANDOR | 119 | 307.5 / 383.0 ms | 25.21% | 5.04% | 123 of 277 |
| Internal, not public | 242 | 252.5 / 286.5 ms | 12.40% | 0.00% | 186 of 539 |

- Latency runs from the end of the user's turn, and counts only endpoints that
fired: read it with the early cut-ins (an endpoint before the turn really
ended) and the misses.
- With the fallback off, Open Yap's mean latency is 436.5 ms, and 16.0% of turns
get no endpoint within 1.2 s.

**Turn state, barge-in and backchannel** (Open Yap 1K test):

| Metric | Value |
| --- | ---: |
| Turn-state macro-F1 | 0.9436 |
| Endpoint ROC-AUC / AP | 0.9839 / 0.8149 |
| Barge-in ROC-AUC / AP | 0.9797 / 0.6657 |
| Backchannel ROC-AUC / AP | 0.8976 / 0.3004 |
| Barge-ins caught within 640 ms / median delay | 25.32% / 400 ms |
| False barge-ins per hour of clean speech | 0.00 (none in 4.72 h; not a guarantee) |
| Backchannels caught / median delay | 14.86% / -160 ms (before the labeled moment) |
| Backchannels fired per hour of listening | 135.08 |

**Speed:** the GRU takes 0.805 ms per frame (median; p90 0.823 ms) on an RTX
4090 at batch 1, with 108.76 MiB of peak GPU memory. That leaves out Nemotron
and waiting for audio. In tyger-s2s, the whole model takes ~4 GB of GPU memory
for 4 conversations.

**Limits:**
- Offline diagnostics only: no live voice-agent benchmark, and no new ASR
  error rates ([NVIDIA's card](upstream/NVIDIA_MODEL_CARD.md) has its own).
- One training seed, and the evaluation sets also guided development. The
  internal set can't be reproduced.
- The Full-Duplex-Bench rows are adapted v1.0 CANDOR replays with a silent
  agent channel, not official full-agent scores. FDB-v3 isn't evaluated.

## License

TYGER's weights and files are OpenMDW-1.1. NVIDIA's original
card, figures and license are unchanged in `upstream/` and `licenses/`, and
`manifest.json` lists every file's size and SHA-256. See
[LICENSE.md](LICENSE.md) and [NOTICE.md](NOTICE.md).

## Links

- **Voice agent:** [nineninesix-ai/tyger-s2s](https://github.com/nineninesix-ai/tyger-s2s)
- **Website:** [nineninesix.ai](https://www.nineninesix.ai/)
- **Discord:** [NineNineSix community](https://discord.gg/NzP3rjB4SB)
- **Training data:** [Open Yap 1K](https://theagenticdatacompany.com/open-yap-1k)
- **ASR:** [NVIDIA Nemotron](https://huggingface.co/nvidia/nemotron-3.5-asr-streaming-0.6b)

## Acknowledgments

We thank NVIDIA for Nemotron, The Agentic Data Company for Open Yap 1K, and the
authors of [Full-Duplex-Bench](https://arxiv.org/abs/2503.04721) for its
evaluation probes. Related work, not bundled or reproduced here:
[Full-Duplex-Bench-v3](https://arxiv.org/abs/2604.04847),
[Moshi](https://arxiv.org/abs/2410.00037),
[SALM-Duplex](https://arxiv.org/abs/2505.15670),
[Freeze-Omni](https://arxiv.org/abs/2411.00774),
[PersonaPlex](https://arxiv.org/abs/2602.06053) and
[Audio Flamingo 3](https://arxiv.org/abs/2507.08128).

## Citation

If you use TYGER, please cite the model and its training data:

```bibtex
@misc{tyger_2026,
  author    = {{NineNineSix}},
  title     = {{TYGER}: Speech Recognition and Duplex Turn Control},
  year      = {2026},
  publisher = {NineNineSix},
  url       = {https://huggingface.co/nineninesix/tyger}
}

@misc{openyap1k,
  title     = {Open Yap 1K: Channel-Separated English Natural Two-Speaker Conversations},
  author    = {{The Agentic Data Company}},
  year      = {2026},
  version   = {1.0},
  publisher = {The Agentic Data Company},
  url       = {https://theagenticdatacompany.com/open-yap-1k}
}
```

## References

<details>
<summary>BibTeX for the ASR component and related work</summary>

```bibtex
@misc{nvidia_nemotron_3_5_asr,
  author = {{NVIDIA}},
  title  = {Nemotron 3.5 ASR Streaming 0.6B},
  url    = {https://huggingface.co/nvidia/nemotron-3.5-asr-streaming-0.6b},
  note   = {Bundled revision ea30d66debe3740a08b573244286791d423d6b3e}
}

@misc{lin2025fullduplexbench,
  title = {Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilities},
  author = {Lin, Guan-Ting and Lian, Jiachen and Li, Tingle and Wang, Qirui and Anumanchipalli, Gopala and Liu, Alexander H. and Lee, Hung-yi},
  year = {2025}, eprint = {2503.04721}, archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.04721}
}

@misc{lin2026fullduplexbenchv3,
  title = {Full-Duplex-Bench-v3: Benchmarking Tool Use for Full-Duplex Voice Agents Under Real-World Disfluency},
  author = {Lin, Guan-Ting and Chen, Chen and Chen, Zhehuai and Lee, Hung-yi},
  year = {2026}, eprint = {2604.04847}, archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2604.04847}
}

@misc{defossez2024moshi,
  title = {Moshi: a speech-text foundation model for real-time dialogue},
  author = {D\'{e}fossez, Alexandre and Mazar\'{e}, Laurent and Orsini, Manu and Royer, Am\'{e}lie and P\'{e}rez, Patrick and J\'{e}gou, Herv\'{e} and Grave, Edouard and Zeghidour, Neil},
  year = {2024}, eprint = {2410.00037}, archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.00037}
}

@misc{hu2025salmduplex,
  title = {SALM-Duplex: Efficient and Direct Duplex Modeling for Speech-to-Speech Language Model},
  author = {Hu, Ke and Hosseini-Asl, Ehsan and Chen, Chen and Casanova, Edresson and Ghosh, Subhankar and {\.{Z}}elasko, Piotr and Chen, Zhehuai and Li, Jason and Balam, Jagadeesh and Ginsburg, Boris},
  year = {2025}, eprint = {2505.15670}, archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.15670}
}

@misc{wang2024freezeomni,
  title = {Freeze-Omni: A Smart and Low Latency Speech-to-speech Dialogue Model with Frozen LLM},
  author = {Wang, Xiong and Li, Yangze and Fu, Chaoyou and Shen, Yunhang and Xie, Lei and Li, Ke and Sun, Xing and Ma, Long},
  year = {2024}, eprint = {2411.00774}, archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.00774}
}

@misc{roy2026personaplex,
  title = {PersonaPlex: Voice and Role Control for Full Duplex Conversational Speech Models},
  author = {Roy, Rajarshi and Raiman, Jonathan and Lee, Sang-gil and Ene, Teodor-Dumitru and Kirby, Robert and Kim, Sungwon and Kim, Jaehyeon and Catanzaro, Bryan},
  year = {2026}, eprint = {2602.06053}, archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.06053}
}

@misc{goel2025audioflamingo3,
  title = {Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Models},
  author = {Goel, Arushi and Ghosh, Sreyan and Kim, Jaehyeon and Kumar, Sonal and Kong, Zhifeng and Lee, Sang-gil and Yang, Chao-Han Huck and Duraiswami, Ramani and Manocha, Dinesh and Valle, Rafael and Catanzaro, Bryan},
  year = {2025}, eprint = {2507.08128}, archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.08128}
}
```

</details>