| Field | Response |
| :------------------------------------------------------------------------------------------------------|:--------------------------------------------------------------------------------- |
| Intended Task/Domain: | Speech To Speech Conversational Chat Agent |
| Model Type: | Speech Encoder (Fast Conformer), Nemotron Nano v2 LLM backbone (hybrid Mamba/Transformer), EAR TTS Speech Decoder (ConvNet, Transformer) |
| Intended Users: | NemotronLabs VoiceChat is targeted for researchers, developers, and professionals in the field of natural language processing (NLP) and speech technology for purposes such as automatic speech recognition (ASR), text-to-speech synthesis (TTS), and voice assistant development. |
| Output: | Speech, Text |
| Describe how the model works: | NemotronLabs VoiceChat is a real-time speech-to-speech conversational model that jointly performs streaming speech understanding and speech generation. The model operates on continuous audio encoded using a fast conformer module before the resulting audio tokens are handed to a Nemotron nano V2 9b LLM backbone to predict text tokens which are fed to an EAR-TTS decoder [2] generating the agent's speech. |
| Name the adversely impacted groups this has been tested to deliver comparable outcomes regardless of: | Not Applicable |
| Technical Limitations & Mitigation: | NemotronLabs VoiceChat is trained with no more than 2-min long audio context windows. Conversational context beyond this window may not be retained reliably. <br>NemotronLabs VoiceChat has been optimized to achieve a trade-off between general knowledge and natural conversation, hence it may not be as performant as the LLM backbone it is derived from, namely NVIDIA Nemotron Nano v2, in terms of knowledge, instruction-following capabilities and safety. <br> The model can make reasoning errors or provide incorrect/incomplete information NemotronLabs VoiceChat was not explicitly trained for reasoning or alignment. Its performance on tasks requiring multi-step reasoning, arithmetic, or safety-aligned behavior may therefore be limited. <br>Further, the model may not yet be able to systematically handle user backchannelling. <br> As far as tool calling is concerned, we recommend a maximum of 5 tools per session; more tools may degrade performance. Also, the model cannot yet reliably call multiple tools simultaneously. Long tool responses may cause delays before the agent speaks (mitigate with on-hold messages). The user cannot interrupt the agent during tool calling execution. Further, in mixed conversations (general chat + tool requests), the model may answer from its own knowledge instead of calling the appropriate tool. <br>Finally, the model is not suitable for noisy or highly reverberant environments, especially where background speech may occur. |
| Verified to have met prescribed NVIDIA quality standards: | Yes |
| Performance Metrics: | Knowledge performance metrics include accuracy (%), GPT-scores (1-5) and refusal rate (%).<br>Conversational dynamics metrics include takeover rate (TOR); latency (seconds) for interruptions, pause handling, and turn-taking; frequency and Jensen–Shannon divergence (JSD) for backchannels; question-answering accuracy (score); and speaker similarity (SSIM). |
| Potential Known Risks: | The model may present errors in speech understanding and/or pronunciation. Additionally, the training data consists exclusively of English speech. The model cannot handle non-English languages. NemotronLabs VoiceChat was trained primarily on assistant-style and general conversational interactions. Its behavior may degrade when applied outside these domains. For security and privacy reasons, NemotronLabs VoiceChat does not support generating voice prompts from real user voice recordings. |
| Licensing: | GOVERNING TERMS: The use of this model is governed by the [OpenMDW License Agreement, version 1.1](https://github.com/OpenMDW/OpenMDW/blob/main/1.1/LICENSE.OpenMDW-1.1). |
