| Intended Task/Domain: |
Text to Speech |
| Model Type: |
Transformer Encoder, Transformer Decoder |
| Intended Users: |
People working with conversational AI models that need to generate speech from text for a single speaker. |
| Output: |
Speech |
| Describe how the model works: |
MagpieTTS is a text-to-speech model that synthesizes speech from input text, conditioned on an audio prompt embedding that captures the target speaker's acoustic characteristics. The model autoregressively predicts discrete audio codec tokens using a transformer encoder-decoder, with a Local Transformer refining the multi-codebook predictions and frame stacking (factor = 2) reducing sequence length. The model supports two inference modes: standard and long-form. In standard mode, the entire input text is passed through the encoder. The encoder output is concatenated with the audio prompt embedding and fed to the decoder, which autoregressively generates the corresponding speech. In long-form mode, the input text is split into sentences, each processed independently while maintaining a rolling window of prior text and attention states. This preserves prosodic continuity across sentence boundaries. |
| Name the adversely impacted groups this has been tested to deliver comparable outcomes regardless of: |
Not Applicable |
| Technical Limitations & Mitigation: |
There are two modes of inference, namely, standard and long-form. In standard mode, this model can generate up to thirty (30) seconds of multilingual (En, Es, De, Fr, Vi, It, Zh, Hi, Ja, Ar, pt-BR, Ko) speech at a time. In long-form mode, the model performs optimally when the input text contains punctuation and capitalization. The model was trained on a mix of publicly available speech datasets and internally recorded datasets in twelve languages. As a result, it is not suitable for speech generation in any language other than the twelve languages mentioned. We have removed zero-shot capabilities of this model for security purposes. Text normalization is required. |
| Verified to have met prescribed NVIDIA quality standards: |
Yes |
| Performance Metrics: |
character error rate (CER) and speaker verification model speaker similarity (SV-SSIM) |
| Potential Known Risks: |
In standard inference mode, if the input text is too long, the model might not produce speech. In long-form inference mode, if the input text does not contain punctuation and capitalization, the generated speech may not be 100% accurate. |
| Licensing: |
GOVERNING TERMS: Your use of this model is governed by the NVIDIA Open Model License Agreement. |
