feb_version_update

README.md

@@ -11,6 +11,8 @@ language:
 - vi
 - zh
 - fr
+- hi
+- ja
 metrics:
 - cer
 library_name: nemo
@@ -58,14 +60,16 @@ img {
 
 💻 **NeMo Framework**: [github.com/NVIDIA/NeMo](https://github.com/NVIDIA/NeMo)
 
+**UPDATED in February with support for 2 new languages (Hindi and Japanese)**
+
 ### Description:
-The model is a text-to-speech model that generates speech in 5 different English speakers - Sofia, Aria, Jason, Leo, [John Van Stan](https://librivox.org/reader/9017?primary_key=9017&search_category=reader&search_page=1&search_form=get_results&search_order=alpha). Each speakers can speak seven different languages (En, Es, De, Fr, Vi, It, Zh). The model predicts discrete audio codec tokens autoregressively using a transformer encoder-decoder architecture. It employs multi-codebook prediction (typically 8 codebooks) with optional local transformer refinement for high-quality audio generation, and leverages techniques like attention priors, classifier-free guidance (CFG), and Group Relative Policy Optimization (GRPO) for improved alignment. The generated codecs are then converted to speech waveform using [NanoCodec](https://huggingface.co/nvidia/nemo-nano-codec-22khz-1.89kbps-21.5fps).
+The model is a text-to-speech model that generates speech in 5 different English speakers - Sofia, Aria, Jason, Leo, [John Van Stan](https://librivox.org/reader/9017?primary_key=9017&search_category=reader&search_page=1&search_form=get_results&search_order=alpha). Each speakers can speak nine different languages (En, Es, De, Fr, Vi, It, Zh, Hi, Ja). The model predicts discrete audio codec tokens autoregressively using a transformer encoder-decoder architecture. It employs multi-codebook prediction (typically 8 codebooks) with optional local transformer refinement for high-quality audio generation, and leverages techniques like attention priors, classifier-free guidance (CFG), and Group Relative Policy Optimization (GRPO) for improved alignment. The generated codecs are then converted to speech waveform using [NanoCodec](https://huggingface.co/nvidia/nemo-nano-codec-22khz-1.89kbps-21.5fps).
 
 This model is ready for commercial use. <br>
 
 ### Key Features of the model
 
-- **Multilingual Support** — Synthesizes natural speech in English, French, Spanish, German, French, Vietnamese, Italian, and Mandarin
+- **Multilingual Support** — Synthesizes natural speech in English, French, Spanish, German, French, Vietnamese, Italian, Mandarin, Hindi, Japanese.
 - **Expressive Voices** — Multiple voice options with emotional tones and gender variations including 4 proprietary voices and 1 public voice
 - **Text Normalization** — Built-in text normalization for handling numbers, abbreviations, and special characters for all languages except Vietnamese
 
@@ -81,7 +85,7 @@ This model is ready for commercial use. <br>
 Global
 
 ### Use Case: <br>
-Wherever NVIDIA’s text-to-speech (TTS) models are used, Multilingual MagpieTTS can generate multilingual speech for a given text.
+For streaming voice agent use-cases, For Offline speech generation from text, in multiple languages.
 
 ## Model Architecture:
 **Architecture Type:** Transformer Encoder, Transformer Decoder, Local Transformer, and feedforward layers  <br>
@@ -291,7 +295,7 @@ When `--run_evaluation` is enabled, the following metrics are computed:
 The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment. <br>
 
 ## Model Version(s):
-Multilingual MagpieTTS-357M <br>
+Multilingual MagpieTTS-357M **V26.02**<br>
 
 ## Training and Evaluation Datasets:
 
@@ -313,6 +317,9 @@ The following datasets were used to train the model, including additional datase
 * [InfoRe-1 Vi](https://huggingface.co/datasets/doof-ferb/infore1_25hours)
 * Internal Vietnamese Dataset
 * Internal Mandarin Dataset
+* [AI4bharat Hi](https://huggingface.co/datasets/ai4bharat/Kathbath)
+* [Emilia YODAS Ja](https://huggingface.co/datasets/amphion/Emilia-Dataset)
+* King ASR Ja: Internal Dataset
 
 
 **Data Modality** <br>
@@ -320,7 +327,7 @@ The following datasets were used to train the model, including additional datase
 
  <!-- 291.6 hifi, 36K hifi2, 585 Libri -->
 **Audio Training Data Size** <br>
-* 60,000 Hours <br>
+* 50,000 Hours <br>
 
 **Data Collection Method by dataset** <br>
 * Publicly available dataset <br>
@@ -333,7 +340,7 @@ The following datasets were used to train the model, including additional datase
 Number of data items in training set: 38k hours
 Modality: Audio (speech signal)
 Nature of the content: Audio books
-Language: Multilingual (En, Es, De, Fr, Vi, It, Zh)
+Language: Multilingual (En, Es, De, Fr, Vi, It, Zh, Hi, Ja)
 Sensor Type: Microphones <br>
 
 ### Evaluation Dataset:
@@ -361,10 +368,10 @@ Sensor Type: Microphones <br>
 
 |                       | CER (%)| SV-SSIM   |
 | --------------------- | ------ | ------ |
-| LibriTTS test-clean   |  0.38  |  0.823 |
+| LibriTTS test-clean   |  0.34  |  0.835 |
 | Spanish CML           |  1.0   |  0.719 |
-| French CML            |  2.8   |  0.708 |
-| German CML            |  1.1   |  0.646 |
+| French CML            |  2.7   |  0.708 |
+| German CML            |  0.66   |  0.646 |
 
 * This result is based on the MagpieTTS model ([Huggingface Checkpoint](https://huggingface.co/nvidia/multilingual_magpietts_2512))
 
@@ -378,7 +385,7 @@ Sensor Type: Microphones <br>
 
 ## Technical Limitations & Mitigation:
 
-There are two modes of inference, namely, standard and long-form. In standard mode, this model can generate up to twenty (20) seconds of multilingual (En, Es, De, Fr, Vi, It, Zh) speech at a time. In long-form mode, the model performs optimally when the input text contains punctuation and capitalization. The model was trained on a mix of publicly available speech datasets and internally recorded datasets in seven languages. As a result, it is not suitable for speech generation in any language other than the seven languages mentioned. We have removed zero-shot capabilities of this model for this release. Text normalization is required.
+There are two modes of inference, namely, standard and long-form. In standard mode, this model can generate up to twenty (20) seconds of multilingual (En, Es, De, Fr, Vi, It, Zh, Hi, Ja) speech at a time. In long-form mode, the model performs optimally when the input text contains punctuation and capitalization. The model was trained on a mix of publicly available speech datasets and internally recorded datasets in nine languages. As a result, it is not suitable for speech generation in any language other than the nine languages mentioned. We have removed zero-shot capabilities of this model for this release. Text normalization is required.
 
 
 ## Ethical Considerations:

magpie_tts_multilingual_357m.nemo

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b254d46a4cf3cdb1d43611cf5d1677b17baab0f7ccd9e2789780d0e5a292c17e
-size 1208944640
+oid sha256:3111c41d88de500dbc0cee70802c0ae7fb54915c46f29a2391a4510081f76a94
+size 1208883200