Mirror of ai4bharat/indic-parler-tts (Apache-2.0) for ungated install

LICENSE

@@ -0,0 +1,202 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

NOTICE.md

@@ -0,0 +1,42 @@
+# NOTICE — provenance of this mirror
+
+This repository is a **verbatim public mirror** of the upstream model:
+
+> **Original**: [`ai4bharat/indic-parler-tts`](https://huggingface.co/ai4bharat/indic-parler-tts)
+> Maintained by [AI4Bharat](https://ai4bharat.iitm.ac.in/), IIT Madras,
+> in collaboration with the Hugging Face Parler-TTS team.
+
+The upstream model is licensed under the **Apache License 2.0**, which
+permits redistribution. This mirror exists so that downstream products
+(notably [prashnam-voice](https://github.com/prashnam/prashnam-voice))
+can install the model without users having to accept Hugging Face's
+gated-access flow on every fresh install.
+
+**No modifications have been made to the model weights or supporting
+files.** Bytes are byte-identical to the upstream snapshot at the time
+of cloning. If you want the canonical, latest version, please use the
+upstream repo and credit AI4Bharat.
+
+## Citation
+
+```
+@misc{ai4bharat-indic-parler-tts,
+  title   = {Indic Parler-TTS: Multilingual TTS for Indian Languages},
+  author  = {AI4Bharat and the Parler-TTS team},
+  year    = {2024},
+  howpublished = {\url{https://huggingface.co/ai4bharat/indic-parler-tts}}
+}
+```
+
+## License terms preserved
+
+This mirror redistributes the upstream model under the same Apache 2.0
+terms. The Apache 2.0 license requires:
+
+- Preservation of the license text and copyright notice (this NOTICE +
+  the upstream README's metadata satisfy that).
+- Documentation of any modifications made — **none have been made**.
+- A statement of changes, if any — **none**.
+
+For the full text of the Apache License 2.0, see
+<https://www.apache.org/licenses/LICENSE-2.0>.

README.md

@@ -1,3 +1,396 @@
----
-license: apache-2.0
----
+---
+library_name: transformers
+tags:
+- text-to-speech
+- annotation
+license: apache-2.0
+language:
+- en
+- as
+- bn
+- gu
+- hi
+- kn
+- ks
+- or
+- ml
+- mr
+- ne
+- pa
+- sa
+- sd
+- ta
+- te
+- ur
+- om
+pipeline_tag: text-to-speech
+inference: false
+datasets:
+- ai4b-hf/GLOBE-annotated
+---
+
+<img src="https://huggingface.co/datasets/parler-tts/images/resolve/main/Indic%20Parler-TTS.png" alt="Indic Parler Logo" width="800" style="margin-left:'auto' margin-right:'auto' display:'block'"/>
+
+# Indic Parler-TTS
+
+<a target="_blank" href="https://huggingface.co/spaces/ai4bharat/indic-parler-tts">
+  <img src="https://huggingface.co/datasets/huggingface/badges/raw/main/open-in-hf-spaces-sm.svg" alt="Open in HuggingFace"/>
+</a>
+
+**Indic Parler-TTS** is a multilingual Indic extension of [Parler-TTS Mini](https://huggingface.co/parler-tts/parler-tts-mini-v1.1).
+
+It is a fine-tuned version of [Indic Parler-TTS Pretrained](https://huggingface.co/ai4bharat/indic-parler-tts-pretrained), trained on a **1,806 hours** multilingual Indic and English dataset.
+
+**Indic Parler-TTS Mini** can officially speak in 20 Indic languages, making it comprehensive for regional language technologies, and in English. The **21 languages** supported are: Assamese, Bengali, Bodo, Dogri, English, Gujarati, Hindi, Kannada, Konkani, Maithili, Malayalam, Manipuri, Marathi, Nepali, Odia, Sanskrit, Santali, Sindhi, Tamil, Telugu, and Urdu.
+
+Thanks to its **better prompt tokenizer**, it can easily be extended to other languages. This tokenizer has a larger vocabulary and handles byte fallback, which simplifies multilingual training.
+
+🚨 This work is the result of a collaboration between the **HuggingFace audio team** and the **[AI4Bharat](https://ai4bharat.iitm.ac.in/) team**. 🚨
+
+
+## 📖 Quick Index
+* [👨‍💻 Installation](#👨‍💻-installation)
+* [🛠️ Key capabilities](#🛠️-key-capabilities)
+* [🎲 Using a random voice](#🎲-random-voice)
+* [🌍 Switching languages](#[🌍-switching-languages)
+* [🎯 Using a specific speaker](#🎯-using-a-specific-speaker)
+* [Some Description Examples](#some-description-examples)
+* [📐Evaluation](#📐-evaluation)
+* [Motivation](#motivation)
+* [Optimizing inference](https://github.com/huggingface/parler-tts/blob/main/INFERENCE.md)
+
+### 👨‍💻 Installation
+
+Using Parler-TTS is as simple as "bonjour". Simply install the library once:
+
+```sh
+pip install git+https://github.com/huggingface/parler-tts.git
+```
+
+## 🛠️ Key capabilities
+
+The model accepts two primary inputs:  
+1. **Transcript** - The text to be converted to speech.  
+2. **Caption** - A detailed description of how the speech should sound, e.g., "Leela speaks in a high-pitched, fast-paced, and cheerful tone, full of energy and happiness. The recording is very high quality with no background noise."  
+
+### Key Features  
+
+1. **Language Support**  
+   - **Officially supported languages**: Assamese, Bengali, Bodo, Dogri, Kannada, Malayalam, Marathi, Sanskrit, Nepali, English, Telugu, Hindi, Gujarati, Konkani, Maithili, Manipuri, Odia, Santali, Sindhi, Tamil, and Urdu.  
+   - **Unofficial support**: Chhattisgarhi, Kashmiri, Punjabi.  
+
+2. **Speaker Diversity**  
+   - **69 unique voices** across the supported languages.  
+   - Supported languages have a set of **recommended voices** optimized for naturalness and intelligibility.  
+   
+3. **Emotion Rendering**  
+   - **10 languages** officially support emotion-specific prompts: Assamese, Bengali, Bodo, Dogri, Kannada, Malayalam, Marathi, Sanskrit, Nepali, and Tamil.  
+   - Emotion support for other languages exists but has not been extensively tested.  
+   - **Available emotions** include: Command, Anger, Narration, Conversation, Disgust, Fear, Happy, Neutral, Proper Noun, News, Sad, and Surprise.  
+
+4. **Accent Flexibility**  
+   - The model **officially supports Indian English accents** through its English voices, providing clear and natural speech.  
+   - For other accents, the model allows customization by specifying accent details, such as "A male British speaker" or "A female American speaker," using style transfer for more dynamic and personalized outputs.  
+
+5. **Customizable Output**  
+   Indic Parler-TTS offers precise control over various speech characteristics using the **caption** input:  
+
+   - **Background Noise**: Adjust the noise level in the audio, from clear to slightly noisy environments.  
+   - **Reverberation**: Control the perceived distance of the voice, from close-sounding to distant-sounding speech.  
+   - **Expressivity**: Specify how dynamic or monotone the speech should be, ranging from expressive to slightly expressive or monotone.  
+   - **Pitch**: Modify the pitch of the speech, including high, low, or balanced tones.  
+   - **Speaking Rate**: Change the speaking rate, from slow to fast.  
+   - **Voice Quality**: Control the overall clarity and naturalness of the speech, adjusting from basic to refined voice quality.
+
+
+## 🎲 Random voice
+
+🚨 Unlike previous versions of Parler-TTS, here we use two tokenizers - one for the prompt and one for the description. 🚨
+
+**Indic Parler-TTS** has been trained to generate speech with features that can be controlled with a simple text prompt, for example:
+
+```py
+import torch
+from parler_tts import ParlerTTSForConditionalGeneration
+from transformers import AutoTokenizer
+import soundfile as sf
+
+device = "cuda:0" if torch.cuda.is_available() else "cpu"
+
+model = ParlerTTSForConditionalGeneration.from_pretrained("ai4bharat/indic-parler-tts").to(device)
+tokenizer = AutoTokenizer.from_pretrained("ai4bharat/indic-parler-tts")
+description_tokenizer = AutoTokenizer.from_pretrained(model.config.text_encoder._name_or_path)
+
+prompt = "Hey, how are you doing today?"
+description = "A female speaker with a British accent delivers a slightly expressive and animated speech with a moderate speed and pitch. The recording is of very high quality, with the speaker's voice sounding clear and very close up."
+
+description_input_ids = description_tokenizer(description, return_tensors="pt").to(device)
+prompt_input_ids = tokenizer(prompt, return_tensors="pt").to(device)
+
+generation = model.generate(input_ids=description_input_ids.input_ids, attention_mask=description_input_ids.attention_mask, prompt_input_ids=prompt_input_ids.input_ids, prompt_attention_mask=prompt_input_ids.attention_mask)
+audio_arr = generation.cpu().numpy().squeeze()
+sf.write("indic_tts_out.wav", audio_arr, model.config.sampling_rate)
+```
+
+Indic Parler-TTS provides highly effective control over key aspects of speech synthesis using descriptive captions. Below is a summary of what each control parameter can achieve:
+
+| **Control Type**        | **Capabilities**                                                                 |
+|--------------------------|----------------------------------------------------------------------------------|
+| **Background Noise**     | Adjusts the level of background noise, supporting clear and slightly noisy environments. |
+| **Reverberation**        | Controls the perceived distance of the speaker’s voice, allowing close or distant sounds. |
+| **Expressivity**         | Modulates the emotional intensity of speech, from monotone to highly expressive. |
+| **Pitch**                | Varies the pitch to achieve high, low, or moderate tonal output.                |
+| **Speaking Rate**        | Changes the speed of speech delivery, ranging from slow to fast-paced.          |
+| **Speech Quality**       | Improves or degrades the overall audio clarity, supporting basic to refined outputs. |
+
+
+## 🌍 Switching languages
+
+The model automatically adapts to the language it detects in the prompt. You don't need to specify the language you want to use. For example, to switch to Hindi, simply use an Hindi prompt:
+
+```py
+import torch
+from parler_tts import ParlerTTSForConditionalGeneration
+from transformers import AutoTokenizer
+import soundfile as sf
+
+device = "cuda:0" if torch.cuda.is_available() else "cpu"
+
+model = ParlerTTSForConditionalGeneration.from_pretrained("ai4bharat/indic-parler-tts").to(device)
+tokenizer = AutoTokenizer.from_pretrained("ai4bharat/indic-parler-tts")
+description_tokenizer = AutoTokenizer.from_pretrained(model.config.text_encoder._name_or_path)
+
+prompt = "अरे, तुम आज कैसे हो?"
+description = "A female speaker delivers a slightly expressive and animated speech with a moderate speed and pitch. The recording is of very high quality, with the speaker's voice sounding clear and very close up."
+
+description_input_ids = description_tokenizer(description, return_tensors="pt").to(device)
+prompt_input_ids = tokenizer(prompt, return_tensors="pt").to(device)
+
+generation = model.generate(input_ids=description_input_ids.input_ids, attention_mask=description_input_ids.attention_mask, prompt_input_ids=prompt_input_ids.input_ids, prompt_attention_mask=prompt_input_ids.attention_mask)
+audio_arr = generation.cpu().numpy().squeeze()
+sf.write("indic_tts_out.wav", audio_arr, model.config.sampling_rate)
+```
+
+## 🎯 Using a specific speaker
+
+To ensure speaker consistency across generations, this checkpoint was also trained on pre-determined speakers, characterized by name (e.g. Rohit, Karan, Leela, Maya, Sita, ...).
+To take advantage of this, simply adapt your text description to specify which speaker to use: `Divya's voice is monotone yet slightly fast in delivery, with a very close recording that almost has no background noise.`
+
+```py
+import torch
+from parler_tts import ParlerTTSForConditionalGeneration
+from transformers import AutoTokenizer
+import soundfile as sf
+
+device = "cuda:0" if torch.cuda.is_available() else "cpu"
+
+model = ParlerTTSForConditionalGeneration.from_pretrained("ai4bharat/indic-parler-tts").to(device)
+tokenizer = AutoTokenizer.from_pretrained("ai4bharat/indic-parler-tts")
+description_tokenizer = AutoTokenizer.from_pretrained(model.config.text_encoder._name_or_path)
+
+prompt = "अरे, तुम आज कैस��� हो?"
+description = "Divya's voice is monotone yet slightly fast in delivery, with a very close recording that almost has no background noise."
+
+description_input_ids = description_tokenizer(description, return_tensors="pt").to(device)
+prompt_input_ids = tokenizer(prompt, return_tensors="pt").to(device)
+
+generation = model.generate(input_ids=description_input_ids.input_ids, attention_mask=description_input_ids.attention_mask, prompt_input_ids=prompt_input_ids.input_ids, prompt_attention_mask=prompt_input_ids.attention_mask)
+audio_arr = generation.cpu().numpy().squeeze()
+sf.write("indic_tts_out.wav", audio_arr, model.config.sampling_rate)
+```
+
+The model includes **69 speakers** across 18 officially supported languages, with each language having a set of recommended voices for optimal performance. Below is a table summarizing the available speakers for each language, along with the recommended ones.
+
+Here is the table based on the provided data:
+
+| **Language**      | **Available Speakers**                                       | **Recommended Speakers**       |
+|--------------------|-------------------------------------------------------------|---------------------------------|
+| Assamese          | Amit, Sita, Poonam, Rakesh                                  | Amit, Sita                     |
+| Bengali           | Arjun, Aditi, Tapan, Rashmi, Arnav, Riya                    | Arjun, Aditi                   |
+| Bodo              | Bikram, Maya, Kalpana                                      | Bikram, Maya                   |
+| Chhattisgarhi     | Bhanu, Champa                                              | Bhanu, Champa                  |
+| Dogri             | Karan                                                      | Karan                          |
+| English           | Thoma, Mary, Swapna, Dinesh, Meera, Jatin, Aakash, Sneha, Kabir, Tisha, Chingkhei, Thoiba, Priya, Tarun, Gauri, Nisha, Raghav, Kavya, Ravi, Vikas, Riya | Thoma, Mary                     |
+| Gujarati          | Yash, Neha                                                 | Yash, Neha                     |
+| Hindi             | Rohit, Divya, Aman, Rani                                   | Rohit, Divya                   |
+| Kannada           | Suresh, Anu, Chetan, Vidya                                 | Suresh, Anu                    |
+| Malayalam         | Anjali, Anju, Harish                                       | Anjali, Harish                   |
+| Manipuri          | Laishram, Ranjit                                           | Laishram, Ranjit               |
+| Marathi           | Sanjay, Sunita, Nikhil, Radha, Varun, Isha                  | Sanjay, Sunita                 |
+| Nepali            | Amrita                                                     | Amrita                         |
+| Odia              | Manas, Debjani                                             | Manas, Debjani                 |
+| Punjabi           | Divjot, Gurpreet                                           | Divjot, Gurpreet               |
+| Sanskrit          | Aryan                                                      | Aryan                          |
+| Tamil             | Kavitha, Jaya                                        | Jaya                 |
+| Telugu            | Prakash, Lalitha, Kiran                                    | Prakash, Lalitha               |
+
+
+**Tips**:
+* We've set up an [inference guide](https://github.com/huggingface/parler-tts/blob/main/INFERENCE.md) to make generation faster. Think SDPA, torch.compile, batching and streaming!
+* Include the term "very clear audio" to generate the highest quality audio, and "very noisy audio" for high levels of background noise
+* Punctuation can be used to control the prosody of the generations, e.g. use commas to add small breaks in speech
+* The remaining speech features (gender, speaking rate, pitch and reverberation) can be controlled directly through the prompt
+
+
+## Some Description Examples
+
+1. **Aditi - Slightly High-Pitched, Expressive Tone**:  
+   *"Aditi speaks with a slightly higher pitch in a close-sounding environment. Her voice is clear, with subtle emotional depth and a normal pace, all captured in high-quality recording."*  
+
+2. **Sita - Rapid, Slightly Monotone**:  
+   *"Sita speaks at a fast pace with a slightly low-pitched voice, captured clearly in a close-sounding environment with excellent recording quality."*  
+
+3. **Tapan - Male, Moderate Pace, Slightly Monotone**:  
+   *"Tapan speaks at a moderate pace with a slightly monotone tone. The recording is clear, with a close sound and only minimal ambient noise."*  
+
+4. **Sunita - High-Pitched, Happy Tone**:  
+   *"Sunita speaks with a high pitch in a close environment. Her voice is clear, with slight dynamic changes, and the recording is of excellent quality."*  
+
+5. **Karan - High-Pitched, Positive Tone**:  
+   *"Karan’s high-pitched, engaging voice is captured in a clear, close-sounding recording. His slightly slower delivery conveys a positive tone."*  
+
+6. **Amrita - High-Pitched, Flat Tone**:  
+   *"Amrita speaks with a high pitch at a slow pace. Her voice is clear, with excellent recording quality and only moderate background noise."*  
+
+7. **Aditi - Slow, Slightly Expressive**:  
+   *"Aditi speaks slowly with a high pitch and expressive tone. The recording is clear, showcasing her energetic and emotive voice."*  
+
+8. **Young Male Speaker, American Accent**:  
+   *"A young male speaker with a high-pitched American accent delivers speech at a slightly fast pace in a clear, close-sounding recording."*  
+
+9. **Bikram - High-Pitched, Urgent Tone**:  
+   *"Bikram speaks with a higher pitch and fast pace, conveying urgency. The recording is clear and intimate, with great emotional depth."*  
+
+10. **Anjali - High-Pitched, Neutral Tone**:  
+   *"Anjali speaks with a high pitch at a normal pace in a clear, close-sounding environment. Her neutral tone is captured with excellent audio quality."*  
+
+
+## 📐 Evaluation
+
+Indic Parler-TTS has been evaluated using a MOS-like framework by native and non-native speakers. The results highlight its exceptional performance in generating natural and intelligible speech, especially for native speakers of Indian languages.  
+
+**NSS** stands for **Native Speaker Score**:
+
+| **Language**   | **NSS Pretrained (%)** | **NSS Finetuned (%)** | **Highlights**                                                                                     |
+|----------------|-------------------------|------------------------|--------------------------------------------------------------------------------------------------|
+| Assamese       | 82.56 ± 1.80           | 87.36 ± 1.81          | Clear, natural synthesis with excellent expressiveness.                                          |
+| Bengali        | 77.41 ± 2.14           | 86.16 ± 1.85          | High-quality outputs with smooth intonation.                                                    |
+| Bodo           | 90.83 ± 4.54           | 94.47 ± 4.12          | Near-perfect accuracy for a lesser-resourced language.                                          |
+| Dogri          | 82.61 ± 4.98           | 88.80 ± 3.57          | Robust and consistent synthesis for Dogri.                                                     |
+| Gujarati       | 75.28 ± 1.94           | 75.36 ± 1.78          | Strong clarity and naturalness even for smaller languages.                                      |
+| Hindi          | 83.43 ± 1.53           | 84.79 ± 2.09          | Reliable and expressive outputs for India's most widely spoken language.                       |
+| Kannada        | 77.97 ± 3.43           | 88.17 ± 2.81          | Highly natural and accurate voices for Kannada.                                                |
+| Konkani        | 87.20 ± 3.58           | 76.60 ± 4.14          | Produces clear and natural outputs for diverse speakers.                                       |
+| Maithili       | 89.07 ± 4.47           | 95.36 ± 2.52          | Exceptionally accurate, showcasing fine-tuning success.                                         |
+| Malayalam      | 82.02 ± 2.06           | 86.54 ± 1.67          | Smooth, high-quality synthesis with expressive outputs.                                        |
+| Manipuri       | 89.58 ± 1.33           | 85.63 ± 2.60          | Natural intonation with minimal errors.                                                        |
+| Marathi        | 73.81 ± 1.93           | 76.96 ± 1.45          | Maintains clarity and naturalness across speakers.                                             |
+| Nepali         | 64.05 ± 8.33           | 80.02 ± 5.75          | Strong synthesis for native and proximal Nepali speakers.                                      |
+| Odia           | 90.28 ± 2.52           | 88.94 ± 3.26          | High expressiveness and quality for Odia speakers.                                             |
+| Sanskrit       | 99.71 ± 0.58           | 99.79 ± 0.34          | Near-perfect synthesis, ideal for classical use cases.                                         |
+| Sindhi         | 76.44 ± 2.26           | 76.46 ± 1.29          | Clear and natural voices for underrepresented languages.                                       |
+| Tamil          | 69.68 ± 2.73           | 75.48 ± 2.18          | Delivers intelligible and expressive speech.                                                   |
+| Telugu         | 89.77 ± 2.20           | 88.54 ± 1.86          | Smooth and natural tonal quality for Telugu.                                                   |
+| Urdu           | 77.15 ± 3.47           | 77.75 ± 3.82          | Produces high-quality speech despite resource constraints.                                     |
+
+**Key Strengths**:  
+- Exceptional performance for native speakers, with top scores for **Maithili (95.36)**, **Sanskrit (99.79)**, and **Bodo (94.47)**.  
+- Competitive results for lesser-resourced and unofficially supported languages like **Kashmiri (55.30)** and **Sindhi (76.46)**.  
+- Adaptability to non-native and anonymous speaker scenarios with consistently high clarity.  
+
+
+## Motivation
+
+Parler-TTS is a reproduction of work from the paper [Natural language guidance of high-fidelity text-to-speech with synthetic annotations](https://www.text-description-to-speech.com) by Dan Lyth and Simon King, from Stability AI and Edinburgh University respectively. 
+
+Parler-TTS was released alongside:
+* [The Parler-TTS repository](https://github.com/huggingface/parler-tts) - you can train and fine-tuned your own version of the model.
+* [The Data-Speech repository](https://github.com/huggingface/dataspeech) - a suite of utility scripts designed to annotate speech datasets.
+* [The Parler-TTS organization](https://huggingface.co/parler-tts) - where you can find the annotated datasets as well as the future checkpoints.
+
+
+## Training dataset
+
+- **Description**:  
+  The model was fine-tuned on a subset of the dataset used to train the pre-trained version: **Indic-Parler Dataset**, a large-scale multilingual speech corpus designed to train the **Indic Parler-TTS** model.
+
+- **Key Statistics**:
+
+| Dataset         | Duration (hrs) | Languages Covered | No. of Utterances | License      |
+|:---------------:|:--------------:|:-----------------:|:-----------------:|:------------:|
+| GLOBE           | 535.0          | 1                 | 581,725           | CC V1        |
+| IndicTTS        | 382.0          | 12                | 220,606           | CC BY 4.0    |
+| LIMMITS         | 568.0          | 7                 | 246,008           | CC BY 4.0    |
+| Rasa            | 288.0          | 9                 | 155,734           | CC BY 4.0    |
+
+- **Languages Covered**:  
+  The dataset supports **16 official languages** of India, along with English and Chhattisgarhi, making it comprehensive for regional language technologies. These languages include Assamese, Bengali, Bodo, Chhattisgarhi, Dogri, English, Gujarati, Hindi, Kannada, Malayalam, Manipuri, Marathi, Nepali, Odia, Punjabi, Sanskrit, Tamil and Telugu.
+
+
+- **Language-Wise Data Breakdown**:
+
+Here’s the table combining the duration (hours) and the number of utterances from the provided stats:
+
+| Language        | Duration (hrs) | No. of Utterances |
+|:---------------:|:--------------:|:-----------------:|
+| Assamese        | 69.78          | 41,210            |
+| Bengali         | 140.04         | 70,305            |
+| Bodo            | 49.14          | 27,012            |
+| Chhattisgarhi   | 80.11          | 38,148            |
+| Dogri           | 16.14          | 7,823             |
+| English         | 802.81         | 735,482           |
+| Gujarati        | 21.24          | 5,679             |
+| Hindi           | 107.00         | 46,135            |
+| Kannada         | 125.01         | 54,575            |
+| Malayalam       | 25.21          | 14,988            |
+| Manipuri        | 20.77          | 19,232            |
+| Marathi         | 122.47         | 54,894            |
+| Nepali          | 28.65          | 16,016            |
+| Odia            | 19.18          | 11,558            |
+| Punjabi         | 11.07          | 6,892             |
+| Sanskrit        | 19.91          | 8,720             |
+| Tamil           | 52.25          | 29,204            |
+| Telugu          | 95.91          | 37,405            |
+
+
+## Citation
+
+If you found this repository useful, please consider citing this work and also the original Stability AI paper:
+
+```
+@inproceedings{sankar25_interspeech,
+  title     = {{Rasmalai : Resources for Adaptive Speech Modeling in IndiAn Languages with Accents and Intonations}},
+  author    = {Ashwin Sankar and Yoach Lacombe and Sherry Thomas and Praveen {Srinivasa Varadhan} and Sanchit Gandhi and Mitesh M. Khapra},
+  year      = {2025},
+  booktitle = {{Interspeech 2025}},
+  pages     = {4128--4132},
+  doi       = {10.21437/Interspeech.2025-2758},
+  issn      = {2958-1796},
+}
+```
+
+```
+@misc{lacombe-etal-2024-parler-tts,
+  author = {Yoach Lacombe and Vaibhav Srivastav and Sanchit Gandhi},
+  title = {Parler-TTS},
+  year = {2024},
+  publisher = {GitHub},
+  journal = {GitHub repository},
+  howpublished = {\url{https://github.com/huggingface/parler-tts}}
+}
+```
+
+```
+@misc{lyth2024natural,
+      title={Natural language guidance of high-fidelity text-to-speech with synthetic annotations},
+      author={Dan Lyth and Simon King},
+      year={2024},
+      eprint={2402.01912},
+      archivePrefix={arXiv},
+      primaryClass={cs.SD}
+}
+```
+
+## License
+
+This model is permissively licensed under the Apache 2.0 license.

config.json

@@ -0,0 +1,274 @@
+{
+  "_name_or_path": "/fsx/yoach/tmp/artefacts/training-multilingual-mini-indic-finetuning-on-base/",
+  "architectures": [
+    "ParlerTTSForConditionalGeneration"
+  ],
+  "audio_encoder": {
+    "_name_or_path": "ylacombe/dac_44khz",
+    "add_cross_attention": false,
+    "architectures": [
+      "DacModel"
+    ],
+    "bad_words_ids": null,
+    "begin_suppress_tokens": null,
+    "bos_token_id": null,
+    "chunk_size_feed_forward": 0,
+    "codebook_dim": 8,
+    "codebook_loss_weight": 1.0,
+    "codebook_size": 1024,
+    "commitment_loss_weight": 0.25,
+    "cross_attention_hidden_size": null,
+    "decoder_hidden_size": 1536,
+    "decoder_start_token_id": null,
+    "diversity_penalty": 0.0,
+    "do_sample": false,
+    "downsampling_ratios": [
+      2,
+      4,
+      8,
+      8
+    ],
+    "early_stopping": false,
+    "encoder_hidden_size": 64,
+    "encoder_no_repeat_ngram_size": 0,
+    "eos_token_id": null,
+    "exponential_decay_length_penalty": null,
+    "finetuning_task": null,
+    "forced_bos_token_id": null,
+    "forced_eos_token_id": null,
+    "hidden_size": 1024,
+    "hop_length": 512,
+    "id2label": {
+      "0": "LABEL_0",
+      "1": "LABEL_1"
+    },
+    "is_decoder": false,
+    "is_encoder_decoder": false,
+    "label2id": {
+      "LABEL_0": 0,
+      "LABEL_1": 1
+    },
+    "length_penalty": 1.0,
+    "max_length": 20,
+    "min_length": 0,
+    "model_type": "dac",
+    "n_codebooks": 9,
+    "no_repeat_ngram_size": 0,
+    "num_beam_groups": 1,
+    "num_beams": 1,
+    "num_return_sequences": 1,
+    "output_attentions": false,
+    "output_hidden_states": false,
+    "output_scores": false,
+    "pad_token_id": null,
+    "prefix": null,
+    "problem_type": null,
+    "pruned_heads": {},
+    "quantizer_dropout": 0.0,
+    "remove_invalid_values": false,
+    "repetition_penalty": 1.0,
+    "return_dict": true,
+    "return_dict_in_generate": false,
+    "sampling_rate": 44100,
+    "sep_token_id": null,
+    "suppress_tokens": null,
+    "task_specific_params": null,
+    "temperature": 1.0,
+    "tf_legacy_loss": false,
+    "tie_encoder_decoder": false,
+    "tie_word_embeddings": true,
+    "tokenizer_class": null,
+    "top_k": 50,
+    "top_p": 1.0,
+    "torch_dtype": "float32",
+    "torchscript": false,
+    "typical_p": 1.0,
+    "upsampling_ratios": [
+      8,
+      8,
+      4,
+      2
+    ],
+    "use_bfloat16": false
+  },
+  "decoder": {
+    "_name_or_path": "/fsx/yoach/tmp/artefacts/parler-tts-mini-v2-empty/decoder",
+    "activation_dropout": 0.0,
+    "activation_function": "gelu",
+    "add_cross_attention": true,
+    "architectures": [
+      "ParlerTTSForCausalLM"
+    ],
+    "attention_dropout": 0.0,
+    "bad_words_ids": null,
+    "begin_suppress_tokens": null,
+    "bos_token_id": 1025,
+    "chunk_size_feed_forward": 0,
+    "codebook_weights": null,
+    "cross_attention_hidden_size": null,
+    "cross_attention_implementation_strategy": null,
+    "decoder_start_token_id": null,
+    "delay_strategy": "delay",
+    "diversity_penalty": 0.0,
+    "do_sample": false,
+    "dropout": 0.1,
+    "early_stopping": false,
+    "encoder_no_repeat_ngram_size": 0,
+    "eos_token_id": 1024,
+    "exponential_decay_length_penalty": null,
+    "ffn_dim": 4096,
+    "finetuning_task": null,
+    "forced_bos_token_id": null,
+    "forced_eos_token_id": null,
+    "hidden_size": 1024,
+    "id2label": {
+      "0": "LABEL_0",
+      "1": "LABEL_1"
+    },
+    "initializer_factor": 0.02,
+    "is_decoder": true,
+    "is_encoder_decoder": false,
+    "label2id": {
+      "LABEL_0": 0,
+      "LABEL_1": 1
+    },
+    "layerdrop": 0.0,
+    "length_penalty": 1.0,
+    "max_length": 20,
+    "max_position_embeddings": 4096,
+    "min_length": 0,
+    "model_type": "parler_tts_decoder",
+    "no_repeat_ngram_size": 0,
+    "num_attention_heads": 16,
+    "num_beam_groups": 1,
+    "num_beams": 1,
+    "num_codebooks": 9,
+    "num_cross_attention_key_value_heads": 16,
+    "num_hidden_layers": 24,
+    "num_key_value_heads": 16,
+    "num_return_sequences": 1,
+    "output_attentions": false,
+    "output_hidden_states": false,
+    "output_scores": false,
+    "pad_token_id": 1024,
+    "prefix": null,
+    "problem_type": null,
+    "pruned_heads": {},
+    "remove_invalid_values": false,
+    "repetition_penalty": 1.0,
+    "return_dict": true,
+    "return_dict_in_generate": false,
+    "rope_embeddings": false,
+    "rope_theta": 10000.0,
+    "scale_embedding": false,
+    "sep_token_id": null,
+    "suppress_tokens": null,
+    "task_specific_params": null,
+    "temperature": 1.0,
+    "tf_legacy_loss": false,
+    "tie_encoder_decoder": false,
+    "tie_word_embeddings": false,
+    "tokenizer_class": null,
+    "top_k": 50,
+    "top_p": 1.0,
+    "torch_dtype": "float32",
+    "torchscript": false,
+    "typical_p": 1.0,
+    "use_bfloat16": false,
+    "use_cache": true,
+    "use_fused_lm_heads": true,
+    "vocab_size": 1088
+  },
+  "decoder_start_token_id": 1025,
+  "is_encoder_decoder": true,
+  "model_type": "parler_tts",
+  "pad_token_id": 1024,
+  "prompt_cross_attention": false,
+  "text_encoder": {
+    "_name_or_path": "google/flan-t5-large",
+    "add_cross_attention": false,
+    "architectures": [
+      "T5ForConditionalGeneration"
+    ],
+    "bad_words_ids": null,
+    "begin_suppress_tokens": null,
+    "bos_token_id": null,
+    "chunk_size_feed_forward": 0,
+    "classifier_dropout": 0.0,
+    "cross_attention_hidden_size": null,
+    "d_ff": 2816,
+    "d_kv": 64,
+    "d_model": 1024,
+    "decoder_start_token_id": 0,
+    "dense_act_fn": "gelu_new",
+    "diversity_penalty": 0.0,
+    "do_sample": false,
+    "dropout_rate": 0.1,
+    "early_stopping": false,
+    "encoder_no_repeat_ngram_size": 0,
+    "eos_token_id": 1,
+    "exponential_decay_length_penalty": null,
+    "feed_forward_proj": "gated-gelu",
+    "finetuning_task": null,
+    "forced_bos_token_id": null,
+    "forced_eos_token_id": null,
+    "id2label": {
+      "0": "LABEL_0",
+      "1": "LABEL_1"
+    },
+    "initializer_factor": 1.0,
+    "is_decoder": false,
+    "is_encoder_decoder": true,
+    "is_gated_act": true,
+    "label2id": {
+      "LABEL_0": 0,
+      "LABEL_1": 1
+    },
+    "layer_norm_epsilon": 1e-06,
+    "length_penalty": 1.0,
+    "max_length": 20,
+    "min_length": 0,
+    "model_type": "t5",
+    "n_positions": 512,
+    "no_repeat_ngram_size": 0,
+    "num_beam_groups": 1,
+    "num_beams": 1,
+    "num_decoder_layers": 24,
+    "num_heads": 16,
+    "num_layers": 24,
+    "num_return_sequences": 1,
+    "output_attentions": false,
+    "output_hidden_states": false,
+    "output_past": true,
+    "output_scores": false,
+    "pad_token_id": 0,
+    "prefix": null,
+    "problem_type": null,
+    "pruned_heads": {},
+    "relative_attention_max_distance": 128,
+    "relative_attention_num_buckets": 32,
+    "remove_invalid_values": false,
+    "repetition_penalty": 1.0,
+    "return_dict": true,
+    "return_dict_in_generate": false,
+    "sep_token_id": null,
+    "suppress_tokens": null,
+    "task_specific_params": null,
+    "temperature": 1.0,
+    "tf_legacy_loss": false,
+    "tie_encoder_decoder": false,
+    "tie_word_embeddings": false,
+    "tokenizer_class": null,
+    "top_k": 50,
+    "top_p": 1.0,
+    "torch_dtype": null,
+    "torchscript": false,
+    "typical_p": 1.0,
+    "use_bfloat16": false,
+    "use_cache": true,
+    "vocab_size": 32128
+  },
+  "torch_dtype": "float32",
+  "transformers_version": "4.46.0.dev0",
+  "vocab_size": 90714
+}

generation_config.json

@@ -0,0 +1,10 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 1025,
+  "decoder_start_token_id": 1025,
+  "do_sample": true,
+  "eos_token_id": 1024,
+  "max_length": 2610,
+  "pad_token_id": 1024,
+  "transformers_version": "4.46.0.dev0"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c68daecb60f80c8f1faf0a6d2e6ddd6de8e224fb19750f3e9a33bca43c552c90
+size 3751321772

preprocessor_config.json

@@ -0,0 +1,9 @@
+{
+  "feature_extractor_type": "DacFeatureExtractor",
+  "feature_size": 1,
+  "hop_length": 512,
+  "padding_side": "right",
+  "padding_value": 0.0,
+  "return_attention_mask": true,
+  "sampling_rate": 44100
+}

special_tokens_map.json

@@ -0,0 +1,30 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bc8fa773221597d09cfadb23a2b1bd717488a0481505469ea56d42cb044de9b5
+size 1795391

tokenizer_config.json

@@ -0,0 +1,43 @@
+{
+  "add_bos_token": true,
+  "add_eos_token": false,
+  "add_prefix_space": true,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "</s>",
+  "legacy": false,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<unk>",
+  "padding_side": "left",
+  "sp_model_kwargs": {},
+  "spaces_between_special_tokens": false,
+  "tokenizer_class": "LlamaTokenizer",
+  "unk_token": "<unk>",
+  "use_default_system_prompt": false
+}