--- license: mit language: - hi - te - kn - bn - ta - ml - mr - gu - en base_model: ResembleAI/chatterbox tags: - tts - text-to-speech - lora - indic - indian-languages - chatterbox - speech-synthesis - voice-cloning library_name: transformers pipeline_tag: text-to-speech --- # Chatterbox Indic LoRA — Indian Language TTS **LoRA adapters + extended tokenizer to add 8 Indian languages to [Chatterbox-Multilingual](https://github.com/resemble-ai/chatterbox) by Resemble AI.** No phoneme engineering. No G2P. Just grapheme-level fine-tuning on 1.4% of the model parameters. > **[Article Series: Teaching an AI to Speak Indian Languages](https://theatomsofai.substack.com/p/teaching-an-ai-to-speak-indian-languages)** # Chatterbox Indic LoRA — Indian Language TTS [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1oIM5jY64cuYGZhwPmbKmYrQ3zHW3RvXN?usp=sharing) --- ## Audio Samples ### Hindi (hi) — CER 0.1058 | Male | Female | |------|--------| | | | ### Telugu (te) — CER 0.2853 | Male | Female | |------|--------| | | | ### Kannada (kn) — CER 0.1434 | Male | Female | |------|--------| | | | ### Bengali (bn) — CER 0.2450 | Male | |------| | | ### Tamil (ta) — CER 0.1608 | Male | Female | |------|--------| | | | ### Malayalam (ml) — CER 0.8593 | Male | Female | |------|--------| | | | ### Marathi (mr) — CER 0.1976 | Male | Female | |------|--------| | | | ### Gujarati (gu) — CER 0.2377 | Male | Female | |------|--------| | | | --- ## Supported Languages | Language | Script | Training Data | CER (mean) | Status | |----------|--------|---------------|:----------:|--------| | Hindi | Devanagari | ~10h (IndicTTS) | 0.1058 | Stable | | Telugu | Telugu | ~52h (IndicTTS + ai4bharat Rasa) | 0.2853 | Trained | | Kannada | Kannada | ~7h (IndicTTS) | 0.1434 | Trained | | Bengali | Bengali | ~15h (IndicTTS) | 0.2450 | Trained | | Tamil | Tamil | ~10h (IndicTTS + ai4bharat Rasa) | 0.1608 | Trained | | Malayalam | Malayalam | ~10h (IndicTTS + ai4bharat Rasa) | 0.8593 | Experimental | | Marathi | Devanagari | ~10h (IndicTTS + ai4bharat Rasa) | 0.1976 | Trained | | Gujarati | Gujarati | ~10h (IndicTTS + ai4bharat Rasa) | 0.2377 | Trained | | English | Latin | — | Preserved | Base model (frozen) | *CER measured via Whisper large-v3 ASR on 100 held-out samples per language.* --- ## How It Works The base Chatterbox-Multilingual model supports 23 languages but no Dravidian or additional Indo-Aryan languages beyond Hindi. This adapter extends it by: 1. **Extended Tokenizer** — Added graphemes for Telugu, Kannada, Bengali, Tamil, Malayalam, Marathi, Gujarati to the MTLTokenizer vocabulary (2454 → 2871 tokens) 2. **Brahmic Warm-Start** — New character embeddings initialized from phonetically equivalent Devanagari characters (e.g., Telugu "క" ← Hindi "क") 3. **LoRA Fine-Tuning** — Rank-32 adapters on q/k/v/o projections of the T3 Llama backbone (~7.8M trainable params / 544M total) 4. **Gradient Masking** — Original embedding rows frozen during training; only new script embeddings update The speech vocabulary, vocoder (S3Gen), and speaker encoder remain completely frozen. Only T3's text understanding is adapted. --- ## Quick Start ### Option A: Python (3 lines) Install from the fork (not `pip install chatterbox-tts` — that has dependency conflicts): ```bash # 1. Install PyTorch for your GPU first (example for CUDA 12.8 / Blackwell / 50-series): pip install torch torchaudio --index-url https://download.pytorch.org/whl/cu128 # 2. Install from fork (relaxed deps, Indic support built in): pip install git+https://github.com/reenigne314/chatterbox-indic-lora.git ``` Then generate speech: ```python import soundfile as sf from chatterbox.mtl_tts import ChatterboxMultilingualTTS # Load base model + LoRA + tokenizer + speaker — all in one call model = ChatterboxMultilingualTTS.from_indic_lora(device="cuda", speaker="te_female") # Generate Telugu speech wav = model.generate("నమస్కారం, మీరు ఎలా ఉన్నారు?", language_id="te") sf.write("output_telugu.wav", wav.squeeze(0).cpu().numpy(), model.sr) ``` ```python # Switch speaker on the fly from chatterbox.mtl_tts import Conditionals model.conds = Conditionals.load("path/to/hi_male.pt").to("cuda") wav = model.generate("नमस्ते, आप कैसे हैं?", language_id="hi") sf.write("output_hindi.wav", wav.squeeze(0).cpu().numpy(), model.sr) ``` ### Option B: Docker (one command) ```bash git clone https://huggingface.co/reenigne314/chatterbox-indic-lora cd chatterbox-indic-lora docker compose up # Open http://localhost:7860 ``` ### Option C: Gradio Web UI ```bash pip install torch torchaudio --index-url https://download.pytorch.org/whl/cu128 pip install git+https://github.com/reenigne314/chatterbox-indic.git pip install gradio>=4.0.0 python app.py # http://localhost:7860 python app.py --share # public link ``` --- ## Available Speakers | File | Language | Gender | |------|----------|--------| | `hi_female.pt` / `hi_male.pt` | Hindi | Female / Male | | `te_female.pt` / `te_male.pt` | Telugu | Female / Male | | `kn_female.pt` / `kn_male.pt` | Kannada | Female / Male | | `bn_female.pt` / `bn_male.pt` | Bengali | Female / Male | | `ta_female.pt` / `ta_male.pt` | Tamil | Female / Male | | `ml_female.pt` / `ml_male.pt` | Malayalam | Female / Male | | `mr_female.pt` / `mr_male.pt` | Marathi | Female / Male | | `gu_female.pt` / `gu_male.pt` | Gujarati | Female / Male | --- ## Included Files ``` . ├── app.py # Gradio Web UI ├── Dockerfile # Docker support ├── docker-compose.yml ├── requirements.txt ├── checkpoints/ │ └── best.pt # LoRA weights + extended embeddings ├── tokenizer/ │ ├── extended_tokenizer.json # Extended vocab (2454 → 2871 tokens) │ └── brahmic_init_map.json # Brahmic → Devanagari mapping ├── conds/ │ ├── {lang}_{gender}.pt # 16 speaker conditioning files │ └── conds_manifest.json # Speaker metadata └── README.md # This file ``` **Base model not included.** `from_indic_lora()` auto-downloads it from `ResembleAI/chatterbox` on first run. --- ## Training Details | Setting | Value | |---------|-------| | Base model | Chatterbox-Multilingual (T3 Llama 520M) | | LoRA rank | 32 | | LoRA alpha | 64 | | LoRA targets | q_proj, k_proj, v_proj, o_proj | | Trainable params | ~7.8M / 544M (1.4%) | | Precision | bf16 | | Hardware | 1x RTX PRO 6000 Blackwell (96GB) | | Primary data | SPRINGLab IndicTTS, ai4bharat Rasa | | Training script | [scripts/train_t3_lora.py](https://github.com/reenigne314/chatterbox-indic/blob/main/scripts/train_t3_lora.py) | ### Training Approach Languages were added incrementally with weighted sampling to prevent catastrophic forgetting: - **Round 1:** Hindi only (validate pipeline) - **Round 2:** Telugu + Hindi (extended vocab, Brahmic warm-start) - **Round 3:** Telugu-heavy with larger dataset (ai4bharat Rasa ~52h) - **Round 4:** Telugu refinement with expanded data - **Round 5:** Kannada + Telugu + Hindi - **Round 6:** All 8 languages (Hi, Te, Kn, Bn, Ta, Ml, Mr, Gu) Hindi CER improved even after adding new languages — no catastrophic forgetting observed. --- ## Limitations - **Malayalam CER is high (0.86).** The model struggles with Malayalam — likely needs more training data or dedicated fine-tuning. Treat Malayalam as experimental. - **CER is the primary metric.** Naturalness (MOS), speaker similarity, and prosody have not been formally evaluated yet. The audio sounds clean to the ear, but systematic subjective evaluation is pending. - **2 speakers per language.** Training data has one male and one female speaker from IndicTTS per language. The model may not generalize well to all voice types. - **No code-mix yet.** Hindi+English or Telugu+English mixed sentences are not specifically trained. This is planned for a future release. - **Single codebook.** Chatterbox uses single-stream S3 tokens (25 Hz). Fine acoustic details may be less sharp than multi-codebook systems. --- ## Citation If you use this model, please cite both this work and the original Chatterbox: ```bibtex @misc{chatterbox_indic_lora_2025, author = {Bharadwaj Kommanamanchi}, title = {Chatterbox Indic LoRA — Indian Language TTS via Grapheme-Level Fine-Tuning}, year = {2025}, howpublished = {\url{https://huggingface.co/reenigne314/chatterbox-indic-lora}}, note = {LoRA adapters for Chatterbox-Multilingual} } @misc{chatterboxtts2025, author = {{Resemble AI}}, title = {{Chatterbox-TTS}}, year = {2025}, howpublished = {\url{https://github.com/resemble-ai/chatterbox}}, note = {GitHub repository} } ``` --- ## Acknowledgements - **[Resemble AI](https://github.com/resemble-ai/chatterbox)** — for open-sourcing Chatterbox under MIT license. This work would not exist without their model and architecture. - **[SPRINGLab / IIT Madras](https://huggingface.co/SPRINGLab)** — IndicTTS dataset - **[ai4bharat](https://ai4bharat.iitm.ac.in/)** — Rasa dataset for Telugu - **[CosyVoice](https://github.com/FunAudioLLM/CosyVoice)** — S3Gen architecture (adapted by Resemble AI) - **[Meta / Llama 3](https://github.com/meta-llama/llama3)** — T3 backbone architecture