Qwen_Indic/README.md

---
base_model: Qwen/Qwen3-TTS-12Hz-0.6B-Base
library_name: peft
pipeline_tag: text-to-speech
license: apache-2.0
language:
- as
- bn
- brx
- doi
- gu
- kn
- kok
- mai
- ml
- mni
- mr
- ne
- or
- pa
- sa
- sat
- sd
- ta
- te
- ur
tags:
- tts
- indic
- multilingual
- lora
- peft
- voice-clone
- audio
- qwen3
- ai4bharat
- rasa
datasets:
- ai4bharat/Rasa
---

# Qwen3-TTS-0.6B-LoRA-Finetuned-Indic-Multilingual

A collection of LoRA adapters fine-tuned on top of [Qwen/Qwen3-TTS-12Hz-0.6B-Base](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-0.6B-Base) for **15 Indic languages** using the [AI4Bharat Rasa](https://huggingface.co/datasets/ai4bharat/Rasa) dataset.

Each language has two gender-specific adapters (female and male), giving **30 adapters in total** — all in a single repository.

> Part of the [aguken-ai/qwen3-tts-indic-finetuned](https://huggingface.co/collections/aguken-ai/qwen3-tts-indic-finetuned) collection.

---

## Supported Languages

| Language   | Code | Female Adapter | Male Adapter |
|------------|------|---------------|--------------|
| Assamese   | `as` | `adapters/assamese_female` | `adapters/assamese_male` |
| Bengali    | `bn` | `adapters/bengali_female`  | `adapters/bengali_male`  |
| Bodo       | `brx`| `adapters/bodo_female`     | `adapters/bodo_male`     |
| Dogri      | `doi`| `adapters/dogri_female`    | `adapters/dogri_male`    |
| Gujarati   | `gu` | `adapters/gujarati_female` | `adapters/gujarati_male` |
| Kannada    | `kn` | `adapters/kannada_female`  | `adapters/kannada_male`  |
| Konkani    | `kok`| `adapters/konkani_female`  | `adapters/konkani_male`  |
| Maithili   | `mai`| `adapters/maithili_female` | `adapters/maithili_male` |
| Malayalam  | `ml` | `adapters/malayalam_female`| `adapters/malayalam_male`|
| Marathi    | `mr` | `adapters/marathi_female`  | `adapters/marathi_male`  |
| Nepali     | `ne` | `adapters/nepali_female`   | `adapters/nepali_male`   |
| Odia       | `or` | `adapters/odia_female`     | `adapters/odia_male`     |
| Punjabi    | `pa` | `adapters/punjabi_female`  | `adapters/punjabi_male`  |
| Tamil      | `ta` | `adapters/tamil_female`    | `adapters/tamil_male`    |
| Telugu     | `te` | `adapters/telugu_female`   | `adapters/telugu_male`   |

---

## Repository Structure

```
Qwen3-TTS-0.6B-LoRA-Finetuned-Indic-Multilingual/
├── README.md
├── config.json                  ┐
├── generation_config.json       │
├── merges.txt                   │
├── model.safetensors            ├─ bundled Qwen3-TTS-0.6B base model
├── preprocessor_config.json     │  (loads standalone, no separate download)
├── tokenizer_config.json        │
├── vocab.json                   │
├── speech_tokenizer/            ┘  12 Hz speech tokenizer (config + weights)
└── adapters/
    ├── assamese_female/
    │   ├── adapter_config.json
    │   ├── adapter_model.safetensors
    │   ├── config.json
    │   ├── ref.wav
    │   ├── ref.txt
    │   └── speaker_embedding.safetensors
    ├── assamese_male/
    │   └── ...
    ├── bengali_female/
    ├── bengali_male/
    │   └── ... (one folder per language × gender)
    ├── tamil_female/         ← trained for 3 epochs
    └── telugu_male/
```

The repo bundles the **full base model** at root (plus `speech_tokenizer/`), so it loads standalone without separately downloading `Qwen/Qwen3-TTS-12Hz-0.6B-Base`.

Each adapter folder contains:
- `adapter_config.json` — LoRA configuration (rank 16, alpha 32, targeting all attention + FFN projections)
- `adapter_model.safetensors` — LoRA weight delta (~45 MB per adapter)
- `config.json` — base model config copy
- `ref.wav` — reference audio clip for voice cloning in that language/gender
- `ref.txt` — transcript of `ref.wav` (pass as `ref_text` for voice cloning)
- `speaker_embedding.safetensors` — learned speaker embedding for this language/gender

---

## Training Details

| Setting | Value |
|---------|-------|
| Base model | `Qwen/Qwen3-TTS-12Hz-0.6B-Base` |
| Dataset | AI4Bharat Rasa (per-language subset) |
| Samples per combo | up to 500 |
| Audio preprocessing | 24 kHz, PCM-16, RMS-normalized to −20 dB |
| LoRA rank | 16 |
| LoRA alpha | 32 |
| LoRA dropout | 0.05 |
| Target modules | `q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj` |
| Learning rate | 2e-6 |
| Epochs | 5 (Tamil female: 3) |
| Batch size | 2 (with gradient accumulation ×16) |
| Precision | bf16 |
| Attention | Flash Attention 2 |

---

## Quickstart

### Installation

```bash
pip install -U qwen-tts peft transformers torch torchaudio
pip install -U flash-attn --no-build-isolation  # optional, for speed
```

### Inference

```python
import torch
import soundfile as sf
from huggingface_hub import snapshot_download, hf_hub_download
from qwen_tts import Qwen3TTSModel
from peft import PeftModel

REPO_ID    = "aguken-ai/Qwen3-TTS-0.6B-LoRA-Finetuned-Indic-Multilingual"
COMBO      = "telugu_male"   # any of the 30 {language}_{gender} folders
LORA_SCALE = 0.35            # tune between 0.1–0.5

# The base Qwen3-TTS weights are bundled in this repo, so it loads standalone
BASE_MODEL   = snapshot_download(REPO_ID, allow_patterns=[
    "config.json", "generation_config.json", "merges.txt", "model.safetensors",
    "preprocessor_config.json", "tokenizer_config.json", "vocab.json", "speech_tokenizer/*",
])
ADAPTER_PATH = snapshot_download(REPO_ID, allow_patterns=[f"adapters/{COMBO}/*"]) + f"/adapters/{COMBO}"

# Load base model
model = Qwen3TTSModel.from_pretrained(
    BASE_MODEL,
    device_map="cuda:0",
    dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",
)

# Load LoRA adapter
model = PeftModel.from_pretrained(model, ADAPTER_PATH)
model.set_adapter_scale(LORA_SCALE)

# Bundled reference audio + transcript for voice cloning (one pair per language/gender)
ref_audio = hf_hub_download(REPO_ID, filename=f"adapters/{COMBO}/ref.wav")
ref_text  = open(hf_hub_download(REPO_ID, filename=f"adapters/{COMBO}/ref.txt"), encoding="utf-8").read().strip()

# Generate
text = "నమస్కారం! నేను Avni మాట్లాడుతున్నాను Synoris Technologies నుండి."
wavs, sr = model.generate_voice_clone(
    text=text,
    language="Telugu",
    ref_audio=ref_audio,
    ref_text=ref_text,
)
sf.write("output.wav", wavs[0], sr)
```

### Notebook

The repo root also includes [Qwen3_TTS_Indic_Inference.ipynb](./Qwen3_TTS_Indic_Inference.ipynb) for a notebook-based inference flow with Telugu male reference audio.

### LoRA Scale Tuning

The `lora_scale` controls how strongly the fine-tuned Indic accent/voice is applied over the base model:

| Scale | Effect |
|-------|--------|
| 0.1–0.2 | Subtle Indic influence, close to base model |
| 0.3–0.4 | Balanced — recommended starting point |
| 0.5 | Strong Indic adaptation |

---

## Speaker IDs

Each adapter was trained with a dedicated `speaker_id` in the format `rasa_{language}_{gender}` (e.g., `rasa_telugu_male`, `rasa_assamese_female`). These are stored in the `speaker_embedding.safetensors` file.

---

## Dataset

Training data is sourced from [AI4Bharat Rasa](https://huggingface.co/datasets/ai4bharat/Rasa) — a large-scale, high-quality speech dataset covering 22 Indian languages with male and female speakers.

Up to 500 utterances per language/gender combination were used, filtered for:
- Duration: 1–15 seconds
- Clean text (no URLs, low digit ratio)
- Audio normalized to −20 dB RMS at 24 kHz

---

## Citation

If you use this work, please cite the base model and dataset:

```bibtex
@article{Qwen3-TTS,
  title={Qwen3-TTS Technical Report},
  author={Hangrui Hu and Xinfa Zhu and Ting He and Dake Guo and Bin Zhang and Xiong Wang and Zhifang Guo and Ziyue Jiang and Hongkun Hao and Zishan Guo and Xinyu Zhang and Pei Zhang and Baosong Yang and Jin Xu and Jingren Zhou and Junyang Lin},
  journal={arXiv preprint arXiv:2601.15621},
  year={2026}
}

@dataset{ai4bharat_rasa,
  title={Rasa: A Large-Scale Indian Language Speech Dataset},
  author={AI4Bharat},
  year={2024},
  url={https://huggingface.co/datasets/ai4bharat/Rasa}
}
```

---

## License

Apache 2.0 — same as the base model. See [LICENSE](https://www.apache.org/licenses/LICENSE-2.0) for details.