Qwen_Indic/README.md

metadata

base_model: Qwen/Qwen3-TTS-12Hz-0.6B-Base
library_name: peft
pipeline_tag: text-to-speech
license: apache-2.0
language:
  - as
  - bn
  - brx
  - doi
  - gu
  - kn
  - kok
  - mai
  - ml
  - mni
  - mr
  - ne
  - or
  - pa
  - sa
  - sat
  - sd
  - ta
  - te
  - ur
tags:
  - tts
  - indic
  - multilingual
  - lora
  - peft
  - voice-clone
  - audio
  - qwen3
  - ai4bharat
  - rasa
datasets:
  - ai4bharat/Rasa

Qwen3-TTS-0.6B-LoRA-Finetuned-Indic-Multilingual

A collection of LoRA adapters fine-tuned on top of Qwen/Qwen3-TTS-12Hz-0.6B-Base for 15 Indic languages using the AI4Bharat Rasa dataset.

Each language has two gender-specific adapters (female and male), giving 30 adapters in total — all in a single repository.

Part of the aguken-ai/qwen3-tts-indic-finetuned collection.

---

Supported Languages

Language	Code	Female Adapter	Male Adapter
Assamese	as	adapters/assamese_female	adapters/assamese_male
Bengali	bn	adapters/bengali_female	adapters/bengali_male
Bodo	brx	adapters/bodo_female	adapters/bodo_male
Dogri	doi	adapters/dogri_female	adapters/dogri_male
Gujarati	gu	adapters/gujarati_female	adapters/gujarati_male
Kannada	kn	adapters/kannada_female	adapters/kannada_male
Konkani	kok	adapters/konkani_female	adapters/konkani_male
Maithili	mai	adapters/maithili_female	adapters/maithili_male
Malayalam	ml	adapters/malayalam_female	adapters/malayalam_male
Marathi	mr	adapters/marathi_female	adapters/marathi_male
Nepali	ne	adapters/nepali_female	adapters/nepali_male
Odia	or	adapters/odia_female	adapters/odia_male
Punjabi	pa	adapters/punjabi_female	adapters/punjabi_male
Tamil	ta	adapters/tamil_female	adapters/tamil_male
Telugu	te	adapters/telugu_female	adapters/telugu_male

---

Repository Structure

Qwen3-TTS-0.6B-LoRA-Finetuned-Indic-Multilingual/
├── README.md
├── config.json                  ┐
├── generation_config.json       │
├── merges.txt                   │
├── model.safetensors            ├─ bundled Qwen3-TTS-0.6B base model
├── preprocessor_config.json     │  (loads standalone, no separate download)
├── tokenizer_config.json        │
├── vocab.json                   │
├── speech_tokenizer/            ┘  12 Hz speech tokenizer (config + weights)
└── adapters/
    ├── assamese_female/
    │   ├── adapter_config.json
    │   ├── adapter_model.safetensors
    │   ├── config.json
    │   ├── ref.wav
    │   ├── ref.txt
    │   └── speaker_embedding.safetensors
    ├── assamese_male/
    │   └── ...
    ├── bengali_female/
    ├── bengali_male/
    │   └── ... (one folder per language × gender)
    ├── tamil_female/         ← trained for 3 epochs
    └── telugu_male/

The repo bundles the full base model at root (plus speech_tokenizer/), so it loads standalone without separately downloading Qwen/Qwen3-TTS-12Hz-0.6B-Base.

Each adapter folder contains:

• adapter_config.json — LoRA configuration (rank 16, alpha 32, targeting all attention + FFN projections)
• adapter_model.safetensors — LoRA weight delta (~45 MB per adapter)
• config.json — base model config copy
• ref.wav — reference audio clip for voice cloning in that language/gender
• ref.txt — transcript of ref.wav (pass as ref_text for voice cloning)
• speaker_embedding.safetensors — learned speaker embedding for this language/gender

---

Training Details

Setting	Value
Base model	Qwen/Qwen3-TTS-12Hz-0.6B-Base
Dataset	AI4Bharat Rasa (per-language subset)
Samples per combo	up to 500
Audio preprocessing	24 kHz, PCM-16, RMS-normalized to −20 dB
LoRA rank	16
LoRA alpha	32
LoRA dropout	0.05
Target modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Learning rate	2e-6
Epochs	5 (Tamil female: 3)
Batch size	2 (with gradient accumulation ×16)
Precision	bf16
Attention	Flash Attention 2

---

Quickstart

Installation

pip install -U qwen-tts peft transformers torch torchaudio
pip install -U flash-attn --no-build-isolation  # optional, for speed

Inference

import torch
import soundfile as sf
from huggingface_hub import snapshot_download, hf_hub_download
from qwen_tts import Qwen3TTSModel
from peft import PeftModel

REPO_ID    = "aguken-ai/Qwen3-TTS-0.6B-LoRA-Finetuned-Indic-Multilingual"
COMBO      = "telugu_male"   # any of the 30 {language}_{gender} folders
LORA_SCALE = 0.35            # tune between 0.1–0.5

# The base Qwen3-TTS weights are bundled in this repo, so it loads standalone
BASE_MODEL   = snapshot_download(REPO_ID, allow_patterns=[
    "config.json", "generation_config.json", "merges.txt", "model.safetensors",
    "preprocessor_config.json", "tokenizer_config.json", "vocab.json", "speech_tokenizer/*",
])
ADAPTER_PATH = snapshot_download(REPO_ID, allow_patterns=[f"adapters/{COMBO}/*"]) + f"/adapters/{COMBO}"

# Load base model
model = Qwen3TTSModel.from_pretrained(
    BASE_MODEL,
    device_map="cuda:0",
    dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",
)

# Load LoRA adapter
model = PeftModel.from_pretrained(model, ADAPTER_PATH)
model.set_adapter_scale(LORA_SCALE)

# Bundled reference audio + transcript for voice cloning (one pair per language/gender)
ref_audio = hf_hub_download(REPO_ID, filename=f"adapters/{COMBO}/ref.wav")
ref_text  = open(hf_hub_download(REPO_ID, filename=f"adapters/{COMBO}/ref.txt"), encoding="utf-8").read().strip()

# Generate
text = "నమస్కారం! నేను Avni మాట్లాడుతున్నాను Synoris Technologies నుండి."
wavs, sr = model.generate_voice_clone(
    text=text,
    language="Telugu",
    ref_audio=ref_audio,
    ref_text=ref_text,
)
sf.write("output.wav", wavs[0], sr)

Notebook

The repo root also includes Qwen3_TTS_Indic_Inference.ipynb for a notebook-based inference flow with Telugu male reference audio.

LoRA Scale Tuning

The lora_scale controls how strongly the fine-tuned Indic accent/voice is applied over the base model:

Scale	Effect
0.1–0.2	Subtle Indic influence, close to base model
0.3–0.4	Balanced — recommended starting point
0.5	Strong Indic adaptation

---

Speaker IDs

Each adapter was trained with a dedicated speaker_id in the format rasa_{language}_{gender} (e.g., rasa_telugu_male, rasa_assamese_female). These are stored in the speaker_embedding.safetensors file.

---

Dataset

Training data is sourced from AI4Bharat Rasa — a large-scale, high-quality speech dataset covering 22 Indian languages with male and female speakers.

Up to 500 utterances per language/gender combination were used, filtered for:

• Duration: 1–15 seconds
• Clean text (no URLs, low digit ratio)
• Audio normalized to −20 dB RMS at 24 kHz

---

Citation

If you use this work, please cite the base model and dataset:

@article{Qwen3-TTS,
  title={Qwen3-TTS Technical Report},
  author={Hangrui Hu and Xinfa Zhu and Ting He and Dake Guo and Bin Zhang and Xiong Wang and Zhifang Guo and Ziyue Jiang and Hongkun Hao and Zishan Guo and Xinyu Zhang and Pei Zhang and Baosong Yang and Jin Xu and Jingren Zhou and Junyang Lin},
  journal={arXiv preprint arXiv:2601.15621},
  year={2026}
}

@dataset{ai4bharat_rasa,
  title={Rasa: A Large-Scale Indian Language Speech Dataset},
  author={AI4Bharat},
  year={2024},
  url={https://huggingface.co/datasets/ai4bharat/Rasa}
}

---

License

Apache 2.0 — same as the base model. See LICENSE for details.