ayushkhadkaa/chatterbox-nepali-tts

Aawaaj AI (आवाज) - High-Performance Nepali-English TTS

This is a self-contained, ultra-fast multilingual Text-to-Speech (TTS) model optimized specifically for perfectly switching between Nepali and English. Built on the Chatterbox architecture and fine-tuned for professional-grade audio quality, it excels at handling complex, mixed-language sentences with natural prosody.

✨ Key Features

• Perfect Mixed-Language Support: Seamlessly handles code-switching between Nepali and English (e.g., "म आज very happy छु।") with natural intonation.
• Ultra-Fast & Low Latency: Optimized for real-time speech generation on modern GPUs, while remaining highly performant on CPUs.
• Zero-Shot Voice Cloning: Clone any voice with just a 5-10 second reference audio clip—no additional training required.
• Pro Audio Quality: Crystal clear 24kHz synthesis with custom silence padding for professional results.
• Zero Configuration: Source code and weights are bundled together for instant deployment via snapshot_download.

🚀 Performance

Hardware	Speed	Latency
NVIDIA GPU (T4/L4/A100)	Ultra-Fast	< 200ms
Modern CPU	Real-Time	< 1s

---

🛠️ Usage Example (The "Perfect" Setup)

This script will automatically download the model and the necessary code to run it.

import sys
from pathlib import Path

import torch
import torchaudio
from huggingface_hub import snapshot_download

# =========================
# DOWNLOAD MODEL (SOURCE OF PATH)
# =========================
REPO_ID = "ayushkhadkaa/chatterbox-nepali-tts"
model_path = Path(snapshot_download(repo_id=REPO_ID))

# Add repo to Python path BEFORE import
sys.path.insert(0, str(model_path))

# Now safe to import
from chatterbox.models.mtl_tts import ChatterboxMultilingualTTS


# =========================
# LOAD MODEL ONCE
# =========================
device = "cuda" if torch.cuda.is_available() else "cpu"
print(f"Loading model on {device}...")

model = ChatterboxMultilingualTTS.from_local(model_path, device=device)
model.eval()

# Built-in reference audio
DEFAULT_REF_AUDIO = model_path / "demo" / "ne_slr143_Voice913.wav"


# =========================
# GENERATE FUNCTION
# =========================
@torch.inference_mode()
def generate_tts(text, lang="ne", output_file="output.wav"):
    if not text.strip():
        raise ValueError("Text is empty")

    wav = model.generate(
        text=text.strip(),
        language_id=lang,
        audio_prompt_path=str(DEFAULT_REF_AUDIO),
        exaggeration=0.5,
        cfg_weight=1.5,
        temperature=0.8,
        top_p=0.9,
        repetition_penalty=1.2,
    )

    wav = torch.clamp(wav, -0.99, 0.99)

    if wav.dim() == 1:
        wav = wav.unsqueeze(0)

    torchaudio.save(output_file, wav.cpu(), model.sr)

    print(f"Saved: {output_file}")


# =========================
# MAIN
# =========================
if __name__ == "__main__":
    generate_tts("नेपाल एक सुन्दर देश हो।")

Acknowledgments

Fine-tuned by ayushkhadkaa.