| --- |
| language: |
| - hi |
| - en |
| tags: |
| - text-to-speech |
| - tts |
| - flow-matching |
| - diffusion-transformer |
| - voice-cloning |
| - zero-shot |
| - elevenlabs-style |
| - pytorch |
| license: apache-2.0 |
| pipeline_tag: text-to-speech |
| --- |
| |
| # 🎙️ ViuAI_TTS_200M: Next-Gen 2026 Speech Generation Model (~190.5M Parameters) |
|
|
| **ViuAI_TTS_200M** is a state-of-the-art (SOTA) Text-to-Speech deep learning architecture designed for human-grade speech synthesis and zero-shot voice cloning. |
|
|
| Inspired by modern breakthrough architectures (Flow Matching, DiT, ElevenLabs Adam, and Grok Voice), ViuAI_TTS_200M eliminates robotic artifacts through: |
| * **Optimal Transport Continuous Normalizing Flows (OT-CFM):** Generates crisp, clean speech in only 10–15 Euler ODE steps. |
| * **Diffusion Transformer (DiT) Backbone:** 9 deep DiT layers (~148.7M params) equipped with **AdaLN-Zero** and multi-head cross-attention. |
| * **Human Prosody Engine ($F_0$ Pitch + Energy + Duration):** Prevents robotic flat pitch by modeling dynamic intonation curves, emotional inflection, and natural micro-breaths. |
| * **In-Context Reference Voice Cloning:** Feed a 3–5 second human audio clip (`--ref_audio sample.wav`) to replicate any target voice's timbre, warmth, and cadence. |
| * **Classifier-Free Guidance (CFG Scale = 2.0x):** Delivers clean studio broadcast clarity. |
| * **24,000 Hz Neural Vocoder:** Multi-receptive field periodic generator producing uncompressed 24kHz 16-bit audio. |
|
|
| --- |
|
|
| ## 📊 Architecture & Parameter Budget Breakdown |
|
|
| | Component | Architecture Specs | Trainable Parameters | |
| | :--- | :--- | :--- | |
| | **Text & Prosody Encoder** | Pre-LN Transformer + RoPE (Multilingual) | **19.43 Million** | |
| | **Human Prosody Engine** | Dynamic $F_0$ Pitch + Energy + Duration ConvNeXt | **8.42 Million** | |
| | **DiT Core Backbone** | 9-layer Diffusion Transformer + AdaLN-Zero + CFG | **148.71 Million** | |
| | **Neural Audio Vocoder** | 24,000 Hz Multi-Receptive Periodic Generator | **13.93 Million** | |
| | **Total Model Parameters** | | **190.49 Million (~190.5M)** | |
| |
| --- |
| |
| ## 🚀 Quickstart & Inference |
| |
| ### 1. Zero-Shot Human Voice Cloning (ElevenLabs / Adam Style) |
| Clone any human voice by supplying a 3–5 second `.wav` audio prompt: |
| |
| ```bash |
| python inference.py \ |
| --text "Hello! This is ViuAI_TTS_200M speaking in a human-like voice." \ |
| --ref_audio "path/to/reference_sample.wav" \ |
| --cfg_scale 2.0 \ |
| --steps 15 \ |
| --output "cloned_output.wav" |
| ``` |
| |
| ### 2. Standard Text-to-Speech |
| ```bash |
| python inference.py \ |
| --text "Namaste! Yeh ViuAI TTS 200M model ka audio output hai." \ |
| --cfg_scale 2.0 \ |
| --steps 15 \ |
| --output "speech.wav" |
| ``` |
| |
| --- |
| |
| ## 🏋️ Training Pipeline |
| |
| Train with Mixed Precision (AMP), AdamW, and Cosine Annealing: |
| ```bash |
| python train.py |
| ``` |
| |
| Checkpoints are automatically saved to `checkpoints/` and can be loaded directly for inference. |
| |
| --- |
| |
| ## 📂 Project Repository Structure |
| ``` |
| ViuAI_TTS_200M/ |
| ├── config/ |
| │ └── model_config.py # 190.5M architectural hyperparameters |
| ├── models/ |
| │ ├── text_encoder.py # Multilingual Transformer with RoPE |
| │ ├── dit_backbone.py # Diffusion Transformer with Voice Conditioning |
| │ ├── duration_predictor.py # Human Prosody Engine (F0, Energy, Duration) |
| │ ├── flow_matching.py # Optimal Transport CFM & Euler ODE Solver |
| │ ├── vocoder.py # 24kHz Neural Audio Vocoder |
| │ └── viuai_tts.py # Unified ViuAITTS200M Model |
| ├── dataset/ |
| │ ├── audio_processing.py # Mel-spectrogram extraction |
| │ └── tokenizer.py # Hindi & English multilingual tokenizer |
| ├── checkpoints/ |
| │ └── viuai_tts_200m_init.pt # Pretrained / initialized model weights |
| ├── train.py # Mixed Precision (AMP) training script |
| ├── inference.py # Text-to-speech audio generator |
| ├── verify_model.py # Architecture verification script |
| └── ROADMAP.md # Master engineering roadmap |
| ``` |
| |
| --- |
| |
| ## 📜 License |
| Apache 2.0 |
| |
