- 🎙️ Tui Voice Mini (Malayalam) — 4.5M Parameter Edge TTS (v0.1-preview)
- ⚡ Key Highlights
- 📦 Released Model Weights & Formats
- 🎧 Audio Showcase & Real-World Demos
- 💻 Quickstart & Python Usage
- 🔬 Architecture Specifications
- 📈 Training Convergence Metrics (Step 40,000)
- 📊 Benchmark Comparison vs. Other Malayalam TTS Models
- 📜 Research Benchmarking Suite
- ⚠️ Current Status & Known Limitations
- ✍️ Citation & Attribution
- ⚡ Key Highlights
🎙️ Tui Voice Mini (Malayalam) — 4.5M Parameter Edge TTS (v0.1-preview)
Experimental Research Prototype (
v0.1-preview): This model is an early research proof-of-concept for ultra-compact, on-device Malayalam speech synthesis. It is not a finished commercial voice. While intelligible and extremely lightweight (~10 MB), it exhibits synthetic timbre artifacts inherent to low-parameter source-filter vocoding. See Current Status & Known Limitations below.
Tui Voice Mini Malayalam (tui-voice-mini-ml) is an experimental, ultra-compact Malayalam Text-to-Speech (TTS) neural research model. Fine-tuned from tiny-voca on high-fidelity studio recordings from the AI4Bharat Rasa Malayalam female corpus, it explores the lower limits of neural speech footprint with just 4,537,291 parameters (~10 MB footprint).
Unlike conventional neural TTS pipelines (e.g., FastPitch + HiFi-GAN at ~110M params or Meta MMS-TTS VITS at ~36M params) requiring dedicated cloud GPUs and large memory buffers, Tui Voice Mini runs 100% locally on-device—in web browsers via WebAssembly, on iOS/Android, and on low-power ARM microcontrollers like Raspberry Pi and ROS robots.
⚡ Key Highlights
- True On-Device Footprint: Only 4.5M parameters. Requires < 30 MB RAM at inference time.
- Integrated Source-Filter Synthesizer: Direct harmonic oscillator and filtered noise excitation—requires zero external neural vocoder (no separate 100 MB HiFi-GAN or Vocos needed).
- Native Runtime Prosody Control: Algebraic
--pitch-shiftand--pitch-rangecontrols to manipulate voice timbre and pitch contour with zero latency. - Full Malayalam G2P Coverage: Explicit 76-slot budget mapping covering all 59 Malayalam phonemes, long/short vowel distinctions, conjunct clusters, and explicit Chillus (ൺ, ൻ, ർ, ൽ, ൾ, ൿ).
- Adversarial Training Lineage: Trained for 40,000 steps with multi-period and multi-resolution adversarial GAN discriminators.
📦 Released Model Weights & Formats
| File Name | Format | Size | Target Deployment |
|---|---|---|---|
tui-voice-mini-ml.pt |
PyTorch FP32 | 18.2 MB | Standard Python / PyTorch deployment |
tui-voice-mini-ml-fp16.pt |
PyTorch FP16 | 9.1 MB | High-speed GPU / low-memory Python inference |
onnx/tui-voice-mini-ml.onnx |
ONNX (opset 17) | 21.4 MB | WebAssembly (Web Browser), Android, iOS, C++ |
onnx/tui-voice-mini-ml-int8.onnx |
Quantized INT8 | ~5.4 MB | Raspberry Pi, ROS Robots, Edge Devices |
config.json |
JSON | 2.1 KB | Model hyperparameters and 76-token vocabulary |
vocab_plan.json |
JSON | 4.2 KB | Upstream phoneme slot substitution record |
checkpoints/tui-voice-mini-ml-step40k.pt |
PyTorch Full | 87.2 MB | Final step-40,000 training state (weights + optimizer) |
checkpoints/tui-voice-mini-ml-step*.pt |
PyTorch Full | ~87 MB | Checkpoint snapshots at steps 5k, 8k, 10k, 15k, 20k, 25k, 30k, 35k |
🎧 Audio Showcase & Real-World Demos
Listen to actual audio clips generated by Tui Voice Mini across 6 distinct linguistic domains:
| Category | Example Malayalam Input Text | Audio Sample |
|---|---|---|
| Friendly Check-in | ഞാൻ ഇപ്പോൾ വീട്ടിൽ എത്തി, അവിടെ കാര്യങ്ങൾ ഒക്കെ ശരിയായോ? | audio_samples/everyday_whatsapp_reached_home.wav |
| Everyday Greeting | ഹലോ, സുഖമാണോ? എവിടെയാണ് ഇപ്പോൾ? | audio_samples/everyday_whatsapp_hello_sugamano.wav |
| Breaking Weather News | പ്രധാന വാർത്തകൾ. സംസ്ഥാനത്ത് അടുത്ത അഞ്ച് ദിവസം ശക്തമായ മഴയ്ക്ക് സാധ്യതയുണ്ടെന്ന് കാലാവസ്ഥാ വകുപ്പ് അറിയിച്ചു. | audio_samples/news_and_media_news_weather_warning.wav |
| Kochi Metro Station | അടുത്ത സ്റ്റേഷൻ എം ജി റോഡ്. വാതിലുകൾ ഇടത്തു വശത്ത് തുറക്കുന്നതാണ്. | audio_samples/public_announcements_metro_station.wav |
| Railway Platform | ശ്രദ്ധിക്കുക, തിരുവനന്തപുരത്തു നിന്നും കോഴിക്കോട്ടേക്കുള്ള ജനശതാബ്ദി എക്സ്പ്രസ്സ് രണ്ടാം നമ്പർ പ്ലാറ്റ്ഫോമിൽ എത്തിച്ചേരുന്നു. | audio_samples/public_announcements_railway_station.wav |
| Customer Support (IVR) | ഞങ്ങളുടെ കസ്റ്റമർ കെയർ സർവീസിലേക്ക് സ്വാഗതം. മലയാളത്തിൽ തുടരുന്നതിനായി ഒന്നിൽ അമർത്തുക. | audio_samples/customer_care_ivr_ivr_welcome.wav |
| On-Device Edge AI | ഇന്റർനെറ്റ് ഇല്ലാതെ തന്നെ ഏത് മൊബൈൽ ഫോണിലും ഈ ശബ്ദ സാങ്കേതികവിദ്യ തത്സമയം പ്രവർത്തിക്കും. | audio_samples/ai_tech_showcase_tech_edge_ai.wav |
(All audio clips are available in the audio_samples/ directory).
💻 Quickstart & Python Usage
1. Installation
# Clone the official repository
git clone https://github.com/Cyrax321/tui-voice.git
cd tui-voice
pip install -r requirements.txt
# Install eSpeak NG for Malayalam phonemization
# Ubuntu/Debian: sudo apt-get install espeak-ng
# macOS: brew install espeak-ng
2. Synthesize Speech with PyTorch
import torch
import soundfile as sf
from tui_voice import phonemes as P
from tiny_voca.config import Config
from tiny_voca.model import TinyVoca
from huggingface_hub import hf_hub_download
import json
# 1. Download model & config from Hugging Face
model_path = hf_hub_download("Cyrax321/tui-voice-mini-ml", "tui-voice-mini-ml.pt")
config_path = hf_hub_download("Cyrax321/tui-voice-mini-ml", "config.json")
plan_path = hf_hub_download("Cyrax321/tui-voice-mini-ml", "vocab_plan.json")
# 2. Initialize model
cfg = Config.from_dict(json.load(open(config_path)))
model = TinyVoca(cfg)
state = torch.load(model_path, map_location="cpu", weights_only=False)
model.load_state_dict(state.get("model", state), strict=False)
model.eval()
# 3. Phonemize & Encode input text
text = "നമസ്കാരം, ഇത് തുയി വോയ്സ് മിനി മലയാളം മോഡൽ ആണ്."
plan = json.load(open(plan_path))
tokens = P.phonemize(text)
token_ids = P.encode(tokens, plan)
tok_tensor = torch.tensor([token_ids], dtype=torch.long)
stress = torch.zeros_like(tok_tensor)
lengths = torch.tensor([len(token_ids)], dtype=torch.long)
speaker = torch.tensor([0], dtype=torch.long)
# 4. Synthesize 24 kHz audio
with torch.inference_mode():
audio, _, _ = model.synthesize(
tok_tensor, stress, lengths, speaker,
speed=1.0,
pitch_shift=0.15, # Locks into natural Malayalam female register
pitch_range=0.85, # Smooth, consistent melodic contour
voiced_vowels=True
)
sf.write("output.wav", audio.squeeze(0).numpy(), 24000)
print("Saved 24 kHz audio to output.wav!")
3. Lightweight ONNX Runtime Inference (Zero PyTorch Dependency)
import onnxruntime as ort
import numpy as np
import soundfile as sf
from huggingface_hub import hf_hub_download
# Download ONNX model (21.4 MB) or INT8 model (5.4 MB)
onnx_path = hf_hub_download("Cyrax321/tui-voice-mini-ml", "onnx/tui-voice-mini-ml.onnx")
session = ort.InferenceSession(onnx_path)
# Run direct graph evaluation
one = lambda v: np.array([v], dtype=np.float32)
audio, durations = session.run(None, {
"tokens": np.array([[12, 45, 23, 67, 18]], dtype=np.int64),
"stress": np.zeros((1, 5), dtype=np.int64),
"voice": np.array([0], dtype=np.int64),
"speed": one(1.0),
"pitch_shift": one(0.15),
"pitch_range": one(0.85),
})
sf.write("onnx_output.wav", audio[0], 24000)
🔬 Architecture Specifications
| Module | Component Name | Parameters | % of Model | Function |
|---|---|---|---|---|
| Embeddings | phoneme_embed, stress_embed, voice_embed |
16,704 | 0.37% | Maps 76 phonemes + stress + voice into 192-dim vector space |
| Text Encoder | encoder |
1,588,864 | 35.02% | 4 Transformer blocks, 192 channels, 4 heads, RoPE attention |
| Duration Head | duration_head, pause_head |
123,906 | 2.73% | Predicts frame durations and punctuation pauses |
| Prosody Predictor | frame_transformer_a + f0_head |
795,971 | 17.54% | 2 Transformer blocks predicting continuous $F_0$ and voicing |
| Cepstral Decoder | frame_transformer_b |
1,985,984 | 43.77% | 5 Transformer blocks predicting acoustic filter coefficients |
| Source-Filter Head | synth_head |
25,862 | 0.57% | Differentiable harmonic pulse-train and noise filter |
| Total Model | Complete Inference Pipeline | 4,537,291 | 100.0% | End-to-End waveform generation (24 kHz) |
📈 Training Convergence Metrics (Step 40,000)
Tui Voice Mini was trained on an NVIDIA L4 GPU with high-RAM for 40,000 steps using AdamW optimizer and cosine learning rate decay:
- Validation Pause Accuracy:
1.000(100% exact punctuation pause placement) - Validation Duration Length Error:
0.097(< 9.8% cadence error) - Validation Voicing Accuracy:
93.4%(crisp distinction between voiced vowels and unvoiced consonants) - Final Pitch Error ($F_0$ RMSE):
3.73 semitones(down from8.61at step 10) - Discriminator GAN Loss:
0.277(stable adversarial equilibrium) - Teacher Forcing Fraction:
0.0(fully autonomous, free-running speech synthesis)
📊 Benchmark Comparison vs. Other Malayalam TTS Models
| Metric | AI4Bharat Indic-TTS | Meta MMS-TTS | Piper Malayalam | Tui Voice Mini (Ours) |
|---|---|---|---|---|
| Architecture | FastPitch + HiFi-GAN | VITS (CVAE) | VITS-derived | Source-Filter + GAN |
| Parameters | ~110 Million | ~36 Million | ~15.7 Million | 4.5 Million (8x – 25x Smaller!) |
| Disk Size | ~350 MB – 450 MB | ~145 MB | ~60 MB | 18.2 MB (FP32) / 5.4 MB (INT8) |
| RAM Footprint | ~1.2 GB+ | ~450 MB | ~200 MB | < 30 MB (Ultra-Lightweight) |
| CPU RTF (Latency) | 0.80 – 1.20 | 0.30 – 0.45 | 0.25 – 0.35 | 0.08 – 0.12 (10x Real-Time) |
| Browser (WASM) Offline | ❌ No | ❌ No | ⚠️ Heavy | ✅ YES (Runs in browser) |
| Raspberry Pi / Robots | ❌ No | ⚠️ Slow | ⚠️ Moderate | ✅ YES (Optimized INT8) |
| Runtime Pitch Control | ❌ No | ❌ No | ❌ No | ✅ Full algebraic control |
📜 Research Benchmarking Suite
The complete benchmarking suite and data splits are included in the repository:
benchmark/data/splits/dev_set.jsonl: 100 sentences for pipeline validation.benchmark/data/splits/main_test_set.jsonl: 500 sentences balanced across Conversational (100), News/Formal (100), Long Compounds (75), Questions (75), Numbers/Currency (75), and Code-Switching (75).benchmark/data/splits/stress_test_set.jsonl: 200 sentences testing rare conjuncts and tongue twisters.- Acoustic & Integrity Metrics: Grapheme CER (
INT-01), WER (INT-02), SER (INT-03), Audio Integrity (AUD-01toAUD-07), F0 RMSE (PRO-01), and Efficiency (EFF-01toEFF-11).
To reproduce the benchmark:
python3 benchmark/scripts/prepare_dataset.py
⚠️ Current Status & Known Limitations
Tui Voice Mini is released as an experimental research prototype (v0.1-preview), not a commercial production system:
- Synthetic Timbre & Vocoding: To achieve a ~10 MB footprint and run without an external neural vocoder, Tui Voice Mini employs a lightweight source-filter harmonic oscillator + noise generator. While speech is intelligible, it produces an audible buzz/robotic timbre compared to 100M+ parameter vocoders (e.g. HiFi-GAN, BigVGAN).
- Single Speaker Profile: The model is currently trained on a single Malayalam female speaker (
female_01from AI4Bharat Rasa). Multi-speaker or male voices are not yet supported. - Number & Symbol Verbalization: Raw digits (e.g.
123,2026) and English symbols (%,$) are not automatically expanded. All inputs must be converted into full Malayalam words (e.g.നൂറ്റിയിരുപത്തിമൂന്ന്) before synthesis. - Prosody on Long Paragraphs: Synthesizing lengthy compound texts without punctuation pauses can result in flat or monotonous intonation contours.
- Pending Benchmarking Verification: Objective intelligibility metrics (CER/WER via IndicConformer) and human subjective listening evaluations (MOS/CMOS) are currently ongoing across the 500-sentence test split.
✍️ Citation & Attribution
@software{tuivoice2026malayalam,
author = {Cyrax321},
title = {Tui Voice Mini: Ultra-Compact 4.5M Parameter Malayalam Text-to-Speech},
year = {2026},
publisher = {Hugging Face / GitHub},
url = {https://huggingface.co/Cyrax321/tui-voice-mini-ml}
}
- Model Author: Cyrax321 (GitHub: Cyrax321/tui-voice)
- Base Architecture: Muhammed Nazeem (tiny-voca)
- Dataset: AI4Bharat (Rasa Malayalam Corpus)
- License: Apache-2.0
- Downloads last month
- 130