README.md

---
title: VibeVoice Hindi Test
emoji: 🎤
colorFrom: yellow
colorTo: red
sdk: gradio
sdk_version: "5.50.0"
app_file: app.py
hardware: zero-a10g
pinned: false
---

# VibeVoice-Hindi-1.5B Test Suite

Test and evaluate the fine-tuned Hindi text-to-speech model from HuggingFace.

## Quick Start

### Option 1: Deploy to HF Spaces (Recommended)

1. **Create a new HF Space:**
   - Go to https://huggingface.co/new-space
   - Space name: `vibevoice-hindi-test`
   - License: MIT
   - Space SDK: Docker
   - Private: No (or Yes if you prefer)

2. **Clone this repo into the Space:**
   ```bash
   git clone https://huggingface.co/spaces/YOUR-USERNAME/vibevoice-hindi-test
   cd vibevoice-hindi-test
   ```

3. **Push files:**
   ```bash
   git add .
   git commit -m "Initial VibeVoice-Hindi test suite"
   git push
   ```

4. **HF Spaces will auto-build and deploy** (takes ~5 min)
   - Access at: `https://huggingface.co/spaces/YOUR-USERNAME/vibevoice-hindi-test`

### Option 2: Run Locally

```bash
pip install -r requirements.txt
python app.py
```

Then open `http://localhost:7860` in your browser.

---

## Test Cases Included

| Test | Input | Purpose |
|------|-------|---------|
| **Greeting** | नमस्ते, आपका स्वागत है। आज आपका दिन कैसा चल रहा है? | Test casual speech, daily conversation |
| **Business** | हमारी कंपनी गुणवत्ता और नवाचार में विश्वास करती है। | Test formal tone, business terminology |
| **Numbers** | साल २०२६ में हमने बीस लाख रुपये की आय प्राप्त की। | Test numeral pronunciation, decimal points |
| **Mixed** | आप YouTube पर हमारा नया वीडियो देख सकते हैं। | Test English words mixed with Hindi |

---

## Evaluation Rubric

After generating speech, rate on 5 dimensions:

- **📖 Understand** — Can you follow what's being said?
- **🎵 Natural** — Does it sound like a real person (not robot)?
- **⚡ Speed** — Is the inference latency acceptable (<5s)?
- **🔊 Clarity** — Can you hear each word clearly?
- **🎯 Consistency** — Same pronunciation throughout utterance?

### Scoring

- **5/5 ✓** → 🟢 Production Ready
- **4/5 ✓** → 🟡 Good, Minor Issues
- **3/5 ✓** → 🟠 Acceptable, Needs Work
- **2/5 ✓** → 🔴 Poor Quality
- **1/5 ✓** → ❌ Archive This Model

---

## Evaluation Results

All evaluations are saved to `evaluation_log.jsonl` with:
- Timestamp
- Test case name
- Dimensions passed (0-5)
- Optional notes
- Device used (CPU/GPU)

---

## Model Info

- **Model ID:** `tarun7r/vibevoice-hindi-1.5B`
- **Base:** Qwen2.5-1.5B (text understanding)
- **Voice:** Diffusion-based speech generation (~600M params)
- **Fine-tuning:** LoRA on LLM + full diffusion head tuning
- **License:** MIT
- **Size:** 3.5GB (fits T4 GPU, ZeroGPU Spaces)

---

## Troubleshooting

**Q: "CUDA out of memory"**
- Reduce batch size or use CPU inference (slower but works on Spaces)

**Q: Audio generation is placeholder only**
- This is a demo scaffold. Integrate actual VibeVoice inference from: https://github.com/tarun7r/vibevoice-tts

**Q: Model not loading**
- Check internet connection (HF model download)
- Verify HF token if model is private (use `huggingface-cli login`)

---

## Next Steps

1. ✅ Run the 4 test cases
2. ✅ Listen and evaluate each dimension
3. ✅ Take notes on pronunciation issues
4. ✅ Save evaluations
5. 📊 Review `evaluation_log.jsonl` for patterns
6. 🎯 Decide: Production (1.5B) or test 7B variant?

**Estimated time:** 15-20 minutes for full evaluation suite