README.md

---
license: apache-2.0
tags:
  - audio
  - vocoder
  - speech-synthesis
  - streaming
  - pytorch-lightning
  - causal-conv
language:
  - en
library_name: pytorch
---

# Streaming Vocos: Neural vocoder for fast streaming applications

**Streaming Vocos** is a streaming-friendly replication of the original **Vocos** neural vocoder design, modified for **causal / streaming inference**. Compared to typical GAN vocoders that generate waveform samples in the time domain, Vocos predicts **spectral coefficients**, enabling fast waveform reconstruction via inverse Fourier transform—making it well-suited for **low-latency** and **real-time** settings.

This implementation replaces vanilla CNN blocks with **causal CNNs** and provides a **streaming interface** with dynamically adjustable chunk size (in multiples of the hop size).

- **Input:** 50 Hz log-mel spectrogram  
  - window = 1024, hop = 320
- **Output:** 16 kHz waveform audio

Training follows the GAN objective as in the original Vocos, while adopting loss functions inspired by Descript’s audio codec.

**Original Vocos resources:**  
- Audio samples: https://gemelo-ai.github.io/vocos/  
- Paper: https://arxiv.org/abs/2306.00814

---

## ⚡ Streaming Latency & Real-Time Performance

We benchmark **Streaming Vocos** in **streaming inference mode** using chunked mel-spectrogram decoding on both CPU and GPU.

### Benchmark setup

- **Audio duration:** 3.24 s  
- **Sample rate:** 16 kHz  
- **Mel hop size:** 320 samples (20 ms per mel frame)  
- **Chunk size:** 5 mel frames (100 ms buffering latency)  
- **Runs:** 100 warm-up + 1000 timed runs  
- **Inference mode:** Streaming (stateful causal decoding)  

**Metrics**
- **Processing time per chunk**
- **End-to-end latency** = chunk buffering + processing time
- **RTF (Real-Time Factor)** = processing time / audio duration

---

### Results

#### Streaming performance (chunk size = 5 frames, 100 ms buffer)

| Device | Avg proc / chunk | First-chunk proc | End-to-end latency | Total proc (3.2 s audio) | RTF |
|------|------------------|------------------|--------------------|---------------------------|-----|
| **CPU** | 14.0 ms | 14.0 ms | **114.0 ms** | 464 ms | 0.14 |
| **GPU (CUDA)** | **3.4 ms** | **3.3 ms** | **103.3 ms** | **113 ms** | **0.035** |

> End-to-end latency includes the **100 ms chunk buffering delay** required for streaming inference.

---

### Interpretation

- **Real-time capable on CPU**  
  Streaming Vocos achieves an RTF of approximately **0.14**, corresponding to inference running ~7× faster than real time.

- **Ultra-low compute overhead on GPU**  
  Chunk processing time is reduced to **~3.4 ms**, making overall latency dominated by buffering rather than computation.

- **Streaming-friendly first-chunk behavior**  
  First-chunk latency closely matches steady-state latency, indicating **no cold-start penalty** during streaming inference.

- **Latency–quality tradeoff**  
  Smaller chunk sizes further reduce buffering latency (e.g., 1–2 frames → <40 ms), at the cost of slightly increased computational overhead.

---

With a **chunk size of 1 frame (20 ms buffering)**, GPU end-to-end latency drops below **25 ms**, making **Streaming Vocos** suitable for **interactive and conversational TTS pipelines**.


## Checkpoints

This repo provides a PyTorch Lightning checkpoint:

- `epoch=3.ckpt`

You can download it from the “Files” tab, or directly via `hf_hub_download` (example below).

---

## Quickstart (inference)

### Install
```bash
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121  # or cpu wheels
pip install lightning librosa scipy matplotlib huggingface_hub
```

Clone the github repo
```bash
git clone https://github.com/warisqr007/vocos.git 
cd vocos
```

### Run inference (offline)
```python
import torch
import librosa
from huggingface_hub import hf_hub_download

from src.modules import VocosVocoderModule  # from the github codebase

ckpt_path = hf_hub_download(
    repo_id="warisqr007/StreamingVocos",
    filename="epoch=3.ckpt",
)

model = VocosVocoderModule.load_from_checkpoint(ckpt_path, map_location="cpu")
model.eval()

wav_path = "your_input.wav"
audio, _ = librosa.load(wav_path, sr=16000, mono=True)

audio_t = torch.from_numpy(audio).unsqueeze(0).unsqueeze(0)  # (B=1,1,T)
mel = model.feature_extractor(audio_t)

with torch.no_grad():
    y = model(mel).squeeze().cpu().numpy()  # reconstructed waveform @ 16kHz

```

### Streaming inference (chunked mel)
```python
import torch

chunk_size = 1  # mel frames per chunk (adjust as desired)

with torch.no_grad(), model.decoder[0].streaming(chunk_size), model.decoder[1].streaming(chunk_size):
    y_chunks = []
    for mel_chunk in mel.split(chunk_size, dim=2):
        y_chunks.append(model(mel_chunk))
    y_stream = torch.cat(y_chunks, dim=2).squeeze().cpu().numpy()
```

### Space demo
A Gradio demo Space is provided [here](https://huggingface.co/spaces/warisqr007/StreamingVocos_16khz)


### Acknowledgements
- [Vocos Repo](https://github.com/gemelo-ai/vocos)
- [Moshi Repo for streaming implementation](https://github.com/kyutai-labs/moshi)
- [descript-audio-codec losses](https://github.com/descriptinc/descript-audio-codec)
- [lightning-template](https://github.com/DavidZhang73/pytorch-lightning-template)