README.md

---
library_name: transformers
tags:
  - audio
  - soundstream
license: mit
language:
  - en
pipeline_tag: audio-to-audio
---

# SoundStream

A PyTorch implementation of the [SoundStream](https://arxiv.org/abs/2107.03312) neural audio codec. Accepts only 16 kHz audio.

Encodes speech into discrete tokens (8 codebooks × 80 tokens/sec) and decodes them back to audio.

## Metrics

Evaluated on LibriSpeech test-clean:

- **STOI**: 0.804
- **NISQA**: 2.276

## Architecture

- **Encoder**: Causal convolutions with residual units and strided downsampling (2, 4, 5, 5 = 200x compression)
- **Quantizer**: Residual Vector Quantizer with 8 codebooks of 1024 entries each
- **Decoder**: Mirrored encoder with transposed convolutions
- **Discriminator** (training only): 3 multi-scale waveform discriminators + 1 STFT-based discriminator

**Model parameters**: 16 kHz, 32 channels, latent dim 512, codebook size 1024, 8 quantizers, 200x downsampling

## Usage

```python
import torchaudio
from transformers import AutoModel

# Load model
model = AutoModel.from_pretrained("timofeiiz/soundstream-impl", trust_remote_code=True)
model.eval()

waveform, sr = torchaudio.load("audio.wav")
assert sr == 16000  # Only 16 kHz sample rate is supported

# Encode to discrete tokens
indices = model.encode(waveform.unsqueeze(0))  # (1, 8, T)

# Decode back to audio
reconstructed = model.decode(indices, original_length=waveform.size(-1))

torchaudio.save("reconstructed.wav", reconstructed.squeeze(0).cpu(), 16000)
```

## License

MIT