| | --- |
| | license: apache-2.0 |
| | --- |
| | |
| | A [SoundStream](https://arxiv.org/abs/2107.03312) decoder to reconstruct audio from a mel-spectrogram. |
| |
|
| | ## Overview |
| |
|
| | This model is a SoundStream decoder which inverts mel-spectrograms computed with the specific hyperparameters defined in the example below. This model was trained on music data and used in [Multi-instrument Music Synthesis with Spectrogram Diffusion](https://arxiv.org/abs/2206.05408) (ISMIR 2022). |
| |
|
| | A typical use-case is to simplify music generation by predicting mel-spectrograms (instead of a raw waveform), and then use this model to reconstruct audio. |
| |
|
| | If you use it, please consider citing: |
| |
|
| | ```bibtex |
| | @article{zeghidour2021soundstream, |
| | title={Soundstream: An end-to-end neural audio codec}, |
| | author={Zeghidour, Neil and Luebs, Alejandro and Omran, Ahmed and Skoglund, Jan and Tagliasacchi, Marco}, |
| | journal={IEEE/ACM Transactions on Audio, Speech, and Language Processing}, |
| | volume={30}, |
| | pages={495--507}, |
| | year={2021}, |
| | publisher={IEEE} |
| | } |
| | ``` |
| |
|
| | ## Example Use |
| |
|
| | ```python |
| | from diffusers import OnnxRuntimeModel |
| | |
| | |
| | SAMPLE_RATE = 16000 |
| | N_FFT = 1024 |
| | HOP_LENGTH = 320 |
| | WIN_LENGTH = 640 |
| | N_MEL_CHANNELS = 128 |
| | MEL_FMIN = 0.0 |
| | MEL_FMAX = int(SAMPLE_RATE // 2) |
| | CLIP_VALUE_MIN = 1e-5 |
| | CLIP_VALUE_MAX = 1e8 |
| | |
| | mel = ... |
| | |
| | melgan = OnnxRuntimeModel.from_pretrained("kashif/soundstream_mel_decoder") |
| | |
| | audio = melgan(input_features=mel.astype(np.float32)) |
| | ``` |
