| --- |
| license: mit |
| library_name: stftvae |
| pipeline_tag: audio-to-audio |
| tags: |
| - audio |
| - codec |
| - vae |
| - stft |
| - neural-audio |
| --- |
| |
| # STFT-VAE |
|
|
| Continuous (VAE) neural audio codec built on the STFT. Audio is transformed to a |
| complex spectrogram, a transformer encoder downsamples it to a low-rate |
| continuous latent, and a transformer decoder + ISTFT reconstruct the waveform — |
| phase is carried by the STFT/ISTFT rather than a learned vocoder. |
|
|
| - **Sample rate:** 24 kHz |
| - **Latent:** 3.125 Hz frame rate, 128-dim continuous (no quantization) |
| - **Params:** ~116M |
| - **Files:** `config.json` (architecture) + `model.safetensors` (fp32 weights) |
|
|
| ## Usage |
|
|
| ```bash |
| pip install "stftvae[cli]" |
| ``` |
|
|
| ```python |
| import torch |
| from stftvae import STFTVAE |
| |
| vae = STFTVAE.from_pretrained("fluxions/stftvae", device="cuda") |
| |
| audio = torch.randn(1, 24000) # (1, T) mono @ 24 kHz |
| recons = vae.reconstruct(audio) # (1, 1, T) |
| |
| # or encode / decode separately |
| latent, length = vae.encode(audio) # (1, 128, T_latent) |
| recons = vae.decode(latent, length=length) |
| ``` |
|
|
| Command line: |
|
|
| ```bash |
| python -m stftvae input.wav output.wav --model fluxions/stftvae |
| ``` |
|
|
| The encoder returns the deterministic posterior mean (no sampling noise at |
| inference). Code: https://github.com/fluxions-ai/stftvae |
|
|
| ## License |
|
|
| MIT |
|
|
