README.md

---
license: mit
library_name: diffusers
pipeline_tag: audio-to-audio
tags:
  - audio
  - audio-autoencoder
  - neural-vocoder
  - audio-reconstruction
  - feature-extraction
  - pytorch
metrics:
  - pesq
  - stoi
  - wer
  - cer
  - fad
---

<!-- TODO before public release: add paper and demo/Space links to Model Sources. -->

# Motif-Audio: A General-Purpose Audio Foundation Model

Motif-Audio is a general-purpose audio foundation model. One model covers work that usually needs
three: understanding sound, compressing and rebuilding it, and giving generative models a space to
build in.

- **Understand.** It reads emotion in a voice, genre in a song, and events in everyday sound, and
  those abilities carry from one task to the next.
- **Reconstruct.** As a continuous neural codec, it compresses audio to a compact continuous
  representation and rebuilds the waveform at quality on par with the best neural codecs.
- **Generate.** It gives text-to-audio and music generation models a foundation to build on. They
  learn to produce its latent, and the model turns that into sound.

The design is a dual-stream autoencoder. One stream follows content, the other follows fine spectral
detail, and a cross-attention fusion brings them together. That separation is what lets a single
model do the work of three.

## Model details

- Developed by: Motif Technologies
- Model type: dual-stream audio autoencoder
- Sample rate: 16 kHz
- Latent: continuous, width 1024
- Parameters: 726M total (605M frozen semantic encoder; 121M trained: acoustic encoder, fusion, decoder)
- License: MIT

## Architecture

<p align="center">
  <img src="assets/architecture.png" alt="Motif-Audio dual-stream architecture: a frozen MSM-MAE semantic stream and a lightweight Conv2d patch-embed acoustic stream are combined by cross-attention fusion into a continuous latent, which a Vocos decoder and iSTFT turn back into a 16 kHz waveform" width="900">
</p>

Motif-Audio runs a raw 16 kHz waveform through two parallel encoders, a cross-attention fusion, a
continuous latent, and a convolutional decoder that inverts back to audio (see the figure
above):

- **Semantic encoder (frozen).** Reads an 80-bin mel spectrogram and captures content: the structure
  of what is being said or played, largely independent of fine acoustic variation.
- **Acoustic encoder (trained).** Reads a higher-resolution 160-bin mel spectrogram and captures fine
  spectral detail: the timbre and texture needed for a faithful waveform.
- **Cross-attention fusion (trained).** Merges the two streams into a single continuous latent.
- **Decoder (trained).** A ConvNeXt backbone with an inverse-STFT head reconstructs the waveform.

Keeping content and acoustic detail in separate streams lets the decoder preserve both, and makes the
latent a semantic representation rather than just a compressed spectrum. Both mel spectrograms are
computed inside the model, so it takes a raw waveform directly, with no external feature extraction.

Training runs in two stages. The semantic encoder is first pretrained on its own with masked
spectrogram modeling, predicting masked time-frequency regions from the surrounding context to learn
content from unlabeled audio. It is then frozen while the acoustic encoder, fusion, and decoder train
jointly to reconstruct the waveform, so the acoustic stream learns only the detail the semantic stream
leaves out.

## How to get started

Install the dependencies:

```bash
pip install torch torchaudio torchcodec diffusers timm
```

Recent torchaudio releases decode audio through `torchcodec`, so `torchaudio.load`
in the example below raises `ImportError` unless it is installed.

```python
import torch
import torchaudio
from diffusers import AutoModel


def pad_for_model(waveform, hop_length=160, patch_multiple=16):
    """Pad to the next hop-aligned length whose frame count divides the patch size."""
    orig_len = waveform.shape[-1]
    frames = orig_len // hop_length + 1
    target_frames = -(-frames // patch_multiple) * patch_multiple
    target_len = hop_length * (target_frames - 1)
    if target_len < orig_len:
        target_len = hop_length * (target_frames + patch_multiple - 1)
    return torch.nn.functional.pad(waveform, (0, target_len - orig_len))


# The modeling code ships with the weights, so trust_remote_code=True is required.
model = AutoModel.from_pretrained("Motif-Technologies/Motif-Audio", trust_remote_code=True)
model = model.to("cuda", torch.bfloat16).eval()

waveform, sr = torchaudio.load("input.wav")
waveform = torchaudio.functional.resample(waveform, sr, 16000)  # model expects 16 kHz
waveform = waveform.mean(0, keepdim=True)                       # mono, (1, num_samples)

orig_len = waveform.shape[-1]
waveform = pad_for_model(waveform)               # frame count must divide the patch size
out = model(waveform.unsqueeze(0).to("cuda"))

reconstructed = out["waveform"][..., :orig_len]  # drop padding -> (1, 1, orig_len)

torchaudio.save("output.wav", reconstructed.squeeze(0).float().cpu(), 16000)
```

Cast to bfloat16 with `.to(...)` as shown. Passing `torch_dtype=torch.bfloat16`
to `from_pretrained` is **not** equivalent: it casts parameters but leaves the
mel filterbank buffers in float32, which does not reproduce the reported scores.

`forward` returns a dict with four keys: `waveform`, the reconstructed 16 kHz
audio, and the latents `z_fused`, `z_sem`, and `z_acou` (the fused and
per-stream continuous representations, usable as downstream features).

### Input requirements

- **Mono, 16 kHz**, shape `(batch, 1, num_samples)`. Both mel spectrograms are
  computed inside the model, so no external feature extraction is needed.
- **Run in bfloat16.** The weights were trained under bfloat16 mixed precision;
  running the model in float32 measurably lowers reconstruction quality.
- **Pad the input** so the mel frame count is divisible by the encoder patch
  sizes (16 for the released model, the LCM of the semantic and acoustic
  temporal patch sizes), then trim the output back to the original length. The
  `pad_for_model` helper in the example above does exactly this.

## Evaluation

### Semantic understanding

Semantic scores come from frozen features with an MLP probe. Columns are accuracy unless the header
names another metric, and higher is better throughout. In each column **bold** marks the best score
and <ins>underline</ins> the second best.

**Speech**

| Model | ASV2015 | CREMA-D | Fluent Speech Commands | LibriCount | LibriSpeech-100h (iWER) | LibriSpeech-MF | RAVDESS | Speech Commands V1 | VocalSound | VoxCeleb1 | VoxLingua33 |
|---|---|---|---|---|---|---|---|---|---|---|---|
| DAC | 0.935 | 0.439 | 0.026 | 0.439 | 0.000 | 0.916 | 0.365 | 0.155 | 0.422 | 0.132 | 0.080 |
| WavLM | 0.954 | 0.452 | 0.961 | 0.504 | 0.671 | 0.760 | 0.326 | 0.898 | 0.712 | 0.045 | 0.409 |
| HuBERT | <ins>0.962</ins> | 0.581 | **0.987** | 0.497 | <ins>0.825</ins> | 0.859 | 0.477 | 0.957 | 0.817 | 0.034 | 0.451 |
| SemantiCodec | 0.937 | 0.603 | 0.464 | 0.661 | 0.000 | **0.981** | 0.550 | 0.880 | 0.843 | 0.610 | 0.267 |
| Whisper Large v3 | **0.979** | <ins>0.713</ins> | <ins>0.978</ins> | 0.644 | **0.900** | 0.949 | <ins>0.685</ins> | **0.978** | **0.915** | 0.248 | **0.974** |
| MSM-MAE | 0.932 | 0.686 | 0.005 | **0.712** | 0.000 | <ins>0.976</ins> | 0.664 | 0.928 | 0.876 | <ins>0.716</ins> | 0.666 |
| **Motif-Audio** | 0.927 | **0.744** | 0.800 | <ins>0.703</ins> | 0.000 | 0.956 | **0.726** | <ins>0.958</ins> | <ins>0.910</ins> | **0.754** | <ins>0.679</ins> |

**Environment**

| Model | Clotho (R@1) | DESED (F1) | ESC-50 | FSD18-Kaggle (mAP) | FSD50k (mAP) | UrbanSound 8k |
|---|---|---|---|---|---|---|
| DAC | 0.007 | 0.202 | 0.312 | 0.164 | 0.078 | 0.503 |
| WavLM | 0.008 | 0.168 | 0.315 | 0.196 | 0.101 | 0.531 |
| HuBERT | 0.017 | 0.036 | 0.374 | 0.265 | 0.174 | 0.574 |
| SemantiCodec | 0.019 | <ins>0.486</ins> | 0.817 | 0.267 | 0.348 | <ins>0.845</ins> |
| Whisper Large v3 | 0.031 | 0.226 | 0.625 | <ins>0.496</ins> | 0.320 | 0.757 |
| MSM-MAE | <ins>0.039</ins> | 0.000 | <ins>0.887</ins> | 0.486 | <ins>0.442</ins> | 0.844 |
| **Motif-Audio** | **0.066** | **0.616** | **0.911** | **0.759** | **0.478** | **0.871** |

**Music**

| Model | FMA | GTZAN Genre | MAESTRO (F1) | NSynth |
|---|---|---|---|---|
| DAC | 0.354 | 0.541 | <ins>0.128</ins> | 0.386 |
| WavLM | 0.361 | 0.481 | 0.000 | 0.401 |
| HuBERT | 0.428 | 0.519 | 0.000 | 0.482 |
| SemantiCodec | 0.579 | 0.667 | 0.086 | 0.681 |
| Whisper Large v3 | <ins>0.589</ins> | 0.718 | 0.000 | 0.635 |
| MSM-MAE | **0.627** | <ins>0.844</ins> | 0.003 | **0.730** |
| **Motif-Audio** | 0.582 | **0.887** | **0.200** | <ins>0.699</ins> |

### Reconstruction

Reconstruction quality across datasets. For each metric, the arrow marks the better direction.

| Model | Dataset | Language | PESQ ↑ | STOI ↑ | WER (%) ↓ | CER (%) ↓ | FAD ↓ |
|---|---|---|---|---|---|---|---|
| X-Codec2 | LibriSpeech test-clean | en | 2.433 | 0.918 | 2.36 | – | 0.3536 |
| EnCodec | LibriSpeech test-clean | en | 2.768 | 0.938 | 2.05 | – | 1.3097 |
| Mimi | LibriSpeech test-clean | en | 3.439 | 0.960 | 1.97 | – | 0.6108 |
| DAC | LibriSpeech test-clean | en | 4.010 | 0.974 | 1.94 | – | 0.2099 |
| **Motif-Audio** | LibriSpeech test-clean | en | 3.768 | 0.980 | 1.97 | – | 0.4506 |
| **Motif-Audio** | FLEURS | ko | 3.731 | 0.967 | – | 5.13 | 0.1448 |

Metrics:

- **PESQ** (Perceptual Evaluation of Speech Quality, ↑): perceptual quality of the reconstructed speech.
- **STOI** (Short-Time Objective Intelligibility, ↑): speech intelligibility.
- **WER / CER** (Word / Character Error Rate, ↓): ASR transcription error on the reconstructed audio; lower means content is better preserved. English WER is scored with Wav2Vec2 (`facebook/wav2vec2-large-960h-lv60-self`), Korean CER with Whisper-large-v3 (`openai/whisper-large-v3`).
- **FAD** (Fréchet Audio Distance, ↓): distance between reconstructed and reference audio, computed with the `frechet-audio-distance` library using VGGish embeddings.

### Qualitative comparison

<p align="center">
  <img src="assets/reconstruction_mel_comparison.png" alt="Log-mel spectrograms of an utterance reconstructed by X-Codec2, EnCodec, Mimi, DAC and Motif-Audio alongside the ground truth. A voiced region with strong harmonics is boxed and enlarged, and a third row shows the absolute difference from the reference" width="900">
</p>

The top row gives the full log-mel spectrogram of an utterance reconstructed by each system. The
dashed box marks a voiced segment in which the harmonics are strong and well separated, and that
region is enlarged in the middle row. The bottom row shows the absolute difference from the
reference over the same region, so a darker panel indicates a closer reconstruction.

In the enlarged region, EnCodec flattens the harmonic peaks, reducing their depth relative to the
surrounding valleys to roughly 60% of the reference, while the other systems preserve it.
Motif-Audio's error panel is the darkest, showing the smallest deviation from the reference over
that region.

## Uses

### Direct use

- Audio reconstruction and neural vocoding for general audio and speech at 16 kHz.
- Using the latent as an input representation for a downstream audio model.

### Out-of-scope use

- Text-to-speech or other text-conditioned synthesis. The model reconstructs existing audio and
  accepts no text conditioning.
- Use as a discrete-token codec, or as a source of audio tokens for token-based
  language models. Its latent is continuous, not a sequence of quantized codebook
  indices, so it does not provide the discrete tokens those systems expect.

## Bias, Risks, and Limitations

- **Bandwidth ceiling.** The model runs at 16 kHz, so it cannot represent content above the 8 kHz
  Nyquist limit. Uses that need full-band fidelity are out of scope.
- **Lossy reconstruction.** The output is a reconstruction, not a bit-exact copy. Subtle perceptual
  artifacts can remain even on in-distribution audio.
- **Out-of-distribution inputs.** Heavily degraded, very noisy, or non-audio inputs can produce
  audible artifacts.
- **Potential for misuse.** Because it reconstructs audio faithfully, it can serve as a component in
  voice-cloning or spoofing pipelines. Use it only on audio you have the right to process, and follow
  applicable law.

## License

Released under the [MIT License](https://opensource.org/license/mit). You can use, modify, and
redistribute the model and its weights, including commercially, as long as you keep the copyright
and license notice. The model is provided as is, with no warranty.

## Citation

```bibtex
@misc{motif_audio,
  title  = {Motif-Audio: A General-Purpose Audio Foundation Model},
  author = {Motif Technologies},
  year   = {2026}
}
```

## Contact

For questions, use the [Motif Technologies organization page on Hugging Face](https://huggingface.co/Motif-Technologies).