---
license: other
base_model: facebook/sam-audio-large
tags:
- onnx
- audio
- sam-audio
- source-separation
- audio-visual
---

# SAM-Audio ONNX (Large)

ONNX-converted models for [SAM-Audio](https://github.com/facebookresearch/sam-audio) (facebook/sam-audio-large) - Meta's Semantic Audio Modeling for audio source separation.

This repository contains both **FP32** and **FP16** versions of the models.

## Model Variants

| Variant | DiT Size | Total Size | Notes |
|---------|----------|------------|-------|
| `fp32/` | 11.76 GB | ~13.9 GB | Full precision |
| `fp16/` | 5.88 GB | ~8.0 GB | Half precision (recommended) |

## Model Files (per variant)

| File | Description | FP32 Size | FP16 Size |
|------|-------------|-----------|-----------|
| `dacvae_encoder.onnx` | Audio encoder (48kHz → latent) | 110 MB | 110 MB |
| `dacvae_decoder.onnx` | Audio decoder (latent → 48kHz) | 320 MB | 320 MB |
| `t5_encoder.onnx` | Text encoder (T5-base) | 440 MB | 440 MB |
| `dit_single_step.onnx` | DiT denoiser (3B params) | 11.76 GB | 5.88 GB |
| `vision_encoder.onnx` | Vision encoder (CLIP-based) | 1.27 GB | 1.27 GB |
| `tokenizer/` | SentencePiece tokenizer files | - | - |

## Installation

```bash
pip install onnxruntime sentencepiece torchaudio torchvision torchcodec soundfile
# For CUDA support (recommended for large model):
pip install onnxruntime-gpu
```

## Usage

### Using FP16 Models (Recommended)
```bash
python onnx_inference.py \
    --video input.mp4 \
    --text "a person speaking" \
    --model-dir fp16 \
    --output target.wav \
    --output-residual residual.wav
```

### Using FP32 Models
```bash
python onnx_inference.py \
    --video input.mp4 \
    --text "keyboard typing" \
    --model-dir fp32 \
    --output target.wav
```

### Audio-Only Mode
```bash
python onnx_inference.py \
    --audio input.wav \
    --text "drums" \
    --model-dir fp16 \
    --output drums.wav
```

## Model Specifications

- **Audio Sample Rate**: 48kHz
- **Audio Hop Length**: 1536 samples
- **Vision Input Size**: 336×336 pixels
- **Text Encoder**: T5-base (768-dim)
- **Vision Encoder**: PE-Core-L14-336 (1024-dim)
- **DiT Parameters**: ~3 billion
- **ODE Solver**: Midpoint method (default 16 steps)

## Exporting Models

### Export FP16 DiT (Recommended)
```bash
python -m onnx_export.export_dit \
    --output-dir ./my_models \
    --model-id facebook/sam-audio-large \
    --fp16 \
    --device cuda
```

### Export Other Components
```bash
python -m onnx_export.export_dacvae --output-dir ./my_models --model-id facebook/sam-audio-large
python -m onnx_export.export_t5 --output-dir ./my_models --model-id facebook/sam-audio-large
python -m onnx_export.export_vision --model facebook/sam-audio-large --output ./my_models
```

## License

SAM-Audio is released under the [CC-BY-NC 4.0 license](https://creativecommons.org/licenses/by-nc/4.0/). See [original repository](https://huggingface.co/facebook/sam-audio-large) for full terms.

## Acknowledgments

Original model by [Meta AI Research](https://github.com/facebookresearch/sam-audio).