---
library_name: pytorch
license: mit
pipeline_tag: text-to-audio
---

# Foley-Omni


[GitHub Code](https://github.com/NJU-Speech/Foley-Omni) | [arXiv](https://arxiv.org/abs/2606.03672) | [Demo](https://ty0402.github.io/Foley-omni-Web/)

## Overview

This repository packages the public inference checkpoint set for **Foley-Omni**.
The release focuses on **Video-to-Soundtrack (V2ST)** generation, where the model jointly generates synchronized **speech**, **sound effects**, and **music** from a video and optional text prompt.

## Model Size
5.5B

## Repository Contents

```text
ckpts/
├── Foley-Omni/
│   └── v2st.pth
├── Wan2.2-TI2V-5B/
│   ├── models_t5_umt5-xxl-enc-bf16.pth
│   └── google/
│       └── umt5-xxl/
│           ├── special_tokens_map.json
│           ├── spiece.model
│           ├── tokenizer.json
│           └── tokenizer_config.json
└── mmaudio/
    └── ext_weights/
        ├── v1-16.pth
        ├── best_netG.pt
        └── synchformer_state_dict.pth
```

What each part is used for:

- `ckpts/Foley-Omni/v2st.pth`: released inference-only Foley-Omni weights
- `ckpts/Wan2.2-TI2V-5B/*`: text encoder and tokenizer for text conditioning
- `ckpts/mmaudio/ext_weights/v1-16.pth`: audio VAE for the 16 kHz inference path
- `ckpts/mmaudio/ext_weights/best_netG.pt`: vocoder for waveform decoding
- `ckpts/mmaudio/ext_weights/synchformer_state_dict.pth`: online visual feature extraction

## Online Feature Extraction

This release supports both:

- direct V2ST inference with pre-extracted `clip_feature_path` and `sync_feature_path`
- V2ST inference without pre-extracted features, using online visual feature extraction

Notes:

- `synchformer_state_dict.pth` is included in this repository because it is required for online Sync feature extraction.
- The CLIP image encoder is loaded by `open_clip` from `apple/DFN5B-CLIP-ViT-H-14-384` on first use. The current code path does not use a separate local CLIP checkpoint file.

## Source Attribution

This repository redistributes a small subset of files from the following upstream releases for convenience:

- **Wan2.2-TI2V-5B**: text encoder and tokenizer files
- **MMAudio**: audio VAE, vocoder, and Synchformer files

Please refer to the original upstream repositories for their licenses, usage terms, and project details.