| --- |
| license: cc-by-nc-4.0 |
| language: |
| - en |
| tags: |
| - audio-generation |
| - video-to-audio |
| - text-to-audio |
| - diffusion-transformer |
| - multimodal |
| pipeline_tag: text-to-audio |
| --- |
| |
|
|
| <h1 align="center">Omni2Sound — Unified Video-Text-to-Audio Generation</h1> |
|
|
| <p align="center"> |
| <a href="https://arxiv.org/pdf/2601.02731"><img src="https://img.shields.io/badge/arXiv-2601.02731-red"></a> |
| <a href="https://omni2sound.github.io//"><img src="https://img.shields.io/badge/Project-Page-blue"></a> |
| <a href="https://github.com/omni2sound/Omni2Sound"><img src="https://img.shields.io/badge/GitHub-Code-black"></a> |
| <a href="https://huggingface.co/datasets/Dalision/Omni2Sound_Benchmark"><img src="https://img.shields.io/badge/HF-Benchmark-yellow"></a> |
| </p> |
|
|
| <p align="center"> |
| <b>CVPR 2026 (Highlight)</b> |
| </p> |
|
|
| <p align="center"> |
| <img src="https://swapforward.github.io/Omni2Sound/src/omnisound.png" width="90%"> |
| </p> |
|
|
| ## Model Description |
|
|
| Omni2Sound is a **unified framework** for generating temporally aligned and semantically faithful audio from **video**, **text**, or **both**. A single model handles three tasks: |
|
|
| - **VT2A** (Video + Text → Audio) |
| - **V2A** (Video → Audio) |
| - **T2A** (Text → Audio) |
|
|
| Omni2Sound achieves **state-of-the-art performance across all three tasks** on the VGGSound-Omni benchmark, surpassing both previous unified models (AudioX, MMAudio) and specialized models (ThinkSound, HunyuanVideo-Foley). |
|
|
| ## Architecture |
|
|
| Omni2Sound is built on a standard **Diffusion Transformer (DiT)** backbone with a decoupled two-branch conditioning design: |
|
|
| - **Semantic Branch ("What")**: Fuses text embeddings from Flan-T5 and visual features from CLIP via cross-attention, providing high-level semantic context. For unimodal tasks (V2A or T2A), the absent modality is simply omitted — no padding needed. |
| - **Temporal Branch ("When")**: Uses a Synchformer to extract fine-grained visual-temporal features, injected globally via Adaptive Layer Normalization (AdaLN) for precise audio-visual synchronization. |
|
|
| The model is trained with a **three-stage progressive multi-task training schedule**: |
|
|
| 1. **Stage 1** — Large-scale T2A pretraining on text-audio pairs |
| 2. **Stage 2** — Multi-task interleaved finetuning (joint VT2A + V2A + T2A) on SoundAtlas |
| 3. **Stage 3** — Decoupled robustness finetuning with off-screen synthesis and text dropout augmentations |
|
|
| ## Key Features |
|
|
| - **Unified SOTA**: A single model achieves state-of-the-art on VT2A, V2A, and T2A simultaneously |
| - **Strong temporal control**: Fine-grained audio-visual synchronization via Synchformer temporal features |
| - **Strong semantic control**: Faithful audio generation guided by text and/or visual semantics |
| - **Robustness**: Handles challenging scenarios including off-screen audio synthesis and incomplete text inputs |
| - **Simple design**: Plain DiT backbone — all gains come from high-quality data (SoundAtlas) and training strategy |
|
|
| ## Model Files |
|
|
| ``` |
| omni2sound/ |
| ├── oob_vae_16k_224410.ckpt # Audio VAE |
| ├── synchformer_state_dict.pth # Synchformer temporal encoder |
| └── vt2a-24-v55vt35-oa15-mq-td15/ |
| ├── args.yaml |
| ├── data_config.yaml |
| ├── model_config.json |
| └── checkpoints/model.ckpt # DiT backbone weights |
| ``` |
|
|
| Additionally, download the following dependencies into `weights/`: |
|
|
| | Model | Source | |
| |---|---| |
| | DFN5B-CLIP-ViT-H-14-384 | [apple/DFN5B-CLIP-ViT-H-14-384](https://huggingface.co/apple/DFN5B-CLIP-ViT-H-14-384) | |
| | flan-t5-base | [google/flan-t5-base](https://huggingface.co/google/flan-t5-base) | |
|
|
| ## Quick Start |
|
|
| ```bash |
| git clone https://github.com/omni2sound/Omni2Sound.git |
| cd Omni2Sound |
| |
| pip install torch==2.1.0 torchaudio==2.1.0 torchvision==0.16.0 \ |
| --index-url https://download.pytorch.org/whl/cu121 |
| pip install -r requirements.txt |
| |
| huggingface-cli download Dalision/Omni2Sound --local-dir weights/omni2sound |
| |
| # Run inference |
| bash scripts/infer_online.sh |
| ``` |
|
|
| See the [GitHub repo](https://github.com/omni2sound/Omni2Sound) for full instructions on inference and finetuning. |
|
|
| ## Links |
|
|
| - **Paper**: [arXiv:2601.02731](https://arxiv.org/pdf/2601.02731) |
| - **Project Page**: [omni2sound.github.io/](https://omni2sound.github.io/) |
| - **Code**: [github.com/omni2sound/Omni2Sound](https://github.com/omni2sound/Omni2Sound) |
| - **Benchmark & Dataset**: [Dalision/Omni2Sound_Benchmark](https://huggingface.co/datasets/Dalision/Omni2Sound_Benchmark) |
| - **Evaluation Results**: [Dalision/Omni2Sound_Result](https://huggingface.co/datasets/Dalision/Omni2Sound_Result) |
|
|
| ## Citation |
|
|
| ```bibtex |
| @article{dai2026omni2sound, |
| title = {Omni2Sound: Towards Unified Video-Text-to-Audio Generation}, |
| author = {Dai, Yusheng and Chen, Zehua and Jiang, Yuxuan and Gao, Baolong and |
| Ke, Qiuhong and Cai, Jianfei and Zhu, Jun}, |
| journal = {arXiv preprint arXiv:2601.02731}, |
| year = {2026} |
| } |
| ``` |
|
|
| ## License |
|
|
| Both the code and model weights are released under [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/) (non-commercial use only). |
|
|
