Add model card for ControlFoley

README.md

@@ -1,7 +1,7 @@
 ---
-license: cc-by-nc-4.0
 language:
 - en
+license: cc-by-nc-4.0
 pipeline_tag: text-to-audio
 tags:
 - text-video-to-audio
@@ -10,232 +10,24 @@ tags:
 - audio-generation
 ---
 
+# ControlFoley: Unified and Controllable Video-to-Audio Generation
 
+[**Paper**](https://huggingface.co/papers/2604.15086) | [**Code**](https://github.com/xiaomi-research/controlfoley) | [**Project Page**](https://yjx-research.github.io/ControlFoley_web_page/) | [**Demo Page**](https://yjx-research.github.io/ControlFoley/)
 
+**ControlFoley** is a unified and controllable multimodal video-to-audio (V2A) generation framework. It enables precise control over generated audio using video, text, and reference audio. Unlike previous methods, ControlFoley is specifically designed to handle complex cross-modal conflicts (e.g., when text descriptions and visual content disagree) and allows for precise timbre control using reference audio while maintaining temporal synchronization with the video.
 
-<!-- ## **ControlFoley** -->
-
-[中文阅读](./README_zh.md)
-
-<div align="center">
-
-# ControlFoley: Unified and Controllable Video-to-Audio Generation with Cross-Modal Conflict Handling
-
-<p align="center">
-  <a href="https://arxiv.org/abs/2604.15086" style="text-decoration:none"><img src="https://img.shields.io/badge/arXiv-2506.21448-b31b1b.svg" alt="arXiv"/></a>
-  &nbsp;
-  <a href="https://github.com/xiaomi-research/controlfoley" style="text-decoration:none"><img src="https://img.shields.io/badge/GitHub.io-Code-blue?logo=Github&style=flat-square" alt="GitHub"/></a>
-  &nbsp;
-  <a href="https://yjx-research.github.io/ControlFoley_web_page/" style="text-decoration:none"><img src="https://img.shields.io/badge/Project Page-Project-blue" alt="Project Page"/></a>
-  &nbsp;
-  <a href="https://yjx-research.github.io/ControlFoley/" style="text-decoration:none"><img src="https://img.shields.io/badge/Demo Page-Demo-blue" alt="Demo Page"/></a>
-  &nbsp;
-  <a href="https://huggingface.co/YJX-Xiaomi/ControlFoley" style="text-decoration:none"><img src="https://img.shields.io/badge/HuggingFace-Models-orange?logo=huggingface" alt="Hugging Face"/></a>
-</p>
-
-</div>
-
-<p align="center">
-If you find this project useful, please consider giving a star ⭐️~
-</p>
-
-
-<div align="center">
-
-<hr style="border: none; border-top: 3px solid #333; margin: 16px 0;">
-
-### 👥 **Authors**
-
-<div>
-    <!-- Row 1: 6 authors -->
-    <div style="margin-bottom: 2px;">
-        Jianxuan Yang<sup>1*†</sup>,&nbsp;
-        Xinyue Guo<sup>1*</sup>,&nbsp;
-        Zhi Cheng<sup>1,2</sup>,&nbsp;
-        Kai Wang<sup>1,2</sup>,&nbsp;
-        Lipan Zhang<sup>1</sup>,&nbsp;
-        Jinjie Hu<sup>1</sup>
-    </div>
-    <!-- Row 2: 7 authors -->
-    <div>
-        Qiang Ji<sup>1</sup>,&nbsp;
-        Yihua Cao<sup>1</sup>,&nbsp;
-        Yihao Meng<sup>1,2</sup>,&nbsp;
-        Zhaoyue Cui<sup>1,2</sup>,&nbsp;
-        Mengmei Liu<sup>1</sup>,&nbsp;
-        Meng Meng<sup>1</sup>,&nbsp;
-        Jian Luan<sup>1</sup>
-    </div>
-</div>
-<!-- Affiliations -->
-<div>
-    <sup>1</sup>MiLM Plus, Xiaomi Inc. &nbsp;&nbsp; <sup>2</sup>Wuhan University
-    <br>
-    *Equal contribution &nbsp;&nbsp; †Corresponding author
-</div>
-</div>
-
-<hr style="border: none; border-top: 3px solid #333; margin: 16px 0;">
-
-## 📰 **News**
-
-- [2026-04] Technical report released on [arXiv](https://arxiv.org/abs/2604.15086).
-- [2026-04] [Project page](https://yjx-research.github.io/ControlFoley_web_page/) is now live.
-- [2026-04] [Inference code](https://github.com/xiaomi-research/controlfoley) and [pretrained models](https://huggingface.co/YJX-Xiaomi/ControlFoley) are released.
-- [2026-04] Online demo is available on [Project Inference Interface](https://yjx-research.github.io/ControlFoley_web_page/#try-gen).
-- [Coming Soon] Skill will be released.
-
-<hr style="border: none; border-top: 3px solid #333; margin: 16px 0;">
-
-## 🔄 **Updates**
-
-- [x] Release technical report on arXiv.
-- [x] Launch project page.
-- [x] Release inference code and pretrained models.
-- [x] Launch online inference demo (available on project page).
-- [ ] Release skill.
-
-<hr style="border: none; border-top: 3px solid #333; margin: 16px 0;">
-
-## 📺 **Intro Video**
-
-https://github.com/user-attachments/assets/d63e9837-a568-4521-9009-58b4105214a9
-
-For more results of our model, visit [Project Page](https://yjx-research.github.io/ControlFoley_web_page/). For comparison with other methods, visit [Demo Page](https://yjx-research.github.io/ControlFoley/).
-
-<hr style="border: none; border-top: 3px solid #333; margin: 16px 0;">
-
-## 🎧 **Overview**
-
-ControlFoley is a unified and controllable multimodal video-to-audio (V2A) generation framework that enables precise control over generated audio using video, text, and reference audio.
-
-Unlike existing methods that rely on a single modality or struggle under conflicting inputs, ControlFoley is designed to handle complex multimodal interactions and maintain strong controllability even when modalities are inconsistent.
-
-<hr style="border: none; border-top: 3px solid #333; margin: 16px 0;">
-
-## 🎨 **Tease Figure**
-
-<div align="center">
-    <img src="assets/tease.png" width="100%">
-    <p style="margin-top: 8px; text-align: center; font-style: italic;">
-        Left: Overview of the ControlFoley framework with three multimodal conditioning modes for controllable video-synchronized audio generation. Right: Performance radar chart of Video-to-Audio models.
-    </p>
-</div>
-
-<hr style="border: none; border-top: 3px solid #333; margin: 16px 0;">
-
-## 🚀 **Capabilities**
-
-ControlFoley supports a wide range of applications:
-
-- 🎬 <strong>Text-Video-to-Audio Generation (TV2A)</strong><br>
-  Video-content-adaptive dubbing and synchronized sound effect generation under text guidance.
-
-- 📝 <strong>Text-Controlled Video-to-Audio (TC-V2A)</strong><br>
-  Audio generation under video–text conflicts, with semantics consistent with text prompts and temporally synchronized with video contents.
-
-- 🎧 <strong>Audio-Controlled Video-to-Audio (AC-V2A)</strong><br>
-  Audio generation conditioned on reference audio, with timbre consistent with the reference audio and temporally synchronized with video contents.
-
-- 📝 <strong>Text-to-Audio Generation (T2A)</strong><br>
-  Generate audio directly from text prompts as an additional capability of the unified framework.
-
-<hr style="border: none; border-top: 3px solid #333; margin: 16px 0;">
-
-## 🧠 **Key Innovations**
-
-<div align="center">
-    <img src="assets/controlfoley.png" width="100%">
-</div>
-
-- <strong>Joint Visual Encoding for Robust Multimodal Control:</strong>
-  Combines CLIP and CAV-MAE-ST representations to capture both vision-language and audio-visual correlations, improving robustness under modality conflict.
-
-- <strong>Timbre-Focused Reference Audio Control:</strong>
-  Extracts global timbre representations while suppressing temporal cues, enabling precise acoustic style control without affecting synchronization.
-
-- <strong>Modality-Robust Training with Unified Alignment:</strong>
-  Introduces all-modality dropout and a unified REPA objective to improve robustness across diverse modality combinations.
-
-- <strong>VGGSound-TVC Benchmark:</strong>
-  A new benchmark for evaluating textual controllability under visual-text semantic conflicts.
-
-<hr style="border: none; border-top: 3px solid #333; margin: 16px 0;">
-
-## 🧪 **VGGSound-TVC Benchmark**
-
-We propose VGGSound-TVC to evaluate text controllability under varying levels of visual-text conflict. In this dataset, textual descriptions of videos are reconstructed in accordance with the rules described below.
-
-- L0 → No conflict, where the textual description is consistent with the video content.
-- L1_subject →  A mild semantic conflict introduced at the subject level, where the action description remains unchanged while the sounding subject is replaced.
-- L1_action → A mild semantic conflict introduced at the action level, where the subject remains unchanged while the action description is modified.
-- L2 → A moderate semantic conflict in which the textual description belongs to a different semantic category while still maintaining a similar temporal structure or acoustic rhythm.
-- L3 → Strong conflict, where the textual description is randomly substituted.
-
-This enables systematic analysis of modality dominance and controllability under increasing inconsistency. Example samples from VGGSound-TVC are as follows.
-<div align="center">
-    <img src="assets/benchmark.png" width="100%">
-</div>
+## Capabilities
 
-<hr style="border: none; border-top: 3px solid #333; margin: 16px 0;">
+ControlFoley supports a wide range of applications through a unified framework:
 
-## 📊 **Performance**
+- 🎬 **Text-Video-to-Audio (TV2A)**: Synchronized sound effect generation based on video content and text guidance.
+- 📝 **Text-Controlled Video-to-Audio (TC-V2A)**: Prioritizes text semantics even when they conflict with the visual content.
+- 🎧 **Audio-Controlled Video-to-Audio (AC-V2A)**: Generates audio where the timbre is derived from a reference audio file, synchronized with the target video.
+- 🔊 **Text-to-Audio (T2A)**: Direct audio generation from text prompts without video input.
 
-ControlFoley achieves strong performance across multiple V2A tasks, demonstrating both high generation quality and robust controllability.
+## Quick Start
 
-🎬 <strong>TV2A</strong>
-
-ControlFoley achieves state-of-the-art performance across multiple benchmarks, including VGGSound-Test, Kling-Audio-Eval, and MovieGen-Audio-Bench.
-
-- Highest CLAP scores (better semantic alignment)
-- Lowest DeSync (better temporal synchronization)
-- Best overall IS (better audio quality)—Up to 27% relative improvement (22.08 vs. 17.36 on VGGSound).
-
-<div align="center">
-    <img src="assets/result1.png" width="80%">
-</div>
-
-📝 <strong>TC-V2A</strong>
-
-ControlFoley demonstrates strong textual controllability under increasing visual-text conflict.
-
-- Maintains high CLAP (text alignment) across conflict levels  
-- Effectively reduces IB under conflict (less reliance on visual bias)  
-- Achieves better balance between controllability and generation quality  
-
-<div align="center">
-    <img src="assets/result2.png" width="60%">
-</div>
-
-🎧 <strong>AC-V2A</strong>
-
-ControlFoley achieves the best performance across all evaluation metrics on the Greatest Hits dataset.
-
-- Better timbre similarity (Resemblyzer)  
-- Better synchronization (DeSync)  
-- Higher audio quality (IS)  
-  
-Notably, it outperforms CondFoleyGen, a specialized in-domain baseline, demonstrating strong generalization ability.
-
-<div align="center">
-    <img src="assets/result3.png" width="50%">
-</div>
-
-##
-ControlFoley also demonstrates competitive or superior performance compared to strong proprietary systems such as Kling-Foley, highlighting its effectiveness as an open and controllable solution.
-
-<hr style="border: none; border-top: 3px solid #333; margin: 16px 0;">
-
-## 🛠 **Quick Start**
-
-### 🔑 **Prerequisites**
-
-- Python 3.10+
-- PyTorch 2.5.1+
-- CUDA 11.8+
-- FFmpeg (conda install -c conda-forge ffmpeg)
-
-### 🧱 **Installation**
+### Installation
 
 ```bash
 # Clone the repository
@@ -254,69 +46,34 @@ pip install huggingface-hub==0.26.2
 huggingface-cli download YJX-Xiaomi/ControlFoley --resume-download --local-dir model_weights --local-dir-use-symlinks False
 ```
 
-Or you can download the weights from [here](https://huggingface.co/YJX-Xiaomi/ControlFoley/tree/main/) and put them in the `model_weights` folder.
-
-### 🎨 **Inference**
-
-```
-python demo.py [OPTIONS]
-
-Options:
-  --video            TEXT       Path to the input video file. (default: None)
-  --audio            TEXT       Path to the input reference audio file. (default: None)
-  --prompt           TEXT       Textual prompt for audio generation. (default: None)
-  --negative_prompt  TEXT       Negative textual prompt for audio generation. (default: None)
-  --duration         FLOAT      Duration of the generated audio in seconds. (default: 8.0)
-  --output           TEXT       Output directory for generated audio files. (default: ./output)
-```
-
-### 📌 **Supported Tasks**
-
-| Task   | video      | audio      | prompt   |
-|--------|------------|------------|----------|
-| TV2A   | required   | None       | required |
-| TC-V2A | required   | None       | required |
-| AC-V2A | required   | required   | optional |
-| V2A    | required   | None       | None     |
-| T2A    | None       | None       | required |
+### Inference
 
-### 📋 **Usage Examples**
-
-- TV2A
+You can run various tasks using the provided `demo.py` script:
 
+**Text-Video-to-Audio (TV2A):**
 ```bash
 python demo.py --video "assets/001.mp4" --prompt "the skateboard wheels scraping and grinding on the ground." --duration 8.0 --output "./output"
 ```
 
-- TC-V2A
-
-```bash
-python demo.py --video "assets/002.mp4" --prompt "man whistling." --duration 8.0 --output "./output"
-```
-
-- AC-V2A
-
+**Audio-Controlled Video-to-Audio (AC-V2A):**
 ```bash
 python demo.py --video "assets/003.mp4" --audio "assets/003.wav" --duration 8.0 --output "./output"
 ```
 
-- V2A
-
-```bash
-python demo.py --video "assets/004.mp4" --duration 8.0 --output "./output"
-```
-
-- T2A
-
+**Text-to-Audio (T2A):**
 ```bash
 python demo.py --prompt "A bird sings melodically in a forest." --duration 8.0 --output "./output"
 ```
 
-<hr style="border: none; border-top: 3px solid #333; margin: 16px 0;">
+## Key Innovations
+
+- **Joint Visual Encoding**: Combines CLIP and CAV-MAE-ST representations to improve robustness under modality conflict.
+- **Temporal-Timbre Decoupling**: Extracts acoustic style from reference audio while suppressing temporal cues to avoid affecting video synchronization.
+- **Modality-Robust Training**: Uses unified representation alignment (REPA) and random modality dropout to handle diverse input combinations.
 
-## 📝 **Citation**
+## Citation
 
-If you find this repository useful, please consider citing our paper:
+If you find this project useful, please consider citing the following paper:
 
 ```bibtex
 @misc{yang2026controlfoleyunifiedcontrollablevideotoaudio,
@@ -330,35 +87,6 @@ If you find this repository useful, please consider citing our paper:
 }
 ```
 
-<hr style="border: none; border-top: 3px solid #333; margin: 16px 0;">
-
-## 🔒 **License**
-
-This repository is licensed under the [Apache License 2.0](./LICENSE) and the [model weights](https://huggingface.co/YJX-Xiaomi/ControlFoley/tree/main/) are licensed under the [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/).
-
-<hr style="border: none; border-top: 3px solid #333; margin: 16px 0;">
-
-## 🙏 **Acknowledgments**
-
-This project uses the following datasets:<br>
-VGGSound, Kling-Audio-Eval, The Greatest Hits (<a href="https://creativecommons.org/licenses/by/4.0/" target="_blank" style="color:#007bff; text-decoration:none;">CC BY 4.0</a>),
-and MovieGen-Audio-Bench (<a href="https://creativecommons.org/licenses/by-nc/4.0/" target="_blank" style="color:#dc3545; text-decoration:none;">CC BY-NC 4.0</a>).<br>
-All resources are used for <strong>academic and non-commercial demonstration purposes only</strong>.
-
-This project is inspired by the following works:<br>
-[stable-audio-tools](https://github.com/Stability-AI/stable-audio-tools), [MMAudio](https://github.com/hkchengrex/MMAudio), [Make-An-Audio 2](https://github.com/bytedance/Make-An-Audio-2), [Synchformer](https://github.com/v-iashin/Synchformer), and [audiocraft](https://github.com/facebookresearch/audiocraft).<br>
-Thanks for their contributions.
-
-<hr style="border: none; border-top: 3px solid #333; margin: 16px 0;">
-
-## 📞 **Contact**
-
-If you have any questions or suggestions, please feel free to contact us at yangjianxuan@xiaomi.com.
-
-<hr style="border: none; border-top: 3px solid #333; margin: 16px 0;">
-
-<div align="center">
-
-2026 ControlFoley Project. All Rights Reserved.
+## License
 
-</div>
+The model weights are licensed under [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/). The code is licensed under the Apache License 2.0.