Improve model card for Hear-Your-Click

README.md

@@ -1,3 +1,63 @@
 ---
 license: cc0-1.0
+pipeline_tag: feature-extraction
 ---
+
+# Hear-Your-Click: Interactive Object-Specific Video-to-Audio Generation
+
+![Hear-Your-Click Demo](https://github.com/user-attachments/assets/2ca49ab5-80ca-42c4-b9a5-9dc7959ac358)
+
+Hear-Your-Click is an interactive framework enabling users to generate sounds for specific objects in a video by simply clicking on the frame. This novel approach overcomes limitations of traditional video-to-audio methods that rely on global video information, often struggling with complex scenes and tailored audio generation.
+
+## Paper
+This model was presented in the paper [Hear-Your-Click: Interactive Object-Specific Video-to-Audio Generation](https://huggingface.co/papers/2507.04959).
+
+## Code
+The official code repository for Hear-Your-Click can be found on GitHub: [SynapGrid/Hear-Your-Click-2024](https://github.com/SynapGrid/Hear-Your-Click-2024)
+
+## Abstract
+Video-to-audio (V2A) generation shows great potential in fields such as film production. Despite significant advances, current V2A methods relying on global video information struggle with complex scenes and generating audio tailored to specific objects. To address these limitations, we introduce Hear-Your-Click, an interactive V2A framework enabling users to generate sounds for specific objects by clicking on the frame. To achieve this, we propose Object-aware Contrastive Audio-Visual Fine-tuning (OCAV) with a Mask-guided Visual Encoder (MVE) to obtain object-level visual features aligned with audio. Furthermore, we tailor two data augmentation strategies, Random Video Stitching (RVS) and Mask-guided Loudness Modulation (MLM), to enhance the model's sensitivity to segmented objects. To measure audio-visual correspondence, we designed a new evaluation metric, the CAV score. Extensive experiments demonstrate that our framework offers more precise control and improves generation performance across various metrics.
+
+## Usage
+To get started with Hear-Your-Click, follow the steps below to set up the environment, download checkpoints, and run inference.
+
+### Installation
+1.  Clone the repository:
+    ```bash
+    git clone https://github.com/SynapGrid/Hear-Your-Click-2024.git
+    cd Hear-Your-Click-2024
+    ```
+
+2.  (Optional) Create a Conda environment:
+    ```bash
+    conda env create -n hyc python=3.9.11
+    conda activate hyc
+    ```
+
+3.  Install dependencies:
+    ```bash
+    pip install -r requirements.txt
+    ```
+
+### Model Checkpoints
+Download the necessary model weights and place them in the specified directories as instructed in the [official GitHub repository's `Model Checkpoints` section](https://github.com/SynapGrid/Hear-Your-Click-2024#model-checkpoints).
+
+### Inference Command
+Launch the inference demo:
+```bash
+python app.py --device cuda:0,1 --sam_model_type vit_b
+```
+
+## Citation
+If you find this work useful for your research or applications, please cite our work:
+```bibtex
+@misc{liang2025hearyourclickinteractivevideotoaudiogeneration,
+      title={Hear-Your-Click: Interactive Video-to-Audio Generation via Object-aware Contrastive Audio-Visual Fine-tuning},
+      author={Yingshan Liang and Keyu Fan and Zhicheng Du and Yiran Wang and Qingyang Shi and Xinyu Zhang and Jiasheng Lu and Peiwu Qin},
+      year={2025},
+      eprint={2507.04959},
+      archivePrefix={arXiv},
+      primaryClass={cs.CV},
+      url={https://arxiv.org/abs/2507.04959},
+}
+```