Enhance model card for VideoTG-R1

README.md

@@ -1,3 +1,139 @@
----
-license: mit
----
+---
+license: mit
+pipeline_tag: video-text-to-text
+library_name: transformers
+datasets:
+  - yeliudev/VideoMind-Dataset
+tags:
+  - video-temporal-grounding
+  - multimodal-llm
+  - reinforcement-learning
+  - curriculum-learning
+---
+
+<h2 align="center">VideoTG-R1: Boosting Video Temporal Grounding via
+Curriculum Reinforcement Learning on Reflected Boundary Annotations</h2>
+
+<p align="center">
+  <a href="https://huggingface.co/papers/2510.23397" target="_blank"><img src="https://img.shields.io/badge/%F0%9F%93%96%20Paper-Hugging%20Face-ffbd45.svg"></a>
+  <a href="https://github.com/ldong1111/VideoTG-R1" target="_blank"><img src="https://img.shields.io/badge/%F0%9F%92%BB%20Code-GitHub-brightgreen.svg"></a>
+</p>
+
+<p align="center">
+  Lu Dong<sup>1,3</sup>, Haiyu Zhang<sup>2,3</sup>, Han Lin<sup>4,3</sup>, Ziang Yan<sup>5,3</sup>, Xiangyu Zeng<sup>6,3</sup>, Hongjie Zhang<sup>3</sup>, Yifei Huang<sup>3</sup>, Yi Wang<sup>3</sup>, Zhen-Hua Ling<sup>1</sup>, Limin Wang<sup>6,3</sup>, Yali Wang<sup>7,3</sup><sup>&dagger;</sup>
+  <p align="center">
+    <sup>1</sup>University of Science and Technology of China &nbsp;&nbsp;
+    <sup>2</sup>Beihang University &nbsp;&nbsp;
+    <sup>3</sup>Shanghai Artificial Intelligence Laboratory<br>
+    <sup>4</sup>Shanghai Jiao Tong University &nbsp;&nbsp;
+    <sup>5</sup>Zhejiang University &nbsp;&nbsp;
+    <sup>6</sup>Nanjing University &nbsp;&nbsp;
+    <sup>7</sup>Chinese Academy of Sciences
+  </p>
+</p>
+
+<p align="center">&dagger; Corresponding author</p>
+
+![xxxxxxxx](./figs/main.png) 
+
+**VideoTG-R1** is a multi-agent system for data-efficient video temporal grounding. It contains three modules: 1) Boundary Reflection Agent filters the training data by identifying and discarding partially annotated samples; 2) Difficulty Estimation Agent estimates the difficulty of each sample via zero-shot evaluation; and 3) Curriculum RL strategy dynamically masks the videos of hard-to-ground samples according to the training steps, to ease the training difficulty of hard-to-ground samples. VideoTG-R1 achieves state-of-the-art performance on Charades-STA and ActivityNet-Captions. Moreover, with only 10% of the training dataset, our method outperforms models trained on the full dataset under both GRPO and SFT paradigms. 
+
+### Abstract
+Video temporal grounding (VTG) aims to locate precise segments in videos based on language queries, which is a fundamental challenge in video understanding. While recent Multimodal Large Language Models (MLLMs) have shown promise in tackling VTG through reinforcement learning (RL), they overlook the challenges arising from both the quality and difficulty of training samples. (1) Partially annotated samples. Many samples contain relevant segments beyond the annotated interval, introducing ambiguous supervision. (2) Hard-to-ground samples. Samples with poor zero-shot performance produce consistently low and indistinguishable rewards during RL training, exhibiting no clear preference among multiple outputs and thus hindering learning efficiency. To address these challenges, we propose VideoTG-R1, a novel curriculum RL framework with reflected boundary annotations, enabling data-efficient training. Specifically, we propose a Boundary Reflection Agent that utilizes MLLMs to predict query-relevant timestamps outside the annotated intervals, allowing us to identify and filter out partially annotated samples, thereby reducing ambiguity. Furthermore, we introduce a Difficulty Estimation Agent to assess the training difficulty of each sample and design a curriculum RL strategy that dynamically masks the videos of hard-to-ground samples according to the training steps, easing the training difficulty and providing clearer preference. Experiments on the VTG and grounded VideoQA tasks demonstrate the effectiveness of our method. Remarkably, with only 10% of the training samples and 21% of the computational budget, VideoTG-R1 outperforms full-data counterparts under both group relative policy optimization (GRPO) and supervised fine-tuning (SFT). The code is available at this https URL .
+
+![xx](./figs/BRA.png) 
+
+![xx](./figs/curriculum_RL.png) 
+
+## Code Environment Preparation
+
+```bash
+pip install requirements.txt
+```
+
+## Dataset Preparation
+
+All training and evaluation datasets can be download in [VideoMind's huggingface](https://huggingface.co/datasets/yeliudev/VideoMind-Dataset/tree/main).
+
+## Evaluation
+1. video grounding
+
+```bash
+cd Eval
+
+your_ckpt=xxx
+dataset_name=charades # or anet
+
+bash video_grounding.sh ${dataset_name}
+```
+2. grounded qa
+```bash
+cd Eval
+
+your_ckpt=xxx
+dataset_name=rextime # or nextgqa
+
+bash grounded_qa.sh ${dataset_name}
+```
+
+## Fast Training
+1. You should download the **intermediate_results**  in the huggingface first.
+```bash
+cd ./videotg_r1
+
+bash grpo_train.sh
+```
+
+## Full Training
+1. **boundary reflection agent**
+    1. evaluate each dataset from 'qvhighlights,didemo,tacos,queryd,hirest_grounding,hirest_step,cosmo_cap,internvid_vtime'
+    2. you can run the code with single/multiple gpus.
+```bash
+cd ./BRA
+
+dataset_name=cosmo_cap  
+each_fold_size=22000
+fold_index=0
+
+bash bra_test.sh ${dataset_name} ${each_fold_size} ${fold_index} 
+
+```
+2. **difficulty estimation agent**
+   1. evaluate each dataset from 'qvhighlights,didemo,tacos,queryd,hirest_grounding,hirest_step,cosmo_cap,internvid_vtime'
+    2. you can run the code with single/multiple gpus.
+```bash
+
+
+cd ./DEA
+
+dataset_name=cosmo_cap  
+each_fold_size=22000
+fold_index=0
+
+bash bra_test.sh ${dataset_name} ${each_fold_size} ${fold_index} 
+
+```
+
+3. **curriculum grpo**
+
+```bash
+
+cd ./videotg_r1
+
+bash grpo_train.sh
+
+```
+
+## Acknowledgement
+VideoTG-R1 is built with reference to the following projects: [VideoMind](https://github.com/yeliudev/VideoMind) and [VideoChat-R1](https://github.com/OpenGVLab/VideoChat-R1). Thanks for their work!
+
+## Citation
+If you find our work helpful or inspiring, please feel free to cite it.
+```bibtex
+@article{shen2024longvu,
+  author ={Shen, Xiaoqian and Xiong, Yunyang and Zhao, Changsheng and Wu, Lemeng and Chen, Jun and Zhu, Chenchen and Liu, Zechun and Xiao, Fanyi and Varadarajan, Balakrishnan and Bordes, Florian and Liu, Zhuang and Xu, Hu and J. Kim, Hyunwoo and Soran, Bilge and Krishnamoorthi, Raghuraman and Elhoseiny, Mohamed and Chandra, Vikas},
+  title = {LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding},
+  journal = {arXiv preprint arXiv:2410.17434},
+  year = {2024},
+}
+```