| --- |
| license: mit |
| datasets: |
| - mjjung/ActivityNet-VTune |
| language: |
| - en |
| --- |
| |
| # TimeChat-7B-ActivityNet-VTune Model |
|
|
| ## Model details |
|
|
| We trained [VideoLLaMA](https://arxiv.org/abs/2306.02858) using VTune, a developed instruction-tuning method specifically designed to account for consistency. |
|
|
| For the tuning, we utilized 10K training videos from ActivityNet-Captions with 205K automatically generated annotations. |
|
|
| ## Evaluation |
| We evaluated the model on ActivtyNet-CON and ActivtyNet-Captions. |
|
|
| - ActivityNet-CON |
| | Metric | Value | |
| |-----------------|-------------| |
| | Ground | 33.0 | |
| | R-Ground | 24.7 (74.8) | |
| | S-Ground | 10.0 (30.2) | |
| | H-Verify | 20.2 (61.1) | |
| | C-Verify | 17.7 (53.7) | |
|
|
| - ActivityNet-Captions |
| | Metric | Value | |
| |-----------------|---------| |
| | R@1 IoU=0.3 | 51.58 | |
| | R@1 IoU=0.5 | 34.38 | |
| | R@1 IoU=0.7 | 19.18 | |
| | mIoU | 36.16 | |
|
|
| **Paper and Code for more information:** |
| [Paper](https://arxiv.org/abs/2411.12951), [Code](https://github.com/minjoong507/consistency-of-video-llm) |
|
|
| ## Citation |
| If you find our research and codes useful, please consider starring our repository and citing our paper: |
|
|
| ``` |
| @article{jung2024consistency, |
| title={On the Consistency of Video Large Language Models in Temporal Comprehension}, |
| author={Jung, Minjoon and Xiao, Junbin and Zhang, Byoung-Tak and Yao, Angela}, |
| journal={arXiv preprint arXiv:2411.12951}, |
| year={2024} |
| } |
| ``` |
