| # Video Captioning |
|
|
| Human labeling of videos is expensive and time-consuming. We adopt powerful image captioning models to generate captions for videos. Although GPT-4V achieves a better performance, its 20s/sample speed is too slow for us. With batch inference, we can achieve a speed of 3s/sample with LLaVA, and the quality is comparable. LLaVA is the second best open-source model in [MMMU](https://mmmu-benchmark.github.io/) and accepts any resolution. |
|
|
|  |
|
|
| ## GPT-4V Captioning |
|
|
| Run the following command to generate captions for videos with GPT-4V: |
|
|
| ```bash |
| python -m tools.caption.caption_gpt4 FOLDER_WITH_VIDEOS output.csv --key $OPENAI_API_KEY |
| ``` |
|
|
| The cost is approximately $0.01 per video (3 frames per video). The output is a CSV file with path and caption. |
|
|
| ## LLaVA Captioning |
|
|
| First, install LLaVA according to their [official instructions](https://github.com/haotian-liu/LLaVA?tab=readme-ov-file#install). We use the `liuhaotian/llava-v1.6-34b` model for captioning, which can be download [here](https://huggingface.co/liuhaotian/llava-v1.6-vicuna-7b). Then, run the following command to generate captions for videos with LLaVA: |
|
|
| ```bash |
| CUDA_VISIBLE_DEVICES=0,1 python -m tools.caption.caption_llava samples output.csv |
| ``` |
|
|
| The Yi-34B requires 2 80GB GPUs and 3s/sample. The output is a CSV file with path and caption. |
|
|
