README.md

---
base_model:
- Qwen/Qwen2.5-Omni-3B
datasets:
- openinterx/UGC-VideoCap
license: mit
metrics:
- bleu
- accuracy
pipeline_tag: video-text-to-text
library_name: transformers
tags:
  - multimodal
  - video-captioning
  - audio-visual
  - ugc
---

# UGC-VideoCaptioner: An Omni UGC Video Detail Caption Model and New Benchmarks

**UGC-VideoCaptioner** is a 3B-parameter captioning model distilled from Gemini-2.5 Flash, specifically designed for detailed, omnimodal captioning of short-form, user-generated videos (UGC). It addresses the crucial role of audio in conjunction with visual content, which is often overlooked in existing video captioning models.

<p align="left">
  <a href="https://huggingface.co/papers/2507.11336" target="_blank">
    <img src="https://img.shields.io/badge/HuggingFace_Paper-2507.11336%F0%9F%93%96-blue">
  </a>
  <a href="http://arxiv.org/abs/2507.11336" target="_blank">
    <img src="https://img.shields.io/badge/arXiv-2507.11336%F0%9F%93%96-bron">
  </a>
  <a href="https://memories.ai/" target="_blank">
    <img src="https://img.shields.io/badge/Project_Page-%F0%9F%8C%90-green">
  </a>
  <a href="https://github.com/openinterx/UGC-VideoCaptioner" target="_blank">
    <img src="https://img.shields.io/badge/GitHub-Code-black?logo=github">
  </a>
</p>

## Abstract

Real-world user-generated videos, especially on platforms like TikTok, often feature rich and intertwined audio visual content. However, existing video captioning benchmarks and models remain predominantly visual centric, overlooking the crucial role of audio in conveying scene dynamics, speaker intent, and narrative context. This lack of omni datasets and lightweight, capable models hampers progress in fine grained, multimodal video understanding. To address these challenges, we introduce UGC-VideoCap, a new benchmark and model framework specifically designed for detailed omnimodal captioning of short form user-generated videos. Unlike prior datasets, UGC-VideoCap emphasizes balanced integration of audio and visual modalities, featuring 1000 TikTok videos annotated through a structured three stage human-in-the-loop pipeline covering audio only, visual only, and joint audio visual semantics. The benchmark also includes 4000 carefully crafted QA pairs probing both unimodal and cross modal understanding. Alongside the dataset, we propose UGC-VideoCaptioner(3B), a 3B parameter captioning model distilled from Gemini 2.5 Flash. Using a novel two-stage training strategy supervised fine tuning followed by Group Relative Policy Optimization (GRPO), our approach enables efficient adaptation from limited data while maintaining competitive performance. Together, our benchmark and model offer a high-quality foundation and a data-efficient solution for advancing omnimodal video captioning in unconstrained real-world UGC settings.

<p align="center">
    <img src="https://github.com/openinterx/UGC-VideoCaptioner/blob/main/tiktok_qa_sample.png?raw=true" alt="UGC-VideoCap">
</p>

## Benchmark Results

<p align="center">
    <img src="https://github.com/openinterx/UGC-VideoCaptioner/blob/main/benchmark.png?raw=true" alt="UGC-VideoCap">
</p>

### Model Zoom

| Model                         | visual | audio | details | average | Link |
|:------------------------------|:------:|:-----:|:-------:|:-------:|:----:|
| Gemini-2.5-pro              |    75.8    |   70.8    |    74.8     |     73.78    | N/A  |
| Gemini-2.5-flash              | **78.8**     |   **74.2**    |    **77.2**    |    **76.73**    | N/A  |
| Qwen2.5-Omni-3B               |   55.6     |  48.2    |   52.6      |   52.18      | N/A  |
| UGC-VideoCaptioner-3B-zero(1k RL)         |    57.8    |  53.0     |    55.4     |    55.40(**+3.22**)    | [google-drive](https://drive.google.com/drive/folders/1R-L4kz4R7UxYpcU4El1ctgvVDQbsMsG6?usp=sharing) |
| Qwen2.5-Omni-3B 1k sft        |    58.4    |   61.4  |   57.0    |     58.96(+6.78)    | [google-drive](https://drive.google.com/drive/folders/1itJ1u4XEJNVfmgbxuKL-fGWCbaW3EAza?usp=sharing) |
| Qwen2.5-Omni-3B 10k sft       |    58.4   |    63.2   |   58.0     |   59.87(+7.69)     | [google-drive](https://drive.google.com/drive/folders/1auQ4mx9CcxIzAIF4SyH034xufzrqe29w?usp=sharing) |
| Qwen2.5-Omni-3B 20k sft       |    59.2   |   64    |    58.4   |     60.50(+8.32)     | [google-drive](https://drive.google.com/drive/folders/11WJZkq8I_807zJUmBCCvwNjSj18F2im9?usp=sharing) |
| UGC-VideoCaptioner-3B (1k SFT + 1k RL)         |   59.4     |    62.4   |    58.2     |    60.01(**+7.83**)   | [google-drive](https://drive.google.com/drive/folders/1LGmIU60cdacErNgUk86D8I5_kiU_ljFz?usp=sharing) |

## Quick Start

You can use this model with the `transformers` library. Below is a quick example demonstrating how to perform inference.
Please note that for full video processing capabilities, you might need to install `decord` and refer to the [official GitHub repository](https://github.com/openinterx/UGC-VideoCaptioner) for detailed video handling steps, especially if `AutoProcessor` doesn't directly handle video file paths for complex scenarios.

### Environment Setup

```bash
pip install transformers torch decord soundfile qwen_omni_utils
```

### Inference

```python
from transformers import AutoProcessor, AutoModelForConditionalGeneration
import torch

# Load model and processor
model = AutoModelForConditionalGeneration.from_pretrained(
    "openinterx/UGC-VideoCaptioner-3B",
    torch_dtype="auto",
    device_map="auto",
    trust_remote_code=True, # Required for custom architectures/processors
)
processor = AutoProcessor.from_pretrained(
    "openinterx/UGC-VideoCaptioner-3B",
    trust_remote_code=True, # Required for custom architectures/processors
)

# Example video path (replace with your actual video file path)
video_path = "path/to/your/video.mp4" 

# Define the detailed captioning prompt
prompt_text = (
    "You are given a short video with both audio and visual content. Write a detailed and coherent paragraph "
    "that naturally integrates all modalities. Your description should include: (1) the primary scene and "
    "background setting; (2) key characters or objects and their actions or interactions; (3) significant "
    "audio cues such as voices, background music, sound effects, and their emotional tone; (4) any on-screen "
    "text (OCR) and its role in the video context; and (5) the overall theme or purpose of the video. "
    "Ensure the output is a fluent and objective paragraph, not a bullet-point list, and captures the video's "
    "content in a human-like, narrative style."
)

# Prepare messages in the chat template format
messages = [
    {
        "role": "user",
        "content": [
            {"type": "video", "video": video_path}, # Pass video path
            {"type": "text", "text": prompt_text},
        ],
    }
]

# Apply chat template and process inputs
text = processor.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)

inputs = processor(
    text=[text],
    videos=[video_path], # Pass video path directly to the processor
    return_tensors="pt",
).to(model.device)

# Generate output
generated_ids = model.generate(**inputs, max_new_tokens=512)

# Decode generated text
output_text = processor.batch_decode(
    generated_ids[:, inputs["input_ids"].shape[1]:], # Exclude prompt from output
    skip_special_tokens=True,
    clean_up_tokenization_spaces=False
)[0]

print(output_text)
```

## Evaluation

### Final Caption prompt (for inference)

```python
prompt = "You are given a short video with both audio and visual content. Write a detailed and coherent paragraph that naturally integrates all modalities. "
"Your description should include: (1) the primary scene and background setting; (2) key characters or objects and their actions or interactions; "
"(3) significant audio cues such as voices, background music, sound effects, and their emotional tone; "
"(4) any on-screen text (OCR) and its role in the video context; and (5) the overall theme or purpose of the video. "
"Ensure the output is a fluent and objective paragraph, not a bullet-point list, and captures the video's content in a human-like, narrative style.
```

### Score

Scores are judged by GPT-4o-2024-08-06.

```bash
python eval_caption.py
```

## Citation

If you find this repository helpful, feel free to cite our paper:

```bibtex
@article{wu2024ugc,
  title={UGC-VideoCaptioner: An Omni UGC Video Detail Caption Model and New Benchmarks},
  author={Wu, Zhenyu and Sun, Qiushi and Zhang, Jiabo and Zhu, Yuyin and Ma, Guojun and Cheng, Kanzhi and Jia, Chengyou and Tan, Jian and Yang, Qing and Wu, Zhiyong},
  journal={arXiv preprint arXiv:2507.11336},
  year={2024}
}
```