|
|
--- |
|
|
base_model: |
|
|
- Qwen/Qwen2.5-VL-7B-Instruct |
|
|
language: |
|
|
- en |
|
|
license: apache-2.0 |
|
|
pipeline_tag: video-text-to-text |
|
|
tags: |
|
|
- multimodal |
|
|
library_name: transformers |
|
|
--- |
|
|
|
|
|
# TimeSearch-R-7B |
|
|
- **Code:** https://github.com/Time-Search/TimeSearch-R |
|
|
- **Paper:** [TimeSearch-R: Adaptive Temporal Search for Long-Form Video Understanding via Self-Verification Reinforcement Learning](https://arxiv.org/abs/2511.05489) |
|
|
|
|
|
## Usage |
|
|
|
|
|
We provide the simple generation process for using our model. For more details, you could refer to [Github](https://github.com/Time-Search/TimeSearch-R). |
|
|
|
|
|
```python |
|
|
import numpy as np |
|
|
import torch |
|
|
from longvu.builder import load_pretrained_model |
|
|
from longvu.constants import ( |
|
|
DEFAULT_IMAGE_TOKEN, |
|
|
IMAGE_TOKEN_INDEX, |
|
|
) |
|
|
from longvu.conversation import conv_templates, SeparatorStyle |
|
|
from longvu.mm_datautils import ( |
|
|
KeywordsStoppingCriteria, |
|
|
process_images, |
|
|
tokenizer_image_token, |
|
|
) |
|
|
from decord import cpu, VideoReader |
|
|
|
|
|
tokenizer, model, image_processor, context_len = load_pretrained_model( |
|
|
"./checkpoints/longvu_qwen", None, "cambrian_qwen", |
|
|
) |
|
|
|
|
|
model.eval() |
|
|
video_path = "./examples/video1.mp4" |
|
|
qs = "Describe this video in detail" |
|
|
|
|
|
vr = VideoReader(video_path, ctx=cpu(0), num_threads=1) |
|
|
fps = float(vr.get_avg_fps()) |
|
|
frame_indices = np.array([i for i in range(0, len(vr), round(fps),)]) |
|
|
video = [] |
|
|
for frame_index in frame_indices: |
|
|
img = vr[frame_index].asnumpy() |
|
|
video.append(img) |
|
|
video = np.stack(video) |
|
|
image_sizes = [video[0].shape[:2]] |
|
|
video = process_images(video, image_processor, model.config) |
|
|
video = [item.unsqueeze(0) for item in video] |
|
|
|
|
|
qs = DEFAULT_IMAGE_TOKEN + " |
|
|
" + qs |
|
|
conv = conv_templates["qwen"].copy() |
|
|
conv.append_message(conv.roles[0], qs) |
|
|
conv.append_message(conv.roles[1], None) |
|
|
prompt = conv.get_prompt() |
|
|
|
|
|
input_ids = tokenizer_image_token(prompt, tokenizer, IMAGE_TOKEN_INDEX, return_tensors="pt").unsqueeze(0).to(model.device) |
|
|
stop_str = conv.sep if conv.sep_style != SeparatorStyle.TWO else conv.sep2 |
|
|
keywords = [stop_str] |
|
|
stopping_criteria = KeywordsStoppingCriteria(keywords, tokenizer, input_ids) |
|
|
with torch.inference_mode(): |
|
|
output_ids = model.generate( |
|
|
input_ids, |
|
|
images=video, |
|
|
image_sizes=image_sizes, |
|
|
do_sample=False, |
|
|
temperature=0.2, |
|
|
max_new_tokens=128, |
|
|
use_cache=True, |
|
|
stopping_criteria=[stopping_criteria], |
|
|
) |
|
|
pred = tokenizer.batch_decode(output_ids, skip_special_tokens=True)[0].strip() |
|
|
``` |
|
|
|
|
|
## Citation |
|
|
|
|
|
If you find our work helpful, feel free to give us a cite. |
|
|
|
|
|
``` |
|
|
@article{timesearch-r, |
|
|
title={TimeSearch-R: Adaptive Temporal Search for Long-Form Video Understanding via Self-Verification Reinforcement Learning}, |
|
|
author={Pan, Junwen and Zhang, Qizhe and Zhang, Rui and Lu, Ming and Wan, Xin and Zhang, Yuan and Liu, Chang and She, Qi}, |
|
|
journal={arXiv preprint arXiv:2511.05489}, |
|
|
year={2025} |
|
|
} |
|
|
``` |
