VideoGPT - A Spatiotemporal Video Captioning Model

Vision Encoder Model: timesformer-base-finetuned-k600
Text Decoder Model: gpt2

Dataset used: MSR-VTT

Results:

Epoch 1 finished with average loss: 3.8702

Epoch 2 finished with average loss: 3.2515

Epoch 3 finished with average loss: 2.8516

Example Inference Code:

import av
import numpy as np
import torch
from transformers import AutoImageProcessor, AutoTokenizer, VisionEncoderDecoderModel

device = "cuda" if torch.cuda.is_available() else "cpu"

# load pretrained processor, tokenizer, and model
image_processor = AutoImageProcessor.from_pretrained("notbdq/videogpt")
tokenizer = AutoTokenizer.from_pretrained("notbdq/videogpt")
model = VisionEncoderDecoderModel.from_pretrained("notbdq/videogpt").to(device)

video_path = "/kaggle/input/darthvader1/darthvadersurfing.mp4"
container = av.open(video_path)

# extract evenly spaced frames from video
seg_len = container.streams.video[0].frames
clip_len = model.config.encoder.num_frames
indices = set(np.linspace(0, seg_len, num=clip_len, endpoint=False).astype(np.int64))
frames = []
container.seek(0)
for i, frame in enumerate(container.decode(video=0)):
    if i in indices:
        frames.append(frame.to_ndarray(format="rgb24"))

# generate caption
gen_kwargs = {
    "max_length": 20, 
}

pixel_values = image_processor(frames, return_tensors="pt").pixel_values.to(device)
tokens = model.generate(pixel_values, **gen_kwargs)
caption = tokenizer.batch_decode(tokens, skip_special_tokens=True)[0]
print(caption) # man is surfing in the ocean and doing tricks on a surfboard

Author Information:

🐙 GitHub
🤝 LinkedIn