File size: 1,937 Bytes
26064ba 53717b1 e6381f0 53717b1 4995ead 699a8cb 53717b1 699a8cb 53717b1 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 |
---
license: apache-2.0
---
# VideoGPT - A Spatiotemporal Video Captioning Model
Vision Encoder Model: [timesformer-base-finetuned-k600](https://huggingface.co/facebook/timesformer-base-finetuned-k600) \
Text Decoder Model: [gpt2](https://huggingface.co/gpt2)
Dataset used: [MSR-VTT](https://paperswithcode.com/dataset/msr-vtt)
#### Results:
Epoch 1 finished with average loss: 3.8702
Epoch 2 finished with average loss: 3.2515
Epoch 3 finished with average loss: 2.8516
#### Example Inference Code:
```python
import av
import numpy as np
import torch
from transformers import AutoImageProcessor, AutoTokenizer, VisionEncoderDecoderModel
device = "cuda" if torch.cuda.is_available() else "cpu"
# load pretrained processor, tokenizer, and model
image_processor = AutoImageProcessor.from_pretrained("notbdq/videogpt")
tokenizer = AutoTokenizer.from_pretrained("notbdq/videogpt")
model = VisionEncoderDecoderModel.from_pretrained("notbdq/videogpt").to(device)
video_path = "/kaggle/input/darthvader1/darthvadersurfing.mp4"
container = av.open(video_path)
# extract evenly spaced frames from video
seg_len = container.streams.video[0].frames
clip_len = model.config.encoder.num_frames
indices = set(np.linspace(0, seg_len, num=clip_len, endpoint=False).astype(np.int64))
frames = []
container.seek(0)
for i, frame in enumerate(container.decode(video=0)):
if i in indices:
frames.append(frame.to_ndarray(format="rgb24"))
# generate caption
gen_kwargs = {
"max_length": 20,
}
pixel_values = image_processor(frames, return_tensors="pt").pixel_values.to(device)
tokens = model.generate(pixel_values, **gen_kwargs)
caption = tokenizer.batch_decode(tokens, skip_special_tokens=True)[0]
print(caption) # man is surfing in the ocean and doing tricks on a surfboard
```
#### Author Information:
🐙 [GitHub](https://github.com/notlober) \
🤝 [LinkedIn](https://www.linkedin.com/in/selahattin-baki-damar-6bb38128a/)
|