File size: 2,969 Bytes
25e57c6 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 | # Contextualized Spatial-Temporal Contrastive Learning with Self-Supervision
[](https://arxiv.org/abs/2112.05181)
This repository contains the official implementation of
[Contextualized Spatio-Temporal Contrastive Learning with Self-Supervision](https://arxiv.org/abs/2112.05181)
in TF2.
<p align="left">
<img src="https://storage.googleapis.com/tf_model_garden/vision/const_cl/CHR_FIG2.png" height=350>
</p>
## Description
Most of existing video-language pre-training methods focus on instance-level
alignment between video clips and captions via global contrastive learning but
neglect rich fine-grained local information, which is of importance to
downstream tasks requiring temporal localization and semantic reasoning. In this
work, we propose a simple yet effective video-language pre-training framework,
namely G-ViLM, to learn discriminative spatiotemporal features. Two novel
designs involving spatiotemporal grounding and temporal grouping promote
learning local region-noun alignment and temporal-aware features simultaneously.
Specifically, spatiotemporal grounding aggregates semantically similar video
tokens and aligns them with noun phrases extracted from the caption to promote
local region-noun correspondences. Moreover, temporal grouping leverages
cut-and-paste to manually create temporal scene changes and then learns
distinguishable features from different scenes. Comprehensive evaluations
demonstrate that G-ViLM performs favorably against existing approaches on four
representative downstream tasks, covering text-video retrieval, video question
answering, video action recognition and temporal action localization. G-ViLM
performs competitively on all evaluated tasks and in particular achieves R@10 of
65.1 on zero-shot MSR-VTT retrieval, over 9% higher than the state-of-the-art
method.
## Pre-trained Model Performance
All models are pre-trained from scratch with `region_generator = RANDOM` and `context_length = 5` as described in the paper.
We report the mean average
precision on AVA v2.2 and AVA-Kinetics validation set and precision/success rate
on Object Tracking Benchmark 2015.
| Method | Parameters | Dataset | Pretrain Steps | AVA(mAP) | AVAK(mAP) | OTB(P/S) |
| :--------------: | :----: | :--: | :--: |:----: |:-----------: | :----------: |
| CVRL | 31.7M | Kinetics-400 | 200k | 18.4% | 24.1% | 75.4/53.7 |
| ConST-CL | 31.7M | Kinetics-400 | 100k | 22.1% | 28.0% | 77.4/54.3 |
| ConST-CL | 31.7M | Kinetics-400 | 200k | 24.1% | 30.5% | 78.1/55.2 |
## Citation
```
@inproceedings{yuan2022constcl,
title={Contexualized Spatio-Temporal Contrastive Learning with Self-Supervision},
author={Yuan, Liangzhe and Qian, Rui and Cui, Yin and Gong, Boqing and Schroff, Florian and Yang, Ming-Hsuan and Adam, Hartwig and Liu, Ting},
journal={CVPR},
year={2022}
}
```
|