Upload 52 files

README.md

@@ -1,3 +1,221 @@
----
-license: cc
----
+# Fine-tuned CLIP models are efficient video learners [CVPR 2023]
+
+
+
+> [**Fine-tuned CLIP models are efficient video learners**](https://arxiv.org/abs/2212.03640)<br>
+> [Hanoona Rasheed*](https://scholar.google.com/citations?user=yhDdEuEAAAAJ&hl=en&authuser=1&oi=sra), [Muhammad Uzair Khattak*](https://scholar.google.com/citations?user=M6fFL4gAAAAJ&hl=en&authuser=1), [Muhammad Maaz](https://scholar.google.com/citations?user=vTy9Te8AAAAJ&hl=en&authuser=1&oi=sra), [Salman Khan](https://salman-h-khan.github.io/), [Fahad Shahbaz Khan](https://scholar.google.es/citations?user=zvaeYnUAAAAJ&hl=en)
+
+*Equally contributing first authors
+
+[![Website](https://img.shields.io/badge/Project-Website-87CEEB)](https://muzairkhattak.github.io/ViFi-CLIP/)
+[![paper](https://img.shields.io/badge/arXiv-Paper-<COLOR>.svg)](https://arxiv.org/abs/2212.03640)
+[![video](https://img.shields.io/badge/Video-Presentation-F9D371)](https://www.youtube.com/watch?v=uqPLPIyWBb0)
+[![slides](https://img.shields.io/badge/Presentation-Slides-B762C1)](https://drive.google.com/file/d/1_CITKY9u_Fh77iqQDP2TrcbVD5_61ArT/view?usp=sharing)
+[![Jupyter Notebook](https://img.shields.io/badge/jupyter-%23FA0F00.svg?style=for-the-badge&logo=jupyter&logoColor=white)](https://github.com/muzairkhattak/ViFi-CLIP/blob/main/ViFi-CLIP_Inference_custom_video.ipynb)
+
+Official implementation of the paper "[Fine-tuned CLIP models are efficient video learners](https://arxiv.org/abs/2212.03640)".
+<hr />
+
+[//]: # ([![PWC]&#40;https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/maple-multi-modal-prompt-learning/prompt-engineering-on-imagenet&#41;]&#40;https://paperswithcode.com/sota/prompt-engineering-on-imagenet?p=maple-multi-modal-prompt-learning&#41;)
+
+[//]: # ([![PWC]&#40;https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/maple-multi-modal-prompt-learning/prompt-engineering-on-sun397&#41;]&#40;https://paperswithcode.com/sota/prompt-engineering-on-sun397?p=maple-multi-modal-prompt-learning&#41;)
+
+[//]: # ([![PWC]&#40;https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/maple-multi-modal-prompt-learning/prompt-engineering-on-eurosat&#41;]&#40;https://paperswithcode.com/sota/prompt-engineering-on-eurosat?p=maple-multi-modal-prompt-learning&#41;)
+
+[//]: # ([![PWC]&#40;https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/maple-multi-modal-prompt-learning/prompt-engineering-on-ucf101&#41;]&#40;https://paperswithcode.com/sota/prompt-engineering-on-ucf101?p=maple-multi-modal-prompt-learning&#41;)
+
+[//]: # ([![PWC]&#40;https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/maple-multi-modal-prompt-learning/prompt-engineering-on-fgvc-aircraft&#41;]&#40;https://paperswithcode.com/sota/prompt-engineering-on-fgvc-aircraft?p=maple-multi-modal-prompt-learning&#41;)
+
+[//]: # ()
+[//]: # ()
+[//]: # (<hr />)
+
+# :rocket: News
+* **(Nov 24, 2023)**
+  * Interactive notebook released. Inference with ViFi-CLIP on custom videos without significant installation dependencies!
+* **(Feb 28, 2023)**
+  * Paper accepted at CVPR 2023 :tada: 
+* **(Dec 6, 2022)** 
+  * Training and evaluation codes for [ViFi-CLIP](https://arxiv.org/abs/2212.03640), along with pretrained models are released.
+  
+<hr />
+
+## Highlights
+
+![main figure](docs/main_figure.png)
+<p align="justify">  This work explores the capability of a simple baseline called ViFi-CLIP (Video Fine-tuned CLIP)
+for adapting image pretrained CLIP to video domain. The figure compares the zero-shot performance of vanilla CLIP
+and several of its variants adapted for videos (trained on Kinetics-400, evaluated on UCF-101 and HMDB-51).
+The t-SNE visualizations of video-embeddings obtained from ViFi-CLIP (4th col.) are compared with embeddings
+from vanilla CLIP (1st col.), individually tuned CLIP text (2nd col.) and image encoder (3rd col.) on videos,
+and recent state-of-the-art work, XCLIP (last col.) (∆ represents difference over XCLIP). The embeddings of
+ViFi-CLIP are better separable, indicating that a simple fine-tuning of CLIP is sufficient to learn suitable 
+video-specific inductive biases, and can perform competitive to more complex approaches having dedicated components 
+designed to model temporal information in videos. </p>
+
+
+> **<p align="justify"> Abstract:** *Large-scale multi-modal training with image-text pairs imparts
+> strong generalization to CLIP model. Since training on a similar scale for videos is infeasible,
+> recent approaches focus on the effective transfer of image-based CLIP to the video domain. In this 
+> pursuit, new parametric modules are added to learn temporal information and inter-frame relationships
+> which require meticulous design efforts. Furthermore, when the resulting models are learned on videos
+> , they tend to overfit on the given task distribution and lack in generalization aspect. This begs the
+> following question: How to effectively transfer image-level CLIP representations to videos? In this work,
+> we show that a simple Video Fine-tuned CLIP (ViFi-CLIP) baseline is generally sufficient to bridge the
+> domain gap from images to videos. Our qualitative analysis illustrates that the frame-level processing 
+> from CLIP image-encoder followed by feature pooling and similarity matching with corresponding text 
+> embeddings helps in implicitly modeling the temporal cues within ViFi-CLIP. Such fine-tuning helps
+> the model to focus on scene dynamics, moving objects and inter-object relationships. For low-data 
+> regimes where full fine-tuning is not viable, we propose a `bridge and prompt' approach that first
+> uses fine-tuning to bridge the domain gap and then learns prompts on language and vision side to 
+> adapt CLIP representations. We extensively evaluate this simple yet strong baseline on zero-shot,
+> base-to-novel generalization, few-shot and fully supervised settings across five video benchmarks.* </p>
+
+## Main Contributions
+
+1) **ViFi-CLIP:** We formulate and show the significance of an often neglected but
+simple baseline for transferring image-based CLIP model to video domain. ViFi-CLIP (Video Fine-tuned CLIP) shows that a simple fine-tuning of CLIP is sufficient to learn suitable video-specific inductive biases,
+and can perform competitive to more complex approaches having dedicated components designed to model temporal information in videos.
+2) **Base-to-novel generalization benchmark:** We introduce base-to-novel generalization benchmark for video-domain for evaluating the generalization ability of models for video action recognition.
+3) **Bridge and Prompt approach:** We show the effectiveness of our proposed ‘bridge and prompt’ approach to first bridge the modality gap through fine-tuning followed by prompt learning in both visual and language branches of the CLIP model
+for low-data regimes.
+
+   
+# Model Zoo
+NOTE: All models in our experiments below uses publicly available ViT/B-16 based CLIP model. The trained model weights against each experiment is provided in tables below.
+
+### Zero-shot results
+All models are trained on Kinetics-400 and then evaluated directly on downstream datasets.
+
+| Name  (configs)                                                           | Input  | HMDB-51 | UCF-101 | Kinetics-600 |                                                                    Model                                                                     |
+|---------------------------------------------------------------------------|:------:|:-------:|:-------:|:------------:|:--------------------------------------------------------------------------------------------------------------------------------------------:|
+| [CLIP image-FT](configs/zero_shot/train/k400/16_16_image_tuned_clip.yaml) | 32x224 |  49.0   |  72.9   |     62.2     | [link](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/EdA6n7TCQEFAse5X1g1I08AByLCWHM69axTyK9OyVZy86Q?e=NaipU1) |
+| [CLIP text-FT](configs/zero_shot/train/k400/16_16_text_tuned_clip.yaml)   | 32x224 |  48.5   |  69.8   |     68.5     | [link](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/Eea6hW-_RBdJo4T5JJ_sWgEBNGFdA91tPTq9MQ-XkO5dMg?e=hGneeQ) |
+| [ViFi-CLIP](configs/zero_shot/train/k400/16_16_vifi_clip.yaml)            | 32x224 |  51.3   |  76.8   |     71.2     | [link](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/EW0shb6XYDxFi3BH6DT70rgBPDwgW_knQ8jDsarxINXezw?e=RbixXc) |
+
+
+### Base-to-novel generalization results
+Here, we divide each dataset into base and novel classes.
+All models are trained on base classes and evaluated on both base and novel classes. Results are averaged over 3 seeds for each experiment.
+
+#### Kinetics-400
+| Name  (configs)                                                | Input  | Base Acc. | Novel Acc. |  HM  |                                                                                                                                                                                                                   Model                                                                                                                                                                                                                   |
+|----------------------------------------------------------------|:------:|:---------:|:----------:|:----:|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:|
+| [CLIP image-FT](configs/base2novel/finetuning_base2novel/k400) | 32x224 |   72.9    |    58.0    | 64.6 | [seed1](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/EXaQGUrODN9DjxtWuSylHJIBbFtAimZHdubKSHPlTT79eg?e=WiNFM9)/[seed2](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/ESKX8BXvQoBHn5jq04EoowEB0zR6iPxlkxjSuWJbHupceg?e=vDniMa)/[seed3](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/Ed1D7oXTF6VKtAVSleSJHowBJxsdu1kNNDRk4LBGOfzokg?e=gqt8en) |
+| [CLIP text-FT](configs/base2novel/finetuning_base2novel/k400)  | 32x224 |   73.4    |    59.7    | 65.8 | [seed1](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/EduVCGSp11tFlwyCKg5ee7wBMJQwGHN9gKNBJozpZCgPEg?e=NPeIjf)/[seed2](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/ERw9FPot9T9PrVw0kxdsQvkBcpDuDYYUnIFLTjm_xqz8zA?e=8dkLY8)/[seed3](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/EU1lFXfDColIuYqGRrujImwBVqz2vP5gpTAM446HPa7erA?e=MCcZ6t) |
+| [ViFi-CLIP](configs/base2novel/finetuning_base2novel/k400)     | 32x224 |   76.4    |    61.1    | 67.9 | [seed1](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/EVEyFxODEvtFt6FVpuIQvNQBgi5bfxce_nqgzqsjuxB48g?e=rOAu0o)/[seed2](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/EcCHHh5FvnlPnlQTHLUk2v0Bv6MMTWHpkluBiQ1MdbZWFA?e=d3NkTX)/[seed3](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/ETG_gS_l-E1Ai6BkPq8WlzgB8L5PDYDoVrgzia9832j3wg?e=rfJzPs) |
+
+#### HMDB-51
+| Name  (configs)                                                | Input  | Base Acc. | Novel Acc. |  HM  |                                                                                                                                                                                                                   Model                                                                                                                                                                                                                   |
+|----------------------------------------------------------------|:------:|:---------:|:----------:|:----:|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:|
+| [CLIP image-FT](configs/base2novel/finetuning_base2novel/hmdb) | 32x224 |   62.6    |    47.5    | 54.0 | [seed1](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/EVNYYAhsZtZMtzoQcfKx7rQBlrEYkvUyDVfauuMobgAA0g?e=GQ2D8z)/[seed2](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/EQaX5EzlLfhGhbZEzsgST0cB4HD0saOuoYgBCW7K8bzaBg?e=tKNkqY)/[seed3](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/EUiGmlJa3M9Fgx6epCvPiRkBtkON4YKMWEtQSkwqC3dXWw?e=72DTbt) |
+| [CLIP text-FT](configs/base2novel/finetuning_base2novel/hmdb)  | 32x224 |   70.0    |    51.2    | 59.1 | [seed1](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/ETJ12FfB_8RLg22CHxKPHosBPmFL52G9kbKGayQiqoHXYQ?e=hTb1tv)/[seed2](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/EXVDioTuv6dKgroWI-qmrEUBZV5njUMUndR_XJDZNTcXcw?e=rgPF49)/[seed3](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/EQMykf375n1Iqm2IgABFT2oBokF2ooseZITmvyx2RKX4TA?e=1XNgtI) |
+| [ViFi-CLIP](configs/base2novel/finetuning_base2novel/hmdb)     | 32x224 |   73.8    |    53.3    | 61.9 | [seed1](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/ETbI3yeoedBNqvAf3oz-faIBeGDy862_Tx_ZQT1soM6hZQ?e=2Y5Vxg)/[seed2](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/EZ-JcyYOVthCu2pU4ou-AWgBHzMYzWsSKC7eL4KBU3xyLg?e=0bj1ed)/[seed3](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/EeUfaRWGtEpPn9hVrpb8pCsBJGAMrGZgXLKOOzNNY1DGqA?e=6B7dJy) |
+
+#### UCF-101
+| Name  (configs)                                               | Input  | Base Acc. | Novel Acc. |  HM  |                                                                                                                                                                                                                   Model                                                                                                                                                                                                                   |
+|---------------------------------------------------------------|:------:|:---------:|:----------:|:----:|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:|
+| [CLIP image-FT](configs/base2novel/finetuning_base2novel/ucf) | 32x224 |   86.4    |    65.3    | 74.4 | [seed1](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/EexxOxwJE8dHtk8ykBn39k4B9OaJK88L-N4c8AYOvj4LNA?e=ZCsYlc)/[seed2](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/EdC89wvjprhJgG-Q3DuzF_0BoKT0fxQWSeRLgJ6urodhaw?e=U3gU8U)/[seed3](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/Efsw7nYSffZKhzwnIpwX89kBsqSSMhexheB-fb-xFn0fOQ?e=Q69d2d) |
+| [CLIP text-FT](configs/base2novel/finetuning_base2novel/ucf)  | 32x224 |   90.9    |    67.4    | 77.4 | [seed1](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/EZmAd-E7FXZBoOKa9XY8RfcBG9Qk7nhlLwHin8oN89IKMg?e=Xnmtn9)/[seed2](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/ERIpU_6ZhUpKjTZ6QQVfKPwBQkUiWLM6yRSOJmFZGOK4-Q?e=pkENDN)/[seed3](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/EVoHq04lVOhIpE1pqaI7lmYBhHoh_6Nndgx7xMCZqeXTMw?e=qkcbFm) |
+| [ViFi-CLIP](configs/base2novel/finetuning_base2novel/ucf)     | 32x224 |   92.9    |    67.7    | 78.3 | [seed1](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/EXwqEdOLKSdIpY6AfTSbRMQB0UqZdTKiaWjw-2gf8Ctcyw?e=h2MvBZ)/[seed2](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/EdOmRlCM4zZJpr-Z497OfB4B5YK8qTiApht1StA7xJ3ClA?e=9zYxfS)/[seed3](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/EdgjBDJ0iXtMpdkNqOE5otcBgWgbfrrQBG1W0wICrD9qiA?e=x7VXl2) |
+
+#### SSv2
+| Name  (configs)                                                | Input  | Base Acc. | Novel Acc. |  HM  |                                                                                                                                                                                                                   Model                                                                                                                                                                                                                   |
+|----------------------------------------------------------------|:------:|:---------:|:----------:|:----:|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:|
+| [CLIP image-FT](configs/base2novel/finetuning_base2novel/ssv2) | 32x224 |    9.2    |    8.5     | 8.8  | [seed1](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/EfLcOFvIHK1Hjj-Yw7z_TQ8BSwmptokbOsPuzWnqAm8iTg?e=3gb20s)/[seed2](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/EfAL5G3trhJHue-6RTF4HhsBStMma3XEvzWv_0wQnh1YlA?e=sTnbDG)/[seed3](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/Eff55gcBtRxDuCGebyc0zTIBoAPgwDusk0U5jg7-ddjDDg?e=bXB25M) |
+| [CLIP text-FT](configs/base2novel/finetuning_base2novel/ssv2)  | 32x224 |   12.4    |    9.5     | 10.8 | [seed1](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/EdYLS33jyZZDsIy71Lk3TfwB76xrHL3BIRrUiNeSvWfnWg?e=ndm1JL)/[seed2](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/EbpzILaqXJBKgPmTKBA32d0BsFrErjRCAwMwaXNKB39G5w?e=FbLCaN)/[seed3](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/EY_VHJNKBhlFuir2dL1frOQB5GbG2UeSoG4p65Wh5wOHNg?e=HncWmy) |
+| [ViFi-CLIP](configs/base2novel/finetuning_base2novel/ssv2)     | 32x224 |   16.2    |    12.1    | 13.9 | [seed1](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/Ee9-LsJzAeROj0rsXZ_Kq2gBWfDTJX9yI3NhsP3Wx9XT7g?e=QTh28B)/[seed2](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/ETWroKKSa3VJmktA1qGcrUIBSWdSK8JaclCD7GpxXWMMRw?e=bNM8PS)/[seed3](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/Efl1L1g_OdJHvLu24Yzh3w4BMrTcdll8DilX13lB6rXaFw?e=lLvOiJ) |
+
+
+#### VL Prompting approach: Base-to-Novel
+ViFi-CLIP is first trained on K400 and then vision and language prompts are further fine-tuned on the downstream datasets.
+
+| Dataset (configs)                                       | Input  | Base Acc. | Novel Acc. |  HM  |                                                                                                                                                                                                                   Model                                                                                                                                                                                                                   |
+|---------------------------------------------------------|:------:|:---------:|:----------:|:----:|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:|
+| [HMDB-51](configs/base2novel/prompting_base2novel/hmdb) | 32x224 |   77.1    |    54.9    | 64.1 | [seed1](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/Ee1tEk7Tw-dNibQEMVZYBPMBhYj2--lFdIceS1DNN55mUQ?e=qzP1vE)/[seed2](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/EWxj-A1_EldJggHhBgVTFPIBdcGAXZn1yiWBATvgTKvLYg?e=WLfYUT)/[seed3](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/EXT2ezu2RZBEnKuzkEwYb48BE9LYaXoh-cT9dNSruYiKyg?e=b5cbmX) |
+| [UCF-101](configs/base2novel/prompting_base2novel/ucf)  | 32x224 |   95.9    |    74.1    | 83.6 | [seed1](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/EYNvOOiV0qZIj-YIZlIH-dcBr-8eALRnPse189llN7QiPQ?e=wbbxDB)/[seed2](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/EeBoMzLQ-YNNtl5YAKS0MmkBoKWpxblQQk3ieT50OtwlQQ?e=16jKbC)/[seed3](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/EWwQJkz41o9KgXkgpDoJnjYBCyCD4bV0pBS9XtAD8VpLoQ?e=VKyBNc) |
+| [SSv2](configs/base2novel/prompting_base2novel/ssv2)    | 32x224 |   15.8    |    11.5    | 13.3 | [seed1](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/ESey1Xo8Ka1HoJtu04xsng0BSTFIRgOty4AwIlnQL7iuJQ?e=n27FNI)/[seed2](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/EeLJ6F4mXxBHgBj0qQEXkjkBOCImmwSns3J51yG9YIkjAQ?e=eoXWyd)/[seed3](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/EQ8Vjdf0t8ZEuJFGlTBwP2sBmDRhM7FWuYmyOh0UZJdhPg?e=ZMppVA) |
+
+
+### Few-shot results
+Below table shows few-shot results of ViFi-CLIP for K=2, 4, 8 and 16.
+
+| Name  (configs)                                                                       | Dataset | K (shots) | Input  | Top-1 Acc. |                                                                    Model                                                                     |
+|---------------------------------------------------------------------------------------|:-------:|:---------:|:-------|:----------:|:--------------------------------------------------------------------------------------------------------------------------------------------:|
+| [ViFi-CLIP](configs/few_shot/finetuning_few_shot/hmdb51/16_32_vifi_clip_2_shot.yaml)  | HMDB-51 |     2     | 32x224 |    57.2    | [link](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/EZfPCFy69GlLms0xE9hacYsBMRDZolyy5-5kh7urW6U5Hg?e=PRR4dj) |
+| [ViFi-CLIP](configs/few_shot/finetuning_few_shot/hmdb51/16_32_vifi_clip_4_shot.yaml)  | HMDB-51 |     4     | 32x224 |    62.7    | [link](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/EYSoKhu-CEdFtDIPDB-9mcYBTocR1z6S4pB2prm8M3y86w?e=MgiPpY) |
+| [ViFi-CLIP](configs/few_shot/finetuning_few_shot/hmdb51/16_32_vifi_clip_8_shot.yaml)  | HMDB-51 |     8     | 32x224 |    64.5    | [link](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/EXLoRgDpJERKnxWf6GGGqzoBy-jbAuO-IcV4QSWmtT2mBg?e=piTDRc) |
+| [ViFi-CLIP](configs/few_shot/finetuning_few_shot/hmdb51/16_32_vifi_clip_16_shot.yaml) | HMDB-51 |    16     | 32x224 |    66.8    | [link](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/EdA4jgYynRBHrhy1ftn-s9gBFRFYCPdaD5y9AQBClaziWg?e=x2tHpP) |
+| [ViFi-CLIP](configs/few_shot/finetuning_few_shot/ucf101/16_32_vifi_clip_2_shot.yaml)  | UCF-101 |     2     | 32x224 |    80.7    | [link](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/ERaxz4xkUBdGkGCKopmsctgBWj0aoxf4eNWRFIQPtZja6A?e=FzpFnl) |
+| [ViFi-CLIP](configs/few_shot/finetuning_few_shot/ucf101/16_32_vifi_clip_4_shot.yaml)  | UCF-101 |     4     | 32x224 |    85.1    | [link](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/ETa1Ym63eYtDt9Fzlq_5YuEBcNCPlUPbD12zhc4YGusGyg?e=Z1Si0j) |
+| [ViFi-CLIP](configs/few_shot/finetuning_few_shot/ucf101/16_32_vifi_clip_8_shot.yaml)  | UCF-101 |     8     | 32x224 |    90.0    | [link](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/EaHr57kr7GBGno5v6Qb7sLUBERvoInzco0yfbO81davqWQ?e=V2Odqn) |
+| [ViFi-CLIP](configs/few_shot/finetuning_few_shot/ucf101/16_32_vifi_clip_16_shot.yaml) | UCF-101 |    16     | 32x224 |    92.7    | [link](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/ERGWnUJHBiVJluMvaUrbDPcB3iIGXAet0W-AfwDJy1bL2w?e=0fSQJb) |
+| [ViFi-CLIP](configs/few_shot/finetuning_few_shot/ssv2/16_32_vifi_clip_2_shot.yaml)    |  SSv2   |     2     | 32x224 |    6.2     | [link](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/EfmVXJyo9VxHheDrVrm7b88BJ_MXRyI_dhuI9pWMUpfPww?e=JPmnt2) |
+| [ViFi-CLIP](configs/few_shot/finetuning_few_shot/ssv2/16_32_vifi_clip_4_shot.yaml)    |  SSv2   |     4     | 32x224 |    7.4     | [link](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/ET1MeS3-C_NLpg-rAJMnf0cBruk16K56NDCwySFwse1tsQ?e=1fV3k2) |
+| [ViFi-CLIP](configs/few_shot/finetuning_few_shot/ssv2/16_32_vifi_clip_8_shot.yaml)    |  SSv2   |     8     | 32x224 |    8.5     | [link](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/EWp7ERV-Dn9GiiTgKWyjDyMBUVoLXyPdHcBpAPah3XvZmw?e=r5Xmii) |
+| [ViFi-CLIP](configs/few_shot/finetuning_few_shot/ssv2/16_32_vifi_clip_16_shot.yaml)   |  SSv2   |    16     | 32x224 |    12.4    | [link](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/EZJB66ssj_VBhZB6e59wI9oB1qHGKujTAhoSKyqvnpEzDw?e=Vdjp5n) |
+
+NOTE: Few-shot results for other CLIP Fine-tuned variants are presented in our main paper (Table 3). Model weights for other variants are provided [here](https://mbzuaiac-my.sharepoint.com/:f:/g/personal/uzair_khattak_mbzuai_ac_ae/Elz1joid4FlAkgDnr_O1ZLMBNxK3jZOlzdAHv5yopYakJQ?e=wyDe8r).
+
+#### VL Prompting approach: Few-shot
+ViFi-CLIP is first trained on K400 and then vision and language prompts are further fine-tuned on the downstream datasets in few-shot manner.
+
+| Dataset (configs)                                     | Input  | K=2  | K=4  | K=8  | K=16 |                                                                                                                                                                                                                                                                                      Model                                                                                                                                                                                                                                                                                       |
+|-------------------------------------------------------|:------:|:-----|:----:|:----:|:----:|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:|
+| [HMDB-51](configs/few_shot/prompting_few_shot/hmdb51) | 32x224 | 63.0 | 65.1 | 69.6 | 72.0 | [K=2](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/EQ0oDcnAJLtJt4CmdhnApVIBiWD2YwAO5x01TYy0mpEmzA?e=iFvoSV)/[K=4](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/Ed3LaBQWcrhLgqStigS5HAsBimR0K6DR5l2x_dI6kWuDCA?e=QfYeRd)/[K=8](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/EbpdVbtqUUlLt86s2Ze5gBoBAvG4KgWJVbYFVMMErX7Smw?e=lxRFPs)/[K=16](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/EQTj4Xe9veRHqmypVCJ6rRMBEMO6Rky3m_V2Q8f7lqrpEw?e=V1DJRH) |
+| [UCF-101](configs/few_shot/prompting_few_shot/ucf101) | 32x224 | 91.0 | 93.7 | 95.0 | 96.4 | [K=2](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/EXz03SRz-NdCmVcWpSt-GEwBxrBWmlGbitXq9iRGz8EczQ?e=zpongw)/[K=4](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/EaFyG9bOXUhEnOsviO0BhowBpjvbRcJb9zCehcgyXdhHRQ?e=fl7H6a)/[K=8](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/ERM8RDkpandOshsedBwL0fQBrdQd26zjbaBGGGw1XhuTuQ?e=z8GDng)/[K=16](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/EbUEylUsTyBOhPZqy99sY8UBMtE0AA46TdY-MTDs8ma0AA?e=g2038u) |
+| [SSv2](configs/few_shot/prompting_few_shot/ssv2)      | 32x224 | 6.7  | 7.9  | 10.2 | 13.5 | [K=2](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/EbBoLMM3RnNAvoZ3TdoGYSMBFCfsB_gfaz3svxtyKUdxEA?e=KeIl1s)/[K=4](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/EbzbcG00RgJFsezk_tnDmQkBCf7wPPIexuKEgUZJKgmMew?e=lEXJ45)/[K=8](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/ESVYfUXIjZ9CppbPt8mgKOABVKSljMNI2JiD9PLkoABSoQ?e=fxI1l1)/[K=16](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/Ec3wlwVsJ4FDprzfChxkZeoBqz4AH7Y4JRF1SjsvMsOWcw?e=ya59zp) |
+
+
+
+### Fully-supervised results on Kinetics-400
+| Name  (configs)                                                            | FLOPS(G) | Input  | Top-1 Acc. | Top-5 Acc. |                                                                    Model                                                                     |
+|----------------------------------------------------------------------------|:--------:|:------:|:----------:|:----------:|:--------------------------------------------------------------------------------------------------------------------------------------------:|
+| [CLIP image-FT](configs/fully_supervised/k400/16_16_image_tuned_clip.yaml) |   281    | 16x224 |    82.8    |    96.2    | [link](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/EdmXN3BQe79BgW1Tuw3Q--QBPbSc4b1N5-ahEIaK-SxRRA?e=e4bLz7) |
+| [CLIP text-FT](configs/fully_supervised/k400/16_16_text_tuned_clip.yaml)   |   281    | 16x224 |    73.1    |    91.2    | [link](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/EeKqDguvX8NPvz5MIKmVPBIBLxL0wkzh0SCmpfs8ZebdZQ?e=2mKBTr) |
+| [ViFi-CLIP](configs/fully_supervised/k400/16_16_vifi_clip.yaml)            |   281    | 16x224 |    83.9    |    96.3    | [link](https://mbzuaiac-my.sharepoint.com/:u:/g/personal/uzair_khattak_mbzuai_ac_ae/EfqisYTGKlVIiPI0QHG-pxMBuBMA0906jX_kPpaRGw9Ksw?e=TdbaBU) |
+
+## Installation 
+For installation and other package requirements, please follow the instructions detailed in [INSTALL.md](docs/INSTALL.md). 
+
+## Data preparation
+Please follow the instructions at [DATASETS.md](docs/DATASETS.md) to prepare all datasets.
+
+# Training 
+For all experiments shown in above tables, we provide config files in `configs` folder. For example, to train ViFi-CLIP (tunes both image and text encoder) on Kinetics-400, run the following command:
+```
+python -m torch.distributed.launch --nproc_per_node=8 \ 
+main.py -cfg configs/fully_supervised/k400/16_16_vifi_clip.yaml --output /PATH/TO/OUTPUT 
+```
+
+**Note:**
+- We recommend keeping the total batch size as mentioned in respective config files. Please use `--accumulation-steps` to maintain the total batch size. Specifically, here the effective total batch size is 8(`GPUs_NUM`) x 4(`TRAIN.BATCH_SIZE`) x 16(`TRAIN.ACCUMULATION_STEPS`) = 512.
+- After setting up the datasets as instructed [DATASETS.md](docs/DATASETS.md), only argument in the config file that should be specified is data path. All other settings in config files are pre-set.
+
+For detailed training instructions for all experimental setup, please refer to [TRAIN.md](docs/TRAIN.md).
+
+# Evaluating models
+To evaluate a model, please use a suitable config and corresponding model weights. For example, to evaluate ViFi-CLIP with 16 frames on Kinetics-400, run the command below:
+```
+python -m torch.distributed.launch --nproc_per_node=8 main.py \
+-cfg configs/fully_supervised/k400/16_16_vifi_clip.yaml --output /PATH/TO/OUTPUT \
+--only_test --resume /PATH/TO/CKPT --opts TEST.NUM_CLIP 4 TEST.NUM_CROP 3
+```
+
+## Contact
+If you have any questions, please create an issue on this repository or contact at uzair.khattak@mbzuai.ac.ae or hanoona.bangalath@mbzuai.ac.ae .
+
+
+# Citation
+If you use our approach (code, model or dataset splits) in your research, please consider citing:
+```
+@inproceedings{hanoonavificlip,
+    title={Finetuned CLIP models are efficient video learners},
+    author={Rasheed, Hanoona and khattak, Muhammad Uzair and Maaz, Muhammad and Khan, Salman and Khan, Fahad Shahbaz},
+    booktitle={The IEEE/CVF Conference on Computer Vision and Pattern Recognition},
+    year={2023}
+}
+```
+
+# Acknowledgements
+Our code is based on [XCLIP's repository](https://github.com/microsoft/VideoX/tree/master/X-CLIP). We sincerely thank the authors for releasing their code. If you use our model and code, please consider citing XCLIP as well.

clip/__init__.py

@@ -0,0 +1 @@
+from .clip import *

clip/bpe_simple_vocab_16e6.txt.gz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:924691ac288e54409236115652ad4aa250f48203de50a9e4722a6ecd48d6804a
+size 1356917

clip/clip.py

@@ -0,0 +1,219 @@
+import hashlib
+import os
+import urllib
+import warnings
+from typing import Union, List
+
+import torch
+from PIL import Image
+from torchvision.transforms import Compose, Resize, CenterCrop, ToTensor, Normalize
+from tqdm import tqdm
+
+from .model import build_model
+from .simple_tokenizer import SimpleTokenizer as _Tokenizer
+
+try:
+    from torchvision.transforms import InterpolationMode
+    BICUBIC = InterpolationMode.BICUBIC
+except ImportError:
+    BICUBIC = Image.BICUBIC
+
+
+if torch.__version__.split(".") < ["1", "7", "1"]:
+    warnings.warn("PyTorch version 1.7.1 or higher is recommended")
+
+
+__all__ = ["available_models", "load", "tokenize"]
+_tokenizer = _Tokenizer()
+
+_MODELS = {
+    "ViT-B/32": "https://openaipublic.azureedge.net/clip/models/40d365715913c9da98579312b702a82c18be219cc2a73407c4526f58eba950af/ViT-B-32.pt",
+    "ViT-B/16": "https://openaipublic.azureedge.net/clip/models/5806e77cd80f8b59890b7e101eabd078d9fb84e6937f9e85e4ecb61988df416f/ViT-B-16.pt",
+    "ViT-L/14": "https://openaipublic.azureedge.net/clip/models/b8cca3fd41ae0c99ba7e8951adf17d267cdb84cd88be6f7c2e0eca1737a03836/ViT-L-14.pt",
+    "ViT-L/14@336px": "https://openaipublic.azureedge.net/clip/models/3035c92b350959924f9f00213499208652fc7ea050643e8b385c2dac08641f02/ViT-L-14-336px.pt",
+}
+
+
+def _download(url: str, root: str = os.path.expanduser("~/.cache/clip")):
+    os.makedirs(root, exist_ok=True)
+    filename = os.path.basename(url)
+
+    expected_sha256 = url.split("/")[-2]
+    download_target = os.path.join(root, filename)
+
+    if os.path.exists(download_target) and not os.path.isfile(download_target):
+        raise RuntimeError(f"{download_target} exists and is not a regular file")
+
+    if os.path.isfile(download_target):
+        if hashlib.sha256(open(download_target, "rb").read()).hexdigest() == expected_sha256:
+            return download_target
+        else:
+            warnings.warn(f"{download_target} exists, but the SHA256 checksum does not match; re-downloading the file")
+
+    with urllib.request.urlopen(url) as source, open(download_target, "wb") as output:
+        with tqdm(total=int(source.info().get("Content-Length")), ncols=80, unit='iB', unit_scale=True) as loop:
+            while True:
+                buffer = source.read(8192)
+                if not buffer:
+                    break
+
+                output.write(buffer)
+                loop.update(len(buffer))
+
+    if hashlib.sha256(open(download_target, "rb").read()).hexdigest() != expected_sha256:
+        raise RuntimeError(f"Model has been downloaded but the SHA256 checksum does not not match")
+
+    return download_target
+
+
+def _transform(n_px):
+    return Compose([
+        Resize(n_px, interpolation=BICUBIC),
+        CenterCrop(n_px),
+        lambda image: image.convert("RGB"),
+        ToTensor(),
+        Normalize((0.48145466, 0.4578275, 0.40821073), (0.26862954, 0.26130258, 0.27577711)),
+    ])
+
+
+def available_models() -> List[str]:
+    """Returns the names of available CLIP models"""
+    return list(_MODELS.keys())
+
+
+def load(name: str, device: Union[str, torch.device] = "cuda" if torch.cuda.is_available() else "cpu", jit=False):
+    """Load a CLIP model
+
+    Parameters
+    ----------
+    name : str
+        A model name listed by `clip.available_models()`, or the path to a model checkpoint containing the state_dict
+
+    device : Union[str, torch.device]
+        The device to put the loaded model
+
+    jit : bool
+        Whether to load the optimized JIT model or more hackable non-JIT model (default).
+
+    Returns
+    -------
+    model : torch.nn.Module
+        The CLIP model
+
+    preprocess : Callable[[PIL.Image], torch.Tensor]
+        A torchvision transform that converts a PIL image into a tensor that the returned model can take as its input
+    """
+    if name in _MODELS:
+        model_path = _download(_MODELS[name])
+    elif os.path.isfile(name):
+        model_path = name
+    else:
+        raise RuntimeError(f"Model {name} not found; available models = {available_models()}")
+
+    try:
+        # loading JIT archive
+        model = torch.jit.load(model_path, map_location=device if jit else "cpu").eval()
+        state_dict = None
+    except RuntimeError:
+        # loading saved state dict
+        if jit:
+            warnings.warn(f"File {model_path} is not a JIT archive. Loading as a state dict instead")
+            jit = False
+        state_dict = torch.load(model_path, map_location="cpu")
+
+    if not jit:
+        model = build_model(state_dict or model.state_dict()).to(device)
+        if str(device) == "cpu":
+            model.float()
+        return model, _transform(model.visual.input_resolution)
+
+    # patch the device names
+    device_holder = torch.jit.trace(lambda: torch.ones([]).to(torch.device(device)), example_inputs=[])
+    device_node = [n for n in device_holder.graph.findAllNodes("prim::Constant") if "Device" in repr(n)][-1]
+
+    def patch_device(module):
+        try:
+            graphs = [module.graph] if hasattr(module, "graph") else []
+        except RuntimeError:
+            graphs = []
+
+        if hasattr(module, "forward1"):
+            graphs.append(module.forward1.graph)
+
+        for graph in graphs:
+            for node in graph.findAllNodes("prim::Constant"):
+                if "value" in node.attributeNames() and str(node["value"]).startswith("cuda"):
+                    node.copyAttributes(device_node)
+
+    model.apply(patch_device)
+    patch_device(model.encode_image)
+    patch_device(model.encode_text)
+
+    # patch dtype to float32 on CPU
+    if str(device) == "cpu":
+        float_holder = torch.jit.trace(lambda: torch.ones([]).float(), example_inputs=[])
+        float_input = list(float_holder.graph.findNode("aten::to").inputs())[1]
+        float_node = float_input.node()
+
+        def patch_float(module):
+            try:
+                graphs = [module.graph] if hasattr(module, "graph") else []
+            except RuntimeError:
+                graphs = []
+
+            if hasattr(module, "forward1"):
+                graphs.append(module.forward1.graph)
+
+            for graph in graphs:
+                for node in graph.findAllNodes("aten::to"):
+                    inputs = list(node.inputs())
+                    for i in [1, 2]:  # dtype can be the second or third argument to aten::to()
+                        if inputs[i].node()["value"] == 5:
+                            inputs[i].node().copyAttributes(float_node)
+
+        model.apply(patch_float)
+        patch_float(model.encode_image)
+        patch_float(model.encode_text)
+
+        model.float()
+
+    return model, _transform(model.input_resolution.item())
+
+
+def tokenize(texts: Union[str, List[str]], context_length: int = 77, truncate: bool = False) -> torch.LongTensor:
+    """
+    Returns the tokenized representation of given input string(s)
+
+    Parameters
+    ----------
+    texts : Union[str, List[str]]
+        An input string or a list of input strings to tokenize
+
+    context_length : int
+        The context length to use; all CLIP models use 77 as the context length
+
+    truncate: bool
+        Whether to truncate the text in case its encoding is longer than the context length
+
+    Returns
+    -------
+    A two-dimensional tensor containing the resulting tokens, shape = [number of input strings, context_length]
+    """
+    if isinstance(texts, str):
+        texts = [texts]
+
+    sot_token = _tokenizer.encoder["<|startoftext|>"]
+    eot_token = _tokenizer.encoder["<|endoftext|>"]
+    all_tokens = [[sot_token] + _tokenizer.encode(text) + [eot_token] for text in texts]
+    result = torch.zeros(len(all_tokens), context_length, dtype=torch.long)
+
+    for i, tokens in enumerate(all_tokens):
+        if len(tokens) > context_length:
+            if truncate:
+                tokens = tokens[:context_length]
+                tokens[-1] = eot_token
+            else:
+                raise RuntimeError(f"Input {texts[i]} is too long for context length {context_length}")
+        result[i, :len(tokens)] = torch.tensor(tokens)
+
+    return result

clip/model.py

@@ -0,0 +1,521 @@
+from collections import OrderedDict
+from typing import Tuple, Union
+
+import numpy as np
+import torch
+import torch.nn.functional as F
+from torch import nn
+
+
+class Bottleneck(nn.Module):
+    expansion = 4
+
+    def __init__(self, inplanes, planes, stride=1):
+        super().__init__()
+
+        # all conv layers have stride 1. an avgpool is performed after the second convolution when stride > 1
+        self.conv1 = nn.Conv2d(inplanes, planes, 1, bias=False)
+        self.bn1 = nn.BatchNorm2d(planes)
+
+        self.conv2 = nn.Conv2d(planes, planes, 3, padding=1, bias=False)
+        self.bn2 = nn.BatchNorm2d(planes)
+
+        self.avgpool = nn.AvgPool2d(stride) if stride > 1 else nn.Identity()
+
+        self.conv3 = nn.Conv2d(planes, planes * self.expansion, 1, bias=False)
+        self.bn3 = nn.BatchNorm2d(planes * self.expansion)
+
+        self.relu = nn.ReLU(inplace=True)
+        self.downsample = None
+        self.stride = stride
+
+        if stride > 1 or inplanes != planes * Bottleneck.expansion:
+            # downsampling layer is prepended with an avgpool, and the subsequent convolution has stride 1
+            self.downsample = nn.Sequential(OrderedDict([
+                ("-1", nn.AvgPool2d(stride)),
+                ("0", nn.Conv2d(inplanes, planes * self.expansion, 1, stride=1, bias=False)),
+                ("1", nn.BatchNorm2d(planes * self.expansion))
+            ]))
+
+    def forward(self, x: torch.Tensor):
+        identity = x
+
+        out = self.relu(self.bn1(self.conv1(x)))
+        out = self.relu(self.bn2(self.conv2(out)))
+        out = self.avgpool(out)
+        out = self.bn3(self.conv3(out))
+
+        if self.downsample is not None:
+            identity = self.downsample(x)
+
+        out += identity
+        out = self.relu(out)
+        return out
+
+
+class AttentionPool2d(nn.Module):
+    def __init__(self, spacial_dim: int, embed_dim: int, num_heads: int, output_dim: int = None):
+        super().__init__()
+        self.positional_embedding = nn.Parameter(torch.randn(spacial_dim ** 2 + 1, embed_dim) / embed_dim ** 0.5)
+        self.k_proj = nn.Linear(embed_dim, embed_dim)
+        self.q_proj = nn.Linear(embed_dim, embed_dim)
+        self.v_proj = nn.Linear(embed_dim, embed_dim)
+        self.c_proj = nn.Linear(embed_dim, output_dim or embed_dim)
+        self.num_heads = num_heads
+
+    def forward(self, x):
+        x = x.reshape(x.shape[0], x.shape[1], x.shape[2] * x.shape[3]).permute(2, 0, 1)  # NCHW -> (HW)NC
+        x = torch.cat([x.mean(dim=0, keepdim=True), x], dim=0)  # (HW+1)NC
+        x = x + self.positional_embedding[:, None, :].to(x.dtype)  # (HW+1)NC
+        x, _ = F.multi_head_attention_forward(
+            query=x, key=x, value=x,
+            embed_dim_to_check=x.shape[-1],
+            num_heads=self.num_heads,
+            q_proj_weight=self.q_proj.weight,
+            k_proj_weight=self.k_proj.weight,
+            v_proj_weight=self.v_proj.weight,
+            in_proj_weight=None,
+            in_proj_bias=torch.cat([self.q_proj.bias, self.k_proj.bias, self.v_proj.bias]),
+            bias_k=None,
+            bias_v=None,
+            add_zero_attn=False,
+            dropout_p=0,
+            out_proj_weight=self.c_proj.weight,
+            out_proj_bias=self.c_proj.bias,
+            use_separate_proj_weight=True,
+            training=self.training,
+            need_weights=False
+        )
+
+        return x[0]
+
+
+class ModifiedResNet(nn.Module):
+    """
+    A ResNet class that is similar to torchvision's but contains the following changes:
+    - There are now 3 "stem" convolutions as opposed to 1, with an average pool instead of a max pool.
+    - Performs anti-aliasing strided convolutions, where an avgpool is prepended to convolutions with stride > 1
+    - The final pooling layer is a QKV attention instead of an average pool
+    """
+
+    def __init__(self, layers, output_dim, heads, input_resolution=224, width=64):
+        super().__init__()
+        self.output_dim = output_dim
+        self.input_resolution = input_resolution
+
+        # the 3-layer stem
+        self.conv1 = nn.Conv2d(3, width // 2, kernel_size=3, stride=2, padding=1, bias=False)
+        self.bn1 = nn.BatchNorm2d(width // 2)
+        self.conv2 = nn.Conv2d(width // 2, width // 2, kernel_size=3, padding=1, bias=False)
+        self.bn2 = nn.BatchNorm2d(width // 2)
+        self.conv3 = nn.Conv2d(width // 2, width, kernel_size=3, padding=1, bias=False)
+        self.bn3 = nn.BatchNorm2d(width)
+        self.avgpool = nn.AvgPool2d(2)
+        self.relu = nn.ReLU(inplace=True)
+
+        # residual layers
+        self._inplanes = width  # this is a *mutable* variable used during construction
+        self.layer1 = self._make_layer(width, layers[0])
+        self.layer2 = self._make_layer(width * 2, layers[1], stride=2)
+        self.layer3 = self._make_layer(width * 4, layers[2], stride=2)
+        self.layer4 = self._make_layer(width * 8, layers[3], stride=2)
+
+        embed_dim = width * 32  # the ResNet feature dimension
+        self.attnpool = AttentionPool2d(input_resolution // 32, embed_dim, heads, output_dim)
+
+    def _make_layer(self, planes, blocks, stride=1):
+        layers = [Bottleneck(self._inplanes, planes, stride)]
+
+        self._inplanes = planes * Bottleneck.expansion
+        for _ in range(1, blocks):
+            layers.append(Bottleneck(self._inplanes, planes))
+
+        return nn.Sequential(*layers)
+
+    def forward(self, x):
+        def stem(x):
+            for conv, bn in [(self.conv1, self.bn1), (self.conv2, self.bn2), (self.conv3, self.bn3)]:
+                x = self.relu(bn(conv(x)))
+            x = self.avgpool(x)
+            return x
+
+        x = x.type(self.conv1.weight.dtype)
+        x = stem(x)
+        x = self.layer1(x)
+        x = self.layer2(x)
+        x = self.layer3(x)
+        x = self.layer4(x)
+        x = self.attnpool(x)
+
+        return x
+
+
+class LayerNorm(nn.LayerNorm):
+    """Subclass torch's LayerNorm to handle fp16."""
+
+    def forward(self, x: torch.Tensor):
+        orig_type = x.dtype
+        ret = super().forward(x.type(torch.float32))
+        return ret.type(orig_type)
+
+
+class QuickGELU(nn.Module):
+    def forward(self, x: torch.Tensor):
+        return x * torch.sigmoid(1.702 * x)
+
+
+class ResidualAttentionBlock_ViFi_CLIP(nn.Module):
+    def __init__(self, d_model: int, n_head: int, attn_mask: torch.Tensor = None, add_prompt=False,
+                 text_layer=False, i=0, design_details=None):
+        super().__init__()
+
+        self.attn = nn.MultiheadAttention(d_model, n_head)
+        self.ln_1 = LayerNorm(d_model)
+        self.mlp = nn.Sequential(OrderedDict([
+            ("c_fc", nn.Linear(d_model, d_model * 4)),
+            ("gelu", QuickGELU()),
+            ("c_proj", nn.Linear(d_model * 4, d_model))
+        ]))
+        self.ln_2 = LayerNorm(d_model)
+        # Only add learnable tokens if flag is set True
+        # For the first iteration i, we should not add the learnable parameters
+        # as it is already been taken care of in the very start, for both text
+        # and the visual branch
+        self.text_layer = text_layer
+        self.attn_mask = attn_mask
+        if i != 0:
+            self.add_prompt = add_prompt
+            if self.add_prompt:
+                if self.text_layer:
+                    self.n_ctx_text = design_details["language_ctx"]  # hyperparameter
+                    ctx_vectors = torch.empty(self.n_ctx_text, d_model)
+                else:
+                    self.n_ctx_visual = design_details["vision_ctx"]  # hyperparameter
+                    ctx_vectors = torch.empty(self.n_ctx_visual, d_model)
+                # Code snippet for per layer visual prompts
+                nn.init.normal_(ctx_vectors, std=0.02)
+                self.VPT_shallow = nn.Parameter(ctx_vectors)
+        else:
+            self.add_prompt = False
+
+    def attention(self, x: torch.Tensor):
+        self.attn_mask = self.attn_mask.to(dtype=x.dtype, device=x.device) if self.attn_mask is not None else None
+        return self.attn(x, x, x, need_weights=False, attn_mask=self.attn_mask)[0]
+
+    def forward(self, x: torch.Tensor):
+        # Will need to append the learnable tokens for this layer here
+        # Check if flag was set for this layer or not
+        if self.add_prompt:
+            # Also see if this is textual transformer layer or not
+            if not self.text_layer:
+                # Remove the outputs produced by learnable tokens of previous layer
+                prefix = x[0:x.shape[0] - self.n_ctx_visual, :, :]
+                # Create/configure learnable tokens of this layer
+                visual_context = self.VPT_shallow.expand(x.shape[1], -1, -1).permute(1, 0, 2).half()
+                # Add the learnable tokens of this layer with the input, by replacing the previous
+                # layer learnable tokens
+                x = torch.cat([prefix, visual_context], dim=0)
+            else:
+                # Appending the learnable tokens in different way
+                # x -> [77, NCLS, DIM]
+                # First remove the learnable tokens from previous layer
+                prefix = x[:1, :, :]
+                suffix = x[1 + self.n_ctx_text:, :, :]
+                # Create/configure learnable tokens of this layer
+                textual_context = self.VPT_shallow.expand(x.shape[1], -1, -1).permute(1, 0, 2).half()
+                # Add the learnable tokens of this layer with the input, replaced by previous
+                # layer learnable tokens
+                x = torch.cat([prefix, textual_context, suffix], dim=0)
+
+        x = x + self.attention(self.ln_1(x))
+        x = x + self.mlp(self.ln_2(x))
+        return x
+
+
+class Transformer(nn.Module):
+    def __init__(self, width: int, layers: int, heads: int, attn_mask: torch.Tensor = None, prompts_needed=0,
+                 text_layer=False, design_details=None):
+        super().__init__()
+        self.width = width
+        self.layers = layers
+        self.resblocks = nn.Sequential(*[ResidualAttentionBlock_ViFi_CLIP(width, heads, attn_mask, True,
+                                                                          text_layer, i,
+                                                                          design_details) if prompts_needed > i
+                                         else ResidualAttentionBlock_ViFi_CLIP(width, heads, attn_mask, False,
+                                                                               text_layer, i, design_details)
+                                         for i in range(layers)])
+
+    def forward(self, x: torch.Tensor):
+        return self.resblocks(x)
+
+
+class VisionTransformer(nn.Module):
+    def __init__(self, input_resolution: int, patch_size: int, width: int, layers: int, heads: int,
+                 output_dim: int, design_details):
+        super().__init__()
+        self.input_resolution = input_resolution
+        self.output_dim = output_dim
+        self.conv1 = nn.Conv2d(in_channels=3, out_channels=width, kernel_size=patch_size, stride=patch_size, bias=False)
+        if design_details["vision_depth"] == 0:
+            self.VPT_shallow = False
+        else:
+            self.VPT_shallow = True
+        if self.VPT_shallow:
+            # Add visual prompt tokens here
+            n_ctx = design_details["vision_ctx"]  # hyperparameter
+            ctx_vectors = torch.empty(n_ctx, width)
+            nn.init.normal_(ctx_vectors, std=0.02)
+            self.VPT = nn.Parameter(ctx_vectors)
+            # self.VPT.half()
+        scale = width ** -0.5
+        self.class_embedding = nn.Parameter(scale * torch.randn(width))
+        self.positional_embedding = nn.Parameter(scale * torch.randn((input_resolution // patch_size) ** 2 + 1, width))
+        self.ln_pre = LayerNorm(width)
+        # hyper-parameter if need to add prompt embeddings inside to the input
+        # of transformer block or not:
+        self.prompt_till_layer_visual = design_details["vision_depth"]
+        self.transformer = Transformer(width, layers, heads, prompts_needed=self.prompt_till_layer_visual,
+                                       design_details=design_details)
+
+        self.ln_post = LayerNorm(width)
+        self.proj = nn.Parameter(scale * torch.randn(width, output_dim))
+
+    def forward(self, x: torch.Tensor):
+        x = self.conv1(x)  # shape = [*, width, grid, grid]
+        x = x.reshape(x.shape[0], x.shape[1], -1)  # shape = [*, width, grid ** 2]
+        x = x.permute(0, 2, 1)  # shape = [*, grid ** 2, width]
+        x = torch.cat(
+            [self.class_embedding.to(x.dtype) + torch.zeros(x.shape[0], 1, x.shape[-1], dtype=x.dtype, device=x.device),
+             x], dim=1)  # shape = [*, grid ** 2 + 1, width]
+        x = x + self.positional_embedding.to(x.dtype)
+        # After positional embeddings, we will attach prompts with the model, remember only those
+        # are trainable parameters here in whole image encoder.
+        if self.VPT_shallow:
+            visual_ctx = self.VPT.expand(x.shape[0], -1, -1).half()
+            x = torch.cat([x, visual_ctx], dim=1)
+        else:
+            assert self.prompt_till_layer_visual == 0
+
+        # Normal code as before
+        x = self.ln_pre(x)
+
+        x = x.permute(1, 0, 2)  # NLD -> LND
+        x = self.transformer(x)
+        x = x.permute(1, 0, 2)  # LND -> NLD
+
+        x = self.ln_post(x[:, 0, :])
+
+        if self.proj is not None:
+            x = x @ self.proj
+
+        return x
+
+
+class CLIP(nn.Module):
+    def __init__(self,
+                 embed_dim: int,
+                 # vision
+                 image_resolution: int,
+                 vision_layers: Union[Tuple[int, int, int, int], int],
+                 vision_width: int,
+                 vision_patch_size: int,
+                 # text
+                 context_length: int,
+                 vocab_size: int,
+                 transformer_width: int,
+                 transformer_heads: int,
+                 transformer_layers: int,
+                 design_details
+                 ):
+        super().__init__()
+
+        self.context_length = context_length
+        trainer = design_details['trainer']
+
+        if isinstance(vision_layers, (tuple, list)):
+            vision_heads = vision_width * 32 // 64
+            self.visual = ModifiedResNet(
+                layers=vision_layers,
+                output_dim=embed_dim,
+                heads=vision_heads,
+                input_resolution=image_resolution,
+                width=vision_width
+            )
+        else:
+            vision_heads = vision_width // 64
+            self.visual = VisionTransformer(
+                input_resolution=image_resolution,
+                patch_size=vision_patch_size,
+                width=vision_width,
+                layers=vision_layers,
+                heads=vision_heads,
+                output_dim=embed_dim,
+                design_details=design_details
+            )
+        # hyper-parameter if need to add prompt embeddings inside to the input
+        # of transformer block or not:
+        prompt_till_layer_text = design_details['language_depth']
+        self.transformer = Transformer(
+            width=transformer_width,
+            layers=transformer_layers,
+            heads=transformer_heads,
+            attn_mask=self.build_attention_mask(),
+            prompts_needed=prompt_till_layer_text,
+            text_layer=True,
+            design_details=design_details
+        )
+
+        self.vocab_size = vocab_size
+        self.token_embedding = nn.Embedding(vocab_size, transformer_width)
+        self.positional_embedding = nn.Parameter(torch.empty(self.context_length, transformer_width))
+        self.ln_final = LayerNorm(transformer_width)
+
+        self.text_projection = nn.Parameter(torch.empty(transformer_width, embed_dim))
+        self.logit_scale = nn.Parameter(torch.ones([]) * np.log(1 / 0.07))
+
+        self.initialize_parameters()
+
+    def initialize_parameters(self):
+        nn.init.normal_(self.token_embedding.weight, std=0.02)
+        nn.init.normal_(self.positional_embedding, std=0.01)
+
+        if isinstance(self.visual, ModifiedResNet):
+            if self.visual.attnpool is not None:
+                std = self.visual.attnpool.c_proj.in_features ** -0.5
+                nn.init.normal_(self.visual.attnpool.q_proj.weight, std=std)
+                nn.init.normal_(self.visual.attnpool.k_proj.weight, std=std)
+                nn.init.normal_(self.visual.attnpool.v_proj.weight, std=std)
+                nn.init.normal_(self.visual.attnpool.c_proj.weight, std=std)
+
+            for resnet_block in [self.visual.layer1, self.visual.layer2, self.visual.layer3, self.visual.layer4]:
+                for name, param in resnet_block.named_parameters():
+                    if name.endswith("bn3.weight"):
+                        nn.init.zeros_(param)
+
+        proj_std = (self.transformer.width ** -0.5) * ((2 * self.transformer.layers) ** -0.5)
+        attn_std = self.transformer.width ** -0.5
+        fc_std = (2 * self.transformer.width) ** -0.5
+        for block in self.transformer.resblocks:
+            nn.init.normal_(block.attn.in_proj_weight, std=attn_std)
+            nn.init.normal_(block.attn.out_proj.weight, std=proj_std)
+            nn.init.normal_(block.mlp.c_fc.weight, std=fc_std)
+            nn.init.normal_(block.mlp.c_proj.weight, std=proj_std)
+
+        if self.text_projection is not None:
+            nn.init.normal_(self.text_projection, std=self.transformer.width ** -0.5)
+
+    def build_attention_mask(self):
+        # lazily create causal attention mask, with full attention between the vision tokens
+        # pytorch uses additive attention mask; fill with -inf
+        mask = torch.empty(self.context_length, self.context_length)
+        mask.fill_(float("-inf"))
+        mask.triu_(1)  # zero out the lower diagonal
+        return mask
+
+    @property
+    def dtype(self):
+        return self.visual.conv1.weight.dtype
+
+    def encode_image(self, image):
+        return self.visual(image.type(self.dtype))
+
+    def encode_text(self, text):
+        x = self.token_embedding(text).type(self.dtype)  # [batch_size, n_ctx, d_model]
+
+        x = x + self.positional_embedding.type(self.dtype)
+        x = x.permute(1, 0, 2)  # NLD -> LND
+        x = self.transformer(x)
+        x = x.permute(1, 0, 2)  # LND -> NLD
+        x = self.ln_final(x).type(self.dtype)
+
+        # x.shape = [batch_size, n_ctx, transformer.width]
+        # take features from the eot embedding (eot_token is the highest number in each sequence)
+        x = x[torch.arange(x.shape[0]), text.argmax(dim=-1)] @ self.text_projection
+
+        return x
+
+    def forward(self, image, text):
+        image_features = self.encode_image(image)
+        text_features = self.encode_text(text)
+
+        # normalized features
+        image_features = image_features / image_features.norm(dim=-1, keepdim=True)
+        text_features = text_features / text_features.norm(dim=-1, keepdim=True)
+
+        # cosine similarity as logits
+        logit_scale = self.logit_scale.exp()
+        logits_per_image = logit_scale * image_features @ text_features.t()
+        logits_per_text = logit_scale * text_features @ image_features.t()
+
+        # shape = [global_batch_size, global_batch_size]
+        return logits_per_image, logits_per_text
+
+
+def convert_weights(model: nn.Module):
+    """Convert applicable model parameters to fp16"""
+
+    def _convert_weights_to_fp16(l):
+        if isinstance(l, (nn.Conv1d, nn.Conv2d, nn.Linear)):
+            l.weight.data = l.weight.data.half()
+            if l.bias is not None:
+                l.bias.data = l.bias.data.half()
+
+        if isinstance(l, nn.MultiheadAttention):
+            for attr in [*[f"{s}_proj_weight" for s in ["in", "q", "k", "v"]], "in_proj_bias", "bias_k", "bias_v"]:
+                tensor = getattr(l, attr)
+                if tensor is not None:
+                    tensor.data = tensor.data.half()
+
+        for name in ["text_projection", "proj"]:
+            if hasattr(l, name):
+                attr = getattr(l, name)
+                if attr is not None:
+                    attr.data = attr.data.half()
+
+    model.apply(_convert_weights_to_fp16)
+
+
+def build_model(state_dict: dict, design_details):
+    vit = "visual.proj" in state_dict
+
+    if vit:
+        vision_width = state_dict["visual.conv1.weight"].shape[0]
+        vision_layers = len(
+            [k for k in state_dict.keys() if k.startswith("visual.") and k.endswith(".attn.in_proj_weight")])
+        vision_patch_size = state_dict["visual.conv1.weight"].shape[-1]
+        grid_size = round((state_dict["visual.positional_embedding"].shape[0] - 1) ** 0.5)
+        image_resolution = vision_patch_size * grid_size
+    else:
+        counts: list = [len(set(k.split(".")[2] for k in state_dict if k.startswith(f"visual.layer{b}"))) for b in
+                        [1, 2, 3, 4]]
+        vision_layers = tuple(counts)
+        vision_width = state_dict["visual.layer1.0.conv1.weight"].shape[0]
+        output_width = round((state_dict["visual.attnpool.positional_embedding"].shape[0] - 1) ** 0.5)
+        vision_patch_size = None
+        assert output_width ** 2 + 1 == state_dict["visual.attnpool.positional_embedding"].shape[0]
+        image_resolution = output_width * 32
+
+    embed_dim = state_dict["text_projection"].shape[1]
+    context_length = state_dict["positional_embedding"].shape[0]
+    vocab_size = state_dict["token_embedding.weight"].shape[0]
+    transformer_width = state_dict["ln_final.weight"].shape[0]
+    transformer_heads = transformer_width // 64
+    transformer_layers = len(set(k.split(".")[2] for k in state_dict if k.startswith(f"transformer.resblocks")))
+
+    model = CLIP(
+        embed_dim,
+        image_resolution, vision_layers, vision_width, vision_patch_size,
+        context_length, vocab_size, transformer_width, transformer_heads, transformer_layers, design_details
+    )
+
+    for key in ["input_resolution", "context_length", "vocab_size"]:
+        if key in state_dict:
+            del state_dict[key]
+
+    convert_weights(model)
+    try:
+        model.load_state_dict(state_dict)
+    except:
+        missing_keys, _ = model.load_state_dict(state_dict, strict=False)
+        print('Weights not found for some missing keys: ', missing_keys)
+    return model.eval()

clip/simple_tokenizer.py

@@ -0,0 +1,132 @@
+import gzip
+import html
+import os
+from functools import lru_cache
+
+import ftfy
+import regex as re
+
+
+@lru_cache()
+def default_bpe():
+    return os.path.join(os.path.dirname(os.path.abspath(__file__)), "bpe_simple_vocab_16e6.txt.gz")
+
+
+@lru_cache()
+def bytes_to_unicode():
+    """
+    Returns list of utf-8 byte and a corresponding list of unicode strings.
+    The reversible bpe codes work on unicode strings.
+    This means you need a large # of unicode characters in your vocab if you want to avoid UNKs.
+    When you're at something like a 10B token dataset you end up needing around 5K for decent coverage.
+    This is a signficant percentage of your normal, say, 32K bpe vocab.
+    To avoid that, we want lookup tables between utf-8 bytes and unicode strings.
+    And avoids mapping to whitespace/control characters the bpe code barfs on.
+    """
+    bs = list(range(ord("!"), ord("~")+1))+list(range(ord("¡"), ord("¬")+1))+list(range(ord("®"), ord("ÿ")+1))
+    cs = bs[:]
+    n = 0
+    for b in range(2**8):
+        if b not in bs:
+            bs.append(b)
+            cs.append(2**8+n)
+            n += 1
+    cs = [chr(n) for n in cs]
+    return dict(zip(bs, cs))
+
+
+def get_pairs(word):
+    """Return set of symbol pairs in a word.
+    Word is represented as tuple of symbols (symbols being variable-length strings).
+    """
+    pairs = set()
+    prev_char = word[0]
+    for char in word[1:]:
+        pairs.add((prev_char, char))
+        prev_char = char
+    return pairs
+
+
+def basic_clean(text):
+    text = ftfy.fix_text(text)
+    text = html.unescape(html.unescape(text))
+    return text.strip()
+
+
+def whitespace_clean(text):
+    text = re.sub(r'\s+', ' ', text)
+    text = text.strip()
+    return text
+
+
+class SimpleTokenizer(object):
+    def __init__(self, bpe_path: str = default_bpe()):
+        self.byte_encoder = bytes_to_unicode()
+        self.byte_decoder = {v: k for k, v in self.byte_encoder.items()}
+        merges = gzip.open(bpe_path).read().decode("utf-8").split('\n')
+        merges = merges[1:49152-256-2+1]
+        merges = [tuple(merge.split()) for merge in merges]
+        vocab = list(bytes_to_unicode().values())
+        vocab = vocab + [v+'</w>' for v in vocab]
+        for merge in merges:
+            vocab.append(''.join(merge))
+        vocab.extend(['<|startoftext|>', '<|endoftext|>'])
+        self.encoder = dict(zip(vocab, range(len(vocab))))
+        self.decoder = {v: k for k, v in self.encoder.items()}
+        self.bpe_ranks = dict(zip(merges, range(len(merges))))
+        self.cache = {'<|startoftext|>': '<|startoftext|>', '<|endoftext|>': '<|endoftext|>'}
+        self.pat = re.compile(r"""<\|startoftext\|>|<\|endoftext\|>|'s|'t|'re|'ve|'m|'ll|'d|[\p{L}]+|[\p{N}]|[^\s\p{L}\p{N}]+""", re.IGNORECASE)
+
+    def bpe(self, token):
+        if token in self.cache:
+            return self.cache[token]
+        word = tuple(token[:-1]) + ( token[-1] + '</w>',)
+        pairs = get_pairs(word)
+
+        if not pairs:
+            return token+'</w>'
+
+        while True:
+            bigram = min(pairs, key = lambda pair: self.bpe_ranks.get(pair, float('inf')))
+            if bigram not in self.bpe_ranks:
+                break
+            first, second = bigram
+            new_word = []
+            i = 0
+            while i < len(word):
+                try:
+                    j = word.index(first, i)
+                    new_word.extend(word[i:j])
+                    i = j
+                except:
+                    new_word.extend(word[i:])
+                    break
+
+                if word[i] == first and i < len(word)-1 and word[i+1] == second:
+                    new_word.append(first+second)
+                    i += 2
+                else:
+                    new_word.append(word[i])
+                    i += 1
+            new_word = tuple(new_word)
+            word = new_word
+            if len(word) == 1:
+                break
+            else:
+                pairs = get_pairs(word)
+        word = ' '.join(word)
+        self.cache[token] = word
+        return word
+
+    def encode(self, text):
+        bpe_tokens = []
+        text = whitespace_clean(basic_clean(text)).lower()
+        for token in re.findall(self.pat, text):
+            token = ''.join(self.byte_encoder[b] for b in token.encode('utf-8'))
+            bpe_tokens.extend(self.encoder[bpe_token] for bpe_token in self.bpe(token).split(' '))
+        return bpe_tokens
+
+    def decode(self, tokens):
+        text = ''.join([self.decoder[token] for token in tokens])
+        text = bytearray([self.byte_decoder[c] for c in text]).decode('utf-8', errors="replace").replace('</w>', ' ')
+        return text

configs/config_challenge_test.yaml

@@ -0,0 +1,26 @@
+DATA:
+    ROOT: '/data/vidlab_datasets/challenge_crop/rgb'
+    TRAIN_FILE: '/data/users/sdas/scripts/ViFi-CLIP/merged_dataset.csv'
+    VAL_FILE: '/data/vidlab_datasets/challenge_crop/split/val.csv'
+    DATASET: kinetics400
+    NUM_FRAMES: 16
+    NUM_CLASSES: 45
+    LABEL_LIST: '/data/users/sdas/scripts/ViFi-CLIP/labels/challenge.csv'
+SAVE_FREQ: 5
+MODEL:
+    ARCH: ViT-B/16
+    RESUME: './work_dirs/challenge_baseline_new/best.pth'
+TRAIN:
+    BATCH_SIZE: 1 # BS 512
+    ACCUMULATION_STEPS: 64
+    EPOCHS: 30
+    LR: 2.2e-05
+TEST:
+    MULTI_VIEW_INFERENCE: False
+    NUM_CLIP: 1
+    NUM_CROP: 1
+    ONLY_TEST: True
+TRAINER:
+  ViFi_CLIP:
+    ZS_EVAL: False # Make True only during test mode to evaluate zero-shot vanilla CLIP performance
+    USE: "both" # both refers to complete fine-tuning of CLIP (text+image encoders)

configs/config_challenge_train.yaml

@@ -0,0 +1,23 @@
+DATA:
+    ROOT: ''
+    TRAIN_FILE: '/data/users/sdas/scripts/ViFi-CLIP/merged_dataset_new.csv'
+    VAL_FILE: '/data/users/sdas/scripts/ViFi-CLIP/merged_dataset_new.csv'
+    DATASET: kinetics400
+    NUM_FRAMES: 16
+    NUM_CLASSES: 45
+    LABEL_LIST: '/data/users/sdas/scripts/ViFi-CLIP/labels/challenge.csv'
+SAVE_FREQ: 30
+MODEL:
+    ARCH: ViT-B/16
+TRAIN:
+    BATCH_SIZE: 8 # BS 512
+    ACCUMULATION_STEPS: 8
+    EPOCHS: 30
+    LR: 2.2e-05
+TEST:
+    MULTI_VIEW_INFERENCE: False
+    ONLY_TEST: False
+TRAINER:
+  ViFi_CLIP:
+    ZS_EVAL: False # Make True only during test mode to evaluate zero-shot vanilla CLIP performance
+    USE: "both" # both refers to complete fine-tuning of CLIP (text+image encoders)

crop_person.py

@@ -0,0 +1,162 @@
+'''
+Imports and cropping utils
+'''
+import matplotlib.pyplot as plt
+import numpy as np
+import cv2
+
+from ultralytics import YOLO
+
+def process_frame(frame, tlc, brc, new_shape=None, slack=48, keypoints=None):
+  '''
+  Crop a frame given a bounding box. Return the cropped frame and adjusted keypoints
+  ** Arguments **
+  frame : np.ndarray
+    The frame to crop
+  tlc : tuple[int]
+    Top Left Corner of bounding box
+  brc : tuple[int]
+    Bottom Right Corner of bounding box
+  new_shape : tuple[int] (optional)
+    Shape to resize the crop to
+  slack : int (optional)
+    Padding to add to the bounding box
+  keypoints : np.ndarray (optional)
+    Keypoints associated with the frame
+  '''
+  frame_w, frame_h = frame.shape[1], frame.shape[0]
+  box_w, box_h = (brc[0] - tlc[0]), (brc[1] - tlc[1])
+
+  assert slack == 0 or slack % 2 == 0, "Slack must be divisible by 2"
+
+  # add slack to bounding box
+  tlc = (tlc[0] - slack, tlc[1] - slack)
+  brc = (brc[0] + slack, brc[1] + slack)
+
+  bbox_contained_in_frame = True
+  if (tlc[0] < 0) or (tlc[1] < 0) or (brc[0] >= frame_w) or (brc[1] >= frame_h):
+    bbox_contained_in_frame = False
+
+  # pad image if bbox extends past frame boundaries
+  if not bbox_contained_in_frame:
+    bsz = (box_h, box_h, box_w, box_w) # border size (top, bot, left, right). We can always assume top=bot and left=right
+    frame = cv2.copyMakeBorder(frame, *bsz, cv2.BORDER_CONSTANT)
+  else:
+    bsz = (0, 0, 0, 0)
+
+  # adjust top-left-corner and bottom-right-corner to match padded image
+  tlc = tlc[0] + bsz[2], tlc[1] + bsz[0]
+  brc = brc[0] + bsz[2], brc[1] + bsz[0]
+
+  frame = frame[tlc[1] : brc[1], 
+                  tlc[0] : brc[0]]
+
+  # adjust frame keypoints to match padded image
+  if keypoints is not None:
+    keypoints[:, 0] += bsz[2] - (tlc[0])
+    keypoints[:, 1] += bsz[0] - (brc[1] - box_h)
+
+  if new_shape:
+    cur_shape = frame.shape
+    frame = cv2.resize(frame, new_shape)
+
+    x_ratio, y_ratio = (new_shape[0] / cur_shape[1]), (new_shape[1] / cur_shape[0])
+
+    if keypoints is not None:
+      keypoints[:, 0] *= x_ratio
+      keypoints[:, 1] *= y_ratio
+
+  return frame, keypoints
+
+'''
+Loading the model
+'''
+# Load a pretrained YOLO model
+model = YOLO("yolo11n.pt")
+
+'''
+!!! This is the code you should change to process the ETRI videos
+This code will perform the cropping on a single video and save the processed video to a directory. This is the function you should call for each video in ETRI.
+'''
+# vid_path = '/data/vidlab_datasets/eval_FO_ids/1986.mp4'
+write_dir = '/data/vidlab_datasets/challenge_crop/' # directory to write processed videos to. filename will be the same as the original video
+write_shape = (224, 224) # shape to resize the processed video to
+
+def crop_and_save_video_yolo(vid_path, write_dir, write_shape):
+    # error logging things
+    num_frames_skipped_nohuman = 0
+
+    # load video and show first frame with matplotlib
+    cap = cv2.VideoCapture(vid_path)
+    if not cap.isOpened():
+        print(f"Failed to open the video file at {video_path}")
+    else:
+        print("Video file opened successfully")
+    w, h, frame_rate, num_frames = int(cap.get(3)), int(cap.get(4)), int(cap.get(5)), int(cap.get(7))
+
+    vid_filename = vid_path.split('/')[-1].replace('.avi', '.mp4')
+    writer = cv2.VideoWriter(f'{write_dir}/{vid_filename}', cv2.VideoWriter_fourcc(*'mp4v'), frame_rate, write_shape, True)
+
+    show_frame = False
+
+    while True:
+        ret, frame = cap.read()
+        if not ret:
+            break
+        
+        if show_frame: # debugging
+            frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
+            plt.imshow(frame)
+            break
+
+        det_results = model(frame, verbose=False)
+        class_names = det_results[0].names # e.g. {0: 'person', 1: 'bicycle', ...}
+
+        boxes = det_results[0].boxes.cpu().numpy()
+        boxes_xyxy = boxes.xyxy.astype(int) # bounding boxes as integers. shape: (n_boxes, N, 4), N is number of boxes detected
+        box_cls_ids = boxes.cls # class ids as integers
+        box_cls_names = [class_names[cls_id] for cls_id in box_cls_ids] # class names as strings
+        box_conf = boxes.conf # confidence scores
+
+        human_box_idxs = [1 if cls_name == 'person' else 0 for cls_name in box_cls_names]
+        human_box_idxs = np.array(human_box_idxs).astype(bool)
+        human_boxes = boxes_xyxy[human_box_idxs]
+
+        if human_boxes.size == 0: # no humans detected
+            print("No humans detected in frame, skipping this frame in the video.")
+            num_frames_skipped_nohuman += 1
+            continue
+
+        if human_boxes.shape[0] > 2: # more than 2 humans detected - but NTU contains 2 humans max
+            # print("More than 2 humans detected in frame, naively selecting the first 2")
+            human_boxes = human_boxes[:2]
+
+        if human_boxes.shape[0] == 2: # two humans detected - one of the NTU actions with 2 people. We combine their box into a single box that encompasses both people
+            tlc = (min(human_boxes[0, 0], human_boxes[1, 0]), min(human_boxes[0, 1], human_boxes[1, 1]))
+            brc = (max(human_boxes[0, 2], human_boxes[1, 2]), max(human_boxes[0, 3], human_boxes[1, 3]))
+        else:
+            tlc = (human_boxes[0, 0], human_boxes[0, 1])
+            brc = (human_boxes[0, 2], human_boxes[0, 3])
+
+        new_frame, _ = process_frame(frame, tlc, brc, new_shape=write_shape)
+
+        writer.write(new_frame)
+
+    print(f"Number of frames skipped ({vid_path}) due to no humans detected: {num_frames_skipped_nohuman}")
+
+    writer.release()
+    cap.release()
+   
+import os
+from pathlib import Path
+
+path = '/data/vidlab_datasets/eval_FO_ids/'
+vid_file = sorted(os.listdir(path))
+#vid_ = vid_file.reverse()
+#print(vid_)
+for video in vid_file[1:]:
+    print(Path(write_dir+video))
+    if Path(write_dir+video).exists():
+        continue
+    else:
+        crop_and_save_video_yolo(path+video, write_dir, write_shape)

datasets/blending.py

@@ -0,0 +1,214 @@
+from abc import ABCMeta, abstractmethod
+
+import torch
+import torch.nn.functional as F
+from torch.distributions.beta import Beta
+import numpy as np
+
+
+def one_hot(x, num_classes, on_value=1., off_value=0., device='cuda'):
+    x = x.long().view(-1, 1)
+    return torch.full((x.size()[0], num_classes), off_value, device=device).scatter_(1, x, on_value)
+
+
+class BaseMiniBatchBlending(metaclass=ABCMeta):
+    """Base class for Image Aliasing."""
+
+    def __init__(self, num_classes, smoothing=0.):
+        self.num_classes = num_classes
+        self.off_value = smoothing / self.num_classes
+        self.on_value = 1. - smoothing + self.off_value
+
+    @abstractmethod
+    def do_blending(self, imgs, label, **kwargs):
+        pass
+
+    def __call__(self, imgs, label, **kwargs):
+        """Blending data in a mini-batch.
+
+        Images are float tensors with the shape of (B, N, C, H, W) for 2D
+        recognizers or (B, N, C, T, H, W) for 3D recognizers.
+
+        Besides, labels are converted from hard labels to soft labels.
+        Hard labels are integer tensors with the shape of (B, 1) and all of the
+        elements are in the range [0, num_classes - 1].
+        Soft labels (probablity distribution over classes) are float tensors
+        with the shape of (B, 1, num_classes) and all of the elements are in
+        the range [0, 1].
+
+        Args:
+            imgs (torch.Tensor): Model input images, float tensor with the
+                shape of (B, N, C, H, W) or (B, N, C, T, H, W).
+            label (torch.Tensor): Hard labels, integer tensor with the shape
+                of (B, 1) and all elements are in range [0, num_classes).
+            kwargs (dict, optional): Other keyword argument to be used to
+                blending imgs and labels in a mini-batch.
+
+        Returns:
+            mixed_imgs (torch.Tensor): Blending images, float tensor with the
+                same shape of the input imgs.
+            mixed_label (torch.Tensor): Blended soft labels, float tensor with
+                the shape of (B, 1, num_classes) and all elements are in range
+                [0, 1].
+        """
+        one_hot_label = one_hot(label, num_classes=self.num_classes, on_value=self.on_value, off_value=self.off_value, device=label.device)
+
+        mixed_imgs, mixed_label = self.do_blending(imgs, one_hot_label,
+                                                   **kwargs)
+
+        return mixed_imgs, mixed_label
+
+
+class MixupBlending(BaseMiniBatchBlending):
+    """Implementing Mixup in a mini-batch.
+
+    This module is proposed in `mixup: Beyond Empirical Risk Minimization
+    <https://arxiv.org/abs/1710.09412>`_.
+    Code Reference https://github.com/open-mmlab/mmclassification/blob/master/mmcls/models/utils/mixup.py # noqa
+
+    Args:
+        num_classes (int): The number of classes.
+        alpha (float): Parameters for Beta distribution.
+    """
+
+    def __init__(self, num_classes, alpha=.2, smoothing=0.):
+        super().__init__(num_classes=num_classes, smoothing=smoothing)
+        self.beta = Beta(alpha, alpha)
+
+    def do_blending(self, imgs, label, **kwargs):
+        """Blending images with mixup."""
+        assert len(kwargs) == 0, f'unexpected kwargs for mixup {kwargs}'
+
+        lam = self.beta.sample()
+        batch_size = imgs.size(0)
+        rand_index = torch.randperm(batch_size)
+
+        mixed_imgs = lam * imgs + (1 - lam) * imgs[rand_index, :]
+        mixed_label = lam * label + (1 - lam) * label[rand_index, :]
+
+        return mixed_imgs, mixed_label
+
+
+class CutmixBlending(BaseMiniBatchBlending):
+    """Implementing Cutmix in a mini-batch.
+    This module is proposed in `CutMix: Regularization Strategy to Train Strong
+    Classifiers with Localizable Features <https://arxiv.org/abs/1905.04899>`_.
+    Code Reference https://github.com/clovaai/CutMix-PyTorch
+    Args:
+        num_classes (int): The number of classes.
+        alpha (float): Parameters for Beta distribution.
+    """
+
+    def __init__(self, num_classes, alpha=.2, smoothing=0.):
+        super().__init__(num_classes=num_classes, smoothing=smoothing)
+        self.beta = Beta(alpha, alpha)
+
+    @staticmethod
+    def rand_bbox(img_size, lam):
+        """Generate a random boudning box."""
+        w = img_size[-1]
+        h = img_size[-2]
+        cut_rat = torch.sqrt(1. - lam)
+        cut_w = torch.tensor(int(w * cut_rat))
+        cut_h = torch.tensor(int(h * cut_rat))
+
+        # uniform
+        cx = torch.randint(w, (1, ))[0]
+        cy = torch.randint(h, (1, ))[0]
+
+        bbx1 = torch.clamp(cx - cut_w // 2, 0, w)
+        bby1 = torch.clamp(cy - cut_h // 2, 0, h)
+        bbx2 = torch.clamp(cx + cut_w // 2, 0, w)
+        bby2 = torch.clamp(cy + cut_h // 2, 0, h)
+
+        return bbx1, bby1, bbx2, bby2
+
+    def do_blending(self, imgs, label, **kwargs):
+        """Blending images with cutmix."""
+        assert len(kwargs) == 0, f'unexpected kwargs for cutmix {kwargs}'
+
+        batch_size = imgs.size(0)
+        rand_index = torch.randperm(batch_size)
+        lam = self.beta.sample()
+
+        bbx1, bby1, bbx2, bby2 = self.rand_bbox(imgs.size(), lam)
+        imgs[:, ..., bby1:bby2, bbx1:bbx2] = imgs[rand_index, ..., bby1:bby2,
+                                                  bbx1:bbx2]
+        lam = 1 - (1.0 * (bbx2 - bbx1) * (bby2 - bby1) /
+                   (imgs.size()[-1] * imgs.size()[-2]))
+
+        label = lam * label + (1 - lam) * label[rand_index, :]
+
+        return imgs, label
+
+
+class LabelSmoothing(BaseMiniBatchBlending):
+    def do_blending(self, imgs, label, **kwargs):
+        return imgs, label
+
+
+class CutmixMixupBlending(BaseMiniBatchBlending):
+    def __init__(self, num_classes=400, smoothing=0.1, mixup_alpha=.8, cutmix_alpha=1, switch_prob=0.5):
+        super().__init__(num_classes=num_classes, smoothing=smoothing)
+        self.mixup_beta = Beta(mixup_alpha, mixup_alpha)
+        self.cutmix_beta = Beta(cutmix_alpha, cutmix_alpha)
+        self.switch_prob = switch_prob
+
+    @staticmethod
+    def rand_bbox(img_size, lam):
+        """Generate a random boudning box."""
+        w = img_size[-1]
+        h = img_size[-2]
+        cut_rat = torch.sqrt(1. - lam)
+        cut_w = torch.tensor(int(w * cut_rat))
+        cut_h = torch.tensor(int(h * cut_rat))
+
+        # uniform
+        cx = torch.randint(w, (1, ))[0]
+        cy = torch.randint(h, (1, ))[0]
+
+        bbx1 = torch.clamp(cx - cut_w // 2, 0, w)
+        bby1 = torch.clamp(cy - cut_h // 2, 0, h)
+        bbx2 = torch.clamp(cx + cut_w // 2, 0, w)
+        bby2 = torch.clamp(cy + cut_h // 2, 0, h)
+
+        return bbx1, bby1, bbx2, bby2
+
+    def do_cutmix(self, imgs, label, **kwargs):
+        """Blending images with cutmix."""
+        assert len(kwargs) == 0, f'unexpected kwargs for cutmix {kwargs}'
+
+        batch_size = imgs.size(0)
+        rand_index = torch.randperm(batch_size)
+        lam = self.cutmix_beta.sample()
+
+        bbx1, bby1, bbx2, bby2 = self.rand_bbox(imgs.size(), lam)
+        imgs[:, ..., bby1:bby2, bbx1:bbx2] = imgs[rand_index, ..., bby1:bby2,
+                                                  bbx1:bbx2]
+        lam = 1 - (1.0 * (bbx2 - bbx1) * (bby2 - bby1) /
+                   (imgs.size()[-1] * imgs.size()[-2]))
+
+        label = lam * label + (1 - lam) * label[rand_index, :]
+        return imgs, label
+
+    def do_mixup(self, imgs, label, **kwargs):
+        """Blending images with mixup."""
+        assert len(kwargs) == 0, f'unexpected kwargs for mixup {kwargs}'
+
+        lam = self.mixup_beta.sample()
+        batch_size = imgs.size(0)
+        rand_index = torch.randperm(batch_size)
+
+        mixed_imgs = lam * imgs + (1 - lam) * imgs[rand_index, :]
+        mixed_label = lam * label + (1 - lam) * label[rand_index, :]
+
+        return mixed_imgs, mixed_label
+
+    def do_blending(self, imgs, label, **kwargs):
+        """Blending images with MViT style. Cutmix for half for mixup for the other half."""
+        assert len(kwargs) == 0, f'unexpected kwargs for cutmix_half_mixup {kwargs}'
+
+        if np.random.rand() < self.switch_prob :
+            return self.do_cutmix(imgs, label)
+        else:
+            return self.do_mixup(imgs, label)

datasets/build.py

@@ -0,0 +1,316 @@
+from logging import Logger
+from torch.utils.data import DataLoader
+import torch.distributed as dist
+import torch
+import numpy as np
+from functools import partial
+import random
+
+import io
+import os
+import os.path as osp
+import shutil
+import warnings
+from collections.abc import Mapping, Sequence
+from mmcv.utils import Registry, build_from_cfg
+from torch.utils.data import Dataset
+import copy
+import os.path as osp
+import warnings
+from abc import ABCMeta, abstractmethod
+from collections import OrderedDict, defaultdict
+import os.path as osp
+import mmcv
+import numpy as np
+import torch
+import tarfile
+from .pipeline import *
+from torch.utils.data import DataLoader
+from torch.utils.data.dataloader import default_collate
+from mmcv.parallel import collate
+import pandas as pd
+
+PIPELINES = Registry('pipeline')
+img_norm_cfg = dict(
+    mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_bgr=False)
+
+
+class BaseDataset(Dataset, metaclass=ABCMeta):
+    def __init__(self,
+                 ann_file,
+                 pipeline,
+                 repeat = 1,
+                 data_prefix=None,
+                 test_mode=False,
+                 multi_class=False,
+                 num_classes=None,
+                 start_index=1,
+                 modality='RGB',
+                 sample_by_class=False,
+                 power=0,
+                 dynamic_length=False,):
+        super().__init__()
+        self.use_tar_format = True if ".tar" in data_prefix else False
+        data_prefix = data_prefix.replace(".tar", "")
+        self.ann_file = ann_file
+        self.repeat = repeat
+        self.data_prefix = osp.realpath(
+            data_prefix) if data_prefix is not None and osp.isdir(
+                data_prefix) else data_prefix
+        self.test_mode = test_mode
+        self.multi_class = multi_class
+        self.num_classes = num_classes
+        self.start_index = start_index
+        self.modality = modality
+        self.sample_by_class = sample_by_class
+        self.power = power
+        self.dynamic_length = dynamic_length
+
+        assert not (self.multi_class and self.sample_by_class)
+
+        self.pipeline = Compose(pipeline)
+        self.video_infos = self.load_annotations()
+        if self.sample_by_class:
+            self.video_infos_by_class = self.parse_by_class()
+
+            class_prob = []
+            for _, samples in self.video_infos_by_class.items():
+                class_prob.append(len(samples) / len(self.video_infos))
+            class_prob = [x**self.power for x in class_prob]
+
+            summ = sum(class_prob)
+            class_prob = [x / summ for x in class_prob]
+
+            self.class_prob = dict(zip(self.video_infos_by_class, class_prob))
+
+    @abstractmethod
+    def load_annotations(self):
+        """Load the annotation according to ann_file into video_infos."""
+
+    # json annotations already looks like video_infos, so for each dataset,
+    # this func should be the same
+    def load_json_annotations(self):
+        """Load json annotation file to get video information."""
+        video_infos = mmcv.load(self.ann_file)
+        num_videos = len(video_infos)
+        path_key = 'frame_dir' if 'frame_dir' in video_infos[0] else 'filename'
+        for i in range(num_videos):
+            path_value = video_infos[i][path_key]
+            if self.data_prefix is not None:
+                path_value = osp.join(self.data_prefix, path_value)
+            video_infos[i][path_key] = path_value
+            if self.multi_class:
+                assert self.num_classes is not None
+            else:
+                assert len(video_infos[i]['label']) == 1
+                video_infos[i]['label'] = video_infos[i]['label'][0]
+        return video_infos
+
+    def parse_by_class(self):
+        video_infos_by_class = defaultdict(list)
+        for item in self.video_infos:
+            label = item['label']
+            video_infos_by_class[label].append(item)
+        return video_infos_by_class
+
+    @staticmethod
+    def label2array(num, label):
+        arr = np.zeros(num, dtype=np.float32)
+        arr[label] = 1.
+        return arr
+
+    @staticmethod
+    def dump_results(results, out):
+        """Dump data to json/yaml/pickle strings or files."""
+        return mmcv.dump(results, out)
+
+    def prepare_train_frames(self, idx):
+        """Prepare the frames for training given the index."""
+        results = copy.deepcopy(self.video_infos[idx])
+        results['modality'] = self.modality
+        results['start_index'] = self.start_index
+
+        # prepare tensor in getitem
+        # If HVU, type(results['label']) is dict
+        if self.multi_class and isinstance(results['label'], list):
+            onehot = torch.zeros(self.num_classes)
+            onehot[results['label']] = 1.
+            results['label'] = onehot
+
+        aug1 = self.pipeline(results)
+        if self.repeat > 1:
+            aug2 = self.pipeline(results)
+            ret = {"imgs": torch.cat((aug1['imgs'], aug2['imgs']), 0),
+                    "label": aug1['label'].repeat(2),
+            }
+            return ret
+        else:
+            return aug1
+
+    def prepare_test_frames(self, idx):
+        """Prepare the frames for testing given the index."""
+        results = copy.deepcopy(self.video_infos[idx])
+        results['modality'] = self.modality
+        results['start_index'] = self.start_index
+
+        # prepare tensor in getitem
+        # If HVU, type(results['label']) is dict
+        if self.multi_class and isinstance(results['label'], list):
+            onehot = torch.zeros(self.num_classes)
+            onehot[results['label']] = 1.
+            results['label'] = onehot
+
+        return self.pipeline(results)
+
+    def __len__(self):
+        """Get the size of the dataset."""
+        return len(self.video_infos)
+
+    def __getitem__(self, idx):
+        """Get the sample for either training or testing given index."""
+        if self.test_mode:
+            return self.prepare_test_frames(idx)
+        
+        return self.prepare_train_frames(idx)
+
+class VideoDataset(BaseDataset):
+    def __init__(self, ann_file, pipeline, labels_file, start_index=0, **kwargs):
+        super().__init__(ann_file, pipeline, start_index=start_index, **kwargs)
+        self.labels_file = labels_file
+
+    @property
+    def classes(self):
+        classes_all = pd.read_csv(self.labels_file)
+        print(classes_all)
+        return classes_all.values.tolist()
+
+    def load_annotations(self):
+        """Load annotation file to get video information."""
+        if self.ann_file.endswith('.json'):
+            return self.load_json_annotations()
+
+        video_infos = []
+        with open(self.ann_file, 'r') as fin:
+            for line in fin:
+                line_split = line.strip().split()
+                if self.multi_class:
+                    assert self.num_classes is not None
+                    filename, label = line_split[0], line_split[1:]
+                    label = list(map(int, label))
+                else:
+                    line_split = line_split[0].split(',')
+                    filename, label = line_split
+                    label = int(label)
+                if self.data_prefix is not None:
+                    filename = osp.join(self.data_prefix, filename)
+                video_infos.append(dict(filename=filename, label=label, tar=self.use_tar_format))
+        return video_infos
+
+
+class SubsetRandomSampler(torch.utils.data.Sampler):
+    r"""Samples elements randomly from a given list of indices, without replacement.
+
+    Arguments:
+        indices (sequence): a sequence of indices
+    """
+
+    def __init__(self, indices):
+        self.epoch = 0
+        self.indices = indices
+
+    def __iter__(self):
+        return (self.indices[i] for i in torch.randperm(len(self.indices)))
+
+    def __len__(self):
+        return len(self.indices)
+
+    def set_epoch(self, epoch):
+        self.epoch = epoch
+
+
+def mmcv_collate(batch, samples_per_gpu=1): 
+    if not isinstance(batch, Sequence):
+        raise TypeError(f'{batch.dtype} is not supported.')
+    if isinstance(batch[0], Sequence):
+        transposed = zip(*batch)
+        return [collate(samples, samples_per_gpu) for samples in transposed]
+    elif isinstance(batch[0], Mapping):
+        return {
+            key: mmcv_collate([d[key] for d in batch], samples_per_gpu)
+            for key in batch[0]
+        }
+    else:
+        return default_collate(batch)
+
+
+def build_dataloader(logger, config):
+    scale_resize = int(256 / 224 * config.DATA.INPUT_SIZE)
+
+    train_pipeline = [
+        dict(type='DecordInit'),
+        dict(type='SampleFrames', clip_len=1, frame_interval=1, num_clips=config.DATA.NUM_FRAMES),
+        dict(type='DecordDecode'),
+        dict(type='Resize', scale=(-1, scale_resize)),
+        dict(
+            type='MultiScaleCrop',
+            input_size=config.DATA.INPUT_SIZE,
+            scales=(1, 0.875, 0.75, 0.66),
+            random_crop=False,
+            max_wh_scale_gap=1),
+        dict(type='Resize', scale=(config.DATA.INPUT_SIZE, config.DATA.INPUT_SIZE), keep_ratio=False),
+        dict(type='Flip', flip_ratio=0.5),
+        dict(type='ColorJitter', p=config.AUG.COLOR_JITTER),
+        dict(type='GrayScale', p=config.AUG.GRAY_SCALE),
+        dict(type='Normalize', **img_norm_cfg),
+        dict(type='FormatShape', input_format='NCHW'),
+        dict(type='Collect', keys=['imgs', 'label'], meta_keys=[]),
+        dict(type='ToTensor', keys=['imgs', 'label']),
+    ]
+        
+    
+    train_data = VideoDataset(ann_file=config.DATA.TRAIN_FILE, data_prefix=config.DATA.ROOT,
+                              labels_file=config.DATA.LABEL_LIST, pipeline=train_pipeline)
+    num_tasks = dist.get_world_size()
+    global_rank = dist.get_rank()
+    sampler_train = torch.utils.data.DistributedSampler(
+        train_data, num_replicas=num_tasks, rank=global_rank, shuffle=True
+    )
+    train_loader = DataLoader(
+        train_data, sampler=sampler_train,
+        batch_size=config.TRAIN.BATCH_SIZE,
+        num_workers=16,
+        pin_memory=True,
+        drop_last=True,
+        collate_fn=partial(mmcv_collate, samples_per_gpu=config.TRAIN.BATCH_SIZE),
+    )
+    
+    val_pipeline = [
+        dict(type='DecordInit'),
+        dict(type='SampleFrames', clip_len=1, frame_interval=1, num_clips=config.DATA.NUM_FRAMES, test_mode=True),
+        dict(type='DecordDecode'),
+        dict(type='Resize', scale=(-1, scale_resize)),
+        dict(type='CenterCrop', crop_size=config.DATA.INPUT_SIZE),
+        dict(type='Normalize', **img_norm_cfg),
+        dict(type='FormatShape', input_format='NCHW'),
+        dict(type='Collect', keys=['imgs', 'label'], meta_keys=[]),
+        dict(type='ToTensor', keys=['imgs'])
+    ]
+    if config.TEST.NUM_CROP == 3:
+        val_pipeline[3] = dict(type='Resize', scale=(-1, config.DATA.INPUT_SIZE))
+        val_pipeline[4] = dict(type='ThreeCrop', crop_size=config.DATA.INPUT_SIZE)
+    if config.TEST.NUM_CLIP > 1:
+        val_pipeline[1] = dict(type='SampleFrames', clip_len=1, frame_interval=1, num_clips=config.DATA.NUM_FRAMES, multiview=config.TEST.NUM_CLIP)
+    
+    val_data = VideoDataset(ann_file=config.DATA.VAL_FILE, data_prefix=config.DATA.ROOT, labels_file=config.DATA.LABEL_LIST, pipeline=val_pipeline)
+    indices = np.arange(dist.get_rank(), len(val_data), dist.get_world_size())
+    sampler_val = SubsetRandomSampler(indices)
+    val_loader = DataLoader(
+        val_data, sampler=indices,
+        batch_size=2,
+        num_workers=16,
+        pin_memory=True,
+        drop_last=True,
+        collate_fn=partial(mmcv_collate, samples_per_gpu=2),
+    )
+
+    return train_data, val_data, train_loader, val_loader

datasets/pipeline.py

@@ -0,0 +1,2362 @@
+import io
+import os
+import os.path as osp
+import shutil
+import warnings
+from collections.abc import Sequence
+from mmcv.utils import Registry, build_from_cfg
+from torch.utils.data import Dataset
+import copy
+import os.path as osp
+import warnings
+from abc import ABCMeta, abstractmethod
+from collections import OrderedDict, defaultdict
+import os.path as osp
+import mmcv
+import numpy as np
+import torch
+import tarfile
+import timm.data as tdata
+from torch.nn.modules.utils import _pair
+import random
+import torchvision
+from PIL import Image
+from .rand_augment import rand_augment_transform
+from torchvision import transforms
+from mmcv.fileio import FileClient
+
+PIPELINES = Registry('pipeline')
+
+def _init_lazy_if_proper(results, lazy):
+    """Initialize lazy operation properly.
+
+    Make sure that a lazy operation is properly initialized,
+    and avoid a non-lazy operation accidentally getting mixed in.
+
+    Required keys in results are "imgs" if "img_shape" not in results,
+    otherwise, Required keys in results are "img_shape", add or modified keys
+    are "img_shape", "lazy".
+    Add or modified keys in "lazy" are "original_shape", "crop_bbox", "flip",
+    "flip_direction", "interpolation".
+
+    Args:
+        results (dict): A dict stores data pipeline result.
+        lazy (bool): Determine whether to apply lazy operation. Default: False.
+    """
+
+    if 'img_shape' not in results:
+        results['img_shape'] = results['imgs'][0].shape[:2]
+    if lazy:
+        if 'lazy' not in results:
+            img_h, img_w = results['img_shape']
+            lazyop = dict()
+            lazyop['original_shape'] = results['img_shape']
+            lazyop['crop_bbox'] = np.array([0, 0, img_w, img_h],
+                                           dtype=np.float32)
+            lazyop['flip'] = False
+            lazyop['flip_direction'] = None
+            lazyop['interpolation'] = None
+            results['lazy'] = lazyop
+    else:
+        assert 'lazy' not in results, 'Use Fuse after lazy operations'
+
+
+def _pil_interp(method):
+    if method == "bicubic":
+        return Image.BICUBIC
+    elif method == "lanczos":
+        return Image.LANCZOS
+    elif method == "hamming":
+        return Image.HAMMING
+    else:
+        return Image.BILINEAR
+
+
+class EntityBoxRescale:
+
+    def __init__(self, scale_factor):
+        raise NotImplementedError(
+            'This component should not be used in the '
+            'data pipeline and is removed in PR #782. Details see '
+            'https://github.com/open-mmlab/mmaction2/pull/782')
+
+
+@PIPELINES.register_module()
+class EntityBoxCrop:
+
+    def __init__(self, crop_bbox):
+        raise NotImplementedError(
+            'This component should not be used in the '
+            'data pipeline and is removed in PR #782. Details see '
+            'https://github.com/open-mmlab/mmaction2/pull/782')
+
+
+@PIPELINES.register_module()
+class EntityBoxFlip:
+
+    def __init__(self, img_shape):
+        raise NotImplementedError(
+            'This component should not be used in the '
+            'data pipeline and is removed in PR #782. Details see '
+            'https://github.com/open-mmlab/mmaction2/pull/782')
+
+
+@PIPELINES.register_module()
+class Imgaug:
+    """Imgaug augmentation.
+    Adds custom transformations from imgaug library.
+    Please visit `https://imgaug.readthedocs.io/en/latest/index.html`
+    to get more information. Two demo configs could be found in tsn and i3d
+    config folder.
+    It's better to use uint8 images as inputs since imgaug works best with
+    numpy dtype uint8 and isn't well tested with other dtypes. It should be
+    noted that not all of the augmenters have the same input and output dtype,
+    which may cause unexpected results.
+    Required keys are "imgs", "img_shape"(if "gt_bboxes" is not None) and
+    "modality", added or modified keys are "imgs", "img_shape", "gt_bboxes"
+    and "proposals".
+    It is worth mentioning that `Imgaug` will NOT create custom keys like
+    "interpolation", "crop_bbox", "flip_direction", etc. So when using
+    `Imgaug` along with other mmaction2 pipelines, we should pay more attention
+    to required keys.
+    Two steps to use `Imgaug` pipeline:
+    1. Create initialization parameter `transforms`. There are three ways
+        to create `transforms`.
+        1) string: only support `default` for now.
+            e.g. `transforms='default'`
+        2) list[dict]: create a list of augmenters by a list of dicts, each
+            dict corresponds to one augmenter. Every dict MUST contain a key
+            named `type`. `type` should be a string(iaa.Augmenter's name) or
+            an iaa.Augmenter subclass.
+            e.g. `transforms=[dict(type='Rotate', rotate=(-20, 20))]`
+            e.g. `transforms=[dict(type=iaa.Rotate, rotate=(-20, 20))]`
+        3) iaa.Augmenter: create an imgaug.Augmenter object.
+            e.g. `transforms=iaa.Rotate(rotate=(-20, 20))`
+    2. Add `Imgaug` in dataset pipeline. It is recommended to insert imgaug
+        pipeline before `Normalize`. A demo pipeline is listed as follows.
+        ```
+        pipeline = [
+            dict(
+                type='SampleFrames',
+                clip_len=1,
+                frame_interval=1,
+                num_clips=16,
+            ),
+            dict(type='RawFrameDecode'),
+            dict(type='Resize', scale=(-1, 256)),
+            dict(
+                type='MultiScaleCrop',
+                input_size=224,
+                scales=(1, 0.875, 0.75, 0.66),
+                random_crop=False,
+                max_wh_scale_gap=1,
+                num_fixed_crops=13),
+            dict(type='Resize', scale=(224, 224), keep_ratio=False),
+            dict(type='Flip', flip_ratio=0.5),
+            dict(type='Imgaug', transforms='default'),
+            # dict(type='Imgaug', transforms=[
+            #     dict(type='Rotate', rotate=(-20, 20))
+            # ]),
+            dict(type='Normalize', **img_norm_cfg),
+            dict(type='FormatShape', input_format='NCHW'),
+            dict(type='Collect', keys=['imgs', 'label'], meta_keys=[]),
+            dict(type='ToTensor', keys=['imgs', 'label'])
+        ]
+        ```
+    Args:
+        transforms (str | list[dict] | :obj:`iaa.Augmenter`): Three different
+            ways to create imgaug augmenter.
+    """
+
+    def __init__(self, transforms):
+        import imgaug.augmenters as iaa
+
+        if transforms == 'default':
+            self.transforms = self.default_transforms()
+        elif isinstance(transforms, list):
+            assert all(isinstance(trans, dict) for trans in transforms)
+            self.transforms = transforms
+        elif isinstance(transforms, iaa.Augmenter):
+            self.aug = self.transforms = transforms
+        else:
+            raise ValueError('transforms must be `default` or a list of dicts'
+                             ' or iaa.Augmenter object')
+
+        if not isinstance(transforms, iaa.Augmenter):
+            self.aug = iaa.Sequential(
+                [self.imgaug_builder(t) for t in self.transforms])
+
+    @staticmethod
+    def default_transforms():
+        """Default transforms for imgaug.
+        Implement RandAugment by imgaug.
+        Plase visit `https://arxiv.org/abs/1909.13719` for more information.
+        Augmenters and hyper parameters are borrowed from the following repo:
+        https://github.com/tensorflow/tpu/blob/master/models/official/efficientnet/autoaugment.py # noqa
+        Miss one augmenter ``SolarizeAdd`` since imgaug doesn't support this.
+        Returns:
+            dict: The constructed RandAugment transforms.
+        """
+        # RandAugment hyper params
+        num_augmenters = 2
+        cur_magnitude, max_magnitude = 9, 10
+        cur_level = 1.0 * cur_magnitude / max_magnitude
+
+        return [
+            dict(
+                type='SomeOf',
+                n=num_augmenters,
+                children=[
+                    dict(
+                        type='ShearX',
+                        shear=17.19 * cur_level * random.choice([-1, 1])),
+                    dict(
+                        type='ShearY',
+                        shear=17.19 * cur_level * random.choice([-1, 1])),
+                    dict(
+                        type='TranslateX',
+                        percent=.2 * cur_level * random.choice([-1, 1])),
+                    dict(
+                        type='TranslateY',
+                        percent=.2 * cur_level * random.choice([-1, 1])),
+                    dict(
+                        type='Rotate',
+                        rotate=30 * cur_level * random.choice([-1, 1])),
+                    dict(type='Posterize', nb_bits=max(1, int(4 * cur_level))),
+                    dict(type='Solarize', threshold=256 * cur_level),
+                    dict(type='EnhanceColor', factor=1.8 * cur_level + .1),
+                    dict(type='EnhanceContrast', factor=1.8 * cur_level + .1),
+                    dict(
+                        type='EnhanceBrightness', factor=1.8 * cur_level + .1),
+                    dict(type='EnhanceSharpness', factor=1.8 * cur_level + .1),
+                    dict(type='Autocontrast', cutoff=0),
+                    dict(type='Equalize'),
+                    dict(type='Invert', p=1.),
+                    dict(
+                        type='Cutout',
+                        nb_iterations=1,
+                        size=0.2 * cur_level,
+                        squared=True)
+                ])
+        ]
+
+    def imgaug_builder(self, cfg):
+        """Import a module from imgaug.
+        It follows the logic of :func:`build_from_cfg`. Use a dict object to
+        create an iaa.Augmenter object.
+        Args:
+            cfg (dict): Config dict. It should at least contain the key "type".
+        Returns:
+            obj:`iaa.Augmenter`: The constructed imgaug augmenter.
+        """
+        import imgaug.augmenters as iaa
+
+        assert isinstance(cfg, dict) and 'type' in cfg
+        args = cfg.copy()
+
+        obj_type = args.pop('type')
+        if mmcv.is_str(obj_type):
+            obj_cls = getattr(iaa, obj_type) if hasattr(iaa, obj_type) \
+                else getattr(iaa.pillike, obj_type)
+        elif issubclass(obj_type, iaa.Augmenter):
+            obj_cls = obj_type
+        else:
+            raise TypeError(
+                f'type must be a str or valid type, but got {type(obj_type)}')
+
+        if 'children' in args:
+            args['children'] = [
+                self.imgaug_builder(child) for child in args['children']
+            ]
+
+        return obj_cls(**args)
+
+    def __repr__(self):
+        repr_str = self.__class__.__name__ + f'(transforms={self.aug})'
+        return repr_str
+
+    def __call__(self, results):
+        assert results['modality'] == 'RGB', 'Imgaug only support RGB images.'
+        in_type = results['imgs'][0].dtype.type
+
+        cur_aug = self.aug.to_deterministic()
+
+        results['imgs'] = [
+            cur_aug.augment_image(frame) for frame in results['imgs']
+        ]
+        img_h, img_w, _ = results['imgs'][0].shape
+
+        out_type = results['imgs'][0].dtype.type
+        assert in_type == out_type, \
+            ('Imgaug input dtype and output dtype are not the same. ',
+             f'Convert from {in_type} to {out_type}')
+
+        if 'gt_bboxes' in results:
+            from imgaug.augmentables import bbs
+            bbox_list = [
+                bbs.BoundingBox(
+                    x1=bbox[0], y1=bbox[1], x2=bbox[2], y2=bbox[3])
+                for bbox in results['gt_bboxes']
+            ]
+            bboxes = bbs.BoundingBoxesOnImage(
+                bbox_list, shape=results['img_shape'])
+            bbox_aug, *_ = cur_aug.augment_bounding_boxes([bboxes])
+            results['gt_bboxes'] = [[
+                max(bbox.x1, 0),
+                max(bbox.y1, 0),
+                min(bbox.x2, img_w),
+                min(bbox.y2, img_h)
+            ] for bbox in bbox_aug.items]
+            if 'proposals' in results:
+                bbox_list = [
+                    bbs.BoundingBox(
+                        x1=bbox[0], y1=bbox[1], x2=bbox[2], y2=bbox[3])
+                    for bbox in results['proposals']
+                ]
+                bboxes = bbs.BoundingBoxesOnImage(
+                    bbox_list, shape=results['img_shape'])
+                bbox_aug, *_ = cur_aug.augment_bounding_boxes([bboxes])
+                results['proposals'] = [[
+                    max(bbox.x1, 0),
+                    max(bbox.y1, 0),
+                    min(bbox.x2, img_w),
+                    min(bbox.y2, img_h)
+                ] for bbox in bbox_aug.items]
+
+        results['img_shape'] = (img_h, img_w)
+
+        return results
+
+@PIPELINES.register_module()
+class RandomErasing(tdata.random_erasing.RandomErasing):
+    def __init__(self, device='cpu', **args):
+        super().__init__(device=device, **args)
+
+    def __call__(self, results):
+        in_type = results['imgs'][0].dtype.type
+
+        rand_state = random.getstate()
+        torchrand_state = torch.get_rng_state()
+        numpyrand_state = np.random.get_state()
+        # not using cuda to preserve the determiness
+
+        out_frame = []
+        for frame in results['imgs']:
+            random.setstate(rand_state)
+            torch.set_rng_state(torchrand_state)
+            np.random.set_state(numpyrand_state)
+            frame = super().__call__(torch.from_numpy(frame).permute(2, 0, 1)).permute(1, 2, 0).numpy()
+            out_frame.append(frame)
+
+        results['imgs'] = out_frame
+        img_h, img_w, _ = results['imgs'][0].shape
+
+        out_type = results['imgs'][0].dtype.type
+        assert in_type == out_type, \
+            ('Timmaug input dtype and output dtype are not the same. ',
+             f'Convert from {in_type} to {out_type}')
+
+        if 'gt_bboxes' in results:
+            raise NotImplementedError('only support recognition now')
+        assert results['img_shape'] == (img_h, img_w)
+
+        return results
+
+
+@PIPELINES.register_module()
+class Fuse:
+    """Fuse lazy operations.
+
+    Fusion order:
+        crop -> resize -> flip
+
+    Required keys are "imgs", "img_shape" and "lazy", added or modified keys
+    are "imgs", "lazy".
+    Required keys in "lazy" are "crop_bbox", "interpolation", "flip_direction".
+    """
+
+    def __call__(self, results):
+        if 'lazy' not in results:
+            raise ValueError('No lazy operation detected')
+        lazyop = results['lazy']
+        imgs = results['imgs']
+
+        # crop
+        left, top, right, bottom = lazyop['crop_bbox'].round().astype(int)
+        imgs = [img[top:bottom, left:right] for img in imgs]
+
+        # resize
+        img_h, img_w = results['img_shape']
+        if lazyop['interpolation'] is None:
+            interpolation = 'bilinear'
+        else:
+            interpolation = lazyop['interpolation']
+        imgs = [
+            mmcv.imresize(img, (img_w, img_h), interpolation=interpolation)
+            for img in imgs
+        ]
+
+        # flip
+        if lazyop['flip']:
+            for img in imgs:
+                mmcv.imflip_(img, lazyop['flip_direction'])
+
+        results['imgs'] = imgs
+        del results['lazy']
+
+        return results
+
+
+@PIPELINES.register_module()
+class RandomScale:
+    """Resize images by a random scale.
+
+    Required keys are "imgs", "img_shape", "modality", added or modified
+    keys are "imgs", "img_shape", "keep_ratio", "scale_factor", "lazy",
+    "scale", "resize_size". Required keys in "lazy" is None, added or
+    modified key is "interpolation".
+
+    Args:
+        scales (tuple[int]): Tuple of scales to be chosen for resize.
+        mode (str): Selection mode for choosing the scale. Options are "range"
+            and "value". If set to "range", The short edge will be randomly
+            chosen from the range of minimum and maximum on the shorter one
+            in all tuples. Otherwise, the longer edge will be randomly chosen
+            from the range of minimum and maximum on the longer one in all
+            tuples. Default: 'range'.
+    """
+
+    def __init__(self, scales, mode='range', **kwargs):
+        warnings.warn('"RandomScale" is deprecated and will be removed in '
+                      'later versions. It is currently not used in MMAction2')
+        self.mode = mode
+        if self.mode not in ['range', 'value']:
+            raise ValueError(f"mode should be 'range' or 'value', "
+                             f'but got {self.mode}')
+        self.scales = scales
+        self.kwargs = kwargs
+
+    def select_scale(self, scales):
+        num_scales = len(scales)
+        if num_scales == 1:
+            # specify a fixed scale
+            scale = scales[0]
+        elif num_scales == 2:
+            if self.mode == 'range':
+                scale_long = [max(s) for s in scales]
+                scale_short = [min(s) for s in scales]
+                long_edge = np.random.randint(
+                    min(scale_long),
+                    max(scale_long) + 1)
+                short_edge = np.random.randint(
+                    min(scale_short),
+                    max(scale_short) + 1)
+                scale = (long_edge, short_edge)
+            elif self.mode == 'value':
+                scale = random.choice(scales)
+        else:
+            if self.mode != 'value':
+                raise ValueError("Only 'value' mode supports more than "
+                                 '2 image scales')
+            scale = random.choice(scales)
+
+        return scale
+
+    def __call__(self, results):
+        scale = self.select_scale(self.scales)
+        results['scale'] = scale
+        resize = Resize(scale, **self.kwargs)
+        results = resize(results)
+        return results
+
+    def __repr__(self):
+        repr_str = (f'{self.__class__.__name__}('
+                    f'scales={self.scales}, mode={self.mode})')
+        return repr_str
+
+
+@PIPELINES.register_module()
+class RandomCrop:
+    """Vanilla square random crop that specifics the output size.
+
+    Required keys in results are "img_shape", "keypoint" (optional), "imgs"
+    (optional), added or modified keys are "keypoint", "imgs", "lazy"; Required
+    keys in "lazy" are "flip", "crop_bbox", added or modified key is
+    "crop_bbox".
+
+    Args:
+        size (int): The output size of the images.
+        lazy (bool): Determine whether to apply lazy operation. Default: False.
+    """
+
+    def __init__(self, size, lazy=False):
+        if not isinstance(size, int):
+            raise TypeError(f'Size must be an int, but got {type(size)}')
+        self.size = size
+        self.lazy = lazy
+
+    @staticmethod
+    def _crop_kps(kps, crop_bbox):
+        return kps - crop_bbox[:2]
+
+    @staticmethod
+    def _crop_imgs(imgs, crop_bbox):
+        x1, y1, x2, y2 = crop_bbox
+        return [img[y1:y2, x1:x2] for img in imgs]
+
+    @staticmethod
+    def _box_crop(box, crop_bbox):
+        """Crop the bounding boxes according to the crop_bbox.
+
+        Args:
+            box (np.ndarray): The bounding boxes.
+            crop_bbox(np.ndarray): The bbox used to crop the original image.
+        """
+
+        x1, y1, x2, y2 = crop_bbox
+        img_w, img_h = x2 - x1, y2 - y1
+
+        box_ = box.copy()
+        box_[..., 0::2] = np.clip(box[..., 0::2] - x1, 0, img_w - 1)
+        box_[..., 1::2] = np.clip(box[..., 1::2] - y1, 0, img_h - 1)
+        return box_
+
+    def _all_box_crop(self, results, crop_bbox):
+        """Crop the gt_bboxes and proposals in results according to crop_bbox.
+
+        Args:
+            results (dict): All information about the sample, which contain
+                'gt_bboxes' and 'proposals' (optional).
+            crop_bbox(np.ndarray): The bbox used to crop the original image.
+        """
+        results['gt_bboxes'] = self._box_crop(results['gt_bboxes'], crop_bbox)
+        if 'proposals' in results and results['proposals'] is not None:
+            assert results['proposals'].shape[1] == 4
+            results['proposals'] = self._box_crop(results['proposals'],
+                                                  crop_bbox)
+        return results
+
+    def __call__(self, results):
+        """Performs the RandomCrop augmentation.
+
+        Args:
+            results (dict): The resulting dict to be modified and passed
+                to the next transform in pipeline.
+        """
+        _init_lazy_if_proper(results, self.lazy)
+        if 'keypoint' in results:
+            assert not self.lazy, ('Keypoint Augmentations are not compatible '
+                                   'with lazy == True')
+
+        img_h, img_w = results['img_shape']
+        assert self.size <= img_h and self.size <= img_w
+
+        y_offset = 0
+        x_offset = 0
+        if img_h > self.size:
+            y_offset = int(np.random.randint(0, img_h - self.size))
+        if img_w > self.size:
+            x_offset = int(np.random.randint(0, img_w - self.size))
+
+        if 'crop_quadruple' not in results:
+            results['crop_quadruple'] = np.array(
+                [0, 0, 1, 1],  # x, y, w, h
+                dtype=np.float32)
+
+        x_ratio, y_ratio = x_offset / img_w, y_offset / img_h
+        w_ratio, h_ratio = self.size / img_w, self.size / img_h
+
+        old_crop_quadruple = results['crop_quadruple']
+        old_x_ratio, old_y_ratio = old_crop_quadruple[0], old_crop_quadruple[1]
+        old_w_ratio, old_h_ratio = old_crop_quadruple[2], old_crop_quadruple[3]
+        new_crop_quadruple = [
+            old_x_ratio + x_ratio * old_w_ratio,
+            old_y_ratio + y_ratio * old_h_ratio, w_ratio * old_w_ratio,
+            h_ratio * old_x_ratio
+        ]
+        results['crop_quadruple'] = np.array(
+            new_crop_quadruple, dtype=np.float32)
+
+        new_h, new_w = self.size, self.size
+
+        crop_bbox = np.array(
+            [x_offset, y_offset, x_offset + new_w, y_offset + new_h])
+        results['crop_bbox'] = crop_bbox
+
+        results['img_shape'] = (new_h, new_w)
+
+        if not self.lazy:
+            if 'keypoint' in results:
+                results['keypoint'] = self._crop_kps(results['keypoint'],
+                                                     crop_bbox)
+            if 'imgs' in results:
+                results['imgs'] = self._crop_imgs(results['imgs'], crop_bbox)
+        else:
+            lazyop = results['lazy']
+            if lazyop['flip']:
+                raise NotImplementedError('Put Flip at last for now')
+
+            # record crop_bbox in lazyop dict to ensure only crop once in Fuse
+            lazy_left, lazy_top, lazy_right, lazy_bottom = lazyop['crop_bbox']
+            left = x_offset * (lazy_right - lazy_left) / img_w
+            right = (x_offset + new_w) * (lazy_right - lazy_left) / img_w
+            top = y_offset * (lazy_bottom - lazy_top) / img_h
+            bottom = (y_offset + new_h) * (lazy_bottom - lazy_top) / img_h
+            lazyop['crop_bbox'] = np.array([(lazy_left + left),
+                                            (lazy_top + top),
+                                            (lazy_left + right),
+                                            (lazy_top + bottom)],
+                                           dtype=np.float32)
+
+        # Process entity boxes
+        if 'gt_bboxes' in results:
+            assert not self.lazy
+            results = self._all_box_crop(results, results['crop_bbox'])
+
+        return results
+
+    def __repr__(self):
+        repr_str = (f'{self.__class__.__name__}(size={self.size}, '
+                    f'lazy={self.lazy})')
+        return repr_str
+
+
+@PIPELINES.register_module()
+class RandomResizedCrop(RandomCrop):
+    """Random crop that specifics the area and height-weight ratio range.
+
+    Required keys in results are "img_shape", "crop_bbox", "imgs" (optional),
+    "keypoint" (optional), added or modified keys are "imgs", "keypoint",
+    "crop_bbox" and "lazy"; Required keys in "lazy" are "flip", "crop_bbox",
+    added or modified key is "crop_bbox".
+
+    Args:
+        area_range (Tuple[float]): The candidate area scales range of
+            output cropped images. Default: (0.08, 1.0).
+        aspect_ratio_range (Tuple[float]): The candidate aspect ratio range of
+            output cropped images. Default: (3 / 4, 4 / 3).
+        lazy (bool): Determine whether to apply lazy operation. Default: False.
+    """
+
+    def __init__(self,
+                 area_range=(0.08, 1.0),
+                 aspect_ratio_range=(3 / 4, 4 / 3),
+                 lazy=False):
+        self.area_range = area_range
+        self.aspect_ratio_range = aspect_ratio_range
+        self.lazy = lazy
+        if not mmcv.is_tuple_of(self.area_range, float):
+            raise TypeError(f'Area_range must be a tuple of float, '
+                            f'but got {type(area_range)}')
+        if not mmcv.is_tuple_of(self.aspect_ratio_range, float):
+            raise TypeError(f'Aspect_ratio_range must be a tuple of float, '
+                            f'but got {type(aspect_ratio_range)}')
+
+    @staticmethod
+    def get_crop_bbox(img_shape,
+                      area_range,
+                      aspect_ratio_range,
+                      max_attempts=10):
+        """Get a crop bbox given the area range and aspect ratio range.
+
+        Args:
+            img_shape (Tuple[int]): Image shape
+            area_range (Tuple[float]): The candidate area scales range of
+                output cropped images. Default: (0.08, 1.0).
+            aspect_ratio_range (Tuple[float]): The candidate aspect
+                ratio range of output cropped images. Default: (3 / 4, 4 / 3).
+                max_attempts (int): The maximum of attempts. Default: 10.
+            max_attempts (int): Max attempts times to generate random candidate
+                bounding box. If it doesn't qualified one, the center bounding
+                box will be used.
+        Returns:
+            (list[int]) A random crop bbox within the area range and aspect
+            ratio range.
+        """
+        assert 0 < area_range[0] <= area_range[1] <= 1
+        assert 0 < aspect_ratio_range[0] <= aspect_ratio_range[1]
+
+        img_h, img_w = img_shape
+        area = img_h * img_w
+
+        min_ar, max_ar = aspect_ratio_range
+        aspect_ratios = np.exp(
+            np.random.uniform(
+                np.log(min_ar), np.log(max_ar), size=max_attempts))
+        target_areas = np.random.uniform(*area_range, size=max_attempts) * area
+        candidate_crop_w = np.round(np.sqrt(target_areas *
+                                            aspect_ratios)).astype(np.int32)
+        candidate_crop_h = np.round(np.sqrt(target_areas /
+                                            aspect_ratios)).astype(np.int32)
+
+        for i in range(max_attempts):
+            crop_w = candidate_crop_w[i]
+            crop_h = candidate_crop_h[i]
+            if crop_h <= img_h and crop_w <= img_w:
+                x_offset = random.randint(0, img_w - crop_w)
+                y_offset = random.randint(0, img_h - crop_h)
+                return x_offset, y_offset, x_offset + crop_w, y_offset + crop_h
+
+        # Fallback
+        crop_size = min(img_h, img_w)
+        x_offset = (img_w - crop_size) // 2
+        y_offset = (img_h - crop_size) // 2
+        return x_offset, y_offset, x_offset + crop_size, y_offset + crop_size
+
+    def __call__(self, results):
+        """Performs the RandomResizeCrop augmentation.
+
+        Args:
+            results (dict): The resulting dict to be modified and passed
+                to the next transform in pipeline.
+        """
+        _init_lazy_if_proper(results, self.lazy)
+        if 'keypoint' in results:
+            assert not self.lazy, ('Keypoint Augmentations are not compatible '
+                                   'with lazy == True')
+
+        img_h, img_w = results['img_shape']
+
+        left, top, right, bottom = self.get_crop_bbox(
+            (img_h, img_w), self.area_range, self.aspect_ratio_range)
+        new_h, new_w = bottom - top, right - left
+
+        if 'crop_quadruple' not in results:
+            results['crop_quadruple'] = np.array(
+                [0, 0, 1, 1],  # x, y, w, h
+                dtype=np.float32)
+
+        x_ratio, y_ratio = left / img_w, top / img_h
+        w_ratio, h_ratio = new_w / img_w, new_h / img_h
+
+        old_crop_quadruple = results['crop_quadruple']
+        old_x_ratio, old_y_ratio = old_crop_quadruple[0], old_crop_quadruple[1]
+        old_w_ratio, old_h_ratio = old_crop_quadruple[2], old_crop_quadruple[3]
+        new_crop_quadruple = [
+            old_x_ratio + x_ratio * old_w_ratio,
+            old_y_ratio + y_ratio * old_h_ratio, w_ratio * old_w_ratio,
+            h_ratio * old_x_ratio
+        ]
+        results['crop_quadruple'] = np.array(
+            new_crop_quadruple, dtype=np.float32)
+
+        crop_bbox = np.array([left, top, right, bottom])
+        results['crop_bbox'] = crop_bbox
+        results['img_shape'] = (new_h, new_w)
+
+        if not self.lazy:
+            if 'keypoint' in results:
+                results['keypoint'] = self._crop_kps(results['keypoint'],
+                                                     crop_bbox)
+            if 'imgs' in results:
+                results['imgs'] = self._crop_imgs(results['imgs'], crop_bbox)
+        else:
+            lazyop = results['lazy']
+            if lazyop['flip']:
+                raise NotImplementedError('Put Flip at last for now')
+
+            # record crop_bbox in lazyop dict to ensure only crop once in Fuse
+            lazy_left, lazy_top, lazy_right, lazy_bottom = lazyop['crop_bbox']
+            left = left * (lazy_right - lazy_left) / img_w
+            right = right * (lazy_right - lazy_left) / img_w
+            top = top * (lazy_bottom - lazy_top) / img_h
+            bottom = bottom * (lazy_bottom - lazy_top) / img_h
+            lazyop['crop_bbox'] = np.array([(lazy_left + left),
+                                            (lazy_top + top),
+                                            (lazy_left + right),
+                                            (lazy_top + bottom)],
+                                           dtype=np.float32)
+
+        if 'gt_bboxes' in results:
+            assert not self.lazy
+            results = self._all_box_crop(results, results['crop_bbox'])
+
+        return results
+
+    def __repr__(self):
+        repr_str = (f'{self.__class__.__name__}('
+                    f'area_range={self.area_range}, '
+                    f'aspect_ratio_range={self.aspect_ratio_range}, '
+                    f'lazy={self.lazy})')
+        return repr_str
+
+
+@PIPELINES.register_module()
+class MultiScaleCrop(RandomCrop):
+    """Crop images with a list of randomly selected scales.
+
+    Randomly select the w and h scales from a list of scales. Scale of 1 means
+    the base size, which is the minimal of image width and height. The scale
+    level of w and h is controlled to be smaller than a certain value to
+    prevent too large or small aspect ratio.
+
+    Required keys are "img_shape", "imgs" (optional), "keypoint" (optional),
+    added or modified keys are "imgs", "crop_bbox", "img_shape", "lazy" and
+    "scales". Required keys in "lazy" are "crop_bbox", added or modified key is
+    "crop_bbox".
+
+    Args:
+        input_size (int | tuple[int]): (w, h) of network input.
+        scales (tuple[float]): width and height scales to be selected.
+        max_wh_scale_gap (int): Maximum gap of w and h scale levels.
+            Default: 1.
+        random_crop (bool): If set to True, the cropping bbox will be randomly
+            sampled, otherwise it will be sampler from fixed regions.
+            Default: False.
+        num_fixed_crops (int): If set to 5, the cropping bbox will keep 5
+            basic fixed regions: "upper left", "upper right", "lower left",
+            "lower right", "center". If set to 13, the cropping bbox will
+            append another 8 fix regions: "center left", "center right",
+            "lower center", "upper center", "upper left quarter",
+            "upper right quarter", "lower left quarter", "lower right quarter".
+            Default: 5.
+        lazy (bool): Determine whether to apply lazy operation. Default: False.
+    """
+
+    def __init__(self,
+                 input_size,
+                 scales=(1, ),
+                 max_wh_scale_gap=1,
+                 random_crop=False,
+                 num_fixed_crops=5,
+                 lazy=False):
+        self.input_size = _pair(input_size)
+        if not mmcv.is_tuple_of(self.input_size, int):
+            raise TypeError(f'Input_size must be int or tuple of int, '
+                            f'but got {type(input_size)}')
+
+        if not isinstance(scales, tuple):
+            raise TypeError(f'Scales must be tuple, but got {type(scales)}')
+
+        if num_fixed_crops not in [5, 13]:
+            raise ValueError(f'Num_fix_crops must be in {[5, 13]}, '
+                             f'but got {num_fixed_crops}')
+
+        self.scales = scales
+        self.max_wh_scale_gap = max_wh_scale_gap
+        self.random_crop = random_crop
+        self.num_fixed_crops = num_fixed_crops
+        self.lazy = lazy
+
+    def __call__(self, results):
+        """Performs the MultiScaleCrop augmentation.
+
+        Args:
+            results (dict): The resulting dict to be modified and passed
+                to the next transform in pipeline.
+        """
+        _init_lazy_if_proper(results, self.lazy)
+        if 'keypoint' in results:
+            assert not self.lazy, ('Keypoint Augmentations are not compatible '
+                                   'with lazy == True')
+
+        img_h, img_w = results['img_shape']
+        base_size = min(img_h, img_w)
+        crop_sizes = [int(base_size * s) for s in self.scales]
+
+        candidate_sizes = []
+        for i, h in enumerate(crop_sizes):
+            for j, w in enumerate(crop_sizes):
+                if abs(i - j) <= self.max_wh_scale_gap:
+                    candidate_sizes.append([w, h])
+
+        crop_size = random.choice(candidate_sizes)
+        for i in range(2):
+            if abs(crop_size[i] - self.input_size[i]) < 3:
+                crop_size[i] = self.input_size[i]
+
+        crop_w, crop_h = crop_size
+
+        if self.random_crop:
+            x_offset = random.randint(0, img_w - crop_w)
+            y_offset = random.randint(0, img_h - crop_h)
+        else:
+            w_step = (img_w - crop_w) // 4
+            h_step = (img_h - crop_h) // 4
+            candidate_offsets = [
+                (0, 0),  # upper left
+                (4 * w_step, 0),  # upper right
+                (0, 4 * h_step),  # lower left
+                (4 * w_step, 4 * h_step),  # lower right
+                (2 * w_step, 2 * h_step),  # center
+            ]
+            if self.num_fixed_crops == 13:
+                extra_candidate_offsets = [
+                    (0, 2 * h_step),  # center left
+                    (4 * w_step, 2 * h_step),  # center right
+                    (2 * w_step, 4 * h_step),  # lower center
+                    (2 * w_step, 0 * h_step),  # upper center
+                    (1 * w_step, 1 * h_step),  # upper left quarter
+                    (3 * w_step, 1 * h_step),  # upper right quarter
+                    (1 * w_step, 3 * h_step),  # lower left quarter
+                    (3 * w_step, 3 * h_step)  # lower right quarter
+                ]
+                candidate_offsets.extend(extra_candidate_offsets)
+            x_offset, y_offset = random.choice(candidate_offsets)
+
+        new_h, new_w = crop_h, crop_w
+
+        crop_bbox = np.array(
+            [x_offset, y_offset, x_offset + new_w, y_offset + new_h])
+        results['crop_bbox'] = crop_bbox
+        results['img_shape'] = (new_h, new_w)
+        results['scales'] = self.scales
+
+        if 'crop_quadruple' not in results:
+            results['crop_quadruple'] = np.array(
+                [0, 0, 1, 1],  # x, y, w, h
+                dtype=np.float32)
+
+        x_ratio, y_ratio = x_offset / img_w, y_offset / img_h
+        w_ratio, h_ratio = new_w / img_w, new_h / img_h
+
+        old_crop_quadruple = results['crop_quadruple']
+        old_x_ratio, old_y_ratio = old_crop_quadruple[0], old_crop_quadruple[1]
+        old_w_ratio, old_h_ratio = old_crop_quadruple[2], old_crop_quadruple[3]
+        new_crop_quadruple = [
+            old_x_ratio + x_ratio * old_w_ratio,
+            old_y_ratio + y_ratio * old_h_ratio, w_ratio * old_w_ratio,
+            h_ratio * old_x_ratio
+        ]
+        results['crop_quadruple'] = np.array(
+            new_crop_quadruple, dtype=np.float32)
+
+        if not self.lazy:
+            if 'keypoint' in results:
+                results['keypoint'] = self._crop_kps(results['keypoint'],
+                                                     crop_bbox)
+            if 'imgs' in results:
+                results['imgs'] = self._crop_imgs(results['imgs'], crop_bbox)
+        else:
+            lazyop = results['lazy']
+            if lazyop['flip']:
+                raise NotImplementedError('Put Flip at last for now')
+
+            # record crop_bbox in lazyop dict to ensure only crop once in Fuse
+            lazy_left, lazy_top, lazy_right, lazy_bottom = lazyop['crop_bbox']
+            left = x_offset * (lazy_right - lazy_left) / img_w
+            right = (x_offset + new_w) * (lazy_right - lazy_left) / img_w
+            top = y_offset * (lazy_bottom - lazy_top) / img_h
+            bottom = (y_offset + new_h) * (lazy_bottom - lazy_top) / img_h
+            lazyop['crop_bbox'] = np.array([(lazy_left + left),
+                                            (lazy_top + top),
+                                            (lazy_left + right),
+                                            (lazy_top + bottom)],
+                                           dtype=np.float32)
+
+        if 'gt_bboxes' in results:
+            assert not self.lazy
+            results = self._all_box_crop(results, results['crop_bbox'])
+
+        return results
+
+    def __repr__(self):
+        repr_str = (f'{self.__class__.__name__}('
+                    f'input_size={self.input_size}, scales={self.scales}, '
+                    f'max_wh_scale_gap={self.max_wh_scale_gap}, '
+                    f'random_crop={self.random_crop}, '
+                    f'num_fixed_crops={self.num_fixed_crops}, '
+                    f'lazy={self.lazy})')
+        return repr_str
+
+
+@PIPELINES.register_module()
+class Resize:
+    """Resize images to a specific size.
+
+    Required keys are "img_shape", "modality", "imgs" (optional), "keypoint"
+    (optional), added or modified keys are "imgs", "img_shape", "keep_ratio",
+    "scale_factor", "lazy", "resize_size". Required keys in "lazy" is None,
+    added or modified key is "interpolation".
+
+    Args:
+        scale (float | Tuple[int]): If keep_ratio is True, it serves as scaling
+            factor or maximum size:
+            If it is a float number, the image will be rescaled by this
+            factor, else if it is a tuple of 2 integers, the image will
+            be rescaled as large as possible within the scale.
+            Otherwise, it serves as (w, h) of output size.
+        keep_ratio (bool): If set to True, Images will be resized without
+            changing the aspect ratio. Otherwise, it will resize images to a
+            given size. Default: True.
+        interpolation (str): Algorithm used for interpolation:
+            "nearest" | "bilinear". Default: "bilinear".
+        lazy (bool): Determine whether to apply lazy operation. Default: False.
+    """
+
+    def __init__(self,
+                 scale,
+                 keep_ratio=True,
+                 interpolation='bilinear',
+                 lazy=False):
+        if isinstance(scale, float):
+            if scale <= 0:
+                raise ValueError(f'Invalid scale {scale}, must be positive.')
+        elif isinstance(scale, tuple):
+            max_long_edge = max(scale)
+            max_short_edge = min(scale)
+            if max_short_edge == -1:
+                # assign np.inf to long edge for rescaling short edge later.
+                scale = (np.inf, max_long_edge)
+        else:
+            raise TypeError(
+                f'Scale must be float or tuple of int, but got {type(scale)}')
+        self.scale = scale
+        self.keep_ratio = keep_ratio
+        self.interpolation = interpolation
+        self.lazy = lazy
+
+    def _resize_imgs(self, imgs, new_w, new_h):
+        return [
+            mmcv.imresize(
+                img, (new_w, new_h), interpolation=self.interpolation)
+            for img in imgs
+        ]
+
+    @staticmethod
+    def _resize_kps(kps, scale_factor):
+        return kps * scale_factor
+
+    @staticmethod
+    def _box_resize(box, scale_factor):
+        """Rescale the bounding boxes according to the scale_factor.
+
+        Args:
+            box (np.ndarray): The bounding boxes.
+            scale_factor (np.ndarray): The scale factor used for rescaling.
+        """
+        assert len(scale_factor) == 2
+        scale_factor = np.concatenate([scale_factor, scale_factor])
+        return box * scale_factor
+
+    def __call__(self, results):
+        """Performs the Resize augmentation.
+
+        Args:
+            results (dict): The resulting dict to be modified and passed
+                to the next transform in pipeline.
+        """
+
+        _init_lazy_if_proper(results, self.lazy)
+        if 'keypoint' in results:
+            assert not self.lazy, ('Keypoint Augmentations are not compatible '
+                                   'with lazy == True')
+
+        if 'scale_factor' not in results:
+            results['scale_factor'] = np.array([1, 1], dtype=np.float32)
+        img_h, img_w = results['img_shape']
+
+        if self.keep_ratio:
+            new_w, new_h = mmcv.rescale_size((img_w, img_h), self.scale)
+        else:
+            new_w, new_h = self.scale
+        self.scale_factor = np.array([new_w / img_w, new_h / img_h],
+                                     dtype=np.float32)
+
+        results['img_shape'] = (new_h, new_w)
+        results['keep_ratio'] = self.keep_ratio
+        results['scale_factor'] = results['scale_factor'] * self.scale_factor
+
+        if not self.lazy:
+            if 'imgs' in results:
+                results['imgs'] = self._resize_imgs(results['imgs'], new_w,
+                                                    new_h)
+            if 'keypoint' in results:
+                results['keypoint'] = self._resize_kps(results['keypoint'],
+                                                       self.scale_factor)
+        else:
+            lazyop = results['lazy']
+            if lazyop['flip']:
+                raise NotImplementedError('Put Flip at last for now')
+            lazyop['interpolation'] = self.interpolation
+
+        if 'gt_bboxes' in results:
+            assert not self.lazy
+            results['gt_bboxes'] = self._box_resize(results['gt_bboxes'],
+                                                    self.scale_factor)
+            if 'proposals' in results and results['proposals'] is not None:
+                assert results['proposals'].shape[1] == 4
+                results['proposals'] = self._box_resize(
+                    results['proposals'], self.scale_factor)
+
+        return results
+
+    def __repr__(self):
+        repr_str = (f'{self.__class__.__name__}('
+                    f'scale={self.scale}, keep_ratio={self.keep_ratio}, '
+                    f'interpolation={self.interpolation}, '
+                    f'lazy={self.lazy})')
+        return repr_str
+
+
+@PIPELINES.register_module()
+class RandomRescale:
+    """Randomly resize images so that the short_edge is resized to a specific
+    size in a given range. The scale ratio is unchanged after resizing.
+
+    Required keys are "imgs", "img_shape", "modality", added or modified
+    keys are "imgs", "img_shape", "keep_ratio", "scale_factor", "resize_size",
+    "short_edge".
+
+    Args:
+        scale_range (tuple[int]): The range of short edge length. A closed
+            interval.
+        interpolation (str): Algorithm used for interpolation:
+            "nearest" | "bilinear". Default: "bilinear".
+    """
+
+    def __init__(self, scale_range, interpolation='bilinear'):
+        self.scale_range = scale_range
+        # make sure scale_range is legal, first make sure the type is OK
+        assert mmcv.is_tuple_of(scale_range, int)
+        assert len(scale_range) == 2
+        assert scale_range[0] < scale_range[1]
+        assert np.all([x > 0 for x in scale_range])
+
+        self.keep_ratio = True
+        self.interpolation = interpolation
+
+    def __call__(self, results):
+        """Performs the Resize augmentation.
+
+        Args:
+            results (dict): The resulting dict to be modified and passed
+                to the next transform in pipeline.
+        """
+        short_edge = np.random.randint(self.scale_range[0],
+                                       self.scale_range[1] + 1)
+        resize = Resize((-1, short_edge),
+                        keep_ratio=True,
+                        interpolation=self.interpolation,
+                        lazy=False)
+        results = resize(results)
+
+        results['short_edge'] = short_edge
+        return results
+
+    def __repr__(self):
+        scale_range = self.scale_range
+        repr_str = (f'{self.__class__.__name__}('
+                    f'scale_range=({scale_range[0]}, {scale_range[1]}), '
+                    f'interpolation={self.interpolation})')
+        return repr_str
+
+
+@PIPELINES.register_module()
+class Flip:
+    """Flip the input images with a probability.
+
+    Reverse the order of elements in the given imgs with a specific direction.
+    The shape of the imgs is preserved, but the elements are reordered.
+
+    Required keys are "img_shape", "modality", "imgs" (optional), "keypoint"
+    (optional), added or modified keys are "imgs", "keypoint", "lazy" and
+    "flip_direction". Required keys in "lazy" is None, added or modified key
+    are "flip" and "flip_direction". The Flip augmentation should be placed
+    after any cropping / reshaping augmentations, to make sure crop_quadruple
+    is calculated properly.
+
+    Args:
+        flip_ratio (float): Probability of implementing flip. Default: 0.5.
+        direction (str): Flip imgs horizontally or vertically. Options are
+            "horizontal" | "vertical". Default: "horizontal".
+        flip_label_map (Dict[int, int] | None): Transform the label of the
+            flipped image with the specific label. Default: None.
+        left_kp (list[int]): Indexes of left keypoints, used to flip keypoints.
+            Default: None.
+        right_kp (list[ind]): Indexes of right keypoints, used to flip
+            keypoints. Default: None.
+        lazy (bool): Determine whether to apply lazy operation. Default: False.
+    """
+    _directions = ['horizontal', 'vertical']
+
+    def __init__(self,
+                 flip_ratio=0.5,
+                 direction='horizontal',
+                 flip_label_map=None,
+                 left_kp=None,
+                 right_kp=None,
+                 lazy=False):
+        if direction not in self._directions:
+            raise ValueError(f'Direction {direction} is not supported. '
+                             f'Currently support ones are {self._directions}')
+        self.flip_ratio = flip_ratio
+        self.direction = direction
+        self.flip_label_map = flip_label_map
+        self.left_kp = left_kp
+        self.right_kp = right_kp
+        self.lazy = lazy
+
+    def _flip_imgs(self, imgs, modality):
+        _ = [mmcv.imflip_(img, self.direction) for img in imgs]
+        lt = len(imgs)
+        if modality == 'Flow':
+            # The 1st frame of each 2 frames is flow-x
+            for i in range(0, lt, 2):
+                imgs[i] = mmcv.iminvert(imgs[i])
+        return imgs
+
+    def _flip_kps(self, kps, kpscores, img_width):
+        kp_x = kps[..., 0]
+        kp_x[kp_x != 0] = img_width - kp_x[kp_x != 0]
+        new_order = list(range(kps.shape[2]))
+        if self.left_kp is not None and self.right_kp is not None:
+            for left, right in zip(self.left_kp, self.right_kp):
+                new_order[left] = right
+                new_order[right] = left
+        kps = kps[:, :, new_order]
+        if kpscores is not None:
+            kpscores = kpscores[:, :, new_order]
+        return kps, kpscores
+
+    @staticmethod
+    def _box_flip(box, img_width):
+        """Flip the bounding boxes given the width of the image.
+
+        Args:
+            box (np.ndarray): The bounding boxes.
+            img_width (int): The img width.
+        """
+        box_ = box.copy()
+        box_[..., 0::4] = img_width - box[..., 2::4]
+        box_[..., 2::4] = img_width - box[..., 0::4]
+        return box_
+
+    def __call__(self, results):
+        """Performs the Flip augmentation.
+
+        Args:
+            results (dict): The resulting dict to be modified and passed
+                to the next transform in pipeline.
+        """
+        _init_lazy_if_proper(results, self.lazy)
+        if 'keypoint' in results:
+            assert not self.lazy, ('Keypoint Augmentations are not compatible '
+                                   'with lazy == True')
+            assert self.direction == 'horizontal', (
+                'Only horizontal flips are'
+                'supported for human keypoints')
+
+        modality = results['modality']
+        if modality == 'Flow':
+            assert self.direction == 'horizontal'
+
+        flip = np.random.rand() < self.flip_ratio
+
+        results['flip'] = flip
+        results['flip_direction'] = self.direction
+        img_width = results['img_shape'][1]
+
+        if self.flip_label_map is not None and flip:
+            results['label'] = self.flip_label_map.get(results['label'],
+                                                       results['label'])
+
+        if not self.lazy:
+            if flip:
+                if 'imgs' in results:
+                    results['imgs'] = self._flip_imgs(results['imgs'],
+                                                      modality)
+                if 'keypoint' in results:
+                    kp = results['keypoint']
+                    kpscore = results.get('keypoint_score', None)
+                    kp, kpscore = self._flip_kps(kp, kpscore, img_width)
+                    results['keypoint'] = kp
+                    if 'keypoint_score' in results:
+                        results['keypoint_score'] = kpscore
+        else:
+            lazyop = results['lazy']
+            if lazyop['flip']:
+                raise NotImplementedError('Use one Flip please')
+            lazyop['flip'] = flip
+            lazyop['flip_direction'] = self.direction
+
+        if 'gt_bboxes' in results and flip:
+            assert not self.lazy and self.direction == 'horizontal'
+            width = results['img_shape'][1]
+            results['gt_bboxes'] = self._box_flip(results['gt_bboxes'], width)
+            if 'proposals' in results and results['proposals'] is not None:
+                assert results['proposals'].shape[1] == 4
+                results['proposals'] = self._box_flip(results['proposals'],
+                                                      width)
+
+        return results
+
+    def __repr__(self):
+        repr_str = (
+            f'{self.__class__.__name__}('
+            f'flip_ratio={self.flip_ratio}, direction={self.direction}, '
+            f'flip_label_map={self.flip_label_map}, lazy={self.lazy})')
+        return repr_str
+
+
+@PIPELINES.register_module()
+class Normalize:
+    """Normalize images with the given mean and std value.
+
+    Required keys are "imgs", "img_shape", "modality", added or modified
+    keys are "imgs" and "img_norm_cfg". If modality is 'Flow', additional
+    keys "scale_factor" is required
+
+    Args:
+        mean (Sequence[float]): Mean values of different channels.
+        std (Sequence[float]): Std values of different channels.
+        to_bgr (bool): Whether to convert channels from RGB to BGR.
+            Default: False.
+        adjust_magnitude (bool): Indicate whether to adjust the flow magnitude
+            on 'scale_factor' when modality is 'Flow'. Default: False.
+    """
+
+    def __init__(self, mean, std, to_bgr=False, adjust_magnitude=False):
+        if not isinstance(mean, Sequence):
+            raise TypeError(
+                f'Mean must be list, tuple or np.ndarray, but got {type(mean)}'
+            )
+
+        if not isinstance(std, Sequence):
+            raise TypeError(
+                f'Std must be list, tuple or np.ndarray, but got {type(std)}')
+
+        self.mean = np.array(mean, dtype=np.float32)
+        self.std = np.array(std, dtype=np.float32)
+        self.to_bgr = to_bgr
+        self.adjust_magnitude = adjust_magnitude
+
+    def __call__(self, results):
+        modality = results['modality']
+
+        if modality == 'RGB':
+            n = len(results['imgs'])
+            h, w, c = results['imgs'][0].shape
+            imgs = np.empty((n, h, w, c), dtype=np.float32)
+            for i, img in enumerate(results['imgs']):
+                imgs[i] = img
+
+            for img in imgs:
+                mmcv.imnormalize_(img, self.mean, self.std, self.to_bgr)
+            results['imgs'] = imgs
+            results['img_norm_cfg'] = dict(
+                mean=self.mean, std=self.std, to_bgr=self.to_bgr)
+            return results
+        if modality == 'Flow':
+            num_imgs = len(results['imgs'])
+            assert num_imgs % 2 == 0
+            assert self.mean.shape[0] == 2
+            assert self.std.shape[0] == 2
+            n = num_imgs // 2
+            h, w = results['imgs'][0].shape
+            x_flow = np.empty((n, h, w), dtype=np.float32)
+            y_flow = np.empty((n, h, w), dtype=np.float32)
+            for i in range(n):
+                x_flow[i] = results['imgs'][2 * i]
+                y_flow[i] = results['imgs'][2 * i + 1]
+            x_flow = (x_flow - self.mean[0]) / self.std[0]
+            y_flow = (y_flow - self.mean[1]) / self.std[1]
+            if self.adjust_magnitude:
+                x_flow = x_flow * results['scale_factor'][0]
+                y_flow = y_flow * results['scale_factor'][1]
+            imgs = np.stack([x_flow, y_flow], axis=-1)
+            results['imgs'] = imgs
+            args = dict(
+                mean=self.mean,
+                std=self.std,
+                to_bgr=self.to_bgr,
+                adjust_magnitude=self.adjust_magnitude)
+            results['img_norm_cfg'] = args
+            return results
+        raise NotImplementedError
+
+    def __repr__(self):
+        repr_str = (f'{self.__class__.__name__}('
+                    f'mean={self.mean}, '
+                    f'std={self.std}, '
+                    f'to_bgr={self.to_bgr}, '
+                    f'adjust_magnitude={self.adjust_magnitude})')
+        return repr_str
+
+@PIPELINES.register_module()
+class CenterCrop(RandomCrop):
+    """Crop the center area from images.
+
+    Required keys are "img_shape", "imgs" (optional), "keypoint" (optional),
+    added or modified keys are "imgs", "keypoint", "crop_bbox", "lazy" and
+    "img_shape". Required keys in "lazy" is "crop_bbox", added or modified key
+    is "crop_bbox".
+
+    Args:
+        crop_size (int | tuple[int]): (w, h) of crop size.
+        lazy (bool): Determine whether to apply lazy operation. Default: False.
+    """
+
+    def __init__(self, crop_size, lazy=False):
+        self.crop_size = _pair(crop_size)
+        self.lazy = lazy
+        if not mmcv.is_tuple_of(self.crop_size, int):
+            raise TypeError(f'Crop_size must be int or tuple of int, '
+                            f'but got {type(crop_size)}')
+
+    def __call__(self, results):
+        """Performs the CenterCrop augmentation.
+
+        Args:
+            results (dict): The resulting dict to be modified and passed
+                to the next transform in pipeline.
+        """
+        _init_lazy_if_proper(results, self.lazy)
+        if 'keypoint' in results:
+            assert not self.lazy, ('Keypoint Augmentations are not compatible '
+                                   'with lazy == True')
+
+        img_h, img_w = results['img_shape']
+        crop_w, crop_h = self.crop_size
+
+        left = (img_w - crop_w) // 2
+        top = (img_h - crop_h) // 2
+        right = left + crop_w
+        bottom = top + crop_h
+        new_h, new_w = bottom - top, right - left
+
+        crop_bbox = np.array([left, top, right, bottom])
+        results['crop_bbox'] = crop_bbox
+        results['img_shape'] = (new_h, new_w)
+
+        if 'crop_quadruple' not in results:
+            results['crop_quadruple'] = np.array(
+                [0, 0, 1, 1],  # x, y, w, h
+                dtype=np.float32)
+
+        x_ratio, y_ratio = left / img_w, top / img_h
+        w_ratio, h_ratio = new_w / img_w, new_h / img_h
+
+        old_crop_quadruple = results['crop_quadruple']
+        old_x_ratio, old_y_ratio = old_crop_quadruple[0], old_crop_quadruple[1]
+        old_w_ratio, old_h_ratio = old_crop_quadruple[2], old_crop_quadruple[3]
+        new_crop_quadruple = [
+            old_x_ratio + x_ratio * old_w_ratio,
+            old_y_ratio + y_ratio * old_h_ratio, w_ratio * old_w_ratio,
+            h_ratio * old_x_ratio
+        ]
+        results['crop_quadruple'] = np.array(
+            new_crop_quadruple, dtype=np.float32)
+
+        if not self.lazy:
+            if 'keypoint' in results:
+                results['keypoint'] = self._crop_kps(results['keypoint'],
+                                                     crop_bbox)
+            if 'imgs' in results:
+                results['imgs'] = self._crop_imgs(results['imgs'], crop_bbox)
+        else:
+            lazyop = results['lazy']
+            if lazyop['flip']:
+                raise NotImplementedError('Put Flip at last for now')
+
+            # record crop_bbox in lazyop dict to ensure only crop once in Fuse
+            lazy_left, lazy_top, lazy_right, lazy_bottom = lazyop['crop_bbox']
+            left = left * (lazy_right - lazy_left) / img_w
+            right = right * (lazy_right - lazy_left) / img_w
+            top = top * (lazy_bottom - lazy_top) / img_h
+            bottom = bottom * (lazy_bottom - lazy_top) / img_h
+            lazyop['crop_bbox'] = np.array([(lazy_left + left),
+                                            (lazy_top + top),
+                                            (lazy_left + right),
+                                            (lazy_top + bottom)],
+                                           dtype=np.float32)
+
+        if 'gt_bboxes' in results:
+            assert not self.lazy
+            results = self._all_box_crop(results, results['crop_bbox'])
+
+        return results
+
+    def __repr__(self):
+        repr_str = (f'{self.__class__.__name__}(crop_size={self.crop_size}, '
+                    f'lazy={self.lazy})')
+        return repr_str
+
+
+@PIPELINES.register_module()
+class ThreeCrop:
+    """Crop images into three crops.
+
+    Crop the images equally into three crops with equal intervals along the
+    shorter side.
+    Required keys are "imgs", "img_shape", added or modified keys are "imgs",
+    "crop_bbox" and "img_shape".
+
+    Args:
+        crop_size(int | tuple[int]): (w, h) of crop size.
+    """
+
+    def __init__(self, crop_size):
+        self.crop_size = _pair(crop_size)
+        if not mmcv.is_tuple_of(self.crop_size, int):
+            raise TypeError(f'Crop_size must be int or tuple of int, '
+                            f'but got {type(crop_size)}')
+
+    def __call__(self, results):
+        """Performs the ThreeCrop augmentation.
+
+        Args:
+            results (dict): The resulting dict to be modified and passed
+                to the next transform in pipeline.
+        """
+        _init_lazy_if_proper(results, False)
+        if 'gt_bboxes' in results or 'proposals' in results:
+            warnings.warn('ThreeCrop cannot process bounding boxes')
+
+        imgs = results['imgs']
+        img_h, img_w = results['imgs'][0].shape[:2]
+
+        crop_w, crop_h = self.crop_size
+        assert crop_h == img_h or crop_w == img_w
+
+        if crop_h == img_h:
+            w_step = (img_w - crop_w) // 2
+            offsets = [
+                (0, 0),  # left
+                (2 * w_step, 0),  # right
+                (w_step, 0),  # middle
+            ]
+        elif crop_w == img_w:
+            h_step = (img_h - crop_h) // 2
+            offsets = [
+                (0, 0),  # top
+                (0, 2 * h_step),  # down
+                (0, h_step),  # middle
+            ]
+
+        cropped = []
+        crop_bboxes = []
+        for x_offset, y_offset in offsets:
+            bbox = [x_offset, y_offset, x_offset + crop_w, y_offset + crop_h]
+            crop = [
+                img[y_offset:y_offset + crop_h, x_offset:x_offset + crop_w]
+                for img in imgs
+            ]
+            cropped.extend(crop)
+            crop_bboxes.extend([bbox for _ in range(len(imgs))])
+
+        crop_bboxes = np.array(crop_bboxes)
+        results['imgs'] = cropped
+        results['crop_bbox'] = crop_bboxes
+        results['img_shape'] = results['imgs'][0].shape[:2]
+
+        return results
+
+    def __repr__(self):
+        repr_str = f'{self.__class__.__name__}(crop_size={self.crop_size})'
+        return repr_str
+
+
+@PIPELINES.register_module()
+class TenCrop:
+    """Crop the images into 10 crops (corner + center + flip).
+
+    Crop the four corners and the center part of the image with the same
+    given crop_size, and flip it horizontally.
+    Required keys are "imgs", "img_shape", added or modified keys are "imgs",
+    "crop_bbox" and "img_shape".
+
+    Args:
+        crop_size(int | tuple[int]): (w, h) of crop size.
+    """
+
+    def __init__(self, crop_size):
+        self.crop_size = _pair(crop_size)
+        if not mmcv.is_tuple_of(self.crop_size, int):
+            raise TypeError(f'Crop_size must be int or tuple of int, '
+                            f'but got {type(crop_size)}')
+
+    def __call__(self, results):
+        """Performs the TenCrop augmentation.
+
+        Args:
+            results (dict): The resulting dict to be modified and passed
+                to the next transform in pipeline.
+        """
+        _init_lazy_if_proper(results, False)
+
+        if 'gt_bboxes' in results or 'proposals' in results:
+            warnings.warn('TenCrop cannot process bounding boxes')
+
+        imgs = results['imgs']
+
+        img_h, img_w = results['imgs'][0].shape[:2]
+        crop_w, crop_h = self.crop_size
+
+        w_step = (img_w - crop_w) // 4
+        h_step = (img_h - crop_h) // 4
+
+        offsets = [
+            (0, 0),  # upper left
+            (4 * w_step, 0),  # upper right
+            (0, 4 * h_step),  # lower left
+            (4 * w_step, 4 * h_step),  # lower right
+            (2 * w_step, 2 * h_step),  # center
+        ]
+
+        img_crops = list()
+        crop_bboxes = list()
+        for x_offset, y_offsets in offsets:
+            crop = [
+                img[y_offsets:y_offsets + crop_h, x_offset:x_offset + crop_w]
+                for img in imgs
+            ]
+            # import pdb
+            # pdb.set_trace()
+            flip_crop = [np.flip(c, axis=1).copy() for c in crop]
+            bbox = [x_offset, y_offsets, x_offset + crop_w, y_offsets + crop_h]
+            img_crops.extend(crop)
+            img_crops.extend(flip_crop)
+            crop_bboxes.extend([bbox for _ in range(len(imgs) * 2)])
+
+        crop_bboxes = np.array(crop_bboxes)
+
+        results['imgs'] = img_crops
+        results['crop_bbox'] = crop_bboxes
+        results['img_shape'] = results['imgs'][0].shape[:2]
+
+        return results
+
+    def __repr__(self):
+        repr_str = f'{self.__class__.__name__}(crop_size={self.crop_size})'
+        return repr_str
+
+
+@PIPELINES.register_module()
+class MultiGroupCrop:
+    """Randomly crop the images into several groups.
+
+    Crop the random region with the same given crop_size and bounding box
+    into several groups.
+    Required keys are "imgs", added or modified keys are "imgs", "crop_bbox"
+    and "img_shape".
+
+    Args:
+        crop_size(int | tuple[int]): (w, h) of crop size.
+        groups(int): Number of groups.
+    """
+
+    def __init__(self, crop_size, groups):
+        self.crop_size = _pair(crop_size)
+        self.groups = groups
+        if not mmcv.is_tuple_of(self.crop_size, int):
+            raise TypeError('Crop size must be int or tuple of int, '
+                            f'but got {type(crop_size)}')
+
+        if not isinstance(groups, int):
+            raise TypeError(f'Groups must be int, but got {type(groups)}.')
+
+        if groups <= 0:
+            raise ValueError('Groups must be positive.')
+
+    def __call__(self, results):
+        """Performs the MultiGroupCrop augmentation.
+
+        Args:
+            results (dict): The resulting dict to be modified and passed
+                to the next transform in pipeline.
+        """
+        if 'gt_bboxes' in results or 'proposals' in results:
+            warnings.warn('MultiGroupCrop cannot process bounding boxes')
+
+        imgs = results['imgs']
+        img_h, img_w = imgs[0].shape[:2]
+        crop_w, crop_h = self.crop_size
+
+        img_crops = []
+        crop_bboxes = []
+        for _ in range(self.groups):
+            x_offset = random.randint(0, img_w - crop_w)
+            y_offset = random.randint(0, img_h - crop_h)
+
+            bbox = [x_offset, y_offset, x_offset + crop_w, y_offset + crop_h]
+            crop = [
+                img[y_offset:y_offset + crop_h, x_offset:x_offset + crop_w]
+                for img in imgs
+            ]
+            img_crops.extend(crop)
+            crop_bboxes.extend([bbox for _ in range(len(imgs))])
+
+        crop_bboxes = np.array(crop_bboxes)
+        results['imgs'] = img_crops
+        results['crop_bbox'] = crop_bboxes
+        results['img_shape'] = results['imgs'][0].shape[:2]
+
+        return results
+
+    def __repr__(self):
+        repr_str = (f'{self.__class__.__name__}'
+                    f'(crop_size={self.crop_size}, '
+                    f'groups={self.groups})')
+        return repr_str
+
+
+@PIPELINES.register_module()
+class ColorJitter:
+    def __init__(self, p=0.8, p_gray=0.2, brightness=0.4, contrast=0.4,saturation=0.2, hue=0.1):
+        self.p = p
+        self.p_gray = p_gray
+        self.worker = torchvision.transforms.ColorJitter(brightness=brightness, contrast=contrast, saturation=saturation, hue=hue)
+        
+
+    def __call__(self, results):
+        imgs = results['imgs']
+        v = random.random()
+        if v < self.p:
+            imgs = [np.asarray(self.worker(Image.fromarray(img))) for img in imgs]
+        
+        results['imgs'] = imgs
+        return results
+
+    def __repr__(self):
+        repr_str = (f'{self.__class__.__name__}')
+        return repr_str
+
+@PIPELINES.register_module()
+class GrayScale:
+    def __init__(self, p=0.2):
+        self.p = p
+        self.worker_gray = torchvision.transforms.Grayscale(num_output_channels=3)
+
+    def __call__(self, results):
+        imgs = results['imgs']
+        v = random.random()
+        if v < self.p:
+            imgs = [np.asarray(self.worker_gray(Image.fromarray(img))) for img in imgs]
+        
+        results['imgs'] = imgs
+        return results
+    
+    def __repr__(self):
+        repr_str = (f'{self.__class__.__name__}')
+        return repr_str  
+
+
+@PIPELINES.register_module()
+class Compose:
+    def __init__(self, transforms):
+        assert isinstance(transforms, Sequence)
+        self.transforms = []
+        for transform in transforms:
+            if isinstance(transform, dict):
+                transform = build_from_cfg(transform, PIPELINES)
+                self.transforms.append(transform)
+            elif callable(transform):
+                self.transforms.append(transform)
+            else:
+                raise TypeError(f'transform must be callable or a dict, '
+                                f'but got {type(transform)}')
+
+    def __call__(self, data):
+        for t in self.transforms:
+            data = t(data)
+            if data is None:
+                return None
+        return data
+
+    def __repr__(self):
+        format_string = self.__class__.__name__ + '('
+        for t in self.transforms:
+            format_string += '\n'
+            format_string += '    {0}'.format(t)
+        format_string += '\n)'
+        return format_string
+
+@PIPELINES.register_module()
+class DecordInit:
+    """Using decord to initialize the video_reader.
+
+    Decord: https://github.com/dmlc/decord
+
+    Required keys are "filename",
+    added or modified keys are "video_reader" and "total_frames".
+    """
+
+    def __init__(self, io_backend='disk', num_threads=1, **kwargs):
+        self.io_backend = io_backend
+        self.num_threads = num_threads
+        self.kwargs = kwargs
+        self.file_client = None
+        self.tarfile = None
+
+    def __call__(self, results):
+        """Perform the Decord initialization.
+
+        Args:
+            results (dict): The resulting dict to be modified and passed
+                to the next transform in pipeline.
+        """
+        try:
+            import decord
+        except ImportError:
+            raise ImportError(
+                'Please run "pip install decord" to install Decord first.')
+        if results['tar'] is False:
+            if self.file_client is None:
+                self.file_client = FileClient(self.io_backend, **self.kwargs)
+
+            file_obj = io.BytesIO(self.file_client.get(results['filename']))
+
+            # print(results['filename'])
+        else:
+            if self.tarfile is None:
+                data_root = os.path.dirname(results['filename']) + '.tar'
+                self.tarfile = tarfile.open(data_root)
+            video_name = results['filename'].split('/')[-1]
+            iob = self.tarfile.extractfile(video_name)
+            iob = iob.read()
+            file_obj = io.BytesIO(iob)
+        try:
+            container = decord.VideoReader(file_obj, num_threads=self.num_threads)
+        except Exception as e:
+            print(results['filename'])
+            original_path = results['filename']
+            new_path = original_path.replace('/challenge_crop/rgb', '/eval_FO_ids')
+            results['filename'] = new_path
+            file_obj = io.BytesIO(self.file_client.get(results['filename']))
+            container = decord.VideoReader(file_obj, num_threads=self.num_threads)
+
+        results['video_reader'] = container
+        results['total_frames'] = len(container)
+        return results
+
+    def __repr__(self):
+        repr_str = (f'{self.__class__.__name__}('
+                    f'io_backend={self.io_backend}, '
+                    f'num_threads={self.num_threads})')
+        return repr_str
+
+@PIPELINES.register_module()
+class DecordDecode:
+    """Using decord to decode the video.
+
+    Decord: https://github.com/dmlc/decord
+
+    Required keys are "video_reader", "filename" and "frame_inds",
+    added or modified keys are "imgs" and "original_shape".
+    """
+
+    def __call__(self, results):
+        """Perform the Decord decoding.
+
+        Args:
+            results (dict): The resulting dict to be modified and passed
+                to the next transform in pipeline.
+        """
+        container = results['video_reader']
+
+        if results['frame_inds'].ndim != 1:
+            results['frame_inds'] = np.squeeze(results['frame_inds'])
+
+        frame_inds = results['frame_inds']
+        # Generate frame index mapping in order
+        frame_dict = {
+            idx: container[idx].asnumpy()
+            for idx in np.unique(frame_inds)
+        }
+
+        imgs = [frame_dict[idx] for idx in frame_inds]
+
+        results['video_reader'] = None
+        del container
+
+        results['imgs'] = imgs
+        results['original_shape'] = imgs[0].shape[:2]
+        results['img_shape'] = imgs[0].shape[:2]
+
+        return results
+
+@PIPELINES.register_module()
+class SampleFrames:
+    """Sample frames from the video.
+
+    Required keys are "total_frames", "start_index" , added or modified keys
+    are "frame_inds", "frame_interval" and "num_clips".
+
+    Args:
+        clip_len (int): Frames of each sampled output clip.
+        frame_interval (int): Temporal interval of adjacent sampled frames.
+            Default: 1.
+        num_clips (int): Number of clips to be sampled. Default: 1.
+        temporal_jitter (bool): Whether to apply temporal jittering.
+            Default: False.
+        twice_sample (bool): Whether to use twice sample when testing.
+            If set to True, it will sample frames with and without fixed shift,
+            which is commonly used for testing in TSM model. Default: False.
+        out_of_bound_opt (str): The way to deal with out of bounds frame
+            indexes. Available options are 'loop', 'repeat_last'.
+            Default: 'loop'.
+        test_mode (bool): Store True when building test or validation dataset.
+            Default: False.
+        start_index (None): This argument is deprecated and moved to dataset
+            class (``BaseDataset``, ``VideoDatset``, ``RawframeDataset``, etc),
+            see this: https://github.com/open-mmlab/mmaction2/pull/89.
+    """
+
+    def __init__(self,
+                 clip_len,
+                 frame_interval=1,
+                 num_clips=1,
+                 temporal_jitter=False,
+                 twice_sample=False,
+                 out_of_bound_opt='loop',
+                 test_mode=False,
+                 start_index=None,
+                 frame_uniform=False,
+                 multiview=1):
+
+        self.clip_len = clip_len
+        self.frame_interval = frame_interval
+        self.num_clips = num_clips
+        self.temporal_jitter = temporal_jitter
+        self.twice_sample = twice_sample
+        self.out_of_bound_opt = out_of_bound_opt
+        self.test_mode = test_mode
+        self.frame_uniform = frame_uniform
+        self.multiview=multiview
+        assert self.out_of_bound_opt in ['loop', 'repeat_last']
+
+        if start_index is not None:
+            warnings.warn('No longer support "start_index" in "SampleFrames", '
+                          'it should be set in dataset class, see this pr: '
+                          'https://github.com/open-mmlab/mmaction2/pull/89')
+
+    def _get_train_clips(self, num_frames):
+        """Get clip offsets in train mode.
+
+        It will calculate the average interval for selected frames,
+        and randomly shift them within offsets between [0, avg_interval].
+        If the total number of frames is smaller than clips num or origin
+        frames length, it will return all zero indices.
+
+        Args:
+            num_frames (int): Total number of frame in the video.
+
+        Returns:
+            np.ndarray: Sampled frame indices in train mode.
+        """
+        ori_clip_len = self.clip_len * self.frame_interval
+        avg_interval = (num_frames - ori_clip_len + 1) // self.num_clips
+
+        if avg_interval > 0:
+            base_offsets = np.arange(self.num_clips) * avg_interval
+            clip_offsets = base_offsets + np.random.randint(
+                avg_interval, size=self.num_clips)
+        elif num_frames > max(self.num_clips, ori_clip_len):
+            clip_offsets = np.sort(
+                np.random.randint(
+                    num_frames - ori_clip_len + 1, size=self.num_clips))
+        elif avg_interval == 0:
+            ratio = (num_frames - ori_clip_len + 1.0) / self.num_clips
+            clip_offsets = np.around(np.arange(self.num_clips) * ratio)
+        else:
+            clip_offsets = np.zeros((self.num_clips, ), dtype=np.int)
+
+        return clip_offsets
+
+    def _get_test_clips(self, num_frames):
+        """Get clip offsets in test mode.
+
+        Calculate the average interval for selected frames, and shift them
+        fixedly by avg_interval/2. If set twice_sample True, it will sample
+        frames together without fixed shift. If the total number of frames is
+        not enough, it will return all zero indices.
+
+        Args:
+            num_frames (int): Total number of frame in the video.
+
+        Returns:
+            np.ndarray: Sampled frame indices in test mode.
+        """
+        ori_clip_len = self.clip_len * self.frame_interval
+        avg_interval = (num_frames - ori_clip_len + 1) / float(self.num_clips)
+        if num_frames > ori_clip_len - 1:
+            base_offsets = np.arange(self.num_clips) * avg_interval
+            clip_offsets = (base_offsets + avg_interval / 2.0).astype(np.int)
+            if self.twice_sample:
+                clip_offsets = np.concatenate([clip_offsets, base_offsets])
+        else:
+            clip_offsets = np.zeros((self.num_clips, ), dtype=np.int)
+        return clip_offsets
+
+    def _sample_clips(self, num_frames):
+        """Choose clip offsets for the video in a given mode.
+
+        Args:
+            num_frames (int): Total number of frame in the video.
+
+        Returns:
+            np.ndarray: Sampled frame indices.
+        """
+        if self.test_mode:
+            clip_offsets = self._get_test_clips(num_frames)
+        else:
+            if self.multiview == 1:
+                clip_offsets = self._get_train_clips(num_frames)
+            else:
+                clip_offsets = np.concatenate([self._get_train_clips(num_frames)  for _ in range(self.multiview)])
+
+        return clip_offsets
+
+    def get_seq_frames(self, num_frames):
+        """
+        Modified from https://github.com/facebookresearch/SlowFast/blob/64abcc90ccfdcbb11cf91d6e525bed60e92a8796/slowfast/datasets/ssv2.py#L159
+        Given the video index, return the list of sampled frame indexes.
+        Args:
+            num_frames (int): Total number of frame in the video.
+        Returns:
+            seq (list): the indexes of frames of sampled from the video.
+        """
+        seg_size = float(num_frames - 1) / self.clip_len
+        seq = []
+        for i in range(self.clip_len):
+            start = int(np.round(seg_size * i))
+            end = int(np.round(seg_size * (i + 1)))
+            if not self.test_mode:
+                seq.append(random.randint(start, end))
+            else:
+                seq.append((start + end) // 2)
+
+        return np.array(seq)
+
+    def __call__(self, results):
+        """Perform the SampleFrames loading.
+
+        Args:
+            results (dict): The resulting dict to be modified and passed
+                to the next transform in pipeline.
+        """
+        total_frames = results['total_frames']
+        if self.frame_uniform:  # sthv2 sampling strategy
+            assert results['start_index'] == 0
+            frame_inds = self.get_seq_frames(total_frames)
+        else:
+            clip_offsets = self._sample_clips(total_frames)
+            frame_inds = clip_offsets[:, None] + np.arange(
+                self.clip_len)[None, :] * self.frame_interval
+            frame_inds = np.concatenate(frame_inds)
+
+            if self.temporal_jitter:
+                perframe_offsets = np.random.randint(
+                    self.frame_interval, size=len(frame_inds))
+                frame_inds += perframe_offsets
+
+            frame_inds = frame_inds.reshape((-1, self.clip_len))
+            if self.out_of_bound_opt == 'loop':
+                frame_inds = np.mod(frame_inds, total_frames)
+            elif self.out_of_bound_opt == 'repeat_last':
+                safe_inds = frame_inds < total_frames
+                unsafe_inds = 1 - safe_inds
+                last_ind = np.max(safe_inds * frame_inds, axis=1)
+                new_inds = (safe_inds * frame_inds + (unsafe_inds.T * last_ind).T)
+                frame_inds = new_inds
+            else:
+                raise ValueError('Illegal out_of_bound option.')
+
+            start_index = results['start_index']
+            frame_inds = np.concatenate(frame_inds) + start_index
+
+        results['frame_inds'] = frame_inds.astype(np.int)
+        results['clip_len'] = self.clip_len
+        results['frame_interval'] = self.frame_interval
+        results['num_clips'] = self.num_clips
+        return results
+
+    def __repr__(self):
+        repr_str = (f'{self.__class__.__name__}('
+                    f'clip_len={self.clip_len}, '
+                    f'frame_interval={self.frame_interval}, '
+                    f'num_clips={self.num_clips}, '
+                    f'temporal_jitter={self.temporal_jitter}, '
+                    f'twice_sample={self.twice_sample}, '
+                    f'out_of_bound_opt={self.out_of_bound_opt}, '
+                    f'test_mode={self.test_mode})')
+        return repr_str
+
+
+@PIPELINES.register_module()
+class FormatShape:
+    """Format final imgs shape to the given input_format.
+
+    Required keys are "imgs", "num_clips" and "clip_len", added or modified
+    keys are "imgs" and "input_shape".
+
+    Args:
+        input_format (str): Define the final imgs format.
+        collapse (bool): To collpase input_format N... to ... (NCTHW to CTHW,
+            etc.) if N is 1. Should be set as True when training and testing
+            detectors. Default: False.
+    """
+
+    def __init__(self, input_format, collapse=False):
+        self.input_format = input_format
+        self.collapse = collapse
+        if self.input_format not in ['NCTHW', 'NCHW', 'NCHW_Flow', 'NPTCHW']:
+            raise ValueError(
+                f'The input format {self.input_format} is invalid.')
+
+    def __call__(self, results):
+        """Performs the FormatShape formating.
+
+        Args:
+            results (dict): The resulting dict to be modified and passed
+                to the next transform in pipeline.
+        """
+        if not isinstance(results['imgs'], np.ndarray):
+            results['imgs'] = np.array(results['imgs'])
+        imgs = results['imgs']
+        # [M x H x W x C]
+        # M = 1 * N_crops * N_clips * L
+        if self.collapse:
+            assert results['num_clips'] == 1
+
+        if self.input_format == 'NCTHW':
+            num_clips = results['num_clips']
+            clip_len = results['clip_len']
+
+            imgs = imgs.reshape((-1, num_clips, clip_len) + imgs.shape[1:])
+            # N_crops x N_clips x L x H x W x C
+            imgs = np.transpose(imgs, (0, 1, 5, 2, 3, 4))
+            # N_crops x N_clips x C x L x H x W
+            imgs = imgs.reshape((-1, ) + imgs.shape[2:])
+            # M' x C x L x H x W
+            # M' = N_crops x N_clips
+        elif self.input_format == 'NCHW':
+            imgs = np.transpose(imgs, (0, 3, 1, 2))
+            # M x C x H x W
+        elif self.input_format == 'NCHW_Flow':
+            num_clips = results['num_clips']
+            clip_len = results['clip_len']
+            imgs = imgs.reshape((-1, num_clips, clip_len) + imgs.shape[1:])
+            # N_crops x N_clips x L x H x W x C
+            imgs = np.transpose(imgs, (0, 1, 2, 5, 3, 4))
+            # N_crops x N_clips x L x C x H x W
+            imgs = imgs.reshape((-1, imgs.shape[2] * imgs.shape[3]) +
+                                imgs.shape[4:])
+            # M' x C' x H x W
+            # M' = N_crops x N_clips
+            # C' = L x C
+        elif self.input_format == 'NPTCHW':
+            num_proposals = results['num_proposals']
+            num_clips = results['num_clips']
+            clip_len = results['clip_len']
+            imgs = imgs.reshape((num_proposals, num_clips * clip_len) +
+                                imgs.shape[1:])
+            # P x M x H x W x C
+            # M = N_clips x L
+            imgs = np.transpose(imgs, (0, 1, 4, 2, 3))
+            # P x M x C x H x W
+        if self.collapse:
+            assert imgs.shape[0] == 1
+            imgs = imgs.squeeze(0)
+
+        results['imgs'] = imgs
+        results['input_shape'] = imgs.shape
+        return results
+
+    def __repr__(self):
+        repr_str = self.__class__.__name__
+        repr_str += f"(input_format='{self.input_format}')"
+        return repr_str
+
+@PIPELINES.register_module()
+class Collect:
+    """Collect data from the loader relevant to the specific task.
+
+    This keeps the items in ``keys`` as it is, and collect items in
+    ``meta_keys`` into a meta item called ``meta_name``.This is usually
+    the last stage of the data loader pipeline.
+    For example, when keys='imgs', meta_keys=('filename', 'label',
+    'original_shape'), meta_name='img_metas', the results will be a dict with
+    keys 'imgs' and 'img_metas', where 'img_metas' is a DataContainer of
+    another dict with keys 'filename', 'label', 'original_shape'.
+
+    Args:
+        keys (Sequence[str]): Required keys to be collected.
+        meta_name (str): The name of the key that contains meta infomation.
+            This key is always populated. Default: "img_metas".
+        meta_keys (Sequence[str]): Keys that are collected under meta_name.
+            The contents of the ``meta_name`` dictionary depends on
+            ``meta_keys``.
+            By default this includes:
+
+            - "filename": path to the image file
+            - "label": label of the image file
+            - "original_shape": original shape of the image as a tuple
+                (h, w, c)
+            - "img_shape": shape of the image input to the network as a tuple
+                (h, w, c).  Note that images may be zero padded on the
+                bottom/right, if the batch tensor is larger than this shape.
+            - "pad_shape": image shape after padding
+            - "flip_direction": a str in ("horiziontal", "vertival") to
+                indicate if the image is fliped horizontally or vertically.
+            - "img_norm_cfg": a dict of normalization information:
+                - mean - per channel mean subtraction
+                - std - per channel std divisor
+                - to_rgb - bool indicating if bgr was converted to rgb
+        nested (bool): If set as True, will apply data[x] = [data[x]] to all
+            items in data. The arg is added for compatibility. Default: False.
+    """
+
+    def __init__(self,
+                 keys,
+                 meta_keys=('filename', 'label', 'original_shape', 'img_shape',
+                            'pad_shape', 'flip_direction', 'img_norm_cfg'),
+                 meta_name='img_metas',
+                 nested=False):
+        self.keys = keys
+        self.meta_keys = meta_keys
+        self.meta_name = meta_name
+        self.nested = nested
+
+    def __call__(self, results):
+        """Performs the Collect formating.
+
+        Args:
+            results (dict): The resulting dict to be modified and passed
+                to the next transform in pipeline.
+        """
+        data = {}
+        for key in self.keys:
+            data[key] = results[key]
+
+        if len(self.meta_keys) != 0:
+            meta = {}
+            for key in self.meta_keys:
+                meta[key] = results[key]
+            data[self.meta_name] = DC(meta, cpu_only=True)
+        if self.nested:
+            for k in data:
+                data[k] = [data[k]]
+
+        return data
+
+    def __repr__(self):
+        return (f'{self.__class__.__name__}('
+                f'keys={self.keys}, meta_keys={self.meta_keys}, '
+                f'nested={self.nested})')
+
+
+def to_tensor(data):
+    """Convert objects of various python types to :obj:`torch.Tensor`.
+
+    Supported types are: :class:`numpy.ndarray`, :class:`torch.Tensor`,
+    :class:`Sequence`, :class:`int` and :class:`float`.
+    """
+    if isinstance(data, torch.Tensor):
+        return data
+    if isinstance(data, np.ndarray):
+        return torch.from_numpy(data)
+    if isinstance(data, Sequence) and not mmcv.is_str(data):
+        return torch.tensor(data)
+    if isinstance(data, int):
+        return torch.LongTensor([data])
+    if isinstance(data, float):
+        return torch.FloatTensor([data])
+    raise TypeError(f'type {type(data)} cannot be converted to tensor.')
+
+
+@PIPELINES.register_module()
+class ToTensor:
+    """Convert some values in results dict to `torch.Tensor` type in data
+    loader pipeline.
+
+    Args:
+        keys (Sequence[str]): Required keys to be converted.
+    """
+
+    def __init__(self, keys):
+        self.keys = keys
+
+    def __call__(self, results):
+        """Performs the ToTensor formating.
+
+        Args:
+            results (dict): The resulting dict to be modified and passed
+                to the next transform in pipeline.
+        """
+        for key in self.keys:
+            results[key] = to_tensor(results[key])
+        return results
+
+    def __repr__(self):
+        return f'{self.__class__.__name__}(keys={self.keys})'
+
+
+
+@PIPELINES.register_module()
+class RandAugment:
+    def __init__(self, auto_augment, input_size=224, interpolation='bicubic', level='video'):
+        if isinstance(input_size, tuple):
+            img_size = input_size[-2:]
+        else:
+            img_size = input_size
+
+        if auto_augment:
+            assert isinstance(auto_augment, str)
+            if isinstance(img_size, tuple):
+                img_size_min = min(img_size)
+            else:
+                img_size_min = img_size
+            aa_params = {"translate_const": int(img_size_min * 0.45)}
+            if interpolation and interpolation != "random":
+                aa_params["interpolation"] = _pil_interp(interpolation)
+        self.auto_augment = auto_augment
+        self.aa_params = aa_params
+        self.level = level
+
+    def do_ops(self, ops, buf):
+        for op in ops:
+            buf = op(buf)
+        return buf
+
+    def get_ops(self, ra_ops, num_layers, choice_weights):
+        return np.random.choice(
+            ra_ops,
+            num_layers,
+            replace=choice_weights is None,
+            p=choice_weights,
+        )
+
+    def __call__(self, results):
+        if self.auto_augment.startswith("rand"):
+            ra_ops, num_layers, choice_weights = rand_augment_transform(self.auto_augment, self.aa_params)
+
+        assert results['modality'] == 'RGB', 'Imgaug only support RGB images.'
+        in_type = results['imgs'][0].dtype.type
+
+        if self.level == 'video':
+            ops = self.get_ops(ra_ops, num_layers, choice_weights)
+            buffer = [
+                transforms.ToPILImage()(frame) for frame in results['imgs']
+            ]
+            results['imgs'] = [
+                np.asarray(self.do_ops(ops, buf)) for buf in buffer
+            ]
+        
+        elif self.level == 'image':
+            buffer = [
+                transforms.ToPILImage()(frame) for frame in results['imgs']
+            ]
+            results['imgs'] = []
+            for buf in buffer:
+                ops = self.get_ops(ra_ops, num_layers, choice_weights)
+                buf = self.do_ops(ops, buf)
+                results['imgs'].append(np.asarray(buf))
+        else:
+            assert False, 'Unknown RandAugment config section'
+
+        img_h, img_w, _ = results['imgs'][0].shape
+        out_type = results['imgs'][0].dtype.type
+        assert in_type == out_type, \
+            ('Imgaug input dtype and output dtype are not the same. ',
+             f'Convert from {in_type} to {out_type}')
+
+        results['img_shape'] = (img_h, img_w)
+
+        return results
+            
+    
+    def __repr__(self):
+        repr_str = self.__class__.__name__ + f'(transforms={self.aug})'
+        return repr_str        

datasets/rand_augment.py

@@ -0,0 +1,532 @@
+"""
+This implementation is based on
+https://github.com/rwightman/pytorch-image-models/blob/master/timm/data/auto_augment.py
+pulished under an Apache License 2.0.
+
+COMMENT FROM ORIGINAL:
+AutoAugment, RandAugment, and AugMix for PyTorch
+This code implements the searched ImageNet policies with various tweaks and
+improvements and does not include any of the search code. AA and RA
+Implementation adapted from:
+    https://github.com/tensorflow/tpu/blob/master/models/official/efficientnet/autoaugment.py
+AugMix adapted from:
+    https://github.com/google-research/augmix
+Papers:
+    AutoAugment: Learning Augmentation Policies from Data
+    https://arxiv.org/abs/1805.09501
+    Learning Data Augmentation Strategies for Object Detection
+    https://arxiv.org/abs/1906.11172
+    RandAugment: Practical automated data augmentation...
+    https://arxiv.org/abs/1909.13719
+    AugMix: A Simple Data Processing Method to Improve Robustness and
+    Uncertainty https://arxiv.org/abs/1912.02781
+
+Hacked together by / Copyright 2020 Ross Wightman
+"""
+
+import math
+import numpy as np
+import random
+import re
+import PIL
+from PIL import Image, ImageEnhance, ImageOps
+
+_PIL_VER = tuple([int(x) for x in PIL.__version__.split(".")[:2]])
+
+_FILL = (128, 128, 128)
+
+# This signifies the max integer that the controller RNN could predict for the
+# augmentation scheme.
+_MAX_LEVEL = 10.0
+
+_HPARAMS_DEFAULT = {
+    "translate_const": 250,
+    "img_mean": _FILL,
+}
+
+_RANDOM_INTERPOLATION = (Image.BILINEAR, Image.BICUBIC)
+
+
+def _interpolation(kwargs):
+    interpolation = kwargs.pop("resample", Image.BILINEAR)
+    if isinstance(interpolation, (list, tuple)):
+        return random.choice(interpolation)
+    else:
+        return interpolation
+
+
+def _check_args_tf(kwargs):
+    if "fillcolor" in kwargs and _PIL_VER < (5, 0):
+        kwargs.pop("fillcolor")
+    kwargs["resample"] = _interpolation(kwargs)
+
+
+def shear_x(img, factor, **kwargs):
+    _check_args_tf(kwargs)
+    return img.transform(
+        img.size, Image.AFFINE, (1, factor, 0, 0, 1, 0), **kwargs
+    )
+
+
+def shear_y(img, factor, **kwargs):
+    _check_args_tf(kwargs)
+    return img.transform(
+        img.size, Image.AFFINE, (1, 0, 0, factor, 1, 0), **kwargs
+    )
+
+
+def translate_x_rel(img, pct, **kwargs):
+    pixels = pct * img.size[0]
+    _check_args_tf(kwargs)
+    return img.transform(
+        img.size, Image.AFFINE, (1, 0, pixels, 0, 1, 0), **kwargs
+    )
+
+
+def translate_y_rel(img, pct, **kwargs):
+    pixels = pct * img.size[1]
+    _check_args_tf(kwargs)
+    return img.transform(
+        img.size, Image.AFFINE, (1, 0, 0, 0, 1, pixels), **kwargs
+    )
+
+
+def translate_x_abs(img, pixels, **kwargs):
+    _check_args_tf(kwargs)
+    return img.transform(
+        img.size, Image.AFFINE, (1, 0, pixels, 0, 1, 0), **kwargs
+    )
+
+
+def translate_y_abs(img, pixels, **kwargs):
+    _check_args_tf(kwargs)
+    return img.transform(
+        img.size, Image.AFFINE, (1, 0, 0, 0, 1, pixels), **kwargs
+    )
+
+
+def rotate(img, degrees, **kwargs):
+    _check_args_tf(kwargs)
+    if _PIL_VER >= (5, 2):
+        return img.rotate(degrees, **kwargs)
+    elif _PIL_VER >= (5, 0):
+        w, h = img.size
+        post_trans = (0, 0)
+        rotn_center = (w / 2.0, h / 2.0)
+        angle = -math.radians(degrees)
+        matrix = [
+            round(math.cos(angle), 15),
+            round(math.sin(angle), 15),
+            0.0,
+            round(-math.sin(angle), 15),
+            round(math.cos(angle), 15),
+            0.0,
+        ]
+
+        def transform(x, y, matrix):
+            (a, b, c, d, e, f) = matrix
+            return a * x + b * y + c, d * x + e * y + f
+
+        matrix[2], matrix[5] = transform(
+            -rotn_center[0] - post_trans[0],
+            -rotn_center[1] - post_trans[1],
+            matrix,
+        )
+        matrix[2] += rotn_center[0]
+        matrix[5] += rotn_center[1]
+        return img.transform(img.size, Image.AFFINE, matrix, **kwargs)
+    else:
+        return img.rotate(degrees, resample=kwargs["resample"])
+
+
+def auto_contrast(img, **__):
+    return ImageOps.autocontrast(img)
+
+
+def invert(img, **__):
+    return ImageOps.invert(img)
+
+
+def equalize(img, **__):
+    return ImageOps.equalize(img)
+
+
+def solarize(img, thresh, **__):
+    return ImageOps.solarize(img, thresh)
+
+
+def solarize_add(img, add, thresh=128, **__):
+    lut = []
+    for i in range(256):
+        if i < thresh:
+            lut.append(min(255, i + add))
+        else:
+            lut.append(i)
+    if img.mode in ("L", "RGB"):
+        if img.mode == "RGB" and len(lut) == 256:
+            lut = lut + lut + lut
+        return img.point(lut)
+    else:
+        return img
+
+
+def posterize(img, bits_to_keep, **__):
+    if bits_to_keep >= 8:
+        return img
+    return ImageOps.posterize(img, bits_to_keep)
+
+
+def contrast(img, factor, **__):
+    return ImageEnhance.Contrast(img).enhance(factor)
+
+
+def color(img, factor, **__):
+    return ImageEnhance.Color(img).enhance(factor)
+
+
+def brightness(img, factor, **__):
+    return ImageEnhance.Brightness(img).enhance(factor)
+
+
+def sharpness(img, factor, **__):
+    return ImageEnhance.Sharpness(img).enhance(factor)
+
+
+def _randomly_negate(v):
+    """With 50% prob, negate the value"""
+    return -v if random.random() > 0.5 else v
+
+
+def _rotate_level_to_arg(level, _hparams):
+    # range [-30, 30]
+    level = (level / _MAX_LEVEL) * 30.0
+    level = _randomly_negate(level)
+    return (level,)
+
+
+def _enhance_level_to_arg(level, _hparams):
+    # range [0.1, 1.9]
+    return ((level / _MAX_LEVEL) * 1.8 + 0.1,)
+
+
+def _enhance_increasing_level_to_arg(level, _hparams):
+    # the 'no change' level is 1.0, moving away from that towards 0. or 2.0 increases the enhancement blend
+    # range [0.1, 1.9]
+    level = (level / _MAX_LEVEL) * 0.9
+    level = 1.0 + _randomly_negate(level)
+    return (level,)
+
+
+def _shear_level_to_arg(level, _hparams):
+    # range [-0.3, 0.3]
+    level = (level / _MAX_LEVEL) * 0.3
+    level = _randomly_negate(level)
+    return (level,)
+
+
+def _translate_abs_level_to_arg(level, hparams):
+    translate_const = hparams["translate_const"]
+    level = (level / _MAX_LEVEL) * float(translate_const)
+    level = _randomly_negate(level)
+    return (level,)
+
+
+def _translate_rel_level_to_arg(level, hparams):
+    # default range [-0.45, 0.45]
+    translate_pct = hparams.get("translate_pct", 0.45)
+    level = (level / _MAX_LEVEL) * translate_pct
+    level = _randomly_negate(level)
+    return (level,)
+
+
+def _posterize_level_to_arg(level, _hparams):
+    # As per Tensorflow TPU EfficientNet impl
+    # range [0, 4], 'keep 0 up to 4 MSB of original image'
+    # intensity/severity of augmentation decreases with level
+    return (int((level / _MAX_LEVEL) * 4),)
+
+
+def _posterize_increasing_level_to_arg(level, hparams):
+    # As per Tensorflow models research and UDA impl
+    # range [4, 0], 'keep 4 down to 0 MSB of original image',
+    # intensity/severity of augmentation increases with level
+    return (4 - _posterize_level_to_arg(level, hparams)[0],)
+
+
+def _posterize_original_level_to_arg(level, _hparams):
+    # As per original AutoAugment paper description
+    # range [4, 8], 'keep 4 up to 8 MSB of image'
+    # intensity/severity of augmentation decreases with level
+    return (int((level / _MAX_LEVEL) * 4) + 4,)
+
+
+def _solarize_level_to_arg(level, _hparams):
+    # range [0, 256]
+    # intensity/severity of augmentation decreases with level
+    return (int((level / _MAX_LEVEL) * 256),)
+
+
+def _solarize_increasing_level_to_arg(level, _hparams):
+    # range [0, 256]
+    # intensity/severity of augmentation increases with level
+    return (256 - _solarize_level_to_arg(level, _hparams)[0],)
+
+
+def _solarize_add_level_to_arg(level, _hparams):
+    # range [0, 110]
+    return (int((level / _MAX_LEVEL) * 110),)
+
+
+LEVEL_TO_ARG = {
+    "AutoContrast": None,
+    "Equalize": None,
+    "Invert": None,
+    "Rotate": _rotate_level_to_arg,
+    # There are several variations of the posterize level scaling in various Tensorflow/Google repositories/papers
+    "Posterize": _posterize_level_to_arg,
+    "PosterizeIncreasing": _posterize_increasing_level_to_arg,
+    "PosterizeOriginal": _posterize_original_level_to_arg,
+    "Solarize": _solarize_level_to_arg,
+    "SolarizeIncreasing": _solarize_increasing_level_to_arg,
+    "SolarizeAdd": _solarize_add_level_to_arg,
+    "Color": _enhance_level_to_arg,
+    "ColorIncreasing": _enhance_increasing_level_to_arg,
+    "Contrast": _enhance_level_to_arg,
+    "ContrastIncreasing": _enhance_increasing_level_to_arg,
+    "Brightness": _enhance_level_to_arg,
+    "BrightnessIncreasing": _enhance_increasing_level_to_arg,
+    "Sharpness": _enhance_level_to_arg,
+    "SharpnessIncreasing": _enhance_increasing_level_to_arg,
+    "ShearX": _shear_level_to_arg,
+    "ShearY": _shear_level_to_arg,
+    "TranslateX": _translate_abs_level_to_arg,
+    "TranslateY": _translate_abs_level_to_arg,
+    "TranslateXRel": _translate_rel_level_to_arg,
+    "TranslateYRel": _translate_rel_level_to_arg,
+}
+
+
+NAME_TO_OP = {
+    "AutoContrast": auto_contrast,
+    "Equalize": equalize,
+    "Invert": invert,
+    "Rotate": rotate,
+    "Posterize": posterize,
+    "PosterizeIncreasing": posterize,
+    "PosterizeOriginal": posterize,
+    "Solarize": solarize,
+    "SolarizeIncreasing": solarize,
+    "SolarizeAdd": solarize_add,
+    "Color": color,
+    "ColorIncreasing": color,
+    "Contrast": contrast,
+    "ContrastIncreasing": contrast,
+    "Brightness": brightness,
+    "BrightnessIncreasing": brightness,
+    "Sharpness": sharpness,
+    "SharpnessIncreasing": sharpness,
+    "ShearX": shear_x,
+    "ShearY": shear_y,
+    "TranslateX": translate_x_abs,
+    "TranslateY": translate_y_abs,
+    "TranslateXRel": translate_x_rel,
+    "TranslateYRel": translate_y_rel,
+}
+
+
+class AugmentOp:
+    """
+    Apply for video.
+    """
+
+    def __init__(self, name, prob=0.5, magnitude=10, hparams=None):
+        hparams = hparams or _HPARAMS_DEFAULT
+        self.aug_fn = NAME_TO_OP[name]
+        self.level_fn = LEVEL_TO_ARG[name]
+        self.prob = prob
+        self.magnitude = magnitude
+        self.hparams = hparams.copy()
+        self.kwargs = {
+            "fillcolor": hparams["img_mean"]
+            if "img_mean" in hparams
+            else _FILL,
+            "resample": hparams["interpolation"]
+            if "interpolation" in hparams
+            else _RANDOM_INTERPOLATION,
+        }
+
+        # If magnitude_std is > 0, we introduce some randomness
+        # in the usually fixed policy and sample magnitude from a normal distribution
+        # with mean `magnitude` and std-dev of `magnitude_std`.
+        # NOTE This is my own hack, being tested, not in papers or reference impls.
+        self.magnitude_std = self.hparams.get("magnitude_std", 0)
+
+    def __call__(self, img_list):
+        if self.prob < 1.0 and random.random() > self.prob:
+            return img_list
+        magnitude = self.magnitude
+        if self.magnitude_std and self.magnitude_std > 0:
+            magnitude = random.gauss(magnitude, self.magnitude_std)
+        magnitude = min(_MAX_LEVEL, max(0, magnitude))  # clip to valid range
+        level_args = (
+            self.level_fn(magnitude, self.hparams)
+            if self.level_fn is not None
+            else ()
+        )
+
+        if isinstance(img_list, list):
+            return [
+                self.aug_fn(img, *level_args, **self.kwargs) for img in img_list
+            ]
+        else:
+            return self.aug_fn(img_list, *level_args, **self.kwargs)
+
+
+_RAND_TRANSFORMS = [
+    "AutoContrast",
+    "Equalize",
+    "Invert",
+    "Rotate",
+    "Posterize",
+    "Solarize",
+    "SolarizeAdd",
+    "Color",
+    "Contrast",
+    "Brightness",
+    "Sharpness",
+    "ShearX",
+    "ShearY",
+    "TranslateXRel",
+    "TranslateYRel",
+]
+
+
+_RAND_INCREASING_TRANSFORMS = [
+    "AutoContrast",
+    "Equalize",
+    "Invert",
+    "Rotate",
+    "PosterizeIncreasing",
+    "SolarizeIncreasing",
+    "SolarizeAdd",
+    "ColorIncreasing",
+    "ContrastIncreasing",
+    "BrightnessIncreasing",
+    "SharpnessIncreasing",
+    "ShearX",
+    "ShearY",
+    "TranslateXRel",
+    "TranslateYRel",
+]
+
+
+# These experimental weights are based loosely on the relative improvements mentioned in paper.
+# They may not result in increased performance, but could likely be tuned to so.
+_RAND_CHOICE_WEIGHTS_0 = {
+    "Rotate": 0.3,
+    "ShearX": 0.2,
+    "ShearY": 0.2,
+    "TranslateXRel": 0.1,
+    "TranslateYRel": 0.1,
+    "Color": 0.025,
+    "Sharpness": 0.025,
+    "AutoContrast": 0.025,
+    "Solarize": 0.005,
+    "SolarizeAdd": 0.005,
+    "Contrast": 0.005,
+    "Brightness": 0.005,
+    "Equalize": 0.005,
+    "Posterize": 0,
+    "Invert": 0,
+}
+
+
+def _select_rand_weights(weight_idx=0, transforms=None):
+    transforms = transforms or _RAND_TRANSFORMS
+    assert weight_idx == 0  # only one set of weights currently
+    rand_weights = _RAND_CHOICE_WEIGHTS_0
+    probs = [rand_weights[k] for k in transforms]
+    probs /= np.sum(probs)
+    return probs
+
+
+def rand_augment_ops(magnitude=10, hparams=None, transforms=None):
+    hparams = hparams or _HPARAMS_DEFAULT
+    transforms = transforms or _RAND_TRANSFORMS
+    return [
+        AugmentOp(name, prob=0.5, magnitude=magnitude, hparams=hparams)
+        for name in transforms
+    ]
+
+
+class RandAugment:
+    def __init__(self, ops, num_layers=2, choice_weights=None):
+        self.ops = ops
+        self.num_layers = num_layers
+        self.choice_weights = choice_weights
+
+    def __call__(self, img):
+        # no replacement when using weighted choice
+        ops = np.random.choice(
+            self.ops,
+            self.num_layers,
+            replace=self.choice_weights is None,
+            p=self.choice_weights,
+        )
+        for op in ops:
+            img = op(img)
+        return img
+
+
+def rand_augment_transform(config_str, hparams):
+    """
+    RandAugment: Practical automated data augmentation... - https://arxiv.org/abs/1909.13719
+
+    Create a RandAugment transform
+    :param config_str: String defining configuration of random augmentation. Consists of multiple sections separated by
+    dashes ('-'). The first section defines the specific variant of rand augment (currently only 'rand'). The remaining
+    sections, not order sepecific determine
+        'm' - integer magnitude of rand augment
+        'n' - integer num layers (number of transform ops selected per image)
+        'w' - integer probabiliy weight index (index of a set of weights to influence choice of op)
+        'mstd' -  float std deviation of magnitude noise applied
+        'inc' - integer (bool), use augmentations that increase in severity with magnitude (default: 0)
+    Ex 'rand-m9-n3-mstd0.5' results in RandAugment with magnitude 9, num_layers 3, magnitude_std 0.5
+    'rand-mstd1-w0' results in magnitude_std 1.0, weights 0, default magnitude of 10 and num_layers 2
+    :param hparams: Other hparams (kwargs) for the RandAugmentation scheme
+    :return: A PyTorch compatible Transform
+    """
+    magnitude = _MAX_LEVEL  # default to _MAX_LEVEL for magnitude (currently 10)
+    num_layers = 2  # default to 2 ops per image
+    weight_idx = None  # default to no probability weights for op choice
+    transforms = _RAND_TRANSFORMS
+    config = config_str.split("-")
+    assert config[0] == "rand"
+    config = config[1:]
+    for c in config:
+        cs = re.split(r"(\d.*)", c)
+        if len(cs) < 2:
+            continue
+        key, val = cs[:2]
+        if key == "mstd":
+            # noise param injected via hparams for now
+            hparams.setdefault("magnitude_std", float(val))
+        elif key == "inc":
+            if bool(val):
+                transforms = _RAND_INCREASING_TRANSFORMS
+        elif key == "m":
+            magnitude = int(val)
+        elif key == "n":
+            num_layers = int(val)
+        elif key == "w":
+            weight_idx = int(val)
+        else:
+            assert NotImplementedError
+    ra_ops = rand_augment_ops(
+        magnitude=magnitude, hparams=hparams, transforms=transforms
+    )
+    choice_weights = (
+        None if weight_idx is None else _select_rand_weights(weight_idx)
+    )
+    return ra_ops, num_layers, choice_weights
+    # return RandAugment(ra_ops, num_layers, choice_weights=choice_weights)

labels/challenge.csv

@@ -0,0 +1,46 @@
+id,name
+0,walking
+1,entering a space
+2,leaving a space
+3,sitting
+4,standing
+5,sitting down
+6,getting up
+7,lying down
+8,cooking
+9,making coffee
+10,making tea
+11,wiping table
+12,spreading bedding
+13,folding bedding
+14,cleaning dishes
+15,vacuuming
+16,looking for something
+17,putting on shoes
+18,taking off shoes
+19,putting on glasses
+20,taking off glasses
+21,cleaning
+22,writing
+23,talking
+24,beckoning
+25,waving a hand
+26,clapping
+27,pointing
+28,shaking hands
+29,phone calls
+30,using a telephone
+31,hugging
+32,washing face
+33,washing hands
+34,brushing teeth
+35,brushing hair
+36,massaging a shoulder oneself
+37,taking medicine
+38,eating
+39,drinking
+40,watching tv
+41,reading
+42,using a laptop
+43,using a tablet
+44,exercising

labels/challenge_composite.csv

@@ -0,0 +1,7 @@
+id,name
+0,locomotion
+1,manipulation
+2,communication
+3,hygiene
+4,eating_drinking
+5,leisure

labels/etri_label.csv

@@ -0,0 +1,56 @@
+id,name
+1,eating food with a fork
+2,pouring water into a cup
+3,taking medicine
+4,drinking water
+5,putting food in the fridge/taking food from the fridge 　　
+6,trimming vegetables
+7,peeling fruit
+8,using a gas stove
+9,cutting vegetable on the cutting board
+10,brushing teeth
+11,washing hands
+12,washing face
+13,wiping face with a towel
+14,putting on cosmetics
+15,putting on lipstick
+16,brushing hair
+17,blow drying hair
+18,putting on a jacket
+19,taking off a jacket
+20,putting on/taking off shoes
+21,putting on/taking off glasses
+22,washing the dishes
+23,vacuumming the floor
+24,scrubbing the floor with a rag
+25,wipping off the dinning table
+26,rubbing up furniture
+27,spreading bedding/folding bedding
+28,washing a towel by hands
+29,hanging out laundry
+30,looking around for something
+31,using a remote control
+32,reading a book
+33,reading a newspaper
+34,handwriting
+35,talking on the phone
+36,playing with a mobile phone
+37,using a computer
+38,smoking
+39,clapping
+40,rubbing face with hands
+41,doing freehand exercise
+42,doing neck roll exercise
+43,massaging a shoulder oneself
+44,taking a bow
+45,talking to each other
+46,handshaking
+47,hugging each other
+48,fighting each other
+49,waving a hand
+50,flapping a hand up and down (beckoning) 　　　　
+51,pointing with a finger
+52,opening the door and walking in
+53,fallen on the floor
+54,sitting up/standing up
+55,lying down

labels/sh_cs_label.csv

@@ -0,0 +1,32 @@
+id,name
+0,Pour.Frombottle
+1,WatchTV
+2,Readbook
+3,Sitdown
+4,Walk
+5,Eat.Snack
+6,Cook.Cleanup
+7,Cook.Stir
+8,Enter
+9,Eat.Attable
+10,Drink.Fromcup
+11,Cook.Cleandishes
+12,Laydown
+13,Drink.Frombottle
+14,Leave
+15,Drink.Fromcan
+16,Getup
+17,Pour.Fromkettle
+18,Usetelephone
+19,Takepills
+20,Maketea.Boilwater
+21,Uselaptop
+22,Cutbread
+23,Cook.Usestove
+24,Makecoffee.Pourgrains
+25,Cook.Cut
+26,Pour.Fromcan
+27,Usetablet
+28,Maketea.Insertteabag
+29,Drink.Fromglass
+30,Makecoffee.Pourwater

main_challenge.py

@@ -0,0 +1,292 @@
+import os
+import torch
+import torch.nn as nn
+import torch.backends.cudnn as cudnn
+import torch.distributed as dist
+import argparse
+import datetime
+import shutil
+from pathlib import Path
+from utils.config import get_config
+from utils.optimizer import build_optimizer, build_scheduler
+from utils.tools import AverageMeter, reduce_tensor, epoch_saving, load_checkpoint, generate_text, auto_resume_helper
+from datasets.build import build_dataloader
+from utils.logger import create_logger
+import time, csv
+import numpy as np
+import random
+from apex import amp
+from timm.loss import LabelSmoothingCrossEntropy, SoftTargetCrossEntropy
+from datasets.blending import CutmixMixupBlending
+from utils.config import get_config
+from trainers import vificlip
+
+
+def parse_option():
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--config', '-cfg', required=True, type=str, default='configs/k400/32_8.yaml')
+    parser.add_argument(
+        "--opts",
+        help="Modify config options by adding 'KEY VALUE' pairs. ",
+        default=None,
+        nargs='+',
+    )
+    parser.add_argument('--output', type=str, default="exp")
+    parser.add_argument('--resume', type=str)
+    parser.add_argument('--pretrained', type=str)
+    parser.add_argument('--only_test', action='store_true')
+    parser.add_argument('--batch-size', type=int)
+    parser.add_argument('--accumulation-steps', type=int)
+
+    parser.add_argument("--local_rank", type=int, default=-1, help='local rank for DistributedDataParallel')
+    args = parser.parse_args()
+
+    config = get_config(args)
+
+    return args, config
+
+
+def main(config):
+    train_data, val_data, train_loader, val_loader = build_dataloader(logger, config)
+    class_names = [class_name for i, class_name in train_data.classes]
+
+    # Custom trainer for different variants of ViFi-CLIP
+    model = vificlip.returnCLIP(config,
+                                logger=logger,
+                                class_names=class_names,)
+
+    model = model.cuda()  # changing to cuda here
+
+    mixup_fn = None
+    if config.AUG.MIXUP > 0:
+        criterion = SoftTargetCrossEntropy()
+        mixup_fn = CutmixMixupBlending(num_classes=config.DATA.NUM_CLASSES,
+                                       smoothing=config.AUG.LABEL_SMOOTH,
+                                       mixup_alpha=config.AUG.MIXUP,
+                                       cutmix_alpha=config.AUG.CUTMIX,
+                                       switch_prob=config.AUG.MIXUP_SWITCH_PROB)
+    elif config.AUG.LABEL_SMOOTH > 0:
+        criterion = LabelSmoothingCrossEntropy(smoothing=config.AUG.LABEL_SMOOTH)
+    else:
+        criterion = nn.CrossEntropyLoss()
+
+    optimizer = build_optimizer(config, model)
+    lr_scheduler = build_scheduler(config, optimizer, len(train_loader))
+    if config.TRAIN.OPT_LEVEL != 'O0':
+        model, optimizer = amp.initialize(models=model, optimizers=optimizer, opt_level=config.TRAIN.OPT_LEVEL)
+
+    model = torch.nn.parallel.DistributedDataParallel(model, device_ids=[config.LOCAL_RANK], broadcast_buffers=False,
+                                                      find_unused_parameters=False)
+
+    start_epoch, max_accuracy = 0, 0.0
+
+    if config.TRAIN.AUTO_RESUME:
+        resume_file = auto_resume_helper(config.OUTPUT)
+        if resume_file:
+            config.defrost()
+            config.MODEL.RESUME = resume_file
+            config.freeze()
+            logger.info(f'auto resuming from {resume_file}')
+        else:
+            logger.info(f'no checkpoint found in {config.OUTPUT}, ignoring auto resume')
+
+    if config.MODEL.RESUME:
+        start_epoch, max_accuracy = load_checkpoint(config, model, optimizer, lr_scheduler, logger)
+        if start_epoch > 1:
+            logger.info("resetting epochs no and max. accuracy to 0 after loading pre-trained weights")
+            start_epoch = 0
+            max_accuracy = 0
+    if config.TEST.ONLY_TEST:
+        acc1 = validate(val_loader, model, config)
+        logger.info(f"Accuracy of the network on the {len(val_data)} test videos: {acc1:.1f}%")
+        return
+
+    for epoch in range(start_epoch, config.TRAIN.EPOCHS):
+        train_loader.sampler.set_epoch(epoch)
+        train_one_epoch(epoch, model, criterion, optimizer, lr_scheduler, train_loader, config, mixup_fn)
+
+        if epoch % config.SAVE_FREQ == 0 or epoch == (config.TRAIN.EPOCHS - 1):
+            acc1 = validate(val_loader, model, config)
+            logger.info(f"Accuracy of the network on the {len(val_data)} test videos: {acc1:.1f}%")
+            is_best = acc1 > max_accuracy
+            max_accuracy = max(max_accuracy, acc1)
+            logger.info(f'Max accuracy: {max_accuracy:.2f}%')
+            if dist.get_rank() == 0 and (
+                    epoch % config.SAVE_FREQ == 0 or epoch == (config.TRAIN.EPOCHS - 1) or is_best):
+                epoch_saving(config, epoch, model, max_accuracy, optimizer, lr_scheduler, logger, config.OUTPUT,
+                             is_best)
+    # Now doing the multi-view inference crop for videos
+    # 4 CLIPs are obtained from each video, and for each CLIP, we get 3 crops (augmentations)
+    multi_view_inference = config.TEST.MULTI_VIEW_INFERENCE
+    if multi_view_inference:
+        config.defrost()
+        config.TEST.NUM_CLIP = 4
+        config.TEST.NUM_CROP = 3
+        config.freeze()
+        train_data, val_data, train_loader, val_loader = build_dataloader(logger, config)
+        acc1 = validate(val_loader, model, config)
+        logger.info(f"Accuracy of the network on the {len(val_data)} test videos: {acc1:.1f}%")
+
+
+def train_one_epoch(epoch, model, criterion, optimizer, lr_scheduler, train_loader, config, mixup_fn):
+    model.train()
+    optimizer.zero_grad()
+
+    num_steps = len(train_loader)
+    batch_time = AverageMeter()
+    tot_loss_meter = AverageMeter()
+
+    start = time.time()
+    end = time.time()
+
+
+    for idx, batch_data in enumerate(train_loader):
+
+        images = batch_data["imgs"].cuda(non_blocking=True)
+        label_id = batch_data["label"].cuda(non_blocking=True)
+        label_id = label_id.reshape(-1)
+        images = images.view((-1, config.DATA.NUM_FRAMES, 3) + images.size()[-2:])
+
+        if mixup_fn is not None:
+            images, label_id = mixup_fn(images, label_id)
+
+        output = model(images)
+
+        total_loss = criterion(output, label_id)
+        total_loss = total_loss / config.TRAIN.ACCUMULATION_STEPS
+
+        if config.TRAIN.ACCUMULATION_STEPS == 1:
+            optimizer.zero_grad()
+        if config.TRAIN.OPT_LEVEL != 'O0':
+            with amp.scale_loss(total_loss, optimizer) as scaled_loss:
+                scaled_loss.backward()
+        else:
+            total_loss.backward()
+        if config.TRAIN.ACCUMULATION_STEPS > 1:
+            if (idx + 1) % config.TRAIN.ACCUMULATION_STEPS == 0:
+                optimizer.step()
+                optimizer.zero_grad()
+                lr_scheduler.step_update(epoch * num_steps + idx)
+        else:
+            optimizer.step()
+            lr_scheduler.step_update(epoch * num_steps + idx)
+
+        torch.cuda.synchronize()
+
+        tot_loss_meter.update(total_loss.item(), len(label_id))
+        batch_time.update(time.time() - end)
+        end = time.time()
+
+        if idx % config.PRINT_FREQ == 0:
+            lr = optimizer.param_groups[0]['lr']
+            memory_used = torch.cuda.max_memory_allocated() / (1024.0 * 1024.0)
+            etas = batch_time.avg * (num_steps - idx)
+            logger.info(
+                f'Train: [{epoch}/{config.TRAIN.EPOCHS}][{idx}/{num_steps}]\t'
+                f'eta {datetime.timedelta(seconds=int(etas))} lr {lr:.9f}\t'
+                f'time {batch_time.val:.4f} ({batch_time.avg:.4f})\t'
+                f'tot_loss {tot_loss_meter.val:.4f} ({tot_loss_meter.avg:.4f})\t'
+                f'mem {memory_used:.0f}MB')
+    epoch_time = time.time() - start
+    logger.info(f"EPOCH {epoch} training takes {datetime.timedelta(seconds=int(epoch_time))}")
+
+
+@torch.no_grad()
+def validate(val_loader, model, config):
+    model.eval()
+    results = []
+    id = 0
+    acc1_meter, acc5_meter = AverageMeter(), AverageMeter()
+    with torch.no_grad():
+        logger.info(f"{config.TEST.NUM_CLIP * config.TEST.NUM_CROP} views inference")
+        for idx, batch_data in enumerate(val_loader):
+            _image = batch_data["imgs"]
+            label_id = batch_data["label"]
+            label_id = label_id.reshape(-1)
+            # print(idx)
+            
+            b, tn, c, h, w = _image.size()
+            t = config.DATA.NUM_FRAMES
+            n = tn // t
+            _image = _image.view(b, n, t, c, h, w)
+
+            tot_similarity = torch.zeros((b, config.DATA.NUM_CLASSES)).cuda()
+            for i in range(n):
+                image = _image[:, i, :, :, :, :]  # [b,t,c,h,w]
+                label_id = label_id.cuda(non_blocking=True)
+                image_input = image.cuda(non_blocking=True)
+
+                if config.TRAIN.OPT_LEVEL == 'O2':
+                    image_input = image_input.half()
+
+                output = model(image_input)
+
+                similarity = output.view(b, -1).softmax(dim=-1)
+                tot_similarity += similarity
+
+            values_1, indices_1 = tot_similarity.topk(1, dim=-1)
+            values_5, indices_5 = tot_similarity.topk(5, dim=-1)
+            for i in range(b):
+                results.append((id, indices_1[i].squeeze().tolist()))
+                id += 1
+            '''
+            acc1, acc5 = 0, 0
+            for i in range(b):
+                if indices_1[i] == label_id[i]:
+                    acc1 += 1
+                if label_id[i] in indices_5[i]:
+                    acc5 += 1
+
+            acc1_meter.update(float(acc1) / b * 100, b)
+            acc5_meter.update(float(acc5) / b * 100, b)
+            if idx % config.PRINT_FREQ == 0:
+                logger.info(
+                    f'Test: [{idx}/{len(val_loader)}]\t'
+                    f'Acc@1: {acc1_meter.avg:.3f}\t'
+                )
+            '''
+    with open('batch_indices.csv', 'w', newline='') as f:
+        writer = csv.writer(f)
+        writer.writerow(['id', 'Top-1 Index'])
+        writer.writerows(results)
+    #acc1_meter.sync()
+    #acc5_meter.sync()
+    #logger.info(f' * Acc@1 {acc1_meter.avg:.3f} Acc@5 {acc5_meter.avg:.3f}')
+    return acc1_meter.avg
+
+
+if __name__ == '__main__':
+    # prepare config
+    args, config = parse_option()
+
+    # init_distributed
+    if 'RANK' in os.environ and 'WORLD_SIZE' in os.environ:
+        rank = int(os.environ["RANK"])
+        world_size = int(os.environ['WORLD_SIZE'])
+        print(f"RANK and WORLD_SIZE in environ: {rank}/{world_size}")
+    else:
+        rank = -1
+        world_size = -1
+    torch.cuda.set_device(args.local_rank)
+    torch.distributed.init_process_group(backend='nccl', init_method='env://', world_size=world_size, rank=rank)
+    torch.distributed.barrier(device_ids=[args.local_rank])
+
+    seed = config.SEED + dist.get_rank()
+    torch.manual_seed(seed)
+    np.random.seed(seed)
+    random.seed(seed)
+    cudnn.benchmark = True
+
+    # create working_dir
+    Path(config.OUTPUT).mkdir(parents=True, exist_ok=True)
+
+    # logger
+    logger = create_logger(output_dir=config.OUTPUT, dist_rank=dist.get_rank(), name=f"{config.MODEL.ARCH}")
+    logger.info(f"working dir: {config.OUTPUT}")
+
+    # save config 
+    if dist.get_rank() == 0:
+        logger.info(config)
+        shutil.copy(args.config, config.OUTPUT)
+
+    main(config)

main_train.py

@@ -0,0 +1,281 @@
+import os
+import torch
+import torch.nn as nn
+import torch.backends.cudnn as cudnn
+import torch.distributed as dist
+import argparse
+import datetime
+import shutil
+from pathlib import Path
+from utils.config import get_config
+from utils.optimizer import build_optimizer, build_scheduler
+from utils.tools import AverageMeter, reduce_tensor, epoch_saving, load_checkpoint, generate_text, auto_resume_helper
+from datasets.build import build_dataloader
+from utils.logger import create_logger
+import time
+import numpy as np
+import random
+from apex import amp
+from timm.loss import LabelSmoothingCrossEntropy, SoftTargetCrossEntropy
+from datasets.blending import CutmixMixupBlending
+from utils.config import get_config
+from trainers import vificlip
+
+
+def parse_option():
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--config', '-cfg', required=True, type=str, default='configs/k400/32_8.yaml')
+    parser.add_argument(
+        "--opts",
+        help="Modify config options by adding 'KEY VALUE' pairs. ",
+        default=None,
+        nargs='+',
+    )
+    parser.add_argument('--output', type=str, default="exp")
+    parser.add_argument('--resume', type=str)
+    parser.add_argument('--pretrained', type=str)
+    parser.add_argument('--only_test', action='store_true')
+    parser.add_argument('--batch-size', type=int)
+    parser.add_argument('--accumulation-steps', type=int)
+
+    parser.add_argument("--local_rank", type=int, default=-1, help='local rank for DistributedDataParallel')
+    args = parser.parse_args()
+
+    config = get_config(args)
+
+    return args, config
+
+
+def main(config):
+    train_data, val_data, train_loader, val_loader = build_dataloader(logger, config)
+    class_names = [class_name for i, class_name in train_data.classes]
+
+    # Custom trainer for different variants of ViFi-CLIP
+    model = vificlip.returnCLIP(config,
+                                logger=logger,
+                                class_names=class_names,)
+
+    model = model.cuda()  # changing to cuda here
+
+    mixup_fn = None
+    if config.AUG.MIXUP > 0:
+        criterion = SoftTargetCrossEntropy()
+        mixup_fn = CutmixMixupBlending(num_classes=config.DATA.NUM_CLASSES,
+                                       smoothing=config.AUG.LABEL_SMOOTH,
+                                       mixup_alpha=config.AUG.MIXUP,
+                                       cutmix_alpha=config.AUG.CUTMIX,
+                                       switch_prob=config.AUG.MIXUP_SWITCH_PROB)
+    elif config.AUG.LABEL_SMOOTH > 0:
+        criterion = LabelSmoothingCrossEntropy(smoothing=config.AUG.LABEL_SMOOTH)
+    else:
+        criterion = nn.CrossEntropyLoss()
+
+    optimizer = build_optimizer(config, model)
+    lr_scheduler = build_scheduler(config, optimizer, len(train_loader))
+    if config.TRAIN.OPT_LEVEL != 'O0':
+        model, optimizer = amp.initialize(models=model, optimizers=optimizer, opt_level=config.TRAIN.OPT_LEVEL)
+
+    model = torch.nn.parallel.DistributedDataParallel(model, device_ids=[config.LOCAL_RANK], broadcast_buffers=False,
+                                                      find_unused_parameters=False)
+
+    start_epoch, max_accuracy = 0, 0.0
+
+    if config.TRAIN.AUTO_RESUME:
+        resume_file = auto_resume_helper(config.OUTPUT)
+        if resume_file:
+            config.defrost()
+            config.MODEL.RESUME = resume_file
+            config.freeze()
+            logger.info(f'auto resuming from {resume_file}')
+        else:
+            logger.info(f'no checkpoint found in {config.OUTPUT}, ignoring auto resume')
+
+    if config.MODEL.RESUME:
+        start_epoch, max_accuracy = load_checkpoint(config, model, optimizer, lr_scheduler, logger)
+        if start_epoch > 1:
+            logger.info("resetting epochs no and max. accuracy to 0 after loading pre-trained weights")
+            start_epoch = 0
+            max_accuracy = 0
+    if config.TEST.ONLY_TEST:
+        acc1 = validate(val_loader, model, config)
+        logger.info(f"Accuracy of the network on the {len(val_data)} test videos: {acc1:.1f}%")
+        return
+
+    for epoch in range(start_epoch, config.TRAIN.EPOCHS):
+        train_loader.sampler.set_epoch(epoch)
+        train_one_epoch(epoch, model, criterion, optimizer, lr_scheduler, train_loader, config, mixup_fn)
+
+        if epoch % config.SAVE_FREQ == 0 or epoch == (config.TRAIN.EPOCHS - 1):
+            acc1 = validate(val_loader, model, config)
+            logger.info(f"Accuracy of the network on the {len(val_data)} test videos: {acc1:.1f}%")
+            is_best = acc1 > max_accuracy
+            max_accuracy = max(max_accuracy, acc1)
+            logger.info(f'Max accuracy: {max_accuracy:.2f}%')
+            if dist.get_rank() == 0 and (
+                    epoch % config.SAVE_FREQ == 0 or epoch == (config.TRAIN.EPOCHS - 1) or is_best):
+                epoch_saving(config, epoch, model, max_accuracy, optimizer, lr_scheduler, logger, config.OUTPUT,
+                             is_best)
+    # Now doing the multi-view inference crop for videos
+    # 4 CLIPs are obtained from each video, and for each CLIP, we get 3 crops (augmentations)
+    multi_view_inference = config.TEST.MULTI_VIEW_INFERENCE
+    if multi_view_inference:
+        config.defrost()
+        config.TEST.NUM_CLIP = 4
+        config.TEST.NUM_CROP = 3
+        config.freeze()
+        train_data, val_data, train_loader, val_loader = build_dataloader(logger, config)
+        acc1 = validate(val_loader, model, config)
+        logger.info(f"Accuracy of the network on the {len(val_data)} test videos: {acc1:.1f}%")
+
+
+def train_one_epoch(epoch, model, criterion, optimizer, lr_scheduler, train_loader, config, mixup_fn):
+    model.train()
+    optimizer.zero_grad()
+
+    num_steps = len(train_loader)
+    batch_time = AverageMeter()
+    tot_loss_meter = AverageMeter()
+
+    start = time.time()
+    end = time.time()
+
+
+    for idx, batch_data in enumerate(train_loader):
+
+        images = batch_data["imgs"].cuda(non_blocking=True)
+        label_id = batch_data["label"].cuda(non_blocking=True)
+        label_id = label_id.reshape(-1)
+        images = images.view((-1, config.DATA.NUM_FRAMES, 3) + images.size()[-2:])
+
+        if mixup_fn is not None:
+            images, label_id = mixup_fn(images, label_id)
+
+        output = model(images)
+
+        total_loss = criterion(output, label_id)
+        total_loss = total_loss / config.TRAIN.ACCUMULATION_STEPS
+
+        if config.TRAIN.ACCUMULATION_STEPS == 1:
+            optimizer.zero_grad()
+        if config.TRAIN.OPT_LEVEL != 'O0':
+            with amp.scale_loss(total_loss, optimizer) as scaled_loss:
+                scaled_loss.backward()
+        else:
+            total_loss.backward()
+        if config.TRAIN.ACCUMULATION_STEPS > 1:
+            if (idx + 1) % config.TRAIN.ACCUMULATION_STEPS == 0:
+                optimizer.step()
+                optimizer.zero_grad()
+                lr_scheduler.step_update(epoch * num_steps + idx)
+        else:
+            optimizer.step()
+            lr_scheduler.step_update(epoch * num_steps + idx)
+
+        torch.cuda.synchronize()
+
+        tot_loss_meter.update(total_loss.item(), len(label_id))
+        batch_time.update(time.time() - end)
+        end = time.time()
+
+        if idx % config.PRINT_FREQ == 0:
+            lr = optimizer.param_groups[0]['lr']
+            memory_used = torch.cuda.max_memory_allocated() / (1024.0 * 1024.0)
+            etas = batch_time.avg * (num_steps - idx)
+            logger.info(
+                f'Train: [{epoch}/{config.TRAIN.EPOCHS}][{idx}/{num_steps}]\t'
+                f'eta {datetime.timedelta(seconds=int(etas))} lr {lr:.9f}\t'
+                f'time {batch_time.val:.4f} ({batch_time.avg:.4f})\t'
+                f'tot_loss {tot_loss_meter.val:.4f} ({tot_loss_meter.avg:.4f})\t'
+                f'mem {memory_used:.0f}MB')
+    epoch_time = time.time() - start
+    logger.info(f"EPOCH {epoch} training takes {datetime.timedelta(seconds=int(epoch_time))}")
+
+
+@torch.no_grad()
+def validate(val_loader, model, config):
+    model.eval()
+
+    acc1_meter, acc5_meter = AverageMeter(), AverageMeter()
+    with torch.no_grad():
+        logger.info(f"{config.TEST.NUM_CLIP * config.TEST.NUM_CROP} views inference")
+        for idx, batch_data in enumerate(val_loader):
+            _image = batch_data["imgs"]
+            label_id = batch_data["label"]
+            label_id = label_id.reshape(-1)
+
+            b, tn, c, h, w = _image.size()
+            t = config.DATA.NUM_FRAMES
+            n = tn // t
+            _image = _image.view(b, n, t, c, h, w)
+
+            tot_similarity = torch.zeros((b, config.DATA.NUM_CLASSES)).cuda()
+            for i in range(n):
+                image = _image[:, i, :, :, :, :]  # [b,t,c,h,w]
+                label_id = label_id.cuda(non_blocking=True)
+                image_input = image.cuda(non_blocking=True)
+
+                if config.TRAIN.OPT_LEVEL == 'O2':
+                    image_input = image_input.half()
+
+                output = model(image_input)
+
+                similarity = output.view(b, -1).softmax(dim=-1)
+                tot_similarity += similarity
+
+            values_1, indices_1 = tot_similarity.topk(1, dim=-1)
+            values_5, indices_5 = tot_similarity.topk(5, dim=-1)
+            acc1, acc5 = 0, 0
+            for i in range(b):
+                if indices_1[i] == label_id[i]:
+                    acc1 += 1
+                if label_id[i] in indices_5[i]:
+                    acc5 += 1
+
+            acc1_meter.update(float(acc1) / b * 100, b)
+            acc5_meter.update(float(acc5) / b * 100, b)
+            if idx % config.PRINT_FREQ == 0:
+                logger.info(
+                    f'Test: [{idx}/{len(val_loader)}]\t'
+                    f'Acc@1: {acc1_meter.avg:.3f}\t'
+                )
+    acc1_meter.sync()
+    acc5_meter.sync()
+    logger.info(f' * Acc@1 {acc1_meter.avg:.3f} Acc@5 {acc5_meter.avg:.3f}')
+    return acc1_meter.avg
+
+
+if __name__ == '__main__':
+    # prepare config
+    args, config = parse_option()
+
+    # init_distributed
+    if 'RANK' in os.environ and 'WORLD_SIZE' in os.environ:
+        rank = int(os.environ["RANK"])
+        world_size = int(os.environ['WORLD_SIZE'])
+        print(f"RANK and WORLD_SIZE in environ: {rank}/{world_size}")
+    else:
+        rank = -1
+        world_size = -1
+    torch.cuda.set_device(args.local_rank)
+    torch.distributed.init_process_group(backend='nccl', init_method='env://', world_size=world_size, rank=rank)
+    torch.distributed.barrier(device_ids=[args.local_rank])
+
+    seed = config.SEED + dist.get_rank()
+    torch.manual_seed(seed)
+    np.random.seed(seed)
+    random.seed(seed)
+    cudnn.benchmark = True
+
+    # create working_dir
+    Path(config.OUTPUT).mkdir(parents=True, exist_ok=True)
+
+    # logger
+    logger = create_logger(output_dir=config.OUTPUT, dist_rank=dist.get_rank(), name=f"{config.MODEL.ARCH}")
+    logger.info(f"working dir: {config.OUTPUT}")
+
+    # save config 
+    if dist.get_rank() == 0:
+        logger.info(config)
+        shutil.copy(args.config, config.OUTPUT)
+
+    main(config)

merge_results.py

@@ -0,0 +1,43 @@
+import pandas as pd
+import csv
+
+# Load the CSV files
+video_mapping_df = pd.read_csv('/data/vidlab_datasets/challenge_crop/split/val.csv', header=None, names=['video', 'index'], skiprows=0)
+prediction_results_df = pd.read_csv('batch_indices.csv', header=None, names=['id', 'top_1_index'], skiprows=1)
+labels_df = pd.read_csv('/data/users/sdas/scripts/ViFi-CLIP/labels/challenge.csv', header=None, names=['id', 'label'], skiprows=1)
+
+# Convert types for accurate merging
+video_mapping_df['index'] = video_mapping_df['index'].astype(int)
+prediction_results_df['id'] = prediction_results_df['id'].astype(int)
+prediction_results_df['top_1_index'] = prediction_results_df['top_1_index'].astype(int)
+labels_df['id'] = labels_df['id'].astype(int)
+
+# Assuming prediction_results_df and video_mapping_df are already defined and properly indexed
+with open('output.csv', mode='w', newline='') as file:
+    writer = csv.writer(file)
+
+    # Write the header
+    writer.writerow(['video_name', 'action_category'])
+
+    # Initialize k to start at the first index of video_mapping_df
+    k = 0
+
+    # Loop through the 'top_1_index' in prediction_results_df
+    for i in prediction_results_df['top_1_index']:
+        if 0 <= i <= 7:
+            action_category = 'locomotion'
+        elif 8 <= i <= 22:
+            action_category = 'manipulation'
+        elif 23 <= i <= 31:
+            action_category = 'communication'
+        elif 32 <= i <= 37:
+            action_category = 'hygiene'
+        elif i in {38, 39}:  # This corrects the logic to check for both values 38 and 39
+            action_category = 'eating_drinking'
+        else:
+            action_category = 'leisure'
+
+        # Write the row to the CSV file
+        writer.writerow([video_mapping_df['video'][k], action_category])
+        k += 1
+

requirements.txt

@@ -0,0 +1,12 @@
+torch==1.11.0
+torchvision==0.12.0
+pathlib
+mmcv-full
+decord
+ftfy
+einops
+termcolor
+timm
+regex
+yacs
+pandas

script_crop.sh

@@ -0,0 +1 @@
+python crop_person.py

script_test.sh

@@ -0,0 +1,4 @@
+export MASTER_ADDR='localhost'  # Adjust this to the IP as required
+export MASTER_PORT=29800  # Ensure this port is available
+python -m torch.distributed.launch --nproc_per_node=1 --master_addr $MASTER_ADDR --master_port $MSTER_PORT main_challenge.py -cfg ./configs/config_challenge_test.yaml --output ./work_dirs/
+python merge_results.py

script_train.sh

@@ -0,0 +1,3 @@
+
+python -m torch.distributed.launch --nproc_per_node=8 main_train.py -cfg ./configs/config_challenge_train.yaml --output ./work_dirs/challenge_baseline_new/ --opts TEST.NUM_CLIP 1 TEST.NUM_CROP 1
+

trainers/vificlip.py

@@ -0,0 +1,248 @@
+import os.path as osp
+from collections import OrderedDict
+import math
+
+import torch
+import torch.nn as nn
+from torch.nn import functional as F
+from torch.cuda.amp import GradScaler, autocast
+
+from clip import clip
+from clip.simple_tokenizer import SimpleTokenizer as _Tokenizer
+
+_tokenizer = _Tokenizer()
+
+
+def load_clip_to_cpu(cfg):
+    backbone_name = cfg.MODEL.ARCH
+    url = clip._MODELS[backbone_name]
+    model_path = clip._download(url)
+
+    try:
+        # loading JIT archive
+        model = torch.jit.load(model_path, map_location="cpu").eval()
+        state_dict = None
+
+    except RuntimeError:
+        state_dict = torch.load(model_path, map_location="cpu")
+    design_details = {"trainer": 'ViFi_CLIP',
+                      "vision_depth": cfg.TRAINER.ViFi_CLIP.PROMPT_DEPTH_VISION,
+                      "language_depth": cfg.TRAINER.ViFi_CLIP.PROMPT_DEPTH_TEXT,
+                      "vision_ctx": cfg.TRAINER.ViFi_CLIP.N_CTX_VISION,
+                      "language_ctx": cfg.TRAINER.ViFi_CLIP.N_CTX_TEXT}
+    model = clip.build_model(state_dict or model.state_dict(), design_details)
+
+    return model
+
+
+class TextEncoder(nn.Module):
+    def __init__(self, clip_model):
+        super().__init__()
+        self.transformer = clip_model.transformer
+        self.positional_embedding = clip_model.positional_embedding
+        self.ln_final = clip_model.ln_final
+        self.text_projection = clip_model.text_projection
+        self.dtype = clip_model.dtype
+
+    def forward(self, prompts, tokenized_prompts):
+        x = prompts + self.positional_embedding.type(self.dtype)
+        x = x.permute(1, 0, 2)  # NLD -> LND
+        x = self.transformer(x)
+        x = x.permute(1, 0, 2)  # LND -> NLD
+        x = self.ln_final(x).type(self.dtype)
+
+        # x.shape = [batch_size, n_ctx, transformer.width]
+        # take features from the eot embedding (eot_token is the highest number in each sequence)
+        x = x[torch.arange(x.shape[0]), tokenized_prompts.argmax(dim=-1)] @ self.text_projection
+
+        return x
+
+
+class VLPromptLearner(nn.Module):
+    def __init__(self, cfg, classnames, clip_model, logger):
+        super().__init__()
+        dtype = clip_model.dtype
+        self.use_prompt_stage = cfg.TRAINER.ViFi_CLIP.PROMPT_MODEL
+        ctx_init = cfg.TRAINER.ViFi_CLIP.CTX_INIT
+        ZS_evaluation = cfg.TRAINER.ViFi_CLIP.ZS_EVAL
+        if ZS_evaluation:
+            text_aug = f"{{}}"
+            tokenized_prompts = torch.cat([clip.tokenize(text_aug.format(c), context_length=77) for c in classnames])
+            embedding = clip_model.token_embedding(tokenized_prompts).type(dtype).cuda()
+            self.register_buffer("complete_text_embeddings", embedding)
+            self.tokenized_prompts = tokenized_prompts  # torch.Tensor
+        elif self.use_prompt_stage:
+            n_cls = len(classnames)
+            # Make sure Language depth >= 1
+            assert cfg.TRAINER.ViFi_CLIP.PROMPT_DEPTH_TEXT >= 1, "In VL prompting, Language prompt depth should be >=1" \
+                                                        "\nPlease use VPT trainer if you want to learn only vision " \
+                                                        "branch  "
+            n_ctx = cfg.TRAINER.ViFi_CLIP.N_CTX_TEXT
+            ctx_dim = clip_model.ln_final.weight.shape[0]
+
+            if ctx_init and (n_ctx) <= 4:
+                # use given words to initialize context vectors
+                ctx_init = ctx_init.replace("_", " ")
+                n_ctx = n_ctx
+                prompt = clip.tokenize(ctx_init)
+                with torch.no_grad():
+                    embedding = clip_model.token_embedding(prompt).type(dtype)
+                ctx_vectors = embedding[0, 1: 1 + n_ctx, :]
+                prompt_prefix = ctx_init
+            else:
+                # random initialization
+                ctx_vectors = torch.empty(n_ctx, ctx_dim, dtype=dtype)
+                nn.init.normal_(ctx_vectors, std=0.02)
+                prompt_prefix = " ".join(["X"] * n_ctx)
+            logger.info(f"V-L design")
+            logger.info(f'Initial text context: "{prompt_prefix}"')
+            logger.info(f"Number of context words (tokens) for Language prompting: {n_ctx}")
+            logger.info(f"Number of context words (tokens) for Vision prompting: {cfg.TRAINER.ViFi_CLIP.N_CTX_VISION}")
+            self.ctx = nn.Parameter(ctx_vectors)
+
+            classnames = [name.replace("_", " ") for name in classnames]
+            prompts = [prompt_prefix + " " + name + "." for name in classnames]
+
+            tokenized_prompts = torch.cat([clip.tokenize(p) for p in prompts])  # (n_cls, n_tkn)
+            with torch.no_grad():
+                embedding = clip_model.token_embedding(tokenized_prompts).type(dtype)
+
+            # These token vectors will be saved when in save_model(),
+            # but they should be ignored in load_model() as we want to use
+            # those computed using the current class names
+            self.register_buffer("token_prefix", embedding[:, :1, :])  # SOS
+            self.register_buffer("token_suffix", embedding[:, 1 + n_ctx:, :])  # CLS, EOS
+            self.n_cls = n_cls
+            self.tokenized_prompts = tokenized_prompts  # torch.Tensor
+        else:
+            # No prompting
+            ctx_init = ctx_init.replace("_", " ")
+            prompt_prefix = ctx_init
+            prompts = [prompt_prefix + " " + name + "." for name in classnames]
+            tokenized_prompts = torch.cat([clip.tokenize(p) for p in prompts])  # (n_cls, n_tkn)
+            with torch.no_grad():
+                embedding = clip_model.token_embedding(tokenized_prompts).type(dtype)
+            self.register_buffer("complete_text_embeddings", embedding)
+            self.tokenized_prompts = tokenized_prompts  # torch.Tensor
+
+    def construct_prompts(self, ctx, prefix, suffix, label=None):
+        # dim0 is either batch_size (during training) or n_cls (during testing)
+        # ctx: context tokens, with shape of (dim0, n_ctx, ctx_dim)
+        # prefix: the sos token, with shape of (n_cls, 1, ctx_dim)
+        # suffix: remaining tokens, with shape of (n_cls, *, ctx_dim)
+
+        if label is not None:
+            prefix = prefix[label]
+            suffix = suffix[label]
+
+        prompts = torch.cat(
+            [
+                prefix,  # (dim0, 1, dim)
+                ctx,  # (dim0, n_ctx, dim)
+                suffix,  # (dim0, *, dim)
+            ],
+            dim=1,
+        )
+
+        return prompts
+
+    def forward(self):
+        if self.use_prompt_stage:
+            ctx = self.ctx
+            if ctx.dim() == 2:
+                ctx = ctx.unsqueeze(0).expand(self.n_cls, -1, -1)
+
+            prefix = self.token_prefix
+            suffix = self.token_suffix
+            prompts = self.construct_prompts(ctx, prefix, suffix)
+        else:
+            prompts = self.complete_text_embeddings
+
+        return prompts
+
+
+class ViFiCLIP(nn.Module):
+    def __init__(self, cfg, classnames, clip_model, logger):
+        super().__init__()
+        self.prompt_learner = VLPromptLearner(cfg, classnames, clip_model, logger)
+        self.tokenized_prompts = self.prompt_learner.tokenized_prompts
+        self.image_encoder = clip_model.visual
+        self.text_encoder = TextEncoder(clip_model)
+        self.logit_scale = clip_model.logit_scale
+        self.dtype = clip_model.dtype
+    def forward(self, image):
+        tokenized_prompts = self.tokenized_prompts
+        logit_scale = self.logit_scale.exp()
+        prompts = self.prompt_learner()
+
+        # b = image.shape[0]
+        # Lets encode the video into required format
+        b, t, c, h, w = image.size()
+        # Remove the batch dimensions
+        image = image.reshape(-1, c, h, w)
+        # Now pass the image into CLIP visual encoder
+        image_features = self.image_encoder(image.type(self.dtype))
+        # Now again attach the batch dimensions
+        image_features = image_features.view(b, t, -1)  # [B, T, 512]
+        # Now take the mean along the temporal direction
+        image_features = image_features.mean(dim=1, keepdim=False)  # image features are now ready
+
+        # Finally, make the text features
+        text_features = self.text_encoder(prompts, tokenized_prompts)
+
+        image_features = image_features / image_features.norm(dim=-1, keepdim=True)
+        text_features = text_features / text_features.norm(dim=-1, keepdim=True)
+        logits = logit_scale * image_features @ text_features.t()
+
+        return logits
+
+
+def returnCLIP(config, logger=None,
+               class_names=None):
+    logger.info(f"Loading CLIP (backbone: {config.MODEL.ARCH})")
+    clip_model = load_clip_to_cpu(config)
+
+    logger.info("Building ViFi-CLIP CLIP")
+    model = ViFiCLIP(config, class_names, clip_model, logger)
+
+    if config.TRAINER.ViFi_CLIP.PROMPT_MODEL:
+        logger.info("Turning off gradients in both the image and the text encoder")
+        name_to_update = "prompt_learner"
+        for name, param in model.named_parameters():
+            if name_to_update not in name:
+                # Make sure that VPT prompts are updated
+                if "VPT" in name:
+                    param.requires_grad_(True)
+                else:
+                    param.requires_grad_(False)
+    else:
+        # Now need to control freezing of CLIP for fine-tuning
+        train_complete_clip = config.TRAINER.ViFi_CLIP.USE
+        if train_complete_clip == "both":
+            logger.info("Turning on gradients for COMPLETE ViFi-CLIP model")
+            for name, param in model.named_parameters():
+                param.requires_grad_(True)
+        else:
+            if train_complete_clip == "image":
+                logger.info("Turning on gradients for image side the ViFi-CLIP model")
+                for name, param in model.named_parameters():
+                    if "image_encoder" in name:  # replace by "text_encoder" incase you want to freeze text
+                        param.requires_grad_(True)
+                    else:
+                        param.requires_grad_(False)
+            else:
+                logger.info("Turning on gradients for TEXT side the ViFi-CLIP model")
+                for name, param in model.named_parameters():
+                    if "text_encoder" in name:  # replace by "text_encoder" incase you want to freeze text
+                        param.requires_grad_(True)
+                    else:
+                        param.requires_grad_(False)
+    # Double check
+    enabled = set()
+    for name, param in model.named_parameters():
+        if param.requires_grad:
+            enabled.add(name)
+    logger.info(f"Parameters to be updated: {enabled}")
+    logger.info(f"Total learnable items: {len(enabled)}")
+    model.float()
+    return model

utils/config.py

@@ -0,0 +1,140 @@
+import os
+import yaml
+from yacs.config import CfgNode as CN
+
+_C = CN()
+
+# Base config files
+_C.BASE = ['']
+
+# -----------------------------------------------------------------------------
+# Data settings
+# -----------------------------------------------------------------------------
+_C.DATA = CN()
+_C.DATA.ROOT = ''
+_C.DATA.TRAIN_FILE = ''
+_C.DATA.VAL_FILE = ''
+_C.DATA.DATASET = 'kinetics400'
+_C.DATA.INPUT_SIZE = 224
+_C.DATA.NUM_FRAMES = 8
+_C.DATA.NUM_CLASSES = 400
+_C.DATA.LABEL_LIST = 'labels/kinetics_400_labels.csv'
+
+# -----------------------------------------------------------------------------
+# Model settings
+# -----------------------------------------------------------------------------
+_C.MODEL = CN()
+_C.MODEL.ARCH = 'ViT-B/32'
+_C.MODEL.DROP_PATH_RATE = 0.
+_C.MODEL.PRETRAINED = None
+_C.MODEL.RESUME = None
+_C.MODEL.FIX_TEXT = True
+
+
+# -----------------------------------------------------------------------------
+# Custom trainer settings
+# -----------------------------------------------------------------------------
+_C.TRAINER = CN()
+# Config for ViFi-CLIP
+_C.TRAINER.ViFi_CLIP = CN()
+_C.TRAINER.ViFi_CLIP.PROMPT_MODEL = False # second stage prompting?
+_C.TRAINER.ViFi_CLIP.N_CTX_VISION = 0  # number of context vectors at the vision branch
+_C.TRAINER.ViFi_CLIP.N_CTX_TEXT = 0  # number of context vectors at the language branch
+_C.TRAINER.ViFi_CLIP.CTX_INIT = "a photo of a"  # initialization words (only for language prompts)
+_C.TRAINER.ViFi_CLIP.PROMPT_DEPTH_VISION = 0  # max 12, min 0, for 0 it will act as shallow vision prompting (first layer)
+_C.TRAINER.ViFi_CLIP.PROMPT_DEPTH_TEXT = 1  # max 12, min 0, for 0 it will act as shallow language prompting (first layer)
+_C.TRAINER.ViFi_CLIP.USE = "both"  # fine-tuning complete CLIP model by default
+_C.TRAINER.ViFi_CLIP.ZS_EVAL = False  # make True only during test mode to evaluate zero-shot vanilla CLIP performance
+# -----------------------------------------------------------------------------
+# Training settings
+# -----------------------------------------------------------------------------
+_C.TRAIN = CN()
+_C.TRAIN.EPOCHS = 30
+_C.TRAIN.WARMUP_EPOCHS = 5
+_C.TRAIN.WEIGHT_DECAY = 0.001
+_C.TRAIN.LR = 8.e-6
+_C.TRAIN.BATCH_SIZE = 8
+_C.TRAIN.ACCUMULATION_STEPS = 1
+_C.TRAIN.LR_SCHEDULER = 'cosine'
+_C.TRAIN.OPTIMIZER = 'adamw'
+_C.TRAIN.OPT_LEVEL = 'O1'
+_C.TRAIN.AUTO_RESUME = False
+_C.TRAIN.USE_CHECKPOINT = False
+
+# -----------------------------------------------------------------------------
+# Augmentation settings
+# -----------------------------------------------------------------------------
+_C.AUG = CN()
+_C.AUG.LABEL_SMOOTH = 0.1
+_C.AUG.COLOR_JITTER = 0.8
+_C.AUG.GRAY_SCALE = 0.2
+_C.AUG.MIXUP = 0.8
+_C.AUG.CUTMIX = 1.0
+_C.AUG.MIXUP_SWITCH_PROB = 0.5
+
+# -----------------------------------------------------------------------------
+# Testing settings
+# -----------------------------------------------------------------------------
+_C.TEST = CN()
+_C.TEST.NUM_CLIP = 1
+_C.TEST.NUM_CROP = 1
+_C.TEST.ONLY_TEST = False
+_C.TEST.MULTI_VIEW_INFERENCE = False
+
+# -----------------------------------------------------------------------------
+# Misc
+# -----------------------------------------------------------------------------
+_C.OUTPUT = ''
+_C.SAVE_FREQ = 1
+_C.PRINT_FREQ = 50
+_C.SEED = 1024
+
+
+
+def _update_config_from_file(config, cfg_file):
+    config.defrost()
+    with open(cfg_file, 'r') as f:
+        yaml_cfg = yaml.load(f, Loader=yaml.FullLoader)
+
+    for cfg in yaml_cfg.setdefault('BASE', ['']):
+        if cfg:
+            _update_config_from_file(
+                config, os.path.join(os.path.dirname(cfg_file), cfg)
+            )
+    print('=> merge config from {}'.format(cfg_file))
+    config.merge_from_file(cfg_file)
+    config.freeze()
+
+
+def update_config(config, args):
+    _update_config_from_file(config, args.config)
+
+    config.defrost()
+    if args.opts:
+        config.merge_from_list(args.opts)
+    # merge from specific arguments
+    if args.batch_size:
+        config.TRAIN.BATCH_SIZE = args.batch_size
+    if args.pretrained:
+        config.MODEL.PRETRAINED = args.pretrained
+    if args.resume:
+        config.MODEL.RESUME = args.resume
+    if args.accumulation_steps:
+        config.TRAIN.ACCUMULATION_STEPS = args.accumulation_steps
+    if args.output:
+        config.OUTPUT = args.output
+    if args.only_test:
+        config.TEST.ONLY_TEST = True
+    # set local rank for distributed training
+    config.LOCAL_RANK = args.local_rank
+    config.freeze()
+
+
+def get_config(args):
+    """Get a yacs CfgNode object with default values."""
+    # Return a clone so that the defaults will not be altered
+    # This is for the "local variable" use pattern
+    config = _C.clone()
+    update_config(config, args)
+
+    return config

utils/logger.py

@@ -0,0 +1,34 @@
+import os
+import sys
+import logging
+import functools
+from termcolor import colored
+
+
+@functools.lru_cache()
+def create_logger(output_dir, dist_rank=0, name=''):
+    # create logger
+    logger = logging.getLogger(name)
+    logger.setLevel(logging.DEBUG)
+    logger.propagate = False
+
+    # create formatter
+    fmt = '[%(asctime)s %(name)s] (%(filename)s %(lineno)d): %(levelname)s %(message)s'
+    color_fmt = colored('[%(asctime)s %(name)s]', 'green') + \
+                colored('(%(filename)s %(lineno)d)', 'yellow') + ': %(levelname)s %(message)s'
+
+    # create console handlers for master process
+    if dist_rank == 0:
+        console_handler = logging.StreamHandler(sys.stdout)
+        console_handler.setLevel(logging.DEBUG)
+        console_handler.setFormatter(
+            logging.Formatter(fmt=color_fmt, datefmt='%Y-%m-%d %H:%M:%S'))
+        logger.addHandler(console_handler)
+
+    # create file handlers
+    file_handler = logging.FileHandler(os.path.join(output_dir, f'log_rank{dist_rank}.txt'), mode='a')
+    file_handler.setLevel(logging.DEBUG)
+    file_handler.setFormatter(logging.Formatter(fmt=fmt, datefmt='%Y-%m-%d %H:%M:%S'))
+    logger.addHandler(file_handler)
+
+    return logger

utils/optimizer.py

@@ -0,0 +1,63 @@
+import copy
+import torch.optim as optim
+from timm.scheduler.cosine_lr import CosineLRScheduler
+import torch.distributed as dist
+
+
+def is_main_process():
+    return dist.get_rank() == 0
+
+
+def check_keywords_in_name(name, keywords=()):
+    isin = False
+    for keyword in keywords:
+        if keyword in name:
+            isin = True
+    return isin
+
+
+def set_weight_decay(model, skip_list=(), skip_keywords=(), weight_decay=0.001, lr=2e-6, have=(), not_have=()):
+    has_decay = []
+    no_decay = []
+    for name, param in model.named_parameters():
+        if not param.requires_grad:
+            continue  # frozen weights
+        if len(have) > 0 and not check_keywords_in_name(name, have):
+            continue
+        if len(not_have) > 0 and check_keywords_in_name(name, not_have):
+            continue
+        if len(param.shape) == 1 or name.endswith(".bias") or (name in skip_list) or \
+                check_keywords_in_name(name, skip_keywords):
+            no_decay.append(param)
+        else:
+            has_decay.append(param)
+
+    return [{'params': has_decay, 'weight_decay': weight_decay, 'lr': lr},
+            {'params': no_decay, 'weight_decay': 0., 'lr': lr}]
+
+
+def build_optimizer(config, model):
+    model = model.module if hasattr(model, 'module') else model
+
+    optimizer = optim.AdamW(model.parameters(), lr=config.TRAIN.LR,
+                            weight_decay=config.TRAIN.WEIGHT_DECAY,
+                            betas=(0.9, 0.98), eps=1e-8, )
+
+    return optimizer
+
+
+def build_scheduler(config, optimizer, n_iter_per_epoch):
+    num_steps = int(config.TRAIN.EPOCHS * n_iter_per_epoch)
+    warmup_steps = int(config.TRAIN.WARMUP_EPOCHS * n_iter_per_epoch)
+
+    lr_scheduler = CosineLRScheduler(
+        optimizer,
+        t_initial=num_steps,
+        lr_min=config.TRAIN.LR / 100,
+        warmup_lr_init=0,
+        warmup_t=warmup_steps,
+        cycle_limit=1,
+        t_in_epochs=False,
+    )
+
+    return lr_scheduler

utils/tools.py

@@ -0,0 +1,123 @@
+import numpy
+import torch.distributed as dist
+import torch
+import clip
+import os
+
+
+def reduce_tensor(tensor, n=None):
+    if n is None:
+        n = dist.get_world_size()
+    rt = tensor.clone()
+    dist.all_reduce(rt, op=dist.ReduceOp.SUM)
+    rt = rt / n
+    return rt
+   
+
+class AverageMeter:
+    """Computes and stores the average and current value"""
+    def __init__(self):
+        self.reset()
+
+    def reset(self):
+        self.val = 0
+        self.avg = 0
+        self.sum = 0
+        self.count = 0
+
+    def update(self, val, n=1):
+        self.val = val
+        self.sum += val * n
+        self.count += n
+        self.avg = self.sum / self.count
+    
+    def sync(self):
+        rank = dist.get_rank()
+        world_size = dist.get_world_size()
+        val = torch.tensor(self.val).cuda()
+        sum_v = torch.tensor(self.sum).cuda()
+        count = torch.tensor(self.count).cuda()
+        self.val = reduce_tensor(val, world_size).item()
+        self.sum = reduce_tensor(sum_v, 1).item()
+        self.count = reduce_tensor(count, 1).item()
+        self.avg = self.sum / self.count
+
+
+def epoch_saving(config, epoch, model,  max_accuracy, optimizer, lr_scheduler, logger, working_dir, is_best):
+    save_state = {'model': model.state_dict(),
+                  'optimizer': optimizer.state_dict(),
+                  'lr_scheduler': lr_scheduler.state_dict(),
+                  'max_accuracy': max_accuracy,
+                  'epoch': epoch,
+                  'config': config}
+    
+    save_path = os.path.join(working_dir, f'ckpt_epoch_{epoch}.pth')
+    logger.info(f"{save_path} saving......")
+    torch.save(save_state, save_path)
+    logger.info(f"{save_path} saved !!!")
+    if is_best:
+        best_path = os.path.join(working_dir, f'best.pth')
+        torch.save(save_state, best_path)
+        logger.info(f"{best_path} saved !!!")
+
+
+def load_checkpoint(config, model, optimizer, lr_scheduler, logger):
+    if os.path.isfile(config.MODEL.RESUME): 
+        logger.info(f"==============> Resuming form {config.MODEL.RESUME}....................")
+        checkpoint = torch.load(config.MODEL.RESUME, map_location='cpu')
+        load_state_dict = checkpoint['model']
+
+        # now remove the unwanted keys:
+        if "module.prompt_learner.token_prefix" in load_state_dict:
+            del load_state_dict["module.prompt_learner.token_prefix"]
+
+        if "module.prompt_learner.token_suffix" in load_state_dict:
+            del load_state_dict["module.prompt_learner.token_suffix"]
+
+        if "module.prompt_learner.complete_text_embeddings" in load_state_dict:
+            del load_state_dict["module.prompt_learner.complete_text_embeddings"]
+
+        msg = model.load_state_dict(load_state_dict, strict=False)
+        logger.info(f"resume model: {msg}")
+
+        try:
+            optimizer.load_state_dict(checkpoint['optimizer'])
+            lr_scheduler.load_state_dict(checkpoint['lr_scheduler'])
+
+            start_epoch = checkpoint['epoch'] + 1
+            max_accuracy = checkpoint['max_accuracy']
+
+            logger.info(f"=> loaded successfully '{config.MODEL.RESUME}' (epoch {checkpoint['epoch']})")
+            
+            del checkpoint
+            torch.cuda.empty_cache()
+
+            return start_epoch, max_accuracy
+        except:
+            del checkpoint
+            torch.cuda.empty_cache()
+            return 0, 0.
+
+    else:
+        logger.info(("=> no checkpoint found at '{}'".format(config.MODEL.RESUME)))
+        return 0, 0
+
+
+def auto_resume_helper(output_dir):
+    checkpoints = os.listdir(output_dir)
+    checkpoints = [ckpt for ckpt in checkpoints if ckpt.endswith('pth')]
+    print(f"All checkpoints founded in {output_dir}: {checkpoints}")
+    if len(checkpoints) > 0:
+        latest_checkpoint = max([os.path.join(output_dir, d) for d in checkpoints], key=os.path.getmtime)
+        print(f"The latest checkpoint founded: {latest_checkpoint}")
+        resume_file = latest_checkpoint
+    else:
+        resume_file = None
+    return resume_file
+
+
+def generate_text(data):
+    text_aug = f"{{}}"
+    classes = torch.cat([clip.tokenize(text_aug.format(c), context_length=77) for i, c in data.classes])
+
+    return classes

val.csv

@@ -0,0 +1,4616 @@
+0000.mp4,0
+0003.mp4,0
+0005.mp4,0
+0006.mp4,0
+0007.mp4,0
+0008.mp4,0
+0009.mp4,0
+0010.mp4,0
+0011.mp4,0
+0012.mp4,0
+0013.mp4,0
+0014.mp4,0
+0015.mp4,0
+0016.mp4,0
+0017.mp4,0
+0018.mp4,0
+0019.mp4,0
+0020.mp4,0
+0021.mp4,0
+0022.mp4,0
+0023.mp4,0
+0024.mp4,0
+0025.mp4,0
+0026.mp4,0
+0027.mp4,0
+0028.mp4,0
+0029.mp4,0
+0030.mp4,0
+0031.mp4,0
+0032.mp4,0
+0033.mp4,0
+0034.mp4,0
+0035.mp4,0
+0036.mp4,0
+0037.mp4,0
+0038.mp4,0
+0039.mp4,0
+0040.mp4,0
+0041.mp4,0
+0042.mp4,0
+0043.mp4,0
+0044.mp4,0
+0045.mp4,0
+0046.mp4,0
+0047.mp4,0
+0048.mp4,0
+0049.mp4,0
+0050.mp4,0
+0051.mp4,0
+0052.mp4,0
+0053.mp4,0
+0054.mp4,0
+0055.mp4,0
+0056.mp4,0
+0057.mp4,0
+0058.mp4,0
+0059.mp4,0
+0060.mp4,0
+0061.mp4,0
+0062.mp4,0
+0063.mp4,0
+0064.mp4,0
+0065.mp4,0
+0066.mp4,0
+0067.mp4,0
+0068.mp4,0
+0069.mp4,0
+0070.mp4,0
+0071.mp4,0
+0072.mp4,0
+0073.mp4,0
+0074.mp4,0
+0075.mp4,0
+0076.mp4,0
+0077.mp4,0
+0078.mp4,0
+0079.mp4,0
+0080.mp4,0
+0081.mp4,0
+0082.mp4,0
+0083.mp4,0
+0084.mp4,0
+0085.mp4,0
+0086.mp4,0
+0087.mp4,0
+0088.mp4,0
+0089.mp4,0
+0090.mp4,0
+0091.mp4,0
+0092.mp4,0
+0093.mp4,0
+0094.mp4,0
+0095.mp4,0
+0096.mp4,0
+0097.mp4,0
+0098.mp4,0
+0099.mp4,0
+0100.mp4,0
+0101.mp4,0
+0102.mp4,0
+0103.mp4,0
+0104.mp4,0
+0105.mp4,0
+0106.mp4,0
+0107.mp4,0
+0108.mp4,0
+0109.mp4,0
+0110.mp4,0
+0111.mp4,0
+0112.mp4,0
+0113.mp4,0
+0114.mp4,0
+0115.mp4,0
+0116.mp4,0
+0117.mp4,0
+0118.mp4,0
+0119.mp4,0
+0120.mp4,0
+0121.mp4,0
+0122.mp4,0
+0123.mp4,0
+0124.mp4,0
+0125.mp4,0
+0126.mp4,0
+0127.mp4,0
+0128.mp4,0
+0129.mp4,0
+0130.mp4,0
+0131.mp4,0
+0132.mp4,0
+0133.mp4,0
+0134.mp4,0
+0135.mp4,0
+0136.mp4,0
+0137.mp4,0
+0138.mp4,0
+0139.mp4,0
+0140.mp4,0
+0141.mp4,0
+0142.mp4,0
+0143.mp4,0
+0144.mp4,0
+0145.mp4,0
+0146.mp4,0
+0147.mp4,0
+0148.mp4,0
+0149.mp4,0
+0150.mp4,0
+0151.mp4,0
+0152.mp4,0
+0153.mp4,0
+0154.mp4,0
+0155.mp4,0
+0156.mp4,0
+0157.mp4,0
+0158.mp4,0
+0159.mp4,0
+0160.mp4,0
+0161.mp4,0
+0162.mp4,0
+0163.mp4,0
+0164.mp4,0
+0165.mp4,0
+0166.mp4,0
+0167.mp4,0
+0168.mp4,0
+0169.mp4,0
+0170.mp4,0
+0171.mp4,0
+0172.mp4,0
+0173.mp4,0
+0174.mp4,0
+0175.mp4,0
+0176.mp4,0
+0177.mp4,0
+0178.mp4,0
+0179.mp4,0
+0180.mp4,0
+0181.mp4,0
+0182.mp4,0
+0183.mp4,0
+0184.mp4,0
+0185.mp4,0
+0186.mp4,0
+0187.mp4,0
+0188.mp4,0
+0189.mp4,0
+0190.mp4,0
+0191.mp4,0
+0192.mp4,0
+0193.mp4,0
+0194.mp4,0
+0195.mp4,0
+0196.mp4,0
+0197.mp4,0
+0198.mp4,0
+0199.mp4,0
+0200.mp4,0
+0201.mp4,0
+0202.mp4,0
+0203.mp4,0
+0204.mp4,0
+0205.mp4,0
+0206.mp4,0
+0207.mp4,0
+0208.mp4,0
+0209.mp4,0
+0210.mp4,0
+0211.mp4,0
+0212.mp4,0
+0213.mp4,0
+0214.mp4,0
+0215.mp4,0
+0216.mp4,0
+0217.mp4,0
+0218.mp4,0
+0219.mp4,0
+0220.mp4,0
+0221.mp4,0
+0222.mp4,0
+0223.mp4,0
+0224.mp4,0
+0225.mp4,0
+0226.mp4,0
+0227.mp4,0
+0228.mp4,0
+0229.mp4,0
+0230.mp4,0
+0231.mp4,0
+0232.mp4,0
+0233.mp4,0
+0234.mp4,0
+0235.mp4,0
+0236.mp4,0
+0237.mp4,0
+0238.mp4,0
+0239.mp4,0
+0240.mp4,0
+0241.mp4,0
+0242.mp4,0
+0243.mp4,0
+0244.mp4,0
+0245.mp4,0
+0246.mp4,0
+0247.mp4,0
+0248.mp4,0
+0249.mp4,0
+0250.mp4,0
+0251.mp4,0
+0252.mp4,0
+0253.mp4,0
+0254.mp4,0
+0255.mp4,0
+0256.mp4,0
+0257.mp4,0
+0258.mp4,0
+0259.mp4,0
+0260.mp4,0
+0261.mp4,0
+0262.mp4,0
+0263.mp4,0
+0264.mp4,0
+0265.mp4,0
+0266.mp4,0
+0267.mp4,0
+0268.mp4,0
+0269.mp4,0
+0270.mp4,0
+0271.mp4,0
+0272.mp4,0
+0273.mp4,0
+0274.mp4,0
+0275.mp4,0
+0276.mp4,0
+0277.mp4,0
+0278.mp4,0
+0279.mp4,0
+0280.mp4,0
+0281.mp4,0
+0282.mp4,0
+0283.mp4,0
+0284.mp4,0
+0285.mp4,0
+0286.mp4,0
+0287.mp4,0
+0288.mp4,0
+0289.mp4,0
+0290.mp4,0
+0291.mp4,0
+0292.mp4,0
+0293.mp4,0
+0294.mp4,0
+0295.mp4,0
+0296.mp4,0
+0297.mp4,0
+0298.mp4,0
+0299.mp4,0
+0300.mp4,0
+0301.mp4,0
+0302.mp4,0
+0303.mp4,0
+0304.mp4,0
+0305.mp4,0
+0306.mp4,0
+0307.mp4,0
+0308.mp4,0
+0309.mp4,0
+0310.mp4,0
+0311.mp4,0
+0312.mp4,0
+0313.mp4,0
+0314.mp4,0
+0315.mp4,0
+0316.mp4,0
+0317.mp4,0
+0318.mp4,0
+0319.mp4,0
+0320.mp4,0
+0321.mp4,0
+0322.mp4,0
+0323.mp4,0
+0324.mp4,0
+0325.mp4,0
+0326.mp4,0
+0327.mp4,0
+0328.mp4,0
+0329.mp4,0
+0330.mp4,0
+0331.mp4,0
+0332.mp4,0
+0333.mp4,0
+0334.mp4,0
+0335.mp4,0
+0336.mp4,0
+0337.mp4,0
+0338.mp4,0
+0339.mp4,0
+0340.mp4,0
+0341.mp4,0
+0342.mp4,0
+0343.mp4,0
+0344.mp4,0
+0345.mp4,0
+0346.mp4,0
+0347.mp4,0
+0348.mp4,0
+0349.mp4,0
+0350.mp4,0
+0351.mp4,0
+0352.mp4,0
+0353.mp4,0
+0354.mp4,0
+0355.mp4,0
+0356.mp4,0
+0357.mp4,0
+0358.mp4,0
+0359.mp4,0
+0360.mp4,0
+0361.mp4,0
+0362.mp4,0
+0363.mp4,0
+0364.mp4,0
+0365.mp4,0
+0366.mp4,0
+0367.mp4,0
+0368.mp4,0
+0369.mp4,0
+0370.mp4,0
+0371.mp4,0
+0372.mp4,0
+0373.mp4,0
+0374.mp4,0
+0375.mp4,0
+0376.mp4,0
+0377.mp4,0
+0378.mp4,0
+0379.mp4,0
+0380.mp4,0
+0381.mp4,0
+0382.mp4,0
+0383.mp4,0
+0384.mp4,0
+0385.mp4,0
+0386.mp4,0
+0387.mp4,0
+0388.mp4,0
+0389.mp4,0
+0390.mp4,0
+0391.mp4,0
+0392.mp4,0
+0393.mp4,0
+0394.mp4,0
+0395.mp4,0
+0396.mp4,0
+0397.mp4,0
+0398.mp4,0
+0399.mp4,0
+0400.mp4,0
+0401.mp4,0
+0402.mp4,0
+0403.mp4,0
+0404.mp4,0
+0405.mp4,0
+0406.mp4,0
+0407.mp4,0
+0408.mp4,0
+0409.mp4,0
+0410.mp4,0
+0411.mp4,0
+0412.mp4,0
+0413.mp4,0
+0414.mp4,0
+0415.mp4,0
+0416.mp4,0
+0417.mp4,0
+0418.mp4,0
+0419.mp4,0
+0420.mp4,0
+0421.mp4,0
+0422.mp4,0
+0423.mp4,0
+0424.mp4,0
+0425.mp4,0
+0426.mp4,0
+0427.mp4,0
+0428.mp4,0
+0429.mp4,0
+0430.mp4,0
+0431.mp4,0
+0432.mp4,0
+0433.mp4,0
+0434.mp4,0
+0435.mp4,0
+0436.mp4,0
+0437.mp4,0
+0438.mp4,0
+0439.mp4,0
+0440.mp4,0
+0441.mp4,0
+0442.mp4,0
+0443.mp4,0
+0444.mp4,0
+0445.mp4,0
+0446.mp4,0
+0447.mp4,0
+0448.mp4,0
+0449.mp4,0
+0450.mp4,0
+0451.mp4,0
+0452.mp4,0
+0453.mp4,0
+0454.mp4,0
+0455.mp4,0
+0456.mp4,0
+0457.mp4,0
+0458.mp4,0
+0459.mp4,0
+0460.mp4,0
+0461.mp4,0
+0462.mp4,0
+0463.mp4,0
+0464.mp4,0
+0465.mp4,0
+0466.mp4,0
+0467.mp4,0
+0468.mp4,0
+0469.mp4,0
+0470.mp4,0
+0471.mp4,0
+0472.mp4,0
+0473.mp4,0
+0474.mp4,0
+0475.mp4,0
+0476.mp4,0
+0477.mp4,0
+0478.mp4,0
+0479.mp4,0
+0480.mp4,0
+0481.mp4,0
+0482.mp4,0
+0483.mp4,0
+0484.mp4,0
+0485.mp4,0
+0486.mp4,0
+0487.mp4,0
+0488.mp4,0
+0489.mp4,0
+0490.mp4,0
+0491.mp4,0
+0492.mp4,0
+0493.mp4,0
+0494.mp4,0
+0495.mp4,0
+0496.mp4,0
+0497.mp4,0
+0498.mp4,0
+0499.mp4,0
+0500.mp4,0
+0501.mp4,0
+0502.mp4,0
+0503.mp4,0
+0504.mp4,0
+0505.mp4,0
+0506.mp4,0
+0507.mp4,0
+0508.mp4,0
+0509.mp4,0
+0510.mp4,0
+0511.mp4,0
+0512.mp4,0
+0513.mp4,0
+0514.mp4,0
+0515.mp4,0
+0516.mp4,0
+0517.mp4,0
+0518.mp4,0
+0519.mp4,0
+0520.mp4,0
+0521.mp4,0
+0522.mp4,0
+0523.mp4,0
+0524.mp4,0
+0525.mp4,0
+0526.mp4,0
+0527.mp4,0
+0528.mp4,0
+0529.mp4,0
+0530.mp4,0
+0531.mp4,0
+0532.mp4,0
+0533.mp4,0
+0534.mp4,0
+0535.mp4,0
+0536.mp4,0
+0537.mp4,0
+0538.mp4,0
+0539.mp4,0
+0540.mp4,0
+0541.mp4,0
+0542.mp4,0
+0543.mp4,0
+0544.mp4,0
+0545.mp4,0
+0546.mp4,0
+0547.mp4,0
+0548.mp4,0
+0549.mp4,0
+0550.mp4,0
+0551.mp4,0
+0552.mp4,0
+0553.mp4,0
+0554.mp4,0
+0555.mp4,0
+0556.mp4,0
+0557.mp4,0
+0558.mp4,0
+0559.mp4,0
+0560.mp4,0
+0561.mp4,0
+0562.mp4,0
+0563.mp4,0
+0564.mp4,0
+0565.mp4,0
+0566.mp4,0
+0567.mp4,0
+0568.mp4,0
+0569.mp4,0
+0570.mp4,0
+0571.mp4,0
+0572.mp4,0
+0573.mp4,0
+0574.mp4,0
+0575.mp4,0
+0576.mp4,0
+0577.mp4,0
+0578.mp4,0
+0579.mp4,0
+0580.mp4,0
+0581.mp4,0
+0582.mp4,0
+0583.mp4,0
+0584.mp4,0
+0585.mp4,0
+0586.mp4,0
+0587.mp4,0
+0588.mp4,0
+0589.mp4,0
+0590.mp4,0
+0591.mp4,0
+0592.mp4,0
+0593.mp4,0
+0594.mp4,0
+0595.mp4,0
+0596.mp4,0
+0597.mp4,0
+0598.mp4,0
+0599.mp4,0
+0600.mp4,0
+0601.mp4,0
+0602.mp4,0
+0603.mp4,0
+0604.mp4,0
+0605.mp4,0
+0606.mp4,0
+0607.mp4,0
+0608.mp4,0
+0609.mp4,0
+0610.mp4,0
+0611.mp4,0
+0612.mp4,0
+0613.mp4,0
+0614.mp4,0
+0615.mp4,0
+0616.mp4,0
+0617.mp4,0
+0618.mp4,0
+0619.mp4,0
+0620.mp4,0
+0621.mp4,0
+0622.mp4,0
+0623.mp4,0
+0624.mp4,0
+0625.mp4,0
+0626.mp4,0
+0627.mp4,0
+0628.mp4,0
+0629.mp4,0
+0630.mp4,0
+0631.mp4,0
+0632.mp4,0
+0633.mp4,0
+0634.mp4,0
+0635.mp4,0
+0636.mp4,0
+0637.mp4,0
+0638.mp4,0
+0639.mp4,0
+0640.mp4,0
+0641.mp4,0
+0642.mp4,0
+0643.mp4,0
+0644.mp4,0
+0645.mp4,0
+0646.mp4,0
+0647.mp4,0
+0648.mp4,0
+0649.mp4,0
+0650.mp4,0
+0651.mp4,0
+0652.mp4,0
+0653.mp4,0
+0654.mp4,0
+0655.mp4,0
+0656.mp4,0
+0657.mp4,0
+0658.mp4,0
+0659.mp4,0
+0660.mp4,0
+0661.mp4,0
+0662.mp4,0
+0663.mp4,0
+0664.mp4,0
+0665.mp4,0
+0666.mp4,0
+0667.mp4,0
+0668.mp4,0
+0669.mp4,0
+0670.mp4,0
+0671.mp4,0
+0672.mp4,0
+0673.mp4,0
+0674.mp4,0
+0675.mp4,0
+0676.mp4,0
+0677.mp4,0
+0678.mp4,0
+0679.mp4,0
+0680.mp4,0
+0681.mp4,0
+0682.mp4,0
+0683.mp4,0
+0684.mp4,0
+0685.mp4,0
+0686.mp4,0
+0687.mp4,0
+0688.mp4,0
+0689.mp4,0
+0690.mp4,0
+0691.mp4,0
+0692.mp4,0
+0693.mp4,0
+0694.mp4,0
+0695.mp4,0
+0696.mp4,0
+0697.mp4,0
+0698.mp4,0
+0699.mp4,0
+0700.mp4,0
+0701.mp4,0
+0702.mp4,0
+0703.mp4,0
+0704.mp4,0
+0705.mp4,0
+0706.mp4,0
+0707.mp4,0
+0708.mp4,0
+0709.mp4,0
+0710.mp4,0
+0711.mp4,0
+0712.mp4,0
+0713.mp4,0
+0714.mp4,0
+0715.mp4,0
+0716.mp4,0
+0717.mp4,0
+0718.mp4,0
+0719.mp4,0
+0720.mp4,0
+0721.mp4,0
+0722.mp4,0
+0723.mp4,0
+0724.mp4,0
+0725.mp4,0
+0726.mp4,0
+0727.mp4,0
+0728.mp4,0
+0729.mp4,0
+0730.mp4,0
+0731.mp4,0
+0732.mp4,0
+0733.mp4,0
+0734.mp4,0
+0735.mp4,0
+0736.mp4,0
+0737.mp4,0
+0738.mp4,0
+0739.mp4,0
+0740.mp4,0
+0741.mp4,0
+0742.mp4,0
+0743.mp4,0
+0744.mp4,0
+0745.mp4,0
+0746.mp4,0
+0747.mp4,0
+0748.mp4,0
+0749.mp4,0
+0750.mp4,0
+0751.mp4,0
+0752.mp4,0
+0753.mp4,0
+0754.mp4,0
+0755.mp4,0
+0756.mp4,0
+0757.mp4,0
+0758.mp4,0
+0759.mp4,0
+0760.mp4,0
+0761.mp4,0
+0762.mp4,0
+0763.mp4,0
+0764.mp4,0
+0765.mp4,0
+0766.mp4,0
+0767.mp4,0
+0768.mp4,0
+0769.mp4,0
+0770.mp4,0
+0771.mp4,0
+0772.mp4,0
+0773.mp4,0
+0774.mp4,0
+0775.mp4,0
+0776.mp4,0
+0777.mp4,0
+0778.mp4,0
+0779.mp4,0
+0780.mp4,0
+0781.mp4,0
+0782.mp4,0
+0783.mp4,0
+0784.mp4,0
+0785.mp4,0
+0786.mp4,0
+0787.mp4,0
+0788.mp4,0
+0789.mp4,0
+0790.mp4,0
+0791.mp4,0
+0792.mp4,0
+0793.mp4,0
+0794.mp4,0
+0795.mp4,0
+0796.mp4,0
+0797.mp4,0
+0798.mp4,0
+0799.mp4,0
+0800.mp4,0
+0801.mp4,0
+0802.mp4,0
+0803.mp4,0
+0804.mp4,0
+0805.mp4,0
+0806.mp4,0
+0807.mp4,0
+0808.mp4,0
+0809.mp4,0
+0810.mp4,0
+0811.mp4,0
+0812.mp4,0
+0813.mp4,0
+0814.mp4,0
+0815.mp4,0
+0816.mp4,0
+0817.mp4,0
+0818.mp4,0
+0819.mp4,0
+0820.mp4,0
+0821.mp4,0
+0822.mp4,0
+0823.mp4,0
+0824.mp4,0
+0825.mp4,0
+0826.mp4,0
+0827.mp4,0
+0828.mp4,0
+0829.mp4,0
+0830.mp4,0
+0831.mp4,0
+0832.mp4,0
+0833.mp4,0
+0834.mp4,0
+0835.mp4,0
+0836.mp4,0
+0837.mp4,0
+0838.mp4,0
+0839.mp4,0
+0840.mp4,0
+0841.mp4,0
+0842.mp4,0
+0843.mp4,0
+0844.mp4,0
+0845.mp4,0
+0846.mp4,0
+0847.mp4,0
+0848.mp4,0
+0849.mp4,0
+0850.mp4,0
+0851.mp4,0
+0852.mp4,0
+0853.mp4,0
+0854.mp4,0
+0855.mp4,0
+0856.mp4,0
+0857.mp4,0
+0858.mp4,0
+0859.mp4,0
+0860.mp4,0
+0861.mp4,0
+0862.mp4,0
+0863.mp4,0
+0864.mp4,0
+0865.mp4,0
+0866.mp4,0
+0867.mp4,0
+0868.mp4,0
+0869.mp4,0
+0870.mp4,0
+0871.mp4,0
+0872.mp4,0
+0873.mp4,0
+0874.mp4,0
+0875.mp4,0
+0876.mp4,0
+0877.mp4,0
+0878.mp4,0
+0879.mp4,0
+0880.mp4,0
+0881.mp4,0
+0882.mp4,0
+0883.mp4,0
+0884.mp4,0
+0885.mp4,0
+0886.mp4,0
+0887.mp4,0
+0888.mp4,0
+0889.mp4,0
+0890.mp4,0
+0891.mp4,0
+0892.mp4,0
+0893.mp4,0
+0894.mp4,0
+0895.mp4,0
+0896.mp4,0
+0897.mp4,0
+0898.mp4,0
+0899.mp4,0
+0900.mp4,0
+0901.mp4,0
+0902.mp4,0
+0903.mp4,0
+0904.mp4,0
+0905.mp4,0
+0906.mp4,0
+0907.mp4,0
+0908.mp4,0
+0909.mp4,0
+0910.mp4,0
+0911.mp4,0
+0912.mp4,0
+0913.mp4,0
+0914.mp4,0
+0915.mp4,0
+0916.mp4,0
+0917.mp4,0
+0918.mp4,0
+0919.mp4,0
+0920.mp4,0
+0921.mp4,0
+0922.mp4,0
+0923.mp4,0
+0924.mp4,0
+0925.mp4,0
+0926.mp4,0
+0927.mp4,0
+0928.mp4,0
+0929.mp4,0
+0930.mp4,0
+0931.mp4,0
+0932.mp4,0
+0933.mp4,0
+0934.mp4,0
+0935.mp4,0
+0936.mp4,0
+0937.mp4,0
+0938.mp4,0
+0939.mp4,0
+0940.mp4,0
+0941.mp4,0
+0942.mp4,0
+0943.mp4,0
+0944.mp4,0
+0945.mp4,0
+0946.mp4,0
+0947.mp4,0
+0948.mp4,0
+0949.mp4,0
+0950.mp4,0
+0951.mp4,0
+0952.mp4,0
+0953.mp4,0
+0954.mp4,0
+0955.mp4,0
+0956.mp4,0
+0957.mp4,0
+0958.mp4,0
+0959.mp4,0
+0960.mp4,0
+0961.mp4,0
+0962.mp4,0
+0963.mp4,0
+0964.mp4,0
+0965.mp4,0
+0966.mp4,0
+0967.mp4,0
+0968.mp4,0
+0969.mp4,0
+0970.mp4,0
+0971.mp4,0
+0972.mp4,0
+0973.mp4,0
+0974.mp4,0
+0975.mp4,0
+0976.mp4,0
+0977.mp4,0
+0978.mp4,0
+0979.mp4,0
+0980.mp4,0
+0981.mp4,0
+0982.mp4,0
+0983.mp4,0
+0984.mp4,0
+0985.mp4,0
+0986.mp4,0
+0987.mp4,0
+0988.mp4,0
+0989.mp4,0
+0990.mp4,0
+0991.mp4,0
+0992.mp4,0
+0993.mp4,0
+0994.mp4,0
+0995.mp4,0
+0996.mp4,0
+0997.mp4,0
+0998.mp4,0
+0999.mp4,0
+1000.mp4,0
+1001.mp4,0
+1002.mp4,0
+1003.mp4,0
+1004.mp4,0
+1005.mp4,0
+1006.mp4,0
+1007.mp4,0
+1008.mp4,0
+1009.mp4,0
+1010.mp4,0
+1011.mp4,0
+1012.mp4,0
+1013.mp4,0
+1014.mp4,0
+1015.mp4,0
+1016.mp4,0
+1017.mp4,0
+1018.mp4,0
+1019.mp4,0
+1020.mp4,0
+1021.mp4,0
+1022.mp4,0
+1023.mp4,0
+1024.mp4,0
+1025.mp4,0
+1026.mp4,0
+1027.mp4,0
+1028.mp4,0
+1029.mp4,0
+1030.mp4,0
+1031.mp4,0
+1032.mp4,0
+1033.mp4,0
+1034.mp4,0
+1035.mp4,0
+1036.mp4,0
+1037.mp4,0
+1038.mp4,0
+1039.mp4,0
+1040.mp4,0
+1041.mp4,0
+1042.mp4,0
+1044.mp4,0
+1045.mp4,0
+1046.mp4,0
+1047.mp4,0
+1048.mp4,0
+1049.mp4,0
+1050.mp4,0
+1052.mp4,0
+1053.mp4,0
+1054.mp4,0
+1055.mp4,0
+1056.mp4,0
+1057.mp4,0
+1058.mp4,0
+1060.mp4,0
+1061.mp4,0
+1062.mp4,0
+1063.mp4,0
+1064.mp4,0
+1066.mp4,0
+1067.mp4,0
+1068.mp4,0
+1070.mp4,0
+1071.mp4,0
+1074.mp4,0
+1075.mp4,0
+1076.mp4,0
+1078.mp4,0
+1079.mp4,0
+1081.mp4,0
+1082.mp4,0
+1083.mp4,0
+1084.mp4,0
+1086.mp4,0
+1087.mp4,0
+1088.mp4,0
+1089.mp4,0
+1090.mp4,0
+1091.mp4,0
+1092.mp4,0
+1093.mp4,0
+1094.mp4,0
+1095.mp4,0
+1096.mp4,0
+1097.mp4,0
+1098.mp4,0
+1099.mp4,0
+1101.mp4,0
+1102.mp4,0
+1103.mp4,0
+1105.mp4,0
+1106.mp4,0
+1107.mp4,0
+1108.mp4,0
+1109.mp4,0
+1110.mp4,0
+1111.mp4,0
+1112.mp4,0
+1113.mp4,0
+1114.mp4,0
+1115.mp4,0
+1117.mp4,0
+1118.mp4,0
+1119.mp4,0
+1121.mp4,0
+1122.mp4,0
+1123.mp4,0
+1124.mp4,0
+1125.mp4,0
+1126.mp4,0
+1127.mp4,0
+1128.mp4,0
+1130.mp4,0
+1131.mp4,0
+1133.mp4,0
+1134.mp4,0
+1135.mp4,0
+1136.mp4,0
+1138.mp4,0
+1139.mp4,0
+1140.mp4,0
+1141.mp4,0
+1142.mp4,0
+1143.mp4,0
+1144.mp4,0
+1145.mp4,0
+1146.mp4,0
+1147.mp4,0
+1149.mp4,0
+1150.mp4,0
+1152.mp4,0
+1153.mp4,0
+1156.mp4,0
+1157.mp4,0
+1158.mp4,0
+1159.mp4,0
+1160.mp4,0
+1161.mp4,0
+1162.mp4,0
+1163.mp4,0
+1164.mp4,0
+1165.mp4,0
+1166.mp4,0
+1167.mp4,0
+1168.mp4,0
+1171.mp4,0
+1172.mp4,0
+1173.mp4,0
+1174.mp4,0
+1176.mp4,0
+1177.mp4,0
+1178.mp4,0
+1180.mp4,0
+1181.mp4,0
+1182.mp4,0
+1183.mp4,0
+1185.mp4,0
+1186.mp4,0
+1187.mp4,0
+1188.mp4,0
+1189.mp4,0
+1192.mp4,0
+1193.mp4,0
+1194.mp4,0
+1195.mp4,0
+1196.mp4,0
+1197.mp4,0
+1198.mp4,0
+1199.mp4,0
+1200.mp4,0
+1201.mp4,0
+1202.mp4,0
+1203.mp4,0
+1204.mp4,0
+1205.mp4,0
+1206.mp4,0
+1207.mp4,0
+1209.mp4,0
+1211.mp4,0
+1212.mp4,0
+1213.mp4,0
+1214.mp4,0
+1215.mp4,0
+1217.mp4,0
+1218.mp4,0
+1219.mp4,0
+1220.mp4,0
+1221.mp4,0
+1222.mp4,0
+1223.mp4,0
+1224.mp4,0
+1225.mp4,0
+1226.mp4,0
+1227.mp4,0
+1228.mp4,0
+1229.mp4,0
+1230.mp4,0
+1231.mp4,0
+1232.mp4,0
+1233.mp4,0
+1234.mp4,0
+1235.mp4,0
+1237.mp4,0
+1238.mp4,0
+1239.mp4,0
+1240.mp4,0
+1241.mp4,0
+1242.mp4,0
+1243.mp4,0
+1244.mp4,0
+1245.mp4,0
+1246.mp4,0
+1248.mp4,0
+1249.mp4,0
+1250.mp4,0
+1252.mp4,0
+1255.mp4,0
+1256.mp4,0
+1257.mp4,0
+1258.mp4,0
+1259.mp4,0
+1260.mp4,0
+1261.mp4,0
+1262.mp4,0
+1263.mp4,0
+1264.mp4,0
+1266.mp4,0
+1267.mp4,0
+1268.mp4,0
+1269.mp4,0
+1270.mp4,0
+1271.mp4,0
+1274.mp4,0
+1275.mp4,0
+1276.mp4,0
+1277.mp4,0
+1279.mp4,0
+1282.mp4,0
+1283.mp4,0
+1284.mp4,0
+1285.mp4,0
+1286.mp4,0
+1287.mp4,0
+1288.mp4,0
+1289.mp4,0
+1290.mp4,0
+1291.mp4,0
+1292.mp4,0
+1293.mp4,0
+1295.mp4,0
+1296.mp4,0
+1297.mp4,0
+1298.mp4,0
+1299.mp4,0
+1300.mp4,0
+1301.mp4,0
+1302.mp4,0
+1303.mp4,0
+1304.mp4,0
+1305.mp4,0
+1306.mp4,0
+1307.mp4,0
+1308.mp4,0
+1309.mp4,0
+1311.mp4,0
+1312.mp4,0
+1313.mp4,0
+1314.mp4,0
+1315.mp4,0
+1316.mp4,0
+1317.mp4,0
+1318.mp4,0
+1319.mp4,0
+1321.mp4,0
+1322.mp4,0
+1323.mp4,0
+1324.mp4,0
+1325.mp4,0
+1326.mp4,0
+1327.mp4,0
+1328.mp4,0
+1329.mp4,0
+1330.mp4,0
+1331.mp4,0
+1332.mp4,0
+1333.mp4,0
+1334.mp4,0
+1335.mp4,0
+1336.mp4,0
+1337.mp4,0
+1338.mp4,0
+1339.mp4,0
+1340.mp4,0
+1342.mp4,0
+1343.mp4,0
+1344.mp4,0
+1345.mp4,0
+1346.mp4,0
+1347.mp4,0
+1348.mp4,0
+1349.mp4,0
+1351.mp4,0
+1352.mp4,0
+1354.mp4,0
+1355.mp4,0
+1356.mp4,0
+1358.mp4,0
+1359.mp4,0
+1360.mp4,0
+1361.mp4,0
+1363.mp4,0
+1364.mp4,0
+1365.mp4,0
+1366.mp4,0
+1367.mp4,0
+1368.mp4,0
+1369.mp4,0
+1370.mp4,0
+1371.mp4,0
+1372.mp4,0
+1373.mp4,0
+1374.mp4,0
+1375.mp4,0
+1376.mp4,0
+1377.mp4,0
+1378.mp4,0
+1379.mp4,0
+1380.mp4,0
+1381.mp4,0
+1382.mp4,0
+1384.mp4,0
+1385.mp4,0
+1386.mp4,0
+1387.mp4,0
+1389.mp4,0
+1390.mp4,0
+1392.mp4,0
+1393.mp4,0
+1394.mp4,0
+1395.mp4,0
+1396.mp4,0
+1397.mp4,0
+1398.mp4,0
+1399.mp4,0
+1400.mp4,0
+1401.mp4,0
+1402.mp4,0
+1403.mp4,0
+1405.mp4,0
+1406.mp4,0
+1408.mp4,0
+1409.mp4,0
+1410.mp4,0
+1411.mp4,0
+1412.mp4,0
+1413.mp4,0
+1414.mp4,0
+1415.mp4,0
+1416.mp4,0
+1417.mp4,0
+1418.mp4,0
+1419.mp4,0
+1420.mp4,0
+1421.mp4,0
+1422.mp4,0
+1423.mp4,0
+1424.mp4,0
+1425.mp4,0
+1426.mp4,0
+1428.mp4,0
+1429.mp4,0
+1430.mp4,0
+1431.mp4,0
+1432.mp4,0
+1433.mp4,0
+1434.mp4,0
+1435.mp4,0
+1436.mp4,0
+1438.mp4,0
+1439.mp4,0
+1440.mp4,0
+1441.mp4,0
+1442.mp4,0
+1443.mp4,0
+1445.mp4,0
+1446.mp4,0
+1447.mp4,0
+1449.mp4,0
+1450.mp4,0
+1451.mp4,0
+1452.mp4,0
+1455.mp4,0
+1456.mp4,0
+1457.mp4,0
+1458.mp4,0
+1459.mp4,0
+1462.mp4,0
+1464.mp4,0
+1465.mp4,0
+1466.mp4,0
+1467.mp4,0
+1468.mp4,0
+1469.mp4,0
+1470.mp4,0
+1472.mp4,0
+1473.mp4,0
+1474.mp4,0
+1475.mp4,0
+1476.mp4,0
+1477.mp4,0
+1478.mp4,0
+1479.mp4,0
+1480.mp4,0
+1481.mp4,0
+1482.mp4,0
+1483.mp4,0
+1484.mp4,0
+1487.mp4,0
+1488.mp4,0
+1489.mp4,0
+1491.mp4,0
+1492.mp4,0
+1493.mp4,0
+1494.mp4,0
+1495.mp4,0
+1496.mp4,0
+1497.mp4,0
+1498.mp4,0
+1499.mp4,0
+1500.mp4,0
+1501.mp4,0
+1502.mp4,0
+1503.mp4,0
+1504.mp4,0
+1505.mp4,0
+1507.mp4,0
+1508.mp4,0
+1510.mp4,0
+1511.mp4,0
+1512.mp4,0
+1514.mp4,0
+1515.mp4,0
+1516.mp4,0
+1517.mp4,0
+1518.mp4,0
+1519.mp4,0
+1520.mp4,0
+1521.mp4,0
+1522.mp4,0
+1524.mp4,0
+1525.mp4,0
+1526.mp4,0
+1527.mp4,0
+1528.mp4,0
+1530.mp4,0
+1531.mp4,0
+1532.mp4,0
+1533.mp4,0
+1534.mp4,0
+1535.mp4,0
+1536.mp4,0
+1537.mp4,0
+1538.mp4,0
+1539.mp4,0
+1541.mp4,0
+1542.mp4,0
+1543.mp4,0
+1544.mp4,0
+1545.mp4,0
+1546.mp4,0
+1547.mp4,0
+1548.mp4,0
+1549.mp4,0
+1550.mp4,0
+1551.mp4,0
+1552.mp4,0
+1553.mp4,0
+1554.mp4,0
+1556.mp4,0
+1557.mp4,0
+1558.mp4,0
+1559.mp4,0
+1560.mp4,0
+1562.mp4,0
+1563.mp4,0
+1564.mp4,0
+1565.mp4,0
+1566.mp4,0
+1567.mp4,0
+1568.mp4,0
+1569.mp4,0
+1570.mp4,0
+1572.mp4,0
+1573.mp4,0
+1574.mp4,0
+1575.mp4,0
+1576.mp4,0
+1578.mp4,0
+1579.mp4,0
+1580.mp4,0
+1581.mp4,0
+1582.mp4,0
+1584.mp4,0
+1585.mp4,0
+1586.mp4,0
+1587.mp4,0
+1588.mp4,0
+1589.mp4,0
+1591.mp4,0
+1592.mp4,0
+1594.mp4,0
+1595.mp4,0
+1596.mp4,0
+1597.mp4,0
+1598.mp4,0
+1599.mp4,0
+1600.mp4,0
+1601.mp4,0
+1602.mp4,0
+1603.mp4,0
+1604.mp4,0
+1605.mp4,0
+1606.mp4,0
+1607.mp4,0
+1608.mp4,0
+1609.mp4,0
+1610.mp4,0
+1611.mp4,0
+1612.mp4,0
+1613.mp4,0
+1614.mp4,0
+1615.mp4,0
+1617.mp4,0
+1618.mp4,0
+1619.mp4,0
+1620.mp4,0
+1621.mp4,0
+1622.mp4,0
+1623.mp4,0
+1624.mp4,0
+1625.mp4,0
+1627.mp4,0
+1628.mp4,0
+1629.mp4,0
+1630.mp4,0
+1633.mp4,0
+1634.mp4,0
+1635.mp4,0
+1636.mp4,0
+1637.mp4,0
+1639.mp4,0
+1640.mp4,0
+1641.mp4,0
+1642.mp4,0
+1643.mp4,0
+1644.mp4,0
+1645.mp4,0
+1646.mp4,0
+1647.mp4,0
+1648.mp4,0
+1649.mp4,0
+1652.mp4,0
+1653.mp4,0
+1654.mp4,0
+1655.mp4,0
+1656.mp4,0
+1657.mp4,0
+1658.mp4,0
+1659.mp4,0
+1660.mp4,0
+1661.mp4,0
+1662.mp4,0
+1663.mp4,0
+1664.mp4,0
+1665.mp4,0
+1667.mp4,0
+1668.mp4,0
+1669.mp4,0
+1671.mp4,0
+1672.mp4,0
+1673.mp4,0
+1674.mp4,0
+1675.mp4,0
+1676.mp4,0
+1677.mp4,0
+1678.mp4,0
+1679.mp4,0
+1680.mp4,0
+1681.mp4,0
+1683.mp4,0
+1684.mp4,0
+1685.mp4,0
+1686.mp4,0
+1687.mp4,0
+1688.mp4,0
+1689.mp4,0
+1690.mp4,0
+1691.mp4,0
+1692.mp4,0
+1693.mp4,0
+1694.mp4,0
+1696.mp4,0
+1697.mp4,0
+1698.mp4,0
+1699.mp4,0
+1701.mp4,0
+1702.mp4,0
+1703.mp4,0
+1704.mp4,0
+1705.mp4,0
+1706.mp4,0
+1707.mp4,0
+1708.mp4,0
+1709.mp4,0
+1710.mp4,0
+1711.mp4,0
+1714.mp4,0
+1715.mp4,0
+1716.mp4,0
+1717.mp4,0
+1718.mp4,0
+1719.mp4,0
+1720.mp4,0
+1721.mp4,0
+1722.mp4,0
+1724.mp4,0
+1725.mp4,0
+1726.mp4,0
+1727.mp4,0
+1729.mp4,0
+1730.mp4,0
+1731.mp4,0
+1732.mp4,0
+1733.mp4,0
+1734.mp4,0
+1735.mp4,0
+1738.mp4,0
+1739.mp4,0
+1740.mp4,0
+1741.mp4,0
+1742.mp4,0
+1743.mp4,0
+1744.mp4,0
+1745.mp4,0
+1746.mp4,0
+1747.mp4,0
+1748.mp4,0
+1749.mp4,0
+1750.mp4,0
+1751.mp4,0
+1753.mp4,0
+1754.mp4,0
+1755.mp4,0
+1756.mp4,0
+1757.mp4,0
+1758.mp4,0
+1759.mp4,0
+1760.mp4,0
+1761.mp4,0
+1762.mp4,0
+1763.mp4,0
+1764.mp4,0
+1765.mp4,0
+1766.mp4,0
+1767.mp4,0
+1768.mp4,0
+1769.mp4,0
+1770.mp4,0
+1771.mp4,0
+1772.mp4,0
+1773.mp4,0
+1774.mp4,0
+1775.mp4,0
+1777.mp4,0
+1778.mp4,0
+1779.mp4,0
+1780.mp4,0
+1781.mp4,0
+1784.mp4,0
+1785.mp4,0
+1786.mp4,0
+1788.mp4,0
+1792.mp4,0
+1794.mp4,0
+1795.mp4,0
+1796.mp4,0
+1797.mp4,0
+1798.mp4,0
+1799.mp4,0
+1801.mp4,0
+1802.mp4,0
+1803.mp4,0
+1804.mp4,0
+1806.mp4,0
+1807.mp4,0
+1808.mp4,0
+1809.mp4,0
+1810.mp4,0
+1811.mp4,0
+1812.mp4,0
+1813.mp4,0
+1814.mp4,0
+1815.mp4,0
+1816.mp4,0
+1817.mp4,0
+1818.mp4,0
+1819.mp4,0
+1820.mp4,0
+1821.mp4,0
+1822.mp4,0
+1823.mp4,0
+1824.mp4,0
+1825.mp4,0
+1826.mp4,0
+1828.mp4,0
+1830.mp4,0
+1831.mp4,0
+1832.mp4,0
+1835.mp4,0
+1836.mp4,0
+1837.mp4,0
+1839.mp4,0
+1840.mp4,0
+1841.mp4,0
+1843.mp4,0
+1844.mp4,0
+1845.mp4,0
+1846.mp4,0
+1847.mp4,0
+1848.mp4,0
+1849.mp4,0
+1850.mp4,0
+1851.mp4,0
+1852.mp4,0
+1853.mp4,0
+1854.mp4,0
+1856.mp4,0
+1858.mp4,0
+1860.mp4,0
+1861.mp4,0
+1862.mp4,0
+1863.mp4,0
+1864.mp4,0
+1865.mp4,0
+1866.mp4,0
+1867.mp4,0
+1868.mp4,0
+1869.mp4,0
+1870.mp4,0
+1871.mp4,0
+1872.mp4,0
+1873.mp4,0
+1874.mp4,0
+1875.mp4,0
+1876.mp4,0
+1877.mp4,0
+1878.mp4,0
+1879.mp4,0
+1880.mp4,0
+1882.mp4,0
+1883.mp4,0
+1884.mp4,0
+1885.mp4,0
+1886.mp4,0
+1887.mp4,0
+1889.mp4,0
+1891.mp4,0
+1892.mp4,0
+1893.mp4,0
+1894.mp4,0
+1895.mp4,0
+1896.mp4,0
+1897.mp4,0
+1898.mp4,0
+1899.mp4,0
+1900.mp4,0
+1901.mp4,0
+1902.mp4,0
+1903.mp4,0
+1905.mp4,0
+1908.mp4,0
+1910.mp4,0
+1911.mp4,0
+1912.mp4,0
+1913.mp4,0
+1914.mp4,0
+1916.mp4,0
+1917.mp4,0
+1918.mp4,0
+1919.mp4,0
+1920.mp4,0
+1921.mp4,0
+1922.mp4,0
+1923.mp4,0
+1924.mp4,0
+1925.mp4,0
+1926.mp4,0
+1928.mp4,0
+1930.mp4,0
+1931.mp4,0
+1932.mp4,0
+1933.mp4,0
+1934.mp4,0
+1937.mp4,0
+1938.mp4,0
+1939.mp4,0
+1940.mp4,0
+1941.mp4,0
+1942.mp4,0
+1943.mp4,0
+1944.mp4,0
+1945.mp4,0
+1946.mp4,0
+1949.mp4,0
+1950.mp4,0
+1951.mp4,0
+1952.mp4,0
+1953.mp4,0
+1954.mp4,0
+1955.mp4,0
+1956.mp4,0
+1958.mp4,0
+1959.mp4,0
+1960.mp4,0
+1961.mp4,0
+1962.mp4,0
+1963.mp4,0
+1964.mp4,0
+1965.mp4,0
+1966.mp4,0
+1967.mp4,0
+1968.mp4,0
+1969.mp4,0
+1970.mp4,0
+1971.mp4,0
+1972.mp4,0
+1974.mp4,0
+1976.mp4,0
+1977.mp4,0
+1978.mp4,0
+1979.mp4,0
+1980.mp4,0
+1981.mp4,0
+1982.mp4,0
+1983.mp4,0
+1985.mp4,0
+1986.mp4,0
+1987.mp4,0
+1988.mp4,0
+1989.mp4,0
+1990.mp4,0
+1991.mp4,0
+1992.mp4,0
+1993.mp4,0
+1994.mp4,0
+1995.mp4,0
+1996.mp4,0
+1998.mp4,0
+1999.mp4,0
+2000.mp4,0
+2001.mp4,0
+2003.mp4,0
+2004.mp4,0
+2005.mp4,0
+2007.mp4,0
+2008.mp4,0
+2010.mp4,0
+2011.mp4,0
+2012.mp4,0
+2013.mp4,0
+2014.mp4,0
+2016.mp4,0
+2019.mp4,0
+2020.mp4,0
+2021.mp4,0
+2022.mp4,0
+2025.mp4,0
+2026.mp4,0
+2027.mp4,0
+2028.mp4,0
+2029.mp4,0
+2030.mp4,0
+2031.mp4,0
+2032.mp4,0
+2035.mp4,0
+2036.mp4,0
+2037.mp4,0
+2039.mp4,0
+2040.mp4,0
+2043.mp4,0
+2044.mp4,0
+2045.mp4,0
+2046.mp4,0
+2047.mp4,0
+2048.mp4,0
+2050.mp4,0
+2053.mp4,0
+2055.mp4,0
+2058.mp4,0
+2059.mp4,0
+2060.mp4,0
+2061.mp4,0
+2063.mp4,0
+2064.mp4,0
+2065.mp4,0
+2066.mp4,0
+2067.mp4,0
+2068.mp4,0
+2069.mp4,0
+2070.mp4,0
+2071.mp4,0
+2072.mp4,0
+2073.mp4,0
+2074.mp4,0
+2075.mp4,0
+2078.mp4,0
+2081.mp4,0
+2082.mp4,0
+2083.mp4,0
+2084.mp4,0
+2085.mp4,0
+2087.mp4,0
+2090.mp4,0
+2091.mp4,0
+2092.mp4,0
+2093.mp4,0
+2094.mp4,0
+2096.mp4,0
+2097.mp4,0
+2098.mp4,0
+2100.mp4,0
+2101.mp4,0
+2102.mp4,0
+2103.mp4,0
+2104.mp4,0
+2105.mp4,0
+2106.mp4,0
+2107.mp4,0
+2108.mp4,0
+2109.mp4,0
+2110.mp4,0
+2111.mp4,0
+2112.mp4,0
+2113.mp4,0
+2114.mp4,0
+2118.mp4,0
+2119.mp4,0
+2120.mp4,0
+2121.mp4,0
+2122.mp4,0
+2123.mp4,0
+2124.mp4,0
+2125.mp4,0
+2126.mp4,0
+2127.mp4,0
+2128.mp4,0
+2129.mp4,0
+2130.mp4,0
+2131.mp4,0
+2133.mp4,0
+2135.mp4,0
+2136.mp4,0
+2138.mp4,0
+2143.mp4,0
+2145.mp4,0
+2146.mp4,0
+2147.mp4,0
+2148.mp4,0
+2149.mp4,0
+2150.mp4,0
+2151.mp4,0
+2152.mp4,0
+2154.mp4,0
+2155.mp4,0
+2156.mp4,0
+2157.mp4,0
+2158.mp4,0
+2160.mp4,0
+2162.mp4,0
+2163.mp4,0
+2164.mp4,0
+2165.mp4,0
+2166.mp4,0
+2168.mp4,0
+2170.mp4,0
+2171.mp4,0
+2172.mp4,0
+2174.mp4,0
+2176.mp4,0
+2177.mp4,0
+2178.mp4,0
+2179.mp4,0
+2183.mp4,0
+2184.mp4,0
+2185.mp4,0
+2186.mp4,0
+2189.mp4,0
+2190.mp4,0
+2191.mp4,0
+2192.mp4,0
+2193.mp4,0
+2194.mp4,0
+2196.mp4,0
+2197.mp4,0
+2198.mp4,0
+2200.mp4,0
+2201.mp4,0
+2204.mp4,0
+2205.mp4,0
+2206.mp4,0
+2207.mp4,0
+2208.mp4,0
+2209.mp4,0
+2210.mp4,0
+2211.mp4,0
+2212.mp4,0
+2213.mp4,0
+2214.mp4,0
+2215.mp4,0
+2216.mp4,0
+2217.mp4,0
+2218.mp4,0
+2219.mp4,0
+2221.mp4,0
+2223.mp4,0
+2224.mp4,0
+2225.mp4,0
+2227.mp4,0
+2229.mp4,0
+2232.mp4,0
+2233.mp4,0
+2235.mp4,0
+2237.mp4,0
+2238.mp4,0
+2239.mp4,0
+2240.mp4,0
+2241.mp4,0
+2242.mp4,0
+2243.mp4,0
+2244.mp4,0
+2245.mp4,0
+2246.mp4,0
+2248.mp4,0
+2249.mp4,0
+2251.mp4,0
+2253.mp4,0
+2254.mp4,0
+2255.mp4,0
+2256.mp4,0
+2257.mp4,0
+2258.mp4,0
+2259.mp4,0
+2260.mp4,0
+2261.mp4,0
+2262.mp4,0
+2264.mp4,0
+2265.mp4,0
+2266.mp4,0
+2268.mp4,0
+2271.mp4,0
+2272.mp4,0
+2273.mp4,0
+2274.mp4,0
+2275.mp4,0
+2276.mp4,0
+2278.mp4,0
+2279.mp4,0
+2283.mp4,0
+2284.mp4,0
+2286.mp4,0
+2287.mp4,0
+2288.mp4,0
+2289.mp4,0
+2290.mp4,0
+2291.mp4,0
+2292.mp4,0
+2293.mp4,0
+2294.mp4,0
+2295.mp4,0
+2296.mp4,0
+2297.mp4,0
+2298.mp4,0
+2299.mp4,0
+2300.mp4,0
+2301.mp4,0
+2304.mp4,0
+2306.mp4,0
+2307.mp4,0
+2308.mp4,0
+2309.mp4,0
+2310.mp4,0
+2311.mp4,0
+2312.mp4,0
+2313.mp4,0
+2314.mp4,0
+2315.mp4,0
+2316.mp4,0
+2317.mp4,0
+2318.mp4,0
+2320.mp4,0
+2321.mp4,0
+2324.mp4,0
+2325.mp4,0
+2326.mp4,0
+2327.mp4,0
+2328.mp4,0
+2329.mp4,0
+2330.mp4,0
+2332.mp4,0
+2333.mp4,0
+2334.mp4,0
+2335.mp4,0
+2336.mp4,0
+2337.mp4,0
+2338.mp4,0
+2339.mp4,0
+2340.mp4,0
+2341.mp4,0
+2343.mp4,0
+2345.mp4,0
+2346.mp4,0
+2347.mp4,0
+2348.mp4,0
+2349.mp4,0
+2350.mp4,0
+2351.mp4,0
+2352.mp4,0
+2353.mp4,0
+2355.mp4,0
+2357.mp4,0
+2358.mp4,0
+2361.mp4,0
+2362.mp4,0
+2363.mp4,0
+2364.mp4,0
+2365.mp4,0
+2366.mp4,0
+2367.mp4,0
+2368.mp4,0
+2369.mp4,0
+2370.mp4,0
+2372.mp4,0
+2373.mp4,0
+2374.mp4,0
+2375.mp4,0
+2377.mp4,0
+2380.mp4,0
+2381.mp4,0
+2385.mp4,0
+2388.mp4,0
+2389.mp4,0
+2390.mp4,0
+2391.mp4,0
+2393.mp4,0
+2394.mp4,0
+2395.mp4,0
+2396.mp4,0
+2397.mp4,0
+2398.mp4,0
+2399.mp4,0
+2400.mp4,0
+2401.mp4,0
+2402.mp4,0
+2403.mp4,0
+2404.mp4,0
+2405.mp4,0
+2407.mp4,0
+2408.mp4,0
+2409.mp4,0
+2410.mp4,0
+2411.mp4,0
+2412.mp4,0
+2413.mp4,0
+2415.mp4,0
+2416.mp4,0
+2418.mp4,0
+2419.mp4,0
+2420.mp4,0
+2421.mp4,0
+2423.mp4,0
+2425.mp4,0
+2428.mp4,0
+2429.mp4,0
+2430.mp4,0
+2431.mp4,0
+2432.mp4,0
+2433.mp4,0
+2434.mp4,0
+2436.mp4,0
+2437.mp4,0
+2438.mp4,0
+2439.mp4,0
+2441.mp4,0
+2442.mp4,0
+2443.mp4,0
+2444.mp4,0
+2445.mp4,0
+2446.mp4,0
+2448.mp4,0
+2450.mp4,0
+2451.mp4,0
+2453.mp4,0
+2454.mp4,0
+2455.mp4,0
+2456.mp4,0
+2457.mp4,0
+2458.mp4,0
+2459.mp4,0
+2460.mp4,0
+2462.mp4,0
+2464.mp4,0
+2465.mp4,0
+2466.mp4,0
+2467.mp4,0
+2468.mp4,0
+2469.mp4,0
+2470.mp4,0
+2471.mp4,0
+2473.mp4,0
+2475.mp4,0
+2476.mp4,0
+2477.mp4,0
+2478.mp4,0
+2479.mp4,0
+2480.mp4,0
+2481.mp4,0
+2482.mp4,0
+2483.mp4,0
+2484.mp4,0
+2486.mp4,0
+2487.mp4,0
+2488.mp4,0
+2489.mp4,0
+2490.mp4,0
+2491.mp4,0
+2492.mp4,0
+2493.mp4,0
+2494.mp4,0
+2495.mp4,0
+2496.mp4,0
+2497.mp4,0
+2498.mp4,0
+2500.mp4,0
+2501.mp4,0
+2502.mp4,0
+2505.mp4,0
+2506.mp4,0
+2508.mp4,0
+2510.mp4,0
+2512.mp4,0
+2513.mp4,0
+2515.mp4,0
+2516.mp4,0
+2519.mp4,0
+2520.mp4,0
+2521.mp4,0
+2522.mp4,0
+2523.mp4,0
+2525.mp4,0
+2526.mp4,0
+2527.mp4,0
+2528.mp4,0
+2534.mp4,0
+2536.mp4,0
+2537.mp4,0
+2538.mp4,0
+2539.mp4,0
+2540.mp4,0
+2541.mp4,0
+2544.mp4,0
+2545.mp4,0
+2546.mp4,0
+2547.mp4,0
+2548.mp4,0
+2549.mp4,0
+2550.mp4,0
+2551.mp4,0
+2552.mp4,0
+2553.mp4,0
+2554.mp4,0
+2555.mp4,0
+2557.mp4,0
+2558.mp4,0
+2559.mp4,0
+2560.mp4,0
+2561.mp4,0
+2563.mp4,0
+2564.mp4,0
+2565.mp4,0
+2566.mp4,0
+2568.mp4,0
+2569.mp4,0
+2570.mp4,0
+2571.mp4,0
+2573.mp4,0
+2574.mp4,0
+2575.mp4,0
+2576.mp4,0
+2577.mp4,0
+2578.mp4,0
+2579.mp4,0
+2580.mp4,0
+2582.mp4,0
+2584.mp4,0
+2585.mp4,0
+2586.mp4,0
+2587.mp4,0
+2588.mp4,0
+2589.mp4,0
+2590.mp4,0
+2592.mp4,0
+2593.mp4,0
+2594.mp4,0
+2595.mp4,0
+2596.mp4,0
+2597.mp4,0
+2598.mp4,0
+2601.mp4,0
+2602.mp4,0
+2603.mp4,0
+2604.mp4,0
+2605.mp4,0
+2606.mp4,0
+2607.mp4,0
+2610.mp4,0
+2612.mp4,0
+2613.mp4,0
+2614.mp4,0
+2615.mp4,0
+2616.mp4,0
+2617.mp4,0
+2618.mp4,0
+2619.mp4,0
+2620.mp4,0
+2621.mp4,0
+2624.mp4,0
+2625.mp4,0
+2626.mp4,0
+2627.mp4,0
+2628.mp4,0
+2629.mp4,0
+2630.mp4,0
+2631.mp4,0
+2632.mp4,0
+2633.mp4,0
+2634.mp4,0
+2636.mp4,0
+2637.mp4,0
+2638.mp4,0
+2639.mp4,0
+2640.mp4,0
+2641.mp4,0
+2643.mp4,0
+2644.mp4,0
+2645.mp4,0
+2646.mp4,0
+2647.mp4,0
+2648.mp4,0
+2649.mp4,0
+2650.mp4,0
+2651.mp4,0
+2652.mp4,0
+2653.mp4,0
+2654.mp4,0
+2655.mp4,0
+2657.mp4,0
+2659.mp4,0
+2660.mp4,0
+2661.mp4,0
+2662.mp4,0
+2663.mp4,0
+2664.mp4,0
+2665.mp4,0
+2666.mp4,0
+2667.mp4,0
+2669.mp4,0
+2670.mp4,0
+2671.mp4,0
+2672.mp4,0
+2676.mp4,0
+2677.mp4,0
+2679.mp4,0
+2681.mp4,0
+2683.mp4,0
+2684.mp4,0
+2685.mp4,0
+2686.mp4,0
+2687.mp4,0
+2688.mp4,0
+2689.mp4,0
+2690.mp4,0
+2691.mp4,0
+2692.mp4,0
+2693.mp4,0
+2694.mp4,0
+2696.mp4,0
+2697.mp4,0
+2698.mp4,0
+2699.mp4,0
+2700.mp4,0
+2701.mp4,0
+2702.mp4,0
+2703.mp4,0
+2705.mp4,0
+2706.mp4,0
+2707.mp4,0
+2708.mp4,0
+2709.mp4,0
+2710.mp4,0
+2711.mp4,0
+2714.mp4,0
+2715.mp4,0
+2716.mp4,0
+2719.mp4,0
+2722.mp4,0
+2723.mp4,0
+2724.mp4,0
+2726.mp4,0
+2727.mp4,0
+2729.mp4,0
+2730.mp4,0
+2731.mp4,0
+2732.mp4,0
+2733.mp4,0
+2734.mp4,0
+2735.mp4,0
+2736.mp4,0
+2737.mp4,0
+2738.mp4,0
+2740.mp4,0
+2743.mp4,0
+2746.mp4,0
+2747.mp4,0
+2748.mp4,0
+2749.mp4,0
+2750.mp4,0
+2751.mp4,0
+2752.mp4,0
+2754.mp4,0
+2758.mp4,0
+2759.mp4,0
+2760.mp4,0
+2762.mp4,0
+2763.mp4,0
+2764.mp4,0
+2765.mp4,0
+2766.mp4,0
+2767.mp4,0
+2768.mp4,0
+2771.mp4,0
+2772.mp4,0
+2773.mp4,0
+2776.mp4,0
+2777.mp4,0
+2778.mp4,0
+2779.mp4,0
+2781.mp4,0
+2782.mp4,0
+2783.mp4,0
+2784.mp4,0
+2785.mp4,0
+2786.mp4,0
+2787.mp4,0
+2788.mp4,0
+2790.mp4,0
+2791.mp4,0
+2792.mp4,0
+2793.mp4,0
+2794.mp4,0
+2795.mp4,0
+2796.mp4,0
+2799.mp4,0
+2800.mp4,0
+2801.mp4,0
+2802.mp4,0
+2803.mp4,0
+2805.mp4,0
+2806.mp4,0
+2807.mp4,0
+2809.mp4,0
+2810.mp4,0
+2811.mp4,0
+2813.mp4,0
+2814.mp4,0
+2816.mp4,0
+2818.mp4,0
+2819.mp4,0
+2820.mp4,0
+2821.mp4,0
+2823.mp4,0
+2830.mp4,0
+2831.mp4,0
+2832.mp4,0
+2833.mp4,0
+2835.mp4,0
+2836.mp4,0
+2837.mp4,0
+2838.mp4,0
+2839.mp4,0
+2841.mp4,0
+2842.mp4,0
+2843.mp4,0
+2844.mp4,0
+2846.mp4,0
+2847.mp4,0
+2849.mp4,0
+2851.mp4,0
+2852.mp4,0
+2854.mp4,0
+2855.mp4,0
+2857.mp4,0
+2859.mp4,0
+2860.mp4,0
+2861.mp4,0
+2863.mp4,0
+2864.mp4,0
+2867.mp4,0
+2868.mp4,0
+2869.mp4,0
+2870.mp4,0
+2871.mp4,0
+2873.mp4,0
+2874.mp4,0
+2875.mp4,0
+2876.mp4,0
+2877.mp4,0
+2878.mp4,0
+2880.mp4,0
+2881.mp4,0
+2882.mp4,0
+2883.mp4,0
+2884.mp4,0
+2885.mp4,0
+2886.mp4,0
+2887.mp4,0
+2888.mp4,0
+2889.mp4,0
+2890.mp4,0
+2893.mp4,0
+2894.mp4,0
+2895.mp4,0
+2896.mp4,0
+2897.mp4,0
+2898.mp4,0
+2899.mp4,0
+2901.mp4,0
+2903.mp4,0
+2904.mp4,0
+2906.mp4,0
+2907.mp4,0
+2908.mp4,0
+2910.mp4,0
+2912.mp4,0
+2914.mp4,0
+2915.mp4,0
+2916.mp4,0
+2917.mp4,0
+2918.mp4,0
+2919.mp4,0
+2921.mp4,0
+2922.mp4,0
+2923.mp4,0
+2924.mp4,0
+2925.mp4,0
+2927.mp4,0
+2928.mp4,0
+2929.mp4,0
+2930.mp4,0
+2932.mp4,0
+2933.mp4,0
+2934.mp4,0
+2935.mp4,0
+2936.mp4,0
+2937.mp4,0
+2939.mp4,0
+2941.mp4,0
+2943.mp4,0
+2944.mp4,0
+2945.mp4,0
+2946.mp4,0
+2948.mp4,0
+2949.mp4,0
+2950.mp4,0
+2951.mp4,0
+2952.mp4,0
+2953.mp4,0
+2954.mp4,0
+2957.mp4,0
+2958.mp4,0
+2960.mp4,0
+2962.mp4,0
+2963.mp4,0
+2964.mp4,0
+2965.mp4,0
+2966.mp4,0
+2967.mp4,0
+2968.mp4,0
+2970.mp4,0
+2973.mp4,0
+2974.mp4,0
+2976.mp4,0
+2977.mp4,0
+2979.mp4,0
+2980.mp4,0
+2981.mp4,0
+2982.mp4,0
+2983.mp4,0
+2985.mp4,0
+2986.mp4,0
+2987.mp4,0
+2989.mp4,0
+2990.mp4,0
+2991.mp4,0
+2992.mp4,0
+2993.mp4,0
+2994.mp4,0
+2996.mp4,0
+2997.mp4,0
+2998.mp4,0
+2999.mp4,0
+3000.mp4,0
+3001.mp4,0
+3005.mp4,0
+3007.mp4,0
+3008.mp4,0
+3009.mp4,0
+3010.mp4,0
+3011.mp4,0
+3012.mp4,0
+3013.mp4,0
+3014.mp4,0
+3015.mp4,0
+3016.mp4,0
+3017.mp4,0
+3018.mp4,0
+3019.mp4,0
+3020.mp4,0
+3021.mp4,0
+3022.mp4,0
+3023.mp4,0
+3024.mp4,0
+3025.mp4,0
+3026.mp4,0
+3027.mp4,0
+3028.mp4,0
+3029.mp4,0
+3030.mp4,0
+3031.mp4,0
+3032.mp4,0
+3033.mp4,0
+3034.mp4,0
+3035.mp4,0
+3036.mp4,0
+3037.mp4,0
+3038.mp4,0
+3039.mp4,0
+3040.mp4,0
+3041.mp4,0
+3042.mp4,0
+3043.mp4,0
+3044.mp4,0
+3045.mp4,0
+3046.mp4,0
+3047.mp4,0
+3048.mp4,0
+3049.mp4,0
+3050.mp4,0
+3051.mp4,0
+3052.mp4,0
+3053.mp4,0
+3054.mp4,0
+3055.mp4,0
+3056.mp4,0
+3057.mp4,0
+3058.mp4,0
+3059.mp4,0
+3060.mp4,0
+3061.mp4,0
+3062.mp4,0
+3063.mp4,0
+3064.mp4,0
+3065.mp4,0
+3066.mp4,0
+3067.mp4,0
+3068.mp4,0
+3069.mp4,0
+3070.mp4,0
+3071.mp4,0
+3072.mp4,0
+3073.mp4,0
+3074.mp4,0
+3075.mp4,0
+3076.mp4,0
+3077.mp4,0
+3078.mp4,0
+3079.mp4,0
+3080.mp4,0
+3081.mp4,0
+3082.mp4,0
+3083.mp4,0
+3084.mp4,0
+3085.mp4,0
+3086.mp4,0
+3087.mp4,0
+3088.mp4,0
+3089.mp4,0
+3090.mp4,0
+3091.mp4,0
+3092.mp4,0
+3093.mp4,0
+3094.mp4,0
+3095.mp4,0
+3096.mp4,0
+3097.mp4,0
+3098.mp4,0
+3099.mp4,0
+3100.mp4,0
+3101.mp4,0
+3102.mp4,0
+3103.mp4,0
+3104.mp4,0
+3105.mp4,0
+3106.mp4,0
+3107.mp4,0
+3108.mp4,0
+3109.mp4,0
+3110.mp4,0
+3111.mp4,0
+3112.mp4,0
+3113.mp4,0
+3114.mp4,0
+3115.mp4,0
+3116.mp4,0
+3117.mp4,0
+3118.mp4,0
+3119.mp4,0
+3120.mp4,0
+3121.mp4,0
+3122.mp4,0
+3123.mp4,0
+3124.mp4,0
+3125.mp4,0
+3126.mp4,0
+3127.mp4,0
+3128.mp4,0
+3129.mp4,0
+3130.mp4,0
+3131.mp4,0
+3132.mp4,0
+3133.mp4,0
+3134.mp4,0
+3135.mp4,0
+3136.mp4,0
+3137.mp4,0
+3138.mp4,0
+3139.mp4,0
+3140.mp4,0
+3141.mp4,0
+3142.mp4,0
+3143.mp4,0
+3144.mp4,0
+3145.mp4,0
+3146.mp4,0
+3147.mp4,0
+3148.mp4,0
+3149.mp4,0
+3150.mp4,0
+3151.mp4,0
+3152.mp4,0
+3153.mp4,0
+3154.mp4,0
+3155.mp4,0
+3156.mp4,0
+3157.mp4,0
+3158.mp4,0
+3159.mp4,0
+3160.mp4,0
+3161.mp4,0
+3162.mp4,0
+3163.mp4,0
+3164.mp4,0
+3165.mp4,0
+3166.mp4,0
+3167.mp4,0
+3168.mp4,0
+3169.mp4,0
+3170.mp4,0
+3171.mp4,0
+3172.mp4,0
+3173.mp4,0
+3174.mp4,0
+3175.mp4,0
+3176.mp4,0
+3177.mp4,0
+3178.mp4,0
+3179.mp4,0
+3180.mp4,0
+3181.mp4,0
+3182.mp4,0
+3183.mp4,0
+3184.mp4,0
+3185.mp4,0
+3186.mp4,0
+3187.mp4,0
+3188.mp4,0
+3189.mp4,0
+3190.mp4,0
+3191.mp4,0
+3192.mp4,0
+3193.mp4,0
+3194.mp4,0
+3195.mp4,0
+3196.mp4,0
+3197.mp4,0
+3198.mp4,0
+3199.mp4,0
+3200.mp4,0
+3201.mp4,0
+3202.mp4,0
+3203.mp4,0
+3204.mp4,0
+3205.mp4,0
+3206.mp4,0
+3207.mp4,0
+3208.mp4,0
+3209.mp4,0
+3210.mp4,0
+3211.mp4,0
+3212.mp4,0
+3213.mp4,0
+3214.mp4,0
+3215.mp4,0
+3216.mp4,0
+3217.mp4,0
+3218.mp4,0
+3219.mp4,0
+3220.mp4,0
+3221.mp4,0
+3222.mp4,0
+3223.mp4,0
+3224.mp4,0
+3225.mp4,0
+3226.mp4,0
+3227.mp4,0
+3228.mp4,0
+3229.mp4,0
+3230.mp4,0
+3231.mp4,0
+3232.mp4,0
+3233.mp4,0
+3234.mp4,0
+3235.mp4,0
+3236.mp4,0
+3237.mp4,0
+3238.mp4,0
+3239.mp4,0
+3240.mp4,0
+3241.mp4,0
+3242.mp4,0
+3243.mp4,0
+3244.mp4,0
+3245.mp4,0
+3246.mp4,0
+3247.mp4,0
+3248.mp4,0
+3249.mp4,0
+3250.mp4,0
+3251.mp4,0
+3252.mp4,0
+3253.mp4,0
+3254.mp4,0
+3255.mp4,0
+3256.mp4,0
+3257.mp4,0
+3258.mp4,0
+3259.mp4,0
+3260.mp4,0
+3261.mp4,0
+3262.mp4,0
+3263.mp4,0
+3264.mp4,0
+3265.mp4,0
+3266.mp4,0
+3267.mp4,0
+3268.mp4,0
+3269.mp4,0
+3270.mp4,0
+3271.mp4,0
+3272.mp4,0
+3273.mp4,0
+3274.mp4,0
+3275.mp4,0
+3276.mp4,0
+3277.mp4,0
+3278.mp4,0
+3279.mp4,0
+3280.mp4,0
+3281.mp4,0
+3282.mp4,0
+3283.mp4,0
+3284.mp4,0
+3285.mp4,0
+3286.mp4,0
+3287.mp4,0
+3288.mp4,0
+3289.mp4,0
+3290.mp4,0
+3291.mp4,0
+3292.mp4,0
+3293.mp4,0
+3294.mp4,0
+3295.mp4,0
+3296.mp4,0
+3297.mp4,0
+3298.mp4,0
+3299.mp4,0
+3300.mp4,0
+3301.mp4,0
+3302.mp4,0
+3303.mp4,0
+3304.mp4,0
+3305.mp4,0
+3306.mp4,0
+3307.mp4,0
+3308.mp4,0
+3309.mp4,0
+3310.mp4,0
+3311.mp4,0
+3312.mp4,0
+3313.mp4,0
+3314.mp4,0
+3315.mp4,0
+3316.mp4,0
+3317.mp4,0
+3318.mp4,0
+3319.mp4,0
+3320.mp4,0
+3321.mp4,0
+3322.mp4,0
+3323.mp4,0
+3324.mp4,0
+3325.mp4,0
+3326.mp4,0
+3327.mp4,0
+3328.mp4,0
+3329.mp4,0
+3330.mp4,0
+3331.mp4,0
+3332.mp4,0
+3333.mp4,0
+3334.mp4,0
+3335.mp4,0
+3336.mp4,0
+3337.mp4,0
+3338.mp4,0
+3339.mp4,0
+3340.mp4,0
+3341.mp4,0
+3342.mp4,0
+3343.mp4,0
+3344.mp4,0
+3345.mp4,0
+3346.mp4,0
+3347.mp4,0
+3348.mp4,0
+3349.mp4,0
+3350.mp4,0
+3351.mp4,0
+3352.mp4,0
+3353.mp4,0
+3354.mp4,0
+3355.mp4,0
+3356.mp4,0
+3357.mp4,0
+3358.mp4,0
+3359.mp4,0
+3360.mp4,0
+3361.mp4,0
+3362.mp4,0
+3363.mp4,0
+3364.mp4,0
+3365.mp4,0
+3366.mp4,0
+3367.mp4,0
+3368.mp4,0
+3369.mp4,0
+3370.mp4,0
+3371.mp4,0
+3372.mp4,0
+3373.mp4,0
+3374.mp4,0
+3375.mp4,0
+3376.mp4,0
+3377.mp4,0
+3378.mp4,0
+3379.mp4,0
+3380.mp4,0
+3381.mp4,0
+3382.mp4,0
+3383.mp4,0
+3384.mp4,0
+3385.mp4,0
+3386.mp4,0
+3387.mp4,0
+3388.mp4,0
+3389.mp4,0
+3390.mp4,0
+3391.mp4,0
+3392.mp4,0
+3393.mp4,0
+3394.mp4,0
+3395.mp4,0
+3396.mp4,0
+3397.mp4,0
+3398.mp4,0
+3399.mp4,0
+3400.mp4,0
+3401.mp4,0
+3402.mp4,0
+3403.mp4,0
+3404.mp4,0
+3405.mp4,0
+3406.mp4,0
+3407.mp4,0
+3408.mp4,0
+3409.mp4,0
+3410.mp4,0
+3411.mp4,0
+3412.mp4,0
+3413.mp4,0
+3414.mp4,0
+3415.mp4,0
+3416.mp4,0
+3417.mp4,0
+3418.mp4,0
+3419.mp4,0
+3420.mp4,0
+3421.mp4,0
+3422.mp4,0
+3423.mp4,0
+3424.mp4,0
+3425.mp4,0
+3426.mp4,0
+3427.mp4,0
+3428.mp4,0
+3429.mp4,0
+3430.mp4,0
+3431.mp4,0
+3432.mp4,0
+3433.mp4,0
+3434.mp4,0
+3435.mp4,0
+3436.mp4,0
+3437.mp4,0
+3438.mp4,0
+3439.mp4,0
+3440.mp4,0
+3441.mp4,0
+3442.mp4,0
+3443.mp4,0
+3444.mp4,0
+3445.mp4,0
+3446.mp4,0
+3447.mp4,0
+3448.mp4,0
+3449.mp4,0
+3450.mp4,0
+3451.mp4,0
+3452.mp4,0
+3453.mp4,0
+3454.mp4,0
+3455.mp4,0
+3456.mp4,0
+3457.mp4,0
+3458.mp4,0
+3459.mp4,0
+3460.mp4,0
+3461.mp4,0
+3462.mp4,0
+3463.mp4,0
+3464.mp4,0
+3465.mp4,0
+3466.mp4,0
+3467.mp4,0
+3468.mp4,0
+3469.mp4,0
+3470.mp4,0
+3471.mp4,0
+3472.mp4,0
+3473.mp4,0
+3474.mp4,0
+3475.mp4,0
+3476.mp4,0
+3477.mp4,0
+3478.mp4,0
+3479.mp4,0
+3480.mp4,0
+3481.mp4,0
+3482.mp4,0
+3483.mp4,0
+3484.mp4,0
+3485.mp4,0
+3486.mp4,0
+3487.mp4,0
+3488.mp4,0
+3489.mp4,0
+3490.mp4,0
+3491.mp4,0
+3492.mp4,0
+3493.mp4,0
+3494.mp4,0
+3495.mp4,0
+3496.mp4,0
+3497.mp4,0
+3498.mp4,0
+3499.mp4,0
+3500.mp4,0
+3501.mp4,0
+3502.mp4,0
+3503.mp4,0
+3504.mp4,0
+3505.mp4,0
+3506.mp4,0
+3507.mp4,0
+3508.mp4,0
+3509.mp4,0
+3510.mp4,0
+3511.mp4,0
+3512.mp4,0
+3513.mp4,0
+3514.mp4,0
+3515.mp4,0
+3516.mp4,0
+3517.mp4,0
+3518.mp4,0
+3519.mp4,0
+3520.mp4,0
+3521.mp4,0
+3522.mp4,0
+3523.mp4,0
+3524.mp4,0
+3525.mp4,0
+3526.mp4,0
+3527.mp4,0
+3528.mp4,0
+3529.mp4,0
+3530.mp4,0
+3531.mp4,0
+3532.mp4,0
+3533.mp4,0
+3534.mp4,0
+3535.mp4,0
+3536.mp4,0
+3537.mp4,0
+3538.mp4,0
+3539.mp4,0
+3540.mp4,0
+3541.mp4,0
+3542.mp4,0
+3543.mp4,0
+3544.mp4,0
+3545.mp4,0
+3546.mp4,0
+3547.mp4,0
+3548.mp4,0
+3549.mp4,0
+3550.mp4,0
+3551.mp4,0
+3552.mp4,0
+3553.mp4,0
+3554.mp4,0
+3555.mp4,0
+3556.mp4,0
+3557.mp4,0
+3558.mp4,0
+3559.mp4,0
+3560.mp4,0
+3561.mp4,0
+3562.mp4,0
+3563.mp4,0
+3564.mp4,0
+3565.mp4,0
+3566.mp4,0
+3567.mp4,0
+3568.mp4,0
+3569.mp4,0
+3570.mp4,0
+3571.mp4,0
+3572.mp4,0
+3573.mp4,0
+3574.mp4,0
+3575.mp4,0
+3576.mp4,0
+3577.mp4,0
+3578.mp4,0
+3579.mp4,0
+3580.mp4,0
+3581.mp4,0
+3582.mp4,0
+3583.mp4,0
+3584.mp4,0
+3585.mp4,0
+3586.mp4,0
+3587.mp4,0
+3588.mp4,0
+3589.mp4,0
+3590.mp4,0
+3591.mp4,0
+3592.mp4,0
+3593.mp4,0
+3594.mp4,0
+3595.mp4,0
+3596.mp4,0
+3597.mp4,0
+3598.mp4,0
+3599.mp4,0
+3600.mp4,0
+3601.mp4,0
+3602.mp4,0
+3603.mp4,0
+3604.mp4,0
+3605.mp4,0
+3606.mp4,0
+3607.mp4,0
+3608.mp4,0
+3609.mp4,0
+3610.mp4,0
+3611.mp4,0
+3612.mp4,0
+3613.mp4,0
+3614.mp4,0
+3615.mp4,0
+3616.mp4,0
+3617.mp4,0
+3618.mp4,0
+3619.mp4,0
+3620.mp4,0
+3621.mp4,0
+3622.mp4,0
+3623.mp4,0
+3624.mp4,0
+3625.mp4,0
+3626.mp4,0
+3627.mp4,0
+3628.mp4,0
+3629.mp4,0
+3630.mp4,0
+3631.mp4,0
+3632.mp4,0
+3633.mp4,0
+3634.mp4,0
+3635.mp4,0
+3636.mp4,0
+3637.mp4,0
+3638.mp4,0
+3639.mp4,0
+3640.mp4,0
+3641.mp4,0
+3642.mp4,0
+3643.mp4,0
+3644.mp4,0
+3645.mp4,0
+3646.mp4,0
+3647.mp4,0
+3648.mp4,0
+3649.mp4,0
+3650.mp4,0
+3651.mp4,0
+3652.mp4,0
+3653.mp4,0
+3654.mp4,0
+3655.mp4,0
+3656.mp4,0
+3657.mp4,0
+3658.mp4,0
+3659.mp4,0
+3660.mp4,0
+3661.mp4,0
+3662.mp4,0
+3663.mp4,0
+3664.mp4,0
+3665.mp4,0
+3666.mp4,0
+3667.mp4,0
+3668.mp4,0
+3669.mp4,0
+3670.mp4,0
+3671.mp4,0
+3672.mp4,0
+3673.mp4,0
+3674.mp4,0
+3675.mp4,0
+3676.mp4,0
+3677.mp4,0
+3678.mp4,0
+3679.mp4,0
+3680.mp4,0
+3681.mp4,0
+3682.mp4,0
+3683.mp4,0
+3684.mp4,0
+3685.mp4,0
+3686.mp4,0
+3687.mp4,0
+3688.mp4,0
+3689.mp4,0
+3690.mp4,0
+3691.mp4,0
+3692.mp4,0
+3693.mp4,0
+3694.mp4,0
+3695.mp4,0
+3696.mp4,0
+3697.mp4,0
+3698.mp4,0
+3699.mp4,0
+3700.mp4,0
+3701.mp4,0
+3702.mp4,0
+3703.mp4,0
+3704.mp4,0
+3705.mp4,0
+3706.mp4,0
+3707.mp4,0
+3708.mp4,0
+3709.mp4,0
+3710.mp4,0
+3711.mp4,0
+3712.mp4,0
+3713.mp4,0
+3714.mp4,0
+3715.mp4,0
+3716.mp4,0
+3717.mp4,0
+3718.mp4,0
+3719.mp4,0
+3720.mp4,0
+3721.mp4,0
+3722.mp4,0
+3723.mp4,0
+3724.mp4,0
+3725.mp4,0
+3726.mp4,0
+3727.mp4,0
+3728.mp4,0
+3729.mp4,0
+3730.mp4,0
+3731.mp4,0
+3732.mp4,0
+3733.mp4,0
+3734.mp4,0
+3735.mp4,0
+3736.mp4,0
+3737.mp4,0
+3738.mp4,0
+3739.mp4,0
+3740.mp4,0
+3741.mp4,0
+3742.mp4,0
+3743.mp4,0
+3744.mp4,0
+3745.mp4,0
+3746.mp4,0
+3747.mp4,0
+3748.mp4,0
+3749.mp4,0
+3750.mp4,0
+3751.mp4,0
+3752.mp4,0
+3753.mp4,0
+3754.mp4,0
+3755.mp4,0
+3756.mp4,0
+3757.mp4,0
+3758.mp4,0
+3759.mp4,0
+3760.mp4,0
+3761.mp4,0
+3762.mp4,0
+3763.mp4,0
+3764.mp4,0
+3765.mp4,0
+3766.mp4,0
+3767.mp4,0
+3768.mp4,0
+3769.mp4,0
+3770.mp4,0
+3771.mp4,0
+3772.mp4,0
+3773.mp4,0
+3774.mp4,0
+3775.mp4,0
+3776.mp4,0
+3777.mp4,0
+3778.mp4,0
+3779.mp4,0
+3780.mp4,0
+3781.mp4,0
+3782.mp4,0
+3783.mp4,0
+3784.mp4,0
+3785.mp4,0
+3786.mp4,0
+3787.mp4,0
+3788.mp4,0
+3789.mp4,0
+3790.mp4,0
+3791.mp4,0
+3792.mp4,0
+3793.mp4,0
+3794.mp4,0
+3795.mp4,0
+3796.mp4,0
+3797.mp4,0
+3798.mp4,0
+3799.mp4,0
+3800.mp4,0
+3801.mp4,0
+3802.mp4,0
+3803.mp4,0
+3804.mp4,0
+3805.mp4,0
+3806.mp4,0
+3807.mp4,0
+3808.mp4,0
+3809.mp4,0
+3810.mp4,0
+3811.mp4,0
+3812.mp4,0
+3813.mp4,0
+3814.mp4,0
+3815.mp4,0
+3816.mp4,0
+3817.mp4,0
+3818.mp4,0
+3819.mp4,0
+3820.mp4,0
+3821.mp4,0
+3822.mp4,0
+3823.mp4,0
+3824.mp4,0
+3825.mp4,0
+3826.mp4,0
+3827.mp4,0
+3828.mp4,0
+3829.mp4,0
+3830.mp4,0
+3831.mp4,0
+3832.mp4,0
+3833.mp4,0
+3834.mp4,0
+3835.mp4,0
+3836.mp4,0
+3837.mp4,0
+3838.mp4,0
+3839.mp4,0
+3840.mp4,0
+3841.mp4,0
+3842.mp4,0
+3843.mp4,0
+3844.mp4,0
+3845.mp4,0
+3846.mp4,0
+3847.mp4,0
+3848.mp4,0
+3849.mp4,0
+3850.mp4,0
+3851.mp4,0
+3852.mp4,0
+3853.mp4,0
+3854.mp4,0
+3855.mp4,0
+3856.mp4,0
+3857.mp4,0
+3858.mp4,0
+3859.mp4,0
+3860.mp4,0
+3861.mp4,0
+3862.mp4,0
+3863.mp4,0
+3864.mp4,0
+3865.mp4,0
+3866.mp4,0
+3867.mp4,0
+3868.mp4,0
+3869.mp4,0
+3870.mp4,0
+3871.mp4,0
+3872.mp4,0
+3873.mp4,0
+3874.mp4,0
+3875.mp4,0
+3876.mp4,0
+3877.mp4,0
+3878.mp4,0
+3879.mp4,0
+3880.mp4,0
+3881.mp4,0
+3882.mp4,0
+3883.mp4,0
+3884.mp4,0
+3885.mp4,0
+3886.mp4,0
+3887.mp4,0
+3888.mp4,0
+3889.mp4,0
+3890.mp4,0
+3891.mp4,0
+3892.mp4,0
+3893.mp4,0
+3894.mp4,0
+3895.mp4,0
+3896.mp4,0
+3897.mp4,0
+3898.mp4,0
+3899.mp4,0
+3900.mp4,0
+3901.mp4,0
+3902.mp4,0
+3903.mp4,0
+3904.mp4,0
+3905.mp4,0
+3906.mp4,0
+3907.mp4,0
+3908.mp4,0
+3909.mp4,0
+3910.mp4,0
+3911.mp4,0
+3912.mp4,0
+3913.mp4,0
+3914.mp4,0
+3915.mp4,0
+3916.mp4,0
+3917.mp4,0
+3918.mp4,0
+3919.mp4,0
+3920.mp4,0
+3921.mp4,0
+3922.mp4,0
+3923.mp4,0
+3924.mp4,0
+3925.mp4,0
+3926.mp4,0
+3927.mp4,0
+3928.mp4,0
+3929.mp4,0
+3930.mp4,0
+3931.mp4,0
+3932.mp4,0
+3933.mp4,0
+3934.mp4,0
+3935.mp4,0
+3936.mp4,0
+3937.mp4,0
+3938.mp4,0
+3939.mp4,0
+3940.mp4,0
+3941.mp4,0
+3942.mp4,0
+3943.mp4,0
+3944.mp4,0
+3945.mp4,0
+3946.mp4,0
+3947.mp4,0
+3948.mp4,0
+3949.mp4,0
+3950.mp4,0
+3951.mp4,0
+3952.mp4,0
+3953.mp4,0
+3954.mp4,0
+3955.mp4,0
+3956.mp4,0
+3957.mp4,0
+3958.mp4,0
+3959.mp4,0
+3960.mp4,0
+3961.mp4,0
+3962.mp4,0
+3963.mp4,0
+3964.mp4,0
+3965.mp4,0
+3966.mp4,0
+3967.mp4,0
+3968.mp4,0
+3969.mp4,0
+3970.mp4,0
+3971.mp4,0
+3972.mp4,0
+3973.mp4,0
+3974.mp4,0
+3975.mp4,0
+3976.mp4,0
+3977.mp4,0
+3978.mp4,0
+3979.mp4,0
+3980.mp4,0
+3981.mp4,0
+3982.mp4,0
+3983.mp4,0
+3984.mp4,0
+3985.mp4,0
+3986.mp4,0
+3987.mp4,0
+3988.mp4,0
+3989.mp4,0
+3990.mp4,0
+3991.mp4,0
+3992.mp4,0
+3993.mp4,0
+3994.mp4,0
+3995.mp4,0
+3996.mp4,0
+3997.mp4,0
+3998.mp4,0
+3999.mp4,0
+4000.mp4,0
+4001.mp4,0
+4002.mp4,0
+4003.mp4,0
+4004.mp4,0
+4005.mp4,0
+4006.mp4,0
+4007.mp4,0
+4008.mp4,0
+4009.mp4,0
+4010.mp4,0
+4011.mp4,0
+4012.mp4,0
+4013.mp4,0
+4014.mp4,0
+4015.mp4,0
+4016.mp4,0
+4017.mp4,0
+4018.mp4,0
+4019.mp4,0
+4020.mp4,0
+4021.mp4,0
+4022.mp4,0
+4023.mp4,0
+4024.mp4,0
+4025.mp4,0
+4026.mp4,0
+4027.mp4,0
+4028.mp4,0
+4029.mp4,0
+4030.mp4,0
+4031.mp4,0
+4032.mp4,0
+4033.mp4,0
+4034.mp4,0
+4035.mp4,0
+4036.mp4,0
+4037.mp4,0
+4038.mp4,0
+4039.mp4,0
+4040.mp4,0
+4041.mp4,0
+4042.mp4,0
+4043.mp4,0
+4044.mp4,0
+4045.mp4,0
+4046.mp4,0
+4047.mp4,0
+4048.mp4,0
+4049.mp4,0
+4050.mp4,0
+4051.mp4,0
+4052.mp4,0
+4053.mp4,0
+4054.mp4,0
+4055.mp4,0
+4056.mp4,0
+4057.mp4,0
+4058.mp4,0
+4059.mp4,0
+4060.mp4,0
+4061.mp4,0
+4062.mp4,0
+4063.mp4,0
+4064.mp4,0
+4065.mp4,0
+4066.mp4,0
+4067.mp4,0
+4068.mp4,0
+4069.mp4,0
+4070.mp4,0
+4071.mp4,0
+4072.mp4,0
+4073.mp4,0
+4074.mp4,0
+4075.mp4,0
+4076.mp4,0
+4077.mp4,0
+4078.mp4,0
+4079.mp4,0
+4080.mp4,0
+4081.mp4,0
+4082.mp4,0
+4083.mp4,0
+4084.mp4,0
+4085.mp4,0
+4086.mp4,0
+4087.mp4,0
+4088.mp4,0
+4089.mp4,0
+4090.mp4,0
+4091.mp4,0
+4092.mp4,0
+4093.mp4,0
+4094.mp4,0
+4095.mp4,0
+4096.mp4,0
+4097.mp4,0
+4098.mp4,0
+4099.mp4,0
+4100.mp4,0
+4101.mp4,0
+4102.mp4,0
+4103.mp4,0
+4104.mp4,0
+4105.mp4,0
+4106.mp4,0
+4107.mp4,0
+4108.mp4,0
+4109.mp4,0
+4110.mp4,0
+4111.mp4,0
+4112.mp4,0
+4113.mp4,0
+4114.mp4,0
+4115.mp4,0
+4116.mp4,0
+4117.mp4,0
+4118.mp4,0
+4119.mp4,0
+4120.mp4,0
+4121.mp4,0
+4122.mp4,0
+4123.mp4,0
+4124.mp4,0
+4125.mp4,0
+4126.mp4,0
+4127.mp4,0
+4128.mp4,0
+4129.mp4,0
+4130.mp4,0
+4131.mp4,0
+4132.mp4,0
+4133.mp4,0
+4134.mp4,0
+4135.mp4,0
+4136.mp4,0
+4137.mp4,0
+4138.mp4,0
+4139.mp4,0
+4140.mp4,0
+4141.mp4,0
+4142.mp4,0
+4143.mp4,0
+4144.mp4,0
+4145.mp4,0
+4146.mp4,0
+4147.mp4,0
+4148.mp4,0
+4149.mp4,0
+4150.mp4,0
+4151.mp4,0
+4152.mp4,0
+4153.mp4,0
+4154.mp4,0
+4155.mp4,0
+4156.mp4,0
+4157.mp4,0
+4158.mp4,0
+4159.mp4,0
+4160.mp4,0
+4161.mp4,0
+4162.mp4,0
+4163.mp4,0
+4164.mp4,0
+4165.mp4,0
+4166.mp4,0
+4167.mp4,0
+4168.mp4,0
+4169.mp4,0
+4170.mp4,0
+4171.mp4,0
+4172.mp4,0
+4173.mp4,0
+4174.mp4,0
+4175.mp4,0
+4176.mp4,0
+4177.mp4,0
+4178.mp4,0
+4179.mp4,0
+4180.mp4,0
+4181.mp4,0
+4182.mp4,0
+4183.mp4,0
+4184.mp4,0
+4185.mp4,0
+4186.mp4,0
+4187.mp4,0
+4188.mp4,0
+4189.mp4,0
+4190.mp4,0
+4191.mp4,0
+4192.mp4,0
+4193.mp4,0
+4194.mp4,0
+4195.mp4,0
+4196.mp4,0
+4197.mp4,0
+4198.mp4,0
+4199.mp4,0
+4200.mp4,0
+4201.mp4,0
+4202.mp4,0
+4203.mp4,0
+4204.mp4,0
+4205.mp4,0
+4206.mp4,0
+4207.mp4,0
+4208.mp4,0
+4209.mp4,0
+4210.mp4,0
+4211.mp4,0
+4212.mp4,0
+4213.mp4,0
+4214.mp4,0
+4215.mp4,0
+4216.mp4,0
+4217.mp4,0
+4218.mp4,0
+4219.mp4,0
+4220.mp4,0
+4221.mp4,0
+4222.mp4,0
+4223.mp4,0
+4224.mp4,0
+4225.mp4,0
+4226.mp4,0
+4227.mp4,0
+4228.mp4,0
+4229.mp4,0
+4230.mp4,0
+4231.mp4,0
+4232.mp4,0
+4233.mp4,0
+4234.mp4,0
+4235.mp4,0
+4236.mp4,0
+4237.mp4,0
+4238.mp4,0
+4239.mp4,0
+4240.mp4,0
+4241.mp4,0
+4242.mp4,0
+4243.mp4,0
+4244.mp4,0
+4245.mp4,0
+4246.mp4,0
+4247.mp4,0
+4248.mp4,0
+4249.mp4,0
+4250.mp4,0
+4251.mp4,0
+4252.mp4,0
+4253.mp4,0
+4254.mp4,0
+4255.mp4,0
+4256.mp4,0
+4257.mp4,0
+4258.mp4,0
+4259.mp4,0
+4260.mp4,0
+4261.mp4,0
+4262.mp4,0
+4263.mp4,0
+4264.mp4,0
+4265.mp4,0
+4266.mp4,0
+4267.mp4,0
+4268.mp4,0
+4269.mp4,0
+4270.mp4,0
+4271.mp4,0
+4272.mp4,0
+4273.mp4,0
+4274.mp4,0
+4275.mp4,0
+4276.mp4,0
+4277.mp4,0
+4278.mp4,0
+4279.mp4,0
+4280.mp4,0
+4281.mp4,0
+4282.mp4,0
+4283.mp4,0
+4284.mp4,0
+4285.mp4,0
+4286.mp4,0
+4287.mp4,0
+4288.mp4,0
+4289.mp4,0
+4290.mp4,0
+4291.mp4,0
+4292.mp4,0
+4293.mp4,0
+4294.mp4,0
+4295.mp4,0
+4296.mp4,0
+4297.mp4,0
+4298.mp4,0
+4299.mp4,0
+4300.mp4,0
+4301.mp4,0
+4302.mp4,0
+4303.mp4,0
+4304.mp4,0
+4305.mp4,0
+4306.mp4,0
+4307.mp4,0
+4308.mp4,0
+4309.mp4,0
+4310.mp4,0
+4311.mp4,0
+4312.mp4,0
+4313.mp4,0
+4314.mp4,0
+4315.mp4,0
+4316.mp4,0
+4317.mp4,0
+4318.mp4,0
+4319.mp4,0
+4320.mp4,0
+4321.mp4,0
+4322.mp4,0
+4323.mp4,0
+4324.mp4,0
+4325.mp4,0
+4326.mp4,0
+4327.mp4,0
+4328.mp4,0
+4329.mp4,0
+4330.mp4,0
+4331.mp4,0
+4332.mp4,0
+4333.mp4,0
+4334.mp4,0
+4335.mp4,0
+4336.mp4,0
+4337.mp4,0
+4338.mp4,0
+4339.mp4,0
+4340.mp4,0
+4341.mp4,0
+4342.mp4,0
+4343.mp4,0
+4344.mp4,0
+4345.mp4,0
+4346.mp4,0
+4347.mp4,0
+4348.mp4,0
+4349.mp4,0
+4350.mp4,0
+4351.mp4,0
+4352.mp4,0
+4353.mp4,0
+4354.mp4,0
+4355.mp4,0
+4356.mp4,0
+4357.mp4,0
+4358.mp4,0
+4359.mp4,0
+4360.mp4,0
+4361.mp4,0
+4362.mp4,0
+4363.mp4,0
+4364.mp4,0
+4365.mp4,0
+4366.mp4,0
+4367.mp4,0
+4368.mp4,0
+4369.mp4,0
+4370.mp4,0
+4371.mp4,0
+4372.mp4,0
+4373.mp4,0
+4374.mp4,0
+4375.mp4,0
+4376.mp4,0
+4377.mp4,0
+4378.mp4,0
+4379.mp4,0
+4380.mp4,0
+4381.mp4,0
+4382.mp4,0
+4383.mp4,0
+4384.mp4,0
+4385.mp4,0
+4386.mp4,0
+4387.mp4,0
+4388.mp4,0
+4389.mp4,0
+4390.mp4,0
+4391.mp4,0
+4392.mp4,0
+4393.mp4,0
+4394.mp4,0
+4395.mp4,0
+4396.mp4,0
+4397.mp4,0
+4398.mp4,0
+4399.mp4,0
+4400.mp4,0
+4401.mp4,0
+4402.mp4,0
+4403.mp4,0
+4404.mp4,0
+4405.mp4,0
+4406.mp4,0
+4407.mp4,0
+4408.mp4,0
+4409.mp4,0
+4410.mp4,0
+4411.mp4,0
+4412.mp4,0
+4413.mp4,0
+4414.mp4,0
+4415.mp4,0
+4416.mp4,0
+4417.mp4,0
+4418.mp4,0
+4419.mp4,0
+4420.mp4,0
+4421.mp4,0
+4422.mp4,0
+4423.mp4,0
+4424.mp4,0
+4425.mp4,0
+4426.mp4,0
+4427.mp4,0
+4428.mp4,0
+4429.mp4,0
+4430.mp4,0
+4431.mp4,0
+4432.mp4,0
+4433.mp4,0
+4434.mp4,0
+4435.mp4,0
+4436.mp4,0
+4437.mp4,0
+4438.mp4,0
+4439.mp4,0
+4440.mp4,0
+4441.mp4,0
+4442.mp4,0
+4443.mp4,0
+4444.mp4,0
+4445.mp4,0
+4446.mp4,0
+4447.mp4,0
+4448.mp4,0
+4449.mp4,0
+4450.mp4,0
+4451.mp4,0
+4452.mp4,0
+4453.mp4,0
+4454.mp4,0
+4455.mp4,0
+4456.mp4,0
+4457.mp4,0
+4458.mp4,0
+4459.mp4,0
+4460.mp4,0
+4461.mp4,0
+4462.mp4,0
+4463.mp4,0
+4464.mp4,0
+4465.mp4,0
+4466.mp4,0
+4467.mp4,0
+4468.mp4,0
+4469.mp4,0
+4470.mp4,0
+4471.mp4,0
+4472.mp4,0
+4473.mp4,0
+4474.mp4,0
+4475.mp4,0
+4476.mp4,0
+4477.mp4,0
+4478.mp4,0
+4479.mp4,0
+4480.mp4,0
+4481.mp4,0
+4482.mp4,0
+4483.mp4,0
+4484.mp4,0
+4485.mp4,0
+4486.mp4,0
+4487.mp4,0
+4488.mp4,0
+4489.mp4,0
+4490.mp4,0
+4491.mp4,0
+4492.mp4,0
+4493.mp4,0
+4494.mp4,0
+4495.mp4,0
+4496.mp4,0
+4497.mp4,0
+4498.mp4,0
+4499.mp4,0
+4500.mp4,0
+4501.mp4,0
+4502.mp4,0
+4503.mp4,0
+4504.mp4,0
+4505.mp4,0
+4506.mp4,0
+4507.mp4,0
+4508.mp4,0
+4509.mp4,0
+4510.mp4,0
+4511.mp4,0
+4512.mp4,0
+4513.mp4,0
+4514.mp4,0
+4515.mp4,0
+4516.mp4,0
+4517.mp4,0
+4518.mp4,0
+4519.mp4,0
+4520.mp4,0
+4521.mp4,0
+4522.mp4,0
+4523.mp4,0
+4524.mp4,0
+4525.mp4,0
+4526.mp4,0
+4527.mp4,0
+4528.mp4,0
+4529.mp4,0
+4530.mp4,0
+4531.mp4,0
+4532.mp4,0
+4533.mp4,0
+4534.mp4,0
+4535.mp4,0
+4536.mp4,0
+4537.mp4,0
+4538.mp4,0
+4539.mp4,0
+4540.mp4,0
+4541.mp4,0
+4542.mp4,0
+4543.mp4,0
+4544.mp4,0
+4545.mp4,0
+4546.mp4,0
+4547.mp4,0
+4548.mp4,0
+4549.mp4,0
+4550.mp4,0
+4551.mp4,0
+4552.mp4,0
+4553.mp4,0
+4554.mp4,0
+4555.mp4,0
+4556.mp4,0
+4557.mp4,0
+4558.mp4,0
+4559.mp4,0
+4560.mp4,0
+4561.mp4,0
+4562.mp4,0
+4563.mp4,0
+4564.mp4,0
+4565.mp4,0
+4566.mp4,0
+4567.mp4,0
+4568.mp4,0
+4569.mp4,0
+4570.mp4,0
+4571.mp4,0
+4572.mp4,0
+4573.mp4,0
+4574.mp4,0
+4575.mp4,0
+4576.mp4,0
+4577.mp4,0
+4578.mp4,0
+4579.mp4,0
+4580.mp4,0
+4581.mp4,0
+4582.mp4,0
+4583.mp4,0
+4584.mp4,0
+4585.mp4,0
+4586.mp4,0
+4587.mp4,0
+4588.mp4,0
+4589.mp4,0
+4590.mp4,0
+4591.mp4,0
+4592.mp4,0
+4593.mp4,0
+4594.mp4,0
+4595.mp4,0
+4596.mp4,0
+4597.mp4,0
+4598.mp4,0
+4599.mp4,0
+4600.mp4,0
+4601.mp4,0
+4602.mp4,0
+4603.mp4,0
+4604.mp4,0
+4605.mp4,0
+4606.mp4,0
+4607.mp4,0
+4608.mp4,0
+4609.mp4,0
+4610.mp4,0
+4611.mp4,0
+4612.mp4,0
+4613.mp4,0
+4614.mp4,0
+4615.mp4,0
+4616.mp4,0
+4617.mp4,0
+4618.mp4,0
+4619.mp4,0
+4620.mp4,0
+4621.mp4,0
+4622.mp4,0
+4623.mp4,0
+4624.mp4,0
+4625.mp4,0
+4626.mp4,0
+4627.mp4,0
+4628.mp4,0
+4629.mp4,0
+4630.mp4,0
+4631.mp4,0
+4632.mp4,0
+4633.mp4,0
+4634.mp4,0
+4635.mp4,0
+4636.mp4,0
+4637.mp4,0
+4638.mp4,0
+4639.mp4,0
+4640.mp4,0
+4641.mp4,0
+4642.mp4,0
+4643.mp4,0
+4644.mp4,0
+4645.mp4,0
+4646.mp4,0
+4647.mp4,0
+4648.mp4,0
+4649.mp4,0
+4650.mp4,0
+4651.mp4,0
+4652.mp4,0
+4653.mp4,0
+4654.mp4,0
+4655.mp4,0
+4656.mp4,0
+4657.mp4,0
+4658.mp4,0
+4659.mp4,0
+4660.mp4,0
+4661.mp4,0
+4662.mp4,0
+4663.mp4,0
+4664.mp4,0
+4665.mp4,0
+4666.mp4,0
+4667.mp4,0
+4668.mp4,0
+4669.mp4,0
+4670.mp4,0
+4671.mp4,0
+4672.mp4,0
+4673.mp4,0
+4674.mp4,0
+4675.mp4,0
+4676.mp4,0
+4677.mp4,0
+4678.mp4,0
+4679.mp4,0
+4680.mp4,0
+4681.mp4,0
+4682.mp4,0
+4683.mp4,0
+4684.mp4,0
+4685.mp4,0
+4686.mp4,0
+4687.mp4,0
+4688.mp4,0
+4689.mp4,0
+4690.mp4,0
+4691.mp4,0
+4692.mp4,0
+4693.mp4,0
+4694.mp4,0
+4695.mp4,0
+4696.mp4,0
+4697.mp4,0
+4698.mp4,0
+4699.mp4,0
+4700.mp4,0
+4701.mp4,0
+4702.mp4,0
+4703.mp4,0
+4704.mp4,0
+4705.mp4,0
+4706.mp4,0
+4707.mp4,0
+4708.mp4,0
+4709.mp4,0
+4710.mp4,0
+4711.mp4,0
+4712.mp4,0
+4713.mp4,0
+4714.mp4,0
+4715.mp4,0
+4716.mp4,0
+4717.mp4,0
+4718.mp4,0
+4719.mp4,0
+4720.mp4,0
+4721.mp4,0
+4722.mp4,0
+4723.mp4,0
+4724.mp4,0
+4725.mp4,0
+4726.mp4,0
+4727.mp4,0
+4728.mp4,0
+4729.mp4,0
+4730.mp4,0
+4731.mp4,0
+4732.mp4,0
+4733.mp4,0
+4734.mp4,0
+4735.mp4,0
+4736.mp4,0
+4737.mp4,0
+4738.mp4,0
+4739.mp4,0
+4740.mp4,0
+4741.mp4,0
+4742.mp4,0
+4743.mp4,0
+4744.mp4,0
+4745.mp4,0
+4746.mp4,0
+4747.mp4,0
+4748.mp4,0
+4749.mp4,0
+4750.mp4,0
+4751.mp4,0
+4752.mp4,0
+4753.mp4,0
+4754.mp4,0
+4755.mp4,0
+4756.mp4,0
+4757.mp4,0
+4758.mp4,0
+4759.mp4,0
+4760.mp4,0
+4761.mp4,0
+4762.mp4,0
+4763.mp4,0
+4764.mp4,0
+4765.mp4,0
+4766.mp4,0
+4767.mp4,0
+4768.mp4,0
+4769.mp4,0
+4770.mp4,0
+4771.mp4,0
+4772.mp4,0
+4773.mp4,0
+4774.mp4,0
+4775.mp4,0
+4776.mp4,0
+4777.mp4,0
+4778.mp4,0
+4779.mp4,0
+4780.mp4,0
+4781.mp4,0
+4782.mp4,0
+4783.mp4,0
+4784.mp4,0
+4785.mp4,0
+4786.mp4,0
+4787.mp4,0
+4788.mp4,0
+4789.mp4,0
+4790.mp4,0
+4791.mp4,0
+4792.mp4,0
+4793.mp4,0
+4794.mp4,0
+4795.mp4,0
+4796.mp4,0
+4797.mp4,0
+4798.mp4,0
+4799.mp4,0
+4800.mp4,0
+4801.mp4,0
+4802.mp4,0
+4803.mp4,0
+4804.mp4,0
+4805.mp4,0
+4806.mp4,0
+4807.mp4,0
+4808.mp4,0
+4809.mp4,0
+4810.mp4,0
+4811.mp4,0
+4812.mp4,0
+4813.mp4,0
+4814.mp4,0
+4815.mp4,0
+4816.mp4,0
+4817.mp4,0
+4818.mp4,0
+4819.mp4,0
+4820.mp4,0
+4821.mp4,0
+4822.mp4,0
+4823.mp4,0
+4824.mp4,0
+4825.mp4,0
+4826.mp4,0
+4827.mp4,0
+4828.mp4,0
+4829.mp4,0
+4830.mp4,0
+4831.mp4,0
+4832.mp4,0
+4833.mp4,0
+4834.mp4,0
+4835.mp4,0
+4836.mp4,0
+4837.mp4,0
+4838.mp4,0
+4839.mp4,0
+4840.mp4,0
+4841.mp4,0
+4842.mp4,0
+4843.mp4,0
+4844.mp4,0
+4845.mp4,0
+4846.mp4,0
+4847.mp4,0
+4848.mp4,0
+4849.mp4,0
+4850.mp4,0
+4851.mp4,0
+4852.mp4,0
+4853.mp4,0
+4854.mp4,0
+4855.mp4,0
+4856.mp4,0
+4857.mp4,0
+4858.mp4,0
+4859.mp4,0
+4860.mp4,0
+4861.mp4,0
+4862.mp4,0
+4863.mp4,0
+4864.mp4,0
+4865.mp4,0
+4866.mp4,0
+4867.mp4,0
+4868.mp4,0
+4869.mp4,0
+4870.mp4,0
+4871.mp4,0
+4872.mp4,0
+4873.mp4,0
+4874.mp4,0
+4875.mp4,0
+4876.mp4,0
+4877.mp4,0
+4878.mp4,0
+4879.mp4,0
+4880.mp4,0
+4881.mp4,0
+4882.mp4,0
+4883.mp4,0
+4884.mp4,0
+4885.mp4,0
+4886.mp4,0
+4887.mp4,0
+4888.mp4,0
+4889.mp4,0
+4890.mp4,0
+4891.mp4,0
+4892.mp4,0
+4893.mp4,0
+4894.mp4,0
+4895.mp4,0
+4896.mp4,0
+4897.mp4,0
+4898.mp4,0
+4899.mp4,0
+4900.mp4,0
+4901.mp4,0
+4902.mp4,0
+4903.mp4,0
+4904.mp4,0
+4905.mp4,0
+4906.mp4,0
+4907.mp4,0
+4908.mp4,0
+4909.mp4,0
+4910.mp4,0
+4911.mp4,0
+4912.mp4,0
+4913.mp4,0
+4914.mp4,0
+4915.mp4,0
+4916.mp4,0
+4917.mp4,0
+4918.mp4,0
+4919.mp4,0
+4920.mp4,0
+4921.mp4,0
+4922.mp4,0
+4923.mp4,0
+4924.mp4,0
+4925.mp4,0
+4926.mp4,0
+4927.mp4,0
+4928.mp4,0
+4929.mp4,0
+4930.mp4,0
+4931.mp4,0
+4932.mp4,0
+4933.mp4,0
+4934.mp4,0
+4935.mp4,0
+4936.mp4,0
+4937.mp4,0
+4938.mp4,0
+4939.mp4,0
+4940.mp4,0
+4941.mp4,0
+4942.mp4,0
+4943.mp4,0
+4944.mp4,0
+4945.mp4,0
+4946.mp4,0
+4947.mp4,0
+4948.mp4,0
+4949.mp4,0
+4950.mp4,0
+4951.mp4,0
+4952.mp4,0
+4953.mp4,0
+4954.mp4,0
+4955.mp4,0
+4956.mp4,0
+4957.mp4,0
+4958.mp4,0
+4959.mp4,0
+4960.mp4,0
+4961.mp4,0
+4962.mp4,0
+4963.mp4,0
+4964.mp4,0
+4965.mp4,0
+4966.mp4,0
+4967.mp4,0
+4968.mp4,0
+4969.mp4,0
+4970.mp4,0
+4971.mp4,0
+4972.mp4,0
+4973.mp4,0
+4974.mp4,0
+4975.mp4,0
+4976.mp4,0
+4977.mp4,0
+4978.mp4,0
+4979.mp4,0
+4980.mp4,0
+4981.mp4,0
+4982.mp4,0
+4983.mp4,0
+4984.mp4,0
+4985.mp4,0
+4986.mp4,0
+4987.mp4,0
+4988.mp4,0
+4989.mp4,0
+4990.mp4,0
+4991.mp4,0
+4992.mp4,0
+4993.mp4,0
+4994.mp4,0
+4995.mp4,0
+4996.mp4,0
+4997.mp4,0
+4998.mp4,0
+4999.mp4,0

work_dirs/challenge_baseline_new/best.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cd0caf0f24c2cc4efe7286242fecd4fc445301ff389cb07a938e4303e47dabd2
+size 1499298208