malper commited on
Commit
89083de
·
verified ·
1 Parent(s): 5f34985

Upload folder using huggingface_hub

Browse files
Files changed (2) hide show
  1. README.md +43 -0
  2. cc_ft.ckpt +3 -0
README.md ADDED
@@ -0,0 +1,43 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: mit
3
+ tags:
4
+ - pytorch-lightning
5
+ - image-captioning
6
+ - vision
7
+ ---
8
+
9
+ # CLIPCap fine-tuned for Human-Human Interaction captioning (cc_ft)
10
+
11
+ This is the best-performing checkpoint from **"Learning Human-Human Interactions in Images from Weak Textual Supervision" (ICCV 2023)**: a [CLIPCap](https://github.com/rmokady/CLIP_prefix_caption) model fine-tuned on pseudo-labels (pHHI) for describing human-human interactions (HHI) in images.
12
+
13
+ - **Paper:** [arXiv:2304.14104](https://arxiv.org/abs/2304.14104)
14
+ - **Code:** [github.com/tau-vailab/learning-interactions](https://github.com/tau-vailab/learning-interactions)
15
+ - **Project page:** https://learning-interactions.github.io/
16
+
17
+ Note: this checkpoint has been computed from scratch and may differ slightly from the results reported in the paper due to randomness during training.
18
+
19
+ ## Usage
20
+
21
+ This checkpoint is a PyTorch Lightning checkpoint (`cc_ft.ckpt`), intended for use with the inference code in the repo above (`modeling/src/test.py`):
22
+
23
+ ```bash
24
+ hf download malper/learning-interactions-cc-ft cc_ft.ckpt --local-dir .
25
+ python src/test.py -d <image_dir> -o <output.json> -c cc_ft.ckpt -cc <clipcap_pretrained_weights>
26
+ ```
27
+
28
+ See the [modeling documentation](https://github.com/tau-vailab/learning-interactions/tree/main/modeling) for full details, including the required CLIPCap pretrained MLP mapping network weights.
29
+
30
+ ## Context
31
+
32
+ This is research code from 2023, prior to the widespread availability of general-purpose vision-language models (VLMs). It is provided as-is for reproducibility of the paper's results.
33
+
34
+ ## Citation
35
+
36
+ ```bibtex
37
+ @InProceedings{alper2023learning,
38
+ author = {Morris Alper and Hadar Averbuch-Elor},
39
+ title = {Learning Human-Human Interactions in Images from Weak Textual Supervision},
40
+ booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
41
+ year = {2023}
42
+ }
43
+ ```
cc_ft.ckpt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:97c5d02189cd8a39d9da027f7b4c85474e2c4bae5c409107b5b944773e8ca8c6
3
+ size 2488951568