Upload README.md

README.md

@@ -1,3 +1,251 @@
----
-license: apache-2.0
----
+# VeCLIP: Improving CLIP Training via Visual-enriched Captions
+
+* A novel CLIP training scheme that achieves the SoTA performance on zero-shot ImageNet classification and COCO image text retreival using limited visual-enriched captions.* [[Paper](https://arxiv.org/abs/2310.07699)]
+
+[Zhengfeng Lai*](https://zjujefflai.github.io/), [Haotian Zhang*](https://haotian-zhang.github.io/) , [Bowen Zhang](https://zbwglory.github.io/), Wentao Wu, Haoping Bai, Aleksei Timofeev, Xianzhi Du, [Zhe Gan](https://zhegan27.github.io/), Jiulong Shan, [Chen-Nee Chuah](https://www.ece.ucdavis.edu/~chuah/rubinet/people/chuah/bio.html), Yinfei Yang, Meng Cao [*: equal contribution]
+
+
+<p align="center">
+    <img src="figs/veclip_diagram.jpg" width="100%"></a> <br>
+    Diagram of VeCap.
+</p>
+
+## Release
+- [03/06/2024] 🔥 We released the VeCLIP & VeCap-DFN [checkpoints](#checkpoints).
+
+## Contents
+- [Install](#install)
+- [Getting Started](#getting-started)
+- [Checkpoints](#checkpoints)
+
+## Install
+
+1. Clone this repository
+```Shell
+git clone https://github.com/apple/ml-veclip
+cd ml-veclip
+```
+
+2. Create an environment and install related packages
+```Shell
+conda create -n veclip python=3.9 -y
+conda activate veclip
+pip install -r requirements.txt
+```
+
+## Getting Started
+
+See the [example notebook](https://github.com/apple/ml-veclip/blob/main/load_veclip.ipynb) for details on how to simply load the different checkpoints using HuggingFace transformers.
+
+
+## Checkpoints
+
+We release the checkpoints for **VeCLIP**, which are trained from scratch on visual-enriched captions VeCap 3M/12M/100M/200M, as reported in the paper. The models are evaluated on COCO/Flickr30k image-text retrieval and ImageNet/ImageNetv2 classification in a zero-shot fashion. Use `wget` or `curl` to download the below checkpoints. 
+
+<table>
+<thead>
+  <tr>
+    <th rowspan="2">Data</th>
+    <th rowspan="2">Model</th>
+    <th rowspan="2">Resolution</th>
+    <th colspan="2">COCO (R@1)</th>
+    <th colspan="2">Flickr30k (R@1)</th>
+    <th rowspan="2">ImageNet</th>
+    <th rowspan="2">ImageNetv2</th>
+  </tr>
+  <tr>
+    <th>I2T</th>
+    <th>T2I</th>
+    <th>I2T</th>
+    <th>T2I</th>
+  </tr>
+</thead>
+<tbody>
+  <tr>
+    <td rowspan="2">VeCap 3M</td>
+    <td>CLIP-B/16</td>
+    <td>224x224</td>
+    <td>5.46</td>
+    <td>3.28</td>
+    <td>12.20</td>
+    <td>6.36</td>
+    <td>5.46</td>
+    <td>7.09</td>
+  </tr>
+  <tr>
+    <td><a href="https://docs-assets.developer.apple.com/ml-research/models/veclip/veclip_b16_3m.zip">VeCLIP-B/16</a></td>
+    <td>224x224</td>
+    <td>22.30</td>
+    <td>13.01</td>
+    <td>40.60</td>
+    <td>27.58</td>
+    <td>15.98</td>
+    <td>13.51</td>
+  </tr>
+  <tr>
+    <td rowspan="2">VeCap 12M</td>
+    <td>CLIP-B/16</td>
+    <td>224x224</td>
+    <td>24.52</td>
+    <td>14.28</td>
+    <td>44.70</td>
+    <td>290.6</td>
+    <td>31.60</td>
+    <td>27.03</td>
+  </tr>
+  <tr>
+    <td><a href="https://docs-assets.developer.apple.com/ml-research/models/veclip/veclip_b16_12m.zip">VeCLIP-B/16</a></td>
+    <td>224x224</td>
+    <td>47.78</td>
+    <td>31.62</td>
+    <td>73.90</td>
+    <td>55.68</td>
+    <td>38.11</td>
+    <td>32.53</td>
+  </tr>
+  <tr>
+    <td rowspan="2">VeCap 100M</td>
+    <td>CLIP-B/16</td>
+    <td>224x224</td>
+    <td>47.24</td>
+    <td>30.61</td>
+    <td>74.40</td>
+    <td>57.16</td>
+    <td>58.64</td>
+    <td>50.96</td>
+  </tr>
+  <tr>
+    <td><a href="https://docs-assets.developer.apple.com/ml-research/models/veclip/veclip_b16_100m.zip">VeCLIP-B/16</a></td>
+    <td>224x224</td>
+    <td>64.82</td>
+    <td>46.12</td>
+    <td>89.30</td>
+    <td>73.10</td>
+    <td>60.77</td>
+    <td>54.17</td>
+  </tr>
+  <tr>
+    <td rowspan="2">VeCap 200M</td>
+    <td>CLIP-B/16</td>
+    <td>224x224</td>
+    <td>52.20</td>
+    <td>34.97</td>
+    <td>80.90</td>
+    <td>63.26</td>
+    <td>63.72</td>
+    <td>56.84</td>
+  </tr>
+  <tr>
+    <td><a href="https://docs-assets.developer.apple.com/ml-research/models/veclip/veclip_b16_200m.zip">VeCLIP-B/16</a></td>
+    <td>224x224</td>
+    <td>67.20</td>
+    <td>48.40</td>
+    <td>91.10</td>
+    <td>76.32</td>
+    <td>64.64</td>
+    <td>57.67</td>
+  </tr>
+</tbody>
+</table>
+
+
+We further found our VeCap can also be complementary to other well-established filtering methods, e.g., [Data Filtering Network (DFN)](ttps://arxiv.org/abs/2309.17425). We also provide thosse checkpoints (referred to as **VeCap-DFN**) and report their performance below. 
+
+<table>
+<thead>
+<tr>
+<th rowspan="2">Backbone</th>
+<th rowspan="2">Resolution</th>
+<th rowspan="2">Data</th>
+<th colspan="2">COCO (R@1)</th>
+<th colspan="2">Flickr30k (R@1)</th>
+<th rowspan="2">ImageNet</th>
+<th rowspan="2">ImageNetV2</th>
+</tr>
+<tr>
+<th>I2T</th>
+<th>T2I</th>
+<th>I2T</th>
+<th>T2I</th>
+
+</tr>
+</thead>
+<tbody>
+<tr>
+<td rowspan="3"><a href="https://docs-assets.developer.apple.com/ml-research/models/veclip/vecapdfn_clip_b16.zip">VeCap-DFN-B/16</a></td>
+<td rowspan="3">224x224</td>
+<td>DFN </td>
+<td>62.96</td>
+<td>43.20</td>
+<td>87.10</td>
+<td>70.44</td>
+<td>76.15</td>
+<td>68.19</td>
+</tr>
+<tr>
+<td>VeCap 300M</td>
+<td>64.74</td>
+<td>44.58</td>
+<td>90.10</td>
+<td>73.14</td>
+<td>46.43</td>
+<td>41.15</td>
+</tr>
+<tr>
+<td>DFN + VeCap 300M</td>
+<td>66.28</td>
+<td>45.12</td>
+<td>88.80</td>
+<td>73.56</td>
+<td>76.19</td>
+<td>69.58</td>
+</tr>
+<tr>
+<td rowspan="1"><a href="https://docs-assets.developer.apple.com/ml-research/models/veclip/vecapdfn_clip_l14.zip">VeCap-DFN-L/14</a></td>
+<td rowspan="1">224x224</td>
+<td>DFN + VeCap 300M</td>
+<td>71.06</td>
+<td>51.13</td>
+<td>93.10</td>
+<td>80.96</td>
+<td>81.95</td>
+<td>75.48</td>
+</tr>
+<tr>
+<td rowspan="2"><a href="https://docs-assets.developer.apple.com/ml-research/models/veclip/vecapdfn_clip_h14_336.zip">VeCap-DFN-H/14</a></td>
+<td rowspan="1">336x336</td>
+<td>DFN + VeCap 300M</td>
+<td>72.78</td>
+<td>52.33</td>
+<td>93.60</td>
+<td>82.64</td>
+<td>83.07</td>
+<td>76.37</td>
+</tr>
+</tbody>
+</table>
+
+
+## Citation
+
+If you find VeCLIP useful, please cite using this BibTeX:
+
+```bibtex
+@article{lai2023scarcity,
+  title={From scarcity to efficiency: Improving clip training via visual-enriched captions},
+  author={Lai, Zhengfeng and Zhang, Haotian and Zhang, Bowen and Wu, Wentao and Bai, Haoping and Timofeev, Aleksei and Du, Xianzhi and Gan, Zhe and Shan, Jiulong and Chuah, Chen-Nee and Yang, Yinfei and others},
+  journal={arXiv preprint arXiv:2310.07699},
+  year={2023}
+}
+@article{fang2023data,
+  title={Data filtering networks},
+  author={Fang, Alex and Jose, Albin Madappally and Jain, Amit and Schmidt, Ludwig and Toshev, Alexander and Shankar, Vaishaal},
+  journal={arXiv preprint arXiv:2309.17425},
+  year={2023}
+}
+```
+
+## Acknowledgement
+
+- [axlearn](https://github.com/apple/axlearn): the codebase we use to train the models. 
+- [huggingface transformers](https://huggingface.co/docs/transformers/en/index): Transformers provides APIs to load our trained models.