| --- |
| license: apache-2.0 |
| --- |
| # ZH-CLIP: A Chinese CLIP Model |
| [](https://huggingface.co/spaces/thu-ml/ZH-CLIPZH-CLIP) |
|
|
| ## Models |
| You can download **ZH-CLIP** model from [🤗 thu-ml/zh-clip-vit-roberta-large-patch14](https://huggingface.co/thu-ml/zh-clip-vit-roberta-large-patch14). The model structure is shown below: |
| * Vision encoder network structure is the same as [openai/clip-vit-large-patch14](https://huggingface.co/openai/clip-vit-large-patch14), and initialize with [laion/CLIP-ViT-L-14-laion2B-s32B-b82K](https://huggingface.co/laion/CLIP-ViT-L-14-laion2B-s32B-b82K). |
| * Text encoder network struceure is the same as [hfl/chinese-roberta-wwm-ext-large](https://huggingface.co/hfl/chinese-roberta-wwm-ext-large) and initialized. |
| ## Results |
|
|
| #### COCO-CN Retrieval (Official Test Set): |
| <table> |
| <thead> |
| <tr> |
| <th rowspan="2">Model</th> |
| <th colspan="4">Text-to-Image</th> |
| <th colspan="4">Image-to-Text</th> |
| </tr> |
| <tr> |
| <th>R@1</th> |
| <th>R@5</th> |
| <th>R@10</th> |
| <th>Mean</th> |
| <th>R@1</th> |
| <th>R@5</th> |
| <th>R@10</th> |
| <th>Mean</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td>Clip-Chinese</td> |
| <td>22.60</td> |
| <td>50.04</td> |
| <td>65.24</td> |
| <td>45.96</td> |
| <td>22.8</td> |
| <td>49.8</td> |
| <td>64.1</td> |
| <td>45.57</td> |
| </tr> |
| <tr> |
| <td>mclip</td> |
| <td>56.51</td> |
| <td>83.57</td> |
| <td>90.79</td> |
| <td>76.95</td> |
| <td>59.9</td> |
| <td>87.3</td> |
| <td>94.1</td> |
| <td>80.43</td> |
| </tr> |
| <tr> |
| <td>Taiyi-CLIP</td> |
| <td>52.52</td> |
| <td>81.10</td> |
| <td>89.93</td> |
| <td>74.52</td> |
| <td>45.80</td> |
| <td>75.80</td> |
| <td>88.10</td> |
| <td>69.90</td> |
| </tr> |
| <tr> |
| <td>CN-CLIP</td> |
| <td>64.10</td> |
| <td>88.79</td> |
| <td>94.40</td> |
| <td>82.43</td> |
| <td>61.00</td> |
| <td>84.40</td> |
| <td>93.10</td> |
| <td>79.5</td> |
| </tr> |
| <tr> |
| <td>altclip-xlmr-l</td> |
| <td>62.87</td> |
| <td>87.18</td> |
| <td>94.01</td> |
| <td>81.35</td> |
| <td>63.3</td> |
| <td>88.3</td> |
| <td>95.3</td> |
| <td>82.3</td> |
| </tr> |
| <tr> |
| <td>ZH-CLIP</td> |
| <td><strong>68.00</strong></td> |
| <td><strong>89.46</strong></td> |
| <td><strong>95.44</strong></td> |
| <td><strong>84.30</strong></td> |
| <td><strong>68.50</strong></td> |
| <td><strong>90.10</strong></td> |
| <td><strong>96.50</strong></td> |
| <td><strong>85.03</strong></td> |
| </tr> |
| </tbody> |
| </table> |
| |
| #### Flickr30K-CN Retrieval (Official Test Set): |
| <table> |
| <thead> |
| <tr> |
| <th rowspan="2">Model</th> |
| <th colspan="4">Text-to-Image</th> |
| <th colspan="4">Image-to-Text</th> |
| </tr> |
| <tr> |
| <th>R@1</th> |
| <th>R@5</th> |
| <th>R@10</th> |
| <th>Mean</th> |
| <th>R@1</th> |
| <th>R@5</th> |
| <th>R@10</th> |
| <th>Mean</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td>Clip-Chinese</td> |
| <td>17.76</td> |
| <td>40.34</td> |
| <td>51.88</td> |
| <td>36.66</td> |
| <td>30.4</td> |
| <td>55.30</td> |
| <td>67.10</td> |
| <td>50.93</td> |
| </tr> |
| <tr> |
| <td>mclip</td> |
| <td>62.3</td> |
| <td>86.42</td> |
| <td>92.58</td> |
| <td>80.43</td> |
| <td>84.4</td> |
| <td>97.3</td> |
| <td>98.9</td> |
| <td>93.53</td> |
| </tr> |
| <tr> |
| <td>Taiyi-CLIP</td> |
| <td>53.5</td> |
| <td>80.5</td> |
| <td>87.24</td> |
| <td>73.75</td> |
| <td>65.4</td> |
| <td>90.6</td> |
| <td>95.7</td> |
| <td>83.9</td> |
| </tr> |
| <tr> |
| <td>CN-CLIP</td> |
| <td>67.98</td> |
| <td>89.54</td> |
| <td>94.46</td> |
| <td>83.99</td> |
| <td>81.2</td> |
| <td>96.6</td> |
| <td>98.2</td> |
| <td>92.0</td> |
| </tr> |
| <tr> |
| <td>altclip-xlmr-l</td> |
| <td>69.16</td> |
| <td>89.94</td> |
| <td><strong>94.5</strong></td> |
| <td>84.53</td> |
| <td>85.1</td> |
| <td><strong>97.7</strong></td> |
| <td><strong>99.2</strong></td> |
| <td>94.0</td> |
| </tr> |
| <tr> |
| <td>ZH-CLIP</td> |
| <td><strong>69.64</strong></td> |
| <td><strong>90.14</strong></td> |
| <td>94.3</td> |
| <td><strong>84.69</strong></td> |
| <td><strong>86.6</strong></td> |
| <td>97.6</td> |
| <td>98.8</td> |
| <td><strong>94.33</strong></td> |
| </tr> |
| </tbody> |
| </table> |
| |
|
|
| #### Muge Text-to-Image Retrieval (Official Validation Set): |
| <table> |
| <thead> |
| <tr> |
| <th rowspan="2">Model</th> |
| <th colspan="4">Text-to-Image</th> |
| </tr> |
| <tr> |
| <th>R@1</th> |
| <th>R@5</th> |
| <th>R@10</th> |
| <th>Mean</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td>Clip-Chinese</td> |
| <td>15.06</td> |
| <td>34.96</td> |
| <td>46.21</td> |
| <td>32.08</td> |
| </tr> |
| <tr> |
| <td>mclip</td> |
| <td>22.34</td> |
| <td>41.15</td> |
| <td>50.26</td> |
| <td>37.92</td> |
| </tr> |
| <tr> |
| <td>Taiyi-CLIP</td> |
| <td>42.09</td> |
| <td>67.75</td> |
| <td>77.21</td> |
| <td>62.35</td> |
| </tr> |
| <tr> |
| <td>cn-clip</td> |
| <td>56.25</td> |
| <td><strong>79.87</strong></td> |
| <td>86.50</td> |
| <td>74.21</td> |
| </tr> |
| <tr> |
| <td>altclip-xlmr-l</td> |
| <td>29.69</td> |
| <td>49.92</td> |
| <td>58.87</td> |
| <td>46.16</td> |
| </tr> |
| <tr> |
| <td>ZH-CLIP</td> |
| <td><strong>56.75</strong></td> |
| <td>79.75</td> |
| <td><strong>86.66</strong></td> |
| <td><strong>74.38</strong></td> |
| </tr> |
| </tbody> |
| </table> |
| |
| #### Zero-shot Image Classification: |
| <table> |
| <thead> |
| <tr> |
| <th rowspan="2">Model</th> |
| <th colspan="11">Zero-shot Classification (ACC1)</th> |
| </tr> |
| <tr> |
| <th>CIFAR10</th> |
| <th>CIFAR100</th> |
| <th>DTD</th> |
| <th>EuroSAT</th> |
| <th>FER</th> |
| <th>FGVC</th> |
| <th>KITTI</th> |
| <th>MNIST</th> |
| <th>PC</th> |
| <th>VOC</th> |
| <th>ImageNet</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td>Clip-Chinese</td> |
| <td>86.85</td> |
| <td>44.21</td> |
| <td>18.40</td> |
| <td>34.86</td> |
| <td>14.21</td> |
| <td>3.87</td> |
| <td>32.63</td> |
| <td>14.37</td> |
| <td>52.49</td> |
| <td>67.73</td> |
| <td>22.22</td> |
| </tr> |
| <tr> |
| <td>mclip</td> |
| <td>92.88</td> |
| <td>65.54</td> |
| <td>29.57</td> |
| <td>46.76</td> |
| <td>41.18</td> |
| <td>7.20</td> |
| <td>23.21</td> |
| <td>52.80</td> |
| <td>51.64</td> |
| <td>77.56</td> |
| <td>42.99</td> |
| </tr> |
| <tr> |
| <td>Taiyi-CLIP</td> |
| <td>95.62</td> |
| <td>73.30</td> |
| <td>40.69</td> |
| <td><strong>61.62</strong></td> |
| <td>36.22</td> |
| <td>13.98</td> |
| <td><strong>41.21</strong></td> |
| <td><strong>73.91</strong></td> |
| <td>50.02</td> |
| <td>75.28</td> |
| <td>49.82</td> |
| </tr> |
| <tr> |
| <td>CN-CLIP</td> |
| <td>94.75</td> |
| <td>75.04</td> |
| <td>44.73</td> |
| <td>52.34</td> |
| <td>48.57</td> |
| <td>20.55</td> |
| <td>20.11</td> |
| <td>61.99</td> |
| <td><strong>62.59</strong></td> |
| <td><strong>79.12</strong></td> |
| <td>53.40</td> |
| </tr> |
| <tr> |
| <td>Altclip-xlmr-l</td> |
| <td>95.49</td> |
| <td>77.29</td> |
| <td>42.07</td> |
| <td>56.96</td> |
| <td><strong>51.52</strong></td> |
| <td><strong>26.85</strong></td> |
| <td>24.89</td> |
| <td>65.68</td> |
| <td>50.02</td> |
| <td>77.99</td> |
| <td><strong>59.21</strong></td> |
| </tr> |
| <tr> |
| <td>ZH-CLIP</td> |
| <td><strong>97.08</strong></td> |
| <td><strong>80.73</strong></td> |
| <td><strong>47.66</strong></td> |
| <td>51.58</td> |
| <td>48.48</td> |
| <td>20.73</td> |
| <td>20.11</td> |
| <td>61.94</td> |
| <td>62.31</td> |
| <td>78.07</td> |
| <td>56.87</td> |
| </tr> |
| </tbody> |
| </table> |
| |
| ## Getting Started |
| ### Dependency |
| * python >= 3.9 |
| * pip install -r requirements.txt |
| ### Inference |
| You can clone code from https://github.com/thu-ml/zh-clip |
| ```python |
| from PIL import Image |
| import requests |
| from models.zhclip import ZhCLIPProcessor, ZhCLIPModel # Code in https://github.com/thu-ml/zh-clip |
| |
| version = 'thu-ml/zh-clip-vit-roberta-large-patch14' |
| model = ZhCLIPModel.from_pretrained(version) |
| processor = ZhCLIPProcessor.from_pretrained(version) |
| |
| url = "http://images.cocodataset.org/val2017/000000039769.jpg" |
| image = Image.open(requests.get(url, stream=True).raw) |
| inputs = processor(text=["一只猫", "一只狗"], images=image, return_tensors="pt", padding=True) |
| |
| outputs = model(**inputs) |
| image_features = outputs.image_features |
| text_features = outputs.text_features |
| text_probs = (image_features @ text_features.T).softmax(dim=-1) |
| ``` |
| ### Other Chinese CLIP Models |
| In addition, to compare the effectiveness of different methods, the inference methods of other Chinese CLIP models have been integrated. For the convenience of use, the inference code has also been made public, and please contact us if there is any infringement. The code only implements models at the same level as clip-vit-large-patch14, but it may be adapted for the use of more different versions of models in the future. |
| | # | model | alias | |
| | :----: | :---------- | :---------- | |
| | 0 | [ZH-CLIP](https://github.com/thu-ml/zh-clip) | zhclip | |
| | 1 | [AltCLIP](https://github.com/FlagAI-Open/FlagAI/tree/master/examples/AltCLIP) | altclip | |
| | 2 | [Chinese-CLIP](https://github.com/OFA-Sys/Chinese-CLIP) | cnclip | |
| | 3 | [TaiyiCLIP](https://github.com/IDEA-CCNL/Fengshenbang-LM) | taiyiclip | |
| | 4 | [Multilingual-CLIP](https://github.com/FreddeFrallan/Multilingual-CLIP) | mclip | |
| | 5 | [CLIP-Chinese](https://github.com/yangjianxin1/CLIP-Chinese) | clip-chinese | |
|
|
| Usage in [inference.py](https://github.com/thu-ml/zh-clip/blob/main/inference.py) |
|
|
