Zero-Shot Image Classification
OpenCLIP
biomedical
contrastive-learning
image-text-retrieval
zero-shot-classification
Instructions to use ChartFoundation/BioLoom-models with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- OpenCLIP
How to use ChartFoundation/BioLoom-models with OpenCLIP:
import open_clip model, preprocess_train, preprocess_val = open_clip.create_model_and_transforms('hf-hub:ChartFoundation/BioLoom-models') tokenizer = open_clip.get_tokenizer('hf-hub:ChartFoundation/BioLoom-models') - Notebooks
- Google Colab
- Kaggle
|
Download README.md from ChartFoundation/BioLoom-models: direct link, hf CLI and curl.
- Browser
- Download file 12.7 kB
-
https://huggingface.co/ChartFoundation/BioLoom-models/resolve/main/README.md
- Command line
-
hf download hf://ChartFoundation/BioLoom-models/README.md
-
curl -L -H "Authorization: Bearer $HF_TOKEN" -o README.md https://huggingface.co/ChartFoundation/BioLoom-models/resolve/main/README.md
12.7 kB
| library_name: open_clip | |
| pipeline_tag: zero-shot-image-classification | |
| tags: | |
| - biomedical | |
| - contrastive-learning | |
| - image-text-retrieval | |
| - zero-shot-classification | |
| # BioLoom Models | |
| BioLoom is a biomedical dual encoder for image–text retrieval and zero-shot classification, trained on 52.4 million image–text pairs from Loom52M. It uses whole figures, panels, captions, and relationships within scientific articles to supervise contrastive learning. The main models share a BioClinical ModernBERT text encoder and a ViT-L/14 vision encoder, with either randomly initialized or DFN-2B-initialized visual weights. | |
| [Loom52M training data](https://huggingface.co/datasets/ChartFoundation/Loom52M) · [Evaluation suite](https://huggingface.co/datasets/ChartFoundation/BioLoom-EvalSuite) · [Implementation](https://github.com/yuweiyang-anu/BioJSON) | |
| ## 📦 Model weights | |
| Each model directory contains the weights, architecture configuration, and evaluation files. | |
| - **Main models** · [BioLoom (Scratch)](https://huggingface.co/ChartFoundation/BioLoom-models/tree/main/Main/Scratch) · [BioLoom (DFN-2B)](https://huggingface.co/ChartFoundation/BioLoom-models/tree/main/Main/DFN-2B) | |
| - **Vision initialization** · [CommonPool](https://huggingface.co/ChartFoundation/BioLoom-models/tree/main/Initialization/CommonPool) · [DINOv2](https://huggingface.co/ChartFoundation/BioLoom-models/tree/main/Initialization/DINOv2) | |
| - **Model size** · [ViT-S](https://huggingface.co/ChartFoundation/BioLoom-models/tree/main/Model-size/ViT-S-14) · [ViT-B](https://huggingface.co/ChartFoundation/BioLoom-models/tree/main/Model-size/ViT-B-14) · [ViT-H](https://huggingface.co/ChartFoundation/BioLoom-models/tree/main/Model-size/ViT-H-14) | |
| <!-- | |
| ## Main results | |
| <table> | |
| <caption>Zero-shot classification across ten biomedical domains. Accuracy (%) on 67 tasks from 43 sources.</caption> | |
| <thead> | |
| <tr><th scope="col">Model</th><th scope="col">Params.</th><th scope="col">Cell</th><th scope="col">Struct.</th><th scope="col">Micro.</th><th scope="col">Cytology</th><th scope="col">Histo.</th><th scope="col">Radiol.</th><th scope="col">Ophth.</th><th scope="col">Derm.</th><th scope="col">Surgery</th><th scope="col">Microb.</th><th scope="col">Overall</th></tr> | |
| </thead> | |
| <tbody> | |
| <tr><th scope="row">Random</th><td>-</td><td>18.65</td><td>17.78</td><td>34.42</td><td>13.77</td><td>28.96</td><td>38.18</td><td>29.67</td><td>13.23</td><td>19.63</td><td>29.23</td><td>24.35</td></tr> | |
| <tr><th scope="row">CoCa<sup>*</sup></th><td>430M</td><td>22.18</td><td>33.45</td><td>56.86</td><td>41.56</td><td>39.80</td><td>36.31</td><td>50.53</td><td>17.37</td><td>23.05</td><td>31.48</td><td>35.26</td></tr> | |
| <tr><th scope="row">OpenCLIP<sup>*</sup></th><td>428M</td><td>24.49</td><td>44.44</td><td>59.04</td><td>10.25</td><td>41.03</td><td>64.86</td><td>42.80</td><td>15.67</td><td>31.26</td><td>44.87</td><td>37.87</td></tr> | |
| <tr><th scope="row">PMC-CLIP<sup>*</sup></th><td>137M</td><td>28.16</td><td>48.95</td><td>69.20</td><td>15.16</td><td>57.30</td><td>53.91</td><td>38.74</td><td>16.34</td><td>29.09</td><td>38.16</td><td>39.50</td></tr> | |
| <tr><th scope="row">MedSigLIP</th><td>800M</td><td>22.74</td><td>44.77</td><td>65.31</td><td>37.28</td><td>38.77</td><td>49.26</td><td>49.24</td><td>20.62</td><td>19.38</td><td><u>58.58</u></td><td>40.59</td></tr> | |
| <tr><th scope="row">BiomedCLIP</th><td>196M</td><td>24.11</td><td>49.59</td><td>74.55</td><td>11.16</td><td>53.02</td><td>55.36</td><td>38.27</td><td>30.48</td><td>41.83</td><td>43.63</td><td>42.20</td></tr> | |
| <tr><th scope="row">Panel2Patch</th><td>428M</td><td>29.40</td><td>47.12</td><td>75.66</td><td>9.50</td><td>52.30</td><td><u>63.31</u></td><td>47.56</td><td>43.76</td><td>37.34</td><td>27.77</td><td>43.37</td></tr> | |
| <tr><th scope="row">BMC-CLIP-CF</th><td>428M</td><td>24.79</td><td>56.53</td><td>69.39</td><td>8.97</td><td>51.34</td><td>60.85</td><td>46.38</td><td>43.76</td><td>41.35</td><td>32.25</td><td>43.56</td></tr> | |
| <tr><th scope="row">BMC-LongCLIP</th><td>454M</td><td>25.62</td><td>51.49</td><td>76.23</td><td>35.61</td><td>53.94</td><td>60.73</td><td>35.53</td><td>40.87</td><td>37.38</td><td>28.62</td><td>44.60</td></tr> | |
| <tr><th scope="row">MedPMC-CLIP</th><td>428M</td><td>28.60</td><td><u>57.60</u></td><td>66.17</td><td><u>38.18</u></td><td>59.49</td><td>55.57</td><td>51.40</td><td><u>45.49</u></td><td>35.11</td><td>50.11</td><td>48.77</td></tr> | |
| <tr><th scope="row">BioLoom (Scratch)<sup>*</sup></th><td>454M</td><td><strong>34.22</strong></td><td>56.69</td><td><strong>79.70</strong></td><td>37.96</td><td><u>61.81</u></td><td><strong>64.57</strong></td><td><u>56.65</u></td><td>38.75</td><td><u>52.65</u></td><td>45.90</td><td><u>52.89</u></td></tr> | |
| <tr><th scope="row">BioLoom (DFN-2B)</th><td>454M</td><td><u>31.05</u></td><td><strong>59.73</strong></td><td><u>76.32</u></td><td><strong>45.61</strong></td><td><strong>62.55</strong></td><td>61.40</td><td><strong>61.64</strong></td><td><strong>48.21</strong></td><td><strong>57.62</strong></td><td><strong>64.09</strong></td><td><strong>56.82</strong></td></tr> | |
| </tbody> | |
| </table> | |
| <table> | |
| <caption>Image–text retrieval on BioLoomBench. Recall (%) for concise and source captions.</caption> | |
| <thead> | |
| <tr><th scope="col" rowspan="3">Model</th><th colspan="6">Concise caption</th><th colspan="6">Source caption</th></tr> | |
| <tr><th colspan="3">I2T</th><th colspan="3">T2I</th><th colspan="3">I2T</th><th colspan="3">T2I</th></tr> | |
| <tr><th scope="col">R@1</th><th scope="col">R@10</th><th scope="col">R@100</th><th scope="col">R@1</th><th scope="col">R@10</th><th scope="col">R@100</th><th scope="col">R@1</th><th scope="col">R@10</th><th scope="col">R@100</th><th scope="col">R@1</th><th scope="col">R@10</th><th scope="col">R@100</th></tr> | |
| </thead> | |
| <tbody> | |
| <tr><th scope="row">CoCa<sup>*</sup></th><td>24.25</td><td>46.37</td><td>74.44</td><td>22.35</td><td>42.85</td><td>71.15</td><td>18.37</td><td>36.88</td><td>64.68</td><td>18.53</td><td>36.72</td><td>62.48</td></tr> | |
| <tr><th scope="row">OpenCLIP<sup>*</sup></th><td>27.65</td><td>50.47</td><td>76.93</td><td>25.69</td><td>47.82</td><td>75.24</td><td>25.34</td><td>47.37</td><td>73.49</td><td>24.68</td><td>45.66</td><td>71.75</td></tr> | |
| <tr><th scope="row">PMC-CLIP<sup>*</sup></th><td>4.75</td><td>14.76</td><td>33.11</td><td>4.84</td><td>15.87</td><td>35.96</td><td>4.47</td><td>13.69</td><td>30.34</td><td>6.05</td><td>18.04</td><td>37.51</td></tr> | |
| <tr><th scope="row">MedSigLIP</th><td>27.39</td><td>56.07</td><td>82.69</td><td>16.38</td><td>38.63</td><td>67.55</td><td>19.88</td><td>44.66</td><td>72.75</td><td>9.43</td><td>26.52</td><td>53.27</td></tr> | |
| <tr><th scope="row">MedPMC-CLIP</th><td>41.27</td><td>70.51</td><td>89.33</td><td>36.84</td><td>63.63</td><td>84.71</td><td>55.42</td><td>80.70</td><td>93.03</td><td>56.14</td><td>81.63</td><td>93.35</td></tr> | |
| <tr><th scope="row">Panel2Patch</th><td>42.43</td><td>71.55</td><td>92.02</td><td>36.96</td><td>64.62</td><td>87.45</td><td>47.75</td><td>74.83</td><td>91.61</td><td>49.57</td><td>77.01</td><td>92.84</td></tr> | |
| <tr><th scope="row">BMC-LongCLIP</th><td>44.39</td><td>78.15</td><td>95.83</td><td>35.47</td><td>68.14</td><td>90.77</td><td>67.35</td><td>87.45</td><td>95.83</td><td>67.17</td><td>87.65</td><td>95.90</td></tr> | |
| <tr><th scope="row">BMC-CLIP-CF</th><td>47.11</td><td>76.90</td><td>94.85</td><td>38.74</td><td>66.15</td><td>88.41</td><td>52.09</td><td>78.96</td><td>92.94</td><td>51.89</td><td>78.97</td><td>93.25</td></tr> | |
| <tr><th scope="row">BiomedCLIP</th><td>51.65</td><td>84.38</td><td>97.67</td><td>41.09</td><td>75.63</td><td>94.71</td><td>69.80</td><td>89.67</td><td>97.11</td><td>69.11</td><td>89.48</td><td>96.97</td></tr> | |
| <tr><th scope="row">BioLoom (Scratch)<sup>*</sup></th><td>61.93</td><td>90.61</td><td>98.79</td><td>52.33</td><td>83.86</td><td>97.44</td><td>75.91</td><td>92.94</td><td>98.09</td><td>74.22</td><td>91.52</td><td>97.43</td></tr> | |
| <tr><th scope="row">BioLoom (DFN-2B)</th><td><strong>66.81</strong></td><td><strong>92.81</strong></td><td><strong>99.28</strong></td><td><strong>59.59</strong></td><td><strong>88.81</strong></td><td><strong>98.77</strong></td><td><strong>79.29</strong></td><td><strong>94.57</strong></td><td><strong>98.39</strong></td><td><strong>78.34</strong></td><td><strong>94.10</strong></td><td><strong>98.41</strong></td></tr> | |
| </tbody> | |
| </table> | |
| <table> | |
| <caption>Image–text retrieval on BioMedFlickr. Recall (%) over 7,185 image–caption pairs under the revised same-caption protocol.</caption> | |
| <thead> | |
| <tr><th scope="col" rowspan="2">Model</th><th colspan="3">Image → Text</th><th colspan="3">Text → Image</th></tr> | |
| <tr><th scope="col">R@1</th><th scope="col">R@10</th><th scope="col">R@100</th><th scope="col">R@1</th><th scope="col">R@10</th><th scope="col">R@100</th></tr> | |
| </thead> | |
| <tbody> | |
| <tr><th scope="row">CoCa<sup>*</sup></th><td>1.64</td><td>5.58</td><td>15.30</td><td>2.06</td><td>7.95</td><td>23.98</td></tr> | |
| <tr><th scope="row">OpenCLIP<sup>*</sup></th><td>2.41</td><td>8.48</td><td>23.42</td><td>3.13</td><td>11.36</td><td>34.25</td></tr> | |
| <tr><th scope="row">PMC-CLIP<sup>*</sup></th><td>2.03</td><td>8.38</td><td>25.61</td><td>2.67</td><td>11.75</td><td>34.38</td></tr> | |
| <tr><th scope="row">MedSigLIP</th><td>3.56</td><td>10.83</td><td>31.45</td><td>4.62</td><td>15.18</td><td>38.78</td></tr> | |
| <tr><th scope="row">BiomedCLIP</th><td>4.76</td><td>14.04</td><td>35.46</td><td>4.68</td><td>19.05</td><td>50.45</td></tr> | |
| <tr><th scope="row">Panel2Patch</th><td>3.60</td><td>12.40</td><td>33.00</td><td>7.00</td><td>19.82</td><td>46.93</td></tr> | |
| <tr><th scope="row">BMC-LongCLIP</th><td>5.00</td><td>15.57</td><td>37.52</td><td>6.67</td><td>20.97</td><td>52.32</td></tr> | |
| <tr><th scope="row">BMC-CLIP-CF</th><td>4.86</td><td>14.18</td><td>35.81</td><td>6.99</td><td>22.95</td><td>50.47</td></tr> | |
| <tr><th scope="row">MedPMC-CLIP</th><td>5.94</td><td>19.94</td><td>45.26</td><td>9.16</td><td>31.61</td><td>61.38</td></tr> | |
| <tr><th scope="row">BioLoom (Scratch)<sup>*</sup></th><td>8.00</td><td>21.85</td><td>45.66</td><td>8.84</td><td>27.63</td><td>58.75</td></tr> | |
| <tr><th scope="row">BioLoom (DFN-2B)</th><td><strong>9.73</strong></td><td><strong>25.58</strong></td><td><strong>49.46</strong></td><td><strong>11.45</strong></td><td><strong>31.90</strong></td><td><strong>63.51</strong></td></tr> | |
| </tbody> | |
| </table> | |
| * denotes random vision-encoder initialization for image-text pretraining. I2T is image-to-text and T2I is text-to-image. | |
| --> | |
| ## 💻 Inference | |
| Install the project’s [OpenCLIP implementation](https://github.com/yuweiyang-anu/BioJSON) and its inference dependencies. From its checkout: | |
| ```bash | |
| pip install -r src/requirements.txt "transformers>=4.48,<5" | |
| export PYTHONPATH="$PWD/src:$PYTHONPATH" | |
| ``` | |
| Select a model folder above. Use its registered architecture and matching checkpoint together. | |
| ```python | |
| import json | |
| import torch | |
| from PIL import Image | |
| from huggingface_hub import hf_hub_download | |
| import open_clip | |
| repo = "ChartFoundation/BioLoom-models" | |
| variant = "Main/DFN-2B" | |
| device = "cuda" if torch.cuda.is_available() else "cpu" | |
| manifest_path = hf_hub_download(repo, f"{variant}/manifest.json") | |
| with open(manifest_path) as handle: | |
| spec = json.load(handle) | |
| architecture = spec["architecture"] | |
| config = hf_hub_download(repo, f"{variant}/{architecture}.json") | |
| checkpoint = hf_hub_download(repo, spec["checkpoint"]) | |
| open_clip.add_model_config(config) | |
| model, _, preprocess = open_clip.create_model_and_transforms( | |
| architecture, pretrained=checkpoint, device=device | |
| ) | |
| model.eval() | |
| tokenizer = open_clip.get_tokenizer(architecture) | |
| images = preprocess(Image.open("figure.jpg").convert("RGB")).unsqueeze(0).to(device) | |
| texts = tokenizer(["a microscopy image", "a radiograph"]).to(device) | |
| with torch.inference_mode(): | |
| image_features = torch.nn.functional.normalize(model.encode_image(images), dim=-1) | |
| text_features = torch.nn.functional.normalize(model.encode_text(texts), dim=-1) | |
| scores = image_features @ text_features.T | |
| print(scores.cpu()) | |
| ``` | |
| ## 🧪 Evaluation | |
| To evaluate a checkpoint, use the data and protocols in [BioLoom-EvalSuite](https://huggingface.co/datasets/ChartFoundation/BioLoom-EvalSuite). Load the checkpoint and its matching architecture as shown above, then follow the zero-shot classification, BioMedFlickr, or BioLoomBench guide. Each model’s `results/` folder contains three files: `zero-shot.json` for all 67 classification tasks, `biomedflickr-retrieval.json` for BioMedFlickr, and `bioloom-retrieval.json` for BioLoomBench. | |