BioLoom-models / README.md
Ruler138's picture
Initial commit
963bc8a
|
Raw History Blame Contribute Delete
12.7 kB
---
library_name: open_clip
pipeline_tag: zero-shot-image-classification
tags:
- biomedical
- contrastive-learning
- image-text-retrieval
- zero-shot-classification
---
# BioLoom Models
BioLoom is a biomedical dual encoder for image–text retrieval and zero-shot classification, trained on 52.4 million image–text pairs from Loom52M. It uses whole figures, panels, captions, and relationships within scientific articles to supervise contrastive learning. The main models share a BioClinical ModernBERT text encoder and a ViT-L/14 vision encoder, with either randomly initialized or DFN-2B-initialized visual weights.
[Loom52M training data](https://huggingface.co/datasets/ChartFoundation/Loom52M) · [Evaluation suite](https://huggingface.co/datasets/ChartFoundation/BioLoom-EvalSuite) · [Implementation](https://github.com/yuweiyang-anu/BioJSON)
## 📦 Model weights
Each model directory contains the weights, architecture configuration, and evaluation files.
- **Main models** · [BioLoom (Scratch)](https://huggingface.co/ChartFoundation/BioLoom-models/tree/main/Main/Scratch) · [BioLoom (DFN-2B)](https://huggingface.co/ChartFoundation/BioLoom-models/tree/main/Main/DFN-2B)
- **Vision initialization** · [CommonPool](https://huggingface.co/ChartFoundation/BioLoom-models/tree/main/Initialization/CommonPool) · [DINOv2](https://huggingface.co/ChartFoundation/BioLoom-models/tree/main/Initialization/DINOv2)
- **Model size** · [ViT-S](https://huggingface.co/ChartFoundation/BioLoom-models/tree/main/Model-size/ViT-S-14) · [ViT-B](https://huggingface.co/ChartFoundation/BioLoom-models/tree/main/Model-size/ViT-B-14) · [ViT-H](https://huggingface.co/ChartFoundation/BioLoom-models/tree/main/Model-size/ViT-H-14)
<!--
## Main results
<table>
<caption>Zero-shot classification across ten biomedical domains. Accuracy (%) on 67 tasks from 43 sources.</caption>
<thead>
<tr><th scope="col">Model</th><th scope="col">Params.</th><th scope="col">Cell</th><th scope="col">Struct.</th><th scope="col">Micro.</th><th scope="col">Cytology</th><th scope="col">Histo.</th><th scope="col">Radiol.</th><th scope="col">Ophth.</th><th scope="col">Derm.</th><th scope="col">Surgery</th><th scope="col">Microb.</th><th scope="col">Overall</th></tr>
</thead>
<tbody>
<tr><th scope="row">Random</th><td>-</td><td>18.65</td><td>17.78</td><td>34.42</td><td>13.77</td><td>28.96</td><td>38.18</td><td>29.67</td><td>13.23</td><td>19.63</td><td>29.23</td><td>24.35</td></tr>
<tr><th scope="row">CoCa<sup>*</sup></th><td>430M</td><td>22.18</td><td>33.45</td><td>56.86</td><td>41.56</td><td>39.80</td><td>36.31</td><td>50.53</td><td>17.37</td><td>23.05</td><td>31.48</td><td>35.26</td></tr>
<tr><th scope="row">OpenCLIP<sup>*</sup></th><td>428M</td><td>24.49</td><td>44.44</td><td>59.04</td><td>10.25</td><td>41.03</td><td>64.86</td><td>42.80</td><td>15.67</td><td>31.26</td><td>44.87</td><td>37.87</td></tr>
<tr><th scope="row">PMC-CLIP<sup>*</sup></th><td>137M</td><td>28.16</td><td>48.95</td><td>69.20</td><td>15.16</td><td>57.30</td><td>53.91</td><td>38.74</td><td>16.34</td><td>29.09</td><td>38.16</td><td>39.50</td></tr>
<tr><th scope="row">MedSigLIP</th><td>800M</td><td>22.74</td><td>44.77</td><td>65.31</td><td>37.28</td><td>38.77</td><td>49.26</td><td>49.24</td><td>20.62</td><td>19.38</td><td><u>58.58</u></td><td>40.59</td></tr>
<tr><th scope="row">BiomedCLIP</th><td>196M</td><td>24.11</td><td>49.59</td><td>74.55</td><td>11.16</td><td>53.02</td><td>55.36</td><td>38.27</td><td>30.48</td><td>41.83</td><td>43.63</td><td>42.20</td></tr>
<tr><th scope="row">Panel2Patch</th><td>428M</td><td>29.40</td><td>47.12</td><td>75.66</td><td>9.50</td><td>52.30</td><td><u>63.31</u></td><td>47.56</td><td>43.76</td><td>37.34</td><td>27.77</td><td>43.37</td></tr>
<tr><th scope="row">BMC-CLIP-CF</th><td>428M</td><td>24.79</td><td>56.53</td><td>69.39</td><td>8.97</td><td>51.34</td><td>60.85</td><td>46.38</td><td>43.76</td><td>41.35</td><td>32.25</td><td>43.56</td></tr>
<tr><th scope="row">BMC-LongCLIP</th><td>454M</td><td>25.62</td><td>51.49</td><td>76.23</td><td>35.61</td><td>53.94</td><td>60.73</td><td>35.53</td><td>40.87</td><td>37.38</td><td>28.62</td><td>44.60</td></tr>
<tr><th scope="row">MedPMC-CLIP</th><td>428M</td><td>28.60</td><td><u>57.60</u></td><td>66.17</td><td><u>38.18</u></td><td>59.49</td><td>55.57</td><td>51.40</td><td><u>45.49</u></td><td>35.11</td><td>50.11</td><td>48.77</td></tr>
<tr><th scope="row">BioLoom (Scratch)<sup>*</sup></th><td>454M</td><td><strong>34.22</strong></td><td>56.69</td><td><strong>79.70</strong></td><td>37.96</td><td><u>61.81</u></td><td><strong>64.57</strong></td><td><u>56.65</u></td><td>38.75</td><td><u>52.65</u></td><td>45.90</td><td><u>52.89</u></td></tr>
<tr><th scope="row">BioLoom (DFN-2B)</th><td>454M</td><td><u>31.05</u></td><td><strong>59.73</strong></td><td><u>76.32</u></td><td><strong>45.61</strong></td><td><strong>62.55</strong></td><td>61.40</td><td><strong>61.64</strong></td><td><strong>48.21</strong></td><td><strong>57.62</strong></td><td><strong>64.09</strong></td><td><strong>56.82</strong></td></tr>
</tbody>
</table>
<table>
<caption>Image–text retrieval on BioLoomBench. Recall (%) for concise and source captions.</caption>
<thead>
<tr><th scope="col" rowspan="3">Model</th><th colspan="6">Concise caption</th><th colspan="6">Source caption</th></tr>
<tr><th colspan="3">I2T</th><th colspan="3">T2I</th><th colspan="3">I2T</th><th colspan="3">T2I</th></tr>
<tr><th scope="col">R@1</th><th scope="col">R@10</th><th scope="col">R@100</th><th scope="col">R@1</th><th scope="col">R@10</th><th scope="col">R@100</th><th scope="col">R@1</th><th scope="col">R@10</th><th scope="col">R@100</th><th scope="col">R@1</th><th scope="col">R@10</th><th scope="col">R@100</th></tr>
</thead>
<tbody>
<tr><th scope="row">CoCa<sup>*</sup></th><td>24.25</td><td>46.37</td><td>74.44</td><td>22.35</td><td>42.85</td><td>71.15</td><td>18.37</td><td>36.88</td><td>64.68</td><td>18.53</td><td>36.72</td><td>62.48</td></tr>
<tr><th scope="row">OpenCLIP<sup>*</sup></th><td>27.65</td><td>50.47</td><td>76.93</td><td>25.69</td><td>47.82</td><td>75.24</td><td>25.34</td><td>47.37</td><td>73.49</td><td>24.68</td><td>45.66</td><td>71.75</td></tr>
<tr><th scope="row">PMC-CLIP<sup>*</sup></th><td>4.75</td><td>14.76</td><td>33.11</td><td>4.84</td><td>15.87</td><td>35.96</td><td>4.47</td><td>13.69</td><td>30.34</td><td>6.05</td><td>18.04</td><td>37.51</td></tr>
<tr><th scope="row">MedSigLIP</th><td>27.39</td><td>56.07</td><td>82.69</td><td>16.38</td><td>38.63</td><td>67.55</td><td>19.88</td><td>44.66</td><td>72.75</td><td>9.43</td><td>26.52</td><td>53.27</td></tr>
<tr><th scope="row">MedPMC-CLIP</th><td>41.27</td><td>70.51</td><td>89.33</td><td>36.84</td><td>63.63</td><td>84.71</td><td>55.42</td><td>80.70</td><td>93.03</td><td>56.14</td><td>81.63</td><td>93.35</td></tr>
<tr><th scope="row">Panel2Patch</th><td>42.43</td><td>71.55</td><td>92.02</td><td>36.96</td><td>64.62</td><td>87.45</td><td>47.75</td><td>74.83</td><td>91.61</td><td>49.57</td><td>77.01</td><td>92.84</td></tr>
<tr><th scope="row">BMC-LongCLIP</th><td>44.39</td><td>78.15</td><td>95.83</td><td>35.47</td><td>68.14</td><td>90.77</td><td>67.35</td><td>87.45</td><td>95.83</td><td>67.17</td><td>87.65</td><td>95.90</td></tr>
<tr><th scope="row">BMC-CLIP-CF</th><td>47.11</td><td>76.90</td><td>94.85</td><td>38.74</td><td>66.15</td><td>88.41</td><td>52.09</td><td>78.96</td><td>92.94</td><td>51.89</td><td>78.97</td><td>93.25</td></tr>
<tr><th scope="row">BiomedCLIP</th><td>51.65</td><td>84.38</td><td>97.67</td><td>41.09</td><td>75.63</td><td>94.71</td><td>69.80</td><td>89.67</td><td>97.11</td><td>69.11</td><td>89.48</td><td>96.97</td></tr>
<tr><th scope="row">BioLoom (Scratch)<sup>*</sup></th><td>61.93</td><td>90.61</td><td>98.79</td><td>52.33</td><td>83.86</td><td>97.44</td><td>75.91</td><td>92.94</td><td>98.09</td><td>74.22</td><td>91.52</td><td>97.43</td></tr>
<tr><th scope="row">BioLoom (DFN-2B)</th><td><strong>66.81</strong></td><td><strong>92.81</strong></td><td><strong>99.28</strong></td><td><strong>59.59</strong></td><td><strong>88.81</strong></td><td><strong>98.77</strong></td><td><strong>79.29</strong></td><td><strong>94.57</strong></td><td><strong>98.39</strong></td><td><strong>78.34</strong></td><td><strong>94.10</strong></td><td><strong>98.41</strong></td></tr>
</tbody>
</table>
<table>
<caption>Image–text retrieval on BioMedFlickr. Recall (%) over 7,185 image–caption pairs under the revised same-caption protocol.</caption>
<thead>
<tr><th scope="col" rowspan="2">Model</th><th colspan="3">Image → Text</th><th colspan="3">Text → Image</th></tr>
<tr><th scope="col">R@1</th><th scope="col">R@10</th><th scope="col">R@100</th><th scope="col">R@1</th><th scope="col">R@10</th><th scope="col">R@100</th></tr>
</thead>
<tbody>
<tr><th scope="row">CoCa<sup>*</sup></th><td>1.64</td><td>5.58</td><td>15.30</td><td>2.06</td><td>7.95</td><td>23.98</td></tr>
<tr><th scope="row">OpenCLIP<sup>*</sup></th><td>2.41</td><td>8.48</td><td>23.42</td><td>3.13</td><td>11.36</td><td>34.25</td></tr>
<tr><th scope="row">PMC-CLIP<sup>*</sup></th><td>2.03</td><td>8.38</td><td>25.61</td><td>2.67</td><td>11.75</td><td>34.38</td></tr>
<tr><th scope="row">MedSigLIP</th><td>3.56</td><td>10.83</td><td>31.45</td><td>4.62</td><td>15.18</td><td>38.78</td></tr>
<tr><th scope="row">BiomedCLIP</th><td>4.76</td><td>14.04</td><td>35.46</td><td>4.68</td><td>19.05</td><td>50.45</td></tr>
<tr><th scope="row">Panel2Patch</th><td>3.60</td><td>12.40</td><td>33.00</td><td>7.00</td><td>19.82</td><td>46.93</td></tr>
<tr><th scope="row">BMC-LongCLIP</th><td>5.00</td><td>15.57</td><td>37.52</td><td>6.67</td><td>20.97</td><td>52.32</td></tr>
<tr><th scope="row">BMC-CLIP-CF</th><td>4.86</td><td>14.18</td><td>35.81</td><td>6.99</td><td>22.95</td><td>50.47</td></tr>
<tr><th scope="row">MedPMC-CLIP</th><td>5.94</td><td>19.94</td><td>45.26</td><td>9.16</td><td>31.61</td><td>61.38</td></tr>
<tr><th scope="row">BioLoom (Scratch)<sup>*</sup></th><td>8.00</td><td>21.85</td><td>45.66</td><td>8.84</td><td>27.63</td><td>58.75</td></tr>
<tr><th scope="row">BioLoom (DFN-2B)</th><td><strong>9.73</strong></td><td><strong>25.58</strong></td><td><strong>49.46</strong></td><td><strong>11.45</strong></td><td><strong>31.90</strong></td><td><strong>63.51</strong></td></tr>
</tbody>
</table>
* denotes random vision-encoder initialization for image-text pretraining. I2T is image-to-text and T2I is text-to-image.
-->
## 💻 Inference
Install the project’s [OpenCLIP implementation](https://github.com/yuweiyang-anu/BioJSON) and its inference dependencies. From its checkout:
```bash
pip install -r src/requirements.txt "transformers>=4.48,<5"
export PYTHONPATH="$PWD/src:$PYTHONPATH"
```
Select a model folder above. Use its registered architecture and matching checkpoint together.
```python
import json
import torch
from PIL import Image
from huggingface_hub import hf_hub_download
import open_clip
repo = "ChartFoundation/BioLoom-models"
variant = "Main/DFN-2B"
device = "cuda" if torch.cuda.is_available() else "cpu"
manifest_path = hf_hub_download(repo, f"{variant}/manifest.json")
with open(manifest_path) as handle:
spec = json.load(handle)
architecture = spec["architecture"]
config = hf_hub_download(repo, f"{variant}/{architecture}.json")
checkpoint = hf_hub_download(repo, spec["checkpoint"])
open_clip.add_model_config(config)
model, _, preprocess = open_clip.create_model_and_transforms(
architecture, pretrained=checkpoint, device=device
)
model.eval()
tokenizer = open_clip.get_tokenizer(architecture)
images = preprocess(Image.open("figure.jpg").convert("RGB")).unsqueeze(0).to(device)
texts = tokenizer(["a microscopy image", "a radiograph"]).to(device)
with torch.inference_mode():
image_features = torch.nn.functional.normalize(model.encode_image(images), dim=-1)
text_features = torch.nn.functional.normalize(model.encode_text(texts), dim=-1)
scores = image_features @ text_features.T
print(scores.cpu())
```
## 🧪 Evaluation
To evaluate a checkpoint, use the data and protocols in [BioLoom-EvalSuite](https://huggingface.co/datasets/ChartFoundation/BioLoom-EvalSuite). Load the checkpoint and its matching architecture as shown above, then follow the zero-shot classification, BioMedFlickr, or BioLoomBench guide. Each model’s `results/` folder contains three files: `zero-shot.json` for all 67 classification tasks, `biomedflickr-retrieval.json` for BioMedFlickr, and `bioloom-retrieval.json` for BioLoomBench.