Instructions to use ChartFoundation/BioLoom-models with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- OpenCLIP
How to use ChartFoundation/BioLoom-models with OpenCLIP:
import open_clip model, preprocess_train, preprocess_val = open_clip.create_model_and_transforms('hf-hub:ChartFoundation/BioLoom-models') tokenizer = open_clip.get_tokenizer('hf-hub:ChartFoundation/BioLoom-models') - Notebooks
- Google Colab
- Kaggle
Download README.md from ChartFoundation/BioLoom-models: direct link, hf CLI and curl.
- Browser
- Download file 12.7 kB
-
https://huggingface.co/ChartFoundation/BioLoom-models/resolve/main/README.md
- Command line
-
hf download hf://ChartFoundation/BioLoom-models/README.md
-
curl -L -H "Authorization: Bearer $HF_TOKEN" -o README.md https://huggingface.co/ChartFoundation/BioLoom-models/resolve/main/README.md
library_name: open_clip
pipeline_tag: zero-shot-image-classification
tags:
- biomedical
- contrastive-learning
- image-text-retrieval
- zero-shot-classification
BioLoom Models
BioLoom is a biomedical dual encoder for image鈥搕ext retrieval and zero-shot classification, trained on 52.4 million image鈥搕ext pairs from Loom52M. It uses whole figures, panels, captions, and relationships within scientific articles to supervise contrastive learning. The main models share a BioClinical ModernBERT text encoder and a ViT-L/14 vision encoder, with either randomly initialized or DFN-2B-initialized visual weights.
Loom52M training data 路 Evaluation suite 路 Implementation
馃摝 Model weights
Each model directory contains the weights, architecture configuration, and evaluation files.
- Main models 路 BioLoom (Scratch) 路 BioLoom (DFN-2B)
- Vision initialization 路 CommonPool 路 DINOv2
- Model size 路 ViT-S 路 ViT-B 路 ViT-H
馃捇 Inference
Install the project鈥檚 OpenCLIP implementation and its inference dependencies. From its checkout:
pip install -r src/requirements.txt "transformers>=4.48,<5"
export PYTHONPATH="$PWD/src:$PYTHONPATH"
Select a model folder above. Use its registered architecture and matching checkpoint together.
import json
import torch
from PIL import Image
from huggingface_hub import hf_hub_download
import open_clip
repo = "ChartFoundation/BioLoom-models"
variant = "Main/DFN-2B"
device = "cuda" if torch.cuda.is_available() else "cpu"
manifest_path = hf_hub_download(repo, f"{variant}/manifest.json")
with open(manifest_path) as handle:
spec = json.load(handle)
architecture = spec["architecture"]
config = hf_hub_download(repo, f"{variant}/{architecture}.json")
checkpoint = hf_hub_download(repo, spec["checkpoint"])
open_clip.add_model_config(config)
model, _, preprocess = open_clip.create_model_and_transforms(
architecture, pretrained=checkpoint, device=device
)
model.eval()
tokenizer = open_clip.get_tokenizer(architecture)
images = preprocess(Image.open("figure.jpg").convert("RGB")).unsqueeze(0).to(device)
texts = tokenizer(["a microscopy image", "a radiograph"]).to(device)
with torch.inference_mode():
image_features = torch.nn.functional.normalize(model.encode_image(images), dim=-1)
text_features = torch.nn.functional.normalize(model.encode_text(texts), dim=-1)
scores = image_features @ text_features.T
print(scores.cpu())
馃И Evaluation
To evaluate a checkpoint, use the data and protocols in BioLoom-EvalSuite. Load the checkpoint and its matching architecture as shown above, then follow the zero-shot classification, BioMedFlickr, or BioLoomBench guide. Each model鈥檚 results/ folder contains three files: zero-shot.json for all 67 classification tasks, biomedflickr-retrieval.json for BioMedFlickr, and bioloom-retrieval.json for BioLoomBench.