You need to agree to share your Hugging Face information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

Log in or Sign Up to review the conditions and access this model content.

BioLoom Models

BioLoom is a biomedical dual encoder for image鈥搕ext retrieval and zero-shot classification, trained on 52.4 million image鈥搕ext pairs from Loom52M. It uses whole figures, panels, captions, and relationships within scientific articles to supervise contrastive learning. The main models share a BioClinical ModernBERT text encoder and a ViT-L/14 vision encoder, with either randomly initialized or DFN-2B-initialized visual weights.

Loom52M training data 路 Evaluation suite 路 Implementation

馃摝 Model weights

Each model directory contains the weights, architecture configuration, and evaluation files.

馃捇 Inference

Install the project鈥檚 OpenCLIP implementation and its inference dependencies. From its checkout:

pip install -r src/requirements.txt "transformers>=4.48,<5"
export PYTHONPATH="$PWD/src:$PYTHONPATH"

Select a model folder above. Use its registered architecture and matching checkpoint together.

import json
import torch
from PIL import Image
from huggingface_hub import hf_hub_download
import open_clip

repo = "ChartFoundation/BioLoom-models"
variant = "Main/DFN-2B"
device = "cuda" if torch.cuda.is_available() else "cpu"

manifest_path = hf_hub_download(repo, f"{variant}/manifest.json")
with open(manifest_path) as handle:
    spec = json.load(handle)
architecture = spec["architecture"]
config = hf_hub_download(repo, f"{variant}/{architecture}.json")
checkpoint = hf_hub_download(repo, spec["checkpoint"])
open_clip.add_model_config(config)
model, _, preprocess = open_clip.create_model_and_transforms(
    architecture, pretrained=checkpoint, device=device
)
model.eval()
tokenizer = open_clip.get_tokenizer(architecture)
images = preprocess(Image.open("figure.jpg").convert("RGB")).unsqueeze(0).to(device)
texts = tokenizer(["a microscopy image", "a radiograph"]).to(device)
with torch.inference_mode():
    image_features = torch.nn.functional.normalize(model.encode_image(images), dim=-1)
    text_features = torch.nn.functional.normalize(model.encode_text(texts), dim=-1)
    scores = image_features @ text_features.T
print(scores.cpu())

馃И Evaluation

To evaluate a checkpoint, use the data and protocols in BioLoom-EvalSuite. Load the checkpoint and its matching architecture as shown above, then follow the zero-shot classification, BioMedFlickr, or BioLoomBench guide. Each model鈥檚 results/ folder contains three files: zero-shot.json for all 67 classification tasks, biomedflickr-retrieval.json for BioMedFlickr, and bioloom-retrieval.json for BioLoomBench.

Downloads last month
-
Inference Providers NEW
This model isn't deployed by any Inference Provider. 馃檵 Ask for provider support