BioLoom-models / README.md
Ruler138's picture
Initial commit
963bc8a
|
Raw History Blame Contribute Delete
12.7 kB
metadata
library_name: open_clip
pipeline_tag: zero-shot-image-classification
tags:
  - biomedical
  - contrastive-learning
  - image-text-retrieval
  - zero-shot-classification

BioLoom Models

BioLoom is a biomedical dual encoder for image鈥搕ext retrieval and zero-shot classification, trained on 52.4 million image鈥搕ext pairs from Loom52M. It uses whole figures, panels, captions, and relationships within scientific articles to supervise contrastive learning. The main models share a BioClinical ModernBERT text encoder and a ViT-L/14 vision encoder, with either randomly initialized or DFN-2B-initialized visual weights.

Loom52M training data 路 Evaluation suite 路 Implementation

馃摝 Model weights

Each model directory contains the weights, architecture configuration, and evaluation files.

馃捇 Inference

Install the project鈥檚 OpenCLIP implementation and its inference dependencies. From its checkout:

pip install -r src/requirements.txt "transformers>=4.48,<5"
export PYTHONPATH="$PWD/src:$PYTHONPATH"

Select a model folder above. Use its registered architecture and matching checkpoint together.

import json
import torch
from PIL import Image
from huggingface_hub import hf_hub_download
import open_clip

repo = "ChartFoundation/BioLoom-models"
variant = "Main/DFN-2B"
device = "cuda" if torch.cuda.is_available() else "cpu"

manifest_path = hf_hub_download(repo, f"{variant}/manifest.json")
with open(manifest_path) as handle:
    spec = json.load(handle)
architecture = spec["architecture"]
config = hf_hub_download(repo, f"{variant}/{architecture}.json")
checkpoint = hf_hub_download(repo, spec["checkpoint"])
open_clip.add_model_config(config)
model, _, preprocess = open_clip.create_model_and_transforms(
    architecture, pretrained=checkpoint, device=device
)
model.eval()
tokenizer = open_clip.get_tokenizer(architecture)
images = preprocess(Image.open("figure.jpg").convert("RGB")).unsqueeze(0).to(device)
texts = tokenizer(["a microscopy image", "a radiograph"]).to(device)
with torch.inference_mode():
    image_features = torch.nn.functional.normalize(model.encode_image(images), dim=-1)
    text_features = torch.nn.functional.normalize(model.encode_text(texts), dim=-1)
    scores = image_features @ text_features.T
print(scores.cpu())

馃И Evaluation

To evaluate a checkpoint, use the data and protocols in BioLoom-EvalSuite. Load the checkpoint and its matching architecture as shown above, then follow the zero-shot classification, BioMedFlickr, or BioLoomBench guide. Each model鈥檚 results/ folder contains three files: zero-shot.json for all 67 classification tasks, biomedflickr-retrieval.json for BioMedFlickr, and bioloom-retrieval.json for BioLoomBench.