DermFM-Zero: A vision–language foundation model for dermatology for zero-shot diagnosis, clinical collaboration and automated concept discovery

About this checkpoint. These weights were trained on 517,455 publicly available image–text pairs, because the corpus used in the paper includes in-house pairs that cannot be redistributed. They perform on par with the paper checkpoint (mean zero-shot score 0.691 vs 0.675 across seven benchmarks); results in the paper refer to the paper checkpoint. Details: technical report.

What is DermFM-Zero?

DermFM-Zero pairs a PanDerm ViT-L/16 vision encoder with a knowledge-enhanced PubMedBERT text encoder, aligned on dermatology image–text pairs with multi-aspect knowledge contrastive learning. It classifies dermoscopic and clinical images from text prompts without task-specific training, retrieves images from text and text from images, serves as a backbone for multimodal fine-tuning and VQA, and exposes interpretable concepts through sparse autoencoders. In the paper it was evaluated on 20 benchmarks and in three multinational reader studies with 761 clinicians.

Quick Start

Clone the code repository and use the bundled open_clip fork (the upstream open_clip package does not load this checkpoint):

git clone https://github.com/SiyuanYan1/DermFM-Zero.git
cd DermFM-Zero
conda create -n dermfm-zero python=3.9.20 && conda activate dermfm-zero
pip install -r requirements.txt

import sys, torch
from PIL import Image
sys.path.insert(0, "src")            # bundled open_clip fork; run from the repo root
import open_clip

model, _, preprocess = open_clip.create_model_and_transforms("hf-hub:redlessone/DermFM-Zero")
tokenizer = open_clip.get_tokenizer("hf-hub:redlessone/DermFM-Zero")
model.eval()

image = preprocess(Image.open("examples/PAT_8_15_820.png")).unsqueeze(0)
classnames = ["nevus", "basal cell carcinoma", "actinic keratosis",
              "seborrheic keratosis", "squamous cell carcinoma", "melanoma"]
text = tokenizer([f"This is a skin image of {c}." for c in classnames])

with torch.no_grad():
    image_features = model.encode_image(image)
    text_features  = model.encode_text(text)
    image_features /= image_features.norm(dim=-1, keepdim=True)
    text_features  /= text_features.norm(dim=-1, keepdim=True)

probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)
print(classnames[probs.argmax().item()])    # → basal cell carcinoma

For best results, average text features over the seven prompt templates used in the paper (Supplementary Table 41):

PROMPT_TEMPLATES = [
    "This is a skin image of {}.",
    "A skin image of {}.",
    "An image of {}, a skin condition.",
    "{}, a skin disorder, is shown in this image.",
    "The skin lesion depicted is {}.",
    "The skin cancer in this image is {}.",
    "This image depicts {}, a type of skin cancer.",
]

Scripts for retrieval, linear probing, multimodal fine-tuning, VQA, concept discovery and the reader-study analyses are in the code repository.

Zero-shot Benchmarks

Top-1 accuracy (macro-F1 for the two binary datasets ISIC2020 and PH2), evaluated at 224 × 224. Full results for all tasks are in the technical report.

Model	HAM (7-D)	PAD (6-C)	ISIC2020 (2-D)	PH2 (2-D)	SNU (134-C)	SD-128 (128-C)	Daffodil (5-D)	Average
Task	Skin Cancer	Skin Cancer	Mel Det.	Mel Det.	DDX	DDX	Rare DX	-
Metric	ACC	ACC	Macro F1	Macro F1	ACC	ACC	ACC	-
CLIP-Large	0.2754	0.3839	0.4896	0.4494	0.0857	0.1210	0.5304	0.3336
BiomedCLIP	0.6347	0.4512	0.5317	0.6292	0.0966	0.1153	0.5785	0.4339
MONET	0.3347	0.4729	0.5208	0.6774	0.1414	0.2028	0.7607	0.4444
MAKE	0.4551	0.5857	0.4259	0.8222	0.3260	0.3886	0.7785	0.5403
DermLIP-PanDerm	0.6281	0.6247	0.5190	0.6799	0.3332	0.3822	0.7812	0.5640
DermFM-Zero (paper checkpoint)	0.7957	0.6941	0.5979	0.7998	0.4450	0.5075	0.8848	0.6750
DermFM-Zero (released)	0.7858	0.7592	0.6112	0.8708	0.4212	0.5196	0.8686	0.6909

Model Details

• Vision encoder: PanDerm-Large (ViT-L/16), initialised from the public PanDerm checkpoint and extended with NaViT-style patch-and-pack for native-resolution input (ScaleJitter during training; all reported results use 224 × 224).
• Text encoder: PubMedBERT-256, pretrained on a Derm1M knowledge tree of 556,372 attribute texts with the AdaSP loss following KEP; also kept frozen as a knowledge encoder.
• Training data: 517,455 image–text pairs (406,831 clinical, 110,624 dermoscopic) from ISIC, BCN20000, MSKCC, DermNet, Fitzpatrick17k, Derm12345, HIBA and the literature, textbook, forum and video sources of Derm1M. The 38,404 ISIC images used are listed in pretrain_image_lists/isic_image_ids.txt.
• Objective: MAKE multi-aspect contrastive alignment (raw caption, disease aspect, concept aspect, sub-captions; patch–subtext fine-grained alignment) plus knowledge distillation to the frozen text encoder.
• Optimisation: AdamW (lr 1×10⁻⁴, weight decay 0.1), 200 warm-up steps, cosine decay, effective batch 2,048, 15 epochs, bf16, 2 × NVIDIA H200 (~1 d 9 h).
• Resolution: 224 × 224 for all reported results; the encoder accepts any image size.

Intended Use and License

Released under CC BY-NC-ND 4.0 for non-commercial academic research: zero-shot dermatological diagnosis and retrieval, label-efficient learning, multimodal fine-tuning, and concept discovery with sparse autoencoders. The weights and derivative checkpoints may not be redistributed. DermFM-Zero is not a medical device and must not be used for clinical diagnosis, patient management or deployment without further validation. Please cite the paper when using the model.

Limitations

• Disease coverage: ~400 skin conditions in pretraining; rare tropical diseases, complex systemic dermatoses, and rare genetic disorders remain underrepresented.
• Source balance: Most of the corpus comes from web and literature sources rather than consecutive clinical cohorts, with curated clinical archives making up a minority of pairs. The corpus is therefore not population-representative, and performance on any specific patient population should be empirically validated before use.
• Retrospective evaluation only: All clinical validation was conducted in store-and-forward teledermatology workflows, not in live patient encounters.
• Skin-tone fairness in collaboration: Standalone fairness was characterized, but skin-tone-stratified fairness within human-AI workflows was not formally analyzed.
• Data overlap: Despite source-level separation and SSCD image-level deduplication, some images may appear in both pretraining and downstream benchmarks via published literature in our PubMed and web corpora. Image-level overlap rates against the zero-shot benchmarks range from 0.14% (SNU) to 13.51% (Daffodil) at a cosine threshold of 0.75. That threshold is permissive: above 0.85 the rates fall sharply (e.g. ISIC2020 1.57% → 0.08%), and no zero-shot benchmark retains a near-exact duplicate (≥ 0.99). Per-dataset counts and the flagged image lists are published in the code repository under data_deduplication/results_released/.

Citation

@misc{yan2026dermfmzero,
  title={A Vision-Language Foundation Model for Zero-shot Clinical Collaboration and Automated Concept Discovery in Dermatology},
  author={Siyuan Yan and Xieji Li and Dan Mo and Philipp Tschandl and Yiwen Jiang and Zhonghua Wang and Ming Hu and Lie Ju and Cristina Vico-Alonso and Yizhen Zheng and Jiahe Liu and Juexiao Zhou and Camilla Chello and Jen G. Cheung and Julien Anriot and Luc Thomas and Clare Primiero and Gin Tan and Aik Beng Ng and Simon See and Xiaoying Tang and Albert Ip and Xiaoyang Liao and Adrian Bowling and Martin Haskett and Shuang Zhao and Monika Janda and H. Peter Soyer and Victoria Mar and Harald Kittler and Zongyuan Ge},
  year={2026},
  eprint={2602.10624},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2602.10624}
}

The vision encoder is initialised from PanDerm; please also cite:

@article{yan2025multimodal,
  title={A multimodal vision foundation model for clinical dermatology},
  author={Yan, Siyuan and Yu, Zhen and Primiero, Clare and Vico-Alonso, Cristina and Wang, Zhonghua and Yang, Litao and Tschandl, Philipp and Hu, Ming and Ju, Lie and Tan, Gin and others},
  journal={Nature Medicine},
  volume={31},
  pages={2691--2702},
  year={2025}
}

Contact

Siyuan Yan — Monash University · siyuan.yan@monash.edu