HandXFM

HandXFM: Semantic-Structural Distillation for Hand Radiograph Foundation Models
Accepted at NeurIPS 2026 · Paper

HandXFM is a foundation model for hand X-ray analysis. Built on DINOv3 ViT-B/16, it learns transferable visual representations through semantic-structural distillation from BiomedCLIP and a chest X-ray ViT. The model can serve as a backbone for bone age estimation, rheumatoid arthritis scoring, abnormality classification, joint detection, bone segmentation, and visual question answering.

This repository provides the pretrained HandXFM vision encoder (without the pretraining teacher models or downstream task-specific heads).

Usage

import torch
from PIL import Image
from transformers import AutoImageProcessor, AutoModel

model_id = "Yuesai/HandXFM"
processor = AutoImageProcessor.from_pretrained(model_id)
model = AutoModel.from_pretrained(model_id).eval()

image = Image.open("hand_xray.png").convert("RGB")
inputs = processor(images=image, return_tensors="pt")

with torch.no_grad():
    tokens = model(**inputs).last_hidden_state

# DINOv3 token order: [CLS, register tokens, patch tokens]
num_register_tokens = getattr(model.config, "num_register_tokens", 0)
cls_embedding = tokens[:, 0, :]                         # [B, 768]
patch_embeddings = tokens[:, 1 + num_register_tokens:, :]  # [B, 196, 768] for 224x224

# Optional: mean-pool CLS + patch tokens (excluding register tokens)
features = torch.cat([cls_embedding.unsqueeze(1), patch_embeddings], dim=1)
pooled_embedding = features.mean(dim=1)                  # [B, 768]

print("CLS:", cls_embedding.shape)
print("Patches:", patch_embeddings.shape)
print("Pooled:", pooled_embedding.shape)

Use cls_embedding for global image features, patch_embeddings for spatial features, or pooled_embedding for a single pooled representation. Register tokens are excluded from patch features.

Intended use: Research on hand radiographs. Not validated for direct clinical diagnosis or treatment decisions. The model weights are distributed subject to the DINOv3 License; see the accompanying license file.

Citation

If you use HandXFM in your research, please cite:

@inproceedings{long2026handxfm,
  title     = {{HandXFM}: Semantic-Structural Distillation for Hand Radiograph Foundation Models},
  author    = {Long, Yuxi and Feng, Ganlin and Chen, Lianghong and O'Neil, Liam J. and Hitchon, Carol A. and Hu, Pingzhao},
  booktitle = {Advances in Neural Information Processing Systems},
  year      = {2026},
  url       = {https://neurips.cc/virtual/2026/loc/sydney/poster/150797}
}