Publications

Scaling up self-supervised learning for improved surgical foundation models

Tim J.M. Jaspers^{1* :email:}, Ronald L.P.D. de Jong^2*, Yiping Li², Carolus H.J. Kusters¹, Franciscus H.A. Bakker⁵, Romy C. van Jaarsveld³, Gino M. Kuipers³, Richard³, Jelle P. Ruurda³, Willem M. Brinkman⁴, Josien P.W. Pluim², Peter H.N. de With¹, Marcel Breeuwer², Yasmina Al Khalil², Fons van der Sommen¹

¹ Department of Electrical Engineering, Video Coding & Architectures, Eindhoven University of Technology
² Department of Biomedical Engineering, Medical Image Analysis, Eindhoven University of Technology, Eindhoven, The Netherlands
³ Department of Surgery, University Medical Center Utrecht, Utrecht, The Netherlands
⁴ Department of Oncological Urology, University Medical Center Utrecht, Utrecht, The Netherlands
⁵ Department of Surgery, Catharina Hospital, Eindhoven, The Netherlands

^* Both authors attributed equally
(^:email:) corresponding author

arxiv
(Article)

Exploring the Effect of Dataset Diversity in Self-Supervised Learning for Surgical Computer Vision

Tim J.M. Jaspers^{1 :email:}, Ronald L.P.D. de Jong², Yasmina Al Khalil², Tijn Zeelenberg ¹, Carolus H.J. Kusters¹, Franciscus H.A. Bakker⁵, Yiping Li², Romy C. van Jaarsveld³, Jelle P. Ruurda³, Willem M. Brinkman⁴, Peter H.N. de With¹, Fons van der Sommen¹,

Second Workshop on Data Engineering in Medical Imaging (DEMI) - Satellite Event MICCAI 2024
(Proceeding)

¹ Department of Electrical Engineering, Video Coding & Architectures, Eindhoven University of Technology
² Department of Biomedical Engineering, Medical Image Analysis, Eindhoven University of Technology, Eindhoven, The Netherlands
³ Department of Surgery, University Medical Center Utrecht, Utrecht, The Netherlands
⁴ Department of Oncological Urology, University Medical Center Utrecht, Utrecht, The Netherlands
⁵ Department of Surgery, Catharina Hospital, Eindhoven, The Netherlands

(^:email:) corresponding author

Abstract

Foundation models have revolutionized computer vision by achieving state-of-the-art performance across diverse tasks through large-scale pretraining on extensive datasets. However, their application in surgical computer vision has been limited. This study addresses this gap by introducing SurgeNetXL, a novel surgical foundation model that sets a new benchmark in surgical computer vision. Trained on the largest reported surgical dataset to date, comprising over 4.7 million video frames, SurgeNetXL achieves consistent top-tier performance across six datasets spanning four surgical procedures and three tasks, including semantic segmentation, phase recognition, and critical view of safety (CVS) classification. Compared to the best-performing surgical foundation models, SurgeNetXL shows mean improvements of 2.4, 8.95, and 12.6% for semantic segmentation, phase recognition, and CVS classification, respectively. Additionally, SurgeNetXL outperforms the best-performing ImageNet-based variants by 14.4, 4.0, and 1.6% in the respective tasks. In addition to advancing model performance, this work provides key insights into scaling pretraining datasets, extending training durations, and optimizing model architectures specifically for surgical computer vision. These findings pave the way for improved generalizability and robustness in data-scarce scenarios, offering a comprehensive framework for future research in this domain.

Results

The following figures are from our publications, showcasing the performance of our introduced foundation model across diverse surgical tasks and procedures. These results demonstrate the model’s state-of-the-art performance on a variety of downstream tasks, reflecting its versatility and robustness in handling datasets from multiple surgical procedures.

Figure 1 and Figure 2 illustrate comparative rankings of our model against existing foundation models, highlighting its superior generalization capabilities across datasets. Figure 3 provides a t-SNE visualization, showcasing the clear cluster separation per specific dataset achieved by the model’s feature embeddings, further emphasizing its effectiveness in capturing meaningful representations.

Models

The models used in this study are based on the MetaFormer architecture. The models are trained using a self-supervised learning approach on the SurgeNetXL dataset and its variations, introduced this in the following paper. All model weights can be downloaded from the table below.

Loading Models

The weights from the teacher network can be used to initialize either your classification or segmentation model using the following code snippet:

import torch
from metaformer import caformer_s18, MetaFormerFPN
from convnextv2 import convnextv2_tiny, ConvNextFPN
from pvtv2 import pvt_v2_b2, PVTV2FPN

urls = {
    "ImageNet1k": "https://huggingface.co/sail/dl/resolve/main/caformer/caformer_s18.pth",
    "SurgeNetXL": "https://huggingface.co/TimJaspersTue/SurgeNetModels/resolve/main/SurgeNetXL_checkpoint_epoch0050_teacher.pth?download=true",
    "SurgeNet": "https://huggingface.co/TimJaspersTue/SurgeNetModels/resolve/main/SurgeNet_checkpoint_epoch0050_teacher.pth?download=true",
    "SurgeNet-Small": "https://huggingface.co/TimJaspersTue/SurgeNetModels/resolve/main/SurgeNetSmall_checkpoint_epoch0050_teacher.pth?download=true",
    "SurgeNet-CHOLEC": "https://huggingface.co/TimJaspersTue/SurgeNetModels/resolve/main/CHOLEC_checkpoint_epoch0050_teacher.pth?download=true",
    "SurgeNet-RAMIE": "https://huggingface.co/TimJaspersTue/SurgeNetModels/resolve/main/RAMIE_checkpoint_epoch0050_teacher.pth?download=true",
    "SurgeNet-RARP": "https://huggingface.co/TimJaspersTue/SurgeNetModels/resolve/main/RARP_checkpoint_epoch0050_teacher.pth?download=true",
    "SurgeNet-Public": "https://huggingface.co/TimJaspersTue/SurgeNetModels/resolve/main/Public_checkpoint0050.pth?download=true",
    "SurgeNet-ConvNextv2": "https://huggingface.co/TimJaspersTue/SurgeNetModels/resolve/main/SurgeNet_ConvNextv2_checkpoint_epoch0050_teacher.pth?download=true",
    "SurgeNet-PVTv2": "https://huggingface.co/TimJaspersTue/SurgeNetModels/resolve/main/SurgeNet_PVTv2_checkpoint_epoch0050_teacher.pth?download=true",
}

# Metaformer model
classification_model = caformer_s18(num_classes=12, pretrained='SurgeNet', pretrained_weights=urls['SurgeNetXL'])
segmentation_model = MetaFormerFPN(num_classes=12, pretrained='SurgeNet', pretrained_weights=urls['SurgeNetXL'])

# ConvNextv2 model
classification_model = convnextv2_tiny(num_classes=12, pretrained_weights=urls['SurgeNet-ConvNextv2'])
segmentation_model = ConvNextFPN(num_classes=12, pretrained_weights=urls['SurgeNet-ConvNextv2'])

# PVTv2 model
classification_model = pvt_v2_b2(num_classes=12, pretrained_weights=urls['SurgeNet-PVTv2'])
segmentation_model = PVTV2FPN(num_classes=12, pretrained_weights=urls['SurgeNet-PVTv2'])

Note: If your want a different version of SurgeNet weights (e.g. SurgeNet-Small), you can replace the `pretrained_weights` argument with the desired url (leave the `pretrained` argument as it is).

Surgical Youtube Dataset

A key contribution of our research is the Surgical YouTube dataset, which enhanced our foundation model's performance. This curated dataset contains 2,074,234 frames sampled from 23 distinct surgical procedures and is publicly available at huggingface datasets. This datasets is a large part of our SurgeNetXL dataset, which also includes other opensource datasets.

Acknowledgements

Our implementation of the feature pyramid network is based on the pytorch segmentation models library. Pretraining on SurgeNet was performed using the code provided with the DINO publication. We have used the code of Schmidgall et al. (2024) to obtain the youtube videos, this code can be found here.

Citation

If you find our work useful in your research please consider citing our paper:

@msc{Jaspers2025,
   title={Scaling up self-supervised learning for improved surgical foundation models},
   year={2025}
    }

@inbook{Jaspers2024,
   title={Exploring the Effect of Dataset Diversity in Self-supervised Learning for Surgical Computer Vision},
   ISBN={9783031737480},
   ISSN={1611-3349},
   url={http://dx.doi.org/10.1007/978-3-031-73748-0_5},
   DOI={10.1007/978-3-031-73748-0_5},
   booktitle={Data Engineering in Medical Imaging},
   publisher={Springer Nature Switzerland},
   author={Jaspers, Tim J. M. and de Jong, Ronald L. P. D. and Al Khalil, Yasmina and Zeelenberg, Tijn and Kusters, Carolus H. J. and Li, Yiping and van Jaarsveld, Romy C. and Bakker, Franciscus H. A. and Ruurda, Jelle P. and Brinkman, Willem M. and De With, Peter H. N. and van der Sommen, Fons},
   year={2024},
   month=oct, pages={43–53} }