README.md

---
license: cc-by-nc-sa-4.0
---

# Leveraging generic foundation models for multimodal surgical data analysis

Model repository for the paper **Leveraging generic foundation models for multimodal surgical data analysis** by Simon
Pezold, Jérôme A. Kurylec, Jan S. Liechti, Beat P. Müller, and Joël L. Lavanchy. For more details, see the [paper](https://arxiv.org/abs/2509.06831) and its
[code repository](https://github.com/DigitalSurgeryLab-Basel/ML-CDS-2025).

## Model weights and usage

* **Finetuned V-JEPA** (ours): We finetuned *V-JEPA ViT-L* on unlabeled endoscopic surgical videos, as described in the paper.
  The resulting model weights can be downloaded [here](./jepa-12h.pth.tar).

* **Pretrained V-JEPA** (third-party): The model weights that formed the basis for finetuning can be downloaded via the links provided in Meta Research's
  [V-JEPA v1 code repository](https://github.com/facebookresearch/jepa) – use the link under *model zoo / pretrained models /
  ViT-L / checkpoint* or [this direct link](https://dl.fbaipublicfiles.com/jepa/vitl16/vitl16.pth.tar). Please note that we are not
  affiliated with Meta Research in any way.

For details on how to use the provided model weights with the approach proposed in our paper, follow the instructions in our
[code repository](https://github.com/DigitalSurgeryLab-Basel/ML-CDS-2025).

## Data, distribution, and licensing

As we utilized public datasets for finetuning, we distribute our *model weights* under the same license:
[CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/).
For more details, please refer to the license file linked above.
For the license of the *code*, see the
[code repository](https://github.com/DigitalSurgeryLab-Basel/ML-CDS-2025).

Videos from the following datasets were used for finetuning:

* [**Heidelberg colorectal (HeiCo)**](https://doi.org/10.7303/syn21903917), as described in
  * Maier-Hein, L., Wagner, M., Ross, T., Reinke, A., Bodenstedt, S., Full, P. M., ... & Müller-Stich, B. P. (2021).
    Heidelberg colorectal data set for surgical data science in the sensor operating room.
    [Scientific data, 8(1), 1-11](https://doi.org/10.1038/s41597-021-00882-2)
  * Roß, T., Reinke, A., Full, P. M., Wagner, M., Kenngott, H., Apitz, M., ... & Maier-Hein, L. (2021). Comparative validation
    of multi-instance instrument segmentation in endoscopy: results of the ROBUST-MIS 2019 challenge.
    [Medical image analysis, 70, 101920](https://doi.org/10.1016/j.media.2020.101920)
* [**Cholec80**](http://camma.u-strasbg.fr/datasets/), as described in

    A.P. Twinanda, S. Shehata, D. Mutter, J. Marescaux, M. de Mathelin, N. Padoy, EndoNet: A Deep Architecture for Recognition
    Tasks on Laparoscopic Videos, IEEE Transactions on Medical Imaging (TMI), [arXiv preprint](https://doi.org/10.48550/arXiv.1602.03012), 2017
* [**MultiBypass140**](https://github.com/CAMMA-public/MultiBypass140), as described in

    Lavanchy, J.L., Ramesh, S., Dall’Alba, D. et al. Challenges in multi-centric generalization: phase and step recognition
    in Roux-en-Y gastric bypass surgery. [Int J CARS 19, 2249–2257 (2024)](https://doi.org/10.1007/s11548-024-03166-3)

We are grateful to all authors for creating and publicly releasing these datasets!

## Acknowledgments

We would like to thank Julia Ruppel for her assistance in obtaining the necessary patient data.
This project was partially funded by the Vontobel Foundation (0867/2024).
It was supported by a grant from the Swiss National Supercomputing Centre (CSCS) under project ID lp52 on Alps.

## Citing

If you find our work useful, please consider citing:
```bibtex
@article{pezold2025leveraging,
    title = {Leveraging Generic Foundation Models for Multimodal Surgical Data Analysis}, 
    author = {Pezold, Simon and Kurylec, Jérôme A. and Liechti, Jan S. and Müller, Beat P. and Lavanchy, Joël L.},
    journal = {arXiv preprint arXiv:2509.06831},
    year = {2025}
}
```