Complete model card
Browse files
README.md
CHANGED
|
@@ -1,9 +1,49 @@
|
|
| 1 |
---
|
| 2 |
license: cc-by-nc-sa-4.0
|
| 3 |
-
arxiv: 2509.06831
|
| 4 |
---
|
| 5 |
|
| 6 |
# Leveraging generic foundation models for multimodal surgical data analysis
|
| 7 |
|
| 8 |
Model repository for the paper **Leveraging generic foundation models for multimodal surgical data analysis** by Simon
|
| 9 |
-
Pezold, Jérôme A. Kurylec, Jan S. Liechti, Beat P. Müller, and Joël L. Lavanchy.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
license: cc-by-nc-sa-4.0
|
|
|
|
| 3 |
---
|
| 4 |
|
| 5 |
# Leveraging generic foundation models for multimodal surgical data analysis
|
| 6 |
|
| 7 |
Model repository for the paper **Leveraging generic foundation models for multimodal surgical data analysis** by Simon
|
| 8 |
+
Pezold, Jérôme A. Kurylec, Jan S. Liechti, Beat P. Müller, and Joël L. Lavanchy. For more details, see the [paper](https://arxiv.org/abs/2509.06831) and its
|
| 9 |
+
[code repository](https://github.com/DigitalSurgeryLab-Basel/ML-CDS-2025).
|
| 10 |
+
|
| 11 |
+
## Model weights and usage
|
| 12 |
+
|
| 13 |
+
* **Finetuned V-JEPA** (ours): We finetuned *V-JEPA ViT-L* on unlabeled endoscopic surgical videos, as described in the paper.
|
| 14 |
+
The resulting model weights can be downloaded [here](./jepa-12h.pth.tar).
|
| 15 |
+
|
| 16 |
+
* **Pretrained V-JEPA** (third-party): The model weights that formed the basis for finetuning can be downloaded via the links provided in Meta Research's
|
| 17 |
+
[V-JEPA v1 code repository](https://github.com/facebookresearch/jepa): use the link under *model zoo / pretrained models /
|
| 18 |
+
ViT-L / checkpoint* or [this direct link](https://dl.fbaipublicfiles.com/jepa/vitl16/vitl16.pth.tar). Please note that we are not
|
| 19 |
+
affiliated with Meta Research in any way.
|
| 20 |
+
|
| 21 |
+
For details on how to use the provided model weights with the approach proposed in the paper, follow the instructions in its
|
| 22 |
+
[code repository](https://github.com/DigitalSurgeryLab-Basel/ML-CDS-2025).
|
| 23 |
+
|
| 24 |
+
## Data, distribution, and licensing
|
| 25 |
+
|
| 26 |
+
As we utilized public datasets for finetuning, we distribute our model weights under the same license:
|
| 27 |
+
[CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/).
|
| 28 |
+
For more details, please refer to the license file linked above.
|
| 29 |
+
|
| 30 |
+
Videos from the following datasets were used for finetuning:
|
| 31 |
+
|
| 32 |
+
* [Heidelberg colorectal (HeiCo)](https://www.synapse.org/Synapse:syn21903917/files/)
|
| 33 |
+
* [Cholec80](http://camma.u-strasbg.fr/datasets/)
|
| 34 |
+
* [MultiBypass140](https://github.com/CAMMA-public/MultiBypass140)
|
| 35 |
+
|
| 36 |
+
We are grateful to all authors for creating and publicly releasing these datasets!
|
| 37 |
+
|
| 38 |
+
## Citing
|
| 39 |
+
|
| 40 |
+
If you find our work useful, please consider citing:
|
| 41 |
+
```bibtex
|
| 42 |
+
@article{pezold2025leveraging,
|
| 43 |
+
title = {Leveraging Generic Foundation Models for Multimodal Surgical Data Analysis},
|
| 44 |
+
author = {Pezold, Simon and Kurylec, Jérôme A. and Liechti, Jan S. and Müller, Beat P. and Lavanchy, Joël L.},
|
| 45 |
+
journal = {arXiv preprint},
|
| 46 |
+
year = {2025},
|
| 47 |
+
doi = {10.48550/arXiv.2509.06831},
|
| 48 |
+
}
|
| 49 |
+
```
|