README.md

---
license: cc-by-nc-4.0
arxiv: 2603.29966
tags:
  - surgical-video
  - video-understanding
  - self-supervised-learning
  - videomae
  - jepa
  - vision-transformer
library_name: pytorch
pipeline_tag: video-classification
---

# SurgRec — Surgical Video Foundation Models

[![arXiv](https://img.shields.io/badge/arXiv-2603.29966-b31b1b.svg)](https://arxiv.org/abs/2603.29966)
[![GitHub](https://img.shields.io/badge/GitHub-SurgRec-181717.svg?logo=github)](https://github.com/Siiichenggg/SurgRec)
[![License: CC BY-NC 4.0](https://img.shields.io/badge/License-CC%20BY--NC%204.0-lightgrey.svg)](https://creativecommons.org/licenses/by-nc/4.0/)

Pretrained backbones from
[**Scaling Video Pretraining for Surgical Foundation Models**](https://arxiv.org/abs/2603.29966).

![SurgRec-MAE versus the best third-party baseline on each of the 16 datasets](comparison.png)

<p align="center"><em>Each dataset is compared against whichever third-party baseline is strongest on it — DINOv3, DINOv3-SurgeNetXL, VideoMAE, or V-JEPA (colour-coded; VideoMAE is never the strongest). Numbers are Table 2 of the paper. cataract-1k is the one dataset where a baseline leads, and cat-21 is a tie where every method scores 15.69.</em></p>

```bash
hf download SichengLu/SurgRec --local-dir checkpoints
```

> These are **headless backbones** — no classification head. Fine-tune them on a
> downstream surgical video dataset with the
> [code repository](https://github.com/Siiichenggg/SurgRec).

**Sicheng Lu, Zikai Xiao, Jianhui Wei, Danyu Sun, Qi Lu, Keli Hu, Yang Feng,
Jian Wu, Zongxin Yang, Zuozhu Liu**

## Highlights

- Pretrained on a multi-source surgical corpus of **10,535 videos / 214.5M frames**
  spanning endoscopy, laparoscopy, cataract, and robotic surgery.
- Evaluated on a standardized **16-dataset** downstream benchmark across four
  clinical domains with consistent splits.
- **SurgRec-MAE is strictly the best of the seven compared methods on 11 of 16
  datasets**, and ties for best on 3 more. On the mean it leads the strongest
  third-party baseline (JEPA) by 11.3 points.

## Results

Top-1 accuracy (%) on the 16-dataset downstream benchmark, from Table 2 of the
[paper](https://arxiv.org/abs/2603.29966). Baselines are DINOv3, DINOv3-Surg
(SurgeNetXL), V-MAE (VideoMAE), SR-MAE without balanced sampling, and JEPA;
**bold** marks the best method on each dataset.

| Dataset | DINOv3 | DINOv3-Surg | V-MAE | SR-MAE (w/o bal.) | JEPA | **SR-MAE** | **SR-JEPA** |
| --- | --- | --- | --- | --- | --- | --- | --- |
| AIxSuture | 32.61 | 39.13 | 39.13 | 39.13 | 39.13 | **43.48** | 39.13 |
| AutoLaparo | 17.74 | 19.35 | 19.35 | 17.74 | 20.97 | **22.58** | 17.74 |
| cat-21 | 15.69 | 15.69 | 15.69 | 15.69 | 15.69 | 15.69 | 15.69 |
| cataract-101 | 17.97 | 19.53 | 19.53 | 21.09 | 24.22 | **35.16** | 23.44 |
| cataract-1k | 9.90 | 14.85 | 11.88 | 16.83 | **19.80** | 18.81 | 17.82 |
| Cholec80 | 22.34 | 16.12 | 14.65 | 24.18 | 26.01 | 29.30 | **31.14** |
| Colonoscopic | 53.33 | 53.33 | 53.33 | 53.33 | 53.33 | **73.33** | 53.33 |
| Hyper-Kvasir | 43.03 | 41.81 | 25.11 | 36.91 | 41.65 | **55.28** | 42.27 |
| JIGSAWS | 18.12 | 17.11 | 15.67 | 21.23 | 29.72 | **47.49** | 32.24 |
| Kvasir-Capsule | 46.34 | 47.56 | 19.51 | 46.34 | 48.78 | **73.17** | 60.98 |
| LapGyn4 | 62.68 | 65.63 | 62.68 | 67.07 | 60.66 | **68.13** | 64.13 |
| LDPolypVideo | 90.35 | 90.35 | 90.35 | 90.35 | 90.35 | **90.76** | 90.42 |
| M2CAI16 | 23.01 | 25.66 | 12.39 | 30.09 | 26.55 | **38.05** | **38.05** |
| MultiBypass140 | 38.62 | 42.63 | 20.54 | 51.56 | 12.88 | **56.92** | 56.47 |
| SurgicalActions160 | 6.25 | 6.25 | 6.25 | **18.75** | 12.50 | **18.75** | 6.25 |
| SAR-RARP50 | 23.01 | 26.88 | 25.97 | 24.37 | 22.10 | **37.59** | 36.67 |
| **Mean** | 32.56 | 33.87 | 28.25 | 35.92 | 34.02 | **45.28** | 39.11 |

SR-MAE (w/o bal.) is an ablation of our own model, not a third-party method.
Every method scores exactly 15.69 on cat-21, consistent with all of them
collapsing to a single class, so that row carries no signal. JEPA leads on
cataract-1k and SurgRec-JEPA on Cholec80.

## Models

| File | Model | Architecture | Params | Pretraining objective | Load with |
| --- | --- | --- | --- | --- | --- |
| `surgrec_mae.pth` | SurgRec-MAE | ViT-B/16 | 86.5 M encoder | Masked video reconstruction (VideoMAE-style) | `model_key=model` |
| `surgrec_jepa.pt` | SurgRec-JEPA | ViT-L/16 | 303.9 M encoder | Latent predictive (V-JEPA-style) | `model_key=encoder` |

<details>
<summary><b>Checkpoint internals, training configuration, and checksums</b></summary>

**`surgrec_mae.pth`** — configuration recorded inside the checkpoint:

- `model = pretrain_videomae_base_patch16_224`
- `mask_type = tube`, `mask_ratio = 0.9`, `decoder_depth = 4`
- `input_size = 224`, `num_frames = 16`, `sampling_rate = 4`
- `lr = 3e-4`, `batch_size = 64`, taken at `epoch = 149`

The file stores 94.2 M parameters in total including the MAE decoder; the
encoder used for fine-tuning is 86.5 M. The archive holds `model`, `optimizer`,
`scaler`, `epoch`, and `args`. The
`model` entry contains both the encoder and the MAE decoder; fine-tuning loads
the encoder and discards `decoder.*`.

**`surgrec_jepa.pt`** — holds `encoder`, `predictor`, and `target_encoder`, each
ViT-L/16; taken at `epoch = 131`, `loss = 0.4225`, trained with
`batch_size = 24` across `world_size = 8`. Load the `encoder` entry; its weights
sit under a `module.backbone.` prefix, which the loader in the code repository
strips.

SHA256, for verifying downloads:

```text
ce2c0298292d636f95c77a12c26fcdeb93de8b4a78e515a68d247de5e61759c5  surgrec_mae.pth
84040aa2e2fc015d03165903e6eec06282507f1d275e76efa97292457cd1bd3e  surgrec_jepa.pt
```

</details>

## Usage

```bash
git clone https://github.com/Siiichenggg/SurgRec.git
cd SurgRec
pip install -r requirements.txt
hf download SichengLu/SurgRec --local-dir checkpoints
```

Fine-tune on a downstream dataset:

```bash
DATASET_ROOT=data/splits_local CKPT_ROOT=checkpoints GPUS=8 \
  bash scripts/train/finetune_surgrec_video.sh cholec80 --backbone surgrec_mae
```

Or reproduce a paper run with its original hyperparameters:

```bash
CKPT_ROOT=checkpoints DATA_ROOT=/path/to/split GPUS=8 \
  bash scripts/paper/finetune/jepa_v3_finetune_8gpu.sh
```

## License and intended use

Released for **non-commercial research**. These weights are a research artifact,
not a medical device, and must not be used for clinical diagnosis or treatment
decisions.

[CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/) — attribution
required, no commercial use. The code derives in part from VideoMAE, which
carries the same obligation; see `LICENSE` and `NOTICE` in the code repository.

## Citation

```bibtex
@article{lu2026surgrec,
  title   = {Scaling Video Pretraining for Surgical Foundation Models},
  author  = {Lu, Sicheng and Xiao, Zikai and Wei, Jianhui and Sun, Danyu and
             Lu, Qi and Hu, Keli and Feng, Yang and Wu, Jian and Yang, Zongxin
             and Liu, Zuozhu},
  journal = {arXiv preprint arXiv:2603.29966},
  year    = {2026},
  url     = {https://arxiv.org/abs/2603.29966}
}
```