--- license: cc-by-nc-4.0 arxiv: 2603.29966 tags: - surgical-video - video-understanding - self-supervised-learning - videomae - jepa - vision-transformer library_name: pytorch pipeline_tag: video-classification --- # SurgRec — Surgical Video Foundation Models [![arXiv](https://img.shields.io/badge/arXiv-2603.29966-b31b1b.svg)](https://arxiv.org/abs/2603.29966) [![GitHub](https://img.shields.io/badge/GitHub-SurgRec-181717.svg?logo=github)](https://github.com/Siiichenggg/SurgRec) [![License: CC BY-NC 4.0](https://img.shields.io/badge/License-CC%20BY--NC%204.0-lightgrey.svg)](https://creativecommons.org/licenses/by-nc/4.0/) Pretrained backbones from [**Scaling Video Pretraining for Surgical Foundation Models**](https://arxiv.org/abs/2603.29966). ![SurgRec-MAE versus the best third-party baseline on each of the 16 datasets](comparison.png)

Each dataset is compared against whichever third-party baseline is strongest on it — DINOv3, DINOv3-SurgeNetXL, VideoMAE, or V-JEPA (colour-coded; VideoMAE is never the strongest). Numbers are Table 2 of the paper. cataract-1k is the one dataset where a baseline leads, and cat-21 is a tie where every method scores 15.69.

```bash hf download SichengLu/SurgRec --local-dir checkpoints ``` > These are **headless backbones** — no classification head. Fine-tune them on a > downstream surgical video dataset with the > [code repository](https://github.com/Siiichenggg/SurgRec). **Sicheng Lu, Zikai Xiao, Jianhui Wei, Danyu Sun, Qi Lu, Keli Hu, Yang Feng, Jian Wu, Zongxin Yang, Zuozhu Liu** ## Highlights - Pretrained on a multi-source surgical corpus of **10,535 videos / 214.5M frames** spanning endoscopy, laparoscopy, cataract, and robotic surgery. - Evaluated on a standardized **16-dataset** downstream benchmark across four clinical domains with consistent splits. - **SurgRec-MAE is strictly the best of the seven compared methods on 11 of 16 datasets**, and ties for best on 3 more. On the mean it leads the strongest third-party baseline (JEPA) by 11.3 points. ## Results Top-1 accuracy (%) on the 16-dataset downstream benchmark, from Table 2 of the [paper](https://arxiv.org/abs/2603.29966). Baselines are DINOv3, DINOv3-Surg (SurgeNetXL), V-MAE (VideoMAE), SR-MAE without balanced sampling, and JEPA; **bold** marks the best method on each dataset. | Dataset | DINOv3 | DINOv3-Surg | V-MAE | SR-MAE (w/o bal.) | JEPA | **SR-MAE** | **SR-JEPA** | | --- | --- | --- | --- | --- | --- | --- | --- | | AIxSuture | 32.61 | 39.13 | 39.13 | 39.13 | 39.13 | **43.48** | 39.13 | | AutoLaparo | 17.74 | 19.35 | 19.35 | 17.74 | 20.97 | **22.58** | 17.74 | | cat-21 | 15.69 | 15.69 | 15.69 | 15.69 | 15.69 | 15.69 | 15.69 | | cataract-101 | 17.97 | 19.53 | 19.53 | 21.09 | 24.22 | **35.16** | 23.44 | | cataract-1k | 9.90 | 14.85 | 11.88 | 16.83 | **19.80** | 18.81 | 17.82 | | Cholec80 | 22.34 | 16.12 | 14.65 | 24.18 | 26.01 | 29.30 | **31.14** | | Colonoscopic | 53.33 | 53.33 | 53.33 | 53.33 | 53.33 | **73.33** | 53.33 | | Hyper-Kvasir | 43.03 | 41.81 | 25.11 | 36.91 | 41.65 | **55.28** | 42.27 | | JIGSAWS | 18.12 | 17.11 | 15.67 | 21.23 | 29.72 | **47.49** | 32.24 | | Kvasir-Capsule | 46.34 | 47.56 | 19.51 | 46.34 | 48.78 | **73.17** | 60.98 | | LapGyn4 | 62.68 | 65.63 | 62.68 | 67.07 | 60.66 | **68.13** | 64.13 | | LDPolypVideo | 90.35 | 90.35 | 90.35 | 90.35 | 90.35 | **90.76** | 90.42 | | M2CAI16 | 23.01 | 25.66 | 12.39 | 30.09 | 26.55 | **38.05** | **38.05** | | MultiBypass140 | 38.62 | 42.63 | 20.54 | 51.56 | 12.88 | **56.92** | 56.47 | | SurgicalActions160 | 6.25 | 6.25 | 6.25 | **18.75** | 12.50 | **18.75** | 6.25 | | SAR-RARP50 | 23.01 | 26.88 | 25.97 | 24.37 | 22.10 | **37.59** | 36.67 | | **Mean** | 32.56 | 33.87 | 28.25 | 35.92 | 34.02 | **45.28** | 39.11 | SR-MAE (w/o bal.) is an ablation of our own model, not a third-party method. Every method scores exactly 15.69 on cat-21, consistent with all of them collapsing to a single class, so that row carries no signal. JEPA leads on cataract-1k and SurgRec-JEPA on Cholec80. ## Models | File | Model | Architecture | Params | Pretraining objective | Load with | | --- | --- | --- | --- | --- | --- | | `surgrec_mae.pth` | SurgRec-MAE | ViT-B/16 | 86.5 M encoder | Masked video reconstruction (VideoMAE-style) | `model_key=model` | | `surgrec_jepa.pt` | SurgRec-JEPA | ViT-L/16 | 303.9 M encoder | Latent predictive (V-JEPA-style) | `model_key=encoder` |
Checkpoint internals, training configuration, and checksums **`surgrec_mae.pth`** — configuration recorded inside the checkpoint: - `model = pretrain_videomae_base_patch16_224` - `mask_type = tube`, `mask_ratio = 0.9`, `decoder_depth = 4` - `input_size = 224`, `num_frames = 16`, `sampling_rate = 4` - `lr = 3e-4`, `batch_size = 64`, taken at `epoch = 149` The file stores 94.2 M parameters in total including the MAE decoder; the encoder used for fine-tuning is 86.5 M. The archive holds `model`, `optimizer`, `scaler`, `epoch`, and `args`. The `model` entry contains both the encoder and the MAE decoder; fine-tuning loads the encoder and discards `decoder.*`. **`surgrec_jepa.pt`** — holds `encoder`, `predictor`, and `target_encoder`, each ViT-L/16; taken at `epoch = 131`, `loss = 0.4225`, trained with `batch_size = 24` across `world_size = 8`. Load the `encoder` entry; its weights sit under a `module.backbone.` prefix, which the loader in the code repository strips. SHA256, for verifying downloads: ```text ce2c0298292d636f95c77a12c26fcdeb93de8b4a78e515a68d247de5e61759c5 surgrec_mae.pth 84040aa2e2fc015d03165903e6eec06282507f1d275e76efa97292457cd1bd3e surgrec_jepa.pt ```
## Usage ```bash git clone https://github.com/Siiichenggg/SurgRec.git cd SurgRec pip install -r requirements.txt hf download SichengLu/SurgRec --local-dir checkpoints ``` Fine-tune on a downstream dataset: ```bash DATASET_ROOT=data/splits_local CKPT_ROOT=checkpoints GPUS=8 \ bash scripts/train/finetune_surgrec_video.sh cholec80 --backbone surgrec_mae ``` Or reproduce a paper run with its original hyperparameters: ```bash CKPT_ROOT=checkpoints DATA_ROOT=/path/to/split GPUS=8 \ bash scripts/paper/finetune/jepa_v3_finetune_8gpu.sh ``` ## License and intended use Released for **non-commercial research**. These weights are a research artifact, not a medical device, and must not be used for clinical diagnosis or treatment decisions. [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/) — attribution required, no commercial use. The code derives in part from VideoMAE, which carries the same obligation; see `LICENSE` and `NOTICE` in the code repository. ## Citation ```bibtex @article{lu2026surgrec, title = {Scaling Video Pretraining for Surgical Foundation Models}, author = {Lu, Sicheng and Xiao, Zikai and Wei, Jianhui and Sun, Danyu and Lu, Qi and Hu, Keli and Feng, Yang and Wu, Jian and Yang, Zongxin and Liu, Zuozhu}, journal = {arXiv preprint arXiv:2603.29966}, year = {2026}, url = {https://arxiv.org/abs/2603.29966} } ```