GOVINDFROM/surgical-world-model

Surgical World Model (EG-SWM)

Embodiment-grounded world models of robot-assisted surgery: an action-conditioned video world model whose action space mirrors the robot — an explicit endoscope ego-motion action plus one latent action per instrument arm, learned from unlabeled in-vivo da Vinci video and bound to the arms through the robot's own system logs.

Code, data pipeline, documentation and diagrams: https://github.com/GIND123/Surgical-World-Model

Checkpoints

One folder per finished run: model.safetensors (EMA weights), config.json (model and training config, step) and the evaluation summaries (replay.json: fidelity; control.json: endoscope control, locality, embodiment consistency; action_use.json: action-use diagnostics). Metrics on the held-out SurgVU test cases, 2.8 s horizon, action replay (protocol).

Folder	Model	Size	Steps	PSNR ↑	LPIPS ↓	FVD ↓	Endoscope gain ↑
S_none/	None (unconditional)	S	60000	18.96	0.249	422	–
S_state_nocam/	Robot state	S	60000	19.29	0.237	358	–
S_state/	Robot state + camera	S	60000	20.44	0.206	266	0.79
S_global_nocam/	Monolithic latent (Genie)	S	60000	22.49	0.154	172	–
S_global/	Monolithic latent + camera	S	60000	22.47	0.154	180	0.16
S_global_vq/	Monolithic VQ + camera (SurgWM-style)	S	60000	21.30	0.183	206	0.81
S_slots/	Slots + camera (FLAM-style)	S	60000	22.77	0.149	169	0.05
S_arms_nocam/	EG-SWM, no camera action	S	60000	21.52	0.176	215	–
S_arms/	EG-SWM	S	60000	22.00	0.164	186	0.70
S_arms_scene/	EG-SWM + scene action	S	60000	22.38	0.156	174	0.22
S_arms_nofwd/	EG-SWM, diffusion loss only	S	60000	21.65	0.172	204	0.69
S_arms_poolglobal/	EG-SWM, arm actions not grounded	S	60000	22.07	0.163	188	0.65
S_arms_shuffled/	EG-SWM, wrong arm binding	S	60000	21.98	0.165	186	0.70
S_arms_ui/	EG-SWM + console readout	S	60000	21.98	0.164	184	0.70
S_arms_hold/	EG-SWM, boxes held across detector gaps	S	60000	22.06	0.163	185	0.65
B_arms/	EG-SWM, size B	B	60000	22.43	0.152	118	0.78
S_arms_seed1/	S_arms_seed1	S	60000	21.99	0.164	188	0.66
S_global_seed1/	S_global_seed1	S	60000	22.46	0.155	177	0.15
S_slots_seed1/	S_slots_seed1	S	60000	22.76	0.149	170	0.03

Checkpoints for the kinematic benchmarks: SurgVU models adapted to a dataset without its kinematics (E2, frozen for the linear probes), and models conditioned on the robot kinematics (kin, E3). Metrics on the test episodes of their own dataset.

Folder	Data	Actions	Initialised from	Steps	PSNR ↑	LPIPS ↓	FVD ↓
SB_kin_arms_f0.1/	suturebot, 10% of the training episodes	kin	S_arms	8000	23.85	0.135	93
SB_kin_arms_f1.0/	suturebot	kin	S_arms	8000	23.77	0.140	117
SB_kin_global_f0.1/	suturebot, 10% of the training episodes	kin	S_global	8000	23.78	0.138	101
SB_kin_global_f1.0/	suturebot	kin	S_global	8000	23.69	0.143	117
SB_kin_scratch_f0.1/	suturebot, 10% of the training episodes	kin	scratch	8000	22.73	0.182	595
SB_kin_scratch_f1.0/	suturebot	kin	scratch	8000	22.43	0.192	647
SRTH_kin_arms_f0.1/	srth, 10% of the training episodes	kin	S_arms	8000	21.03	0.148	227
SRTH_kin_arms_f1.0/	srth	kin	S_arms	8000	20.89	0.152	242
SRTH_kin_global_f0.1/	srth, 10% of the training episodes	kin	S_global	8000	20.96	0.152	240
SRTH_kin_global_f1.0/	srth	kin	S_global	8000	20.95	0.152	243
SRTH_kin_scratch_f0.1/	srth, 10% of the training episodes	kin	scratch	8000	19.95	0.197	621
SRTH_kin_scratch_f1.0/	srth	kin	scratch	8000	19.88	0.199	642

Loading (with the swm package from the code repository):

import json
from safetensors.torch import load_model
from swm.models.world_model import WMConfig, WorldModel

meta = json.load(open("S_arms/config.json"))
model = WorldModel(WMConfig.from_dict(meta["model"])).eval()
load_model(model, "S_arms/model.safetensors")

The SurgWM re-implementation (surgwm/) has one file per stage (tokenizer, lam, dynamics), loaded into the classes of swm.baselines.surgwm with SurgWMConfig(**meta["model"]).

Training data

• SurgVU (Zia et al., Surgical Visual Understanding (SurgVU) Dataset, arXiv:2501.09209): 155 da Vinci porcine training-lab cases, 837.8 h, with robot-log instrument and endoscope occupancy per arm.
• For kinematic grounding / benchmarks: Open-H-Embodiment subsets (CC-BY-4.0), JIGSAWS (Gao et al., M2CAI 2014; data-use agreement), SAR-RARP50 (CC BY-NC-SA 4.0).

Licence and intended use

Weights are released under CC BY-NC-SA 4.0 for non-commercial research, in line with the licences of the training data. The models simulate endoscopic video; they are not medical devices and must not be used for clinical decision-making or to control a robot on a patient.