sync run artifacts: vjepa/inflated/d003_lora_ema_capilvd1689m

vjepa/inflated/d003_lora_ema_capilvd1689m/best.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:41461fe488c16c640a57eb52a18130dea1556588c08944b13b46ab941914569a
+size 2577727345

vjepa/inflated/d003_lora_ema_capilvd1689m/git-info.txt

@@ -0,0 +1,2 @@
+branch: main
+commit: 3736c05d8720ac2e94ea9c532b582a5c6706c923

vjepa/inflated/d003_lora_ema_capilvd1689m/job_22606429/22606429_0_log.err

@@ -0,0 +1,113 @@
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
+  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
+  return func(*args, **kwargs)
+[rank0]:[W509 17:03:27.368187417 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
+wandb: Currently logged in as: dgcnz (uvjepa) to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
+wandb: setting up run s7pbl9w1
+wandb: Tracking run with wandb version 0.23.1
+wandb: Run data is saved locally in /scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/wandb/run-20260509_170336-s7pbl9w1
+wandb: Run `wandb offline` to turn off syncing.
+wandb: Syncing run d003_lora_ema_capilvd1689m
+wandb: ⭐️ View project at https://wandb.ai/uvjepa/vjepa_ablation
+wandb: 🚀 View run at https://wandb.ai/uvjepa/vjepa_ablation/runs/s7pbl9w1
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
+  scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
+  return func(*args, **kwargs)
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
+  return func(*args, **kwargs)
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
+  return func(*args, **kwargs)
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
+  return func(*args, **kwargs)
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
+  return func(*args, **kwargs)
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
+  return func(*args, **kwargs)
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
+  return func(*args, **kwargs)
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+submitit WARNING (2026-05-10 04:55:29,236) - Caught signal SIGUSR2 on gcn74.local.snellius.surf.nl: this job is timed-out.
+srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
+submitit WARNING (2026-05-10 04:55:29,468) - Bypassing signal SIGCONT
+[2026-05-10T04:55:29.469] error: *** JOB 22606429 ON gcn74 CANCELLED AT 2026-05-10T04:55:29 DUE TO JOB REQUEUE ***
+[2026-05-10T04:55:29.469] error: *** STEP 22606429.0 ON gcn74 CANCELLED AT 2026-05-10T04:55:29 DUE TO JOB REQUEUE ***
+submitit WARNING (2026-05-10 04:55:29,470) - Bypassing signal SIGCONT
+submitit WARNING (2026-05-10 04:55:29,472) - Bypassing signal SIGTERM
+[rank0]:[W510 04:55:32.117177654 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
+[2026-05-10T04:55:45.027] error: Failed to send MESSAGE_TASK_EXIT: Connection refused
+[2026-05-10T04:56:03.030] error: namespace_p_join: open failed for /slurm/22606429/.ns: No such file or directory
+[2026-05-10T04:56:03.032] error: namespace_g_join(JobId=22606429 SLUID=s8FXKR6KX9EK00): No such file or directory
+[2026-05-10T04:56:04.669] error: Failed to send MESSAGE_TASK_EXIT: Connection refused
+[2026-05-10T04:56:06.654] error: namespace_p_join: open failed for /slurm/22606429/.ns: No such file or directory
+[2026-05-10T04:56:06.656] error: namespace_g_join(JobId=22606429 SLUID=s8FXKR6KX9EK00): No such file or directory
+[2026-05-10T04:56:06.795] error: namespace_p_join: open failed for /slurm/22606429/.ns: No such file or directory
+[2026-05-10T04:56:06.795] error: namespace_g_join(JobId=22606429 SLUID=s8FXKR6KX9EK00): No such file or directory
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
+  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
+  return func(*args, **kwargs)
+[rank0]:[W510 05:04:28.354215764 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
+wandb: Currently logged in as: dgcnz (uvjepa) to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
+wandb: setting up run s7pbl9w1
+wandb: Tracking run with wandb version 0.23.1
+wandb: Run data is saved locally in /scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/wandb/run-20260510_050433-s7pbl9w1
+wandb: Run `wandb offline` to turn off syncing.
+wandb: Resuming run d003_lora_ema_capilvd1689m
+wandb: ⭐️ View project at https://wandb.ai/uvjepa/vjepa_ablation
+wandb: 🚀 View run at https://wandb.ai/uvjepa/vjepa_ablation/runs/s7pbl9w1
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
+  scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+wandb: WARNING Tried to log to step 9900 that is less than the current step 9981. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+wandb: WARNING Tried to log to step 9910 that is less than the current step 9981. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+wandb: WARNING Tried to log to step 9920 that is less than the current step 9981. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+wandb: WARNING Tried to log to step 9930 that is less than the current step 9981. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+wandb: WARNING Tried to log to step 9940 that is less than the current step 9981. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+wandb: WARNING Tried to log to step 9950 that is less than the current step 9981. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+wandb: WARNING Tried to log to step 9960 that is less than the current step 9981. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+wandb: WARNING Tried to log to step 9970 that is less than the current step 9981. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+wandb: WARNING Tried to log to step 9980 that is less than the current step 9981. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
+  return func(*args, **kwargs)
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
+  return func(*args, **kwargs)
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
+  return func(*args, **kwargs)
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
+  return func(*args, **kwargs)
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
+[2026-05-10T09:26:30.522] error: *** JOB 22606429 ON gcn74 CANCELLED AT 2026-05-10T09:26:30 DUE to SIGNAL Terminated ***
+[2026-05-10T09:26:30.522] error: *** STEP 22606429.0 ON gcn74 CANCELLED AT 2026-05-10T09:26:30 DUE to SIGNAL Terminated ***
+submitit WARNING (2026-05-10 09:26:31,185) - Bypassing signal SIGTERM
+submitit WARNING (2026-05-10 09:26:31,187) - Bypassing signal SIGCONT
+[2026-05-10T09:27:04.137] error: namespace_p_join: open failed for /slurm/22606429/.ns: No such file or directory
+[2026-05-10T09:27:04.137] error: namespace_g_join(JobId=22606429 SLUID=s8FXKR7X2B6T00): No such file or directory
+[2026-05-10T09:27:05.775] error: Failed to send MESSAGE_TASK_EXIT: Connection refused
+[2026-05-10T09:27:09.432] error: namespace_p_join: open failed for /slurm/22606429/.ns: No such file or directory
+[2026-05-10T09:27:09.432] error: namespace_g_join(JobId=22606429 SLUID=s8FXKR7X2B6T00): No such file or directory
+[2026-05-10T09:27:09.572] error: namespace_p_join: open failed for /slurm/22606429/.ns: No such file or directory
+[2026-05-10T09:27:09.572] error: namespace_g_join(JobId=22606429 SLUID=s8FXKR7X2B6T00): No such file or directory
+[2026-05-10T09:27:09.611] error: namespace_p_join: open failed for /slurm/22606429/.ns: No such file or directory
+[2026-05-10T09:27:09.611] error: namespace_g_join(JobId=22606429 SLUID=s8FXKR7X2B6T00): No such file or directory

vjepa/inflated/d003_lora_ema_capilvd1689m/job_22606429/22606429_10_log.err

@@ -0,0 +1,295 @@
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
+  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
+[rank10]:[W509 17:02:07.865310197 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
+  scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+submitit WARNING (2026-05-10 04:55:29,448) - Caught signal SIGUSR2 on gcn76.local.snellius.surf.nl: this job is timed-out.
+submitit WARNING (2026-05-10 04:55:29,448) - Bypassing signal SIGCONT
+submitit WARNING (2026-05-10 04:55:29,547) - Bypassing signal SIGTERM
+submitit WARNING (2026-05-10 04:55:29,547) - Bypassing signal SIGCONT
+[rank10]:[W510 04:55:34.969346976 TCPStore.cpp:125] [c10d] recvValue failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:40526, remote=[gcn74.local.snellius.surf.nl]:37129): Failed to recv, got 0 bytes. Connection was likely closed. Did the remote server shutdown or crash?
+Exception raised from recvBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:682 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14aa51f08fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a3325d (0x14aa9609725d in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x273 (0x14aa960951f3 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14aa53177eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14ab926e1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14aba315319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14aba31d8100 in /lib64/libc.so.6)
+
+[rank10]:[W510 04:55:34.973105120 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 10] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Failed to recv, got 0 bytes. Connection was likely closed. Did the remote server shutdown or crash?
+[rank10]:[W510 04:55:35.973247482 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:40526, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14aa51f08fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14aa960966d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14aa960951cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14aa53177eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14ab926e1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14aba315319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14aba31d8100 in /lib64/libc.so.6)
+
+[rank10]:[W510 04:55:35.976427499 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 10] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank10]:[W510 04:55:36.976573201 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:40526, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14aa51f08fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14aa960966d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14aa960951cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14aa53177eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14ab926e1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14aba315319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14aba31d8100 in /lib64/libc.so.6)
+
+[rank10]:[W510 04:55:36.978892321 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 10] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank10]:[W510 04:55:37.979057774 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:40526, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14aa51f08fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14aa960966d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14aa960951cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14aa53177eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14ab926e1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14aba315319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14aba31d8100 in /lib64/libc.so.6)
+
+[rank10]:[W510 04:55:37.981444994 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 10] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank10]:[W510 04:55:38.981573466 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:40526, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14aa51f08fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14aa960966d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14aa960951cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14aa53177eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14ab926e1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14aba315319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14aba31d8100 in /lib64/libc.so.6)
+
+[rank10]:[W510 04:55:38.983868177 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 10] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank10]:[W510 04:55:39.983995509 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:40526, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14aa51f08fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14aa960966d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14aa960951cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14aa53177eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14ab926e1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14aba315319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14aba31d8100 in /lib64/libc.so.6)
+
+[rank10]:[W510 04:55:39.986284640 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 10] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank10]:[W510 04:55:40.986419272 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:40526, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14aa51f08fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14aa960966d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14aa960951cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14aa53177eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14ab926e1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14aba315319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14aba31d8100 in /lib64/libc.so.6)
+
+[rank10]:[W510 04:55:40.988632463 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 10] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank10]:[W510 04:55:41.988763086 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:40526, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14aa51f08fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14aa960966d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14aa960951cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14aa53177eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14ab926e1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14aba315319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14aba31d8100 in /lib64/libc.so.6)
+
+[rank10]:[W510 04:55:41.991172835 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 10] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank10]:[W510 04:55:42.991308038 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:40526, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14aa51f08fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14aa960966d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14aa960951cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14aa53177eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14ab926e1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14aba315319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14aba31d8100 in /lib64/libc.so.6)
+
+[rank10]:[W510 04:55:42.993587608 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 10] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank10]:[W510 04:55:43.993720001 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:40526, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14aa51f08fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14aa960966d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14aa960951cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14aa53177eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14ab926e1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14aba315319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14aba31d8100 in /lib64/libc.so.6)
+
+[rank10]:[W510 04:55:43.996033411 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 10] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank10]:[W510 04:55:44.996173104 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:40526, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14aa51f08fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14aa960966d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14aa960951cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14aa53177eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14ab926e1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14aba315319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14aba31d8100 in /lib64/libc.so.6)
+
+[rank10]:[W510 04:55:44.998503064 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 10] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank10]:[W510 04:55:45.998630326 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:40526, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14aa51f08fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14aa960966d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14aa960951cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14aa53177eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14ab926e1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14aba315319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14aba31d8100 in /lib64/libc.so.6)
+
+[rank10]:[W510 04:55:45.000842617 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 10] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank10]:[W510 04:55:46.000974640 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:40526, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14aa51f08fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14aa960966d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14aa960951cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14aa53177eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14ab926e1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14aba315319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14aba31d8100 in /lib64/libc.so.6)
+
+[rank10]:[W510 04:55:46.003230220 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 10] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank10]:[W510 04:55:47.003351623 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:40526, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14aa51f08fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14aa960966d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14aa960951cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14aa53177eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14ab926e1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14aba315319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14aba31d8100 in /lib64/libc.so.6)
+
+[rank10]:[W510 04:55:47.005706103 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 10] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank10]:[W510 04:55:48.005829715 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:40526, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14aa51f08fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14aa960966d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14aa960951cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14aa53177eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14ab926e1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14aba315319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14aba31d8100 in /lib64/libc.so.6)
+
+[rank10]:[W510 04:55:48.008092626 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 10] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank10]:[W510 04:55:49.008221528 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:40526, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14aa51f08fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14aa960966d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14aa960951cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14aa53177eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14ab926e1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14aba315319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14aba31d8100 in /lib64/libc.so.6)
+
+[rank10]:[W510 04:55:49.010584168 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 10] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank10]:[W510 04:55:50.010712631 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:40526, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14aa51f08fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14aa960966d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14aa960951cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14aa53177eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14ab926e1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14aba315319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14aba31d8100 in /lib64/libc.so.6)
+
+[rank10]:[W510 04:55:50.012963632 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 10] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank10]:[W510 04:55:51.013089874 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:40526, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14aa51f08fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14aa960966d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14aa960951cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14aa53177eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14ab926e1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14aba315319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14aba31d8100 in /lib64/libc.so.6)
+
+[rank10]:[W510 04:55:51.015435304 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 10] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank10]:[W510 04:55:52.015568757 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:40526, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14aa51f08fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14aa960966d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14aa960951cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14aa53177eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14ab926e1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14aba315319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14aba31d8100 in /lib64/libc.so.6)
+
+[rank10]:[W510 04:55:52.018026856 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 10] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank10]:[W510 04:55:53.018151939 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:40526, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14aa51f08fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14aa960966d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14aa960951cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14aa53177eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14ab926e1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14aba315319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14aba31d8100 in /lib64/libc.so.6)
+
+[rank10]:[W510 04:55:53.020438619 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 10] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank10]:[W510 04:55:54.020567582 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:40526, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14aa51f08fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14aa960966d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14aa960951cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14aa53177eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14ab926e1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14aba315319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14aba31d8100 in /lib64/libc.so.6)
+
+[rank10]:[W510 04:55:54.022859212 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 10] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank10]:[W510 04:55:55.022978515 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:40526, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14aa51f08fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14aa960966d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14aa960951cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14aa53177eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14ab926e1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14aba315319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14aba31d8100 in /lib64/libc.so.6)
+
+[rank10]:[W510 04:55:55.025359205 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 10] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank10]:[W510 04:55:56.025482668 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:40526, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14aa51f08fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14aa960966d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14aa960951cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14aa53177eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14ab926e1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14aba315319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14aba31d8100 in /lib64/libc.so.6)
+
+[rank10]:[W510 04:55:56.027841477 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 10] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank10]:[W510 04:55:57.027973610 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:40526, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14aa51f08fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14aa960966d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14aa960951cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14aa53177eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14ab926e1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14aba315319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14aba31d8100 in /lib64/libc.so.6)
+
+[rank10]:[W510 04:55:57.030273490 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 10] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank10]:[W510 04:55:58.030393613 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:40526, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14aa51f08fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14aa960966d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14aa960951cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14aa53177eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14ab926e1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14aba315319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14aba31d8100 in /lib64/libc.so.6)
+
+[rank10]:[W510 04:55:58.032816303 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 10] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
+  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
+[rank10]:[W510 05:02:24.175193731 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
+  scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+submitit WARNING (2026-05-10 09:26:31,100) - Bypassing signal SIGTERM
+submitit WARNING (2026-05-10 09:26:31,100) - Bypassing signal SIGCONT

vjepa/inflated/d003_lora_ema_capilvd1689m/job_22606429/22606429_10_log.out

@@ -0,0 +1,522 @@
+submitit INFO (2026-05-09 16:57:34,481) - Starting with JobEnvironment(job_id=22606429, hostname=gcn76.local.snellius.surf.nl, local_rank=2(4), node=2(4), global_rank=10(16))
+submitit INFO (2026-05-09 16:57:34,481) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/job_22606429/22606429_submitted.pkl
+INFO:root:loaded pretrain params...
+{   'app': 'vjepa',
+    'cpus_per_task': 16,
+    'data': {   'batch_size': 64,
+                'crop_size': 224,
+                'dataset_fpcs': [16, 16],
+                'dataset_type': 'VideoDataset',
+                'datasets': [   '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
+                                '/scratch-shared/dcanez/data/ssv2/train.csv'],
+                'datasets_weights': [0.65, 0.35],
+                'fps': 4,
+                'num_workers': 10,
+                'patch_size': 14,
+                'persistent_workers': True,
+                'pin_mem': True,
+                'stage': [   {   'dest': 'kinetics_240',
+                                 'format': 'targz_parts',
+                                 'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
+                             {   'dest': 'ssv2',
+                                 'format': 'multipart_tar',
+                                 'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
+                'tubelet_size': 1},
+    'data_aug': {   'auto_augment': False,
+                    'motion_shift': False,
+                    'random_resize_aspect_ratio': [0.75, 1.35],
+                    'random_resize_scale': [0.3, 1.0],
+                    'reprob': 0.0},
+    'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m',
+    'loss': {'loss_exp': 1.0},
+    'mask': [   {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 8,
+                    'spatial_scale': [0.15, 0.15],
+                    'temporal_scale': [1.0, 1.0]},
+                {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 2,
+                    'spatial_scale': [0.7, 0.7],
+                    'temporal_scale': [1.0, 1.0]}],
+    'mem_per_gpu': '180G',
+    'meta': {   'dtype': 'bfloat16',
+                'knn_eval_epoch0': False,
+                'knn_eval_freq': 5,
+                'knn_eval_presets': [   {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
+                                                          'eval_videos_per_class': 25,
+                                                          'num_workers': 8,
+                                                          'train_videos_per_class': 100},
+                                            'preset': 'ucf101'},
+                                        {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
+                                                          'eval_videos_per_class': 100,
+                                                          'linear_probe': True,
+                                                          'num_workers': 8,
+                                                          'pool_type': 'temporal_concat',
+                                                          'train_videos_per_class': 500},
+                                            'preset': 'ssv2_coarse10'}],
+                'load_checkpoint': True,
+                'read_checkpoint': None,
+                'save_every_freq': 5,
+                'seed': 239,
+                'use_sdpa': True,
+                'use_wandb': True,
+                'wandb_project': 'vjepa_ablation'},
+    'metrics': {'sigreg': {}, 'std': {}},
+    'model': {   'class_token': False,
+                 'freeze_backbone': True,
+                 'is_causal': False,
+                 'lora_alpha': 16,
+                 'lora_rank': 8,
+                 'lora_targets': ['attn.qkv', 'attn.proj'],
+                 'model_name': 'vit_large_patch14_capi_lvd1689m',
+                 'num_reg_tokens': 16,
+                 'pred_depth': 6,
+                 'pred_embed_dim': 384,
+                 'pred_num_heads': 12,
+                 'rope_type': 'capi2d',
+                 'stem_type': '2d',
+                 'target_kind': 'ema',
+                 'uniform_power': True,
+                 'use_activation_checkpointing': True,
+                 'use_lora': True,
+                 'use_mask_tokens': True,
+                 'use_rope': True,
+                 'use_sdpa': True,
+                 'zero_init_mask_tokens': True},
+    'nodes': 4,
+    'optimization': {   'ema': [0.99925, 0.99925],
+                        'epochs': 100,
+                        'final_lr': 0.0001,
+                        'final_weight_decay': 0.04,
+                        'ipe': 300,
+                        'ipe_scale': 1.0,
+                        'lr': 0.0005,
+                        'start_lr': 0.0001,
+                        'warmup': 10,
+                        'weight_decay': 0.04},
+    'tasks_per_node': 4}
+INFO:root:Running pre-training of app: vjepa
+[INFO    ][2026-05-09 16:57:51][app.vjepa.train     ][main                     ] which_dtype='bfloat16'
+[INFO    ][2026-05-09 16:57:51][app.vjepa.train     ][main                     ] Disabling persistent_workers (incompatible with KNN eval)
+[INFO    ][2026-05-09 16:57:51][app.vjepa.train     ][main                     ] NCCL_SOCKET_IFNAME=eno
+[INFO    ][2026-05-09 16:59:02][app.vjepa.train     ][main                     ] Initialized (rank/world-size) 10/16, tasks_per_node=4
+[INFO    ][2026-05-09 16:59:02][root                ][stage_datasets           ] [local_rank 2/4] Staging kinetics_240 (targz_parts)
+[INFO    ][2026-05-09 16:59:02][root                ][_stage_targz_parts       ] [rank 2] Extracting 26/103 tar.gz parts to /scratch-node/dcanez.22606429/kinetics_240
+[INFO    ][2026-05-09 16:59:16][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 2/26 parts
+[INFO    ][2026-05-09 16:59:30][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 4/26 parts
+[INFO    ][2026-05-09 16:59:45][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 6/26 parts
+[INFO    ][2026-05-09 16:59:59][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 8/26 parts
+[INFO    ][2026-05-09 17:00:13][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 10/26 parts
+[INFO    ][2026-05-09 17:00:27][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 12/26 parts
+[INFO    ][2026-05-09 17:00:41][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 14/26 parts
+[INFO    ][2026-05-09 17:00:56][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 16/26 parts
+[INFO    ][2026-05-09 17:01:10][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 18/26 parts
+[INFO    ][2026-05-09 17:01:24][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 20/26 parts
+[INFO    ][2026-05-09 17:01:39][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 22/26 parts
+[INFO    ][2026-05-09 17:01:53][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 24/26 parts
+[INFO    ][2026-05-09 17:02:07][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 26/26 parts
+[INFO    ][2026-05-09 17:02:07][root                ][stage_datasets           ] [local_rank 2/4] Staging ssv2 (multipart_tar)
+[INFO    ][2026-05-09 17:03:31][app.vjepa.train     ][main                     ] Staged datasets: ['/scratch-node/dcanez.22606429/kinetics_240/train.csv', '/scratch-node/dcanez.22606429/ssv2/train.csv']
+[INFO    ][2026-05-09 17:03:50][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
+[INFO    ][2026-05-09 17:03:53][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA applied: 48 modules replaced (rank=8, alpha=16, dropout=0.0, targets=['attn.qkv', 'attn.proj'], scope=blocks)
+[INFO    ][2026-05-09 17:03:53][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA trainable params: 1,179,648
+[INFO    ][2026-05-09 17:03:53][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-09 17:03:53][root                ][init_video_model         ] VJEPAPredictorMultiSeqWrapper(
+  (backbone): VisionTransformerPredictor(
+    (predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
+    (mask_tokens): ParameterList(
+        (0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+    )
+    (predictor_blocks): ModuleList(
+      (0-5): 6 x Block(
+        (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (attn): RoPEAttention(
+          (qkv): Linear(in_features=384, out_features=1152, bias=True)
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): Linear(in_features=384, out_features=384, bias=True)
+          (proj_drop): Dropout(p=0.0, inplace=False)
+        )
+        (drop_path): Identity()
+        (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (mlp): MLP(
+          (fc1): Linear(in_features=384, out_features=1536, bias=True)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=1536, out_features=384, bias=True)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+    (predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
+  )
+)
+[INFO    ][2026-05-09 17:03:53][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-09 17:03:53][root                ][init_video_model         ] Encoder number of parameters: 1179648
+[INFO    ][2026-05-09 17:03:53][root                ][init_video_model         ] Predictor number of parameters: 11436928
+[INFO    ][2026-05-09 17:03:53][root                ][init_video_model         ] Target encoder number of parameters: 1179648
+[INFO    ][2026-05-09 17:04:00][root                ][make_videodataset        ] VideoDataset dataset created
+[INFO    ][2026-05-09 17:04:00][WeightedSampler     ][__init__                 ] Using DistributedWeightedSampler with rank 10 / 16
+[INFO    ][2026-05-09 17:04:00][root                ][make_videodataset        ] VideoDataset unsupervised data loader created
+[INFO    ][2026-05-09 17:04:00][app.vjepa.train     ][main                     ] iterations per epoch/dataset length: 300/399
+[INFO    ][2026-05-09 17:04:00][app.vjepa.train     ][main                     ] Wrapping models in DDP (rank 10)...
+[INFO    ][2026-05-09 17:05:26][app.vjepa.train     ][main                     ] Initializing loader...
+submitit INFO (2026-05-10 04:55:29,447) - Job has timed out. Ran 718 minutes out of requested 720 minutes.
+[INFO    ][2026-05-10 04:55:29][submitit            ][has_timed_out            ] Job has timed out. Ran 718 minutes out of requested 720 minutes.
+submitit WARNING (2026-05-10 04:55:29,448) - Caught signal SIGUSR2 on gcn76.local.snellius.surf.nl: this job is timed-out.
+[WARNING ][2026-05-10 04:55:29][submitit            ][checkpoint_and_try_requeue] Caught signal SIGUSR2 on gcn76.local.snellius.surf.nl: this job is timed-out.
+submitit INFO (2026-05-10 04:55:29,448) - Not checkpointing nor requeuing since I am a slave (procid=10).
+[INFO    ][2026-05-10 04:55:29][submitit            ][checkpoint_and_try_requeue] Not checkpointing nor requeuing since I am a slave (procid=10).
+submitit WARNING (2026-05-10 04:55:29,448) - Bypassing signal SIGCONT
+[WARNING ][2026-05-10 04:55:29][submitit            ][bypass                   ] Bypassing signal SIGCONT
+submitit WARNING (2026-05-10 04:55:29,547) - Bypassing signal SIGTERM
+[WARNING ][2026-05-10 04:55:29][submitit            ][bypass                   ] Bypassing signal SIGTERM
+submitit WARNING (2026-05-10 04:55:29,547) - Bypassing signal SIGCONT
+[WARNING ][2026-05-10 04:55:29][submitit            ][bypass                   ] Bypassing signal SIGCONT
+submitit INFO (2026-05-10 04:59:12,611) - Starting with JobEnvironment(job_id=22606429, hostname=gcn76.local.snellius.surf.nl, local_rank=2(4), node=2(4), global_rank=10(16))
+submitit INFO (2026-05-10 04:59:12,611) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/job_22606429/22606429_submitted.pkl
+INFO:root:loaded pretrain params...
+{   'app': 'vjepa',
+    'cpus_per_task': 16,
+    'data': {   'batch_size': 64,
+                'crop_size': 224,
+                'dataset_fpcs': [16, 16],
+                'dataset_type': 'VideoDataset',
+                'datasets': [   '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
+                                '/scratch-shared/dcanez/data/ssv2/train.csv'],
+                'datasets_weights': [0.65, 0.35],
+                'fps': 4,
+                'num_workers': 10,
+                'patch_size': 14,
+                'persistent_workers': True,
+                'pin_mem': True,
+                'stage': [   {   'dest': 'kinetics_240',
+                                 'format': 'targz_parts',
+                                 'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
+                             {   'dest': 'ssv2',
+                                 'format': 'multipart_tar',
+                                 'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
+                'tubelet_size': 1},
+    'data_aug': {   'auto_augment': False,
+                    'motion_shift': False,
+                    'random_resize_aspect_ratio': [0.75, 1.35],
+                    'random_resize_scale': [0.3, 1.0],
+                    'reprob': 0.0},
+    'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m',
+    'loss': {'loss_exp': 1.0},
+    'mask': [   {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 8,
+                    'spatial_scale': [0.15, 0.15],
+                    'temporal_scale': [1.0, 1.0]},
+                {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 2,
+                    'spatial_scale': [0.7, 0.7],
+                    'temporal_scale': [1.0, 1.0]}],
+    'mem_per_gpu': '180G',
+    'meta': {   'dtype': 'bfloat16',
+                'knn_eval_epoch0': False,
+                'knn_eval_freq': 5,
+                'knn_eval_presets': [   {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
+                                                          'eval_videos_per_class': 25,
+                                                          'num_workers': 8,
+                                                          'train_videos_per_class': 100},
+                                            'preset': 'ucf101'},
+                                        {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
+                                                          'eval_videos_per_class': 100,
+                                                          'linear_probe': True,
+                                                          'num_workers': 8,
+                                                          'pool_type': 'temporal_concat',
+                                                          'train_videos_per_class': 500},
+                                            'preset': 'ssv2_coarse10'}],
+                'load_checkpoint': True,
+                'read_checkpoint': None,
+                'save_every_freq': 5,
+                'seed': 239,
+                'use_sdpa': True,
+                'use_wandb': True,
+                'wandb_project': 'vjepa_ablation'},
+    'metrics': {'sigreg': {}, 'std': {}},
+    'model': {   'class_token': False,
+                 'freeze_backbone': True,
+                 'is_causal': False,
+                 'lora_alpha': 16,
+                 'lora_rank': 8,
+                 'lora_targets': ['attn.qkv', 'attn.proj'],
+                 'model_name': 'vit_large_patch14_capi_lvd1689m',
+                 'num_reg_tokens': 16,
+                 'pred_depth': 6,
+                 'pred_embed_dim': 384,
+                 'pred_num_heads': 12,
+                 'rope_type': 'capi2d',
+                 'stem_type': '2d',
+                 'target_kind': 'ema',
+                 'uniform_power': True,
+                 'use_activation_checkpointing': True,
+                 'use_lora': True,
+                 'use_mask_tokens': True,
+                 'use_rope': True,
+                 'use_sdpa': True,
+                 'zero_init_mask_tokens': True},
+    'nodes': 4,
+    'optimization': {   'ema': [0.99925, 0.99925],
+                        'epochs': 100,
+                        'final_lr': 0.0001,
+                        'final_weight_decay': 0.04,
+                        'ipe': 300,
+                        'ipe_scale': 1.0,
+                        'lr': 0.0005,
+                        'start_lr': 0.0001,
+                        'warmup': 10,
+                        'weight_decay': 0.04},
+    'tasks_per_node': 4}
+INFO:root:Running pre-training of app: vjepa
+[INFO    ][2026-05-10 04:59:17][app.vjepa.train     ][main                     ] which_dtype='bfloat16'
+[INFO    ][2026-05-10 04:59:17][app.vjepa.train     ][main                     ] Disabling persistent_workers (incompatible with KNN eval)
+[INFO    ][2026-05-10 04:59:17][app.vjepa.train     ][main                     ] NCCL_SOCKET_IFNAME=eno
+[INFO    ][2026-05-10 04:59:19][app.vjepa.train     ][main                     ] Initialized (rank/world-size) 10/16, tasks_per_node=4
+[INFO    ][2026-05-10 04:59:19][root                ][stage_datasets           ] [local_rank 2/4] Staging kinetics_240 (targz_parts)
+[INFO    ][2026-05-10 04:59:19][root                ][_stage_targz_parts       ] [rank 2] Extracting 26/103 tar.gz parts to /scratch-node/dcanez.22606429/kinetics_240
+[INFO    ][2026-05-10 04:59:33][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 2/26 parts
+[INFO    ][2026-05-10 04:59:47][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 4/26 parts
+[INFO    ][2026-05-10 05:00:02][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 6/26 parts
+[INFO    ][2026-05-10 05:00:16][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 8/26 parts
+[INFO    ][2026-05-10 05:00:31][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 10/26 parts
+[INFO    ][2026-05-10 05:00:45][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 12/26 parts
+[INFO    ][2026-05-10 05:00:59][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 14/26 parts
+[INFO    ][2026-05-10 05:01:13][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 16/26 parts
+[INFO    ][2026-05-10 05:01:27][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 18/26 parts
+[INFO    ][2026-05-10 05:01:41][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 20/26 parts
+[INFO    ][2026-05-10 05:01:56][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 22/26 parts
+[INFO    ][2026-05-10 05:02:10][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 24/26 parts
+[INFO    ][2026-05-10 05:02:24][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 26/26 parts
+[INFO    ][2026-05-10 05:02:24][root                ][stage_datasets           ] [local_rank 2/4] Staging ssv2 (multipart_tar)
+[INFO    ][2026-05-10 05:04:33][app.vjepa.train     ][main                     ] Staged datasets: ['/scratch-node/dcanez.22606429/kinetics_240/train.csv', '/scratch-node/dcanez.22606429/ssv2/train.csv']
+[INFO    ][2026-05-10 05:04:47][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
+[INFO    ][2026-05-10 05:04:51][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA applied: 48 modules replaced (rank=8, alpha=16, dropout=0.0, targets=['attn.qkv', 'attn.proj'], scope=blocks)
+[INFO    ][2026-05-10 05:04:51][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA trainable params: 1,179,648
+[INFO    ][2026-05-10 05:04:52][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-10 05:04:52][root                ][init_video_model         ] VJEPAPredictorMultiSeqWrapper(
+  (backbone): VisionTransformerPredictor(
+    (predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
+    (mask_tokens): ParameterList(
+        (0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+    )
+    (predictor_blocks): ModuleList(
+      (0-5): 6 x Block(
+        (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (attn): RoPEAttention(
+          (qkv): Linear(in_features=384, out_features=1152, bias=True)
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): Linear(in_features=384, out_features=384, bias=True)
+          (proj_drop): Dropout(p=0.0, inplace=False)
+        )
+        (drop_path): Identity()
+        (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (mlp): MLP(
+          (fc1): Linear(in_features=384, out_features=1536, bias=True)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=1536, out_features=384, bias=True)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+    (predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
+  )
+)
+[INFO    ][2026-05-10 05:04:52][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-10 05:04:52][root                ][init_video_model         ] Encoder number of parameters: 1179648
+[INFO    ][2026-05-10 05:04:52][root                ][init_video_model         ] Predictor number of parameters: 11436928
+[INFO    ][2026-05-10 05:04:52][root                ][init_video_model         ] Target encoder number of parameters: 1179648
+[INFO    ][2026-05-10 05:04:53][root                ][make_videodataset        ] VideoDataset dataset created
+[INFO    ][2026-05-10 05:04:53][WeightedSampler     ][__init__                 ] Using DistributedWeightedSampler with rank 10 / 16
+[INFO    ][2026-05-10 05:04:53][root                ][make_videodataset        ] VideoDataset unsupervised data loader created
+[INFO    ][2026-05-10 05:04:53][app.vjepa.train     ][main                     ] iterations per epoch/dataset length: 300/399
+[INFO    ][2026-05-10 05:04:53][app.vjepa.train     ][main                     ] Wrapping models in DDP (rank 10)...
+[INFO    ][2026-05-10 05:06:08][root                ][load_checkpoint          ] Loading checkpoint from /scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/latest.pt
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] loaded pretrained encoder from epoch 33 with msg: <All keys matched successfully>
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] loaded pretrained predictor from epoch 33 with msg: <All keys matched successfully>
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] loaded pretrained target encoder from epoch 33 with msg: <All keys matched successfully>
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] loaded optimizers from epoch 33
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] read-path: /scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/latest.pt
+[INFO    ][2026-05-10 05:06:09][app.vjepa.train     ][main                     ] Initializing loader...
+submitit WARNING (2026-05-10 09:26:31,100) - Bypassing signal SIGTERM
+[WARNING ][2026-05-10 09:26:31][submitit            ][bypass                   ] Bypassing signal SIGTERM
+submitit WARNING (2026-05-10 09:26:31,100) - Bypassing signal SIGCONT
+[WARNING ][2026-05-10 09:26:31][submitit            ][bypass                   ] Bypassing signal SIGCONT

vjepa/inflated/d003_lora_ema_capilvd1689m/job_22606429/22606429_11_log.err

@@ -0,0 +1,295 @@
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
+  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
+[rank11]:[W509 17:02:18.677817728 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
+  scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+submitit WARNING (2026-05-10 04:55:29,279) - Caught signal SIGUSR2 on gcn76.local.snellius.surf.nl: this job is timed-out.
+submitit WARNING (2026-05-10 04:55:29,279) - Bypassing signal SIGCONT
+submitit WARNING (2026-05-10 04:55:29,547) - Bypassing signal SIGTERM
+submitit WARNING (2026-05-10 04:55:29,547) - Bypassing signal SIGCONT
+[rank11]:[W510 04:55:34.969268456 TCPStore.cpp:125] [c10d] recvValue failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:37978, remote=[gcn74.local.snellius.surf.nl]:37129): Failed to recv, got 0 bytes. Connection was likely closed. Did the remote server shutdown or crash?
+Exception raised from recvBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:682 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14876a16bfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a3325d (0x1487ae2fa25d in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x273 (0x1487ae2f81f3 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14876b3daeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1488aa944164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1488bb3b619a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1488bb43b100 in /lib64/libc.so.6)
+
+[rank11]:[W510 04:55:34.972870931 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 11] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Failed to recv, got 0 bytes. Connection was likely closed. Did the remote server shutdown or crash?
+[rank11]:[W510 04:55:35.973044903 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:37978, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14876a16bfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1487ae2f96d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1487ae2f81cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14876b3daeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1488aa944164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1488bb3b619a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1488bb43b100 in /lib64/libc.so.6)
+
+[rank11]:[W510 04:55:35.976429809 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 11] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank11]:[W510 04:55:36.976564161 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:37978, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14876a16bfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1487ae2f96d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1487ae2f81cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14876b3daeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1488aa944164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1488bb3b619a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1488bb43b100 in /lib64/libc.so.6)
+
+[rank11]:[W510 04:55:36.978872911 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 11] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank11]:[W510 04:55:37.979049364 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:37978, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14876a16bfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1487ae2f96d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1487ae2f81cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14876b3daeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1488aa944164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1488bb3b619a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1488bb43b100 in /lib64/libc.so.6)
+
+[rank11]:[W510 04:55:37.981367574 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 11] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank11]:[W510 04:55:38.981495827 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:37978, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14876a16bfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1487ae2f96d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1487ae2f81cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14876b3daeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1488aa944164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1488bb3b619a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1488bb43b100 in /lib64/libc.so.6)
+
+[rank11]:[W510 04:55:38.983901377 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 11] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank11]:[W510 04:55:39.984073909 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:37978, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14876a16bfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1487ae2f96d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1487ae2f81cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14876b3daeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1488aa944164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1488bb3b619a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1488bb43b100 in /lib64/libc.so.6)
+
+[rank11]:[W510 04:55:39.986408839 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 11] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank11]:[W510 04:55:40.986540182 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:37978, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14876a16bfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1487ae2f96d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1487ae2f81cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14876b3daeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1488aa944164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1488bb3b619a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1488bb43b100 in /lib64/libc.so.6)
+
+[rank11]:[W510 04:55:40.988904562 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 11] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank11]:[W510 04:55:41.989086324 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:37978, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14876a16bfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1487ae2f96d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1487ae2f81cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14876b3daeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1488aa944164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1488bb3b619a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1488bb43b100 in /lib64/libc.so.6)
+
+[rank11]:[W510 04:55:41.991445104 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 11] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank11]:[W510 04:55:42.991581567 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:37978, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14876a16bfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1487ae2f96d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1487ae2f81cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14876b3daeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1488aa944164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1488bb3b619a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1488bb43b100 in /lib64/libc.so.6)
+
+[rank11]:[W510 04:55:42.993875897 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 11] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank11]:[W510 04:55:43.993998740 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:37978, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14876a16bfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1487ae2f96d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1487ae2f81cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14876b3daeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1488aa944164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1488bb3b619a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1488bb43b100 in /lib64/libc.so.6)
+
+[rank11]:[W510 04:55:43.996332880 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 11] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank11]:[W510 04:55:44.996485622 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:37978, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14876a16bfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1487ae2f96d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1487ae2f81cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14876b3daeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1488aa944164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1488bb3b619a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1488bb43b100 in /lib64/libc.so.6)
+
+[rank11]:[W510 04:55:44.998835292 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 11] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank11]:[W510 04:55:45.998964655 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:37978, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14876a16bfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1487ae2f96d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1487ae2f81cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14876b3daeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1488aa944164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1488bb3b619a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1488bb43b100 in /lib64/libc.so.6)
+
+[rank11]:[W510 04:55:45.001370215 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 11] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank11]:[W510 04:55:46.001492737 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:37978, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14876a16bfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1487ae2f96d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1487ae2f81cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14876b3daeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1488aa944164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1488bb3b619a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1488bb43b100 in /lib64/libc.so.6)
+
+[rank11]:[W510 04:55:46.003724978 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 11] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank11]:[W510 04:55:47.003840681 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:37978, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14876a16bfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1487ae2f96d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1487ae2f81cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14876b3daeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1488aa944164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1488bb3b619a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1488bb43b100 in /lib64/libc.so.6)
+
+[rank11]:[W510 04:55:47.006249850 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 11] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank11]:[W510 04:55:48.006367503 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:37978, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14876a16bfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1487ae2f96d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1487ae2f81cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14876b3daeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1488aa944164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1488bb3b619a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1488bb43b100 in /lib64/libc.so.6)
+
+[rank11]:[W510 04:55:48.008712613 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 11] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank11]:[W510 04:55:49.008835076 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:37978, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14876a16bfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1487ae2f96d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1487ae2f81cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14876b3daeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1488aa944164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1488bb3b619a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1488bb43b100 in /lib64/libc.so.6)
+
+[rank11]:[W510 04:55:49.011241586 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 11] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank11]:[W510 04:55:50.011379118 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:37978, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14876a16bfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1487ae2f96d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1487ae2f81cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14876b3daeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1488aa944164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1488bb3b619a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1488bb43b100 in /lib64/libc.so.6)
+
+[rank11]:[W510 04:55:50.013723848 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 11] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank11]:[W510 04:55:51.013847911 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:37978, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14876a16bfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1487ae2f96d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1487ae2f81cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14876b3daeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1488aa944164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1488bb3b619a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1488bb43b100 in /lib64/libc.so.6)
+
+[rank11]:[W510 04:55:51.016240561 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 11] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank11]:[W510 04:55:52.016360694 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:37978, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14876a16bfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1487ae2f96d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1487ae2f81cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14876b3daeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1488aa944164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1488bb3b619a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1488bb43b100 in /lib64/libc.so.6)
+
+[rank11]:[W510 04:55:52.018649734 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 11] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank11]:[W510 04:55:53.018761387 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:37978, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14876a16bfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1487ae2f96d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1487ae2f81cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14876b3daeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1488aa944164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1488bb3b619a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1488bb43b100 in /lib64/libc.so.6)
+
+[rank11]:[W510 04:55:53.021294666 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 11] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank11]:[W510 04:55:54.021419738 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:37978, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14876a16bfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1487ae2f96d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1487ae2f81cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14876b3daeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1488aa944164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1488bb3b619a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1488bb43b100 in /lib64/libc.so.6)
+
+[rank11]:[W510 04:55:54.023701789 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 11] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank11]:[W510 04:55:55.023818541 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:37978, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14876a16bfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1487ae2f96d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1487ae2f81cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14876b3daeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1488aa944164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1488bb3b619a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1488bb43b100 in /lib64/libc.so.6)
+
+[rank11]:[W510 04:55:55.026104602 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 11] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank11]:[W510 04:55:56.026247404 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:37978, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14876a16bfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1487ae2f96d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1487ae2f81cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14876b3daeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1488aa944164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1488bb3b619a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1488bb43b100 in /lib64/libc.so.6)
+
+[rank11]:[W510 04:55:56.028634784 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 11] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank11]:[W510 04:55:57.028763937 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:37978, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14876a16bfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1487ae2f96d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1487ae2f81cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14876b3daeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1488aa944164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1488bb3b619a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1488bb43b100 in /lib64/libc.so.6)
+
+[rank11]:[W510 04:55:57.031027807 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 11] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank11]:[W510 04:55:58.031160720 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:37978, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14876a16bfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1487ae2f96d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1487ae2f81cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14876b3daeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1488aa944164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1488bb3b619a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1488bb43b100 in /lib64/libc.so.6)
+
+[rank11]:[W510 04:55:58.033411590 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 11] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
+  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
+[rank11]:[W510 05:02:19.111911008 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
+  scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+submitit WARNING (2026-05-10 09:26:31,193) - Bypassing signal SIGTERM
+submitit WARNING (2026-05-10 09:26:31,285) - Bypassing signal SIGCONT

vjepa/inflated/d003_lora_ema_capilvd1689m/job_22606429/22606429_11_log.out

@@ -0,0 +1,522 @@
+submitit INFO (2026-05-09 16:57:34,481) - Starting with JobEnvironment(job_id=22606429, hostname=gcn76.local.snellius.surf.nl, local_rank=3(4), node=2(4), global_rank=11(16))
+submitit INFO (2026-05-09 16:57:34,481) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/job_22606429/22606429_submitted.pkl
+INFO:root:loaded pretrain params...
+{   'app': 'vjepa',
+    'cpus_per_task': 16,
+    'data': {   'batch_size': 64,
+                'crop_size': 224,
+                'dataset_fpcs': [16, 16],
+                'dataset_type': 'VideoDataset',
+                'datasets': [   '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
+                                '/scratch-shared/dcanez/data/ssv2/train.csv'],
+                'datasets_weights': [0.65, 0.35],
+                'fps': 4,
+                'num_workers': 10,
+                'patch_size': 14,
+                'persistent_workers': True,
+                'pin_mem': True,
+                'stage': [   {   'dest': 'kinetics_240',
+                                 'format': 'targz_parts',
+                                 'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
+                             {   'dest': 'ssv2',
+                                 'format': 'multipart_tar',
+                                 'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
+                'tubelet_size': 1},
+    'data_aug': {   'auto_augment': False,
+                    'motion_shift': False,
+                    'random_resize_aspect_ratio': [0.75, 1.35],
+                    'random_resize_scale': [0.3, 1.0],
+                    'reprob': 0.0},
+    'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m',
+    'loss': {'loss_exp': 1.0},
+    'mask': [   {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 8,
+                    'spatial_scale': [0.15, 0.15],
+                    'temporal_scale': [1.0, 1.0]},
+                {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 2,
+                    'spatial_scale': [0.7, 0.7],
+                    'temporal_scale': [1.0, 1.0]}],
+    'mem_per_gpu': '180G',
+    'meta': {   'dtype': 'bfloat16',
+                'knn_eval_epoch0': False,
+                'knn_eval_freq': 5,
+                'knn_eval_presets': [   {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
+                                                          'eval_videos_per_class': 25,
+                                                          'num_workers': 8,
+                                                          'train_videos_per_class': 100},
+                                            'preset': 'ucf101'},
+                                        {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
+                                                          'eval_videos_per_class': 100,
+                                                          'linear_probe': True,
+                                                          'num_workers': 8,
+                                                          'pool_type': 'temporal_concat',
+                                                          'train_videos_per_class': 500},
+                                            'preset': 'ssv2_coarse10'}],
+                'load_checkpoint': True,
+                'read_checkpoint': None,
+                'save_every_freq': 5,
+                'seed': 239,
+                'use_sdpa': True,
+                'use_wandb': True,
+                'wandb_project': 'vjepa_ablation'},
+    'metrics': {'sigreg': {}, 'std': {}},
+    'model': {   'class_token': False,
+                 'freeze_backbone': True,
+                 'is_causal': False,
+                 'lora_alpha': 16,
+                 'lora_rank': 8,
+                 'lora_targets': ['attn.qkv', 'attn.proj'],
+                 'model_name': 'vit_large_patch14_capi_lvd1689m',
+                 'num_reg_tokens': 16,
+                 'pred_depth': 6,
+                 'pred_embed_dim': 384,
+                 'pred_num_heads': 12,
+                 'rope_type': 'capi2d',
+                 'stem_type': '2d',
+                 'target_kind': 'ema',
+                 'uniform_power': True,
+                 'use_activation_checkpointing': True,
+                 'use_lora': True,
+                 'use_mask_tokens': True,
+                 'use_rope': True,
+                 'use_sdpa': True,
+                 'zero_init_mask_tokens': True},
+    'nodes': 4,
+    'optimization': {   'ema': [0.99925, 0.99925],
+                        'epochs': 100,
+                        'final_lr': 0.0001,
+                        'final_weight_decay': 0.04,
+                        'ipe': 300,
+                        'ipe_scale': 1.0,
+                        'lr': 0.0005,
+                        'start_lr': 0.0001,
+                        'warmup': 10,
+                        'weight_decay': 0.04},
+    'tasks_per_node': 4}
+INFO:root:Running pre-training of app: vjepa
+[INFO    ][2026-05-09 16:57:51][app.vjepa.train     ][main                     ] which_dtype='bfloat16'
+[INFO    ][2026-05-09 16:57:51][app.vjepa.train     ][main                     ] Disabling persistent_workers (incompatible with KNN eval)
+[INFO    ][2026-05-09 16:57:51][app.vjepa.train     ][main                     ] NCCL_SOCKET_IFNAME=eno
+[INFO    ][2026-05-09 16:59:18][app.vjepa.train     ][main                     ] Initialized (rank/world-size) 11/16, tasks_per_node=4
+[INFO    ][2026-05-09 16:59:18][root                ][stage_datasets           ] [local_rank 3/4] Staging kinetics_240 (targz_parts)
+[INFO    ][2026-05-09 16:59:18][root                ][_stage_targz_parts       ] [rank 3] Extracting 25/103 tar.gz parts to /scratch-node/dcanez.22606429/kinetics_240
+[INFO    ][2026-05-09 16:59:33][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 2/25 parts
+[INFO    ][2026-05-09 16:59:47][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 4/25 parts
+[INFO    ][2026-05-09 17:00:01][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 6/25 parts
+[INFO    ][2026-05-09 17:00:15][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 8/25 parts
+[INFO    ][2026-05-09 17:00:30][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 10/25 parts
+[INFO    ][2026-05-09 17:00:44][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 12/25 parts
+[INFO    ][2026-05-09 17:00:59][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 14/25 parts
+[INFO    ][2026-05-09 17:01:13][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 16/25 parts
+[INFO    ][2026-05-09 17:01:27][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 18/25 parts
+[INFO    ][2026-05-09 17:01:42][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 20/25 parts
+[INFO    ][2026-05-09 17:01:56][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 22/25 parts
+[INFO    ][2026-05-09 17:02:10][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 24/25 parts
+[INFO    ][2026-05-09 17:02:18][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 25/25 parts
+[INFO    ][2026-05-09 17:02:18][root                ][stage_datasets           ] [local_rank 3/4] Staging ssv2 (multipart_tar)
+[INFO    ][2026-05-09 17:03:31][app.vjepa.train     ][main                     ] Staged datasets: ['/scratch-node/dcanez.22606429/kinetics_240/train.csv', '/scratch-node/dcanez.22606429/ssv2/train.csv']
+[INFO    ][2026-05-09 17:03:49][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
+[INFO    ][2026-05-09 17:03:53][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA applied: 48 modules replaced (rank=8, alpha=16, dropout=0.0, targets=['attn.qkv', 'attn.proj'], scope=blocks)
+[INFO    ][2026-05-09 17:03:53][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA trainable params: 1,179,648
+[INFO    ][2026-05-09 17:03:53][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-09 17:03:53][root                ][init_video_model         ] VJEPAPredictorMultiSeqWrapper(
+  (backbone): VisionTransformerPredictor(
+    (predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
+    (mask_tokens): ParameterList(
+        (0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+    )
+    (predictor_blocks): ModuleList(
+      (0-5): 6 x Block(
+        (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (attn): RoPEAttention(
+          (qkv): Linear(in_features=384, out_features=1152, bias=True)
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): Linear(in_features=384, out_features=384, bias=True)
+          (proj_drop): Dropout(p=0.0, inplace=False)
+        )
+        (drop_path): Identity()
+        (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (mlp): MLP(
+          (fc1): Linear(in_features=384, out_features=1536, bias=True)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=1536, out_features=384, bias=True)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+    (predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
+  )
+)
+[INFO    ][2026-05-09 17:03:53][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-09 17:03:53][root                ][init_video_model         ] Encoder number of parameters: 1179648
+[INFO    ][2026-05-09 17:03:53][root                ][init_video_model         ] Predictor number of parameters: 11436928
+[INFO    ][2026-05-09 17:03:53][root                ][init_video_model         ] Target encoder number of parameters: 1179648
+[INFO    ][2026-05-09 17:04:04][root                ][make_videodataset        ] VideoDataset dataset created
+[INFO    ][2026-05-09 17:04:04][WeightedSampler     ][__init__                 ] Using DistributedWeightedSampler with rank 11 / 16
+[INFO    ][2026-05-09 17:04:04][root                ][make_videodataset        ] VideoDataset unsupervised data loader created
+[INFO    ][2026-05-09 17:04:04][app.vjepa.train     ][main                     ] iterations per epoch/dataset length: 300/399
+[INFO    ][2026-05-09 17:04:04][app.vjepa.train     ][main                     ] Wrapping models in DDP (rank 11)...
+[INFO    ][2026-05-09 17:05:26][app.vjepa.train     ][main                     ] Initializing loader...
+submitit INFO (2026-05-10 04:55:29,279) - Job has timed out. Ran 718 minutes out of requested 720 minutes.
+[INFO    ][2026-05-10 04:55:29][submitit            ][has_timed_out            ] Job has timed out. Ran 718 minutes out of requested 720 minutes.
+submitit WARNING (2026-05-10 04:55:29,279) - Caught signal SIGUSR2 on gcn76.local.snellius.surf.nl: this job is timed-out.
+[WARNING ][2026-05-10 04:55:29][submitit            ][checkpoint_and_try_requeue] Caught signal SIGUSR2 on gcn76.local.snellius.surf.nl: this job is timed-out.
+submitit INFO (2026-05-10 04:55:29,279) - Not checkpointing nor requeuing since I am a slave (procid=11).
+[INFO    ][2026-05-10 04:55:29][submitit            ][checkpoint_and_try_requeue] Not checkpointing nor requeuing since I am a slave (procid=11).
+submitit WARNING (2026-05-10 04:55:29,279) - Bypassing signal SIGCONT
+[WARNING ][2026-05-10 04:55:29][submitit            ][bypass                   ] Bypassing signal SIGCONT
+submitit WARNING (2026-05-10 04:55:29,547) - Bypassing signal SIGTERM
+[WARNING ][2026-05-10 04:55:29][submitit            ][bypass                   ] Bypassing signal SIGTERM
+submitit WARNING (2026-05-10 04:55:29,547) - Bypassing signal SIGCONT
+[WARNING ][2026-05-10 04:55:29][submitit            ][bypass                   ] Bypassing signal SIGCONT
+submitit INFO (2026-05-10 04:59:12,611) - Starting with JobEnvironment(job_id=22606429, hostname=gcn76.local.snellius.surf.nl, local_rank=3(4), node=2(4), global_rank=11(16))
+submitit INFO (2026-05-10 04:59:12,611) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/job_22606429/22606429_submitted.pkl
+INFO:root:loaded pretrain params...
+{   'app': 'vjepa',
+    'cpus_per_task': 16,
+    'data': {   'batch_size': 64,
+                'crop_size': 224,
+                'dataset_fpcs': [16, 16],
+                'dataset_type': 'VideoDataset',
+                'datasets': [   '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
+                                '/scratch-shared/dcanez/data/ssv2/train.csv'],
+                'datasets_weights': [0.65, 0.35],
+                'fps': 4,
+                'num_workers': 10,
+                'patch_size': 14,
+                'persistent_workers': True,
+                'pin_mem': True,
+                'stage': [   {   'dest': 'kinetics_240',
+                                 'format': 'targz_parts',
+                                 'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
+                             {   'dest': 'ssv2',
+                                 'format': 'multipart_tar',
+                                 'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
+                'tubelet_size': 1},
+    'data_aug': {   'auto_augment': False,
+                    'motion_shift': False,
+                    'random_resize_aspect_ratio': [0.75, 1.35],
+                    'random_resize_scale': [0.3, 1.0],
+                    'reprob': 0.0},
+    'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m',
+    'loss': {'loss_exp': 1.0},
+    'mask': [   {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 8,
+                    'spatial_scale': [0.15, 0.15],
+                    'temporal_scale': [1.0, 1.0]},
+                {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 2,
+                    'spatial_scale': [0.7, 0.7],
+                    'temporal_scale': [1.0, 1.0]}],
+    'mem_per_gpu': '180G',
+    'meta': {   'dtype': 'bfloat16',
+                'knn_eval_epoch0': False,
+                'knn_eval_freq': 5,
+                'knn_eval_presets': [   {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
+                                                          'eval_videos_per_class': 25,
+                                                          'num_workers': 8,
+                                                          'train_videos_per_class': 100},
+                                            'preset': 'ucf101'},
+                                        {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
+                                                          'eval_videos_per_class': 100,
+                                                          'linear_probe': True,
+                                                          'num_workers': 8,
+                                                          'pool_type': 'temporal_concat',
+                                                          'train_videos_per_class': 500},
+                                            'preset': 'ssv2_coarse10'}],
+                'load_checkpoint': True,
+                'read_checkpoint': None,
+                'save_every_freq': 5,
+                'seed': 239,
+                'use_sdpa': True,
+                'use_wandb': True,
+                'wandb_project': 'vjepa_ablation'},
+    'metrics': {'sigreg': {}, 'std': {}},
+    'model': {   'class_token': False,
+                 'freeze_backbone': True,
+                 'is_causal': False,
+                 'lora_alpha': 16,
+                 'lora_rank': 8,
+                 'lora_targets': ['attn.qkv', 'attn.proj'],
+                 'model_name': 'vit_large_patch14_capi_lvd1689m',
+                 'num_reg_tokens': 16,
+                 'pred_depth': 6,
+                 'pred_embed_dim': 384,
+                 'pred_num_heads': 12,
+                 'rope_type': 'capi2d',
+                 'stem_type': '2d',
+                 'target_kind': 'ema',
+                 'uniform_power': True,
+                 'use_activation_checkpointing': True,
+                 'use_lora': True,
+                 'use_mask_tokens': True,
+                 'use_rope': True,
+                 'use_sdpa': True,
+                 'zero_init_mask_tokens': True},
+    'nodes': 4,
+    'optimization': {   'ema': [0.99925, 0.99925],
+                        'epochs': 100,
+                        'final_lr': 0.0001,
+                        'final_weight_decay': 0.04,
+                        'ipe': 300,
+                        'ipe_scale': 1.0,
+                        'lr': 0.0005,
+                        'start_lr': 0.0001,
+                        'warmup': 10,
+                        'weight_decay': 0.04},
+    'tasks_per_node': 4}
+INFO:root:Running pre-training of app: vjepa
+[INFO    ][2026-05-10 04:59:17][app.vjepa.train     ][main                     ] which_dtype='bfloat16'
+[INFO    ][2026-05-10 04:59:17][app.vjepa.train     ][main                     ] Disabling persistent_workers (incompatible with KNN eval)
+[INFO    ][2026-05-10 04:59:17][app.vjepa.train     ][main                     ] NCCL_SOCKET_IFNAME=eno
+[INFO    ][2026-05-10 04:59:22][app.vjepa.train     ][main                     ] Initialized (rank/world-size) 11/16, tasks_per_node=4
+[INFO    ][2026-05-10 04:59:22][root                ][stage_datasets           ] [local_rank 3/4] Staging kinetics_240 (targz_parts)
+[INFO    ][2026-05-10 04:59:22][root                ][_stage_targz_parts       ] [rank 3] Extracting 25/103 tar.gz parts to /scratch-node/dcanez.22606429/kinetics_240
+[INFO    ][2026-05-10 04:59:36][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 2/25 parts
+[INFO    ][2026-05-10 04:59:50][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 4/25 parts
+[INFO    ][2026-05-10 05:00:04][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 6/25 parts
+[INFO    ][2026-05-10 05:00:19][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 8/25 parts
+[INFO    ][2026-05-10 05:00:33][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 10/25 parts
+[INFO    ][2026-05-10 05:00:47][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 12/25 parts
+[INFO    ][2026-05-10 05:01:01][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 14/25 parts
+[INFO    ][2026-05-10 05:01:15][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 16/25 parts
+[INFO    ][2026-05-10 05:01:29][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 18/25 parts
+[INFO    ][2026-05-10 05:01:44][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 20/25 parts
+[INFO    ][2026-05-10 05:01:58][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 22/25 parts
+[INFO    ][2026-05-10 05:02:12][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 24/25 parts
+[INFO    ][2026-05-10 05:02:19][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 25/25 parts
+[INFO    ][2026-05-10 05:02:19][root                ][stage_datasets           ] [local_rank 3/4] Staging ssv2 (multipart_tar)
+[INFO    ][2026-05-10 05:04:33][app.vjepa.train     ][main                     ] Staged datasets: ['/scratch-node/dcanez.22606429/kinetics_240/train.csv', '/scratch-node/dcanez.22606429/ssv2/train.csv']
+[INFO    ][2026-05-10 05:04:47][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
+[INFO    ][2026-05-10 05:04:51][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA applied: 48 modules replaced (rank=8, alpha=16, dropout=0.0, targets=['attn.qkv', 'attn.proj'], scope=blocks)
+[INFO    ][2026-05-10 05:04:51][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA trainable params: 1,179,648
+[INFO    ][2026-05-10 05:04:52][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-10 05:04:52][root                ][init_video_model         ] VJEPAPredictorMultiSeqWrapper(
+  (backbone): VisionTransformerPredictor(
+    (predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
+    (mask_tokens): ParameterList(
+        (0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+    )
+    (predictor_blocks): ModuleList(
+      (0-5): 6 x Block(
+        (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (attn): RoPEAttention(
+          (qkv): Linear(in_features=384, out_features=1152, bias=True)
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): Linear(in_features=384, out_features=384, bias=True)
+          (proj_drop): Dropout(p=0.0, inplace=False)
+        )
+        (drop_path): Identity()
+        (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (mlp): MLP(
+          (fc1): Linear(in_features=384, out_features=1536, bias=True)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=1536, out_features=384, bias=True)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+    (predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
+  )
+)
+[INFO    ][2026-05-10 05:04:52][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-10 05:04:52][root                ][init_video_model         ] Encoder number of parameters: 1179648
+[INFO    ][2026-05-10 05:04:52][root                ][init_video_model         ] Predictor number of parameters: 11436928
+[INFO    ][2026-05-10 05:04:52][root                ][init_video_model         ] Target encoder number of parameters: 1179648
+[INFO    ][2026-05-10 05:04:53][root                ][make_videodataset        ] VideoDataset dataset created
+[INFO    ][2026-05-10 05:04:53][WeightedSampler     ][__init__                 ] Using DistributedWeightedSampler with rank 11 / 16
+[INFO    ][2026-05-10 05:04:53][root                ][make_videodataset        ] VideoDataset unsupervised data loader created
+[INFO    ][2026-05-10 05:04:53][app.vjepa.train     ][main                     ] iterations per epoch/dataset length: 300/399
+[INFO    ][2026-05-10 05:04:53][app.vjepa.train     ][main                     ] Wrapping models in DDP (rank 11)...
+[INFO    ][2026-05-10 05:06:08][root                ][load_checkpoint          ] Loading checkpoint from /scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/latest.pt
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] loaded pretrained encoder from epoch 33 with msg: <All keys matched successfully>
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] loaded pretrained predictor from epoch 33 with msg: <All keys matched successfully>
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] loaded pretrained target encoder from epoch 33 with msg: <All keys matched successfully>
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] loaded optimizers from epoch 33
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] read-path: /scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/latest.pt
+[INFO    ][2026-05-10 05:06:09][app.vjepa.train     ][main                     ] Initializing loader...
+submitit WARNING (2026-05-10 09:26:31,193) - Bypassing signal SIGTERM
+[WARNING ][2026-05-10 09:26:31][submitit            ][bypass                   ] Bypassing signal SIGTERM
+submitit WARNING (2026-05-10 09:26:31,285) - Bypassing signal SIGCONT
+[WARNING ][2026-05-10 09:26:31][submitit            ][bypass                   ] Bypassing signal SIGCONT

vjepa/inflated/d003_lora_ema_capilvd1689m/job_22606429/22606429_12_log.err

@@ -0,0 +1,314 @@
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
+  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
+[rank12]:[W509 17:03:12.192440797 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
+  scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+submitit WARNING (2026-05-10 04:55:29,284) - Caught signal SIGUSR2 on gcn78.local.snellius.surf.nl: this job is timed-out.
+submitit WARNING (2026-05-10 04:55:29,284) - Bypassing signal SIGCONT
+submitit WARNING (2026-05-10 04:55:29,547) - Bypassing signal SIGTERM
+submitit WARNING (2026-05-10 04:55:29,558) - Bypassing signal SIGCONT
+[rank12]:[W510 04:55:34.254366026 TCPStore.cpp:125] [c10d] recvValue failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:57910, remote=[gcn74.local.snellius.surf.nl]:37129): Failed to recv, got 0 bytes. Connection was likely closed. Did the remote server shutdown or crash?
+Exception raised from recvBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:682 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x147175c34fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a3325d (0x1471b9dc325d in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x273 (0x1471b9dc11f3 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x147176ea3eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1472b630d164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1472c6d7f19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1472c6e04100 in /lib64/libc.so.6)
+
+[rank12]:[W510 04:55:34.359653283 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 12] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Failed to recv, got 0 bytes. Connection was likely closed. Did the remote server shutdown or crash?
+[rank12]:[W510 04:55:35.359864595 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:57910, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x147175c34fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1471b9dc26d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1471b9dc11cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x147176ea3eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1472b630d164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1472c6d7f19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1472c6e04100 in /lib64/libc.so.6)
+
+[rank12]:[W510 04:55:35.362837481 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 12] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank12]:[W510 04:55:36.362981274 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:57910, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x147175c34fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1471b9dc26d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1471b9dc11cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x147176ea3eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1472b630d164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1472c6d7f19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1472c6e04100 in /lib64/libc.so.6)
+
+[rank12]:[W510 04:55:36.365408892 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 12] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank12]:[W510 04:55:37.365546085 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:57910, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x147175c34fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1471b9dc26d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1471b9dc11cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x147176ea3eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1472b630d164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1472c6d7f19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1472c6e04100 in /lib64/libc.so.6)
+
+[rank12]:[W510 04:55:37.367818644 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 12] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank12]:[W510 04:55:38.367945707 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:57910, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x147175c34fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1471b9dc26d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1471b9dc11cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x147176ea3eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1472b630d164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1472c6d7f19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1472c6e04100 in /lib64/libc.so.6)
+
+[rank12]:[W510 04:55:38.370198576 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 12] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank12]:[W510 04:55:39.370327129 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:57910, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x147175c34fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1471b9dc26d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1471b9dc11cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x147176ea3eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1472b630d164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1472c6d7f19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1472c6e04100 in /lib64/libc.so.6)
+
+[rank12]:[W510 04:55:39.372612828 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 12] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank12]:[W510 04:55:40.372733250 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:57910, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x147175c34fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1471b9dc26d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1471b9dc11cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x147176ea3eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1472b630d164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1472c6d7f19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1472c6e04100 in /lib64/libc.so.6)
+
+[rank12]:[W510 04:55:40.375042829 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 12] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank12]:[W510 04:55:41.375164132 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:57910, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x147175c34fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1471b9dc26d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1471b9dc11cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x147176ea3eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1472b630d164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1472c6d7f19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1472c6e04100 in /lib64/libc.so.6)
+
+[rank12]:[W510 04:55:41.377393501 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 12] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank12]:[W510 04:55:42.377510274 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:57910, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x147175c34fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1471b9dc26d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1471b9dc11cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x147176ea3eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1472b630d164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1472c6d7f19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1472c6e04100 in /lib64/libc.so.6)
+
+[rank12]:[W510 04:55:42.379692544 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 12] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank12]:[W510 04:55:43.379808417 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:57910, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x147175c34fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1471b9dc26d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1471b9dc11cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x147176ea3eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1472b630d164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1472c6d7f19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1472c6e04100 in /lib64/libc.so.6)
+
+[rank12]:[W510 04:55:43.382024966 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 12] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank12]:[W510 04:55:44.382153389 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:57910, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x147175c34fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1471b9dc26d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1471b9dc11cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x147176ea3eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1472b630d164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1472c6d7f19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1472c6e04100 in /lib64/libc.so.6)
+
+[rank12]:[W510 04:55:44.384424358 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 12] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank12]:[W510 04:55:45.384548851 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:57910, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x147175c34fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1471b9dc26d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1471b9dc11cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x147176ea3eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1472b630d164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1472c6d7f19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1472c6e04100 in /lib64/libc.so.6)
+
+[rank12]:[W510 04:55:45.386780370 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 12] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank12]:[W510 04:55:46.386923453 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:57910, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x147175c34fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1471b9dc26d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1471b9dc11cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x147176ea3eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1472b630d164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1472c6d7f19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1472c6e04100 in /lib64/libc.so.6)
+
+[rank12]:[W510 04:55:46.389259112 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 12] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank12]:[W510 04:55:47.389385624 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:57910, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x147175c34fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1471b9dc26d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1471b9dc11cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x147176ea3eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1472b630d164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1472c6d7f19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1472c6e04100 in /lib64/libc.so.6)
+
+[rank12]:[W510 04:55:47.391558854 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 12] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank12]:[W510 04:55:48.391684417 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:57910, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x147175c34fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1471b9dc26d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1471b9dc11cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x147176ea3eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1472b630d164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1472c6d7f19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1472c6e04100 in /lib64/libc.so.6)
+
+[rank12]:[W510 04:55:48.394222215 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 12] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank12]:[W510 04:55:49.394351887 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:57910, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x147175c34fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1471b9dc26d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1471b9dc11cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x147176ea3eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1472b630d164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1472c6d7f19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1472c6e04100 in /lib64/libc.so.6)
+
+[rank12]:[W510 04:55:49.396690206 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 12] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank12]:[W510 04:55:50.396826689 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:57910, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x147175c34fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1471b9dc26d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1471b9dc11cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x147176ea3eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1472b630d164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1472c6d7f19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1472c6e04100 in /lib64/libc.so.6)
+
+[rank12]:[W510 04:55:50.399190517 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 12] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank12]:[W510 04:55:51.399314350 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:57910, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x147175c34fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1471b9dc26d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1471b9dc11cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x147176ea3eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1472b630d164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1472c6d7f19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1472c6e04100 in /lib64/libc.so.6)
+
+[rank12]:[W510 04:55:51.401509560 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 12] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank12]:[W510 04:55:52.401655042 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:57910, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x147175c34fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1471b9dc26d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1471b9dc11cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x147176ea3eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1472b630d164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1472c6d7f19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1472c6e04100 in /lib64/libc.so.6)
+
+[rank12]:[W510 04:55:52.403964861 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 12] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank12]:[W510 04:55:53.404086604 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:57910, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x147175c34fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1471b9dc26d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1471b9dc11cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x147176ea3eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1472b630d164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1472c6d7f19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1472c6e04100 in /lib64/libc.so.6)
+
+[rank12]:[W510 04:55:53.406365073 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 12] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank12]:[W510 04:55:54.406483026 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:57910, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x147175c34fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1471b9dc26d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1471b9dc11cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x147176ea3eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1472b630d164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1472c6d7f19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1472c6e04100 in /lib64/libc.so.6)
+
+[rank12]:[W510 04:55:54.408727965 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 12] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank12]:[W510 04:55:55.408847838 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:57910, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x147175c34fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1471b9dc26d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1471b9dc11cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x147176ea3eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1472b630d164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1472c6d7f19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1472c6e04100 in /lib64/libc.so.6)
+
+[rank12]:[W510 04:55:55.411035338 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 12] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank12]:[W510 04:55:56.411183670 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:57910, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x147175c34fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1471b9dc26d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1471b9dc11cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x147176ea3eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1472b630d164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1472c6d7f19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1472c6e04100 in /lib64/libc.so.6)
+
+[rank12]:[W510 04:55:56.413474859 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 12] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank12]:[W510 04:55:57.413605542 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:57910, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x147175c34fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1471b9dc26d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1471b9dc11cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x147176ea3eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1472b630d164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1472c6d7f19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1472c6e04100 in /lib64/libc.so.6)
+
+[rank12]:[W510 04:55:57.415944131 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 12] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank12]:[W510 04:55:58.416067494 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:57910, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x147175c34fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1471b9dc26d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1471b9dc11cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x147176ea3eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1472b630d164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1472c6d7f19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1472c6e04100 in /lib64/libc.so.6)
+
+[rank12]:[W510 04:55:58.418303283 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 12] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[2026-05-10T04:56:03.718] error: namespace_p_join: open failed for /slurm/22606429/.ns: No such file or directory
+[2026-05-10T04:56:03.718] error: namespace_g_join(JobId=22606429 SLUID=s8FXKR6KX9EK00): No such file or directory
+[2026-05-10T04:56:05.369] error: Failed to send MESSAGE_TASK_EXIT: Connection refused
+[2026-05-10T04:56:05.372] error: namespace_p_join: open failed for /slurm/22606429/.ns: No such file or directory
+[2026-05-10T04:56:05.372] error: namespace_g_join(JobId=22606429 SLUID=s8FXKR6KX9EK00): No such file or directory
+[2026-05-10T04:56:05.510] error: namespace_p_join: open failed for /slurm/22606429/.ns: No such file or directory
+[2026-05-10T04:56:05.510] error: namespace_g_join(JobId=22606429 SLUID=s8FXKR6KX9EK00): No such file or directory
+[2026-05-10T04:56:05.548] error: namespace_p_join: open failed for /slurm/22606429/.ns: No such file or directory
+[2026-05-10T04:56:05.549] error: namespace_g_join(JobId=22606429 SLUID=s8FXKR6KX9EK00): No such file or directory
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
+  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
+[rank12]:[W510 05:04:31.499237153 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
+  scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+submitit WARNING (2026-05-10 09:26:31,169) - Bypassing signal SIGTERM
+submitit WARNING (2026-05-10 09:26:31,170) - Bypassing signal SIGCONT
+[2026-05-10T09:27:03.099] error: namespace_p_join: open failed for /slurm/22606429/.ns: No such file or directory
+[2026-05-10T09:27:03.411] error: namespace_g_join(JobId=22606429 SLUID=s8FXKR7X2B6T00): No such file or directory
+[2026-05-10T09:27:05.069] error: Failed to send MESSAGE_TASK_EXIT: Connection refused
+[2026-05-10T09:27:05.072] error: namespace_p_join: open failed for /slurm/22606429/.ns: No such file or directory
+[2026-05-10T09:27:05.073] error: namespace_g_join(JobId=22606429 SLUID=s8FXKR7X2B6T00): No such file or directory
+[2026-05-10T09:27:06.710] error: Failed to send MESSAGE_TASK_EXIT: Connection refused
+[2026-05-10T09:27:07.306] error: namespace_p_join: open failed for /slurm/22606429/.ns: No such file or directory
+[2026-05-10T09:27:07.306] error: namespace_g_join(JobId=22606429 SLUID=s8FXKR7X2B6T00): No such file or directory
+[2026-05-10T09:27:07.446] error: namespace_p_join: open failed for /slurm/22606429/.ns: No such file or directory
+[2026-05-10T09:27:07.446] error: namespace_g_join(JobId=22606429 SLUID=s8FXKR7X2B6T00): No such file or directory

vjepa/inflated/d003_lora_ema_capilvd1689m/job_22606429/22606429_12_log.out

@@ -0,0 +1,530 @@
+submitit INFO (2026-05-09 16:57:36,055) - Starting with JobEnvironment(job_id=22606429, hostname=gcn78.local.snellius.surf.nl, local_rank=0(4), node=3(4), global_rank=12(16))
+submitit INFO (2026-05-09 16:57:36,055) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/job_22606429/22606429_submitted.pkl
+INFO:root:loaded pretrain params...
+{   'app': 'vjepa',
+    'cpus_per_task': 16,
+    'data': {   'batch_size': 64,
+                'crop_size': 224,
+                'dataset_fpcs': [16, 16],
+                'dataset_type': 'VideoDataset',
+                'datasets': [   '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
+                                '/scratch-shared/dcanez/data/ssv2/train.csv'],
+                'datasets_weights': [0.65, 0.35],
+                'fps': 4,
+                'num_workers': 10,
+                'patch_size': 14,
+                'persistent_workers': True,
+                'pin_mem': True,
+                'stage': [   {   'dest': 'kinetics_240',
+                                 'format': 'targz_parts',
+                                 'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
+                             {   'dest': 'ssv2',
+                                 'format': 'multipart_tar',
+                                 'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
+                'tubelet_size': 1},
+    'data_aug': {   'auto_augment': False,
+                    'motion_shift': False,
+                    'random_resize_aspect_ratio': [0.75, 1.35],
+                    'random_resize_scale': [0.3, 1.0],
+                    'reprob': 0.0},
+    'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m',
+    'loss': {'loss_exp': 1.0},
+    'mask': [   {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 8,
+                    'spatial_scale': [0.15, 0.15],
+                    'temporal_scale': [1.0, 1.0]},
+                {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 2,
+                    'spatial_scale': [0.7, 0.7],
+                    'temporal_scale': [1.0, 1.0]}],
+    'mem_per_gpu': '180G',
+    'meta': {   'dtype': 'bfloat16',
+                'knn_eval_epoch0': False,
+                'knn_eval_freq': 5,
+                'knn_eval_presets': [   {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
+                                                          'eval_videos_per_class': 25,
+                                                          'num_workers': 8,
+                                                          'train_videos_per_class': 100},
+                                            'preset': 'ucf101'},
+                                        {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
+                                                          'eval_videos_per_class': 100,
+                                                          'linear_probe': True,
+                                                          'num_workers': 8,
+                                                          'pool_type': 'temporal_concat',
+                                                          'train_videos_per_class': 500},
+                                            'preset': 'ssv2_coarse10'}],
+                'load_checkpoint': True,
+                'read_checkpoint': None,
+                'save_every_freq': 5,
+                'seed': 239,
+                'use_sdpa': True,
+                'use_wandb': True,
+                'wandb_project': 'vjepa_ablation'},
+    'metrics': {'sigreg': {}, 'std': {}},
+    'model': {   'class_token': False,
+                 'freeze_backbone': True,
+                 'is_causal': False,
+                 'lora_alpha': 16,
+                 'lora_rank': 8,
+                 'lora_targets': ['attn.qkv', 'attn.proj'],
+                 'model_name': 'vit_large_patch14_capi_lvd1689m',
+                 'num_reg_tokens': 16,
+                 'pred_depth': 6,
+                 'pred_embed_dim': 384,
+                 'pred_num_heads': 12,
+                 'rope_type': 'capi2d',
+                 'stem_type': '2d',
+                 'target_kind': 'ema',
+                 'uniform_power': True,
+                 'use_activation_checkpointing': True,
+                 'use_lora': True,
+                 'use_mask_tokens': True,
+                 'use_rope': True,
+                 'use_sdpa': True,
+                 'zero_init_mask_tokens': True},
+    'nodes': 4,
+    'optimization': {   'ema': [0.99925, 0.99925],
+                        'epochs': 100,
+                        'final_lr': 0.0001,
+                        'final_weight_decay': 0.04,
+                        'ipe': 300,
+                        'ipe_scale': 1.0,
+                        'lr': 0.0005,
+                        'start_lr': 0.0001,
+                        'warmup': 10,
+                        'weight_decay': 0.04},
+    'tasks_per_node': 4}
+INFO:root:Running pre-training of app: vjepa
+[INFO    ][2026-05-09 16:58:12][app.vjepa.train     ][main                     ] which_dtype='bfloat16'
+[INFO    ][2026-05-09 16:58:12][app.vjepa.train     ][main                     ] Disabling persistent_workers (incompatible with KNN eval)
+[INFO    ][2026-05-09 16:58:12][app.vjepa.train     ][main                     ] NCCL_SOCKET_IFNAME=eno
+[INFO    ][2026-05-09 16:59:12][app.vjepa.train     ][main                     ] Initialized (rank/world-size) 12/16, tasks_per_node=4
+[INFO    ][2026-05-09 16:59:12][root                ][stage_datasets           ] [local_rank 0/4] Staging kinetics_240 (targz_parts)
+[INFO    ][2026-05-09 16:59:12][root                ][_stage_targz_parts       ] [rank 0] Extracting 26/103 tar.gz parts to /scratch-node/dcanez.22606429/kinetics_240
+[INFO    ][2026-05-09 16:59:26][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 2/26 parts
+[INFO    ][2026-05-09 16:59:40][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 4/26 parts
+[INFO    ][2026-05-09 16:59:54][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 6/26 parts
+[INFO    ][2026-05-09 17:00:08][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 8/26 parts
+[INFO    ][2026-05-09 17:00:22][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 10/26 parts
+[INFO    ][2026-05-09 17:00:36][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 12/26 parts
+[INFO    ][2026-05-09 17:00:51][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 14/26 parts
+[INFO    ][2026-05-09 17:01:05][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 16/26 parts
+[INFO    ][2026-05-09 17:01:20][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 18/26 parts
+[INFO    ][2026-05-09 17:01:34][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 20/26 parts
+[INFO    ][2026-05-09 17:01:48][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 22/26 parts
+[INFO    ][2026-05-09 17:02:02][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 24/26 parts
+[INFO    ][2026-05-09 17:02:16][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 26/26 parts
+[INFO    ][2026-05-09 17:02:16][root                ][stage_datasets           ] [local_rank 0/4] Staging ssv2 (multipart_tar)
+[INFO    ][2026-05-09 17:02:16][root                ][_stage_multipart_tar     ] [rank 0] Extracting multipart tar (2 files) to /scratch-node/dcanez.22606429/ssv2
+[INFO    ][2026-05-09 17:03:30][root                ][stage_datasets           ] Data staging completed in 258.1s (4.3min)
+[INFO    ][2026-05-09 17:03:30][root                ][_rewrite_csv             ] Wrote local CSV: /scratch-node/dcanez.22606429/kinetics_240/train.csv (239789 entries)
+[INFO    ][2026-05-09 17:03:31][root                ][_rewrite_csv             ] Wrote local CSV: /scratch-node/dcanez.22606429/ssv2/train.csv (168913 entries)
+[INFO    ][2026-05-09 17:03:31][app.vjepa.train     ][main                     ] Staged datasets: ['/scratch-node/dcanez.22606429/kinetics_240/train.csv', '/scratch-node/dcanez.22606429/ssv2/train.csv']
+[INFO    ][2026-05-09 17:03:51][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
+[INFO    ][2026-05-09 17:03:53][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA applied: 48 modules replaced (rank=8, alpha=16, dropout=0.0, targets=['attn.qkv', 'attn.proj'], scope=blocks)
+[INFO    ][2026-05-09 17:03:53][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA trainable params: 1,179,648
+[INFO    ][2026-05-09 17:03:53][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-09 17:03:53][root                ][init_video_model         ] VJEPAPredictorMultiSeqWrapper(
+  (backbone): VisionTransformerPredictor(
+    (predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
+    (mask_tokens): ParameterList(
+        (0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+    )
+    (predictor_blocks): ModuleList(
+      (0-5): 6 x Block(
+        (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (attn): RoPEAttention(
+          (qkv): Linear(in_features=384, out_features=1152, bias=True)
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): Linear(in_features=384, out_features=384, bias=True)
+          (proj_drop): Dropout(p=0.0, inplace=False)
+        )
+        (drop_path): Identity()
+        (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (mlp): MLP(
+          (fc1): Linear(in_features=384, out_features=1536, bias=True)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=1536, out_features=384, bias=True)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+    (predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
+  )
+)
+[INFO    ][2026-05-09 17:03:53][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-09 17:03:53][root                ][init_video_model         ] Encoder number of parameters: 1179648
+[INFO    ][2026-05-09 17:03:53][root                ][init_video_model         ] Predictor number of parameters: 11436928
+[INFO    ][2026-05-09 17:03:53][root                ][init_video_model         ] Target encoder number of parameters: 1179648
+[INFO    ][2026-05-09 17:03:58][root                ][make_videodataset        ] VideoDataset dataset created
+[INFO    ][2026-05-09 17:03:58][WeightedSampler     ][__init__                 ] Using DistributedWeightedSampler with rank 12 / 16
+[INFO    ][2026-05-09 17:03:58][root                ][make_videodataset        ] VideoDataset unsupervised data loader created
+[INFO    ][2026-05-09 17:03:58][app.vjepa.train     ][main                     ] iterations per epoch/dataset length: 300/399
+[INFO    ][2026-05-09 17:03:59][app.vjepa.train     ][main                     ] Wrapping models in DDP (rank 12)...
+[INFO    ][2026-05-09 17:05:26][app.vjepa.train     ][main                     ] Initializing loader...
+submitit INFO (2026-05-10 04:55:29,283) - Job has timed out. Ran 718 minutes out of requested 720 minutes.
+[INFO    ][2026-05-10 04:55:29][submitit            ][has_timed_out            ] Job has timed out. Ran 718 minutes out of requested 720 minutes.
+submitit WARNING (2026-05-10 04:55:29,284) - Caught signal SIGUSR2 on gcn78.local.snellius.surf.nl: this job is timed-out.
+[WARNING ][2026-05-10 04:55:29][submitit            ][checkpoint_and_try_requeue] Caught signal SIGUSR2 on gcn78.local.snellius.surf.nl: this job is timed-out.
+submitit INFO (2026-05-10 04:55:29,284) - Not checkpointing nor requeuing since I am a slave (procid=12).
+[INFO    ][2026-05-10 04:55:29][submitit            ][checkpoint_and_try_requeue] Not checkpointing nor requeuing since I am a slave (procid=12).
+submitit WARNING (2026-05-10 04:55:29,284) - Bypassing signal SIGCONT
+[WARNING ][2026-05-10 04:55:29][submitit            ][bypass                   ] Bypassing signal SIGCONT
+submitit WARNING (2026-05-10 04:55:29,547) - Bypassing signal SIGTERM
+[WARNING ][2026-05-10 04:55:29][submitit            ][bypass                   ] Bypassing signal SIGTERM
+submitit WARNING (2026-05-10 04:55:29,558) - Bypassing signal SIGCONT
+[WARNING ][2026-05-10 04:55:29][submitit            ][bypass                   ] Bypassing signal SIGCONT
+submitit INFO (2026-05-10 04:59:17,804) - Starting with JobEnvironment(job_id=22606429, hostname=gcn77.local.snellius.surf.nl, local_rank=0(4), node=3(4), global_rank=12(16))
+submitit INFO (2026-05-10 04:59:17,813) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/job_22606429/22606429_submitted.pkl
+INFO:root:loaded pretrain params...
+{   'app': 'vjepa',
+    'cpus_per_task': 16,
+    'data': {   'batch_size': 64,
+                'crop_size': 224,
+                'dataset_fpcs': [16, 16],
+                'dataset_type': 'VideoDataset',
+                'datasets': [   '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
+                                '/scratch-shared/dcanez/data/ssv2/train.csv'],
+                'datasets_weights': [0.65, 0.35],
+                'fps': 4,
+                'num_workers': 10,
+                'patch_size': 14,
+                'persistent_workers': True,
+                'pin_mem': True,
+                'stage': [   {   'dest': 'kinetics_240',
+                                 'format': 'targz_parts',
+                                 'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
+                             {   'dest': 'ssv2',
+                                 'format': 'multipart_tar',
+                                 'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
+                'tubelet_size': 1},
+    'data_aug': {   'auto_augment': False,
+                    'motion_shift': False,
+                    'random_resize_aspect_ratio': [0.75, 1.35],
+                    'random_resize_scale': [0.3, 1.0],
+                    'reprob': 0.0},
+    'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m',
+    'loss': {'loss_exp': 1.0},
+    'mask': [   {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 8,
+                    'spatial_scale': [0.15, 0.15],
+                    'temporal_scale': [1.0, 1.0]},
+                {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 2,
+                    'spatial_scale': [0.7, 0.7],
+                    'temporal_scale': [1.0, 1.0]}],
+    'mem_per_gpu': '180G',
+    'meta': {   'dtype': 'bfloat16',
+                'knn_eval_epoch0': False,
+                'knn_eval_freq': 5,
+                'knn_eval_presets': [   {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
+                                                          'eval_videos_per_class': 25,
+                                                          'num_workers': 8,
+                                                          'train_videos_per_class': 100},
+                                            'preset': 'ucf101'},
+                                        {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
+                                                          'eval_videos_per_class': 100,
+                                                          'linear_probe': True,
+                                                          'num_workers': 8,
+                                                          'pool_type': 'temporal_concat',
+                                                          'train_videos_per_class': 500},
+                                            'preset': 'ssv2_coarse10'}],
+                'load_checkpoint': True,
+                'read_checkpoint': None,
+                'save_every_freq': 5,
+                'seed': 239,
+                'use_sdpa': True,
+                'use_wandb': True,
+                'wandb_project': 'vjepa_ablation'},
+    'metrics': {'sigreg': {}, 'std': {}},
+    'model': {   'class_token': False,
+                 'freeze_backbone': True,
+                 'is_causal': False,
+                 'lora_alpha': 16,
+                 'lora_rank': 8,
+                 'lora_targets': ['attn.qkv', 'attn.proj'],
+                 'model_name': 'vit_large_patch14_capi_lvd1689m',
+                 'num_reg_tokens': 16,
+                 'pred_depth': 6,
+                 'pred_embed_dim': 384,
+                 'pred_num_heads': 12,
+                 'rope_type': 'capi2d',
+                 'stem_type': '2d',
+                 'target_kind': 'ema',
+                 'uniform_power': True,
+                 'use_activation_checkpointing': True,
+                 'use_lora': True,
+                 'use_mask_tokens': True,
+                 'use_rope': True,
+                 'use_sdpa': True,
+                 'zero_init_mask_tokens': True},
+    'nodes': 4,
+    'optimization': {   'ema': [0.99925, 0.99925],
+                        'epochs': 100,
+                        'final_lr': 0.0001,
+                        'final_weight_decay': 0.04,
+                        'ipe': 300,
+                        'ipe_scale': 1.0,
+                        'lr': 0.0005,
+                        'start_lr': 0.0001,
+                        'warmup': 10,
+                        'weight_decay': 0.04},
+    'tasks_per_node': 4}
+INFO:root:Running pre-training of app: vjepa
+[INFO    ][2026-05-10 05:00:27][app.vjepa.train     ][main                     ] which_dtype='bfloat16'
+[INFO    ][2026-05-10 05:00:27][app.vjepa.train     ][main                     ] Disabling persistent_workers (incompatible with KNN eval)
+[INFO    ][2026-05-10 05:00:27][app.vjepa.train     ][main                     ] NCCL_SOCKET_IFNAME=eno
+[INFO    ][2026-05-10 05:00:30][app.vjepa.train     ][main                     ] Initialized (rank/world-size) 12/16, tasks_per_node=4
+[INFO    ][2026-05-10 05:00:32][root                ][stage_datasets           ] [local_rank 0/4] Staging kinetics_240 (targz_parts)
+[INFO    ][2026-05-10 05:00:32][root                ][_stage_targz_parts       ] [rank 0] Extracting 26/103 tar.gz parts to /scratch-node/dcanez.22606429/kinetics_240
+[INFO    ][2026-05-10 05:00:46][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 2/26 parts
+[INFO    ][2026-05-10 05:01:00][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 4/26 parts
+[INFO    ][2026-05-10 05:01:14][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 6/26 parts
+[INFO    ][2026-05-10 05:01:28][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 8/26 parts
+[INFO    ][2026-05-10 05:01:42][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 10/26 parts
+[INFO    ][2026-05-10 05:01:57][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 12/26 parts
+[INFO    ][2026-05-10 05:02:11][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 14/26 parts
+[INFO    ][2026-05-10 05:02:25][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 16/26 parts
+[INFO    ][2026-05-10 05:02:39][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 18/26 parts
+[INFO    ][2026-05-10 05:02:53][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 20/26 parts
+[INFO    ][2026-05-10 05:03:07][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 22/26 parts
+[INFO    ][2026-05-10 05:03:22][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 24/26 parts
+[INFO    ][2026-05-10 05:03:36][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 26/26 parts
+[INFO    ][2026-05-10 05:03:36][root                ][stage_datasets           ] [local_rank 0/4] Staging ssv2 (multipart_tar)
+[INFO    ][2026-05-10 05:03:36][root                ][_stage_multipart_tar     ] [rank 0] Extracting multipart tar (2 files) to /scratch-node/dcanez.22606429/ssv2
+[INFO    ][2026-05-10 05:04:32][root                ][stage_datasets           ] Data staging completed in 240.5s (4.0min)
+[INFO    ][2026-05-10 05:04:32][root                ][_rewrite_csv             ] Wrote local CSV: /scratch-node/dcanez.22606429/kinetics_240/train.csv (239789 entries)
+[INFO    ][2026-05-10 05:04:33][root                ][_rewrite_csv             ] Wrote local CSV: /scratch-node/dcanez.22606429/ssv2/train.csv (168913 entries)
+[INFO    ][2026-05-10 05:04:33][app.vjepa.train     ][main                     ] Staged datasets: ['/scratch-node/dcanez.22606429/kinetics_240/train.csv', '/scratch-node/dcanez.22606429/ssv2/train.csv']
+[INFO    ][2026-05-10 05:06:01][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
+[INFO    ][2026-05-10 05:06:05][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA applied: 48 modules replaced (rank=8, alpha=16, dropout=0.0, targets=['attn.qkv', 'attn.proj'], scope=blocks)
+[INFO    ][2026-05-10 05:06:05][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA trainable params: 1,179,648
+[INFO    ][2026-05-10 05:06:05][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-10 05:06:05][root                ][init_video_model         ] VJEPAPredictorMultiSeqWrapper(
+  (backbone): VisionTransformerPredictor(
+    (predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
+    (mask_tokens): ParameterList(
+        (0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+    )
+    (predictor_blocks): ModuleList(
+      (0-5): 6 x Block(
+        (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (attn): RoPEAttention(
+          (qkv): Linear(in_features=384, out_features=1152, bias=True)
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): Linear(in_features=384, out_features=384, bias=True)
+          (proj_drop): Dropout(p=0.0, inplace=False)
+        )
+        (drop_path): Identity()
+        (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (mlp): MLP(
+          (fc1): Linear(in_features=384, out_features=1536, bias=True)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=1536, out_features=384, bias=True)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+    (predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
+  )
+)
+[INFO    ][2026-05-10 05:06:05][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-10 05:06:05][root                ][init_video_model         ] Encoder number of parameters: 1179648
+[INFO    ][2026-05-10 05:06:05][root                ][init_video_model         ] Predictor number of parameters: 11436928
+[INFO    ][2026-05-10 05:06:05][root                ][init_video_model         ] Target encoder number of parameters: 1179648
+[INFO    ][2026-05-10 05:06:07][root                ][make_videodataset        ] VideoDataset dataset created
+[INFO    ][2026-05-10 05:06:07][WeightedSampler     ][__init__                 ] Using DistributedWeightedSampler with rank 12 / 16
+[INFO    ][2026-05-10 05:06:07][root                ][make_videodataset        ] VideoDataset unsupervised data loader created
+[INFO    ][2026-05-10 05:06:07][app.vjepa.train     ][main                     ] iterations per epoch/dataset length: 300/399
+[INFO    ][2026-05-10 05:06:07][app.vjepa.train     ][main                     ] Wrapping models in DDP (rank 12)...
+[INFO    ][2026-05-10 05:06:08][root                ][load_checkpoint          ] Loading checkpoint from /scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/latest.pt
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] loaded pretrained encoder from epoch 33 with msg: <All keys matched successfully>
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] loaded pretrained predictor from epoch 33 with msg: <All keys matched successfully>
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] loaded pretrained target encoder from epoch 33 with msg: <All keys matched successfully>
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] loaded optimizers from epoch 33
+[INFO    ][2026-05-10 05:06:10][root                ][load_checkpoint          ] read-path: /scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/latest.pt
+[INFO    ][2026-05-10 05:06:10][app.vjepa.train     ][main                     ] Initializing loader...
+submitit WARNING (2026-05-10 09:26:31,169) - Bypassing signal SIGTERM
+[WARNING ][2026-05-10 09:26:31][submitit            ][bypass                   ] Bypassing signal SIGTERM
+submitit WARNING (2026-05-10 09:26:31,170) - Bypassing signal SIGCONT
+[WARNING ][2026-05-10 09:26:31][submitit            ][bypass                   ] Bypassing signal SIGCONT

vjepa/inflated/d003_lora_ema_capilvd1689m/job_22606429/22606429_13_log.err

@@ -0,0 +1,294 @@
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
+  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
+[rank13]:[W509 17:02:16.000828039 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
+  scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+submitit WARNING (2026-05-10 04:55:29,473) - Caught signal SIGUSR2 on gcn78.local.snellius.surf.nl: this job is timed-out.
+submitit WARNING (2026-05-10 04:55:29,473) - Bypassing signal SIGTERM
+submitit WARNING (2026-05-10 04:55:29,473) - Bypassing signal SIGCONT
+[rank13]:[W510 04:55:34.254340396 TCPStore.cpp:125] [c10d] recvValue failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:47930, remote=[gcn74.local.snellius.surf.nl]:37129): Failed to recv, got 0 bytes. Connection was likely closed. Did the remote server shutdown or crash?
+Exception raised from recvBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:682 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x153b4cfd8fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a3325d (0x153b9116725d in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x273 (0x153b911651f3 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x153b4e247eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153c8d6b1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x153c9e12319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x153c9e1a8100 in /lib64/libc.so.6)
+
+[rank13]:[W510 04:55:34.359659353 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 13] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Failed to recv, got 0 bytes. Connection was likely closed. Did the remote server shutdown or crash?
+[rank13]:[W510 04:55:35.359865665 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:47930, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x153b4cfd8fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x153b911666d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x153b911651cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x153b4e247eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153c8d6b1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x153c9e12319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x153c9e1a8100 in /lib64/libc.so.6)
+
+[rank13]:[W510 04:55:35.362832181 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 13] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank13]:[W510 04:55:36.362966214 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:47930, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x153b4cfd8fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x153b911666d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x153b911651cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x153b4e247eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153c8d6b1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x153c9e12319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x153c9e1a8100 in /lib64/libc.so.6)
+
+[rank13]:[W510 04:55:36.365483102 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 13] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank13]:[W510 04:55:37.365628164 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:47930, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x153b4cfd8fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x153b911666d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x153b911651cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x153b4e247eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153c8d6b1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x153c9e12319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x153c9e1a8100 in /lib64/libc.so.6)
+
+[rank13]:[W510 04:55:37.367886814 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 13] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank13]:[W510 04:55:38.368018296 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:47930, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x153b4cfd8fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x153b911666d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x153b911651cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x153b4e247eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153c8d6b1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x153c9e12319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x153c9e1a8100 in /lib64/libc.so.6)
+
+[rank13]:[W510 04:55:38.370232666 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 13] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank13]:[W510 04:55:39.370356139 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:47930, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x153b4cfd8fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x153b911666d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x153b911651cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x153b4e247eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153c8d6b1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x153c9e12319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x153c9e1a8100 in /lib64/libc.so.6)
+
+[rank13]:[W510 04:55:39.372663437 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 13] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank13]:[W510 04:55:40.372788040 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:47930, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x153b4cfd8fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x153b911666d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x153b911651cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x153b4e247eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153c8d6b1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x153c9e12319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x153c9e1a8100 in /lib64/libc.so.6)
+
+[rank13]:[W510 04:55:40.375137979 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 13] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank13]:[W510 04:55:41.375262792 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:47930, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x153b4cfd8fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x153b911666d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x153b911651cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x153b4e247eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153c8d6b1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x153c9e12319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x153c9e1a8100 in /lib64/libc.so.6)
+
+[rank13]:[W510 04:55:41.377434101 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 13] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank13]:[W510 04:55:42.377559204 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:47930, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x153b4cfd8fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x153b911666d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x153b911651cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x153b4e247eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153c8d6b1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x153c9e12319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x153c9e1a8100 in /lib64/libc.so.6)
+
+[rank13]:[W510 04:55:42.379796853 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 13] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank13]:[W510 04:55:43.379920466 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:47930, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x153b4cfd8fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x153b911666d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x153b911651cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x153b4e247eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153c8d6b1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x153c9e12319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x153c9e1a8100 in /lib64/libc.so.6)
+
+[rank13]:[W510 04:55:43.382117406 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 13] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank13]:[W510 04:55:44.382244398 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:47930, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x153b4cfd8fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x153b911666d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x153b911651cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x153b4e247eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153c8d6b1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x153c9e12319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x153c9e1a8100 in /lib64/libc.so.6)
+
+[rank13]:[W510 04:55:44.384619667 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 13] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank13]:[W510 04:55:45.384760700 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:47930, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x153b4cfd8fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x153b911666d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x153b911651cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x153b4e247eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153c8d6b1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x153c9e12319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x153c9e1a8100 in /lib64/libc.so.6)
+
+[rank13]:[W510 04:55:45.387052649 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 13] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank13]:[W510 04:55:46.387187331 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:47930, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x153b4cfd8fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x153b911666d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x153b911651cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x153b4e247eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153c8d6b1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x153c9e12319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x153c9e1a8100 in /lib64/libc.so.6)
+
+[rank13]:[W510 04:55:46.389414401 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 13] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank13]:[W510 04:55:47.389542634 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:47930, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x153b4cfd8fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x153b911666d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x153b911651cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x153b4e247eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153c8d6b1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x153c9e12319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x153c9e1a8100 in /lib64/libc.so.6)
+
+[rank13]:[W510 04:55:47.391894452 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 13] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank13]:[W510 04:55:48.392032875 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:47930, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x153b4cfd8fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x153b911666d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x153b911651cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x153b4e247eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153c8d6b1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x153c9e12319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x153c9e1a8100 in /lib64/libc.so.6)
+
+[rank13]:[W510 04:55:48.394653273 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 13] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank13]:[W510 04:55:49.394778005 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:47930, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x153b4cfd8fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x153b911666d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x153b911651cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x153b4e247eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153c8d6b1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x153c9e12319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x153c9e1a8100 in /lib64/libc.so.6)
+
+[rank13]:[W510 04:55:49.397036555 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 13] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank13]:[W510 04:55:50.397148567 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:47930, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x153b4cfd8fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x153b911666d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x153b911651cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x153b4e247eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153c8d6b1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x153c9e12319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x153c9e1a8100 in /lib64/libc.so.6)
+
+[rank13]:[W510 04:55:50.399321327 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 13] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank13]:[W510 04:55:51.399434840 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:47930, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x153b4cfd8fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x153b911666d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x153b911651cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x153b4e247eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153c8d6b1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x153c9e12319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x153c9e1a8100 in /lib64/libc.so.6)
+
+[rank13]:[W510 04:55:51.401713189 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 13] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank13]:[W510 04:55:52.401825712 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:47930, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x153b4cfd8fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x153b911666d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x153b911651cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x153b4e247eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153c8d6b1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x153c9e12319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x153c9e1a8100 in /lib64/libc.so.6)
+
+[rank13]:[W510 04:55:52.403957621 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 13] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank13]:[W510 04:55:53.404070414 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:47930, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x153b4cfd8fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x153b911666d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x153b911651cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x153b4e247eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153c8d6b1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x153c9e12319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x153c9e1a8100 in /lib64/libc.so.6)
+
+[rank13]:[W510 04:55:53.406342833 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 13] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank13]:[W510 04:55:54.406473486 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:47930, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x153b4cfd8fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x153b911666d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x153b911651cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x153b4e247eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153c8d6b1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x153c9e12319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x153c9e1a8100 in /lib64/libc.so.6)
+
+[rank13]:[W510 04:55:54.408732735 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 13] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank13]:[W510 04:55:55.408865238 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:47930, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x153b4cfd8fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x153b911666d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x153b911651cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x153b4e247eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153c8d6b1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x153c9e12319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x153c9e1a8100 in /lib64/libc.so.6)
+
+[rank13]:[W510 04:55:55.411029068 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 13] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank13]:[W510 04:55:56.411184940 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:47930, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x153b4cfd8fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x153b911666d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x153b911651cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x153b4e247eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153c8d6b1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x153c9e12319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x153c9e1a8100 in /lib64/libc.so.6)
+
+[rank13]:[W510 04:55:56.497458818 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 13] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank13]:[W510 04:55:57.497627231 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:47930, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x153b4cfd8fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x153b911666d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x153b911651cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x153b4e247eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153c8d6b1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x153c9e12319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x153c9e1a8100 in /lib64/libc.so.6)
+
+[rank13]:[W510 04:55:57.500024499 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 13] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank13]:[W510 04:55:58.500158342 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:47930, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x153b4cfd8fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x153b911666d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x153b911651cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x153b4e247eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153c8d6b1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x153c9e12319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x153c9e1a8100 in /lib64/libc.so.6)
+
+[rank13]:[W510 04:55:58.502425421 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 13] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
+  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
+[rank13]:[W510 05:03:35.904124913 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
+  scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+submitit WARNING (2026-05-10 09:26:31,174) - Bypassing signal SIGTERM
+submitit WARNING (2026-05-10 09:26:31,175) - Bypassing signal SIGCONT

vjepa/inflated/d003_lora_ema_capilvd1689m/job_22606429/22606429_13_log.out

@@ -0,0 +1,520 @@
+submitit INFO (2026-05-09 16:57:36,055) - Starting with JobEnvironment(job_id=22606429, hostname=gcn78.local.snellius.surf.nl, local_rank=1(4), node=3(4), global_rank=13(16))
+submitit INFO (2026-05-09 16:57:36,055) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/job_22606429/22606429_submitted.pkl
+INFO:root:loaded pretrain params...
+{   'app': 'vjepa',
+    'cpus_per_task': 16,
+    'data': {   'batch_size': 64,
+                'crop_size': 224,
+                'dataset_fpcs': [16, 16],
+                'dataset_type': 'VideoDataset',
+                'datasets': [   '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
+                                '/scratch-shared/dcanez/data/ssv2/train.csv'],
+                'datasets_weights': [0.65, 0.35],
+                'fps': 4,
+                'num_workers': 10,
+                'patch_size': 14,
+                'persistent_workers': True,
+                'pin_mem': True,
+                'stage': [   {   'dest': 'kinetics_240',
+                                 'format': 'targz_parts',
+                                 'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
+                             {   'dest': 'ssv2',
+                                 'format': 'multipart_tar',
+                                 'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
+                'tubelet_size': 1},
+    'data_aug': {   'auto_augment': False,
+                    'motion_shift': False,
+                    'random_resize_aspect_ratio': [0.75, 1.35],
+                    'random_resize_scale': [0.3, 1.0],
+                    'reprob': 0.0},
+    'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m',
+    'loss': {'loss_exp': 1.0},
+    'mask': [   {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 8,
+                    'spatial_scale': [0.15, 0.15],
+                    'temporal_scale': [1.0, 1.0]},
+                {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 2,
+                    'spatial_scale': [0.7, 0.7],
+                    'temporal_scale': [1.0, 1.0]}],
+    'mem_per_gpu': '180G',
+    'meta': {   'dtype': 'bfloat16',
+                'knn_eval_epoch0': False,
+                'knn_eval_freq': 5,
+                'knn_eval_presets': [   {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
+                                                          'eval_videos_per_class': 25,
+                                                          'num_workers': 8,
+                                                          'train_videos_per_class': 100},
+                                            'preset': 'ucf101'},
+                                        {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
+                                                          'eval_videos_per_class': 100,
+                                                          'linear_probe': True,
+                                                          'num_workers': 8,
+                                                          'pool_type': 'temporal_concat',
+                                                          'train_videos_per_class': 500},
+                                            'preset': 'ssv2_coarse10'}],
+                'load_checkpoint': True,
+                'read_checkpoint': None,
+                'save_every_freq': 5,
+                'seed': 239,
+                'use_sdpa': True,
+                'use_wandb': True,
+                'wandb_project': 'vjepa_ablation'},
+    'metrics': {'sigreg': {}, 'std': {}},
+    'model': {   'class_token': False,
+                 'freeze_backbone': True,
+                 'is_causal': False,
+                 'lora_alpha': 16,
+                 'lora_rank': 8,
+                 'lora_targets': ['attn.qkv', 'attn.proj'],
+                 'model_name': 'vit_large_patch14_capi_lvd1689m',
+                 'num_reg_tokens': 16,
+                 'pred_depth': 6,
+                 'pred_embed_dim': 384,
+                 'pred_num_heads': 12,
+                 'rope_type': 'capi2d',
+                 'stem_type': '2d',
+                 'target_kind': 'ema',
+                 'uniform_power': True,
+                 'use_activation_checkpointing': True,
+                 'use_lora': True,
+                 'use_mask_tokens': True,
+                 'use_rope': True,
+                 'use_sdpa': True,
+                 'zero_init_mask_tokens': True},
+    'nodes': 4,
+    'optimization': {   'ema': [0.99925, 0.99925],
+                        'epochs': 100,
+                        'final_lr': 0.0001,
+                        'final_weight_decay': 0.04,
+                        'ipe': 300,
+                        'ipe_scale': 1.0,
+                        'lr': 0.0005,
+                        'start_lr': 0.0001,
+                        'warmup': 10,
+                        'weight_decay': 0.04},
+    'tasks_per_node': 4}
+INFO:root:Running pre-training of app: vjepa
+[INFO    ][2026-05-09 16:58:12][app.vjepa.train     ][main                     ] which_dtype='bfloat16'
+[INFO    ][2026-05-09 16:58:12][app.vjepa.train     ][main                     ] Disabling persistent_workers (incompatible with KNN eval)
+[INFO    ][2026-05-09 16:58:12][app.vjepa.train     ][main                     ] NCCL_SOCKET_IFNAME=eno
+[INFO    ][2026-05-09 16:59:15][app.vjepa.train     ][main                     ] Initialized (rank/world-size) 13/16, tasks_per_node=4
+[INFO    ][2026-05-09 16:59:15][root                ][stage_datasets           ] [local_rank 1/4] Staging kinetics_240 (targz_parts)
+[INFO    ][2026-05-09 16:59:15][root                ][_stage_targz_parts       ] [rank 1] Extracting 26/103 tar.gz parts to /scratch-node/dcanez.22606429/kinetics_240
+[INFO    ][2026-05-09 16:59:27][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 2/26 parts
+[INFO    ][2026-05-09 16:59:41][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 4/26 parts
+[INFO    ][2026-05-09 16:59:55][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 6/26 parts
+[INFO    ][2026-05-09 17:00:09][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 8/26 parts
+[INFO    ][2026-05-09 17:00:23][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 10/26 parts
+[INFO    ][2026-05-09 17:00:38][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 12/26 parts
+[INFO    ][2026-05-09 17:00:51][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 14/26 parts
+[INFO    ][2026-05-09 17:01:05][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 16/26 parts
+[INFO    ][2026-05-09 17:01:19][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 18/26 parts
+[INFO    ][2026-05-09 17:01:33][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 20/26 parts
+[INFO    ][2026-05-09 17:01:48][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 22/26 parts
+[INFO    ][2026-05-09 17:02:02][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 24/26 parts
+[INFO    ][2026-05-09 17:02:16][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 26/26 parts
+[INFO    ][2026-05-09 17:02:16][root                ][stage_datasets           ] [local_rank 1/4] Staging ssv2 (multipart_tar)
+[INFO    ][2026-05-09 17:03:31][app.vjepa.train     ][main                     ] Staged datasets: ['/scratch-node/dcanez.22606429/kinetics_240/train.csv', '/scratch-node/dcanez.22606429/ssv2/train.csv']
+[INFO    ][2026-05-09 17:03:52][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
+[INFO    ][2026-05-09 17:03:53][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA applied: 48 modules replaced (rank=8, alpha=16, dropout=0.0, targets=['attn.qkv', 'attn.proj'], scope=blocks)
+[INFO    ][2026-05-09 17:03:53][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA trainable params: 1,179,648
+[INFO    ][2026-05-09 17:03:53][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-09 17:03:53][root                ][init_video_model         ] VJEPAPredictorMultiSeqWrapper(
+  (backbone): VisionTransformerPredictor(
+    (predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
+    (mask_tokens): ParameterList(
+        (0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+    )
+    (predictor_blocks): ModuleList(
+      (0-5): 6 x Block(
+        (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (attn): RoPEAttention(
+          (qkv): Linear(in_features=384, out_features=1152, bias=True)
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): Linear(in_features=384, out_features=384, bias=True)
+          (proj_drop): Dropout(p=0.0, inplace=False)
+        )
+        (drop_path): Identity()
+        (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (mlp): MLP(
+          (fc1): Linear(in_features=384, out_features=1536, bias=True)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=1536, out_features=384, bias=True)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+    (predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
+  )
+)
+[INFO    ][2026-05-09 17:03:53][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-09 17:03:53][root                ][init_video_model         ] Encoder number of parameters: 1179648
+[INFO    ][2026-05-09 17:03:53][root                ][init_video_model         ] Predictor number of parameters: 11436928
+[INFO    ][2026-05-09 17:03:53][root                ][init_video_model         ] Target encoder number of parameters: 1179648
+[INFO    ][2026-05-09 17:04:03][root                ][make_videodataset        ] VideoDataset dataset created
+[INFO    ][2026-05-09 17:04:03][WeightedSampler     ][__init__                 ] Using DistributedWeightedSampler with rank 13 / 16
+[INFO    ][2026-05-09 17:04:03][root                ][make_videodataset        ] VideoDataset unsupervised data loader created
+[INFO    ][2026-05-09 17:04:03][app.vjepa.train     ][main                     ] iterations per epoch/dataset length: 300/399
+[INFO    ][2026-05-09 17:04:03][app.vjepa.train     ][main                     ] Wrapping models in DDP (rank 13)...
+[INFO    ][2026-05-09 17:05:26][app.vjepa.train     ][main                     ] Initializing loader...
+submitit INFO (2026-05-10 04:55:29,473) - Job has timed out. Ran 718 minutes out of requested 720 minutes.
+[INFO    ][2026-05-10 04:55:29][submitit            ][has_timed_out            ] Job has timed out. Ran 718 minutes out of requested 720 minutes.
+submitit WARNING (2026-05-10 04:55:29,473) - Caught signal SIGUSR2 on gcn78.local.snellius.surf.nl: this job is timed-out.
+[WARNING ][2026-05-10 04:55:29][submitit            ][checkpoint_and_try_requeue] Caught signal SIGUSR2 on gcn78.local.snellius.surf.nl: this job is timed-out.
+submitit INFO (2026-05-10 04:55:29,473) - Not checkpointing nor requeuing since I am a slave (procid=13).
+[INFO    ][2026-05-10 04:55:29][submitit            ][checkpoint_and_try_requeue] Not checkpointing nor requeuing since I am a slave (procid=13).
+submitit WARNING (2026-05-10 04:55:29,473) - Bypassing signal SIGTERM
+[WARNING ][2026-05-10 04:55:29][submitit            ][bypass                   ] Bypassing signal SIGTERM
+submitit WARNING (2026-05-10 04:55:29,473) - Bypassing signal SIGCONT
+[WARNING ][2026-05-10 04:55:29][submitit            ][bypass                   ] Bypassing signal SIGCONT
+submitit INFO (2026-05-10 04:59:17,804) - Starting with JobEnvironment(job_id=22606429, hostname=gcn77.local.snellius.surf.nl, local_rank=1(4), node=3(4), global_rank=13(16))
+submitit INFO (2026-05-10 04:59:17,820) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/job_22606429/22606429_submitted.pkl
+INFO:root:loaded pretrain params...
+{   'app': 'vjepa',
+    'cpus_per_task': 16,
+    'data': {   'batch_size': 64,
+                'crop_size': 224,
+                'dataset_fpcs': [16, 16],
+                'dataset_type': 'VideoDataset',
+                'datasets': [   '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
+                                '/scratch-shared/dcanez/data/ssv2/train.csv'],
+                'datasets_weights': [0.65, 0.35],
+                'fps': 4,
+                'num_workers': 10,
+                'patch_size': 14,
+                'persistent_workers': True,
+                'pin_mem': True,
+                'stage': [   {   'dest': 'kinetics_240',
+                                 'format': 'targz_parts',
+                                 'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
+                             {   'dest': 'ssv2',
+                                 'format': 'multipart_tar',
+                                 'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
+                'tubelet_size': 1},
+    'data_aug': {   'auto_augment': False,
+                    'motion_shift': False,
+                    'random_resize_aspect_ratio': [0.75, 1.35],
+                    'random_resize_scale': [0.3, 1.0],
+                    'reprob': 0.0},
+    'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m',
+    'loss': {'loss_exp': 1.0},
+    'mask': [   {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 8,
+                    'spatial_scale': [0.15, 0.15],
+                    'temporal_scale': [1.0, 1.0]},
+                {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 2,
+                    'spatial_scale': [0.7, 0.7],
+                    'temporal_scale': [1.0, 1.0]}],
+    'mem_per_gpu': '180G',
+    'meta': {   'dtype': 'bfloat16',
+                'knn_eval_epoch0': False,
+                'knn_eval_freq': 5,
+                'knn_eval_presets': [   {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
+                                                          'eval_videos_per_class': 25,
+                                                          'num_workers': 8,
+                                                          'train_videos_per_class': 100},
+                                            'preset': 'ucf101'},
+                                        {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
+                                                          'eval_videos_per_class': 100,
+                                                          'linear_probe': True,
+                                                          'num_workers': 8,
+                                                          'pool_type': 'temporal_concat',
+                                                          'train_videos_per_class': 500},
+                                            'preset': 'ssv2_coarse10'}],
+                'load_checkpoint': True,
+                'read_checkpoint': None,
+                'save_every_freq': 5,
+                'seed': 239,
+                'use_sdpa': True,
+                'use_wandb': True,
+                'wandb_project': 'vjepa_ablation'},
+    'metrics': {'sigreg': {}, 'std': {}},
+    'model': {   'class_token': False,
+                 'freeze_backbone': True,
+                 'is_causal': False,
+                 'lora_alpha': 16,
+                 'lora_rank': 8,
+                 'lora_targets': ['attn.qkv', 'attn.proj'],
+                 'model_name': 'vit_large_patch14_capi_lvd1689m',
+                 'num_reg_tokens': 16,
+                 'pred_depth': 6,
+                 'pred_embed_dim': 384,
+                 'pred_num_heads': 12,
+                 'rope_type': 'capi2d',
+                 'stem_type': '2d',
+                 'target_kind': 'ema',
+                 'uniform_power': True,
+                 'use_activation_checkpointing': True,
+                 'use_lora': True,
+                 'use_mask_tokens': True,
+                 'use_rope': True,
+                 'use_sdpa': True,
+                 'zero_init_mask_tokens': True},
+    'nodes': 4,
+    'optimization': {   'ema': [0.99925, 0.99925],
+                        'epochs': 100,
+                        'final_lr': 0.0001,
+                        'final_weight_decay': 0.04,
+                        'ipe': 300,
+                        'ipe_scale': 1.0,
+                        'lr': 0.0005,
+                        'start_lr': 0.0001,
+                        'warmup': 10,
+                        'weight_decay': 0.04},
+    'tasks_per_node': 4}
+INFO:root:Running pre-training of app: vjepa
+[INFO    ][2026-05-10 05:00:27][app.vjepa.train     ][main                     ] which_dtype='bfloat16'
+[INFO    ][2026-05-10 05:00:27][app.vjepa.train     ][main                     ] Disabling persistent_workers (incompatible with KNN eval)
+[INFO    ][2026-05-10 05:00:27][app.vjepa.train     ][main                     ] NCCL_SOCKET_IFNAME=eno
+[INFO    ][2026-05-10 05:00:30][app.vjepa.train     ][main                     ] Initialized (rank/world-size) 13/16, tasks_per_node=4
+[INFO    ][2026-05-10 05:00:32][root                ][stage_datasets           ] [local_rank 1/4] Staging kinetics_240 (targz_parts)
+[INFO    ][2026-05-10 05:00:32][root                ][_stage_targz_parts       ] [rank 1] Extracting 26/103 tar.gz parts to /scratch-node/dcanez.22606429/kinetics_240
+[INFO    ][2026-05-10 05:00:45][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 2/26 parts
+[INFO    ][2026-05-10 05:00:58][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 4/26 parts
+[INFO    ][2026-05-10 05:01:13][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 6/26 parts
+[INFO    ][2026-05-10 05:01:27][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 8/26 parts
+[INFO    ][2026-05-10 05:01:41][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 10/26 parts
+[INFO    ][2026-05-10 05:01:56][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 12/26 parts
+[INFO    ][2026-05-10 05:02:09][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 14/26 parts
+[INFO    ][2026-05-10 05:02:23][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 16/26 parts
+[INFO    ][2026-05-10 05:02:37][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 18/26 parts
+[INFO    ][2026-05-10 05:02:52][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 20/26 parts
+[INFO    ][2026-05-10 05:03:06][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 22/26 parts
+[INFO    ][2026-05-10 05:03:21][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 24/26 parts
+[INFO    ][2026-05-10 05:03:35][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 26/26 parts
+[INFO    ][2026-05-10 05:03:35][root                ][stage_datasets           ] [local_rank 1/4] Staging ssv2 (multipart_tar)
+[INFO    ][2026-05-10 05:04:33][app.vjepa.train     ][main                     ] Staged datasets: ['/scratch-node/dcanez.22606429/kinetics_240/train.csv', '/scratch-node/dcanez.22606429/ssv2/train.csv']
+[INFO    ][2026-05-10 05:06:01][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
+[INFO    ][2026-05-10 05:06:05][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA applied: 48 modules replaced (rank=8, alpha=16, dropout=0.0, targets=['attn.qkv', 'attn.proj'], scope=blocks)
+[INFO    ][2026-05-10 05:06:05][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA trainable params: 1,179,648
+[INFO    ][2026-05-10 05:06:05][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-10 05:06:05][root                ][init_video_model         ] VJEPAPredictorMultiSeqWrapper(
+  (backbone): VisionTransformerPredictor(
+    (predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
+    (mask_tokens): ParameterList(
+        (0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+    )
+    (predictor_blocks): ModuleList(
+      (0-5): 6 x Block(
+        (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (attn): RoPEAttention(
+          (qkv): Linear(in_features=384, out_features=1152, bias=True)
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): Linear(in_features=384, out_features=384, bias=True)
+          (proj_drop): Dropout(p=0.0, inplace=False)
+        )
+        (drop_path): Identity()
+        (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (mlp): MLP(
+          (fc1): Linear(in_features=384, out_features=1536, bias=True)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=1536, out_features=384, bias=True)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+    (predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
+  )
+)
+[INFO    ][2026-05-10 05:06:05][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-10 05:06:05][root                ][init_video_model         ] Encoder number of parameters: 1179648
+[INFO    ][2026-05-10 05:06:05][root                ][init_video_model         ] Predictor number of parameters: 11436928
+[INFO    ][2026-05-10 05:06:05][root                ][init_video_model         ] Target encoder number of parameters: 1179648
+[INFO    ][2026-05-10 05:06:07][root                ][make_videodataset        ] VideoDataset dataset created
+[INFO    ][2026-05-10 05:06:07][WeightedSampler     ][__init__                 ] Using DistributedWeightedSampler with rank 13 / 16
+[INFO    ][2026-05-10 05:06:07][root                ][make_videodataset        ] VideoDataset unsupervised data loader created
+[INFO    ][2026-05-10 05:06:07][app.vjepa.train     ][main                     ] iterations per epoch/dataset length: 300/399
+[INFO    ][2026-05-10 05:06:07][app.vjepa.train     ][main                     ] Wrapping models in DDP (rank 13)...
+[INFO    ][2026-05-10 05:06:08][root                ][load_checkpoint          ] Loading checkpoint from /scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/latest.pt
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] loaded pretrained encoder from epoch 33 with msg: <All keys matched successfully>
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] loaded pretrained predictor from epoch 33 with msg: <All keys matched successfully>
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] loaded pretrained target encoder from epoch 33 with msg: <All keys matched successfully>
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] loaded optimizers from epoch 33
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] read-path: /scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/latest.pt
+[INFO    ][2026-05-10 05:06:09][app.vjepa.train     ][main                     ] Initializing loader...
+submitit WARNING (2026-05-10 09:26:31,174) - Bypassing signal SIGTERM
+[WARNING ][2026-05-10 09:26:31][submitit            ][bypass                   ] Bypassing signal SIGTERM
+submitit WARNING (2026-05-10 09:26:31,175) - Bypassing signal SIGCONT
+[WARNING ][2026-05-10 09:26:31][submitit            ][bypass                   ] Bypassing signal SIGCONT

vjepa/inflated/d003_lora_ema_capilvd1689m/job_22606429/22606429_14_log.err

@@ -0,0 +1,294 @@
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
+  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
+[rank14]:[W509 17:02:02.877473887 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
+  scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+submitit WARNING (2026-05-10 04:55:29,481) - Caught signal SIGUSR2 on gcn78.local.snellius.surf.nl: this job is timed-out.
+submitit WARNING (2026-05-10 04:55:29,482) - Bypassing signal SIGTERM
+submitit WARNING (2026-05-10 04:55:29,482) - Bypassing signal SIGCONT
+[rank14]:[W510 04:55:34.723267926 TCPStore.cpp:125] [c10d] recvValue failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:40564, remote=[gcn74.local.snellius.surf.nl]:37129): Failed to recv, got 0 bytes. Connection was likely closed. Did the remote server shutdown or crash?
+Exception raised from recvBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:682 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14a82a19efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a3325d (0x14a86e32d25d in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x273 (0x14a86e32b1f3 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14a82b40deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14a96a877164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14a97b2e919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14a97b36e100 in /lib64/libc.so.6)
+
+[rank14]:[W510 04:55:34.726288942 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 14] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Failed to recv, got 0 bytes. Connection was likely closed. Did the remote server shutdown or crash?
+[rank14]:[W510 04:55:35.726465154 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:40564, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14a82a19efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14a86e32c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14a86e32b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14a82b40deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14a96a877164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14a97b2e919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14a97b36e100 in /lib64/libc.so.6)
+
+[rank14]:[W510 04:55:35.729843608 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 14] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank14]:[W510 04:55:36.730009561 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:40564, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14a82a19efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14a86e32c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14a86e32b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14a82b40deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14a96a877164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14a97b2e919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14a97b36e100 in /lib64/libc.so.6)
+
+[rank14]:[W510 04:55:36.732279350 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 14] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank14]:[W510 04:55:37.732431582 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:40564, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14a82a19efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14a86e32c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14a86e32b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14a82b40deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14a96a877164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14a97b2e919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14a97b36e100 in /lib64/libc.so.6)
+
+[rank14]:[W510 04:55:37.734767491 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 14] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank14]:[W510 04:55:38.734913404 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:40564, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14a82a19efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14a86e32c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14a86e32b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14a82b40deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14a96a877164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14a97b2e919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14a97b36e100 in /lib64/libc.so.6)
+
+[rank14]:[W510 04:55:38.737266602 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 14] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank14]:[W510 04:55:39.737413165 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:40564, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14a82a19efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14a86e32c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14a86e32b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14a82b40deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14a96a877164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14a97b2e919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14a97b36e100 in /lib64/libc.so.6)
+
+[rank14]:[W510 04:55:39.739727054 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 14] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank14]:[W510 04:55:40.739890467 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:40564, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14a82a19efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14a86e32c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14a86e32b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14a82b40deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14a96a877164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14a97b2e919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14a97b36e100 in /lib64/libc.so.6)
+
+[rank14]:[W510 04:55:40.742347045 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 14] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank14]:[W510 04:55:41.742497288 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:40564, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14a82a19efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14a86e32c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14a86e32b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14a82b40deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14a96a877164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14a97b2e919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14a97b36e100 in /lib64/libc.so.6)
+
+[rank14]:[W510 04:55:41.744815417 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 14] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank14]:[W510 04:55:42.744958699 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:40564, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14a82a19efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14a86e32c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14a86e32b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14a82b40deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14a96a877164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14a97b2e919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14a97b36e100 in /lib64/libc.so.6)
+
+[rank14]:[W510 04:55:42.747292698 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 14] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank14]:[W510 04:55:43.747433041 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:40564, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14a82a19efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14a86e32c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14a86e32b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14a82b40deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14a96a877164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14a97b2e919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14a97b36e100 in /lib64/libc.so.6)
+
+[rank14]:[W510 04:55:43.749757980 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 14] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank14]:[W510 04:55:44.749918802 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:40564, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14a82a19efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14a86e32c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14a86e32b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14a82b40deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14a96a877164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14a97b2e919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14a97b36e100 in /lib64/libc.so.6)
+
+[rank14]:[W510 04:55:44.752323341 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 14] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank14]:[W510 04:55:45.752472313 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:40564, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14a82a19efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14a86e32c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14a86e32b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14a82b40deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14a96a877164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14a97b2e919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14a97b36e100 in /lib64/libc.so.6)
+
+[rank14]:[W510 04:55:45.754797932 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 14] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank14]:[W510 04:55:46.754939075 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:40564, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14a82a19efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14a86e32c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14a86e32b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14a82b40deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14a96a877164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14a97b2e919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14a97b36e100 in /lib64/libc.so.6)
+
+[rank14]:[W510 04:55:46.757314144 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 14] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank14]:[W510 04:55:47.757465386 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:40564, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14a82a19efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14a86e32c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14a86e32b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14a82b40deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14a96a877164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14a97b2e919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14a97b36e100 in /lib64/libc.so.6)
+
+[rank14]:[W510 04:55:47.759865655 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 14] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank14]:[W510 04:55:48.759998707 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:40564, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14a82a19efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14a86e32c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14a86e32b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14a82b40deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14a96a877164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14a97b2e919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14a97b36e100 in /lib64/libc.so.6)
+
+[rank14]:[W510 04:55:48.762291376 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 14] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank14]:[W510 04:55:49.762434629 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:40564, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14a82a19efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14a86e32c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14a86e32b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14a82b40deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14a96a877164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14a97b2e919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14a97b36e100 in /lib64/libc.so.6)
+
+[rank14]:[W510 04:55:49.764926947 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 14] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank14]:[W510 04:55:50.765077670 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:40564, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14a82a19efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14a86e32c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14a86e32b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14a82b40deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14a96a877164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14a97b2e919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14a97b36e100 in /lib64/libc.so.6)
+
+[rank14]:[W510 04:55:50.767526508 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 14] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank14]:[W510 04:55:51.767670861 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:40564, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14a82a19efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14a86e32c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14a86e32b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14a82b40deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14a96a877164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14a97b2e919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14a97b36e100 in /lib64/libc.so.6)
+
+[rank14]:[W510 04:55:51.769967200 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 14] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank14]:[W510 04:55:52.770127462 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:40564, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14a82a19efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14a86e32c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14a86e32b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14a82b40deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14a96a877164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14a97b2e919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14a97b36e100 in /lib64/libc.so.6)
+
+[rank14]:[W510 04:55:52.772566971 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 14] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank14]:[W510 04:55:53.772712903 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:40564, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14a82a19efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14a86e32c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14a86e32b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14a82b40deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14a96a877164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14a97b2e919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14a97b36e100 in /lib64/libc.so.6)
+
+[rank14]:[W510 04:55:53.775055372 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 14] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank14]:[W510 04:55:54.775204305 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:40564, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14a82a19efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14a86e32c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14a86e32b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14a82b40deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14a96a877164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14a97b2e919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14a97b36e100 in /lib64/libc.so.6)
+
+[rank14]:[W510 04:55:54.777497594 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 14] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank14]:[W510 04:55:55.777647057 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:40564, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14a82a19efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14a86e32c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14a86e32b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14a82b40deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14a96a877164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14a97b2e919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14a97b36e100 in /lib64/libc.so.6)
+
+[rank14]:[W510 04:55:55.779943566 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 14] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank14]:[W510 04:55:56.780125168 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:40564, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14a82a19efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14a86e32c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14a86e32b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14a82b40deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14a96a877164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14a97b2e919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14a97b36e100 in /lib64/libc.so.6)
+
+[rank14]:[W510 04:55:56.782497677 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 14] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank14]:[W510 04:55:57.782644029 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:40564, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14a82a19efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14a86e32c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14a86e32b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14a82b40deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14a96a877164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14a97b2e919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14a97b36e100 in /lib64/libc.so.6)
+
+[rank14]:[W510 04:55:57.784993478 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 14] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank14]:[W510 04:55:58.785147701 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:40564, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14a82a19efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14a86e32c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14a86e32b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14a82b40deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14a96a877164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14a97b2e919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14a97b36e100 in /lib64/libc.so.6)
+
+[rank14]:[W510 04:55:58.787478040 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 14] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
+  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
+[rank14]:[W510 05:03:37.250195466 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
+  scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+submitit WARNING (2026-05-10 09:26:31,225) - Bypassing signal SIGTERM
+submitit WARNING (2026-05-10 09:26:31,225) - Bypassing signal SIGCONT

vjepa/inflated/d003_lora_ema_capilvd1689m/job_22606429/22606429_14_log.out

@@ -0,0 +1,520 @@
+submitit INFO (2026-05-09 16:57:36,055) - Starting with JobEnvironment(job_id=22606429, hostname=gcn78.local.snellius.surf.nl, local_rank=2(4), node=3(4), global_rank=14(16))
+submitit INFO (2026-05-09 16:57:36,055) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/job_22606429/22606429_submitted.pkl
+INFO:root:loaded pretrain params...
+{   'app': 'vjepa',
+    'cpus_per_task': 16,
+    'data': {   'batch_size': 64,
+                'crop_size': 224,
+                'dataset_fpcs': [16, 16],
+                'dataset_type': 'VideoDataset',
+                'datasets': [   '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
+                                '/scratch-shared/dcanez/data/ssv2/train.csv'],
+                'datasets_weights': [0.65, 0.35],
+                'fps': 4,
+                'num_workers': 10,
+                'patch_size': 14,
+                'persistent_workers': True,
+                'pin_mem': True,
+                'stage': [   {   'dest': 'kinetics_240',
+                                 'format': 'targz_parts',
+                                 'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
+                             {   'dest': 'ssv2',
+                                 'format': 'multipart_tar',
+                                 'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
+                'tubelet_size': 1},
+    'data_aug': {   'auto_augment': False,
+                    'motion_shift': False,
+                    'random_resize_aspect_ratio': [0.75, 1.35],
+                    'random_resize_scale': [0.3, 1.0],
+                    'reprob': 0.0},
+    'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m',
+    'loss': {'loss_exp': 1.0},
+    'mask': [   {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 8,
+                    'spatial_scale': [0.15, 0.15],
+                    'temporal_scale': [1.0, 1.0]},
+                {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 2,
+                    'spatial_scale': [0.7, 0.7],
+                    'temporal_scale': [1.0, 1.0]}],
+    'mem_per_gpu': '180G',
+    'meta': {   'dtype': 'bfloat16',
+                'knn_eval_epoch0': False,
+                'knn_eval_freq': 5,
+                'knn_eval_presets': [   {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
+                                                          'eval_videos_per_class': 25,
+                                                          'num_workers': 8,
+                                                          'train_videos_per_class': 100},
+                                            'preset': 'ucf101'},
+                                        {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
+                                                          'eval_videos_per_class': 100,
+                                                          'linear_probe': True,
+                                                          'num_workers': 8,
+                                                          'pool_type': 'temporal_concat',
+                                                          'train_videos_per_class': 500},
+                                            'preset': 'ssv2_coarse10'}],
+                'load_checkpoint': True,
+                'read_checkpoint': None,
+                'save_every_freq': 5,
+                'seed': 239,
+                'use_sdpa': True,
+                'use_wandb': True,
+                'wandb_project': 'vjepa_ablation'},
+    'metrics': {'sigreg': {}, 'std': {}},
+    'model': {   'class_token': False,
+                 'freeze_backbone': True,
+                 'is_causal': False,
+                 'lora_alpha': 16,
+                 'lora_rank': 8,
+                 'lora_targets': ['attn.qkv', 'attn.proj'],
+                 'model_name': 'vit_large_patch14_capi_lvd1689m',
+                 'num_reg_tokens': 16,
+                 'pred_depth': 6,
+                 'pred_embed_dim': 384,
+                 'pred_num_heads': 12,
+                 'rope_type': 'capi2d',
+                 'stem_type': '2d',
+                 'target_kind': 'ema',
+                 'uniform_power': True,
+                 'use_activation_checkpointing': True,
+                 'use_lora': True,
+                 'use_mask_tokens': True,
+                 'use_rope': True,
+                 'use_sdpa': True,
+                 'zero_init_mask_tokens': True},
+    'nodes': 4,
+    'optimization': {   'ema': [0.99925, 0.99925],
+                        'epochs': 100,
+                        'final_lr': 0.0001,
+                        'final_weight_decay': 0.04,
+                        'ipe': 300,
+                        'ipe_scale': 1.0,
+                        'lr': 0.0005,
+                        'start_lr': 0.0001,
+                        'warmup': 10,
+                        'weight_decay': 0.04},
+    'tasks_per_node': 4}
+INFO:root:Running pre-training of app: vjepa
+[INFO    ][2026-05-09 16:58:12][app.vjepa.train     ][main                     ] which_dtype='bfloat16'
+[INFO    ][2026-05-09 16:58:12][app.vjepa.train     ][main                     ] Disabling persistent_workers (incompatible with KNN eval)
+[INFO    ][2026-05-09 16:58:12][app.vjepa.train     ][main                     ] NCCL_SOCKET_IFNAME=eno
+[INFO    ][2026-05-09 16:58:57][app.vjepa.train     ][main                     ] Initialized (rank/world-size) 14/16, tasks_per_node=4
+[INFO    ][2026-05-09 16:58:59][root                ][stage_datasets           ] [local_rank 2/4] Staging kinetics_240 (targz_parts)
+[INFO    ][2026-05-09 16:58:59][root                ][_stage_targz_parts       ] [rank 2] Extracting 26/103 tar.gz parts to /scratch-node/dcanez.22606429/kinetics_240
+[INFO    ][2026-05-09 16:59:12][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 2/26 parts
+[INFO    ][2026-05-09 16:59:26][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 4/26 parts
+[INFO    ][2026-05-09 16:59:41][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 6/26 parts
+[INFO    ][2026-05-09 16:59:55][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 8/26 parts
+[INFO    ][2026-05-09 17:00:10][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 10/26 parts
+[INFO    ][2026-05-09 17:00:23][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 12/26 parts
+[INFO    ][2026-05-09 17:00:37][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 14/26 parts
+[INFO    ][2026-05-09 17:00:51][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 16/26 parts
+[INFO    ][2026-05-09 17:01:05][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 18/26 parts
+[INFO    ][2026-05-09 17:01:19][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 20/26 parts
+[INFO    ][2026-05-09 17:01:34][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 22/26 parts
+[INFO    ][2026-05-09 17:01:48][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 24/26 parts
+[INFO    ][2026-05-09 17:02:02][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 26/26 parts
+[INFO    ][2026-05-09 17:02:02][root                ][stage_datasets           ] [local_rank 2/4] Staging ssv2 (multipart_tar)
+[INFO    ][2026-05-09 17:03:31][app.vjepa.train     ][main                     ] Staged datasets: ['/scratch-node/dcanez.22606429/kinetics_240/train.csv', '/scratch-node/dcanez.22606429/ssv2/train.csv']
+[INFO    ][2026-05-09 17:03:54][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
+[INFO    ][2026-05-09 17:03:54][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA applied: 48 modules replaced (rank=8, alpha=16, dropout=0.0, targets=['attn.qkv', 'attn.proj'], scope=blocks)
+[INFO    ][2026-05-09 17:03:54][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA trainable params: 1,179,648
+[INFO    ][2026-05-09 17:03:54][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-09 17:03:54][root                ][init_video_model         ] VJEPAPredictorMultiSeqWrapper(
+  (backbone): VisionTransformerPredictor(
+    (predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
+    (mask_tokens): ParameterList(
+        (0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+    )
+    (predictor_blocks): ModuleList(
+      (0-5): 6 x Block(
+        (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (attn): RoPEAttention(
+          (qkv): Linear(in_features=384, out_features=1152, bias=True)
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): Linear(in_features=384, out_features=384, bias=True)
+          (proj_drop): Dropout(p=0.0, inplace=False)
+        )
+        (drop_path): Identity()
+        (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (mlp): MLP(
+          (fc1): Linear(in_features=384, out_features=1536, bias=True)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=1536, out_features=384, bias=True)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+    (predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
+  )
+)
+[INFO    ][2026-05-09 17:03:54][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-09 17:03:54][root                ][init_video_model         ] Encoder number of parameters: 1179648
+[INFO    ][2026-05-09 17:03:54][root                ][init_video_model         ] Predictor number of parameters: 11436928
+[INFO    ][2026-05-09 17:03:54][root                ][init_video_model         ] Target encoder number of parameters: 1179648
+[INFO    ][2026-05-09 17:04:03][root                ][make_videodataset        ] VideoDataset dataset created
+[INFO    ][2026-05-09 17:04:03][WeightedSampler     ][__init__                 ] Using DistributedWeightedSampler with rank 14 / 16
+[INFO    ][2026-05-09 17:04:03][root                ][make_videodataset        ] VideoDataset unsupervised data loader created
+[INFO    ][2026-05-09 17:04:03][app.vjepa.train     ][main                     ] iterations per epoch/dataset length: 300/399
+[INFO    ][2026-05-09 17:04:03][app.vjepa.train     ][main                     ] Wrapping models in DDP (rank 14)...
+[INFO    ][2026-05-09 17:05:26][app.vjepa.train     ][main                     ] Initializing loader...
+submitit INFO (2026-05-10 04:55:29,481) - Job has timed out. Ran 718 minutes out of requested 720 minutes.
+[INFO    ][2026-05-10 04:55:29][submitit            ][has_timed_out            ] Job has timed out. Ran 718 minutes out of requested 720 minutes.
+submitit WARNING (2026-05-10 04:55:29,481) - Caught signal SIGUSR2 on gcn78.local.snellius.surf.nl: this job is timed-out.
+[WARNING ][2026-05-10 04:55:29][submitit            ][checkpoint_and_try_requeue] Caught signal SIGUSR2 on gcn78.local.snellius.surf.nl: this job is timed-out.
+submitit INFO (2026-05-10 04:55:29,482) - Not checkpointing nor requeuing since I am a slave (procid=14).
+[INFO    ][2026-05-10 04:55:29][submitit            ][checkpoint_and_try_requeue] Not checkpointing nor requeuing since I am a slave (procid=14).
+submitit WARNING (2026-05-10 04:55:29,482) - Bypassing signal SIGTERM
+[WARNING ][2026-05-10 04:55:29][submitit            ][bypass                   ] Bypassing signal SIGTERM
+submitit WARNING (2026-05-10 04:55:29,482) - Bypassing signal SIGCONT
+[WARNING ][2026-05-10 04:55:29][submitit            ][bypass                   ] Bypassing signal SIGCONT
+submitit INFO (2026-05-10 04:59:17,804) - Starting with JobEnvironment(job_id=22606429, hostname=gcn77.local.snellius.surf.nl, local_rank=2(4), node=3(4), global_rank=14(16))
+submitit INFO (2026-05-10 04:59:17,820) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/job_22606429/22606429_submitted.pkl
+INFO:root:loaded pretrain params...
+{   'app': 'vjepa',
+    'cpus_per_task': 16,
+    'data': {   'batch_size': 64,
+                'crop_size': 224,
+                'dataset_fpcs': [16, 16],
+                'dataset_type': 'VideoDataset',
+                'datasets': [   '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
+                                '/scratch-shared/dcanez/data/ssv2/train.csv'],
+                'datasets_weights': [0.65, 0.35],
+                'fps': 4,
+                'num_workers': 10,
+                'patch_size': 14,
+                'persistent_workers': True,
+                'pin_mem': True,
+                'stage': [   {   'dest': 'kinetics_240',
+                                 'format': 'targz_parts',
+                                 'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
+                             {   'dest': 'ssv2',
+                                 'format': 'multipart_tar',
+                                 'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
+                'tubelet_size': 1},
+    'data_aug': {   'auto_augment': False,
+                    'motion_shift': False,
+                    'random_resize_aspect_ratio': [0.75, 1.35],
+                    'random_resize_scale': [0.3, 1.0],
+                    'reprob': 0.0},
+    'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m',
+    'loss': {'loss_exp': 1.0},
+    'mask': [   {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 8,
+                    'spatial_scale': [0.15, 0.15],
+                    'temporal_scale': [1.0, 1.0]},
+                {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 2,
+                    'spatial_scale': [0.7, 0.7],
+                    'temporal_scale': [1.0, 1.0]}],
+    'mem_per_gpu': '180G',
+    'meta': {   'dtype': 'bfloat16',
+                'knn_eval_epoch0': False,
+                'knn_eval_freq': 5,
+                'knn_eval_presets': [   {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
+                                                          'eval_videos_per_class': 25,
+                                                          'num_workers': 8,
+                                                          'train_videos_per_class': 100},
+                                            'preset': 'ucf101'},
+                                        {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
+                                                          'eval_videos_per_class': 100,
+                                                          'linear_probe': True,
+                                                          'num_workers': 8,
+                                                          'pool_type': 'temporal_concat',
+                                                          'train_videos_per_class': 500},
+                                            'preset': 'ssv2_coarse10'}],
+                'load_checkpoint': True,
+                'read_checkpoint': None,
+                'save_every_freq': 5,
+                'seed': 239,
+                'use_sdpa': True,
+                'use_wandb': True,
+                'wandb_project': 'vjepa_ablation'},
+    'metrics': {'sigreg': {}, 'std': {}},
+    'model': {   'class_token': False,
+                 'freeze_backbone': True,
+                 'is_causal': False,
+                 'lora_alpha': 16,
+                 'lora_rank': 8,
+                 'lora_targets': ['attn.qkv', 'attn.proj'],
+                 'model_name': 'vit_large_patch14_capi_lvd1689m',
+                 'num_reg_tokens': 16,
+                 'pred_depth': 6,
+                 'pred_embed_dim': 384,
+                 'pred_num_heads': 12,
+                 'rope_type': 'capi2d',
+                 'stem_type': '2d',
+                 'target_kind': 'ema',
+                 'uniform_power': True,
+                 'use_activation_checkpointing': True,
+                 'use_lora': True,
+                 'use_mask_tokens': True,
+                 'use_rope': True,
+                 'use_sdpa': True,
+                 'zero_init_mask_tokens': True},
+    'nodes': 4,
+    'optimization': {   'ema': [0.99925, 0.99925],
+                        'epochs': 100,
+                        'final_lr': 0.0001,
+                        'final_weight_decay': 0.04,
+                        'ipe': 300,
+                        'ipe_scale': 1.0,
+                        'lr': 0.0005,
+                        'start_lr': 0.0001,
+                        'warmup': 10,
+                        'weight_decay': 0.04},
+    'tasks_per_node': 4}
+INFO:root:Running pre-training of app: vjepa
+[INFO    ][2026-05-10 05:00:27][app.vjepa.train     ][main                     ] which_dtype='bfloat16'
+[INFO    ][2026-05-10 05:00:27][app.vjepa.train     ][main                     ] Disabling persistent_workers (incompatible with KNN eval)
+[INFO    ][2026-05-10 05:00:27][app.vjepa.train     ][main                     ] NCCL_SOCKET_IFNAME=eno
+[INFO    ][2026-05-10 05:00:31][app.vjepa.train     ][main                     ] Initialized (rank/world-size) 14/16, tasks_per_node=4
+[INFO    ][2026-05-10 05:00:32][root                ][stage_datasets           ] [local_rank 2/4] Staging kinetics_240 (targz_parts)
+[INFO    ][2026-05-10 05:00:32][root                ][_stage_targz_parts       ] [rank 2] Extracting 26/103 tar.gz parts to /scratch-node/dcanez.22606429/kinetics_240
+[INFO    ][2026-05-10 05:00:45][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 2/26 parts
+[INFO    ][2026-05-10 05:00:59][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 4/26 parts
+[INFO    ][2026-05-10 05:01:14][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 6/26 parts
+[INFO    ][2026-05-10 05:01:28][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 8/26 parts
+[INFO    ][2026-05-10 05:01:43][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 10/26 parts
+[INFO    ][2026-05-10 05:01:57][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 12/26 parts
+[INFO    ][2026-05-10 05:02:11][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 14/26 parts
+[INFO    ][2026-05-10 05:02:26][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 16/26 parts
+[INFO    ][2026-05-10 05:02:40][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 18/26 parts
+[INFO    ][2026-05-10 05:02:54][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 20/26 parts
+[INFO    ][2026-05-10 05:03:09][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 22/26 parts
+[INFO    ][2026-05-10 05:03:23][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 24/26 parts
+[INFO    ][2026-05-10 05:03:37][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 26/26 parts
+[INFO    ][2026-05-10 05:03:37][root                ][stage_datasets           ] [local_rank 2/4] Staging ssv2 (multipart_tar)
+[INFO    ][2026-05-10 05:04:33][app.vjepa.train     ][main                     ] Staged datasets: ['/scratch-node/dcanez.22606429/kinetics_240/train.csv', '/scratch-node/dcanez.22606429/ssv2/train.csv']
+[INFO    ][2026-05-10 05:06:01][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
+[INFO    ][2026-05-10 05:06:05][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA applied: 48 modules replaced (rank=8, alpha=16, dropout=0.0, targets=['attn.qkv', 'attn.proj'], scope=blocks)
+[INFO    ][2026-05-10 05:06:05][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA trainable params: 1,179,648
+[INFO    ][2026-05-10 05:06:05][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-10 05:06:05][root                ][init_video_model         ] VJEPAPredictorMultiSeqWrapper(
+  (backbone): VisionTransformerPredictor(
+    (predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
+    (mask_tokens): ParameterList(
+        (0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+    )
+    (predictor_blocks): ModuleList(
+      (0-5): 6 x Block(
+        (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (attn): RoPEAttention(
+          (qkv): Linear(in_features=384, out_features=1152, bias=True)
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): Linear(in_features=384, out_features=384, bias=True)
+          (proj_drop): Dropout(p=0.0, inplace=False)
+        )
+        (drop_path): Identity()
+        (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (mlp): MLP(
+          (fc1): Linear(in_features=384, out_features=1536, bias=True)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=1536, out_features=384, bias=True)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+    (predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
+  )
+)
+[INFO    ][2026-05-10 05:06:05][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-10 05:06:05][root                ][init_video_model         ] Encoder number of parameters: 1179648
+[INFO    ][2026-05-10 05:06:05][root                ][init_video_model         ] Predictor number of parameters: 11436928
+[INFO    ][2026-05-10 05:06:05][root                ][init_video_model         ] Target encoder number of parameters: 1179648
+[INFO    ][2026-05-10 05:06:07][root                ][make_videodataset        ] VideoDataset dataset created
+[INFO    ][2026-05-10 05:06:07][WeightedSampler     ][__init__                 ] Using DistributedWeightedSampler with rank 14 / 16
+[INFO    ][2026-05-10 05:06:07][root                ][make_videodataset        ] VideoDataset unsupervised data loader created
+[INFO    ][2026-05-10 05:06:07][app.vjepa.train     ][main                     ] iterations per epoch/dataset length: 300/399
+[INFO    ][2026-05-10 05:06:07][app.vjepa.train     ][main                     ] Wrapping models in DDP (rank 14)...
+[INFO    ][2026-05-10 05:06:08][root                ][load_checkpoint          ] Loading checkpoint from /scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/latest.pt
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] loaded pretrained encoder from epoch 33 with msg: <All keys matched successfully>
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] loaded pretrained predictor from epoch 33 with msg: <All keys matched successfully>
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] loaded pretrained target encoder from epoch 33 with msg: <All keys matched successfully>
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] loaded optimizers from epoch 33
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] read-path: /scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/latest.pt
+[INFO    ][2026-05-10 05:06:09][app.vjepa.train     ][main                     ] Initializing loader...
+submitit WARNING (2026-05-10 09:26:31,225) - Bypassing signal SIGTERM
+[WARNING ][2026-05-10 09:26:31][submitit            ][bypass                   ] Bypassing signal SIGTERM
+submitit WARNING (2026-05-10 09:26:31,225) - Bypassing signal SIGCONT
+[WARNING ][2026-05-10 09:26:31][submitit            ][bypass                   ] Bypassing signal SIGCONT

vjepa/inflated/d003_lora_ema_capilvd1689m/job_22606429/22606429_15_log.err

@@ -0,0 +1,295 @@
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
+  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
+[rank15]:[W509 17:02:01.928089011 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
+  scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+submitit WARNING (2026-05-10 04:55:29,468) - Caught signal SIGUSR2 on gcn78.local.snellius.surf.nl: this job is timed-out.
+submitit WARNING (2026-05-10 04:55:29,468) - Bypassing signal SIGCONT
+submitit WARNING (2026-05-10 04:55:29,471) - Bypassing signal SIGCONT
+submitit WARNING (2026-05-10 04:55:29,471) - Bypassing signal SIGTERM
+[rank15]:[W510 04:55:34.723225336 TCPStore.cpp:125] [c10d] recvValue failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:40572, remote=[gcn74.local.snellius.surf.nl]:37129): Failed to recv, got 0 bytes. Connection was likely closed. Did the remote server shutdown or crash?
+Exception raised from recvBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:682 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1468836fefdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a3325d (0x1468c788d25d in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x273 (0x1468c788b1f3 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14688496deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1469c3dd7164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1469d484919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1469d48ce100 in /lib64/libc.so.6)
+
+[rank15]:[W510 04:55:34.726045383 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 15] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Failed to recv, got 0 bytes. Connection was likely closed. Did the remote server shutdown or crash?
+[rank15]:[W510 04:55:35.726227275 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:40572, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1468836fefdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1468c788c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1468c788b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14688496deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1469c3dd7164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1469d484919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1469d48ce100 in /lib64/libc.so.6)
+
+[rank15]:[W510 04:55:35.729718119 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 15] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank15]:[W510 04:55:36.729870651 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:40572, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1468836fefdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1468c788c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1468c788b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14688496deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1469c3dd7164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1469d484919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1469d48ce100 in /lib64/libc.so.6)
+
+[rank15]:[W510 04:55:36.732268510 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 15] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank15]:[W510 04:55:37.732424132 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:40572, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1468836fefdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1468c788c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1468c788b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14688496deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1469c3dd7164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1469d484919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1469d48ce100 in /lib64/libc.so.6)
+
+[rank15]:[W510 04:55:37.734840831 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 15] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank15]:[W510 04:55:38.734990193 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:40572, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1468836fefdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1468c788c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1468c788b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14688496deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1469c3dd7164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1469d484919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1469d48ce100 in /lib64/libc.so.6)
+
+[rank15]:[W510 04:55:38.737330172 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 15] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank15]:[W510 04:55:39.737477365 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:40572, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1468836fefdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1468c788c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1468c788b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14688496deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1469c3dd7164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1469d484919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1469d48ce100 in /lib64/libc.so.6)
+
+[rank15]:[W510 04:55:39.739774944 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 15] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank15]:[W510 04:55:40.739915757 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:40572, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1468836fefdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1468c788c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1468c788b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14688496deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1469c3dd7164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1469d484919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1469d48ce100 in /lib64/libc.so.6)
+
+[rank15]:[W510 04:55:40.742218446 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 15] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank15]:[W510 04:55:41.742347918 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:40572, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1468836fefdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1468c788c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1468c788b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14688496deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1469c3dd7164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1469d484919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1469d48ce100 in /lib64/libc.so.6)
+
+[rank15]:[W510 04:55:41.744715227 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 15] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank15]:[W510 04:55:42.744852740 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:40572, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1468836fefdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1468c788c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1468c788b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14688496deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1469c3dd7164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1469d484919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1469d48ce100 in /lib64/libc.so.6)
+
+[rank15]:[W510 04:55:42.747334498 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 15] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank15]:[W510 04:55:43.747474260 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:40572, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1468836fefdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1468c788c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1468c788b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14688496deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1469c3dd7164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1469d484919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1469d48ce100 in /lib64/libc.so.6)
+
+[rank15]:[W510 04:55:43.749885789 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 15] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank15]:[W510 04:55:44.750032682 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:40572, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1468836fefdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1468c788c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1468c788b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14688496deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1469c3dd7164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1469d484919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1469d48ce100 in /lib64/libc.so.6)
+
+[rank15]:[W510 04:55:44.752310031 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 15] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank15]:[W510 04:55:45.752446053 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:40572, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1468836fefdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1468c788c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1468c788b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14688496deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1469c3dd7164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1469d484919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1469d48ce100 in /lib64/libc.so.6)
+
+[rank15]:[W510 04:55:45.754728723 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 15] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank15]:[W510 04:55:46.754866075 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:40572, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1468836fefdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1468c788c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1468c788b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14688496deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1469c3dd7164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1469d484919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1469d48ce100 in /lib64/libc.so.6)
+
+[rank15]:[W510 04:55:46.757182934 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 15] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank15]:[W510 04:55:47.757325747 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:40572, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1468836fefdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1468c788c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1468c788b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14688496deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1469c3dd7164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1469d484919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1469d48ce100 in /lib64/libc.so.6)
+
+[rank15]:[W510 04:55:47.759693135 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 15] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank15]:[W510 04:55:48.759820478 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:40572, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1468836fefdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1468c788c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1468c788b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14688496deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1469c3dd7164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1469d484919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1469d48ce100 in /lib64/libc.so.6)
+
+[rank15]:[W510 04:55:48.762140947 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 15] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank15]:[W510 04:55:49.762275270 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:40572, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1468836fefdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1468c788c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1468c788b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14688496deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1469c3dd7164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1469d484919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1469d48ce100 in /lib64/libc.so.6)
+
+[rank15]:[W510 04:55:49.764675798 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 15] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank15]:[W510 04:55:50.764812791 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:40572, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1468836fefdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1468c788c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1468c788b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14688496deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1469c3dd7164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1469d484919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1469d48ce100 in /lib64/libc.so.6)
+
+[rank15]:[W510 04:55:50.767140340 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 15] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank15]:[W510 04:55:51.767280053 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:40572, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1468836fefdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1468c788c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1468c788b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14688496deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1469c3dd7164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1469d484919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1469d48ce100 in /lib64/libc.so.6)
+
+[rank15]:[W510 04:55:51.769578542 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 15] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank15]:[W510 04:55:52.769726354 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:40572, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1468836fefdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1468c788c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1468c788b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14688496deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1469c3dd7164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1469d484919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1469d48ce100 in /lib64/libc.so.6)
+
+[rank15]:[W510 04:55:52.772073853 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 15] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank15]:[W510 04:55:53.772211936 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:40572, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1468836fefdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1468c788c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1468c788b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14688496deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1469c3dd7164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1469d484919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1469d48ce100 in /lib64/libc.so.6)
+
+[rank15]:[W510 04:55:53.774506635 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 15] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank15]:[W510 04:55:54.774643167 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:40572, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1468836fefdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1468c788c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1468c788b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14688496deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1469c3dd7164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1469d484919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1469d48ce100 in /lib64/libc.so.6)
+
+[rank15]:[W510 04:55:54.776901687 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 15] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank15]:[W510 04:55:55.777056129 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:40572, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1468836fefdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1468c788c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1468c788b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14688496deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1469c3dd7164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1469d484919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1469d48ce100 in /lib64/libc.so.6)
+
+[rank15]:[W510 04:55:55.779378008 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 15] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank15]:[W510 04:55:56.779548001 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:40572, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1468836fefdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1468c788c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1468c788b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14688496deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1469c3dd7164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1469d484919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1469d48ce100 in /lib64/libc.so.6)
+
+[rank15]:[W510 04:55:56.781905680 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 15] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank15]:[W510 04:55:57.782101242 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:40572, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1468836fefdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1468c788c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1468c788b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14688496deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1469c3dd7164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1469d484919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1469d48ce100 in /lib64/libc.so.6)
+
+[rank15]:[W510 04:55:57.784313601 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 15] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank15]:[W510 04:55:58.784453844 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn78.local.snellius.surf.nl]:40572, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1468836fefdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1468c788c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1468c788b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14688496deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1469c3dd7164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1469d484919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1469d48ce100 in /lib64/libc.so.6)
+
+[rank15]:[W510 04:55:58.786761633 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 15] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
+  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
+[rank15]:[W510 05:03:32.621951061 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
+  scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+submitit WARNING (2026-05-10 09:26:31,216) - Bypassing signal SIGTERM
+submitit WARNING (2026-05-10 09:26:31,216) - Bypassing signal SIGCONT

vjepa/inflated/d003_lora_ema_capilvd1689m/job_22606429/22606429_15_log.out

@@ -0,0 +1,522 @@
+submitit INFO (2026-05-09 16:57:36,055) - Starting with JobEnvironment(job_id=22606429, hostname=gcn78.local.snellius.surf.nl, local_rank=3(4), node=3(4), global_rank=15(16))
+submitit INFO (2026-05-09 16:57:36,055) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/job_22606429/22606429_submitted.pkl
+INFO:root:loaded pretrain params...
+{   'app': 'vjepa',
+    'cpus_per_task': 16,
+    'data': {   'batch_size': 64,
+                'crop_size': 224,
+                'dataset_fpcs': [16, 16],
+                'dataset_type': 'VideoDataset',
+                'datasets': [   '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
+                                '/scratch-shared/dcanez/data/ssv2/train.csv'],
+                'datasets_weights': [0.65, 0.35],
+                'fps': 4,
+                'num_workers': 10,
+                'patch_size': 14,
+                'persistent_workers': True,
+                'pin_mem': True,
+                'stage': [   {   'dest': 'kinetics_240',
+                                 'format': 'targz_parts',
+                                 'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
+                             {   'dest': 'ssv2',
+                                 'format': 'multipart_tar',
+                                 'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
+                'tubelet_size': 1},
+    'data_aug': {   'auto_augment': False,
+                    'motion_shift': False,
+                    'random_resize_aspect_ratio': [0.75, 1.35],
+                    'random_resize_scale': [0.3, 1.0],
+                    'reprob': 0.0},
+    'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m',
+    'loss': {'loss_exp': 1.0},
+    'mask': [   {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 8,
+                    'spatial_scale': [0.15, 0.15],
+                    'temporal_scale': [1.0, 1.0]},
+                {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 2,
+                    'spatial_scale': [0.7, 0.7],
+                    'temporal_scale': [1.0, 1.0]}],
+    'mem_per_gpu': '180G',
+    'meta': {   'dtype': 'bfloat16',
+                'knn_eval_epoch0': False,
+                'knn_eval_freq': 5,
+                'knn_eval_presets': [   {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
+                                                          'eval_videos_per_class': 25,
+                                                          'num_workers': 8,
+                                                          'train_videos_per_class': 100},
+                                            'preset': 'ucf101'},
+                                        {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
+                                                          'eval_videos_per_class': 100,
+                                                          'linear_probe': True,
+                                                          'num_workers': 8,
+                                                          'pool_type': 'temporal_concat',
+                                                          'train_videos_per_class': 500},
+                                            'preset': 'ssv2_coarse10'}],
+                'load_checkpoint': True,
+                'read_checkpoint': None,
+                'save_every_freq': 5,
+                'seed': 239,
+                'use_sdpa': True,
+                'use_wandb': True,
+                'wandb_project': 'vjepa_ablation'},
+    'metrics': {'sigreg': {}, 'std': {}},
+    'model': {   'class_token': False,
+                 'freeze_backbone': True,
+                 'is_causal': False,
+                 'lora_alpha': 16,
+                 'lora_rank': 8,
+                 'lora_targets': ['attn.qkv', 'attn.proj'],
+                 'model_name': 'vit_large_patch14_capi_lvd1689m',
+                 'num_reg_tokens': 16,
+                 'pred_depth': 6,
+                 'pred_embed_dim': 384,
+                 'pred_num_heads': 12,
+                 'rope_type': 'capi2d',
+                 'stem_type': '2d',
+                 'target_kind': 'ema',
+                 'uniform_power': True,
+                 'use_activation_checkpointing': True,
+                 'use_lora': True,
+                 'use_mask_tokens': True,
+                 'use_rope': True,
+                 'use_sdpa': True,
+                 'zero_init_mask_tokens': True},
+    'nodes': 4,
+    'optimization': {   'ema': [0.99925, 0.99925],
+                        'epochs': 100,
+                        'final_lr': 0.0001,
+                        'final_weight_decay': 0.04,
+                        'ipe': 300,
+                        'ipe_scale': 1.0,
+                        'lr': 0.0005,
+                        'start_lr': 0.0001,
+                        'warmup': 10,
+                        'weight_decay': 0.04},
+    'tasks_per_node': 4}
+INFO:root:Running pre-training of app: vjepa
+[INFO    ][2026-05-09 16:58:12][app.vjepa.train     ][main                     ] which_dtype='bfloat16'
+[INFO    ][2026-05-09 16:58:12][app.vjepa.train     ][main                     ] Disabling persistent_workers (incompatible with KNN eval)
+[INFO    ][2026-05-09 16:58:12][app.vjepa.train     ][main                     ] NCCL_SOCKET_IFNAME=eno
+[INFO    ][2026-05-09 16:59:01][app.vjepa.train     ][main                     ] Initialized (rank/world-size) 15/16, tasks_per_node=4
+[INFO    ][2026-05-09 16:59:01][root                ][stage_datasets           ] [local_rank 3/4] Staging kinetics_240 (targz_parts)
+[INFO    ][2026-05-09 16:59:01][root                ][_stage_targz_parts       ] [rank 3] Extracting 25/103 tar.gz parts to /scratch-node/dcanez.22606429/kinetics_240
+[INFO    ][2026-05-09 16:59:15][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 2/25 parts
+[INFO    ][2026-05-09 16:59:30][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 4/25 parts
+[INFO    ][2026-05-09 16:59:44][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 6/25 parts
+[INFO    ][2026-05-09 16:59:58][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 8/25 parts
+[INFO    ][2026-05-09 17:00:12][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 10/25 parts
+[INFO    ][2026-05-09 17:00:27][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 12/25 parts
+[INFO    ][2026-05-09 17:00:41][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 14/25 parts
+[INFO    ][2026-05-09 17:00:55][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 16/25 parts
+[INFO    ][2026-05-09 17:01:09][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 18/25 parts
+[INFO    ][2026-05-09 17:01:24][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 20/25 parts
+[INFO    ][2026-05-09 17:01:38][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 22/25 parts
+[INFO    ][2026-05-09 17:01:52][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 24/25 parts
+[INFO    ][2026-05-09 17:01:59][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 25/25 parts
+[INFO    ][2026-05-09 17:02:01][root                ][stage_datasets           ] [local_rank 3/4] Staging ssv2 (multipart_tar)
+[INFO    ][2026-05-09 17:03:31][app.vjepa.train     ][main                     ] Staged datasets: ['/scratch-node/dcanez.22606429/kinetics_240/train.csv', '/scratch-node/dcanez.22606429/ssv2/train.csv']
+[INFO    ][2026-05-09 17:03:52][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
+[INFO    ][2026-05-09 17:03:53][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA applied: 48 modules replaced (rank=8, alpha=16, dropout=0.0, targets=['attn.qkv', 'attn.proj'], scope=blocks)
+[INFO    ][2026-05-09 17:03:53][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA trainable params: 1,179,648
+[INFO    ][2026-05-09 17:03:53][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-09 17:03:53][root                ][init_video_model         ] VJEPAPredictorMultiSeqWrapper(
+  (backbone): VisionTransformerPredictor(
+    (predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
+    (mask_tokens): ParameterList(
+        (0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+    )
+    (predictor_blocks): ModuleList(
+      (0-5): 6 x Block(
+        (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (attn): RoPEAttention(
+          (qkv): Linear(in_features=384, out_features=1152, bias=True)
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): Linear(in_features=384, out_features=384, bias=True)
+          (proj_drop): Dropout(p=0.0, inplace=False)
+        )
+        (drop_path): Identity()
+        (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (mlp): MLP(
+          (fc1): Linear(in_features=384, out_features=1536, bias=True)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=1536, out_features=384, bias=True)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+    (predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
+  )
+)
+[INFO    ][2026-05-09 17:03:53][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-09 17:03:53][root                ][init_video_model         ] Encoder number of parameters: 1179648
+[INFO    ][2026-05-09 17:03:53][root                ][init_video_model         ] Predictor number of parameters: 11436928
+[INFO    ][2026-05-09 17:03:53][root                ][init_video_model         ] Target encoder number of parameters: 1179648
+[INFO    ][2026-05-09 17:03:58][root                ][make_videodataset        ] VideoDataset dataset created
+[INFO    ][2026-05-09 17:03:58][WeightedSampler     ][__init__                 ] Using DistributedWeightedSampler with rank 15 / 16
+[INFO    ][2026-05-09 17:03:58][root                ][make_videodataset        ] VideoDataset unsupervised data loader created
+[INFO    ][2026-05-09 17:03:58][app.vjepa.train     ][main                     ] iterations per epoch/dataset length: 300/399
+[INFO    ][2026-05-09 17:03:59][app.vjepa.train     ][main                     ] Wrapping models in DDP (rank 15)...
+[INFO    ][2026-05-09 17:05:26][app.vjepa.train     ][main                     ] Initializing loader...
+submitit INFO (2026-05-10 04:55:29,468) - Job has timed out. Ran 718 minutes out of requested 720 minutes.
+[INFO    ][2026-05-10 04:55:29][submitit            ][has_timed_out            ] Job has timed out. Ran 718 minutes out of requested 720 minutes.
+submitit WARNING (2026-05-10 04:55:29,468) - Caught signal SIGUSR2 on gcn78.local.snellius.surf.nl: this job is timed-out.
+[WARNING ][2026-05-10 04:55:29][submitit            ][checkpoint_and_try_requeue] Caught signal SIGUSR2 on gcn78.local.snellius.surf.nl: this job is timed-out.
+submitit INFO (2026-05-10 04:55:29,468) - Not checkpointing nor requeuing since I am a slave (procid=15).
+[INFO    ][2026-05-10 04:55:29][submitit            ][checkpoint_and_try_requeue] Not checkpointing nor requeuing since I am a slave (procid=15).
+submitit WARNING (2026-05-10 04:55:29,468) - Bypassing signal SIGCONT
+[WARNING ][2026-05-10 04:55:29][submitit            ][bypass                   ] Bypassing signal SIGCONT
+submitit WARNING (2026-05-10 04:55:29,471) - Bypassing signal SIGCONT
+[WARNING ][2026-05-10 04:55:29][submitit            ][bypass                   ] Bypassing signal SIGCONT
+submitit WARNING (2026-05-10 04:55:29,471) - Bypassing signal SIGTERM
+[WARNING ][2026-05-10 04:55:29][submitit            ][bypass                   ] Bypassing signal SIGTERM
+submitit INFO (2026-05-10 04:59:17,804) - Starting with JobEnvironment(job_id=22606429, hostname=gcn77.local.snellius.surf.nl, local_rank=3(4), node=3(4), global_rank=15(16))
+submitit INFO (2026-05-10 04:59:17,901) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/job_22606429/22606429_submitted.pkl
+INFO:root:loaded pretrain params...
+{   'app': 'vjepa',
+    'cpus_per_task': 16,
+    'data': {   'batch_size': 64,
+                'crop_size': 224,
+                'dataset_fpcs': [16, 16],
+                'dataset_type': 'VideoDataset',
+                'datasets': [   '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
+                                '/scratch-shared/dcanez/data/ssv2/train.csv'],
+                'datasets_weights': [0.65, 0.35],
+                'fps': 4,
+                'num_workers': 10,
+                'patch_size': 14,
+                'persistent_workers': True,
+                'pin_mem': True,
+                'stage': [   {   'dest': 'kinetics_240',
+                                 'format': 'targz_parts',
+                                 'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
+                             {   'dest': 'ssv2',
+                                 'format': 'multipart_tar',
+                                 'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
+                'tubelet_size': 1},
+    'data_aug': {   'auto_augment': False,
+                    'motion_shift': False,
+                    'random_resize_aspect_ratio': [0.75, 1.35],
+                    'random_resize_scale': [0.3, 1.0],
+                    'reprob': 0.0},
+    'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m',
+    'loss': {'loss_exp': 1.0},
+    'mask': [   {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 8,
+                    'spatial_scale': [0.15, 0.15],
+                    'temporal_scale': [1.0, 1.0]},
+                {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 2,
+                    'spatial_scale': [0.7, 0.7],
+                    'temporal_scale': [1.0, 1.0]}],
+    'mem_per_gpu': '180G',
+    'meta': {   'dtype': 'bfloat16',
+                'knn_eval_epoch0': False,
+                'knn_eval_freq': 5,
+                'knn_eval_presets': [   {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
+                                                          'eval_videos_per_class': 25,
+                                                          'num_workers': 8,
+                                                          'train_videos_per_class': 100},
+                                            'preset': 'ucf101'},
+                                        {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
+                                                          'eval_videos_per_class': 100,
+                                                          'linear_probe': True,
+                                                          'num_workers': 8,
+                                                          'pool_type': 'temporal_concat',
+                                                          'train_videos_per_class': 500},
+                                            'preset': 'ssv2_coarse10'}],
+                'load_checkpoint': True,
+                'read_checkpoint': None,
+                'save_every_freq': 5,
+                'seed': 239,
+                'use_sdpa': True,
+                'use_wandb': True,
+                'wandb_project': 'vjepa_ablation'},
+    'metrics': {'sigreg': {}, 'std': {}},
+    'model': {   'class_token': False,
+                 'freeze_backbone': True,
+                 'is_causal': False,
+                 'lora_alpha': 16,
+                 'lora_rank': 8,
+                 'lora_targets': ['attn.qkv', 'attn.proj'],
+                 'model_name': 'vit_large_patch14_capi_lvd1689m',
+                 'num_reg_tokens': 16,
+                 'pred_depth': 6,
+                 'pred_embed_dim': 384,
+                 'pred_num_heads': 12,
+                 'rope_type': 'capi2d',
+                 'stem_type': '2d',
+                 'target_kind': 'ema',
+                 'uniform_power': True,
+                 'use_activation_checkpointing': True,
+                 'use_lora': True,
+                 'use_mask_tokens': True,
+                 'use_rope': True,
+                 'use_sdpa': True,
+                 'zero_init_mask_tokens': True},
+    'nodes': 4,
+    'optimization': {   'ema': [0.99925, 0.99925],
+                        'epochs': 100,
+                        'final_lr': 0.0001,
+                        'final_weight_decay': 0.04,
+                        'ipe': 300,
+                        'ipe_scale': 1.0,
+                        'lr': 0.0005,
+                        'start_lr': 0.0001,
+                        'warmup': 10,
+                        'weight_decay': 0.04},
+    'tasks_per_node': 4}
+INFO:root:Running pre-training of app: vjepa
+[INFO    ][2026-05-10 05:00:27][app.vjepa.train     ][main                     ] which_dtype='bfloat16'
+[INFO    ][2026-05-10 05:00:27][app.vjepa.train     ][main                     ] Disabling persistent_workers (incompatible with KNN eval)
+[INFO    ][2026-05-10 05:00:27][app.vjepa.train     ][main                     ] NCCL_SOCKET_IFNAME=eno
+[INFO    ][2026-05-10 05:00:29][app.vjepa.train     ][main                     ] Initialized (rank/world-size) 15/16, tasks_per_node=4
+[INFO    ][2026-05-10 05:00:32][root                ][stage_datasets           ] [local_rank 3/4] Staging kinetics_240 (targz_parts)
+[INFO    ][2026-05-10 05:00:32][root                ][_stage_targz_parts       ] [rank 3] Extracting 25/103 tar.gz parts to /scratch-node/dcanez.22606429/kinetics_240
+[INFO    ][2026-05-10 05:00:46][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 2/25 parts
+[INFO    ][2026-05-10 05:01:00][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 4/25 parts
+[INFO    ][2026-05-10 05:01:15][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 6/25 parts
+[INFO    ][2026-05-10 05:01:29][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 8/25 parts
+[INFO    ][2026-05-10 05:01:44][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 10/25 parts
+[INFO    ][2026-05-10 05:01:58][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 12/25 parts
+[INFO    ][2026-05-10 05:02:12][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 14/25 parts
+[INFO    ][2026-05-10 05:02:27][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 16/25 parts
+[INFO    ][2026-05-10 05:02:41][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 18/25 parts
+[INFO    ][2026-05-10 05:02:55][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 20/25 parts
+[INFO    ][2026-05-10 05:03:10][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 22/25 parts
+[INFO    ][2026-05-10 05:03:24][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 24/25 parts
+[INFO    ][2026-05-10 05:03:31][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 25/25 parts
+[INFO    ][2026-05-10 05:03:31][root                ][stage_datasets           ] [local_rank 3/4] Staging ssv2 (multipart_tar)
+[INFO    ][2026-05-10 05:04:33][app.vjepa.train     ][main                     ] Staged datasets: ['/scratch-node/dcanez.22606429/kinetics_240/train.csv', '/scratch-node/dcanez.22606429/ssv2/train.csv']
+[INFO    ][2026-05-10 05:06:01][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
+[INFO    ][2026-05-10 05:06:05][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA applied: 48 modules replaced (rank=8, alpha=16, dropout=0.0, targets=['attn.qkv', 'attn.proj'], scope=blocks)
+[INFO    ][2026-05-10 05:06:05][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA trainable params: 1,179,648
+[INFO    ][2026-05-10 05:06:05][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-10 05:06:05][root                ][init_video_model         ] VJEPAPredictorMultiSeqWrapper(
+  (backbone): VisionTransformerPredictor(
+    (predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
+    (mask_tokens): ParameterList(
+        (0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+    )
+    (predictor_blocks): ModuleList(
+      (0-5): 6 x Block(
+        (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (attn): RoPEAttention(
+          (qkv): Linear(in_features=384, out_features=1152, bias=True)
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): Linear(in_features=384, out_features=384, bias=True)
+          (proj_drop): Dropout(p=0.0, inplace=False)
+        )
+        (drop_path): Identity()
+        (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (mlp): MLP(
+          (fc1): Linear(in_features=384, out_features=1536, bias=True)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=1536, out_features=384, bias=True)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+    (predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
+  )
+)
+[INFO    ][2026-05-10 05:06:05][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-10 05:06:05][root                ][init_video_model         ] Encoder number of parameters: 1179648
+[INFO    ][2026-05-10 05:06:05][root                ][init_video_model         ] Predictor number of parameters: 11436928
+[INFO    ][2026-05-10 05:06:05][root                ][init_video_model         ] Target encoder number of parameters: 1179648
+[INFO    ][2026-05-10 05:06:07][root                ][make_videodataset        ] VideoDataset dataset created
+[INFO    ][2026-05-10 05:06:07][WeightedSampler     ][__init__                 ] Using DistributedWeightedSampler with rank 15 / 16
+[INFO    ][2026-05-10 05:06:07][root                ][make_videodataset        ] VideoDataset unsupervised data loader created
+[INFO    ][2026-05-10 05:06:07][app.vjepa.train     ][main                     ] iterations per epoch/dataset length: 300/399
+[INFO    ][2026-05-10 05:06:07][app.vjepa.train     ][main                     ] Wrapping models in DDP (rank 15)...
+[INFO    ][2026-05-10 05:06:08][root                ][load_checkpoint          ] Loading checkpoint from /scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/latest.pt
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] loaded pretrained encoder from epoch 33 with msg: <All keys matched successfully>
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] loaded pretrained predictor from epoch 33 with msg: <All keys matched successfully>
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] loaded pretrained target encoder from epoch 33 with msg: <All keys matched successfully>
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] loaded optimizers from epoch 33
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] read-path: /scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/latest.pt
+[INFO    ][2026-05-10 05:06:09][app.vjepa.train     ][main                     ] Initializing loader...
+submitit WARNING (2026-05-10 09:26:31,216) - Bypassing signal SIGTERM
+[WARNING ][2026-05-10 09:26:31][submitit            ][bypass                   ] Bypassing signal SIGTERM
+submitit WARNING (2026-05-10 09:26:31,216) - Bypassing signal SIGCONT
+[WARNING ][2026-05-10 09:26:31][submitit            ][bypass                   ] Bypassing signal SIGCONT

vjepa/inflated/d003_lora_ema_capilvd1689m/job_22606429/22606429_1_log.err

@@ -0,0 +1,313 @@
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
+  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
+[rank1]:[W509 17:01:59.820396447 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
+  scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+submitit WARNING (2026-05-10 04:55:29,177) - Caught signal SIGUSR2 on gcn74.local.snellius.surf.nl: this job is timed-out.
+submitit WARNING (2026-05-10 04:55:29,178) - Bypassing signal SIGCONT
+submitit WARNING (2026-05-10 04:55:29,547) - Bypassing signal SIGTERM
+submitit WARNING (2026-05-10 04:55:29,547) - Bypassing signal SIGCONT
+[rank1]:[W510 04:55:34.394824634 TCPStore.cpp:125] [c10d] recvValue failed on SocketImpl(fd=3, addr=[gcn74.local.snellius.surf.nl]:52144, remote=[gcn74.local.snellius.surf.nl]:37129): Failed to recv, got 0 bytes. Connection was likely closed. Did the remote server shutdown or crash?
+Exception raised from recvBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:682 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14a0fd2a9fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a3325d (0x14a14143825d in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x273 (0x14a1414361f3 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14a0fe518eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14a23d982164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14a24e3f419a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14a24e479100 in /lib64/libc.so.6)
+
+[rank1]:[W510 04:55:34.454977412 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 1] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Failed to recv, got 0 bytes. Connection was likely closed. Did the remote server shutdown or crash?
+[rank1]:[W510 04:55:35.455124149 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn74.local.snellius.surf.nl]:52144, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14a0fd2a9fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14a1414376d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14a1414361cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14a0fe518eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14a23d982164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14a24e3f419a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14a24e479100 in /lib64/libc.so.6)
+
+[rank1]:[W510 04:55:35.457356583 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 1] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank1]:[W510 04:55:36.457487650 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn74.local.snellius.surf.nl]:52144, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14a0fd2a9fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14a1414376d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14a1414361cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14a0fe518eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14a23d982164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14a24e3f419a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14a24e479100 in /lib64/libc.so.6)
+
+[rank1]:[W510 04:55:36.459700985 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 1] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank1]:[W510 04:55:37.459835991 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn74.local.snellius.surf.nl]:52144, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14a0fd2a9fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14a1414376d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14a1414361cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14a0fe518eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14a23d982164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14a24e3f419a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14a24e479100 in /lib64/libc.so.6)
+
+[rank1]:[W510 04:55:37.462164165 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 1] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank1]:[W510 04:55:38.462282842 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn74.local.snellius.surf.nl]:52144, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14a0fd2a9fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14a1414376d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14a1414361cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14a0fe518eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14a23d982164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14a24e3f419a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14a24e479100 in /lib64/libc.so.6)
+
+[rank1]:[W510 04:55:38.464606826 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 1] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank1]:[W510 04:55:39.464723273 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn74.local.snellius.surf.nl]:52144, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14a0fd2a9fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14a1414376d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14a1414361cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14a0fe518eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14a23d982164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14a24e3f419a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14a24e479100 in /lib64/libc.so.6)
+
+[rank1]:[W510 04:55:39.467029667 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 1] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank1]:[W510 04:55:40.467147964 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn74.local.snellius.surf.nl]:52144, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14a0fd2a9fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14a1414376d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14a1414361cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14a0fe518eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14a23d982164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14a24e3f419a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14a24e479100 in /lib64/libc.so.6)
+
+[rank1]:[W510 04:55:40.469489298 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 1] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank1]:[W510 04:55:41.469633225 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn74.local.snellius.surf.nl]:52144, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14a0fd2a9fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14a1414376d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14a1414361cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14a0fe518eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14a23d982164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14a24e3f419a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14a24e479100 in /lib64/libc.so.6)
+
+[rank1]:[W510 04:55:41.471947929 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 1] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank1]:[W510 04:55:42.472090516 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn74.local.snellius.surf.nl]:52144, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14a0fd2a9fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14a1414376d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14a1414361cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14a0fe518eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14a23d982164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14a24e3f419a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14a24e479100 in /lib64/libc.so.6)
+
+[rank1]:[W510 04:55:42.474380030 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 1] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank1]:[W510 04:55:43.474516707 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn74.local.snellius.surf.nl]:52144, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14a0fd2a9fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14a1414376d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14a1414361cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14a0fe518eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14a23d982164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14a24e3f419a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14a24e479100 in /lib64/libc.so.6)
+
+[rank1]:[W510 04:55:43.476761731 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 1] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank1]:[W510 04:55:44.476907248 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn74.local.snellius.surf.nl]:52144, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14a0fd2a9fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14a1414376d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14a1414361cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14a0fe518eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14a23d982164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14a24e3f419a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14a24e479100 in /lib64/libc.so.6)
+
+[rank1]:[W510 04:55:44.479249802 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 1] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank1]:[W510 04:55:45.479382169 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn74.local.snellius.surf.nl]:52144, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14a0fd2a9fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14a1414376d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14a1414361cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14a0fe518eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14a23d982164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14a24e3f419a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14a24e479100 in /lib64/libc.so.6)
+
+[rank1]:[W510 04:55:45.481671083 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 1] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank1]:[W510 04:55:46.481800990 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn74.local.snellius.surf.nl]:52144, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14a0fd2a9fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14a1414376d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14a1414361cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14a0fe518eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14a23d982164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14a24e3f419a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14a24e479100 in /lib64/libc.so.6)
+
+[rank1]:[W510 04:55:46.484088454 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 1] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank1]:[W510 04:55:47.484223251 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn74.local.snellius.surf.nl]:52144, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14a0fd2a9fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14a1414376d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14a1414361cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14a0fe518eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14a23d982164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14a24e3f419a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14a24e479100 in /lib64/libc.so.6)
+
+[rank1]:[W510 04:55:47.486564305 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 1] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank1]:[W510 04:55:48.486694422 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn74.local.snellius.surf.nl]:52144, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14a0fd2a9fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14a1414376d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14a1414361cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14a0fe518eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14a23d982164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14a24e3f419a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14a24e479100 in /lib64/libc.so.6)
+
+[rank1]:[W510 04:55:48.489014716 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 1] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank1]:[W510 04:55:49.489155263 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn74.local.snellius.surf.nl]:52144, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14a0fd2a9fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14a1414376d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14a1414361cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14a0fe518eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14a23d982164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14a24e3f419a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14a24e479100 in /lib64/libc.so.6)
+
+[rank1]:[W510 04:55:49.491491777 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 1] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank1]:[W510 04:55:50.491630913 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn74.local.snellius.surf.nl]:52144, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14a0fd2a9fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14a1414376d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14a1414361cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14a0fe518eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14a23d982164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14a24e3f419a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14a24e479100 in /lib64/libc.so.6)
+
+[rank1]:[W510 04:55:50.493943658 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 1] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank1]:[W510 04:55:51.494080534 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn74.local.snellius.surf.nl]:52144, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14a0fd2a9fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14a1414376d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14a1414361cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14a0fe518eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14a23d982164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14a24e3f419a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14a24e479100 in /lib64/libc.so.6)
+
+[rank1]:[W510 04:55:51.496381539 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 1] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank1]:[W510 04:55:52.496527015 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn74.local.snellius.surf.nl]:52144, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14a0fd2a9fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14a1414376d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14a1414361cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14a0fe518eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14a23d982164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14a24e3f419a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14a24e479100 in /lib64/libc.so.6)
+
+[rank1]:[W510 04:55:52.498859050 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 1] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank1]:[W510 04:55:53.498991916 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn74.local.snellius.surf.nl]:52144, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14a0fd2a9fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14a1414376d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14a1414361cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14a0fe518eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14a23d982164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14a24e3f419a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14a24e479100 in /lib64/libc.so.6)
+
+[rank1]:[W510 04:55:53.501291991 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 1] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank1]:[W510 04:55:54.501426937 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn74.local.snellius.surf.nl]:52144, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14a0fd2a9fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14a1414376d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14a1414361cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14a0fe518eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14a23d982164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14a24e3f419a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14a24e479100 in /lib64/libc.so.6)
+
+[rank1]:[W510 04:55:54.503751532 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 1] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank1]:[W510 04:55:55.503884078 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn74.local.snellius.surf.nl]:52144, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14a0fd2a9fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14a1414376d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14a1414361cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14a0fe518eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14a23d982164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14a24e3f419a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14a24e479100 in /lib64/libc.so.6)
+
+[rank1]:[W510 04:55:55.506197123 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 1] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank1]:[W510 04:55:56.506335129 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn74.local.snellius.surf.nl]:52144, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14a0fd2a9fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14a1414376d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14a1414361cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14a0fe518eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14a23d982164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14a24e3f419a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14a24e479100 in /lib64/libc.so.6)
+
+[rank1]:[W510 04:55:56.508629734 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 1] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank1]:[W510 04:55:57.508777350 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn74.local.snellius.surf.nl]:52144, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14a0fd2a9fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14a1414376d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14a1414361cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14a0fe518eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14a23d982164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14a24e3f419a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14a24e479100 in /lib64/libc.so.6)
+
+[rank1]:[W510 04:55:57.511163764 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 1] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank1]:[W510 04:55:58.511292591 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn74.local.snellius.surf.nl]:52144, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14a0fd2a9fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14a1414376d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14a1414361cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14a0fe518eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14a23d982164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14a24e3f419a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14a24e479100 in /lib64/libc.so.6)
+
+[rank1]:[W510 04:55:58.513570786 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 1] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
+  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
+[rank1]:[W510 05:02:23.918270987 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
+  scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+submitit WARNING (2026-05-10 09:26:31,095) - Bypassing signal SIGTERM
+submitit WARNING (2026-05-10 09:26:31,111) - Bypassing signal SIGCONT

vjepa/inflated/d003_lora_ema_capilvd1689m/job_22606429/22606429_1_log.out

@@ -0,0 +1,522 @@
+submitit INFO (2026-05-09 16:57:36,581) - Starting with JobEnvironment(job_id=22606429, hostname=gcn74.local.snellius.surf.nl, local_rank=1(4), node=0(4), global_rank=1(16))
+submitit INFO (2026-05-09 16:57:36,581) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/job_22606429/22606429_submitted.pkl
+INFO:root:loaded pretrain params...
+{   'app': 'vjepa',
+    'cpus_per_task': 16,
+    'data': {   'batch_size': 64,
+                'crop_size': 224,
+                'dataset_fpcs': [16, 16],
+                'dataset_type': 'VideoDataset',
+                'datasets': [   '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
+                                '/scratch-shared/dcanez/data/ssv2/train.csv'],
+                'datasets_weights': [0.65, 0.35],
+                'fps': 4,
+                'num_workers': 10,
+                'patch_size': 14,
+                'persistent_workers': True,
+                'pin_mem': True,
+                'stage': [   {   'dest': 'kinetics_240',
+                                 'format': 'targz_parts',
+                                 'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
+                             {   'dest': 'ssv2',
+                                 'format': 'multipart_tar',
+                                 'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
+                'tubelet_size': 1},
+    'data_aug': {   'auto_augment': False,
+                    'motion_shift': False,
+                    'random_resize_aspect_ratio': [0.75, 1.35],
+                    'random_resize_scale': [0.3, 1.0],
+                    'reprob': 0.0},
+    'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m',
+    'loss': {'loss_exp': 1.0},
+    'mask': [   {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 8,
+                    'spatial_scale': [0.15, 0.15],
+                    'temporal_scale': [1.0, 1.0]},
+                {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 2,
+                    'spatial_scale': [0.7, 0.7],
+                    'temporal_scale': [1.0, 1.0]}],
+    'mem_per_gpu': '180G',
+    'meta': {   'dtype': 'bfloat16',
+                'knn_eval_epoch0': False,
+                'knn_eval_freq': 5,
+                'knn_eval_presets': [   {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
+                                                          'eval_videos_per_class': 25,
+                                                          'num_workers': 8,
+                                                          'train_videos_per_class': 100},
+                                            'preset': 'ucf101'},
+                                        {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
+                                                          'eval_videos_per_class': 100,
+                                                          'linear_probe': True,
+                                                          'num_workers': 8,
+                                                          'pool_type': 'temporal_concat',
+                                                          'train_videos_per_class': 500},
+                                            'preset': 'ssv2_coarse10'}],
+                'load_checkpoint': True,
+                'read_checkpoint': None,
+                'save_every_freq': 5,
+                'seed': 239,
+                'use_sdpa': True,
+                'use_wandb': True,
+                'wandb_project': 'vjepa_ablation'},
+    'metrics': {'sigreg': {}, 'std': {}},
+    'model': {   'class_token': False,
+                 'freeze_backbone': True,
+                 'is_causal': False,
+                 'lora_alpha': 16,
+                 'lora_rank': 8,
+                 'lora_targets': ['attn.qkv', 'attn.proj'],
+                 'model_name': 'vit_large_patch14_capi_lvd1689m',
+                 'num_reg_tokens': 16,
+                 'pred_depth': 6,
+                 'pred_embed_dim': 384,
+                 'pred_num_heads': 12,
+                 'rope_type': 'capi2d',
+                 'stem_type': '2d',
+                 'target_kind': 'ema',
+                 'uniform_power': True,
+                 'use_activation_checkpointing': True,
+                 'use_lora': True,
+                 'use_mask_tokens': True,
+                 'use_rope': True,
+                 'use_sdpa': True,
+                 'zero_init_mask_tokens': True},
+    'nodes': 4,
+    'optimization': {   'ema': [0.99925, 0.99925],
+                        'epochs': 100,
+                        'final_lr': 0.0001,
+                        'final_weight_decay': 0.04,
+                        'ipe': 300,
+                        'ipe_scale': 1.0,
+                        'lr': 0.0005,
+                        'start_lr': 0.0001,
+                        'warmup': 10,
+                        'weight_decay': 0.04},
+    'tasks_per_node': 4}
+INFO:root:Running pre-training of app: vjepa
+[INFO    ][2026-05-09 16:58:54][app.vjepa.train     ][main                     ] which_dtype='bfloat16'
+[INFO    ][2026-05-09 16:58:54][app.vjepa.train     ][main                     ] Disabling persistent_workers (incompatible with KNN eval)
+[INFO    ][2026-05-09 16:58:54][app.vjepa.train     ][main                     ] NCCL_SOCKET_IFNAME=eno
+[INFO    ][2026-05-09 16:58:56][app.vjepa.train     ][main                     ] Initialized (rank/world-size) 1/16, tasks_per_node=4
+[INFO    ][2026-05-09 16:58:56][root                ][stage_datasets           ] [local_rank 1/4] Staging kinetics_240 (targz_parts)
+[INFO    ][2026-05-09 16:58:56][root                ][_stage_targz_parts       ] [rank 1] Extracting 26/103 tar.gz parts to /scratch-node/dcanez.22606429/kinetics_240
+[INFO    ][2026-05-09 16:59:08][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 2/26 parts
+[INFO    ][2026-05-09 16:59:22][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 4/26 parts
+[INFO    ][2026-05-09 16:59:36][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 6/26 parts
+[INFO    ][2026-05-09 16:59:50][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 8/26 parts
+[INFO    ][2026-05-09 17:00:05][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 10/26 parts
+[INFO    ][2026-05-09 17:00:19][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 12/26 parts
+[INFO    ][2026-05-09 17:00:33][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 14/26 parts
+[INFO    ][2026-05-09 17:00:47][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 16/26 parts
+[INFO    ][2026-05-09 17:01:01][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 18/26 parts
+[INFO    ][2026-05-09 17:01:16][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 20/26 parts
+[INFO    ][2026-05-09 17:01:30][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 22/26 parts
+[INFO    ][2026-05-09 17:01:45][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 24/26 parts
+[INFO    ][2026-05-09 17:01:59][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 26/26 parts
+[INFO    ][2026-05-09 17:01:59][root                ][stage_datasets           ] [local_rank 1/4] Staging ssv2 (multipart_tar)
+[INFO    ][2026-05-09 17:03:31][app.vjepa.train     ][main                     ] Staged datasets: ['/scratch-node/dcanez.22606429/kinetics_240/train.csv', '/scratch-node/dcanez.22606429/ssv2/train.csv']
+[INFO    ][2026-05-09 17:05:15][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
+[INFO    ][2026-05-09 17:05:19][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA applied: 48 modules replaced (rank=8, alpha=16, dropout=0.0, targets=['attn.qkv', 'attn.proj'], scope=blocks)
+[INFO    ][2026-05-09 17:05:19][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA trainable params: 1,179,648
+[INFO    ][2026-05-09 17:05:19][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-09 17:05:19][root                ][init_video_model         ] VJEPAPredictorMultiSeqWrapper(
+  (backbone): VisionTransformerPredictor(
+    (predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
+    (mask_tokens): ParameterList(
+        (0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+    )
+    (predictor_blocks): ModuleList(
+      (0-5): 6 x Block(
+        (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (attn): RoPEAttention(
+          (qkv): Linear(in_features=384, out_features=1152, bias=True)
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): Linear(in_features=384, out_features=384, bias=True)
+          (proj_drop): Dropout(p=0.0, inplace=False)
+        )
+        (drop_path): Identity()
+        (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (mlp): MLP(
+          (fc1): Linear(in_features=384, out_features=1536, bias=True)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=1536, out_features=384, bias=True)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+    (predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
+  )
+)
+[INFO    ][2026-05-09 17:05:19][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-09 17:05:19][root                ][init_video_model         ] Encoder number of parameters: 1179648
+[INFO    ][2026-05-09 17:05:19][root                ][init_video_model         ] Predictor number of parameters: 11436928
+[INFO    ][2026-05-09 17:05:19][root                ][init_video_model         ] Target encoder number of parameters: 1179648
+[INFO    ][2026-05-09 17:05:21][root                ][make_videodataset        ] VideoDataset dataset created
+[INFO    ][2026-05-09 17:05:21][WeightedSampler     ][__init__                 ] Using DistributedWeightedSampler with rank 1 / 16
+[INFO    ][2026-05-09 17:05:21][root                ][make_videodataset        ] VideoDataset unsupervised data loader created
+[INFO    ][2026-05-09 17:05:21][app.vjepa.train     ][main                     ] iterations per epoch/dataset length: 300/399
+[INFO    ][2026-05-09 17:05:21][app.vjepa.train     ][main                     ] Wrapping models in DDP (rank 1)...
+[INFO    ][2026-05-09 17:05:26][app.vjepa.train     ][main                     ] Initializing loader...
+submitit INFO (2026-05-10 04:55:29,165) - Job has timed out. Ran 718 minutes out of requested 720 minutes.
+[INFO    ][2026-05-10 04:55:29][submitit            ][has_timed_out            ] Job has timed out. Ran 718 minutes out of requested 720 minutes.
+submitit WARNING (2026-05-10 04:55:29,177) - Caught signal SIGUSR2 on gcn74.local.snellius.surf.nl: this job is timed-out.
+[WARNING ][2026-05-10 04:55:29][submitit            ][checkpoint_and_try_requeue] Caught signal SIGUSR2 on gcn74.local.snellius.surf.nl: this job is timed-out.
+submitit INFO (2026-05-10 04:55:29,178) - Not checkpointing nor requeuing since I am a slave (procid=1).
+[INFO    ][2026-05-10 04:55:29][submitit            ][checkpoint_and_try_requeue] Not checkpointing nor requeuing since I am a slave (procid=1).
+submitit WARNING (2026-05-10 04:55:29,178) - Bypassing signal SIGCONT
+[WARNING ][2026-05-10 04:55:29][submitit            ][bypass                   ] Bypassing signal SIGCONT
+submitit WARNING (2026-05-10 04:55:29,547) - Bypassing signal SIGTERM
+[WARNING ][2026-05-10 04:55:29][submitit            ][bypass                   ] Bypassing signal SIGTERM
+submitit WARNING (2026-05-10 04:55:29,547) - Bypassing signal SIGCONT
+[WARNING ][2026-05-10 04:55:29][submitit            ][bypass                   ] Bypassing signal SIGCONT
+submitit INFO (2026-05-10 04:59:12,566) - Starting with JobEnvironment(job_id=22606429, hostname=gcn74.local.snellius.surf.nl, local_rank=1(4), node=0(4), global_rank=1(16))
+submitit INFO (2026-05-10 04:59:12,566) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/job_22606429/22606429_submitted.pkl
+INFO:root:loaded pretrain params...
+{   'app': 'vjepa',
+    'cpus_per_task': 16,
+    'data': {   'batch_size': 64,
+                'crop_size': 224,
+                'dataset_fpcs': [16, 16],
+                'dataset_type': 'VideoDataset',
+                'datasets': [   '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
+                                '/scratch-shared/dcanez/data/ssv2/train.csv'],
+                'datasets_weights': [0.65, 0.35],
+                'fps': 4,
+                'num_workers': 10,
+                'patch_size': 14,
+                'persistent_workers': True,
+                'pin_mem': True,
+                'stage': [   {   'dest': 'kinetics_240',
+                                 'format': 'targz_parts',
+                                 'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
+                             {   'dest': 'ssv2',
+                                 'format': 'multipart_tar',
+                                 'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
+                'tubelet_size': 1},
+    'data_aug': {   'auto_augment': False,
+                    'motion_shift': False,
+                    'random_resize_aspect_ratio': [0.75, 1.35],
+                    'random_resize_scale': [0.3, 1.0],
+                    'reprob': 0.0},
+    'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m',
+    'loss': {'loss_exp': 1.0},
+    'mask': [   {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 8,
+                    'spatial_scale': [0.15, 0.15],
+                    'temporal_scale': [1.0, 1.0]},
+                {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 2,
+                    'spatial_scale': [0.7, 0.7],
+                    'temporal_scale': [1.0, 1.0]}],
+    'mem_per_gpu': '180G',
+    'meta': {   'dtype': 'bfloat16',
+                'knn_eval_epoch0': False,
+                'knn_eval_freq': 5,
+                'knn_eval_presets': [   {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
+                                                          'eval_videos_per_class': 25,
+                                                          'num_workers': 8,
+                                                          'train_videos_per_class': 100},
+                                            'preset': 'ucf101'},
+                                        {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
+                                                          'eval_videos_per_class': 100,
+                                                          'linear_probe': True,
+                                                          'num_workers': 8,
+                                                          'pool_type': 'temporal_concat',
+                                                          'train_videos_per_class': 500},
+                                            'preset': 'ssv2_coarse10'}],
+                'load_checkpoint': True,
+                'read_checkpoint': None,
+                'save_every_freq': 5,
+                'seed': 239,
+                'use_sdpa': True,
+                'use_wandb': True,
+                'wandb_project': 'vjepa_ablation'},
+    'metrics': {'sigreg': {}, 'std': {}},
+    'model': {   'class_token': False,
+                 'freeze_backbone': True,
+                 'is_causal': False,
+                 'lora_alpha': 16,
+                 'lora_rank': 8,
+                 'lora_targets': ['attn.qkv', 'attn.proj'],
+                 'model_name': 'vit_large_patch14_capi_lvd1689m',
+                 'num_reg_tokens': 16,
+                 'pred_depth': 6,
+                 'pred_embed_dim': 384,
+                 'pred_num_heads': 12,
+                 'rope_type': 'capi2d',
+                 'stem_type': '2d',
+                 'target_kind': 'ema',
+                 'uniform_power': True,
+                 'use_activation_checkpointing': True,
+                 'use_lora': True,
+                 'use_mask_tokens': True,
+                 'use_rope': True,
+                 'use_sdpa': True,
+                 'zero_init_mask_tokens': True},
+    'nodes': 4,
+    'optimization': {   'ema': [0.99925, 0.99925],
+                        'epochs': 100,
+                        'final_lr': 0.0001,
+                        'final_weight_decay': 0.04,
+                        'ipe': 300,
+                        'ipe_scale': 1.0,
+                        'lr': 0.0005,
+                        'start_lr': 0.0001,
+                        'warmup': 10,
+                        'weight_decay': 0.04},
+    'tasks_per_node': 4}
+INFO:root:Running pre-training of app: vjepa
+[INFO    ][2026-05-10 04:59:18][app.vjepa.train     ][main                     ] which_dtype='bfloat16'
+[INFO    ][2026-05-10 04:59:18][app.vjepa.train     ][main                     ] Disabling persistent_workers (incompatible with KNN eval)
+[INFO    ][2026-05-10 04:59:18][app.vjepa.train     ][main                     ] NCCL_SOCKET_IFNAME=eno
+[INFO    ][2026-05-10 04:59:21][app.vjepa.train     ][main                     ] Initialized (rank/world-size) 1/16, tasks_per_node=4
+[INFO    ][2026-05-10 04:59:21][root                ][stage_datasets           ] [local_rank 1/4] Staging kinetics_240 (targz_parts)
+[INFO    ][2026-05-10 04:59:21][root                ][_stage_targz_parts       ] [rank 1] Extracting 26/103 tar.gz parts to /scratch-node/dcanez.22606429/kinetics_240
+[INFO    ][2026-05-10 04:59:33][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 2/26 parts
+[INFO    ][2026-05-10 04:59:47][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 4/26 parts
+[INFO    ][2026-05-10 05:00:01][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 6/26 parts
+[INFO    ][2026-05-10 05:00:15][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 8/26 parts
+[INFO    ][2026-05-10 05:00:30][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 10/26 parts
+[INFO    ][2026-05-10 05:00:44][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 12/26 parts
+[INFO    ][2026-05-10 05:00:58][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 14/26 parts
+[INFO    ][2026-05-10 05:01:12][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 16/26 parts
+[INFO    ][2026-05-10 05:01:26][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 18/26 parts
+[INFO    ][2026-05-10 05:01:40][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 20/26 parts
+[INFO    ][2026-05-10 05:01:55][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 22/26 parts
+[INFO    ][2026-05-10 05:02:09][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 24/26 parts
+[INFO    ][2026-05-10 05:02:23][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 26/26 parts
+[INFO    ][2026-05-10 05:02:23][root                ][stage_datasets           ] [local_rank 1/4] Staging ssv2 (multipart_tar)
+[INFO    ][2026-05-10 05:04:33][app.vjepa.train     ][main                     ] Staged datasets: ['/scratch-node/dcanez.22606429/kinetics_240/train.csv', '/scratch-node/dcanez.22606429/ssv2/train.csv']
+[INFO    ][2026-05-10 05:05:22][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
+[INFO    ][2026-05-10 05:05:26][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA applied: 48 modules replaced (rank=8, alpha=16, dropout=0.0, targets=['attn.qkv', 'attn.proj'], scope=blocks)
+[INFO    ][2026-05-10 05:05:26][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA trainable params: 1,179,648
+[INFO    ][2026-05-10 05:05:27][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-10 05:05:27][root                ][init_video_model         ] VJEPAPredictorMultiSeqWrapper(
+  (backbone): VisionTransformerPredictor(
+    (predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
+    (mask_tokens): ParameterList(
+        (0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+    )
+    (predictor_blocks): ModuleList(
+      (0-5): 6 x Block(
+        (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (attn): RoPEAttention(
+          (qkv): Linear(in_features=384, out_features=1152, bias=True)
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): Linear(in_features=384, out_features=384, bias=True)
+          (proj_drop): Dropout(p=0.0, inplace=False)
+        )
+        (drop_path): Identity()
+        (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (mlp): MLP(
+          (fc1): Linear(in_features=384, out_features=1536, bias=True)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=1536, out_features=384, bias=True)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+    (predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
+  )
+)
+[INFO    ][2026-05-10 05:05:27][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-10 05:05:27][root                ][init_video_model         ] Encoder number of parameters: 1179648
+[INFO    ][2026-05-10 05:05:27][root                ][init_video_model         ] Predictor number of parameters: 11436928
+[INFO    ][2026-05-10 05:05:27][root                ][init_video_model         ] Target encoder number of parameters: 1179648
+[INFO    ][2026-05-10 05:05:28][root                ][make_videodataset        ] VideoDataset dataset created
+[INFO    ][2026-05-10 05:05:28][WeightedSampler     ][__init__                 ] Using DistributedWeightedSampler with rank 1 / 16
+[INFO    ][2026-05-10 05:05:28][root                ][make_videodataset        ] VideoDataset unsupervised data loader created
+[INFO    ][2026-05-10 05:05:28][app.vjepa.train     ][main                     ] iterations per epoch/dataset length: 300/399
+[INFO    ][2026-05-10 05:05:28][app.vjepa.train     ][main                     ] Wrapping models in DDP (rank 1)...
+[INFO    ][2026-05-10 05:06:08][root                ][load_checkpoint          ] Loading checkpoint from /scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/latest.pt
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] loaded pretrained encoder from epoch 33 with msg: <All keys matched successfully>
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] loaded pretrained predictor from epoch 33 with msg: <All keys matched successfully>
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] loaded pretrained target encoder from epoch 33 with msg: <All keys matched successfully>
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] loaded optimizers from epoch 33
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] read-path: /scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/latest.pt
+[INFO    ][2026-05-10 05:06:09][app.vjepa.train     ][main                     ] Initializing loader...
+submitit WARNING (2026-05-10 09:26:31,095) - Bypassing signal SIGTERM
+[WARNING ][2026-05-10 09:26:31][submitit            ][bypass                   ] Bypassing signal SIGTERM
+submitit WARNING (2026-05-10 09:26:31,111) - Bypassing signal SIGCONT
+[WARNING ][2026-05-10 09:26:31][submitit            ][bypass                   ] Bypassing signal SIGCONT

vjepa/inflated/d003_lora_ema_capilvd1689m/job_22606429/22606429_2_log.err

@@ -0,0 +1,313 @@
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
+  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
+[rank2]:[W509 17:02:02.793102444 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
+  scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+submitit WARNING (2026-05-10 04:55:29,245) - Caught signal SIGUSR2 on gcn74.local.snellius.surf.nl: this job is timed-out.
+submitit WARNING (2026-05-10 04:55:29,245) - Bypassing signal SIGCONT
+submitit WARNING (2026-05-10 04:55:29,547) - Bypassing signal SIGTERM
+submitit WARNING (2026-05-10 04:55:29,547) - Bypassing signal SIGCONT
+[rank2]:[W510 04:55:34.560308863 TCPStore.cpp:125] [c10d] recvValue failed on SocketImpl(fd=4, addr=[gcn74.local.snellius.surf.nl]:52154, remote=[gcn74.local.snellius.surf.nl]:37129): Failed to recv, got 0 bytes. Connection was likely closed. Did the remote server shutdown or crash?
+Exception raised from recvBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:682 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15302b86dfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a3325d (0x15306f9fc25d in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x273 (0x15306f9fa1f3 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15302cadceec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15316bf46164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15317c9b819a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15317ca3d100 in /lib64/libc.so.6)
+
+[rank2]:[W510 04:55:34.563393286 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 2] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Failed to recv, got 0 bytes. Connection was likely closed. Did the remote server shutdown or crash?
+[rank2]:[W510 04:55:35.563541282 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn74.local.snellius.surf.nl]:52154, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15302b86dfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15306f9fb6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15306f9fa1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15302cadceec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15316bf46164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15317c9b819a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15317ca3d100 in /lib64/libc.so.6)
+
+[rank2]:[W510 04:55:35.565850107 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 2] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank2]:[W510 04:55:36.565943503 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn74.local.snellius.surf.nl]:52154, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15302b86dfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15306f9fb6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15306f9fa1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15302cadceec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15316bf46164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15317c9b819a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15317ca3d100 in /lib64/libc.so.6)
+
+[rank2]:[W510 04:55:36.568184308 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 2] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank2]:[W510 04:55:37.568314255 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn74.local.snellius.surf.nl]:52154, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15302b86dfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15306f9fb6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15306f9fa1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15302cadceec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15316bf46164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15317c9b819a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15317ca3d100 in /lib64/libc.so.6)
+
+[rank2]:[W510 04:55:37.570686259 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 2] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank2]:[W510 04:55:38.570825245 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn74.local.snellius.surf.nl]:52154, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15302b86dfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15306f9fb6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15306f9fa1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15302cadceec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15316bf46164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15317c9b819a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15317ca3d100 in /lib64/libc.so.6)
+
+[rank2]:[W510 04:55:38.573171570 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 2] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank2]:[W510 04:55:39.573312246 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn74.local.snellius.surf.nl]:52154, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15302b86dfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15306f9fb6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15306f9fa1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15302cadceec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15316bf46164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15317c9b819a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15317ca3d100 in /lib64/libc.so.6)
+
+[rank2]:[W510 04:55:39.575660421 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 2] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank2]:[W510 04:55:40.575789517 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn74.local.snellius.surf.nl]:52154, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15302b86dfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15306f9fb6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15306f9fa1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15302cadceec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15316bf46164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15317c9b819a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15317ca3d100 in /lib64/libc.so.6)
+
+[rank2]:[W510 04:55:40.578147842 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 2] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank2]:[W510 04:55:41.578265748 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn74.local.snellius.surf.nl]:52154, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15302b86dfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15306f9fb6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15306f9fa1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15302cadceec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15316bf46164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15317c9b819a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15317ca3d100 in /lib64/libc.so.6)
+
+[rank2]:[W510 04:55:41.580669902 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 2] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank2]:[W510 04:55:42.580822889 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn74.local.snellius.surf.nl]:52154, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15302b86dfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15306f9fb6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15306f9fa1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15302cadceec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15316bf46164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15317c9b819a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15317ca3d100 in /lib64/libc.so.6)
+
+[rank2]:[W510 04:55:42.583122063 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 2] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank2]:[W510 04:55:43.583267190 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn74.local.snellius.surf.nl]:52154, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15302b86dfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15306f9fb6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15306f9fa1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15302cadceec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15316bf46164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15317c9b819a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15317ca3d100 in /lib64/libc.so.6)
+
+[rank2]:[W510 04:55:43.585584824 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 2] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank2]:[W510 04:55:44.585729321 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn74.local.snellius.surf.nl]:52154, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15302b86dfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15306f9fb6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15306f9fa1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15302cadceec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15316bf46164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15317c9b819a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15317ca3d100 in /lib64/libc.so.6)
+
+[rank2]:[W510 04:55:44.588012455 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 2] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank2]:[W510 04:55:45.588163182 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn74.local.snellius.surf.nl]:52154, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15302b86dfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15306f9fb6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15306f9fa1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15302cadceec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15316bf46164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15317c9b819a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15317ca3d100 in /lib64/libc.so.6)
+
+[rank2]:[W510 04:55:45.590550266 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 2] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank2]:[W510 04:55:46.590697632 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn74.local.snellius.surf.nl]:52154, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15302b86dfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15306f9fb6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15306f9fa1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15302cadceec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15316bf46164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15317c9b819a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15317ca3d100 in /lib64/libc.so.6)
+
+[rank2]:[W510 04:55:46.593009957 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 2] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank2]:[W510 04:55:47.593147323 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn74.local.snellius.surf.nl]:52154, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15302b86dfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15306f9fb6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15306f9fa1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15302cadceec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15316bf46164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15317c9b819a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15317ca3d100 in /lib64/libc.so.6)
+
+[rank2]:[W510 04:55:47.595412808 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 2] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank2]:[W510 04:55:48.595557224 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn74.local.snellius.surf.nl]:52154, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15302b86dfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15306f9fb6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15306f9fa1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15302cadceec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15316bf46164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15317c9b819a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15317ca3d100 in /lib64/libc.so.6)
+
+[rank2]:[W510 04:55:48.597994379 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 2] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank2]:[W510 04:55:49.598151015 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn74.local.snellius.surf.nl]:52154, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15302b86dfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15306f9fb6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15306f9fa1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15302cadceec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15316bf46164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15317c9b819a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15317ca3d100 in /lib64/libc.so.6)
+
+[rank2]:[W510 04:55:49.600474449 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 2] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank2]:[W510 04:55:50.600604966 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn74.local.snellius.surf.nl]:52154, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15302b86dfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15306f9fb6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15306f9fa1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15302cadceec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15316bf46164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15317c9b819a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15317ca3d100 in /lib64/libc.so.6)
+
+[rank2]:[W510 04:55:50.602857501 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 2] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank2]:[W510 04:55:51.602998597 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn74.local.snellius.surf.nl]:52154, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15302b86dfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15306f9fb6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15306f9fa1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15302cadceec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15316bf46164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15317c9b819a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15317ca3d100 in /lib64/libc.so.6)
+
+[rank2]:[W510 04:55:51.605348641 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 2] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank2]:[W510 04:55:52.605494028 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn74.local.snellius.surf.nl]:52154, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15302b86dfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15306f9fb6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15306f9fa1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15302cadceec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15316bf46164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15317c9b819a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15317ca3d100 in /lib64/libc.so.6)
+
+[rank2]:[W510 04:55:52.607923172 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 2] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank2]:[W510 04:55:53.608068219 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn74.local.snellius.surf.nl]:52154, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15302b86dfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15306f9fb6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15306f9fa1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15302cadceec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15316bf46164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15317c9b819a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15317ca3d100 in /lib64/libc.so.6)
+
+[rank2]:[W510 04:55:53.610391043 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 2] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank2]:[W510 04:55:54.610520939 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn74.local.snellius.surf.nl]:52154, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15302b86dfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15306f9fb6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15306f9fa1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15302cadceec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15316bf46164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15317c9b819a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15317ca3d100 in /lib64/libc.so.6)
+
+[rank2]:[W510 04:55:54.612861984 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 2] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank2]:[W510 04:55:55.612992190 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn74.local.snellius.surf.nl]:52154, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15302b86dfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15306f9fb6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15306f9fa1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15302cadceec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15316bf46164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15317c9b819a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15317ca3d100 in /lib64/libc.so.6)
+
+[rank2]:[W510 04:55:55.615379905 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 2] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank2]:[W510 04:55:56.615526691 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn74.local.snellius.surf.nl]:52154, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15302b86dfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15306f9fb6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15306f9fa1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15302cadceec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15316bf46164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15317c9b819a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15317ca3d100 in /lib64/libc.so.6)
+
+[rank2]:[W510 04:55:56.617919735 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 2] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank2]:[W510 04:55:57.618084692 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn74.local.snellius.surf.nl]:52154, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15302b86dfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15306f9fb6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15306f9fa1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15302cadceec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15316bf46164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15317c9b819a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15317ca3d100 in /lib64/libc.so.6)
+
+[rank2]:[W510 04:55:57.620404726 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 2] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank2]:[W510 04:55:58.620544253 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn74.local.snellius.surf.nl]:52154, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15302b86dfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15306f9fb6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15306f9fa1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15302cadceec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15316bf46164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15317c9b819a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15317ca3d100 in /lib64/libc.so.6)
+
+[rank2]:[W510 04:55:58.622884447 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 2] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
+  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
+[rank2]:[W510 05:02:24.914809986 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
+  scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+submitit WARNING (2026-05-10 09:26:31,103) - Bypassing signal SIGTERM
+submitit WARNING (2026-05-10 09:26:31,150) - Bypassing signal SIGCONT

vjepa/inflated/d003_lora_ema_capilvd1689m/job_22606429/22606429_2_log.out

@@ -0,0 +1,522 @@
+submitit INFO (2026-05-09 16:57:36,581) - Starting with JobEnvironment(job_id=22606429, hostname=gcn74.local.snellius.surf.nl, local_rank=2(4), node=0(4), global_rank=2(16))
+submitit INFO (2026-05-09 16:57:36,581) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/job_22606429/22606429_submitted.pkl
+INFO:root:loaded pretrain params...
+{   'app': 'vjepa',
+    'cpus_per_task': 16,
+    'data': {   'batch_size': 64,
+                'crop_size': 224,
+                'dataset_fpcs': [16, 16],
+                'dataset_type': 'VideoDataset',
+                'datasets': [   '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
+                                '/scratch-shared/dcanez/data/ssv2/train.csv'],
+                'datasets_weights': [0.65, 0.35],
+                'fps': 4,
+                'num_workers': 10,
+                'patch_size': 14,
+                'persistent_workers': True,
+                'pin_mem': True,
+                'stage': [   {   'dest': 'kinetics_240',
+                                 'format': 'targz_parts',
+                                 'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
+                             {   'dest': 'ssv2',
+                                 'format': 'multipart_tar',
+                                 'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
+                'tubelet_size': 1},
+    'data_aug': {   'auto_augment': False,
+                    'motion_shift': False,
+                    'random_resize_aspect_ratio': [0.75, 1.35],
+                    'random_resize_scale': [0.3, 1.0],
+                    'reprob': 0.0},
+    'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m',
+    'loss': {'loss_exp': 1.0},
+    'mask': [   {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 8,
+                    'spatial_scale': [0.15, 0.15],
+                    'temporal_scale': [1.0, 1.0]},
+                {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 2,
+                    'spatial_scale': [0.7, 0.7],
+                    'temporal_scale': [1.0, 1.0]}],
+    'mem_per_gpu': '180G',
+    'meta': {   'dtype': 'bfloat16',
+                'knn_eval_epoch0': False,
+                'knn_eval_freq': 5,
+                'knn_eval_presets': [   {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
+                                                          'eval_videos_per_class': 25,
+                                                          'num_workers': 8,
+                                                          'train_videos_per_class': 100},
+                                            'preset': 'ucf101'},
+                                        {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
+                                                          'eval_videos_per_class': 100,
+                                                          'linear_probe': True,
+                                                          'num_workers': 8,
+                                                          'pool_type': 'temporal_concat',
+                                                          'train_videos_per_class': 500},
+                                            'preset': 'ssv2_coarse10'}],
+                'load_checkpoint': True,
+                'read_checkpoint': None,
+                'save_every_freq': 5,
+                'seed': 239,
+                'use_sdpa': True,
+                'use_wandb': True,
+                'wandb_project': 'vjepa_ablation'},
+    'metrics': {'sigreg': {}, 'std': {}},
+    'model': {   'class_token': False,
+                 'freeze_backbone': True,
+                 'is_causal': False,
+                 'lora_alpha': 16,
+                 'lora_rank': 8,
+                 'lora_targets': ['attn.qkv', 'attn.proj'],
+                 'model_name': 'vit_large_patch14_capi_lvd1689m',
+                 'num_reg_tokens': 16,
+                 'pred_depth': 6,
+                 'pred_embed_dim': 384,
+                 'pred_num_heads': 12,
+                 'rope_type': 'capi2d',
+                 'stem_type': '2d',
+                 'target_kind': 'ema',
+                 'uniform_power': True,
+                 'use_activation_checkpointing': True,
+                 'use_lora': True,
+                 'use_mask_tokens': True,
+                 'use_rope': True,
+                 'use_sdpa': True,
+                 'zero_init_mask_tokens': True},
+    'nodes': 4,
+    'optimization': {   'ema': [0.99925, 0.99925],
+                        'epochs': 100,
+                        'final_lr': 0.0001,
+                        'final_weight_decay': 0.04,
+                        'ipe': 300,
+                        'ipe_scale': 1.0,
+                        'lr': 0.0005,
+                        'start_lr': 0.0001,
+                        'warmup': 10,
+                        'weight_decay': 0.04},
+    'tasks_per_node': 4}
+INFO:root:Running pre-training of app: vjepa
+[INFO    ][2026-05-09 16:58:54][app.vjepa.train     ][main                     ] which_dtype='bfloat16'
+[INFO    ][2026-05-09 16:58:54][app.vjepa.train     ][main                     ] Disabling persistent_workers (incompatible with KNN eval)
+[INFO    ][2026-05-09 16:58:54][app.vjepa.train     ][main                     ] NCCL_SOCKET_IFNAME=eno
+[INFO    ][2026-05-09 16:58:57][app.vjepa.train     ][main                     ] Initialized (rank/world-size) 2/16, tasks_per_node=4
+[INFO    ][2026-05-09 16:58:57][root                ][stage_datasets           ] [local_rank 2/4] Staging kinetics_240 (targz_parts)
+[INFO    ][2026-05-09 16:58:57][root                ][_stage_targz_parts       ] [rank 2] Extracting 26/103 tar.gz parts to /scratch-node/dcanez.22606429/kinetics_240
+[INFO    ][2026-05-09 16:59:11][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 2/26 parts
+[INFO    ][2026-05-09 16:59:25][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 4/26 parts
+[INFO    ][2026-05-09 16:59:39][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 6/26 parts
+[INFO    ][2026-05-09 16:59:54][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 8/26 parts
+[INFO    ][2026-05-09 17:00:08][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 10/26 parts
+[INFO    ][2026-05-09 17:00:22][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 12/26 parts
+[INFO    ][2026-05-09 17:00:36][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 14/26 parts
+[INFO    ][2026-05-09 17:00:50][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 16/26 parts
+[INFO    ][2026-05-09 17:01:05][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 18/26 parts
+[INFO    ][2026-05-09 17:01:19][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 20/26 parts
+[INFO    ][2026-05-09 17:01:34][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 22/26 parts
+[INFO    ][2026-05-09 17:01:48][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 24/26 parts
+[INFO    ][2026-05-09 17:02:02][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 26/26 parts
+[INFO    ][2026-05-09 17:02:02][root                ][stage_datasets           ] [local_rank 2/4] Staging ssv2 (multipart_tar)
+[INFO    ][2026-05-09 17:03:31][app.vjepa.train     ][main                     ] Staged datasets: ['/scratch-node/dcanez.22606429/kinetics_240/train.csv', '/scratch-node/dcanez.22606429/ssv2/train.csv']
+[INFO    ][2026-05-09 17:05:15][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
+[INFO    ][2026-05-09 17:05:19][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA applied: 48 modules replaced (rank=8, alpha=16, dropout=0.0, targets=['attn.qkv', 'attn.proj'], scope=blocks)
+[INFO    ][2026-05-09 17:05:19][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA trainable params: 1,179,648
+[INFO    ][2026-05-09 17:05:19][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-09 17:05:19][root                ][init_video_model         ] VJEPAPredictorMultiSeqWrapper(
+  (backbone): VisionTransformerPredictor(
+    (predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
+    (mask_tokens): ParameterList(
+        (0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+    )
+    (predictor_blocks): ModuleList(
+      (0-5): 6 x Block(
+        (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (attn): RoPEAttention(
+          (qkv): Linear(in_features=384, out_features=1152, bias=True)
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): Linear(in_features=384, out_features=384, bias=True)
+          (proj_drop): Dropout(p=0.0, inplace=False)
+        )
+        (drop_path): Identity()
+        (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (mlp): MLP(
+          (fc1): Linear(in_features=384, out_features=1536, bias=True)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=1536, out_features=384, bias=True)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+    (predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
+  )
+)
+[INFO    ][2026-05-09 17:05:19][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-09 17:05:19][root                ][init_video_model         ] Encoder number of parameters: 1179648
+[INFO    ][2026-05-09 17:05:19][root                ][init_video_model         ] Predictor number of parameters: 11436928
+[INFO    ][2026-05-09 17:05:19][root                ][init_video_model         ] Target encoder number of parameters: 1179648
+[INFO    ][2026-05-09 17:05:21][root                ][make_videodataset        ] VideoDataset dataset created
+[INFO    ][2026-05-09 17:05:21][WeightedSampler     ][__init__                 ] Using DistributedWeightedSampler with rank 2 / 16
+[INFO    ][2026-05-09 17:05:21][root                ][make_videodataset        ] VideoDataset unsupervised data loader created
+[INFO    ][2026-05-09 17:05:21][app.vjepa.train     ][main                     ] iterations per epoch/dataset length: 300/399
+[INFO    ][2026-05-09 17:05:21][app.vjepa.train     ][main                     ] Wrapping models in DDP (rank 2)...
+[INFO    ][2026-05-09 17:05:26][app.vjepa.train     ][main                     ] Initializing loader...
+submitit INFO (2026-05-10 04:55:29,226) - Job has timed out. Ran 718 minutes out of requested 720 minutes.
+[INFO    ][2026-05-10 04:55:29][submitit            ][has_timed_out            ] Job has timed out. Ran 718 minutes out of requested 720 minutes.
+submitit WARNING (2026-05-10 04:55:29,245) - Caught signal SIGUSR2 on gcn74.local.snellius.surf.nl: this job is timed-out.
+[WARNING ][2026-05-10 04:55:29][submitit            ][checkpoint_and_try_requeue] Caught signal SIGUSR2 on gcn74.local.snellius.surf.nl: this job is timed-out.
+submitit INFO (2026-05-10 04:55:29,245) - Not checkpointing nor requeuing since I am a slave (procid=2).
+[INFO    ][2026-05-10 04:55:29][submitit            ][checkpoint_and_try_requeue] Not checkpointing nor requeuing since I am a slave (procid=2).
+submitit WARNING (2026-05-10 04:55:29,245) - Bypassing signal SIGCONT
+[WARNING ][2026-05-10 04:55:29][submitit            ][bypass                   ] Bypassing signal SIGCONT
+submitit WARNING (2026-05-10 04:55:29,547) - Bypassing signal SIGTERM
+[WARNING ][2026-05-10 04:55:29][submitit            ][bypass                   ] Bypassing signal SIGTERM
+submitit WARNING (2026-05-10 04:55:29,547) - Bypassing signal SIGCONT
+[WARNING ][2026-05-10 04:55:29][submitit            ][bypass                   ] Bypassing signal SIGCONT
+submitit INFO (2026-05-10 04:59:12,566) - Starting with JobEnvironment(job_id=22606429, hostname=gcn74.local.snellius.surf.nl, local_rank=2(4), node=0(4), global_rank=2(16))
+submitit INFO (2026-05-10 04:59:12,566) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/job_22606429/22606429_submitted.pkl
+INFO:root:loaded pretrain params...
+{   'app': 'vjepa',
+    'cpus_per_task': 16,
+    'data': {   'batch_size': 64,
+                'crop_size': 224,
+                'dataset_fpcs': [16, 16],
+                'dataset_type': 'VideoDataset',
+                'datasets': [   '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
+                                '/scratch-shared/dcanez/data/ssv2/train.csv'],
+                'datasets_weights': [0.65, 0.35],
+                'fps': 4,
+                'num_workers': 10,
+                'patch_size': 14,
+                'persistent_workers': True,
+                'pin_mem': True,
+                'stage': [   {   'dest': 'kinetics_240',
+                                 'format': 'targz_parts',
+                                 'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
+                             {   'dest': 'ssv2',
+                                 'format': 'multipart_tar',
+                                 'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
+                'tubelet_size': 1},
+    'data_aug': {   'auto_augment': False,
+                    'motion_shift': False,
+                    'random_resize_aspect_ratio': [0.75, 1.35],
+                    'random_resize_scale': [0.3, 1.0],
+                    'reprob': 0.0},
+    'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m',
+    'loss': {'loss_exp': 1.0},
+    'mask': [   {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 8,
+                    'spatial_scale': [0.15, 0.15],
+                    'temporal_scale': [1.0, 1.0]},
+                {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 2,
+                    'spatial_scale': [0.7, 0.7],
+                    'temporal_scale': [1.0, 1.0]}],
+    'mem_per_gpu': '180G',
+    'meta': {   'dtype': 'bfloat16',
+                'knn_eval_epoch0': False,
+                'knn_eval_freq': 5,
+                'knn_eval_presets': [   {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
+                                                          'eval_videos_per_class': 25,
+                                                          'num_workers': 8,
+                                                          'train_videos_per_class': 100},
+                                            'preset': 'ucf101'},
+                                        {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
+                                                          'eval_videos_per_class': 100,
+                                                          'linear_probe': True,
+                                                          'num_workers': 8,
+                                                          'pool_type': 'temporal_concat',
+                                                          'train_videos_per_class': 500},
+                                            'preset': 'ssv2_coarse10'}],
+                'load_checkpoint': True,
+                'read_checkpoint': None,
+                'save_every_freq': 5,
+                'seed': 239,
+                'use_sdpa': True,
+                'use_wandb': True,
+                'wandb_project': 'vjepa_ablation'},
+    'metrics': {'sigreg': {}, 'std': {}},
+    'model': {   'class_token': False,
+                 'freeze_backbone': True,
+                 'is_causal': False,
+                 'lora_alpha': 16,
+                 'lora_rank': 8,
+                 'lora_targets': ['attn.qkv', 'attn.proj'],
+                 'model_name': 'vit_large_patch14_capi_lvd1689m',
+                 'num_reg_tokens': 16,
+                 'pred_depth': 6,
+                 'pred_embed_dim': 384,
+                 'pred_num_heads': 12,
+                 'rope_type': 'capi2d',
+                 'stem_type': '2d',
+                 'target_kind': 'ema',
+                 'uniform_power': True,
+                 'use_activation_checkpointing': True,
+                 'use_lora': True,
+                 'use_mask_tokens': True,
+                 'use_rope': True,
+                 'use_sdpa': True,
+                 'zero_init_mask_tokens': True},
+    'nodes': 4,
+    'optimization': {   'ema': [0.99925, 0.99925],
+                        'epochs': 100,
+                        'final_lr': 0.0001,
+                        'final_weight_decay': 0.04,
+                        'ipe': 300,
+                        'ipe_scale': 1.0,
+                        'lr': 0.0005,
+                        'start_lr': 0.0001,
+                        'warmup': 10,
+                        'weight_decay': 0.04},
+    'tasks_per_node': 4}
+INFO:root:Running pre-training of app: vjepa
+[INFO    ][2026-05-10 04:59:18][app.vjepa.train     ][main                     ] which_dtype='bfloat16'
+[INFO    ][2026-05-10 04:59:18][app.vjepa.train     ][main                     ] Disabling persistent_workers (incompatible with KNN eval)
+[INFO    ][2026-05-10 04:59:18][app.vjepa.train     ][main                     ] NCCL_SOCKET_IFNAME=eno
+[INFO    ][2026-05-10 04:59:20][app.vjepa.train     ][main                     ] Initialized (rank/world-size) 2/16, tasks_per_node=4
+[INFO    ][2026-05-10 04:59:20][root                ][stage_datasets           ] [local_rank 2/4] Staging kinetics_240 (targz_parts)
+[INFO    ][2026-05-10 04:59:20][root                ][_stage_targz_parts       ] [rank 2] Extracting 26/103 tar.gz parts to /scratch-node/dcanez.22606429/kinetics_240
+[INFO    ][2026-05-10 04:59:34][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 2/26 parts
+[INFO    ][2026-05-10 04:59:47][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 4/26 parts
+[INFO    ][2026-05-10 05:00:02][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 6/26 parts
+[INFO    ][2026-05-10 05:00:17][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 8/26 parts
+[INFO    ][2026-05-10 05:00:31][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 10/26 parts
+[INFO    ][2026-05-10 05:00:45][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 12/26 parts
+[INFO    ][2026-05-10 05:00:58][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 14/26 parts
+[INFO    ][2026-05-10 05:01:13][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 16/26 parts
+[INFO    ][2026-05-10 05:01:27][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 18/26 parts
+[INFO    ][2026-05-10 05:01:41][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 20/26 parts
+[INFO    ][2026-05-10 05:01:56][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 22/26 parts
+[INFO    ][2026-05-10 05:02:10][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 24/26 parts
+[INFO    ][2026-05-10 05:02:24][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 26/26 parts
+[INFO    ][2026-05-10 05:02:24][root                ][stage_datasets           ] [local_rank 2/4] Staging ssv2 (multipart_tar)
+[INFO    ][2026-05-10 05:04:33][app.vjepa.train     ][main                     ] Staged datasets: ['/scratch-node/dcanez.22606429/kinetics_240/train.csv', '/scratch-node/dcanez.22606429/ssv2/train.csv']
+[INFO    ][2026-05-10 05:05:22][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
+[INFO    ][2026-05-10 05:05:26][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA applied: 48 modules replaced (rank=8, alpha=16, dropout=0.0, targets=['attn.qkv', 'attn.proj'], scope=blocks)
+[INFO    ][2026-05-10 05:05:26][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA trainable params: 1,179,648
+[INFO    ][2026-05-10 05:05:27][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-10 05:05:27][root                ][init_video_model         ] VJEPAPredictorMultiSeqWrapper(
+  (backbone): VisionTransformerPredictor(
+    (predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
+    (mask_tokens): ParameterList(
+        (0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+    )
+    (predictor_blocks): ModuleList(
+      (0-5): 6 x Block(
+        (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (attn): RoPEAttention(
+          (qkv): Linear(in_features=384, out_features=1152, bias=True)
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): Linear(in_features=384, out_features=384, bias=True)
+          (proj_drop): Dropout(p=0.0, inplace=False)
+        )
+        (drop_path): Identity()
+        (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (mlp): MLP(
+          (fc1): Linear(in_features=384, out_features=1536, bias=True)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=1536, out_features=384, bias=True)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+    (predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
+  )
+)
+[INFO    ][2026-05-10 05:05:27][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-10 05:05:27][root                ][init_video_model         ] Encoder number of parameters: 1179648
+[INFO    ][2026-05-10 05:05:27][root                ][init_video_model         ] Predictor number of parameters: 11436928
+[INFO    ][2026-05-10 05:05:27][root                ][init_video_model         ] Target encoder number of parameters: 1179648
+[INFO    ][2026-05-10 05:05:28][root                ][make_videodataset        ] VideoDataset dataset created
+[INFO    ][2026-05-10 05:05:28][WeightedSampler     ][__init__                 ] Using DistributedWeightedSampler with rank 2 / 16
+[INFO    ][2026-05-10 05:05:28][root                ][make_videodataset        ] VideoDataset unsupervised data loader created
+[INFO    ][2026-05-10 05:05:28][app.vjepa.train     ][main                     ] iterations per epoch/dataset length: 300/399
+[INFO    ][2026-05-10 05:05:28][app.vjepa.train     ][main                     ] Wrapping models in DDP (rank 2)...
+[INFO    ][2026-05-10 05:06:08][root                ][load_checkpoint          ] Loading checkpoint from /scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/latest.pt
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] loaded pretrained encoder from epoch 33 with msg: <All keys matched successfully>
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] loaded pretrained predictor from epoch 33 with msg: <All keys matched successfully>
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] loaded pretrained target encoder from epoch 33 with msg: <All keys matched successfully>
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] loaded optimizers from epoch 33
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] read-path: /scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/latest.pt
+[INFO    ][2026-05-10 05:06:09][app.vjepa.train     ][main                     ] Initializing loader...
+submitit WARNING (2026-05-10 09:26:31,103) - Bypassing signal SIGTERM
+[WARNING ][2026-05-10 09:26:31][submitit            ][bypass                   ] Bypassing signal SIGTERM
+submitit WARNING (2026-05-10 09:26:31,150) - Bypassing signal SIGCONT
+[WARNING ][2026-05-10 09:26:31][submitit            ][bypass                   ] Bypassing signal SIGCONT

vjepa/inflated/d003_lora_ema_capilvd1689m/job_22606429/22606429_3_log.err

@@ -0,0 +1,313 @@
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
+  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
+[rank3]:[W509 17:01:56.086943966 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
+  scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+submitit WARNING (2026-05-10 04:55:29,308) - Caught signal SIGUSR2 on gcn74.local.snellius.surf.nl: this job is timed-out.
+submitit WARNING (2026-05-10 04:55:29,308) - Bypassing signal SIGCONT
+submitit WARNING (2026-05-10 04:55:29,547) - Bypassing signal SIGTERM
+submitit WARNING (2026-05-10 04:55:29,547) - Bypassing signal SIGCONT
+[rank3]:[W510 04:55:34.495110837 TCPStore.cpp:125] [c10d] recvValue failed on SocketImpl(fd=4, addr=[gcn74.local.snellius.surf.nl]:52158, remote=[gcn74.local.snellius.surf.nl]:37129): Failed to recv, got 0 bytes. Connection was likely closed. Did the remote server shutdown or crash?
+Exception raised from recvBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:682 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x146d9da38fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a3325d (0x146de1bc725d in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x273 (0x146de1bc51f3 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x146d9eca7eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x146ede111164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x146eeeb8319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x146eeec08100 in /lib64/libc.so.6)
+
+[rank3]:[W510 04:55:34.498257870 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 3] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Failed to recv, got 0 bytes. Connection was likely closed. Did the remote server shutdown or crash?
+[rank3]:[W510 04:55:35.498414636 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn74.local.snellius.surf.nl]:52158, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x146d9da38fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x146de1bc66d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x146de1bc51cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x146d9eca7eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x146ede111164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x146eeeb8319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x146eeec08100 in /lib64/libc.so.6)
+
+[rank3]:[W510 04:55:35.500757561 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 3] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank3]:[W510 04:55:36.500911207 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn74.local.snellius.surf.nl]:52158, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x146d9da38fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x146de1bc66d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x146de1bc51cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x146d9eca7eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x146ede111164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x146eeeb8319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x146eeec08100 in /lib64/libc.so.6)
+
+[rank3]:[W510 04:55:36.503237612 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 3] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank3]:[W510 04:55:37.503382708 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn74.local.snellius.surf.nl]:52158, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x146d9da38fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x146de1bc66d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x146de1bc51cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x146d9eca7eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x146ede111164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x146eeeb8319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x146eeec08100 in /lib64/libc.so.6)
+
+[rank3]:[W510 04:55:37.505652773 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 3] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank3]:[W510 04:55:38.505790159 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn74.local.snellius.surf.nl]:52158, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x146d9da38fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x146de1bc66d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x146de1bc51cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x146d9eca7eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x146ede111164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x146eeeb8319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x146eeec08100 in /lib64/libc.so.6)
+
+[rank3]:[W510 04:55:38.508070124 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 3] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank3]:[W510 04:55:39.508213330 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn74.local.snellius.surf.nl]:52158, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x146d9da38fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x146de1bc66d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x146de1bc51cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x146d9eca7eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x146ede111164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x146eeeb8319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x146eeec08100 in /lib64/libc.so.6)
+
+[rank3]:[W510 04:55:39.510525225 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 3] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank3]:[W510 04:55:40.510666701 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn74.local.snellius.surf.nl]:52158, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x146d9da38fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x146de1bc66d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x146de1bc51cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x146d9eca7eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x146ede111164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x146eeeb8319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x146eeec08100 in /lib64/libc.so.6)
+
+[rank3]:[W510 04:55:40.512981056 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 3] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank3]:[W510 04:55:41.513124252 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn74.local.snellius.surf.nl]:52158, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x146d9da38fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x146de1bc66d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x146de1bc51cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x146d9eca7eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x146ede111164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x146eeeb8319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x146eeec08100 in /lib64/libc.so.6)
+
+[rank3]:[W510 04:55:41.515485896 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 3] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank3]:[W510 04:55:42.515614163 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn74.local.snellius.surf.nl]:52158, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x146d9da38fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x146de1bc66d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x146de1bc51cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x146d9eca7eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x146ede111164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x146eeeb8319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x146eeec08100 in /lib64/libc.so.6)
+
+[rank3]:[W510 04:55:42.517822918 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 3] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank3]:[W510 04:55:43.517960224 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn74.local.snellius.surf.nl]:52158, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x146d9da38fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x146de1bc66d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x146de1bc51cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x146d9eca7eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x146ede111164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x146eeeb8319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x146eeec08100 in /lib64/libc.so.6)
+
+[rank3]:[W510 04:55:43.520285159 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 3] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank3]:[W510 04:55:44.520408995 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn74.local.snellius.surf.nl]:52158, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x146d9da38fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x146de1bc66d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x146de1bc51cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x146d9eca7eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x146ede111164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x146eeeb8319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x146eeec08100 in /lib64/libc.so.6)
+
+[rank3]:[W510 04:55:44.522721510 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 3] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank3]:[W510 04:55:45.522846026 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn74.local.snellius.surf.nl]:52158, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x146d9da38fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x146de1bc66d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x146de1bc51cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x146d9eca7eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x146ede111164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x146eeeb8319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x146eeec08100 in /lib64/libc.so.6)
+
+[rank3]:[W510 04:55:45.525223370 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 3] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank3]:[W510 04:55:46.525358957 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn74.local.snellius.surf.nl]:52158, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x146d9da38fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x146de1bc66d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x146de1bc51cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x146d9eca7eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x146ede111164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x146eeeb8319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x146eeec08100 in /lib64/libc.so.6)
+
+[rank3]:[W510 04:55:46.527741071 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 3] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank3]:[W510 04:55:47.527874207 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn74.local.snellius.surf.nl]:52158, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x146d9da38fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x146de1bc66d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x146de1bc51cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x146d9eca7eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x146ede111164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x146eeeb8319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x146eeec08100 in /lib64/libc.so.6)
+
+[rank3]:[W510 04:55:47.530236332 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 3] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank3]:[W510 04:55:48.530371908 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn74.local.snellius.surf.nl]:52158, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x146d9da38fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x146de1bc66d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x146de1bc51cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x146d9eca7eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x146ede111164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x146eeeb8319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x146eeec08100 in /lib64/libc.so.6)
+
+[rank3]:[W510 04:55:48.532747463 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 3] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank3]:[W510 04:55:49.532882039 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn74.local.snellius.surf.nl]:52158, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x146d9da38fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x146de1bc66d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x146de1bc51cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x146d9eca7eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x146ede111164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x146eeeb8319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x146eeec08100 in /lib64/libc.so.6)
+
+[rank3]:[W510 04:55:49.535247864 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 3] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank3]:[W510 04:55:50.535369700 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn74.local.snellius.surf.nl]:52158, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x146d9da38fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x146de1bc66d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x146de1bc51cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x146d9eca7eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x146ede111164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x146eeeb8319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x146eeec08100 in /lib64/libc.so.6)
+
+[rank3]:[W510 04:55:50.537759174 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 3] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank3]:[W510 04:55:51.537882041 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn74.local.snellius.surf.nl]:52158, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x146d9da38fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x146de1bc66d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x146de1bc51cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x146d9eca7eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x146ede111164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x146eeeb8319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x146eeec08100 in /lib64/libc.so.6)
+
+[rank3]:[W510 04:55:51.540163255 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 3] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank3]:[W510 04:55:52.540288062 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn74.local.snellius.surf.nl]:52158, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x146d9da38fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x146de1bc66d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x146de1bc51cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x146d9eca7eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x146ede111164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x146eeeb8319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x146eeec08100 in /lib64/libc.so.6)
+
+[rank3]:[W510 04:55:52.542647336 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 3] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank3]:[W510 04:55:53.542769903 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn74.local.snellius.surf.nl]:52158, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x146d9da38fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x146de1bc66d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x146de1bc51cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x146d9eca7eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x146ede111164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x146eeeb8319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x146eeec08100 in /lib64/libc.so.6)
+
+[rank3]:[W510 04:55:53.545099747 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 3] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank3]:[W510 04:55:54.545212684 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn74.local.snellius.surf.nl]:52158, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x146d9da38fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x146de1bc66d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x146de1bc51cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x146d9eca7eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x146ede111164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x146eeeb8319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x146eeec08100 in /lib64/libc.so.6)
+
+[rank3]:[W510 04:55:54.547539408 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 3] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank3]:[W510 04:55:55.547675665 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn74.local.snellius.surf.nl]:52158, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x146d9da38fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x146de1bc66d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x146de1bc51cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x146d9eca7eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x146ede111164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x146eeeb8319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x146eeec08100 in /lib64/libc.so.6)
+
+[rank3]:[W510 04:55:55.550020529 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 3] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank3]:[W510 04:55:56.550145366 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn74.local.snellius.surf.nl]:52158, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x146d9da38fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x146de1bc66d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x146de1bc51cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x146d9eca7eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x146ede111164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x146eeeb8319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x146eeec08100 in /lib64/libc.so.6)
+
+[rank3]:[W510 04:55:56.552428670 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 3] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank3]:[W510 04:55:57.552550107 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn74.local.snellius.surf.nl]:52158, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x146d9da38fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x146de1bc66d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x146de1bc51cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x146d9eca7eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x146ede111164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x146eeeb8319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x146eeec08100 in /lib64/libc.so.6)
+
+[rank3]:[W510 04:55:57.554850521 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 3] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank3]:[W510 04:55:58.554973978 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn74.local.snellius.surf.nl]:52158, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x146d9da38fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x146de1bc66d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x146de1bc51cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x146d9eca7eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x146ede111164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x146eeeb8319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x146eeec08100 in /lib64/libc.so.6)
+
+[rank3]:[W510 04:55:58.557345232 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 3] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
+  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
+[rank3]:[W510 05:02:19.994442964 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
+  scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+submitit WARNING (2026-05-10 09:26:31,097) - Bypassing signal SIGTERM
+submitit WARNING (2026-05-10 09:26:31,113) - Bypassing signal SIGCONT

vjepa/inflated/d003_lora_ema_capilvd1689m/job_22606429/22606429_3_log.out

@@ -0,0 +1,522 @@
+submitit INFO (2026-05-09 16:57:36,581) - Starting with JobEnvironment(job_id=22606429, hostname=gcn74.local.snellius.surf.nl, local_rank=3(4), node=0(4), global_rank=3(16))
+submitit INFO (2026-05-09 16:57:36,581) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/job_22606429/22606429_submitted.pkl
+INFO:root:loaded pretrain params...
+{   'app': 'vjepa',
+    'cpus_per_task': 16,
+    'data': {   'batch_size': 64,
+                'crop_size': 224,
+                'dataset_fpcs': [16, 16],
+                'dataset_type': 'VideoDataset',
+                'datasets': [   '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
+                                '/scratch-shared/dcanez/data/ssv2/train.csv'],
+                'datasets_weights': [0.65, 0.35],
+                'fps': 4,
+                'num_workers': 10,
+                'patch_size': 14,
+                'persistent_workers': True,
+                'pin_mem': True,
+                'stage': [   {   'dest': 'kinetics_240',
+                                 'format': 'targz_parts',
+                                 'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
+                             {   'dest': 'ssv2',
+                                 'format': 'multipart_tar',
+                                 'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
+                'tubelet_size': 1},
+    'data_aug': {   'auto_augment': False,
+                    'motion_shift': False,
+                    'random_resize_aspect_ratio': [0.75, 1.35],
+                    'random_resize_scale': [0.3, 1.0],
+                    'reprob': 0.0},
+    'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m',
+    'loss': {'loss_exp': 1.0},
+    'mask': [   {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 8,
+                    'spatial_scale': [0.15, 0.15],
+                    'temporal_scale': [1.0, 1.0]},
+                {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 2,
+                    'spatial_scale': [0.7, 0.7],
+                    'temporal_scale': [1.0, 1.0]}],
+    'mem_per_gpu': '180G',
+    'meta': {   'dtype': 'bfloat16',
+                'knn_eval_epoch0': False,
+                'knn_eval_freq': 5,
+                'knn_eval_presets': [   {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
+                                                          'eval_videos_per_class': 25,
+                                                          'num_workers': 8,
+                                                          'train_videos_per_class': 100},
+                                            'preset': 'ucf101'},
+                                        {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
+                                                          'eval_videos_per_class': 100,
+                                                          'linear_probe': True,
+                                                          'num_workers': 8,
+                                                          'pool_type': 'temporal_concat',
+                                                          'train_videos_per_class': 500},
+                                            'preset': 'ssv2_coarse10'}],
+                'load_checkpoint': True,
+                'read_checkpoint': None,
+                'save_every_freq': 5,
+                'seed': 239,
+                'use_sdpa': True,
+                'use_wandb': True,
+                'wandb_project': 'vjepa_ablation'},
+    'metrics': {'sigreg': {}, 'std': {}},
+    'model': {   'class_token': False,
+                 'freeze_backbone': True,
+                 'is_causal': False,
+                 'lora_alpha': 16,
+                 'lora_rank': 8,
+                 'lora_targets': ['attn.qkv', 'attn.proj'],
+                 'model_name': 'vit_large_patch14_capi_lvd1689m',
+                 'num_reg_tokens': 16,
+                 'pred_depth': 6,
+                 'pred_embed_dim': 384,
+                 'pred_num_heads': 12,
+                 'rope_type': 'capi2d',
+                 'stem_type': '2d',
+                 'target_kind': 'ema',
+                 'uniform_power': True,
+                 'use_activation_checkpointing': True,
+                 'use_lora': True,
+                 'use_mask_tokens': True,
+                 'use_rope': True,
+                 'use_sdpa': True,
+                 'zero_init_mask_tokens': True},
+    'nodes': 4,
+    'optimization': {   'ema': [0.99925, 0.99925],
+                        'epochs': 100,
+                        'final_lr': 0.0001,
+                        'final_weight_decay': 0.04,
+                        'ipe': 300,
+                        'ipe_scale': 1.0,
+                        'lr': 0.0005,
+                        'start_lr': 0.0001,
+                        'warmup': 10,
+                        'weight_decay': 0.04},
+    'tasks_per_node': 4}
+INFO:root:Running pre-training of app: vjepa
+[INFO    ][2026-05-09 16:58:54][app.vjepa.train     ][main                     ] which_dtype='bfloat16'
+[INFO    ][2026-05-09 16:58:54][app.vjepa.train     ][main                     ] Disabling persistent_workers (incompatible with KNN eval)
+[INFO    ][2026-05-09 16:58:54][app.vjepa.train     ][main                     ] NCCL_SOCKET_IFNAME=eno
+[INFO    ][2026-05-09 16:58:56][app.vjepa.train     ][main                     ] Initialized (rank/world-size) 3/16, tasks_per_node=4
+[INFO    ][2026-05-09 16:58:56][root                ][stage_datasets           ] [local_rank 3/4] Staging kinetics_240 (targz_parts)
+[INFO    ][2026-05-09 16:58:56][root                ][_stage_targz_parts       ] [rank 3] Extracting 25/103 tar.gz parts to /scratch-node/dcanez.22606429/kinetics_240
+[INFO    ][2026-05-09 16:59:11][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 2/25 parts
+[INFO    ][2026-05-09 16:59:25][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 4/25 parts
+[INFO    ][2026-05-09 16:59:39][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 6/25 parts
+[INFO    ][2026-05-09 16:59:54][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 8/25 parts
+[INFO    ][2026-05-09 17:00:08][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 10/25 parts
+[INFO    ][2026-05-09 17:00:22][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 12/25 parts
+[INFO    ][2026-05-09 17:00:37][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 14/25 parts
+[INFO    ][2026-05-09 17:00:51][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 16/25 parts
+[INFO    ][2026-05-09 17:01:05][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 18/25 parts
+[INFO    ][2026-05-09 17:01:19][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 20/25 parts
+[INFO    ][2026-05-09 17:01:34][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 22/25 parts
+[INFO    ][2026-05-09 17:01:48][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 24/25 parts
+[INFO    ][2026-05-09 17:01:55][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 25/25 parts
+[INFO    ][2026-05-09 17:01:55][root                ][stage_datasets           ] [local_rank 3/4] Staging ssv2 (multipart_tar)
+[INFO    ][2026-05-09 17:03:31][app.vjepa.train     ][main                     ] Staged datasets: ['/scratch-node/dcanez.22606429/kinetics_240/train.csv', '/scratch-node/dcanez.22606429/ssv2/train.csv']
+[INFO    ][2026-05-09 17:05:15][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
+[INFO    ][2026-05-09 17:05:19][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA applied: 48 modules replaced (rank=8, alpha=16, dropout=0.0, targets=['attn.qkv', 'attn.proj'], scope=blocks)
+[INFO    ][2026-05-09 17:05:19][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA trainable params: 1,179,648
+[INFO    ][2026-05-09 17:05:19][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-09 17:05:19][root                ][init_video_model         ] VJEPAPredictorMultiSeqWrapper(
+  (backbone): VisionTransformerPredictor(
+    (predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
+    (mask_tokens): ParameterList(
+        (0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+    )
+    (predictor_blocks): ModuleList(
+      (0-5): 6 x Block(
+        (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (attn): RoPEAttention(
+          (qkv): Linear(in_features=384, out_features=1152, bias=True)
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): Linear(in_features=384, out_features=384, bias=True)
+          (proj_drop): Dropout(p=0.0, inplace=False)
+        )
+        (drop_path): Identity()
+        (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (mlp): MLP(
+          (fc1): Linear(in_features=384, out_features=1536, bias=True)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=1536, out_features=384, bias=True)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+    (predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
+  )
+)
+[INFO    ][2026-05-09 17:05:19][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-09 17:05:19][root                ][init_video_model         ] Encoder number of parameters: 1179648
+[INFO    ][2026-05-09 17:05:19][root                ][init_video_model         ] Predictor number of parameters: 11436928
+[INFO    ][2026-05-09 17:05:19][root                ][init_video_model         ] Target encoder number of parameters: 1179648
+[INFO    ][2026-05-09 17:05:21][root                ][make_videodataset        ] VideoDataset dataset created
+[INFO    ][2026-05-09 17:05:21][WeightedSampler     ][__init__                 ] Using DistributedWeightedSampler with rank 3 / 16
+[INFO    ][2026-05-09 17:05:21][root                ][make_videodataset        ] VideoDataset unsupervised data loader created
+[INFO    ][2026-05-09 17:05:21][app.vjepa.train     ][main                     ] iterations per epoch/dataset length: 300/399
+[INFO    ][2026-05-09 17:05:21][app.vjepa.train     ][main                     ] Wrapping models in DDP (rank 3)...
+[INFO    ][2026-05-09 17:05:26][app.vjepa.train     ][main                     ] Initializing loader...
+submitit INFO (2026-05-10 04:55:29,195) - Job has timed out. Ran 718 minutes out of requested 720 minutes.
+[INFO    ][2026-05-10 04:55:29][submitit            ][has_timed_out            ] Job has timed out. Ran 718 minutes out of requested 720 minutes.
+submitit WARNING (2026-05-10 04:55:29,308) - Caught signal SIGUSR2 on gcn74.local.snellius.surf.nl: this job is timed-out.
+[WARNING ][2026-05-10 04:55:29][submitit            ][checkpoint_and_try_requeue] Caught signal SIGUSR2 on gcn74.local.snellius.surf.nl: this job is timed-out.
+submitit INFO (2026-05-10 04:55:29,308) - Not checkpointing nor requeuing since I am a slave (procid=3).
+[INFO    ][2026-05-10 04:55:29][submitit            ][checkpoint_and_try_requeue] Not checkpointing nor requeuing since I am a slave (procid=3).
+submitit WARNING (2026-05-10 04:55:29,308) - Bypassing signal SIGCONT
+[WARNING ][2026-05-10 04:55:29][submitit            ][bypass                   ] Bypassing signal SIGCONT
+submitit WARNING (2026-05-10 04:55:29,547) - Bypassing signal SIGTERM
+[WARNING ][2026-05-10 04:55:29][submitit            ][bypass                   ] Bypassing signal SIGTERM
+submitit WARNING (2026-05-10 04:55:29,547) - Bypassing signal SIGCONT
+[WARNING ][2026-05-10 04:55:29][submitit            ][bypass                   ] Bypassing signal SIGCONT
+submitit INFO (2026-05-10 04:59:12,566) - Starting with JobEnvironment(job_id=22606429, hostname=gcn74.local.snellius.surf.nl, local_rank=3(4), node=0(4), global_rank=3(16))
+submitit INFO (2026-05-10 04:59:12,566) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/job_22606429/22606429_submitted.pkl
+INFO:root:loaded pretrain params...
+{   'app': 'vjepa',
+    'cpus_per_task': 16,
+    'data': {   'batch_size': 64,
+                'crop_size': 224,
+                'dataset_fpcs': [16, 16],
+                'dataset_type': 'VideoDataset',
+                'datasets': [   '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
+                                '/scratch-shared/dcanez/data/ssv2/train.csv'],
+                'datasets_weights': [0.65, 0.35],
+                'fps': 4,
+                'num_workers': 10,
+                'patch_size': 14,
+                'persistent_workers': True,
+                'pin_mem': True,
+                'stage': [   {   'dest': 'kinetics_240',
+                                 'format': 'targz_parts',
+                                 'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
+                             {   'dest': 'ssv2',
+                                 'format': 'multipart_tar',
+                                 'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
+                'tubelet_size': 1},
+    'data_aug': {   'auto_augment': False,
+                    'motion_shift': False,
+                    'random_resize_aspect_ratio': [0.75, 1.35],
+                    'random_resize_scale': [0.3, 1.0],
+                    'reprob': 0.0},
+    'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m',
+    'loss': {'loss_exp': 1.0},
+    'mask': [   {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 8,
+                    'spatial_scale': [0.15, 0.15],
+                    'temporal_scale': [1.0, 1.0]},
+                {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 2,
+                    'spatial_scale': [0.7, 0.7],
+                    'temporal_scale': [1.0, 1.0]}],
+    'mem_per_gpu': '180G',
+    'meta': {   'dtype': 'bfloat16',
+                'knn_eval_epoch0': False,
+                'knn_eval_freq': 5,
+                'knn_eval_presets': [   {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
+                                                          'eval_videos_per_class': 25,
+                                                          'num_workers': 8,
+                                                          'train_videos_per_class': 100},
+                                            'preset': 'ucf101'},
+                                        {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
+                                                          'eval_videos_per_class': 100,
+                                                          'linear_probe': True,
+                                                          'num_workers': 8,
+                                                          'pool_type': 'temporal_concat',
+                                                          'train_videos_per_class': 500},
+                                            'preset': 'ssv2_coarse10'}],
+                'load_checkpoint': True,
+                'read_checkpoint': None,
+                'save_every_freq': 5,
+                'seed': 239,
+                'use_sdpa': True,
+                'use_wandb': True,
+                'wandb_project': 'vjepa_ablation'},
+    'metrics': {'sigreg': {}, 'std': {}},
+    'model': {   'class_token': False,
+                 'freeze_backbone': True,
+                 'is_causal': False,
+                 'lora_alpha': 16,
+                 'lora_rank': 8,
+                 'lora_targets': ['attn.qkv', 'attn.proj'],
+                 'model_name': 'vit_large_patch14_capi_lvd1689m',
+                 'num_reg_tokens': 16,
+                 'pred_depth': 6,
+                 'pred_embed_dim': 384,
+                 'pred_num_heads': 12,
+                 'rope_type': 'capi2d',
+                 'stem_type': '2d',
+                 'target_kind': 'ema',
+                 'uniform_power': True,
+                 'use_activation_checkpointing': True,
+                 'use_lora': True,
+                 'use_mask_tokens': True,
+                 'use_rope': True,
+                 'use_sdpa': True,
+                 'zero_init_mask_tokens': True},
+    'nodes': 4,
+    'optimization': {   'ema': [0.99925, 0.99925],
+                        'epochs': 100,
+                        'final_lr': 0.0001,
+                        'final_weight_decay': 0.04,
+                        'ipe': 300,
+                        'ipe_scale': 1.0,
+                        'lr': 0.0005,
+                        'start_lr': 0.0001,
+                        'warmup': 10,
+                        'weight_decay': 0.04},
+    'tasks_per_node': 4}
+INFO:root:Running pre-training of app: vjepa
+[INFO    ][2026-05-10 04:59:18][app.vjepa.train     ][main                     ] which_dtype='bfloat16'
+[INFO    ][2026-05-10 04:59:18][app.vjepa.train     ][main                     ] Disabling persistent_workers (incompatible with KNN eval)
+[INFO    ][2026-05-10 04:59:18][app.vjepa.train     ][main                     ] NCCL_SOCKET_IFNAME=eno
+[INFO    ][2026-05-10 04:59:20][app.vjepa.train     ][main                     ] Initialized (rank/world-size) 3/16, tasks_per_node=4
+[INFO    ][2026-05-10 04:59:20][root                ][stage_datasets           ] [local_rank 3/4] Staging kinetics_240 (targz_parts)
+[INFO    ][2026-05-10 04:59:20][root                ][_stage_targz_parts       ] [rank 3] Extracting 25/103 tar.gz parts to /scratch-node/dcanez.22606429/kinetics_240
+[INFO    ][2026-05-10 04:59:34][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 2/25 parts
+[INFO    ][2026-05-10 04:59:48][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 4/25 parts
+[INFO    ][2026-05-10 05:00:03][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 6/25 parts
+[INFO    ][2026-05-10 05:00:17][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 8/25 parts
+[INFO    ][2026-05-10 05:00:32][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 10/25 parts
+[INFO    ][2026-05-10 05:00:46][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 12/25 parts
+[INFO    ][2026-05-10 05:01:00][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 14/25 parts
+[INFO    ][2026-05-10 05:01:15][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 16/25 parts
+[INFO    ][2026-05-10 05:01:28][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 18/25 parts
+[INFO    ][2026-05-10 05:01:43][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 20/25 parts
+[INFO    ][2026-05-10 05:01:58][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 22/25 parts
+[INFO    ][2026-05-10 05:02:12][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 24/25 parts
+[INFO    ][2026-05-10 05:02:19][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 25/25 parts
+[INFO    ][2026-05-10 05:02:19][root                ][stage_datasets           ] [local_rank 3/4] Staging ssv2 (multipart_tar)
+[INFO    ][2026-05-10 05:04:33][app.vjepa.train     ][main                     ] Staged datasets: ['/scratch-node/dcanez.22606429/kinetics_240/train.csv', '/scratch-node/dcanez.22606429/ssv2/train.csv']
+[INFO    ][2026-05-10 05:05:22][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
+[INFO    ][2026-05-10 05:05:26][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA applied: 48 modules replaced (rank=8, alpha=16, dropout=0.0, targets=['attn.qkv', 'attn.proj'], scope=blocks)
+[INFO    ][2026-05-10 05:05:26][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA trainable params: 1,179,648
+[INFO    ][2026-05-10 05:05:27][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-10 05:05:27][root                ][init_video_model         ] VJEPAPredictorMultiSeqWrapper(
+  (backbone): VisionTransformerPredictor(
+    (predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
+    (mask_tokens): ParameterList(
+        (0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+    )
+    (predictor_blocks): ModuleList(
+      (0-5): 6 x Block(
+        (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (attn): RoPEAttention(
+          (qkv): Linear(in_features=384, out_features=1152, bias=True)
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): Linear(in_features=384, out_features=384, bias=True)
+          (proj_drop): Dropout(p=0.0, inplace=False)
+        )
+        (drop_path): Identity()
+        (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (mlp): MLP(
+          (fc1): Linear(in_features=384, out_features=1536, bias=True)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=1536, out_features=384, bias=True)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+    (predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
+  )
+)
+[INFO    ][2026-05-10 05:05:27][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-10 05:05:27][root                ][init_video_model         ] Encoder number of parameters: 1179648
+[INFO    ][2026-05-10 05:05:27][root                ][init_video_model         ] Predictor number of parameters: 11436928
+[INFO    ][2026-05-10 05:05:27][root                ][init_video_model         ] Target encoder number of parameters: 1179648
+[INFO    ][2026-05-10 05:05:28][root                ][make_videodataset        ] VideoDataset dataset created
+[INFO    ][2026-05-10 05:05:28][WeightedSampler     ][__init__                 ] Using DistributedWeightedSampler with rank 3 / 16
+[INFO    ][2026-05-10 05:05:28][root                ][make_videodataset        ] VideoDataset unsupervised data loader created
+[INFO    ][2026-05-10 05:05:28][app.vjepa.train     ][main                     ] iterations per epoch/dataset length: 300/399
+[INFO    ][2026-05-10 05:05:28][app.vjepa.train     ][main                     ] Wrapping models in DDP (rank 3)...
+[INFO    ][2026-05-10 05:06:08][root                ][load_checkpoint          ] Loading checkpoint from /scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/latest.pt
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] loaded pretrained encoder from epoch 33 with msg: <All keys matched successfully>
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] loaded pretrained predictor from epoch 33 with msg: <All keys matched successfully>
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] loaded pretrained target encoder from epoch 33 with msg: <All keys matched successfully>
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] loaded optimizers from epoch 33
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] read-path: /scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/latest.pt
+[INFO    ][2026-05-10 05:06:09][app.vjepa.train     ][main                     ] Initializing loader...
+submitit WARNING (2026-05-10 09:26:31,097) - Bypassing signal SIGTERM
+[WARNING ][2026-05-10 09:26:31][submitit            ][bypass                   ] Bypassing signal SIGTERM
+submitit WARNING (2026-05-10 09:26:31,113) - Bypassing signal SIGCONT
+[WARNING ][2026-05-10 09:26:31][submitit            ][bypass                   ] Bypassing signal SIGCONT

vjepa/inflated/d003_lora_ema_capilvd1689m/job_22606429/22606429_4_log.err

@@ -0,0 +1,314 @@
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
+  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
+[rank4]:[W509 17:03:12.718263143 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
+  scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+submitit WARNING (2026-05-10 04:55:29,326) - Caught signal SIGUSR2 on gcn75.local.snellius.surf.nl: this job is timed-out.
+submitit WARNING (2026-05-10 04:55:29,326) - Bypassing signal SIGCONT
+submitit WARNING (2026-05-10 04:55:29,547) - Bypassing signal SIGTERM
+submitit WARNING (2026-05-10 04:55:29,596) - Bypassing signal SIGCONT
+[rank4]:[W510 04:55:33.255921249 TCPStore.cpp:125] [c10d] recvValue failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:58522, remote=[gcn74.local.snellius.surf.nl]:37129): Failed to recv, got 0 bytes. Connection was likely closed. Did the remote server shutdown or crash?
+Exception raised from recvBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:682 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1461ffd33fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a3325d (0x146243ec225d in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x273 (0x146243ec01f3 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x146200fa2eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14634040c164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x146350e7e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x146350f03100 in /lib64/libc.so.6)
+
+[rank4]:[W510 04:55:33.258371367 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 4] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Failed to recv, got 0 bytes. Connection was likely closed. Did the remote server shutdown or crash?
+[rank4]:[W510 04:55:34.258504298 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:58522, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1461ffd33fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x146243ec16d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x146243ec01cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x146200fa2eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14634040c164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x146350e7e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x146350f03100 in /lib64/libc.so.6)
+
+[rank4]:[W510 04:55:34.261872331 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 4] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank4]:[W510 04:55:35.262012247 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:58522, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1461ffd33fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x146243ec16d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x146243ec01cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x146200fa2eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14634040c164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x146350e7e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x146350f03100 in /lib64/libc.so.6)
+
+[rank4]:[W510 04:55:35.264390915 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 4] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank4]:[W510 04:55:36.264515070 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:58522, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1461ffd33fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x146243ec16d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x146243ec01cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x146200fa2eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14634040c164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x146350e7e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x146350f03100 in /lib64/libc.so.6)
+
+[rank4]:[W510 04:55:36.266749349 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 4] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank4]:[W510 04:55:37.266869954 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:58522, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1461ffd33fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x146243ec16d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x146243ec01cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x146200fa2eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14634040c164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x146350e7e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x146350f03100 in /lib64/libc.so.6)
+
+[rank4]:[W510 04:55:37.269130862 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 4] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank4]:[W510 04:55:38.269258867 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:58522, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1461ffd33fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x146243ec16d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x146243ec01cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x146200fa2eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14634040c164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x146350e7e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x146350f03100 in /lib64/libc.so.6)
+
+[rank4]:[W510 04:55:38.271644195 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 4] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank4]:[W510 04:55:39.271770730 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:58522, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1461ffd33fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x146243ec16d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x146243ec01cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x146200fa2eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14634040c164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x146350e7e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x146350f03100 in /lib64/libc.so.6)
+
+[rank4]:[W510 04:55:39.274082358 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 4] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank4]:[W510 04:55:40.274212223 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:58522, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1461ffd33fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x146243ec16d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x146243ec01cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x146200fa2eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14634040c164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x146350e7e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x146350f03100 in /lib64/libc.so.6)
+
+[rank4]:[W510 04:55:40.276422492 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 4] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank4]:[W510 04:55:41.276546327 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:58522, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1461ffd33fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x146243ec16d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x146243ec01cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x146200fa2eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14634040c164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x146350e7e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x146350f03100 in /lib64/libc.so.6)
+
+[rank4]:[W510 04:55:41.278799966 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 4] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank4]:[W510 04:55:42.278916780 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:58522, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1461ffd33fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x146243ec16d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x146243ec01cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x146200fa2eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14634040c164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x146350e7e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x146350f03100 in /lib64/libc.so.6)
+
+[rank4]:[W510 04:55:42.281184429 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 4] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank4]:[W510 04:55:43.281312264 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:58522, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1461ffd33fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x146243ec16d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x146243ec01cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x146200fa2eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14634040c164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x146350e7e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x146350f03100 in /lib64/libc.so.6)
+
+[rank4]:[W510 04:55:43.283514423 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 4] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank4]:[W510 04:55:44.283631988 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:58522, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1461ffd33fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x146243ec16d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x146243ec01cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x146200fa2eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14634040c164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x146350e7e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x146350f03100 in /lib64/libc.so.6)
+
+[rank4]:[W510 04:55:44.285941696 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 4] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank4]:[W510 04:55:45.286065021 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:58522, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1461ffd33fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x146243ec16d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x146243ec01cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x146200fa2eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14634040c164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x146350e7e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x146350f03100 in /lib64/libc.so.6)
+
+[rank4]:[W510 04:55:45.288285890 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 4] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank4]:[W510 04:55:46.288416945 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:58522, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1461ffd33fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x146243ec16d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x146243ec01cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x146200fa2eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14634040c164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x146350e7e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x146350f03100 in /lib64/libc.so.6)
+
+[rank4]:[W510 04:55:46.290697703 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 4] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank4]:[W510 04:55:48.290815678 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:58522, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1461ffd33fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x146243ec16d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x146243ec01cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x146200fa2eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14634040c164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x146350e7e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x146350f03100 in /lib64/libc.so.6)
+
+[rank4]:[W510 04:55:48.293075917 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 4] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank4]:[W510 04:55:49.293202562 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:58522, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1461ffd33fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x146243ec16d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x146243ec01cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x146200fa2eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14634040c164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x146350e7e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x146350f03100 in /lib64/libc.so.6)
+
+[rank4]:[W510 04:55:49.295595809 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 4] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank4]:[W510 04:55:50.295711624 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:58522, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1461ffd33fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x146243ec16d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x146243ec01cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x146200fa2eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14634040c164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x146350e7e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x146350f03100 in /lib64/libc.so.6)
+
+[rank4]:[W510 04:55:50.298046123 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 4] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank4]:[W510 04:55:51.298160617 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:58522, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1461ffd33fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x146243ec16d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x146243ec01cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x146200fa2eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14634040c164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x146350e7e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x146350f03100 in /lib64/libc.so.6)
+
+[rank4]:[W510 04:55:51.300473036 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 4] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank4]:[W510 04:55:52.300592471 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:58522, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1461ffd33fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x146243ec16d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x146243ec01cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x146200fa2eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14634040c164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x146350e7e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x146350f03100 in /lib64/libc.so.6)
+
+[rank4]:[W510 04:55:52.302828310 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 4] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank4]:[W510 04:55:53.302942844 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:58522, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1461ffd33fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x146243ec16d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x146243ec01cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x146200fa2eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14634040c164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x146350e7e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x146350f03100 in /lib64/libc.so.6)
+
+[rank4]:[W510 04:55:53.305320203 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 4] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank4]:[W510 04:55:54.305460197 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:58522, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1461ffd33fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x146243ec16d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x146243ec01cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x146200fa2eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14634040c164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x146350e7e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x146350f03100 in /lib64/libc.so.6)
+
+[rank4]:[W510 04:55:54.307902565 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 4] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank4]:[W510 04:55:55.308055000 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:58522, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1461ffd33fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x146243ec16d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x146243ec01cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x146200fa2eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14634040c164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x146350e7e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x146350f03100 in /lib64/libc.so.6)
+
+[rank4]:[W510 04:55:55.310368548 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 4] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank4]:[W510 04:55:56.310484953 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:58522, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1461ffd33fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x146243ec16d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x146243ec01cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x146200fa2eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14634040c164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x146350e7e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x146350f03100 in /lib64/libc.so.6)
+
+[rank4]:[W510 04:55:56.312789491 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 4] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank4]:[W510 04:55:57.312905686 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:58522, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1461ffd33fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x146243ec16d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x146243ec01cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x146200fa2eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14634040c164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x146350e7e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x146350f03100 in /lib64/libc.so.6)
+
+[rank4]:[W510 04:55:57.500578983 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 4] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank4]:[W510 04:55:58.500718068 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:58522, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1461ffd33fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x146243ec16d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x146243ec01cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x146200fa2eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14634040c164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x146350e7e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x146350f03100 in /lib64/libc.so.6)
+
+[rank4]:[W510 04:55:58.502987547 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 4] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[2026-05-10T04:56:02.628] error: namespace_p_join: open failed for /slurm/22606429/.ns: No such file or directory
+[2026-05-10T04:56:02.629] error: namespace_g_join(JobId=22606429 SLUID=s8FXKR6KX9EK00): No such file or directory
+[2026-05-10T04:56:04.278] error: Failed to send MESSAGE_TASK_EXIT: Connection refused
+[2026-05-10T04:56:04.777] error: namespace_p_join: open failed for /slurm/22606429/.ns: No such file or directory
+[2026-05-10T04:56:04.778] error: namespace_g_join(JobId=22606429 SLUID=s8FXKR6KX9EK00): No such file or directory
+[2026-05-10T04:56:04.916] error: namespace_p_join: open failed for /slurm/22606429/.ns: No such file or directory
+[2026-05-10T04:56:04.916] error: namespace_g_join(JobId=22606429 SLUID=s8FXKR6KX9EK00): No such file or directory
+[2026-05-10T04:56:04.954] error: namespace_p_join: open failed for /slurm/22606429/.ns: No such file or directory
+[2026-05-10T04:56:04.955] error: namespace_g_join(JobId=22606429 SLUID=s8FXKR6KX9EK00): No such file or directory
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
+  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
+[rank4]:[W510 05:03:21.629382948 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
+  scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+submitit WARNING (2026-05-10 09:26:31,171) - Bypassing signal SIGTERM
+submitit WARNING (2026-05-10 09:26:31,171) - Bypassing signal SIGCONT
+[2026-05-10T09:27:02.626] error: namespace_p_join: open failed for /slurm/22606429/.ns: No such file or directory
+[2026-05-10T09:27:02.626] error: namespace_g_join(JobId=22606429 SLUID=s8FXKR7X2B6T00): No such file or directory
+[2026-05-10T09:27:04.270] error: Failed to send MESSAGE_TASK_EXIT: Connection refused
+[2026-05-10T09:27:04.746] error: namespace_p_join: open failed for /slurm/22606429/.ns: No such file or directory
+[2026-05-10T09:27:04.746] error: namespace_g_join(JobId=22606429 SLUID=s8FXKR7X2B6T00): No such file or directory
+[2026-05-10T09:27:06.384] error: Failed to send MESSAGE_TASK_EXIT: Connection refused
+[2026-05-10T09:27:06.386] error: namespace_p_join: open failed for /slurm/22606429/.ns: No such file or directory
+[2026-05-10T09:27:06.386] error: namespace_g_join(JobId=22606429 SLUID=s8FXKR7X2B6T00): No such file or directory
+[2026-05-10T09:27:06.525] error: namespace_p_join: open failed for /slurm/22606429/.ns: No such file or directory
+[2026-05-10T09:27:06.525] error: namespace_g_join(JobId=22606429 SLUID=s8FXKR7X2B6T00): No such file or directory

vjepa/inflated/d003_lora_ema_capilvd1689m/job_22606429/22606429_4_log.out

@@ -0,0 +1,530 @@
+submitit INFO (2026-05-09 16:57:36,008) - Starting with JobEnvironment(job_id=22606429, hostname=gcn75.local.snellius.surf.nl, local_rank=0(4), node=1(4), global_rank=4(16))
+submitit INFO (2026-05-09 16:57:36,008) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/job_22606429/22606429_submitted.pkl
+INFO:root:loaded pretrain params...
+{   'app': 'vjepa',
+    'cpus_per_task': 16,
+    'data': {   'batch_size': 64,
+                'crop_size': 224,
+                'dataset_fpcs': [16, 16],
+                'dataset_type': 'VideoDataset',
+                'datasets': [   '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
+                                '/scratch-shared/dcanez/data/ssv2/train.csv'],
+                'datasets_weights': [0.65, 0.35],
+                'fps': 4,
+                'num_workers': 10,
+                'patch_size': 14,
+                'persistent_workers': True,
+                'pin_mem': True,
+                'stage': [   {   'dest': 'kinetics_240',
+                                 'format': 'targz_parts',
+                                 'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
+                             {   'dest': 'ssv2',
+                                 'format': 'multipart_tar',
+                                 'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
+                'tubelet_size': 1},
+    'data_aug': {   'auto_augment': False,
+                    'motion_shift': False,
+                    'random_resize_aspect_ratio': [0.75, 1.35],
+                    'random_resize_scale': [0.3, 1.0],
+                    'reprob': 0.0},
+    'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m',
+    'loss': {'loss_exp': 1.0},
+    'mask': [   {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 8,
+                    'spatial_scale': [0.15, 0.15],
+                    'temporal_scale': [1.0, 1.0]},
+                {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 2,
+                    'spatial_scale': [0.7, 0.7],
+                    'temporal_scale': [1.0, 1.0]}],
+    'mem_per_gpu': '180G',
+    'meta': {   'dtype': 'bfloat16',
+                'knn_eval_epoch0': False,
+                'knn_eval_freq': 5,
+                'knn_eval_presets': [   {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
+                                                          'eval_videos_per_class': 25,
+                                                          'num_workers': 8,
+                                                          'train_videos_per_class': 100},
+                                            'preset': 'ucf101'},
+                                        {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
+                                                          'eval_videos_per_class': 100,
+                                                          'linear_probe': True,
+                                                          'num_workers': 8,
+                                                          'pool_type': 'temporal_concat',
+                                                          'train_videos_per_class': 500},
+                                            'preset': 'ssv2_coarse10'}],
+                'load_checkpoint': True,
+                'read_checkpoint': None,
+                'save_every_freq': 5,
+                'seed': 239,
+                'use_sdpa': True,
+                'use_wandb': True,
+                'wandb_project': 'vjepa_ablation'},
+    'metrics': {'sigreg': {}, 'std': {}},
+    'model': {   'class_token': False,
+                 'freeze_backbone': True,
+                 'is_causal': False,
+                 'lora_alpha': 16,
+                 'lora_rank': 8,
+                 'lora_targets': ['attn.qkv', 'attn.proj'],
+                 'model_name': 'vit_large_patch14_capi_lvd1689m',
+                 'num_reg_tokens': 16,
+                 'pred_depth': 6,
+                 'pred_embed_dim': 384,
+                 'pred_num_heads': 12,
+                 'rope_type': 'capi2d',
+                 'stem_type': '2d',
+                 'target_kind': 'ema',
+                 'uniform_power': True,
+                 'use_activation_checkpointing': True,
+                 'use_lora': True,
+                 'use_mask_tokens': True,
+                 'use_rope': True,
+                 'use_sdpa': True,
+                 'zero_init_mask_tokens': True},
+    'nodes': 4,
+    'optimization': {   'ema': [0.99925, 0.99925],
+                        'epochs': 100,
+                        'final_lr': 0.0001,
+                        'final_weight_decay': 0.04,
+                        'ipe': 300,
+                        'ipe_scale': 1.0,
+                        'lr': 0.0005,
+                        'start_lr': 0.0001,
+                        'warmup': 10,
+                        'weight_decay': 0.04},
+    'tasks_per_node': 4}
+INFO:root:Running pre-training of app: vjepa
+[INFO    ][2026-05-09 16:58:12][app.vjepa.train     ][main                     ] which_dtype='bfloat16'
+[INFO    ][2026-05-09 16:58:12][app.vjepa.train     ][main                     ] Disabling persistent_workers (incompatible with KNN eval)
+[INFO    ][2026-05-09 16:58:12][app.vjepa.train     ][main                     ] NCCL_SOCKET_IFNAME=eno
+[INFO    ][2026-05-09 16:59:11][app.vjepa.train     ][main                     ] Initialized (rank/world-size) 4/16, tasks_per_node=4
+[INFO    ][2026-05-09 16:59:11][root                ][stage_datasets           ] [local_rank 0/4] Staging kinetics_240 (targz_parts)
+[INFO    ][2026-05-09 16:59:11][root                ][_stage_targz_parts       ] [rank 0] Extracting 26/103 tar.gz parts to /scratch-node/dcanez.22606429/kinetics_240
+[INFO    ][2026-05-09 16:59:25][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 2/26 parts
+[INFO    ][2026-05-09 16:59:39][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 4/26 parts
+[INFO    ][2026-05-09 16:59:53][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 6/26 parts
+[INFO    ][2026-05-09 17:00:07][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 8/26 parts
+[INFO    ][2026-05-09 17:00:22][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 10/26 parts
+[INFO    ][2026-05-09 17:00:36][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 12/26 parts
+[INFO    ][2026-05-09 17:00:51][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 14/26 parts
+[INFO    ][2026-05-09 17:01:05][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 16/26 parts
+[INFO    ][2026-05-09 17:01:20][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 18/26 parts
+[INFO    ][2026-05-09 17:01:34][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 20/26 parts
+[INFO    ][2026-05-09 17:01:48][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 22/26 parts
+[INFO    ][2026-05-09 17:02:02][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 24/26 parts
+[INFO    ][2026-05-09 17:02:17][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 26/26 parts
+[INFO    ][2026-05-09 17:02:17][root                ][stage_datasets           ] [local_rank 0/4] Staging ssv2 (multipart_tar)
+[INFO    ][2026-05-09 17:02:17][root                ][_stage_multipart_tar     ] [rank 0] Extracting multipart tar (2 files) to /scratch-node/dcanez.22606429/ssv2
+[INFO    ][2026-05-09 17:03:30][root                ][stage_datasets           ] Data staging completed in 259.3s (4.3min)
+[INFO    ][2026-05-09 17:03:30][root                ][_rewrite_csv             ] Wrote local CSV: /scratch-node/dcanez.22606429/kinetics_240/train.csv (239789 entries)
+[INFO    ][2026-05-09 17:03:31][root                ][_rewrite_csv             ] Wrote local CSV: /scratch-node/dcanez.22606429/ssv2/train.csv (168913 entries)
+[INFO    ][2026-05-09 17:03:31][app.vjepa.train     ][main                     ] Staged datasets: ['/scratch-node/dcanez.22606429/kinetics_240/train.csv', '/scratch-node/dcanez.22606429/ssv2/train.csv']
+[INFO    ][2026-05-09 17:03:52][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
+[INFO    ][2026-05-09 17:03:53][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA applied: 48 modules replaced (rank=8, alpha=16, dropout=0.0, targets=['attn.qkv', 'attn.proj'], scope=blocks)
+[INFO    ][2026-05-09 17:03:53][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA trainable params: 1,179,648
+[INFO    ][2026-05-09 17:03:53][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-09 17:03:53][root                ][init_video_model         ] VJEPAPredictorMultiSeqWrapper(
+  (backbone): VisionTransformerPredictor(
+    (predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
+    (mask_tokens): ParameterList(
+        (0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+    )
+    (predictor_blocks): ModuleList(
+      (0-5): 6 x Block(
+        (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (attn): RoPEAttention(
+          (qkv): Linear(in_features=384, out_features=1152, bias=True)
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): Linear(in_features=384, out_features=384, bias=True)
+          (proj_drop): Dropout(p=0.0, inplace=False)
+        )
+        (drop_path): Identity()
+        (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (mlp): MLP(
+          (fc1): Linear(in_features=384, out_features=1536, bias=True)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=1536, out_features=384, bias=True)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+    (predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
+  )
+)
+[INFO    ][2026-05-09 17:03:53][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-09 17:03:53][root                ][init_video_model         ] Encoder number of parameters: 1179648
+[INFO    ][2026-05-09 17:03:53][root                ][init_video_model         ] Predictor number of parameters: 11436928
+[INFO    ][2026-05-09 17:03:53][root                ][init_video_model         ] Target encoder number of parameters: 1179648
+[INFO    ][2026-05-09 17:04:02][root                ][make_videodataset        ] VideoDataset dataset created
+[INFO    ][2026-05-09 17:04:02][WeightedSampler     ][__init__                 ] Using DistributedWeightedSampler with rank 4 / 16
+[INFO    ][2026-05-09 17:04:02][root                ][make_videodataset        ] VideoDataset unsupervised data loader created
+[INFO    ][2026-05-09 17:04:02][app.vjepa.train     ][main                     ] iterations per epoch/dataset length: 300/399
+[INFO    ][2026-05-09 17:04:02][app.vjepa.train     ][main                     ] Wrapping models in DDP (rank 4)...
+[INFO    ][2026-05-09 17:05:26][app.vjepa.train     ][main                     ] Initializing loader...
+submitit INFO (2026-05-10 04:55:29,325) - Job has timed out. Ran 718 minutes out of requested 720 minutes.
+[INFO    ][2026-05-10 04:55:29][submitit            ][has_timed_out            ] Job has timed out. Ran 718 minutes out of requested 720 minutes.
+submitit WARNING (2026-05-10 04:55:29,326) - Caught signal SIGUSR2 on gcn75.local.snellius.surf.nl: this job is timed-out.
+[WARNING ][2026-05-10 04:55:29][submitit            ][checkpoint_and_try_requeue] Caught signal SIGUSR2 on gcn75.local.snellius.surf.nl: this job is timed-out.
+submitit INFO (2026-05-10 04:55:29,326) - Not checkpointing nor requeuing since I am a slave (procid=4).
+[INFO    ][2026-05-10 04:55:29][submitit            ][checkpoint_and_try_requeue] Not checkpointing nor requeuing since I am a slave (procid=4).
+submitit WARNING (2026-05-10 04:55:29,326) - Bypassing signal SIGCONT
+[WARNING ][2026-05-10 04:55:29][submitit            ][bypass                   ] Bypassing signal SIGCONT
+submitit WARNING (2026-05-10 04:55:29,547) - Bypassing signal SIGTERM
+[WARNING ][2026-05-10 04:55:29][submitit            ][bypass                   ] Bypassing signal SIGTERM
+submitit WARNING (2026-05-10 04:55:29,596) - Bypassing signal SIGCONT
+[WARNING ][2026-05-10 04:55:29][submitit            ][bypass                   ] Bypassing signal SIGCONT
+submitit INFO (2026-05-10 04:59:12,592) - Starting with JobEnvironment(job_id=22606429, hostname=gcn75.local.snellius.surf.nl, local_rank=0(4), node=1(4), global_rank=4(16))
+submitit INFO (2026-05-10 04:59:12,593) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/job_22606429/22606429_submitted.pkl
+INFO:root:loaded pretrain params...
+{   'app': 'vjepa',
+    'cpus_per_task': 16,
+    'data': {   'batch_size': 64,
+                'crop_size': 224,
+                'dataset_fpcs': [16, 16],
+                'dataset_type': 'VideoDataset',
+                'datasets': [   '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
+                                '/scratch-shared/dcanez/data/ssv2/train.csv'],
+                'datasets_weights': [0.65, 0.35],
+                'fps': 4,
+                'num_workers': 10,
+                'patch_size': 14,
+                'persistent_workers': True,
+                'pin_mem': True,
+                'stage': [   {   'dest': 'kinetics_240',
+                                 'format': 'targz_parts',
+                                 'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
+                             {   'dest': 'ssv2',
+                                 'format': 'multipart_tar',
+                                 'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
+                'tubelet_size': 1},
+    'data_aug': {   'auto_augment': False,
+                    'motion_shift': False,
+                    'random_resize_aspect_ratio': [0.75, 1.35],
+                    'random_resize_scale': [0.3, 1.0],
+                    'reprob': 0.0},
+    'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m',
+    'loss': {'loss_exp': 1.0},
+    'mask': [   {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 8,
+                    'spatial_scale': [0.15, 0.15],
+                    'temporal_scale': [1.0, 1.0]},
+                {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 2,
+                    'spatial_scale': [0.7, 0.7],
+                    'temporal_scale': [1.0, 1.0]}],
+    'mem_per_gpu': '180G',
+    'meta': {   'dtype': 'bfloat16',
+                'knn_eval_epoch0': False,
+                'knn_eval_freq': 5,
+                'knn_eval_presets': [   {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
+                                                          'eval_videos_per_class': 25,
+                                                          'num_workers': 8,
+                                                          'train_videos_per_class': 100},
+                                            'preset': 'ucf101'},
+                                        {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
+                                                          'eval_videos_per_class': 100,
+                                                          'linear_probe': True,
+                                                          'num_workers': 8,
+                                                          'pool_type': 'temporal_concat',
+                                                          'train_videos_per_class': 500},
+                                            'preset': 'ssv2_coarse10'}],
+                'load_checkpoint': True,
+                'read_checkpoint': None,
+                'save_every_freq': 5,
+                'seed': 239,
+                'use_sdpa': True,
+                'use_wandb': True,
+                'wandb_project': 'vjepa_ablation'},
+    'metrics': {'sigreg': {}, 'std': {}},
+    'model': {   'class_token': False,
+                 'freeze_backbone': True,
+                 'is_causal': False,
+                 'lora_alpha': 16,
+                 'lora_rank': 8,
+                 'lora_targets': ['attn.qkv', 'attn.proj'],
+                 'model_name': 'vit_large_patch14_capi_lvd1689m',
+                 'num_reg_tokens': 16,
+                 'pred_depth': 6,
+                 'pred_embed_dim': 384,
+                 'pred_num_heads': 12,
+                 'rope_type': 'capi2d',
+                 'stem_type': '2d',
+                 'target_kind': 'ema',
+                 'uniform_power': True,
+                 'use_activation_checkpointing': True,
+                 'use_lora': True,
+                 'use_mask_tokens': True,
+                 'use_rope': True,
+                 'use_sdpa': True,
+                 'zero_init_mask_tokens': True},
+    'nodes': 4,
+    'optimization': {   'ema': [0.99925, 0.99925],
+                        'epochs': 100,
+                        'final_lr': 0.0001,
+                        'final_weight_decay': 0.04,
+                        'ipe': 300,
+                        'ipe_scale': 1.0,
+                        'lr': 0.0005,
+                        'start_lr': 0.0001,
+                        'warmup': 10,
+                        'weight_decay': 0.04},
+    'tasks_per_node': 4}
+INFO:root:Running pre-training of app: vjepa
+[INFO    ][2026-05-10 04:59:17][app.vjepa.train     ][main                     ] which_dtype='bfloat16'
+[INFO    ][2026-05-10 04:59:17][app.vjepa.train     ][main                     ] Disabling persistent_workers (incompatible with KNN eval)
+[INFO    ][2026-05-10 04:59:17][app.vjepa.train     ][main                     ] NCCL_SOCKET_IFNAME=eno
+[INFO    ][2026-05-10 04:59:22][app.vjepa.train     ][main                     ] Initialized (rank/world-size) 4/16, tasks_per_node=4
+[INFO    ][2026-05-10 04:59:22][root                ][stage_datasets           ] [local_rank 0/4] Staging kinetics_240 (targz_parts)
+[INFO    ][2026-05-10 04:59:22][root                ][_stage_targz_parts       ] [rank 0] Extracting 26/103 tar.gz parts to /scratch-node/dcanez.22606429/kinetics_240
+[INFO    ][2026-05-10 04:59:36][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 2/26 parts
+[INFO    ][2026-05-10 04:59:50][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 4/26 parts
+[INFO    ][2026-05-10 05:00:04][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 6/26 parts
+[INFO    ][2026-05-10 05:00:18][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 8/26 parts
+[INFO    ][2026-05-10 05:00:32][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 10/26 parts
+[INFO    ][2026-05-10 05:00:46][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 12/26 parts
+[INFO    ][2026-05-10 05:01:00][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 14/26 parts
+[INFO    ][2026-05-10 05:01:15][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 16/26 parts
+[INFO    ][2026-05-10 05:01:29][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 18/26 parts
+[INFO    ][2026-05-10 05:01:43][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 20/26 parts
+[INFO    ][2026-05-10 05:01:57][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 22/26 parts
+[INFO    ][2026-05-10 05:02:12][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 24/26 parts
+[INFO    ][2026-05-10 05:02:26][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 26/26 parts
+[INFO    ][2026-05-10 05:02:26][root                ][stage_datasets           ] [local_rank 0/4] Staging ssv2 (multipart_tar)
+[INFO    ][2026-05-10 05:02:26][root                ][_stage_multipart_tar     ] [rank 0] Extracting multipart tar (2 files) to /scratch-node/dcanez.22606429/ssv2
+[INFO    ][2026-05-10 05:04:32][root                ][stage_datasets           ] Data staging completed in 310.2s (5.2min)
+[INFO    ][2026-05-10 05:04:32][root                ][_rewrite_csv             ] Wrote local CSV: /scratch-node/dcanez.22606429/kinetics_240/train.csv (239789 entries)
+[INFO    ][2026-05-10 05:04:33][root                ][_rewrite_csv             ] Wrote local CSV: /scratch-node/dcanez.22606429/ssv2/train.csv (168913 entries)
+[INFO    ][2026-05-10 05:04:33][app.vjepa.train     ][main                     ] Staged datasets: ['/scratch-node/dcanez.22606429/kinetics_240/train.csv', '/scratch-node/dcanez.22606429/ssv2/train.csv']
+[INFO    ][2026-05-10 05:04:47][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
+[INFO    ][2026-05-10 05:04:51][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA applied: 48 modules replaced (rank=8, alpha=16, dropout=0.0, targets=['attn.qkv', 'attn.proj'], scope=blocks)
+[INFO    ][2026-05-10 05:04:51][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA trainable params: 1,179,648
+[INFO    ][2026-05-10 05:04:52][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-10 05:04:52][root                ][init_video_model         ] VJEPAPredictorMultiSeqWrapper(
+  (backbone): VisionTransformerPredictor(
+    (predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
+    (mask_tokens): ParameterList(
+        (0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+    )
+    (predictor_blocks): ModuleList(
+      (0-5): 6 x Block(
+        (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (attn): RoPEAttention(
+          (qkv): Linear(in_features=384, out_features=1152, bias=True)
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): Linear(in_features=384, out_features=384, bias=True)
+          (proj_drop): Dropout(p=0.0, inplace=False)
+        )
+        (drop_path): Identity()
+        (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (mlp): MLP(
+          (fc1): Linear(in_features=384, out_features=1536, bias=True)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=1536, out_features=384, bias=True)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+    (predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
+  )
+)
+[INFO    ][2026-05-10 05:04:52][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-10 05:04:52][root                ][init_video_model         ] Encoder number of parameters: 1179648
+[INFO    ][2026-05-10 05:04:52][root                ][init_video_model         ] Predictor number of parameters: 11436928
+[INFO    ][2026-05-10 05:04:52][root                ][init_video_model         ] Target encoder number of parameters: 1179648
+[INFO    ][2026-05-10 05:04:53][root                ][make_videodataset        ] VideoDataset dataset created
+[INFO    ][2026-05-10 05:04:53][WeightedSampler     ][__init__                 ] Using DistributedWeightedSampler with rank 4 / 16
+[INFO    ][2026-05-10 05:04:53][root                ][make_videodataset        ] VideoDataset unsupervised data loader created
+[INFO    ][2026-05-10 05:04:53][app.vjepa.train     ][main                     ] iterations per epoch/dataset length: 300/399
+[INFO    ][2026-05-10 05:04:53][app.vjepa.train     ][main                     ] Wrapping models in DDP (rank 4)...
+[INFO    ][2026-05-10 05:06:08][root                ][load_checkpoint          ] Loading checkpoint from /scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/latest.pt
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] loaded pretrained encoder from epoch 33 with msg: <All keys matched successfully>
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] loaded pretrained predictor from epoch 33 with msg: <All keys matched successfully>
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] loaded pretrained target encoder from epoch 33 with msg: <All keys matched successfully>
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] loaded optimizers from epoch 33
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] read-path: /scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/latest.pt
+[INFO    ][2026-05-10 05:06:09][app.vjepa.train     ][main                     ] Initializing loader...
+submitit WARNING (2026-05-10 09:26:31,171) - Bypassing signal SIGTERM
+[WARNING ][2026-05-10 09:26:31][submitit            ][bypass                   ] Bypassing signal SIGTERM
+submitit WARNING (2026-05-10 09:26:31,171) - Bypassing signal SIGCONT
+[WARNING ][2026-05-10 09:26:31][submitit            ][bypass                   ] Bypassing signal SIGCONT

vjepa/inflated/d003_lora_ema_capilvd1689m/job_22606429/22606429_5_log.err

@@ -0,0 +1,306 @@
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
+  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
+[rank5]:[W509 17:02:13.724036467 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
+  scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+submitit WARNING (2026-05-10 04:55:29,312) - Caught signal SIGUSR2 on gcn75.local.snellius.surf.nl: this job is timed-out.
+submitit WARNING (2026-05-10 04:55:29,312) - Bypassing signal SIGCONT
+submitit WARNING (2026-05-10 04:55:29,547) - Bypassing signal SIGTERM
+submitit WARNING (2026-05-10 04:55:29,547) - Bypassing signal SIGCONT
+[rank5]:[W510 04:55:33.255941989 TCPStore.cpp:125] [c10d] recvValue failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:58514, remote=[gcn74.local.snellius.surf.nl]:37129): Failed to recv, got 0 bytes. Connection was likely closed. Did the remote server shutdown or crash?
+Exception raised from recvBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:682 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1511c9af3fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a3325d (0x15120dc8225d in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x273 (0x15120dc801f3 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1511cad62eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15130a1cc164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15131ac3e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15131acc3100 in /lib64/libc.so.6)
+
+[rank5]:[W510 04:55:33.258392227 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 5] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Failed to recv, got 0 bytes. Connection was likely closed. Did the remote server shutdown or crash?
+[rank5]:[W510 04:55:34.258532498 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:58514, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1511c9af3fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15120dc816d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15120dc801cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1511cad62eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15130a1cc164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15131ac3e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15131acc3100 in /lib64/libc.so.6)
+
+[rank5]:[W510 04:55:34.261898101 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 5] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank5]:[W510 04:55:35.262042957 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:58514, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1511c9af3fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15120dc816d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15120dc801cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1511cad62eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15130a1cc164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15131ac3e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15131acc3100 in /lib64/libc.so.6)
+
+[rank5]:[W510 04:55:35.264343575 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 5] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank5]:[W510 04:55:36.264476220 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:58514, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1511c9af3fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15120dc816d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15120dc801cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1511cad62eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15130a1cc164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15131ac3e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15131acc3100 in /lib64/libc.so.6)
+
+[rank5]:[W510 04:55:36.266742339 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 5] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank5]:[W510 04:55:37.266870094 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:58514, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1511c9af3fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15120dc816d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15120dc801cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1511cad62eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15130a1cc164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15131ac3e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15131acc3100 in /lib64/libc.so.6)
+
+[rank5]:[W510 04:55:37.269200332 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 5] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank5]:[W510 04:55:38.269319357 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:58514, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1511c9af3fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15120dc816d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15120dc801cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1511cad62eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15130a1cc164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15131ac3e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15131acc3100 in /lib64/libc.so.6)
+
+[rank5]:[W510 04:55:38.271603425 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 5] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank5]:[W510 04:55:39.271748370 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:58514, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1511c9af3fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15120dc816d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15120dc801cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1511cad62eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15130a1cc164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15131ac3e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15131acc3100 in /lib64/libc.so.6)
+
+[rank5]:[W510 04:55:39.274120178 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 5] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank5]:[W510 04:55:40.274237343 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:58514, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1511c9af3fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15120dc816d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15120dc801cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1511cad62eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15130a1cc164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15131ac3e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15131acc3100 in /lib64/libc.so.6)
+
+[rank5]:[W510 04:55:40.276591171 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 5] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank5]:[W510 04:55:41.276730646 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:58514, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1511c9af3fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15120dc816d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15120dc801cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1511cad62eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15130a1cc164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15131ac3e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15131acc3100 in /lib64/libc.so.6)
+
+[rank5]:[W510 04:55:41.279101954 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 5] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank5]:[W510 04:55:42.279218709 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:58514, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1511c9af3fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15120dc816d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15120dc801cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1511cad62eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15130a1cc164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15131ac3e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15131acc3100 in /lib64/libc.so.6)
+
+[rank5]:[W510 04:55:42.281453538 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 5] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank5]:[W510 04:55:43.281586363 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:58514, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1511c9af3fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15120dc816d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15120dc801cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1511cad62eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15130a1cc164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15131ac3e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15131acc3100 in /lib64/libc.so.6)
+
+[rank5]:[W510 04:55:43.283824321 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 5] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank5]:[W510 04:55:44.283960636 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:58514, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1511c9af3fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15120dc816d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15120dc801cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1511cad62eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15130a1cc164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15131ac3e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15131acc3100 in /lib64/libc.so.6)
+
+[rank5]:[W510 04:55:44.286215035 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 5] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank5]:[W510 04:55:45.286348979 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:58514, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1511c9af3fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15120dc816d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15120dc801cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1511cad62eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15130a1cc164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15131ac3e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15131acc3100 in /lib64/libc.so.6)
+
+[rank5]:[W510 04:55:45.288581178 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 5] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank5]:[W510 04:55:46.288716823 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:58514, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1511c9af3fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15120dc816d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15120dc801cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1511cad62eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15130a1cc164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15131ac3e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15131acc3100 in /lib64/libc.so.6)
+
+[rank5]:[W510 04:55:47.291054721 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 5] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank5]:[W510 04:55:48.291191666 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:58514, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1511c9af3fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15120dc816d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15120dc801cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1511cad62eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15130a1cc164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15131ac3e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15131acc3100 in /lib64/libc.so.6)
+
+[rank5]:[W510 04:55:48.293461325 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 5] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank5]:[W510 04:55:49.293579520 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:58514, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1511c9af3fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15120dc816d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15120dc801cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1511cad62eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15130a1cc164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15131ac3e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15131acc3100 in /lib64/libc.so.6)
+
+[rank5]:[W510 04:55:49.295923038 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 5] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank5]:[W510 04:55:50.296066003 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:58514, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1511c9af3fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15120dc816d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15120dc801cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1511cad62eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15130a1cc164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15131ac3e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15131acc3100 in /lib64/libc.so.6)
+
+[rank5]:[W510 04:55:50.298464761 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 5] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank5]:[W510 04:55:51.298590625 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:58514, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1511c9af3fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15120dc816d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15120dc801cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1511cad62eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15130a1cc164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15131ac3e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15131acc3100 in /lib64/libc.so.6)
+
+[rank5]:[W510 04:55:51.300814544 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 5] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank5]:[W510 04:55:52.300946499 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:58514, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1511c9af3fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15120dc816d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15120dc801cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1511cad62eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15130a1cc164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15131ac3e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15131acc3100 in /lib64/libc.so.6)
+
+[rank5]:[W510 04:55:52.303191838 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 5] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank5]:[W510 04:55:53.303320553 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:58514, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1511c9af3fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15120dc816d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15120dc801cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1511cad62eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15130a1cc164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15131ac3e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15131acc3100 in /lib64/libc.so.6)
+
+[rank5]:[W510 04:55:53.305670421 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 5] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank5]:[W510 04:55:54.305800976 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:58514, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1511c9af3fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15120dc816d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15120dc801cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1511cad62eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15130a1cc164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15131ac3e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15131acc3100 in /lib64/libc.so.6)
+
+[rank5]:[W510 04:55:54.308065544 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 5] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank5]:[W510 04:55:55.308188989 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:58514, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1511c9af3fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15120dc816d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15120dc801cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1511cad62eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15130a1cc164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15131ac3e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15131acc3100 in /lib64/libc.so.6)
+
+[rank5]:[W510 04:55:55.310425168 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 5] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank5]:[W510 04:55:56.310555473 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:58514, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1511c9af3fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15120dc816d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15120dc801cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1511cad62eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15130a1cc164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15131ac3e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15131acc3100 in /lib64/libc.so.6)
+
+[rank5]:[W510 04:55:56.312900861 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 5] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank5]:[W510 04:55:57.313049386 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:58514, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1511c9af3fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15120dc816d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15120dc801cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1511cad62eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15130a1cc164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15131ac3e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15131acc3100 in /lib64/libc.so.6)
+
+[rank5]:[W510 04:55:57.315260305 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 5] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank5]:[W510 04:55:58.315391259 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:58514, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1511c9af3fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15120dc816d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15120dc801cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1511cad62eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15130a1cc164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15131ac3e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15131acc3100 in /lib64/libc.so.6)
+
+[rank5]:[W510 04:55:58.317586728 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 5] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank5]:[W510 04:55:59.317719093 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:58514, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1511c9af3fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15120dc816d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15120dc801cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1511cad62eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15130a1cc164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15131ac3e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15131acc3100 in /lib64/libc.so.6)
+
+[rank5]:[W510 04:55:59.320101621 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 5] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
+  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
+[rank5]:[W510 05:02:23.053068628 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
+  scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+submitit WARNING (2026-05-10 09:26:31,170) - Bypassing signal SIGTERM
+submitit WARNING (2026-05-10 09:26:31,171) - Bypassing signal SIGCONT

vjepa/inflated/d003_lora_ema_capilvd1689m/job_22606429/22606429_5_log.out

@@ -0,0 +1,522 @@
+submitit INFO (2026-05-09 16:57:36,008) - Starting with JobEnvironment(job_id=22606429, hostname=gcn75.local.snellius.surf.nl, local_rank=1(4), node=1(4), global_rank=5(16))
+submitit INFO (2026-05-09 16:57:36,008) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/job_22606429/22606429_submitted.pkl
+INFO:root:loaded pretrain params...
+{   'app': 'vjepa',
+    'cpus_per_task': 16,
+    'data': {   'batch_size': 64,
+                'crop_size': 224,
+                'dataset_fpcs': [16, 16],
+                'dataset_type': 'VideoDataset',
+                'datasets': [   '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
+                                '/scratch-shared/dcanez/data/ssv2/train.csv'],
+                'datasets_weights': [0.65, 0.35],
+                'fps': 4,
+                'num_workers': 10,
+                'patch_size': 14,
+                'persistent_workers': True,
+                'pin_mem': True,
+                'stage': [   {   'dest': 'kinetics_240',
+                                 'format': 'targz_parts',
+                                 'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
+                             {   'dest': 'ssv2',
+                                 'format': 'multipart_tar',
+                                 'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
+                'tubelet_size': 1},
+    'data_aug': {   'auto_augment': False,
+                    'motion_shift': False,
+                    'random_resize_aspect_ratio': [0.75, 1.35],
+                    'random_resize_scale': [0.3, 1.0],
+                    'reprob': 0.0},
+    'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m',
+    'loss': {'loss_exp': 1.0},
+    'mask': [   {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 8,
+                    'spatial_scale': [0.15, 0.15],
+                    'temporal_scale': [1.0, 1.0]},
+                {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 2,
+                    'spatial_scale': [0.7, 0.7],
+                    'temporal_scale': [1.0, 1.0]}],
+    'mem_per_gpu': '180G',
+    'meta': {   'dtype': 'bfloat16',
+                'knn_eval_epoch0': False,
+                'knn_eval_freq': 5,
+                'knn_eval_presets': [   {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
+                                                          'eval_videos_per_class': 25,
+                                                          'num_workers': 8,
+                                                          'train_videos_per_class': 100},
+                                            'preset': 'ucf101'},
+                                        {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
+                                                          'eval_videos_per_class': 100,
+                                                          'linear_probe': True,
+                                                          'num_workers': 8,
+                                                          'pool_type': 'temporal_concat',
+                                                          'train_videos_per_class': 500},
+                                            'preset': 'ssv2_coarse10'}],
+                'load_checkpoint': True,
+                'read_checkpoint': None,
+                'save_every_freq': 5,
+                'seed': 239,
+                'use_sdpa': True,
+                'use_wandb': True,
+                'wandb_project': 'vjepa_ablation'},
+    'metrics': {'sigreg': {}, 'std': {}},
+    'model': {   'class_token': False,
+                 'freeze_backbone': True,
+                 'is_causal': False,
+                 'lora_alpha': 16,
+                 'lora_rank': 8,
+                 'lora_targets': ['attn.qkv', 'attn.proj'],
+                 'model_name': 'vit_large_patch14_capi_lvd1689m',
+                 'num_reg_tokens': 16,
+                 'pred_depth': 6,
+                 'pred_embed_dim': 384,
+                 'pred_num_heads': 12,
+                 'rope_type': 'capi2d',
+                 'stem_type': '2d',
+                 'target_kind': 'ema',
+                 'uniform_power': True,
+                 'use_activation_checkpointing': True,
+                 'use_lora': True,
+                 'use_mask_tokens': True,
+                 'use_rope': True,
+                 'use_sdpa': True,
+                 'zero_init_mask_tokens': True},
+    'nodes': 4,
+    'optimization': {   'ema': [0.99925, 0.99925],
+                        'epochs': 100,
+                        'final_lr': 0.0001,
+                        'final_weight_decay': 0.04,
+                        'ipe': 300,
+                        'ipe_scale': 1.0,
+                        'lr': 0.0005,
+                        'start_lr': 0.0001,
+                        'warmup': 10,
+                        'weight_decay': 0.04},
+    'tasks_per_node': 4}
+INFO:root:Running pre-training of app: vjepa
+[INFO    ][2026-05-09 16:58:12][app.vjepa.train     ][main                     ] which_dtype='bfloat16'
+[INFO    ][2026-05-09 16:58:12][app.vjepa.train     ][main                     ] Disabling persistent_workers (incompatible with KNN eval)
+[INFO    ][2026-05-09 16:58:12][app.vjepa.train     ][main                     ] NCCL_SOCKET_IFNAME=eno
+[INFO    ][2026-05-09 16:59:09][app.vjepa.train     ][main                     ] Initialized (rank/world-size) 5/16, tasks_per_node=4
+[INFO    ][2026-05-09 16:59:09][root                ][stage_datasets           ] [local_rank 1/4] Staging kinetics_240 (targz_parts)
+[INFO    ][2026-05-09 16:59:09][root                ][_stage_targz_parts       ] [rank 1] Extracting 26/103 tar.gz parts to /scratch-node/dcanez.22606429/kinetics_240
+[INFO    ][2026-05-09 16:59:22][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 2/26 parts
+[INFO    ][2026-05-09 16:59:35][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 4/26 parts
+[INFO    ][2026-05-09 16:59:50][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 6/26 parts
+[INFO    ][2026-05-09 17:00:04][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 8/26 parts
+[INFO    ][2026-05-09 17:00:19][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 10/26 parts
+[INFO    ][2026-05-09 17:00:33][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 12/26 parts
+[INFO    ][2026-05-09 17:00:47][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 14/26 parts
+[INFO    ][2026-05-09 17:01:01][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 16/26 parts
+[INFO    ][2026-05-09 17:01:15][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 18/26 parts
+[INFO    ][2026-05-09 17:01:30][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 20/26 parts
+[INFO    ][2026-05-09 17:01:44][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 22/26 parts
+[INFO    ][2026-05-09 17:01:59][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 24/26 parts
+[INFO    ][2026-05-09 17:02:13][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 26/26 parts
+[INFO    ][2026-05-09 17:02:13][root                ][stage_datasets           ] [local_rank 1/4] Staging ssv2 (multipart_tar)
+[INFO    ][2026-05-09 17:03:31][app.vjepa.train     ][main                     ] Staged datasets: ['/scratch-node/dcanez.22606429/kinetics_240/train.csv', '/scratch-node/dcanez.22606429/ssv2/train.csv']
+[INFO    ][2026-05-09 17:03:52][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
+[INFO    ][2026-05-09 17:03:53][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA applied: 48 modules replaced (rank=8, alpha=16, dropout=0.0, targets=['attn.qkv', 'attn.proj'], scope=blocks)
+[INFO    ][2026-05-09 17:03:53][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA trainable params: 1,179,648
+[INFO    ][2026-05-09 17:03:53][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-09 17:03:53][root                ][init_video_model         ] VJEPAPredictorMultiSeqWrapper(
+  (backbone): VisionTransformerPredictor(
+    (predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
+    (mask_tokens): ParameterList(
+        (0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+    )
+    (predictor_blocks): ModuleList(
+      (0-5): 6 x Block(
+        (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (attn): RoPEAttention(
+          (qkv): Linear(in_features=384, out_features=1152, bias=True)
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): Linear(in_features=384, out_features=384, bias=True)
+          (proj_drop): Dropout(p=0.0, inplace=False)
+        )
+        (drop_path): Identity()
+        (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (mlp): MLP(
+          (fc1): Linear(in_features=384, out_features=1536, bias=True)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=1536, out_features=384, bias=True)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+    (predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
+  )
+)
+[INFO    ][2026-05-09 17:03:53][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-09 17:03:53][root                ][init_video_model         ] Encoder number of parameters: 1179648
+[INFO    ][2026-05-09 17:03:53][root                ][init_video_model         ] Predictor number of parameters: 11436928
+[INFO    ][2026-05-09 17:03:53][root                ][init_video_model         ] Target encoder number of parameters: 1179648
+[INFO    ][2026-05-09 17:04:02][root                ][make_videodataset        ] VideoDataset dataset created
+[INFO    ][2026-05-09 17:04:02][WeightedSampler     ][__init__                 ] Using DistributedWeightedSampler with rank 5 / 16
+[INFO    ][2026-05-09 17:04:02][root                ][make_videodataset        ] VideoDataset unsupervised data loader created
+[INFO    ][2026-05-09 17:04:02][app.vjepa.train     ][main                     ] iterations per epoch/dataset length: 300/399
+[INFO    ][2026-05-09 17:04:02][app.vjepa.train     ][main                     ] Wrapping models in DDP (rank 5)...
+[INFO    ][2026-05-09 17:05:26][app.vjepa.train     ][main                     ] Initializing loader...
+submitit INFO (2026-05-10 04:55:29,312) - Job has timed out. Ran 718 minutes out of requested 720 minutes.
+[INFO    ][2026-05-10 04:55:29][submitit            ][has_timed_out            ] Job has timed out. Ran 718 minutes out of requested 720 minutes.
+submitit WARNING (2026-05-10 04:55:29,312) - Caught signal SIGUSR2 on gcn75.local.snellius.surf.nl: this job is timed-out.
+[WARNING ][2026-05-10 04:55:29][submitit            ][checkpoint_and_try_requeue] Caught signal SIGUSR2 on gcn75.local.snellius.surf.nl: this job is timed-out.
+submitit INFO (2026-05-10 04:55:29,312) - Not checkpointing nor requeuing since I am a slave (procid=5).
+[INFO    ][2026-05-10 04:55:29][submitit            ][checkpoint_and_try_requeue] Not checkpointing nor requeuing since I am a slave (procid=5).
+submitit WARNING (2026-05-10 04:55:29,312) - Bypassing signal SIGCONT
+[WARNING ][2026-05-10 04:55:29][submitit            ][bypass                   ] Bypassing signal SIGCONT
+submitit WARNING (2026-05-10 04:55:29,547) - Bypassing signal SIGTERM
+[WARNING ][2026-05-10 04:55:29][submitit            ][bypass                   ] Bypassing signal SIGTERM
+submitit WARNING (2026-05-10 04:55:29,547) - Bypassing signal SIGCONT
+[WARNING ][2026-05-10 04:55:29][submitit            ][bypass                   ] Bypassing signal SIGCONT
+submitit INFO (2026-05-10 04:59:12,592) - Starting with JobEnvironment(job_id=22606429, hostname=gcn75.local.snellius.surf.nl, local_rank=1(4), node=1(4), global_rank=5(16))
+submitit INFO (2026-05-10 04:59:12,592) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/job_22606429/22606429_submitted.pkl
+INFO:root:loaded pretrain params...
+{   'app': 'vjepa',
+    'cpus_per_task': 16,
+    'data': {   'batch_size': 64,
+                'crop_size': 224,
+                'dataset_fpcs': [16, 16],
+                'dataset_type': 'VideoDataset',
+                'datasets': [   '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
+                                '/scratch-shared/dcanez/data/ssv2/train.csv'],
+                'datasets_weights': [0.65, 0.35],
+                'fps': 4,
+                'num_workers': 10,
+                'patch_size': 14,
+                'persistent_workers': True,
+                'pin_mem': True,
+                'stage': [   {   'dest': 'kinetics_240',
+                                 'format': 'targz_parts',
+                                 'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
+                             {   'dest': 'ssv2',
+                                 'format': 'multipart_tar',
+                                 'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
+                'tubelet_size': 1},
+    'data_aug': {   'auto_augment': False,
+                    'motion_shift': False,
+                    'random_resize_aspect_ratio': [0.75, 1.35],
+                    'random_resize_scale': [0.3, 1.0],
+                    'reprob': 0.0},
+    'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m',
+    'loss': {'loss_exp': 1.0},
+    'mask': [   {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 8,
+                    'spatial_scale': [0.15, 0.15],
+                    'temporal_scale': [1.0, 1.0]},
+                {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 2,
+                    'spatial_scale': [0.7, 0.7],
+                    'temporal_scale': [1.0, 1.0]}],
+    'mem_per_gpu': '180G',
+    'meta': {   'dtype': 'bfloat16',
+                'knn_eval_epoch0': False,
+                'knn_eval_freq': 5,
+                'knn_eval_presets': [   {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
+                                                          'eval_videos_per_class': 25,
+                                                          'num_workers': 8,
+                                                          'train_videos_per_class': 100},
+                                            'preset': 'ucf101'},
+                                        {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
+                                                          'eval_videos_per_class': 100,
+                                                          'linear_probe': True,
+                                                          'num_workers': 8,
+                                                          'pool_type': 'temporal_concat',
+                                                          'train_videos_per_class': 500},
+                                            'preset': 'ssv2_coarse10'}],
+                'load_checkpoint': True,
+                'read_checkpoint': None,
+                'save_every_freq': 5,
+                'seed': 239,
+                'use_sdpa': True,
+                'use_wandb': True,
+                'wandb_project': 'vjepa_ablation'},
+    'metrics': {'sigreg': {}, 'std': {}},
+    'model': {   'class_token': False,
+                 'freeze_backbone': True,
+                 'is_causal': False,
+                 'lora_alpha': 16,
+                 'lora_rank': 8,
+                 'lora_targets': ['attn.qkv', 'attn.proj'],
+                 'model_name': 'vit_large_patch14_capi_lvd1689m',
+                 'num_reg_tokens': 16,
+                 'pred_depth': 6,
+                 'pred_embed_dim': 384,
+                 'pred_num_heads': 12,
+                 'rope_type': 'capi2d',
+                 'stem_type': '2d',
+                 'target_kind': 'ema',
+                 'uniform_power': True,
+                 'use_activation_checkpointing': True,
+                 'use_lora': True,
+                 'use_mask_tokens': True,
+                 'use_rope': True,
+                 'use_sdpa': True,
+                 'zero_init_mask_tokens': True},
+    'nodes': 4,
+    'optimization': {   'ema': [0.99925, 0.99925],
+                        'epochs': 100,
+                        'final_lr': 0.0001,
+                        'final_weight_decay': 0.04,
+                        'ipe': 300,
+                        'ipe_scale': 1.0,
+                        'lr': 0.0005,
+                        'start_lr': 0.0001,
+                        'warmup': 10,
+                        'weight_decay': 0.04},
+    'tasks_per_node': 4}
+INFO:root:Running pre-training of app: vjepa
+[INFO    ][2026-05-10 04:59:17][app.vjepa.train     ][main                     ] which_dtype='bfloat16'
+[INFO    ][2026-05-10 04:59:17][app.vjepa.train     ][main                     ] Disabling persistent_workers (incompatible with KNN eval)
+[INFO    ][2026-05-10 04:59:17][app.vjepa.train     ][main                     ] NCCL_SOCKET_IFNAME=eno
+[INFO    ][2026-05-10 04:59:21][app.vjepa.train     ][main                     ] Initialized (rank/world-size) 5/16, tasks_per_node=4
+[INFO    ][2026-05-10 04:59:21][root                ][stage_datasets           ] [local_rank 1/4] Staging kinetics_240 (targz_parts)
+[INFO    ][2026-05-10 04:59:21][root                ][_stage_targz_parts       ] [rank 1] Extracting 26/103 tar.gz parts to /scratch-node/dcanez.22606429/kinetics_240
+[INFO    ][2026-05-10 04:59:33][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 2/26 parts
+[INFO    ][2026-05-10 04:59:47][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 4/26 parts
+[INFO    ][2026-05-10 05:00:01][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 6/26 parts
+[INFO    ][2026-05-10 05:00:15][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 8/26 parts
+[INFO    ][2026-05-10 05:00:30][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 10/26 parts
+[INFO    ][2026-05-10 05:00:44][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 12/26 parts
+[INFO    ][2026-05-10 05:00:57][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 14/26 parts
+[INFO    ][2026-05-10 05:01:12][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 16/26 parts
+[INFO    ][2026-05-10 05:01:26][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 18/26 parts
+[INFO    ][2026-05-10 05:01:40][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 20/26 parts
+[INFO    ][2026-05-10 05:01:55][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 22/26 parts
+[INFO    ][2026-05-10 05:02:09][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 24/26 parts
+[INFO    ][2026-05-10 05:02:23][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 26/26 parts
+[INFO    ][2026-05-10 05:02:23][root                ][stage_datasets           ] [local_rank 1/4] Staging ssv2 (multipart_tar)
+[INFO    ][2026-05-10 05:04:33][app.vjepa.train     ][main                     ] Staged datasets: ['/scratch-node/dcanez.22606429/kinetics_240/train.csv', '/scratch-node/dcanez.22606429/ssv2/train.csv']
+[INFO    ][2026-05-10 05:04:47][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
+[INFO    ][2026-05-10 05:04:51][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA applied: 48 modules replaced (rank=8, alpha=16, dropout=0.0, targets=['attn.qkv', 'attn.proj'], scope=blocks)
+[INFO    ][2026-05-10 05:04:51][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA trainable params: 1,179,648
+[INFO    ][2026-05-10 05:04:52][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-10 05:04:52][root                ][init_video_model         ] VJEPAPredictorMultiSeqWrapper(
+  (backbone): VisionTransformerPredictor(
+    (predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
+    (mask_tokens): ParameterList(
+        (0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+    )
+    (predictor_blocks): ModuleList(
+      (0-5): 6 x Block(
+        (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (attn): RoPEAttention(
+          (qkv): Linear(in_features=384, out_features=1152, bias=True)
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): Linear(in_features=384, out_features=384, bias=True)
+          (proj_drop): Dropout(p=0.0, inplace=False)
+        )
+        (drop_path): Identity()
+        (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (mlp): MLP(
+          (fc1): Linear(in_features=384, out_features=1536, bias=True)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=1536, out_features=384, bias=True)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+    (predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
+  )
+)
+[INFO    ][2026-05-10 05:04:52][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-10 05:04:52][root                ][init_video_model         ] Encoder number of parameters: 1179648
+[INFO    ][2026-05-10 05:04:52][root                ][init_video_model         ] Predictor number of parameters: 11436928
+[INFO    ][2026-05-10 05:04:52][root                ][init_video_model         ] Target encoder number of parameters: 1179648
+[INFO    ][2026-05-10 05:04:53][root                ][make_videodataset        ] VideoDataset dataset created
+[INFO    ][2026-05-10 05:04:53][WeightedSampler     ][__init__                 ] Using DistributedWeightedSampler with rank 5 / 16
+[INFO    ][2026-05-10 05:04:53][root                ][make_videodataset        ] VideoDataset unsupervised data loader created
+[INFO    ][2026-05-10 05:04:53][app.vjepa.train     ][main                     ] iterations per epoch/dataset length: 300/399
+[INFO    ][2026-05-10 05:04:53][app.vjepa.train     ][main                     ] Wrapping models in DDP (rank 5)...
+[INFO    ][2026-05-10 05:06:08][root                ][load_checkpoint          ] Loading checkpoint from /scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/latest.pt
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] loaded pretrained encoder from epoch 33 with msg: <All keys matched successfully>
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] loaded pretrained predictor from epoch 33 with msg: <All keys matched successfully>
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] loaded pretrained target encoder from epoch 33 with msg: <All keys matched successfully>
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] loaded optimizers from epoch 33
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] read-path: /scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/latest.pt
+[INFO    ][2026-05-10 05:06:09][app.vjepa.train     ][main                     ] Initializing loader...
+submitit WARNING (2026-05-10 09:26:31,170) - Bypassing signal SIGTERM
+[WARNING ][2026-05-10 09:26:31][submitit            ][bypass                   ] Bypassing signal SIGTERM
+submitit WARNING (2026-05-10 09:26:31,171) - Bypassing signal SIGCONT
+[WARNING ][2026-05-10 09:26:31][submitit            ][bypass                   ] Bypassing signal SIGCONT

vjepa/inflated/d003_lora_ema_capilvd1689m/job_22606429/22606429_6_log.err

@@ -0,0 +1,294 @@
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
+  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
+[rank6]:[W509 17:02:07.766984011 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
+  scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+submitit WARNING (2026-05-10 04:55:29,547) - Caught signal SIGUSR2 on gcn75.local.snellius.surf.nl: this job is timed-out.
+submitit WARNING (2026-05-10 04:55:29,547) - Bypassing signal SIGTERM
+submitit WARNING (2026-05-10 04:55:29,547) - Bypassing signal SIGCONT
+[rank6]:[W510 04:55:34.907852517 TCPStore.cpp:125] [c10d] recvValue failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:37394, remote=[gcn74.local.snellius.surf.nl]:37129): Failed to recv, got 0 bytes. Connection was likely closed. Did the remote server shutdown or crash?
+Exception raised from recvBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:682 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x151efd382fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a3325d (0x151f4151125d in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x273 (0x151f4150f1f3 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x151efe5f1eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15203da5b164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15204e4cd19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15204e552100 in /lib64/libc.so.6)
+
+[rank6]:[W510 04:55:34.910464484 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 6] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Failed to recv, got 0 bytes. Connection was likely closed. Did the remote server shutdown or crash?
+[rank6]:[W510 04:55:35.910621713 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:37394, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x151efd382fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x151f415106d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x151f4150f1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x151efe5f1eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15203da5b164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15204e4cd19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15204e552100 in /lib64/libc.so.6)
+
+[rank6]:[W510 04:55:35.913940336 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 6] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank6]:[W510 04:55:36.914088071 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:37394, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x151efd382fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x151f415106d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x151f4150f1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x151efe5f1eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15203da5b164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15204e4cd19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15204e552100 in /lib64/libc.so.6)
+
+[rank6]:[W510 04:55:36.916404119 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 6] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank6]:[W510 04:55:37.916545694 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:37394, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x151efd382fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x151f415106d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x151f4150f1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x151efe5f1eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15203da5b164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15204e4cd19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15204e552100 in /lib64/libc.so.6)
+
+[rank6]:[W510 04:55:37.918959252 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 6] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank6]:[W510 04:55:38.919098926 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:37394, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x151efd382fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x151f415106d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x151f4150f1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x151efe5f1eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15203da5b164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15204e4cd19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15204e552100 in /lib64/libc.so.6)
+
+[rank6]:[W510 04:55:38.921394225 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 6] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank6]:[W510 04:55:39.921538300 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:37394, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x151efd382fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x151f415106d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x151f4150f1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x151efe5f1eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15203da5b164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15204e4cd19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15204e552100 in /lib64/libc.so.6)
+
+[rank6]:[W510 04:55:39.923886038 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 6] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank6]:[W510 04:55:40.924063473 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:37394, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x151efd382fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x151f415106d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x151f4150f1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x151efe5f1eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15203da5b164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15204e4cd19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15204e552100 in /lib64/libc.so.6)
+
+[rank6]:[W510 04:55:40.926435541 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 6] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank6]:[W510 04:55:41.926581395 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:37394, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x151efd382fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x151f415106d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x151f4150f1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x151efe5f1eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15203da5b164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15204e4cd19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15204e552100 in /lib64/libc.so.6)
+
+[rank6]:[W510 04:55:41.928938684 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 6] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank6]:[W510 04:55:42.929073918 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:37394, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x151efd382fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x151f415106d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x151f4150f1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x151efe5f1eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15203da5b164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15204e4cd19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15204e552100 in /lib64/libc.so.6)
+
+[rank6]:[W510 04:55:42.931442376 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 6] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank6]:[W510 04:55:43.931587771 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:37394, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x151efd382fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x151f415106d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x151f4150f1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x151efe5f1eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15203da5b164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15204e4cd19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15204e552100 in /lib64/libc.so.6)
+
+[rank6]:[W510 04:55:43.934042009 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 6] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank6]:[W510 04:55:44.934176634 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:37394, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x151efd382fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x151f415106d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x151f4150f1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x151efe5f1eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15203da5b164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15204e4cd19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15204e552100 in /lib64/libc.so.6)
+
+[rank6]:[W510 04:55:44.936501532 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 6] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank6]:[W510 04:55:45.936620737 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:37394, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x151efd382fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x151f415106d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x151f4150f1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x151efe5f1eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15203da5b164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15204e4cd19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15204e552100 in /lib64/libc.so.6)
+
+[rank6]:[W510 04:55:45.938918765 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 6] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank6]:[W510 04:55:46.939052820 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:37394, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x151efd382fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x151f415106d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x151f4150f1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x151efe5f1eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15203da5b164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15204e4cd19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15204e552100 in /lib64/libc.so.6)
+
+[rank6]:[W510 04:55:46.941408408 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 6] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank6]:[W510 04:55:47.941540083 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:37394, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x151efd382fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x151f415106d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x151f4150f1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x151efe5f1eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15203da5b164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15204e4cd19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15204e552100 in /lib64/libc.so.6)
+
+[rank6]:[W510 04:55:47.943882951 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 6] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank6]:[W510 04:55:48.944017116 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:37394, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x151efd382fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x151f415106d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x151f4150f1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x151efe5f1eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15203da5b164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15204e4cd19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15204e552100 in /lib64/libc.so.6)
+
+[rank6]:[W510 04:55:48.946332354 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 6] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank6]:[W510 04:55:49.946460489 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:37394, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x151efd382fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x151f415106d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x151f4150f1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x151efe5f1eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15203da5b164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15204e4cd19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15204e552100 in /lib64/libc.so.6)
+
+[rank6]:[W510 04:55:49.948833977 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 6] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank6]:[W510 04:55:50.948972202 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:37394, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x151efd382fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x151f415106d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x151f4150f1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x151efe5f1eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15203da5b164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15204e4cd19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15204e552100 in /lib64/libc.so.6)
+
+[rank6]:[W510 04:55:50.951316300 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 6] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank6]:[W510 04:55:51.951448495 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:37394, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x151efd382fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x151f415106d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x151f4150f1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x151efe5f1eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15203da5b164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15204e4cd19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15204e552100 in /lib64/libc.so.6)
+
+[rank6]:[W510 04:55:51.953850573 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 6] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank6]:[W510 04:55:52.953974788 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:37394, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x151efd382fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x151f415106d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x151f4150f1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x151efe5f1eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15203da5b164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15204e4cd19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15204e552100 in /lib64/libc.so.6)
+
+[rank6]:[W510 04:55:52.956415396 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 6] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank6]:[W510 04:55:53.956543080 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:37394, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x151efd382fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x151f415106d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x151f4150f1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x151efe5f1eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15203da5b164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15204e4cd19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15204e552100 in /lib64/libc.so.6)
+
+[rank6]:[W510 04:55:53.958860879 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 6] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank6]:[W510 04:55:54.958991784 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:37394, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x151efd382fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x151f415106d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x151f4150f1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x151efe5f1eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15203da5b164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15204e4cd19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15204e552100 in /lib64/libc.so.6)
+
+[rank6]:[W510 04:55:54.961296892 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 6] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank6]:[W510 04:55:55.961422117 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:37394, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x151efd382fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x151f415106d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x151f4150f1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x151efe5f1eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15203da5b164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15204e4cd19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15204e552100 in /lib64/libc.so.6)
+
+[rank6]:[W510 04:55:55.963872715 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 6] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank6]:[W510 04:55:56.963994599 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:37394, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x151efd382fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x151f415106d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x151f4150f1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x151efe5f1eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15203da5b164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15204e4cd19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15204e552100 in /lib64/libc.so.6)
+
+[rank6]:[W510 04:55:56.966371988 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 6] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank6]:[W510 04:55:57.966491262 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:37394, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x151efd382fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x151f415106d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x151f4150f1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x151efe5f1eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15203da5b164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15204e4cd19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15204e552100 in /lib64/libc.so.6)
+
+[rank6]:[W510 04:55:57.968799371 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 6] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank6]:[W510 04:55:58.968919666 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:37394, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x151efd382fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x151f415106d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x151f4150f1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x151efe5f1eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15203da5b164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15204e4cd19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15204e552100 in /lib64/libc.so.6)
+
+[rank6]:[W510 04:55:58.971258594 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 6] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
+  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
+[rank6]:[W510 05:02:26.542989124 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
+  scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+submitit WARNING (2026-05-10 09:26:31,175) - Bypassing signal SIGTERM
+submitit WARNING (2026-05-10 09:26:31,175) - Bypassing signal SIGCONT

vjepa/inflated/d003_lora_ema_capilvd1689m/job_22606429/22606429_6_log.out

@@ -0,0 +1,520 @@
+submitit INFO (2026-05-09 16:57:36,008) - Starting with JobEnvironment(job_id=22606429, hostname=gcn75.local.snellius.surf.nl, local_rank=2(4), node=1(4), global_rank=6(16))
+submitit INFO (2026-05-09 16:57:36,008) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/job_22606429/22606429_submitted.pkl
+INFO:root:loaded pretrain params...
+{   'app': 'vjepa',
+    'cpus_per_task': 16,
+    'data': {   'batch_size': 64,
+                'crop_size': 224,
+                'dataset_fpcs': [16, 16],
+                'dataset_type': 'VideoDataset',
+                'datasets': [   '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
+                                '/scratch-shared/dcanez/data/ssv2/train.csv'],
+                'datasets_weights': [0.65, 0.35],
+                'fps': 4,
+                'num_workers': 10,
+                'patch_size': 14,
+                'persistent_workers': True,
+                'pin_mem': True,
+                'stage': [   {   'dest': 'kinetics_240',
+                                 'format': 'targz_parts',
+                                 'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
+                             {   'dest': 'ssv2',
+                                 'format': 'multipart_tar',
+                                 'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
+                'tubelet_size': 1},
+    'data_aug': {   'auto_augment': False,
+                    'motion_shift': False,
+                    'random_resize_aspect_ratio': [0.75, 1.35],
+                    'random_resize_scale': [0.3, 1.0],
+                    'reprob': 0.0},
+    'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m',
+    'loss': {'loss_exp': 1.0},
+    'mask': [   {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 8,
+                    'spatial_scale': [0.15, 0.15],
+                    'temporal_scale': [1.0, 1.0]},
+                {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 2,
+                    'spatial_scale': [0.7, 0.7],
+                    'temporal_scale': [1.0, 1.0]}],
+    'mem_per_gpu': '180G',
+    'meta': {   'dtype': 'bfloat16',
+                'knn_eval_epoch0': False,
+                'knn_eval_freq': 5,
+                'knn_eval_presets': [   {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
+                                                          'eval_videos_per_class': 25,
+                                                          'num_workers': 8,
+                                                          'train_videos_per_class': 100},
+                                            'preset': 'ucf101'},
+                                        {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
+                                                          'eval_videos_per_class': 100,
+                                                          'linear_probe': True,
+                                                          'num_workers': 8,
+                                                          'pool_type': 'temporal_concat',
+                                                          'train_videos_per_class': 500},
+                                            'preset': 'ssv2_coarse10'}],
+                'load_checkpoint': True,
+                'read_checkpoint': None,
+                'save_every_freq': 5,
+                'seed': 239,
+                'use_sdpa': True,
+                'use_wandb': True,
+                'wandb_project': 'vjepa_ablation'},
+    'metrics': {'sigreg': {}, 'std': {}},
+    'model': {   'class_token': False,
+                 'freeze_backbone': True,
+                 'is_causal': False,
+                 'lora_alpha': 16,
+                 'lora_rank': 8,
+                 'lora_targets': ['attn.qkv', 'attn.proj'],
+                 'model_name': 'vit_large_patch14_capi_lvd1689m',
+                 'num_reg_tokens': 16,
+                 'pred_depth': 6,
+                 'pred_embed_dim': 384,
+                 'pred_num_heads': 12,
+                 'rope_type': 'capi2d',
+                 'stem_type': '2d',
+                 'target_kind': 'ema',
+                 'uniform_power': True,
+                 'use_activation_checkpointing': True,
+                 'use_lora': True,
+                 'use_mask_tokens': True,
+                 'use_rope': True,
+                 'use_sdpa': True,
+                 'zero_init_mask_tokens': True},
+    'nodes': 4,
+    'optimization': {   'ema': [0.99925, 0.99925],
+                        'epochs': 100,
+                        'final_lr': 0.0001,
+                        'final_weight_decay': 0.04,
+                        'ipe': 300,
+                        'ipe_scale': 1.0,
+                        'lr': 0.0005,
+                        'start_lr': 0.0001,
+                        'warmup': 10,
+                        'weight_decay': 0.04},
+    'tasks_per_node': 4}
+INFO:root:Running pre-training of app: vjepa
+[INFO    ][2026-05-09 16:58:12][app.vjepa.train     ][main                     ] which_dtype='bfloat16'
+[INFO    ][2026-05-09 16:58:12][app.vjepa.train     ][main                     ] Disabling persistent_workers (incompatible with KNN eval)
+[INFO    ][2026-05-09 16:58:12][app.vjepa.train     ][main                     ] NCCL_SOCKET_IFNAME=eno
+[INFO    ][2026-05-09 16:59:04][app.vjepa.train     ][main                     ] Initialized (rank/world-size) 6/16, tasks_per_node=4
+[INFO    ][2026-05-09 16:59:04][root                ][stage_datasets           ] [local_rank 2/4] Staging kinetics_240 (targz_parts)
+[INFO    ][2026-05-09 16:59:04][root                ][_stage_targz_parts       ] [rank 2] Extracting 26/103 tar.gz parts to /scratch-node/dcanez.22606429/kinetics_240
+[INFO    ][2026-05-09 16:59:18][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 2/26 parts
+[INFO    ][2026-05-09 16:59:32][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 4/26 parts
+[INFO    ][2026-05-09 16:59:47][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 6/26 parts
+[INFO    ][2026-05-09 17:00:01][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 8/26 parts
+[INFO    ][2026-05-09 17:00:15][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 10/26 parts
+[INFO    ][2026-05-09 17:00:29][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 12/26 parts
+[INFO    ][2026-05-09 17:00:43][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 14/26 parts
+[INFO    ][2026-05-09 17:00:57][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 16/26 parts
+[INFO    ][2026-05-09 17:01:11][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 18/26 parts
+[INFO    ][2026-05-09 17:01:25][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 20/26 parts
+[INFO    ][2026-05-09 17:01:39][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 22/26 parts
+[INFO    ][2026-05-09 17:01:53][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 24/26 parts
+[INFO    ][2026-05-09 17:02:07][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 26/26 parts
+[INFO    ][2026-05-09 17:02:07][root                ][stage_datasets           ] [local_rank 2/4] Staging ssv2 (multipart_tar)
+[INFO    ][2026-05-09 17:03:31][app.vjepa.train     ][main                     ] Staged datasets: ['/scratch-node/dcanez.22606429/kinetics_240/train.csv', '/scratch-node/dcanez.22606429/ssv2/train.csv']
+[INFO    ][2026-05-09 17:03:54][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
+[INFO    ][2026-05-09 17:03:54][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA applied: 48 modules replaced (rank=8, alpha=16, dropout=0.0, targets=['attn.qkv', 'attn.proj'], scope=blocks)
+[INFO    ][2026-05-09 17:03:54][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA trainable params: 1,179,648
+[INFO    ][2026-05-09 17:03:55][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-09 17:03:55][root                ][init_video_model         ] VJEPAPredictorMultiSeqWrapper(
+  (backbone): VisionTransformerPredictor(
+    (predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
+    (mask_tokens): ParameterList(
+        (0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+    )
+    (predictor_blocks): ModuleList(
+      (0-5): 6 x Block(
+        (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (attn): RoPEAttention(
+          (qkv): Linear(in_features=384, out_features=1152, bias=True)
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): Linear(in_features=384, out_features=384, bias=True)
+          (proj_drop): Dropout(p=0.0, inplace=False)
+        )
+        (drop_path): Identity()
+        (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (mlp): MLP(
+          (fc1): Linear(in_features=384, out_features=1536, bias=True)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=1536, out_features=384, bias=True)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+    (predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
+  )
+)
+[INFO    ][2026-05-09 17:03:55][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-09 17:03:55][root                ][init_video_model         ] Encoder number of parameters: 1179648
+[INFO    ][2026-05-09 17:03:55][root                ][init_video_model         ] Predictor number of parameters: 11436928
+[INFO    ][2026-05-09 17:03:55][root                ][init_video_model         ] Target encoder number of parameters: 1179648
+[INFO    ][2026-05-09 17:04:02][root                ][make_videodataset        ] VideoDataset dataset created
+[INFO    ][2026-05-09 17:04:02][WeightedSampler     ][__init__                 ] Using DistributedWeightedSampler with rank 6 / 16
+[INFO    ][2026-05-09 17:04:02][root                ][make_videodataset        ] VideoDataset unsupervised data loader created
+[INFO    ][2026-05-09 17:04:02][app.vjepa.train     ][main                     ] iterations per epoch/dataset length: 300/399
+[INFO    ][2026-05-09 17:04:02][app.vjepa.train     ][main                     ] Wrapping models in DDP (rank 6)...
+[INFO    ][2026-05-09 17:05:26][app.vjepa.train     ][main                     ] Initializing loader...
+submitit INFO (2026-05-10 04:55:29,547) - Job has timed out. Ran 718 minutes out of requested 720 minutes.
+[INFO    ][2026-05-10 04:55:29][submitit            ][has_timed_out            ] Job has timed out. Ran 718 minutes out of requested 720 minutes.
+submitit WARNING (2026-05-10 04:55:29,547) - Caught signal SIGUSR2 on gcn75.local.snellius.surf.nl: this job is timed-out.
+[WARNING ][2026-05-10 04:55:29][submitit            ][checkpoint_and_try_requeue] Caught signal SIGUSR2 on gcn75.local.snellius.surf.nl: this job is timed-out.
+submitit INFO (2026-05-10 04:55:29,547) - Not checkpointing nor requeuing since I am a slave (procid=6).
+[INFO    ][2026-05-10 04:55:29][submitit            ][checkpoint_and_try_requeue] Not checkpointing nor requeuing since I am a slave (procid=6).
+submitit WARNING (2026-05-10 04:55:29,547) - Bypassing signal SIGTERM
+[WARNING ][2026-05-10 04:55:29][submitit            ][bypass                   ] Bypassing signal SIGTERM
+submitit WARNING (2026-05-10 04:55:29,547) - Bypassing signal SIGCONT
+[WARNING ][2026-05-10 04:55:29][submitit            ][bypass                   ] Bypassing signal SIGCONT
+submitit INFO (2026-05-10 04:59:12,592) - Starting with JobEnvironment(job_id=22606429, hostname=gcn75.local.snellius.surf.nl, local_rank=2(4), node=1(4), global_rank=6(16))
+submitit INFO (2026-05-10 04:59:12,592) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/job_22606429/22606429_submitted.pkl
+INFO:root:loaded pretrain params...
+{   'app': 'vjepa',
+    'cpus_per_task': 16,
+    'data': {   'batch_size': 64,
+                'crop_size': 224,
+                'dataset_fpcs': [16, 16],
+                'dataset_type': 'VideoDataset',
+                'datasets': [   '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
+                                '/scratch-shared/dcanez/data/ssv2/train.csv'],
+                'datasets_weights': [0.65, 0.35],
+                'fps': 4,
+                'num_workers': 10,
+                'patch_size': 14,
+                'persistent_workers': True,
+                'pin_mem': True,
+                'stage': [   {   'dest': 'kinetics_240',
+                                 'format': 'targz_parts',
+                                 'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
+                             {   'dest': 'ssv2',
+                                 'format': 'multipart_tar',
+                                 'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
+                'tubelet_size': 1},
+    'data_aug': {   'auto_augment': False,
+                    'motion_shift': False,
+                    'random_resize_aspect_ratio': [0.75, 1.35],
+                    'random_resize_scale': [0.3, 1.0],
+                    'reprob': 0.0},
+    'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m',
+    'loss': {'loss_exp': 1.0},
+    'mask': [   {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 8,
+                    'spatial_scale': [0.15, 0.15],
+                    'temporal_scale': [1.0, 1.0]},
+                {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 2,
+                    'spatial_scale': [0.7, 0.7],
+                    'temporal_scale': [1.0, 1.0]}],
+    'mem_per_gpu': '180G',
+    'meta': {   'dtype': 'bfloat16',
+                'knn_eval_epoch0': False,
+                'knn_eval_freq': 5,
+                'knn_eval_presets': [   {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
+                                                          'eval_videos_per_class': 25,
+                                                          'num_workers': 8,
+                                                          'train_videos_per_class': 100},
+                                            'preset': 'ucf101'},
+                                        {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
+                                                          'eval_videos_per_class': 100,
+                                                          'linear_probe': True,
+                                                          'num_workers': 8,
+                                                          'pool_type': 'temporal_concat',
+                                                          'train_videos_per_class': 500},
+                                            'preset': 'ssv2_coarse10'}],
+                'load_checkpoint': True,
+                'read_checkpoint': None,
+                'save_every_freq': 5,
+                'seed': 239,
+                'use_sdpa': True,
+                'use_wandb': True,
+                'wandb_project': 'vjepa_ablation'},
+    'metrics': {'sigreg': {}, 'std': {}},
+    'model': {   'class_token': False,
+                 'freeze_backbone': True,
+                 'is_causal': False,
+                 'lora_alpha': 16,
+                 'lora_rank': 8,
+                 'lora_targets': ['attn.qkv', 'attn.proj'],
+                 'model_name': 'vit_large_patch14_capi_lvd1689m',
+                 'num_reg_tokens': 16,
+                 'pred_depth': 6,
+                 'pred_embed_dim': 384,
+                 'pred_num_heads': 12,
+                 'rope_type': 'capi2d',
+                 'stem_type': '2d',
+                 'target_kind': 'ema',
+                 'uniform_power': True,
+                 'use_activation_checkpointing': True,
+                 'use_lora': True,
+                 'use_mask_tokens': True,
+                 'use_rope': True,
+                 'use_sdpa': True,
+                 'zero_init_mask_tokens': True},
+    'nodes': 4,
+    'optimization': {   'ema': [0.99925, 0.99925],
+                        'epochs': 100,
+                        'final_lr': 0.0001,
+                        'final_weight_decay': 0.04,
+                        'ipe': 300,
+                        'ipe_scale': 1.0,
+                        'lr': 0.0005,
+                        'start_lr': 0.0001,
+                        'warmup': 10,
+                        'weight_decay': 0.04},
+    'tasks_per_node': 4}
+INFO:root:Running pre-training of app: vjepa
+[INFO    ][2026-05-10 04:59:17][app.vjepa.train     ][main                     ] which_dtype='bfloat16'
+[INFO    ][2026-05-10 04:59:17][app.vjepa.train     ][main                     ] Disabling persistent_workers (incompatible with KNN eval)
+[INFO    ][2026-05-10 04:59:17][app.vjepa.train     ][main                     ] NCCL_SOCKET_IFNAME=eno
+[INFO    ][2026-05-10 04:59:21][app.vjepa.train     ][main                     ] Initialized (rank/world-size) 6/16, tasks_per_node=4
+[INFO    ][2026-05-10 04:59:21][root                ][stage_datasets           ] [local_rank 2/4] Staging kinetics_240 (targz_parts)
+[INFO    ][2026-05-10 04:59:21][root                ][_stage_targz_parts       ] [rank 2] Extracting 26/103 tar.gz parts to /scratch-node/dcanez.22606429/kinetics_240
+[INFO    ][2026-05-10 04:59:35][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 2/26 parts
+[INFO    ][2026-05-10 04:59:49][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 4/26 parts
+[INFO    ][2026-05-10 05:00:03][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 6/26 parts
+[INFO    ][2026-05-10 05:00:18][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 8/26 parts
+[INFO    ][2026-05-10 05:00:32][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 10/26 parts
+[INFO    ][2026-05-10 05:00:46][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 12/26 parts
+[INFO    ][2026-05-10 05:01:00][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 14/26 parts
+[INFO    ][2026-05-10 05:01:14][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 16/26 parts
+[INFO    ][2026-05-10 05:01:28][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 18/26 parts
+[INFO    ][2026-05-10 05:01:43][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 20/26 parts
+[INFO    ][2026-05-10 05:01:57][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 22/26 parts
+[INFO    ][2026-05-10 05:02:11][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 24/26 parts
+[INFO    ][2026-05-10 05:02:26][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 26/26 parts
+[INFO    ][2026-05-10 05:02:26][root                ][stage_datasets           ] [local_rank 2/4] Staging ssv2 (multipart_tar)
+[INFO    ][2026-05-10 05:04:33][app.vjepa.train     ][main                     ] Staged datasets: ['/scratch-node/dcanez.22606429/kinetics_240/train.csv', '/scratch-node/dcanez.22606429/ssv2/train.csv']
+[INFO    ][2026-05-10 05:04:47][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
+[INFO    ][2026-05-10 05:04:51][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA applied: 48 modules replaced (rank=8, alpha=16, dropout=0.0, targets=['attn.qkv', 'attn.proj'], scope=blocks)
+[INFO    ][2026-05-10 05:04:51][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA trainable params: 1,179,648
+[INFO    ][2026-05-10 05:04:52][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-10 05:04:52][root                ][init_video_model         ] VJEPAPredictorMultiSeqWrapper(
+  (backbone): VisionTransformerPredictor(
+    (predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
+    (mask_tokens): ParameterList(
+        (0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+    )
+    (predictor_blocks): ModuleList(
+      (0-5): 6 x Block(
+        (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (attn): RoPEAttention(
+          (qkv): Linear(in_features=384, out_features=1152, bias=True)
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): Linear(in_features=384, out_features=384, bias=True)
+          (proj_drop): Dropout(p=0.0, inplace=False)
+        )
+        (drop_path): Identity()
+        (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (mlp): MLP(
+          (fc1): Linear(in_features=384, out_features=1536, bias=True)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=1536, out_features=384, bias=True)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+    (predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
+  )
+)
+[INFO    ][2026-05-10 05:04:52][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-10 05:04:52][root                ][init_video_model         ] Encoder number of parameters: 1179648
+[INFO    ][2026-05-10 05:04:52][root                ][init_video_model         ] Predictor number of parameters: 11436928
+[INFO    ][2026-05-10 05:04:52][root                ][init_video_model         ] Target encoder number of parameters: 1179648
+[INFO    ][2026-05-10 05:04:53][root                ][make_videodataset        ] VideoDataset dataset created
+[INFO    ][2026-05-10 05:04:53][WeightedSampler     ][__init__                 ] Using DistributedWeightedSampler with rank 6 / 16
+[INFO    ][2026-05-10 05:04:53][root                ][make_videodataset        ] VideoDataset unsupervised data loader created
+[INFO    ][2026-05-10 05:04:53][app.vjepa.train     ][main                     ] iterations per epoch/dataset length: 300/399
+[INFO    ][2026-05-10 05:04:53][app.vjepa.train     ][main                     ] Wrapping models in DDP (rank 6)...
+[INFO    ][2026-05-10 05:06:08][root                ][load_checkpoint          ] Loading checkpoint from /scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/latest.pt
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] loaded pretrained encoder from epoch 33 with msg: <All keys matched successfully>
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] loaded pretrained predictor from epoch 33 with msg: <All keys matched successfully>
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] loaded pretrained target encoder from epoch 33 with msg: <All keys matched successfully>
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] loaded optimizers from epoch 33
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] read-path: /scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/latest.pt
+[INFO    ][2026-05-10 05:06:09][app.vjepa.train     ][main                     ] Initializing loader...
+submitit WARNING (2026-05-10 09:26:31,175) - Bypassing signal SIGTERM
+[WARNING ][2026-05-10 09:26:31][submitit            ][bypass                   ] Bypassing signal SIGTERM
+submitit WARNING (2026-05-10 09:26:31,175) - Bypassing signal SIGCONT
+[WARNING ][2026-05-10 09:26:31][submitit            ][bypass                   ] Bypassing signal SIGCONT

vjepa/inflated/d003_lora_ema_capilvd1689m/job_22606429/22606429_7_log.err

@@ -0,0 +1,294 @@
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
+  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
+[rank7]:[W509 17:01:56.471144805 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
+  scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+submitit WARNING (2026-05-10 04:55:29,543) - Caught signal SIGUSR2 on gcn75.local.snellius.surf.nl: this job is timed-out.
+submitit WARNING (2026-05-10 04:55:29,543) - Bypassing signal SIGTERM
+submitit WARNING (2026-05-10 04:55:29,543) - Bypassing signal SIGCONT
+[rank7]:[W510 04:55:34.749079484 TCPStore.cpp:125] [c10d] recvValue failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:37388, remote=[gcn74.local.snellius.surf.nl]:37129): Failed to recv, got 0 bytes. Connection was likely closed. Did the remote server shutdown or crash?
+Exception raised from recvBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:682 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15383906ffdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a3325d (0x15387d1fe25d in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x273 (0x15387d1fc1f3 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15383a2deeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153979748164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15398a1ba19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15398a23f100 in /lib64/libc.so.6)
+
+[rank7]:[W510 04:55:34.751641581 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 7] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Failed to recv, got 0 bytes. Connection was likely closed. Did the remote server shutdown or crash?
+[rank7]:[W510 04:55:35.751792881 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:37388, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15383906ffdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15387d1fd6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15387d1fc1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15383a2deeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153979748164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15398a1ba19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15398a23f100 in /lib64/libc.so.6)
+
+[rank7]:[W510 04:55:35.755100684 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 7] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank7]:[W510 04:55:36.755243829 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:37388, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15383906ffdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15387d1fd6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15387d1fc1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15383a2deeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153979748164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15398a1ba19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15398a23f100 in /lib64/libc.so.6)
+
+[rank7]:[W510 04:55:36.757575487 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 7] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank7]:[W510 04:55:37.757711282 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:37388, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15383906ffdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15387d1fd6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15387d1fc1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15383a2deeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153979748164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15398a1ba19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15398a23f100 in /lib64/libc.so.6)
+
+[rank7]:[W510 04:55:37.760119900 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 7] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank7]:[W510 04:55:38.760273344 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:37388, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15383906ffdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15387d1fd6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15387d1fc1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15383a2deeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153979748164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15398a1ba19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15398a23f100 in /lib64/libc.so.6)
+
+[rank7]:[W510 04:55:38.762624323 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 7] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank7]:[W510 04:55:39.762738888 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:37388, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15383906ffdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15387d1fd6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15387d1fc1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15383a2deeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153979748164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15398a1ba19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15398a23f100 in /lib64/libc.so.6)
+
+[rank7]:[W510 04:55:39.765148635 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 7] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank7]:[W510 04:55:40.765290980 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:37388, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15383906ffdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15387d1fd6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15387d1fc1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15383a2deeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153979748164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15398a1ba19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15398a23f100 in /lib64/libc.so.6)
+
+[rank7]:[W510 04:55:40.767737228 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 7] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank7]:[W510 04:55:41.767878953 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:37388, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15383906ffdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15387d1fd6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15387d1fc1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15383a2deeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153979748164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15398a1ba19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15398a23f100 in /lib64/libc.so.6)
+
+[rank7]:[W510 04:55:41.770168621 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 7] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank7]:[W510 04:55:42.770310826 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:37388, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15383906ffdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15387d1fd6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15387d1fc1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15383a2deeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153979748164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15398a1ba19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15398a23f100 in /lib64/libc.so.6)
+
+[rank7]:[W510 04:55:42.772608834 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 7] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank7]:[W510 04:55:43.772749809 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:37388, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15383906ffdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15387d1fd6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15387d1fc1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15383a2deeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153979748164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15398a1ba19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15398a23f100 in /lib64/libc.so.6)
+
+[rank7]:[W510 04:55:43.775102897 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 7] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank7]:[W510 04:55:44.775237512 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:37388, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15383906ffdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15387d1fd6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15387d1fc1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15383a2deeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153979748164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15398a1ba19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15398a23f100 in /lib64/libc.so.6)
+
+[rank7]:[W510 04:55:44.777523061 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 7] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank7]:[W510 04:55:45.777663945 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:37388, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15383906ffdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15387d1fd6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15387d1fc1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15383a2deeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153979748164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15398a1ba19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15398a23f100 in /lib64/libc.so.6)
+
+[rank7]:[W510 04:55:45.779967934 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 7] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank7]:[W510 04:55:46.780095229 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:37388, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15383906ffdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15387d1fd6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15387d1fc1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15383a2deeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153979748164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15398a1ba19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15398a23f100 in /lib64/libc.so.6)
+
+[rank7]:[W510 04:55:46.782484217 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 7] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank7]:[W510 04:55:47.782613611 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:37388, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15383906ffdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15387d1fd6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15387d1fc1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15383a2deeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153979748164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15398a1ba19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15398a23f100 in /lib64/libc.so.6)
+
+[rank7]:[W510 04:55:47.784904940 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 7] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank7]:[W510 04:55:48.785051925 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:37388, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15383906ffdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15387d1fd6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15387d1fc1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15383a2deeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153979748164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15398a1ba19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15398a23f100 in /lib64/libc.so.6)
+
+[rank7]:[W510 04:55:48.787434093 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 7] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank7]:[W510 04:55:49.787587847 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:37388, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15383906ffdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15387d1fd6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15387d1fc1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15383a2deeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153979748164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15398a1ba19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15398a23f100 in /lib64/libc.so.6)
+
+[rank7]:[W510 04:55:49.789997885 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 7] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank7]:[W510 04:55:50.790137880 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:37388, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15383906ffdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15387d1fd6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15387d1fc1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15383a2deeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153979748164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15398a1ba19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15398a23f100 in /lib64/libc.so.6)
+
+[rank7]:[W510 04:55:50.792445579 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 7] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank7]:[W510 04:55:51.792581273 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:37388, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15383906ffdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15387d1fd6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15387d1fc1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15383a2deeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153979748164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15398a1ba19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15398a23f100 in /lib64/libc.so.6)
+
+[rank7]:[W510 04:55:51.794899452 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 7] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank7]:[W510 04:55:52.795035616 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:37388, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15383906ffdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15387d1fd6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15387d1fc1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15383a2deeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153979748164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15398a1ba19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15398a23f100 in /lib64/libc.so.6)
+
+[rank7]:[W510 04:55:52.797427504 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 7] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank7]:[W510 04:55:53.797548239 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:37388, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15383906ffdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15387d1fd6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15387d1fc1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15383a2deeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153979748164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15398a1ba19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15398a23f100 in /lib64/libc.so.6)
+
+[rank7]:[W510 04:55:53.799791458 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 7] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank7]:[W510 04:55:54.799918393 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:37388, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15383906ffdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15387d1fd6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15387d1fc1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15383a2deeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153979748164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15398a1ba19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15398a23f100 in /lib64/libc.so.6)
+
+[rank7]:[W510 04:55:54.802176092 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 7] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank7]:[W510 04:55:55.802299016 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:37388, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15383906ffdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15387d1fd6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15387d1fc1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15383a2deeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153979748164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15398a1ba19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15398a23f100 in /lib64/libc.so.6)
+
+[rank7]:[W510 04:55:55.804721724 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 7] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank7]:[W510 04:55:56.804847399 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:37388, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15383906ffdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15387d1fd6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15387d1fc1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15383a2deeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153979748164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15398a1ba19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15398a23f100 in /lib64/libc.so.6)
+
+[rank7]:[W510 04:55:56.807245827 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 7] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank7]:[W510 04:55:57.807384162 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:37388, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15383906ffdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15387d1fd6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15387d1fc1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15383a2deeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153979748164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15398a1ba19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15398a23f100 in /lib64/libc.so.6)
+
+[rank7]:[W510 04:55:57.809672680 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 7] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank7]:[W510 04:55:58.809825785 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:37388, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15383906ffdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15387d1fd6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15387d1fc1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15383a2deeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153979748164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15398a1ba19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15398a23f100 in /lib64/libc.so.6)
+
+[rank7]:[W510 04:55:58.812226163 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 7] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
+  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
+[rank7]:[W510 05:02:19.040885243 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
+  scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+submitit WARNING (2026-05-10 09:26:31,163) - Bypassing signal SIGTERM
+submitit WARNING (2026-05-10 09:26:31,163) - Bypassing signal SIGCONT

vjepa/inflated/d003_lora_ema_capilvd1689m/job_22606429/22606429_7_log.out

@@ -0,0 +1,520 @@
+submitit INFO (2026-05-09 16:57:36,008) - Starting with JobEnvironment(job_id=22606429, hostname=gcn75.local.snellius.surf.nl, local_rank=3(4), node=1(4), global_rank=7(16))
+submitit INFO (2026-05-09 16:57:36,008) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/job_22606429/22606429_submitted.pkl
+INFO:root:loaded pretrain params...
+{   'app': 'vjepa',
+    'cpus_per_task': 16,
+    'data': {   'batch_size': 64,
+                'crop_size': 224,
+                'dataset_fpcs': [16, 16],
+                'dataset_type': 'VideoDataset',
+                'datasets': [   '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
+                                '/scratch-shared/dcanez/data/ssv2/train.csv'],
+                'datasets_weights': [0.65, 0.35],
+                'fps': 4,
+                'num_workers': 10,
+                'patch_size': 14,
+                'persistent_workers': True,
+                'pin_mem': True,
+                'stage': [   {   'dest': 'kinetics_240',
+                                 'format': 'targz_parts',
+                                 'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
+                             {   'dest': 'ssv2',
+                                 'format': 'multipart_tar',
+                                 'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
+                'tubelet_size': 1},
+    'data_aug': {   'auto_augment': False,
+                    'motion_shift': False,
+                    'random_resize_aspect_ratio': [0.75, 1.35],
+                    'random_resize_scale': [0.3, 1.0],
+                    'reprob': 0.0},
+    'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m',
+    'loss': {'loss_exp': 1.0},
+    'mask': [   {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 8,
+                    'spatial_scale': [0.15, 0.15],
+                    'temporal_scale': [1.0, 1.0]},
+                {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 2,
+                    'spatial_scale': [0.7, 0.7],
+                    'temporal_scale': [1.0, 1.0]}],
+    'mem_per_gpu': '180G',
+    'meta': {   'dtype': 'bfloat16',
+                'knn_eval_epoch0': False,
+                'knn_eval_freq': 5,
+                'knn_eval_presets': [   {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
+                                                          'eval_videos_per_class': 25,
+                                                          'num_workers': 8,
+                                                          'train_videos_per_class': 100},
+                                            'preset': 'ucf101'},
+                                        {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
+                                                          'eval_videos_per_class': 100,
+                                                          'linear_probe': True,
+                                                          'num_workers': 8,
+                                                          'pool_type': 'temporal_concat',
+                                                          'train_videos_per_class': 500},
+                                            'preset': 'ssv2_coarse10'}],
+                'load_checkpoint': True,
+                'read_checkpoint': None,
+                'save_every_freq': 5,
+                'seed': 239,
+                'use_sdpa': True,
+                'use_wandb': True,
+                'wandb_project': 'vjepa_ablation'},
+    'metrics': {'sigreg': {}, 'std': {}},
+    'model': {   'class_token': False,
+                 'freeze_backbone': True,
+                 'is_causal': False,
+                 'lora_alpha': 16,
+                 'lora_rank': 8,
+                 'lora_targets': ['attn.qkv', 'attn.proj'],
+                 'model_name': 'vit_large_patch14_capi_lvd1689m',
+                 'num_reg_tokens': 16,
+                 'pred_depth': 6,
+                 'pred_embed_dim': 384,
+                 'pred_num_heads': 12,
+                 'rope_type': 'capi2d',
+                 'stem_type': '2d',
+                 'target_kind': 'ema',
+                 'uniform_power': True,
+                 'use_activation_checkpointing': True,
+                 'use_lora': True,
+                 'use_mask_tokens': True,
+                 'use_rope': True,
+                 'use_sdpa': True,
+                 'zero_init_mask_tokens': True},
+    'nodes': 4,
+    'optimization': {   'ema': [0.99925, 0.99925],
+                        'epochs': 100,
+                        'final_lr': 0.0001,
+                        'final_weight_decay': 0.04,
+                        'ipe': 300,
+                        'ipe_scale': 1.0,
+                        'lr': 0.0005,
+                        'start_lr': 0.0001,
+                        'warmup': 10,
+                        'weight_decay': 0.04},
+    'tasks_per_node': 4}
+INFO:root:Running pre-training of app: vjepa
+[INFO    ][2026-05-09 16:58:12][app.vjepa.train     ][main                     ] which_dtype='bfloat16'
+[INFO    ][2026-05-09 16:58:12][app.vjepa.train     ][main                     ] Disabling persistent_workers (incompatible with KNN eval)
+[INFO    ][2026-05-09 16:58:12][app.vjepa.train     ][main                     ] NCCL_SOCKET_IFNAME=eno
+[INFO    ][2026-05-09 16:58:57][app.vjepa.train     ][main                     ] Initialized (rank/world-size) 7/16, tasks_per_node=4
+[INFO    ][2026-05-09 16:58:58][root                ][stage_datasets           ] [local_rank 3/4] Staging kinetics_240 (targz_parts)
+[INFO    ][2026-05-09 16:58:58][root                ][_stage_targz_parts       ] [rank 3] Extracting 25/103 tar.gz parts to /scratch-node/dcanez.22606429/kinetics_240
+[INFO    ][2026-05-09 16:59:13][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 2/25 parts
+[INFO    ][2026-05-09 16:59:27][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 4/25 parts
+[INFO    ][2026-05-09 16:59:41][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 6/25 parts
+[INFO    ][2026-05-09 16:59:55][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 8/25 parts
+[INFO    ][2026-05-09 17:00:10][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 10/25 parts
+[INFO    ][2026-05-09 17:00:24][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 12/25 parts
+[INFO    ][2026-05-09 17:00:38][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 14/25 parts
+[INFO    ][2026-05-09 17:00:52][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 16/25 parts
+[INFO    ][2026-05-09 17:01:06][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 18/25 parts
+[INFO    ][2026-05-09 17:01:20][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 20/25 parts
+[INFO    ][2026-05-09 17:01:34][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 22/25 parts
+[INFO    ][2026-05-09 17:01:48][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 24/25 parts
+[INFO    ][2026-05-09 17:01:55][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 25/25 parts
+[INFO    ][2026-05-09 17:01:55][root                ][stage_datasets           ] [local_rank 3/4] Staging ssv2 (multipart_tar)
+[INFO    ][2026-05-09 17:03:31][app.vjepa.train     ][main                     ] Staged datasets: ['/scratch-node/dcanez.22606429/kinetics_240/train.csv', '/scratch-node/dcanez.22606429/ssv2/train.csv']
+[INFO    ][2026-05-09 17:03:51][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
+[INFO    ][2026-05-09 17:03:53][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA applied: 48 modules replaced (rank=8, alpha=16, dropout=0.0, targets=['attn.qkv', 'attn.proj'], scope=blocks)
+[INFO    ][2026-05-09 17:03:53][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA trainable params: 1,179,648
+[INFO    ][2026-05-09 17:03:53][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-09 17:03:53][root                ][init_video_model         ] VJEPAPredictorMultiSeqWrapper(
+  (backbone): VisionTransformerPredictor(
+    (predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
+    (mask_tokens): ParameterList(
+        (0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+    )
+    (predictor_blocks): ModuleList(
+      (0-5): 6 x Block(
+        (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (attn): RoPEAttention(
+          (qkv): Linear(in_features=384, out_features=1152, bias=True)
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): Linear(in_features=384, out_features=384, bias=True)
+          (proj_drop): Dropout(p=0.0, inplace=False)
+        )
+        (drop_path): Identity()
+        (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (mlp): MLP(
+          (fc1): Linear(in_features=384, out_features=1536, bias=True)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=1536, out_features=384, bias=True)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+    (predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
+  )
+)
+[INFO    ][2026-05-09 17:03:53][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-09 17:03:53][root                ][init_video_model         ] Encoder number of parameters: 1179648
+[INFO    ][2026-05-09 17:03:53][root                ][init_video_model         ] Predictor number of parameters: 11436928
+[INFO    ][2026-05-09 17:03:53][root                ][init_video_model         ] Target encoder number of parameters: 1179648
+[INFO    ][2026-05-09 17:04:02][root                ][make_videodataset        ] VideoDataset dataset created
+[INFO    ][2026-05-09 17:04:02][WeightedSampler     ][__init__                 ] Using DistributedWeightedSampler with rank 7 / 16
+[INFO    ][2026-05-09 17:04:02][root                ][make_videodataset        ] VideoDataset unsupervised data loader created
+[INFO    ][2026-05-09 17:04:02][app.vjepa.train     ][main                     ] iterations per epoch/dataset length: 300/399
+[INFO    ][2026-05-09 17:04:02][app.vjepa.train     ][main                     ] Wrapping models in DDP (rank 7)...
+[INFO    ][2026-05-09 17:05:26][app.vjepa.train     ][main                     ] Initializing loader...
+submitit INFO (2026-05-10 04:55:29,542) - Job has timed out. Ran 718 minutes out of requested 720 minutes.
+[INFO    ][2026-05-10 04:55:29][submitit            ][has_timed_out            ] Job has timed out. Ran 718 minutes out of requested 720 minutes.
+submitit WARNING (2026-05-10 04:55:29,543) - Caught signal SIGUSR2 on gcn75.local.snellius.surf.nl: this job is timed-out.
+[WARNING ][2026-05-10 04:55:29][submitit            ][checkpoint_and_try_requeue] Caught signal SIGUSR2 on gcn75.local.snellius.surf.nl: this job is timed-out.
+submitit INFO (2026-05-10 04:55:29,543) - Not checkpointing nor requeuing since I am a slave (procid=7).
+[INFO    ][2026-05-10 04:55:29][submitit            ][checkpoint_and_try_requeue] Not checkpointing nor requeuing since I am a slave (procid=7).
+submitit WARNING (2026-05-10 04:55:29,543) - Bypassing signal SIGTERM
+[WARNING ][2026-05-10 04:55:29][submitit            ][bypass                   ] Bypassing signal SIGTERM
+submitit WARNING (2026-05-10 04:55:29,543) - Bypassing signal SIGCONT
+[WARNING ][2026-05-10 04:55:29][submitit            ][bypass                   ] Bypassing signal SIGCONT
+submitit INFO (2026-05-10 04:59:12,592) - Starting with JobEnvironment(job_id=22606429, hostname=gcn75.local.snellius.surf.nl, local_rank=3(4), node=1(4), global_rank=7(16))
+submitit INFO (2026-05-10 04:59:12,592) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/job_22606429/22606429_submitted.pkl
+INFO:root:loaded pretrain params...
+{   'app': 'vjepa',
+    'cpus_per_task': 16,
+    'data': {   'batch_size': 64,
+                'crop_size': 224,
+                'dataset_fpcs': [16, 16],
+                'dataset_type': 'VideoDataset',
+                'datasets': [   '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
+                                '/scratch-shared/dcanez/data/ssv2/train.csv'],
+                'datasets_weights': [0.65, 0.35],
+                'fps': 4,
+                'num_workers': 10,
+                'patch_size': 14,
+                'persistent_workers': True,
+                'pin_mem': True,
+                'stage': [   {   'dest': 'kinetics_240',
+                                 'format': 'targz_parts',
+                                 'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
+                             {   'dest': 'ssv2',
+                                 'format': 'multipart_tar',
+                                 'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
+                'tubelet_size': 1},
+    'data_aug': {   'auto_augment': False,
+                    'motion_shift': False,
+                    'random_resize_aspect_ratio': [0.75, 1.35],
+                    'random_resize_scale': [0.3, 1.0],
+                    'reprob': 0.0},
+    'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m',
+    'loss': {'loss_exp': 1.0},
+    'mask': [   {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 8,
+                    'spatial_scale': [0.15, 0.15],
+                    'temporal_scale': [1.0, 1.0]},
+                {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 2,
+                    'spatial_scale': [0.7, 0.7],
+                    'temporal_scale': [1.0, 1.0]}],
+    'mem_per_gpu': '180G',
+    'meta': {   'dtype': 'bfloat16',
+                'knn_eval_epoch0': False,
+                'knn_eval_freq': 5,
+                'knn_eval_presets': [   {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
+                                                          'eval_videos_per_class': 25,
+                                                          'num_workers': 8,
+                                                          'train_videos_per_class': 100},
+                                            'preset': 'ucf101'},
+                                        {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
+                                                          'eval_videos_per_class': 100,
+                                                          'linear_probe': True,
+                                                          'num_workers': 8,
+                                                          'pool_type': 'temporal_concat',
+                                                          'train_videos_per_class': 500},
+                                            'preset': 'ssv2_coarse10'}],
+                'load_checkpoint': True,
+                'read_checkpoint': None,
+                'save_every_freq': 5,
+                'seed': 239,
+                'use_sdpa': True,
+                'use_wandb': True,
+                'wandb_project': 'vjepa_ablation'},
+    'metrics': {'sigreg': {}, 'std': {}},
+    'model': {   'class_token': False,
+                 'freeze_backbone': True,
+                 'is_causal': False,
+                 'lora_alpha': 16,
+                 'lora_rank': 8,
+                 'lora_targets': ['attn.qkv', 'attn.proj'],
+                 'model_name': 'vit_large_patch14_capi_lvd1689m',
+                 'num_reg_tokens': 16,
+                 'pred_depth': 6,
+                 'pred_embed_dim': 384,
+                 'pred_num_heads': 12,
+                 'rope_type': 'capi2d',
+                 'stem_type': '2d',
+                 'target_kind': 'ema',
+                 'uniform_power': True,
+                 'use_activation_checkpointing': True,
+                 'use_lora': True,
+                 'use_mask_tokens': True,
+                 'use_rope': True,
+                 'use_sdpa': True,
+                 'zero_init_mask_tokens': True},
+    'nodes': 4,
+    'optimization': {   'ema': [0.99925, 0.99925],
+                        'epochs': 100,
+                        'final_lr': 0.0001,
+                        'final_weight_decay': 0.04,
+                        'ipe': 300,
+                        'ipe_scale': 1.0,
+                        'lr': 0.0005,
+                        'start_lr': 0.0001,
+                        'warmup': 10,
+                        'weight_decay': 0.04},
+    'tasks_per_node': 4}
+INFO:root:Running pre-training of app: vjepa
+[INFO    ][2026-05-10 04:59:17][app.vjepa.train     ][main                     ] which_dtype='bfloat16'
+[INFO    ][2026-05-10 04:59:17][app.vjepa.train     ][main                     ] Disabling persistent_workers (incompatible with KNN eval)
+[INFO    ][2026-05-10 04:59:17][app.vjepa.train     ][main                     ] NCCL_SOCKET_IFNAME=eno
+[INFO    ][2026-05-10 04:59:20][app.vjepa.train     ][main                     ] Initialized (rank/world-size) 7/16, tasks_per_node=4
+[INFO    ][2026-05-10 04:59:20][root                ][stage_datasets           ] [local_rank 3/4] Staging kinetics_240 (targz_parts)
+[INFO    ][2026-05-10 04:59:20][root                ][_stage_targz_parts       ] [rank 3] Extracting 25/103 tar.gz parts to /scratch-node/dcanez.22606429/kinetics_240
+[INFO    ][2026-05-10 04:59:34][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 2/25 parts
+[INFO    ][2026-05-10 04:59:48][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 4/25 parts
+[INFO    ][2026-05-10 05:00:03][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 6/25 parts
+[INFO    ][2026-05-10 05:00:17][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 8/25 parts
+[INFO    ][2026-05-10 05:00:32][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 10/25 parts
+[INFO    ][2026-05-10 05:00:46][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 12/25 parts
+[INFO    ][2026-05-10 05:01:00][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 14/25 parts
+[INFO    ][2026-05-10 05:01:14][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 16/25 parts
+[INFO    ][2026-05-10 05:01:28][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 18/25 parts
+[INFO    ][2026-05-10 05:01:43][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 20/25 parts
+[INFO    ][2026-05-10 05:01:58][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 22/25 parts
+[INFO    ][2026-05-10 05:02:12][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 24/25 parts
+[INFO    ][2026-05-10 05:02:19][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 25/25 parts
+[INFO    ][2026-05-10 05:02:19][root                ][stage_datasets           ] [local_rank 3/4] Staging ssv2 (multipart_tar)
+[INFO    ][2026-05-10 05:04:33][app.vjepa.train     ][main                     ] Staged datasets: ['/scratch-node/dcanez.22606429/kinetics_240/train.csv', '/scratch-node/dcanez.22606429/ssv2/train.csv']
+[INFO    ][2026-05-10 05:04:47][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
+[INFO    ][2026-05-10 05:04:51][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA applied: 48 modules replaced (rank=8, alpha=16, dropout=0.0, targets=['attn.qkv', 'attn.proj'], scope=blocks)
+[INFO    ][2026-05-10 05:04:51][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA trainable params: 1,179,648
+[INFO    ][2026-05-10 05:04:52][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-10 05:04:52][root                ][init_video_model         ] VJEPAPredictorMultiSeqWrapper(
+  (backbone): VisionTransformerPredictor(
+    (predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
+    (mask_tokens): ParameterList(
+        (0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+    )
+    (predictor_blocks): ModuleList(
+      (0-5): 6 x Block(
+        (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (attn): RoPEAttention(
+          (qkv): Linear(in_features=384, out_features=1152, bias=True)
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): Linear(in_features=384, out_features=384, bias=True)
+          (proj_drop): Dropout(p=0.0, inplace=False)
+        )
+        (drop_path): Identity()
+        (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (mlp): MLP(
+          (fc1): Linear(in_features=384, out_features=1536, bias=True)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=1536, out_features=384, bias=True)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+    (predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
+  )
+)
+[INFO    ][2026-05-10 05:04:52][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-10 05:04:52][root                ][init_video_model         ] Encoder number of parameters: 1179648
+[INFO    ][2026-05-10 05:04:52][root                ][init_video_model         ] Predictor number of parameters: 11436928
+[INFO    ][2026-05-10 05:04:52][root                ][init_video_model         ] Target encoder number of parameters: 1179648
+[INFO    ][2026-05-10 05:04:53][root                ][make_videodataset        ] VideoDataset dataset created
+[INFO    ][2026-05-10 05:04:53][WeightedSampler     ][__init__                 ] Using DistributedWeightedSampler with rank 7 / 16
+[INFO    ][2026-05-10 05:04:53][root                ][make_videodataset        ] VideoDataset unsupervised data loader created
+[INFO    ][2026-05-10 05:04:53][app.vjepa.train     ][main                     ] iterations per epoch/dataset length: 300/399
+[INFO    ][2026-05-10 05:04:53][app.vjepa.train     ][main                     ] Wrapping models in DDP (rank 7)...
+[INFO    ][2026-05-10 05:06:08][root                ][load_checkpoint          ] Loading checkpoint from /scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/latest.pt
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] loaded pretrained encoder from epoch 33 with msg: <All keys matched successfully>
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] loaded pretrained predictor from epoch 33 with msg: <All keys matched successfully>
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] loaded pretrained target encoder from epoch 33 with msg: <All keys matched successfully>
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] loaded optimizers from epoch 33
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] read-path: /scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/latest.pt
+[INFO    ][2026-05-10 05:06:09][app.vjepa.train     ][main                     ] Initializing loader...
+submitit WARNING (2026-05-10 09:26:31,163) - Bypassing signal SIGTERM
+[WARNING ][2026-05-10 09:26:31][submitit            ][bypass                   ] Bypassing signal SIGTERM
+submitit WARNING (2026-05-10 09:26:31,163) - Bypassing signal SIGCONT
+[WARNING ][2026-05-10 09:26:31][submitit            ][bypass                   ] Bypassing signal SIGCONT

vjepa/inflated/d003_lora_ema_capilvd1689m/job_22606429/22606429_8_log.err

@@ -0,0 +1,314 @@
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
+  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
+[rank8]:[W509 17:03:29.608991782 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
+  scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+submitit WARNING (2026-05-10 04:55:29,447) - Caught signal SIGUSR2 on gcn76.local.snellius.surf.nl: this job is timed-out.
+submitit WARNING (2026-05-10 04:55:29,448) - Bypassing signal SIGCONT
+submitit WARNING (2026-05-10 04:55:29,547) - Bypassing signal SIGTERM
+submitit WARNING (2026-05-10 04:55:29,620) - Bypassing signal SIGCONT
+[rank8]:[W510 04:55:34.500204443 TCPStore.cpp:125] [c10d] recvValue failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:38552, remote=[gcn74.local.snellius.surf.nl]:37129): Failed to recv, got 0 bytes. Connection was likely closed. Did the remote server shutdown or crash?
+Exception raised from recvBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:682 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14953630efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a3325d (0x14957a49d25d in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x273 (0x14957a49b1f3 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14953757deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x149676ae7164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14968755919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1496875de100 in /lib64/libc.so.6)
+
+[rank8]:[W510 04:55:34.549990331 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 8] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Failed to recv, got 0 bytes. Connection was likely closed. Did the remote server shutdown or crash?
+[rank8]:[W510 04:55:35.550185793 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:38552, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14953630efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14957a49c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14957a49b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14953757deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x149676ae7164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14968755919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1496875de100 in /lib64/libc.so.6)
+
+[rank8]:[W510 04:55:35.553259190 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 8] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank8]:[W510 04:55:36.553438413 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:38552, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14953630efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14957a49c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14957a49b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14953757deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x149676ae7164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14968755919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1496875de100 in /lib64/libc.so.6)
+
+[rank8]:[W510 04:55:36.555763293 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 8] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank8]:[W510 04:55:37.555912485 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:38552, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14953630efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14957a49c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14957a49b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14953757deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x149676ae7164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14968755919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1496875de100 in /lib64/libc.so.6)
+
+[rank8]:[W510 04:55:37.558290125 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 8] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank8]:[W510 04:55:38.558436348 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:38552, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14953630efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14957a49c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14957a49b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14953757deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x149676ae7164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14968755919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1496875de100 in /lib64/libc.so.6)
+
+[rank8]:[W510 04:55:38.560707738 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 8] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank8]:[W510 04:55:39.560865901 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:38552, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14953630efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14957a49c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14957a49b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14953757deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x149676ae7164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14968755919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1496875de100 in /lib64/libc.so.6)
+
+[rank8]:[W510 04:55:39.563225971 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 8] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank8]:[W510 04:55:40.563408863 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:38552, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14953630efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14957a49c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14957a49b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14953757deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x149676ae7164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14968755919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1496875de100 in /lib64/libc.so.6)
+
+[rank8]:[W510 04:55:40.565876372 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 8] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank8]:[W510 04:55:41.566039885 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:38552, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14953630efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14957a49c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14957a49b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14953757deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x149676ae7164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14968755919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1496875de100 in /lib64/libc.so.6)
+
+[rank8]:[W510 04:55:41.568277555 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 8] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank8]:[W510 04:55:42.568427568 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:38552, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14953630efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14957a49c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14957a49b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14953757deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x149676ae7164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14968755919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1496875de100 in /lib64/libc.so.6)
+
+[rank8]:[W510 04:55:42.570726958 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 8] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank8]:[W510 04:55:43.570877211 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:38552, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14953630efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14957a49c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14957a49b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14953757deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x149676ae7164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14968755919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1496875de100 in /lib64/libc.so.6)
+
+[rank8]:[W510 04:55:43.573243871 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 8] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank8]:[W510 04:55:44.573416183 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:38552, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14953630efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14957a49c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14957a49b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14953757deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x149676ae7164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14968755919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1496875de100 in /lib64/libc.so.6)
+
+[rank8]:[W510 04:55:44.575774603 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 8] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank8]:[W510 04:55:45.575927636 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:38552, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14953630efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14957a49c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14957a49b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14953757deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x149676ae7164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14968755919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1496875de100 in /lib64/libc.so.6)
+
+[rank8]:[W510 04:55:45.578130246 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 8] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank8]:[W510 04:55:46.578285089 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:38552, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14953630efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14957a49c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14957a49b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14953757deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x149676ae7164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14968755919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1496875de100 in /lib64/libc.so.6)
+
+[rank8]:[W510 04:55:46.580703459 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 8] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank8]:[W510 04:55:47.580861191 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:38552, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14953630efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14957a49c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14957a49b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14953757deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x149676ae7164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14968755919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1496875de100 in /lib64/libc.so.6)
+
+[rank8]:[W510 04:55:47.583169921 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 8] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank8]:[W510 04:55:48.583322004 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:38552, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14953630efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14957a49c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14957a49b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14953757deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x149676ae7164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14968755919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1496875de100 in /lib64/libc.so.6)
+
+[rank8]:[W510 04:55:48.585552464 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 8] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank8]:[W510 04:55:49.585710017 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:38552, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14953630efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14957a49c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14957a49b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14953757deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x149676ae7164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14968755919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1496875de100 in /lib64/libc.so.6)
+
+[rank8]:[W510 04:55:49.588082377 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 8] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank8]:[W510 04:55:50.588251499 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:38552, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14953630efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14957a49c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14957a49b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14953757deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x149676ae7164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14968755919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1496875de100 in /lib64/libc.so.6)
+
+[rank8]:[W510 04:55:50.590547660 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 8] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank8]:[W510 04:55:51.590725782 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:38552, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14953630efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14957a49c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14957a49b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14953757deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x149676ae7164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14968755919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1496875de100 in /lib64/libc.so.6)
+
+[rank8]:[W510 04:55:51.593010212 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 8] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank8]:[W510 04:55:52.593171585 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:38552, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14953630efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14957a49c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14957a49b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14953757deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x149676ae7164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14968755919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1496875de100 in /lib64/libc.so.6)
+
+[rank8]:[W510 04:55:52.595552805 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 8] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank8]:[W510 04:55:53.595696237 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:38552, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14953630efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14957a49c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14957a49b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14953757deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x149676ae7164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14968755919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1496875de100 in /lib64/libc.so.6)
+
+[rank8]:[W510 04:55:53.597939848 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 8] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank8]:[W510 04:55:54.598100200 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:38552, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14953630efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14957a49c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14957a49b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14953757deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x149676ae7164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14968755919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1496875de100 in /lib64/libc.so.6)
+
+[rank8]:[W510 04:55:54.600385121 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 8] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank8]:[W510 04:55:55.600528423 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:38552, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14953630efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14957a49c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14957a49b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14953757deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x149676ae7164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14968755919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1496875de100 in /lib64/libc.so.6)
+
+[rank8]:[W510 04:55:55.602867203 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 8] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank8]:[W510 04:55:56.603014956 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:38552, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14953630efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14957a49c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14957a49b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14953757deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x149676ae7164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14968755919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1496875de100 in /lib64/libc.so.6)
+
+[rank8]:[W510 04:55:56.605250256 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 8] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank8]:[W510 04:55:57.605396319 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:38552, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14953630efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14957a49c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14957a49b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14953757deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x149676ae7164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14968755919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1496875de100 in /lib64/libc.so.6)
+
+[rank8]:[W510 04:55:57.607639279 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 8] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank8]:[W510 04:55:58.607800162 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:38552, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14953630efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14957a49c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14957a49b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14953757deec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x149676ae7164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14968755919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1496875de100 in /lib64/libc.so.6)
+
+[rank8]:[W510 04:55:58.610154832 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 8] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[2026-05-10T04:56:01.587] error: namespace_p_join: open failed for /slurm/22606429/.ns: No such file or directory
+[2026-05-10T04:56:01.587] error: namespace_g_join(JobId=22606429 SLUID=s8FXKR6KX9EK00): No such file or directory
+[2026-05-10T04:56:03.239] error: Failed to send MESSAGE_TASK_EXIT: Connection refused
+[2026-05-10T04:56:03.241] error: namespace_p_join: open failed for /slurm/22606429/.ns: No such file or directory
+[2026-05-10T04:56:03.241] error: namespace_g_join(JobId=22606429 SLUID=s8FXKR6KX9EK00): No such file or directory
+[2026-05-10T04:56:04.880] error: Failed to send MESSAGE_TASK_EXIT: Connection refused
+[2026-05-10T04:56:04.882] error: namespace_p_join: open failed for /slurm/22606429/.ns: No such file or directory
+[2026-05-10T04:56:04.882] error: namespace_g_join(JobId=22606429 SLUID=s8FXKR6KX9EK00): No such file or directory
+[2026-05-10T04:56:05.021] error: namespace_p_join: open failed for /slurm/22606429/.ns: No such file or directory
+[2026-05-10T04:56:05.021] error: namespace_g_join(JobId=22606429 SLUID=s8FXKR6KX9EK00): No such file or directory
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
+  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
+[rank8]:[W510 05:03:21.565247986 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
+  scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+submitit WARNING (2026-05-10 09:26:31,152) - Bypassing signal SIGTERM
+submitit WARNING (2026-05-10 09:26:31,152) - Bypassing signal SIGCONT
+[2026-05-10T09:27:03.751] error: namespace_p_join: open failed for /slurm/22606429/.ns: No such file or directory
+[2026-05-10T09:27:03.752] error: namespace_g_join(JobId=22606429 SLUID=s8FXKR7X2B6T00): No such file or directory
+[2026-05-10T09:27:05.396] error: Failed to send MESSAGE_TASK_EXIT: Connection refused
+[2026-05-10T09:27:05.857] error: namespace_p_join: open failed for /slurm/22606429/.ns: No such file or directory
+[2026-05-10T09:27:05.857] error: namespace_g_join(JobId=22606429 SLUID=s8FXKR7X2B6T00): No such file or directory
+[2026-05-10T09:27:05.997] error: namespace_p_join: open failed for /slurm/22606429/.ns: No such file or directory
+[2026-05-10T09:27:05.997] error: namespace_g_join(JobId=22606429 SLUID=s8FXKR7X2B6T00): No such file or directory
+[2026-05-10T09:27:06.036] error: namespace_p_join: open failed for /slurm/22606429/.ns: No such file or directory
+[2026-05-10T09:27:06.036] error: namespace_g_join(JobId=22606429 SLUID=s8FXKR7X2B6T00): No such file or directory

vjepa/inflated/d003_lora_ema_capilvd1689m/job_22606429/22606429_8_log.out

@@ -0,0 +1,530 @@
+submitit INFO (2026-05-09 16:57:34,481) - Starting with JobEnvironment(job_id=22606429, hostname=gcn76.local.snellius.surf.nl, local_rank=0(4), node=2(4), global_rank=8(16))
+submitit INFO (2026-05-09 16:57:34,481) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/job_22606429/22606429_submitted.pkl
+INFO:root:loaded pretrain params...
+{   'app': 'vjepa',
+    'cpus_per_task': 16,
+    'data': {   'batch_size': 64,
+                'crop_size': 224,
+                'dataset_fpcs': [16, 16],
+                'dataset_type': 'VideoDataset',
+                'datasets': [   '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
+                                '/scratch-shared/dcanez/data/ssv2/train.csv'],
+                'datasets_weights': [0.65, 0.35],
+                'fps': 4,
+                'num_workers': 10,
+                'patch_size': 14,
+                'persistent_workers': True,
+                'pin_mem': True,
+                'stage': [   {   'dest': 'kinetics_240',
+                                 'format': 'targz_parts',
+                                 'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
+                             {   'dest': 'ssv2',
+                                 'format': 'multipart_tar',
+                                 'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
+                'tubelet_size': 1},
+    'data_aug': {   'auto_augment': False,
+                    'motion_shift': False,
+                    'random_resize_aspect_ratio': [0.75, 1.35],
+                    'random_resize_scale': [0.3, 1.0],
+                    'reprob': 0.0},
+    'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m',
+    'loss': {'loss_exp': 1.0},
+    'mask': [   {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 8,
+                    'spatial_scale': [0.15, 0.15],
+                    'temporal_scale': [1.0, 1.0]},
+                {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 2,
+                    'spatial_scale': [0.7, 0.7],
+                    'temporal_scale': [1.0, 1.0]}],
+    'mem_per_gpu': '180G',
+    'meta': {   'dtype': 'bfloat16',
+                'knn_eval_epoch0': False,
+                'knn_eval_freq': 5,
+                'knn_eval_presets': [   {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
+                                                          'eval_videos_per_class': 25,
+                                                          'num_workers': 8,
+                                                          'train_videos_per_class': 100},
+                                            'preset': 'ucf101'},
+                                        {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
+                                                          'eval_videos_per_class': 100,
+                                                          'linear_probe': True,
+                                                          'num_workers': 8,
+                                                          'pool_type': 'temporal_concat',
+                                                          'train_videos_per_class': 500},
+                                            'preset': 'ssv2_coarse10'}],
+                'load_checkpoint': True,
+                'read_checkpoint': None,
+                'save_every_freq': 5,
+                'seed': 239,
+                'use_sdpa': True,
+                'use_wandb': True,
+                'wandb_project': 'vjepa_ablation'},
+    'metrics': {'sigreg': {}, 'std': {}},
+    'model': {   'class_token': False,
+                 'freeze_backbone': True,
+                 'is_causal': False,
+                 'lora_alpha': 16,
+                 'lora_rank': 8,
+                 'lora_targets': ['attn.qkv', 'attn.proj'],
+                 'model_name': 'vit_large_patch14_capi_lvd1689m',
+                 'num_reg_tokens': 16,
+                 'pred_depth': 6,
+                 'pred_embed_dim': 384,
+                 'pred_num_heads': 12,
+                 'rope_type': 'capi2d',
+                 'stem_type': '2d',
+                 'target_kind': 'ema',
+                 'uniform_power': True,
+                 'use_activation_checkpointing': True,
+                 'use_lora': True,
+                 'use_mask_tokens': True,
+                 'use_rope': True,
+                 'use_sdpa': True,
+                 'zero_init_mask_tokens': True},
+    'nodes': 4,
+    'optimization': {   'ema': [0.99925, 0.99925],
+                        'epochs': 100,
+                        'final_lr': 0.0001,
+                        'final_weight_decay': 0.04,
+                        'ipe': 300,
+                        'ipe_scale': 1.0,
+                        'lr': 0.0005,
+                        'start_lr': 0.0001,
+                        'warmup': 10,
+                        'weight_decay': 0.04},
+    'tasks_per_node': 4}
+INFO:root:Running pre-training of app: vjepa
+[INFO    ][2026-05-09 16:57:51][app.vjepa.train     ][main                     ] which_dtype='bfloat16'
+[INFO    ][2026-05-09 16:57:51][app.vjepa.train     ][main                     ] Disabling persistent_workers (incompatible with KNN eval)
+[INFO    ][2026-05-09 16:57:51][app.vjepa.train     ][main                     ] NCCL_SOCKET_IFNAME=eno
+[INFO    ][2026-05-09 16:59:28][app.vjepa.train     ][main                     ] Initialized (rank/world-size) 8/16, tasks_per_node=4
+[INFO    ][2026-05-09 16:59:28][root                ][stage_datasets           ] [local_rank 0/4] Staging kinetics_240 (targz_parts)
+[INFO    ][2026-05-09 16:59:28][root                ][_stage_targz_parts       ] [rank 0] Extracting 26/103 tar.gz parts to /scratch-node/dcanez.22606429/kinetics_240
+[INFO    ][2026-05-09 16:59:42][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 2/26 parts
+[INFO    ][2026-05-09 16:59:56][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 4/26 parts
+[INFO    ][2026-05-09 17:00:10][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 6/26 parts
+[INFO    ][2026-05-09 17:00:24][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 8/26 parts
+[INFO    ][2026-05-09 17:00:38][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 10/26 parts
+[INFO    ][2026-05-09 17:00:53][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 12/26 parts
+[INFO    ][2026-05-09 17:01:07][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 14/26 parts
+[INFO    ][2026-05-09 17:01:22][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 16/26 parts
+[INFO    ][2026-05-09 17:01:36][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 18/26 parts
+[INFO    ][2026-05-09 17:01:50][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 20/26 parts
+[INFO    ][2026-05-09 17:02:04][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 22/26 parts
+[INFO    ][2026-05-09 17:02:18][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 24/26 parts
+[INFO    ][2026-05-09 17:02:32][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 26/26 parts
+[INFO    ][2026-05-09 17:02:32][root                ][stage_datasets           ] [local_rank 0/4] Staging ssv2 (multipart_tar)
+[INFO    ][2026-05-09 17:02:33][root                ][_stage_multipart_tar     ] [rank 0] Extracting multipart tar (2 files) to /scratch-node/dcanez.22606429/ssv2
+[INFO    ][2026-05-09 17:03:30][root                ][stage_datasets           ] Data staging completed in 242.3s (4.0min)
+[INFO    ][2026-05-09 17:03:30][root                ][_rewrite_csv             ] Wrote local CSV: /scratch-node/dcanez.22606429/kinetics_240/train.csv (239789 entries)
+[INFO    ][2026-05-09 17:03:31][root                ][_rewrite_csv             ] Wrote local CSV: /scratch-node/dcanez.22606429/ssv2/train.csv (168913 entries)
+[INFO    ][2026-05-09 17:03:31][app.vjepa.train     ][main                     ] Staged datasets: ['/scratch-node/dcanez.22606429/kinetics_240/train.csv', '/scratch-node/dcanez.22606429/ssv2/train.csv']
+[INFO    ][2026-05-09 17:03:50][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
+[INFO    ][2026-05-09 17:03:53][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA applied: 48 modules replaced (rank=8, alpha=16, dropout=0.0, targets=['attn.qkv', 'attn.proj'], scope=blocks)
+[INFO    ][2026-05-09 17:03:53][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA trainable params: 1,179,648
+[INFO    ][2026-05-09 17:03:53][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-09 17:03:53][root                ][init_video_model         ] VJEPAPredictorMultiSeqWrapper(
+  (backbone): VisionTransformerPredictor(
+    (predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
+    (mask_tokens): ParameterList(
+        (0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+    )
+    (predictor_blocks): ModuleList(
+      (0-5): 6 x Block(
+        (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (attn): RoPEAttention(
+          (qkv): Linear(in_features=384, out_features=1152, bias=True)
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): Linear(in_features=384, out_features=384, bias=True)
+          (proj_drop): Dropout(p=0.0, inplace=False)
+        )
+        (drop_path): Identity()
+        (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (mlp): MLP(
+          (fc1): Linear(in_features=384, out_features=1536, bias=True)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=1536, out_features=384, bias=True)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+    (predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
+  )
+)
+[INFO    ][2026-05-09 17:03:53][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-09 17:03:53][root                ][init_video_model         ] Encoder number of parameters: 1179648
+[INFO    ][2026-05-09 17:03:53][root                ][init_video_model         ] Predictor number of parameters: 11436928
+[INFO    ][2026-05-09 17:03:53][root                ][init_video_model         ] Target encoder number of parameters: 1179648
+[INFO    ][2026-05-09 17:04:04][root                ][make_videodataset        ] VideoDataset dataset created
+[INFO    ][2026-05-09 17:04:04][WeightedSampler     ][__init__                 ] Using DistributedWeightedSampler with rank 8 / 16
+[INFO    ][2026-05-09 17:04:04][root                ][make_videodataset        ] VideoDataset unsupervised data loader created
+[INFO    ][2026-05-09 17:04:04][app.vjepa.train     ][main                     ] iterations per epoch/dataset length: 300/399
+[INFO    ][2026-05-09 17:04:04][app.vjepa.train     ][main                     ] Wrapping models in DDP (rank 8)...
+[INFO    ][2026-05-09 17:05:26][app.vjepa.train     ][main                     ] Initializing loader...
+submitit INFO (2026-05-10 04:55:29,447) - Job has timed out. Ran 718 minutes out of requested 720 minutes.
+[INFO    ][2026-05-10 04:55:29][submitit            ][has_timed_out            ] Job has timed out. Ran 718 minutes out of requested 720 minutes.
+submitit WARNING (2026-05-10 04:55:29,447) - Caught signal SIGUSR2 on gcn76.local.snellius.surf.nl: this job is timed-out.
+[WARNING ][2026-05-10 04:55:29][submitit            ][checkpoint_and_try_requeue] Caught signal SIGUSR2 on gcn76.local.snellius.surf.nl: this job is timed-out.
+submitit INFO (2026-05-10 04:55:29,448) - Not checkpointing nor requeuing since I am a slave (procid=8).
+[INFO    ][2026-05-10 04:55:29][submitit            ][checkpoint_and_try_requeue] Not checkpointing nor requeuing since I am a slave (procid=8).
+submitit WARNING (2026-05-10 04:55:29,448) - Bypassing signal SIGCONT
+[WARNING ][2026-05-10 04:55:29][submitit            ][bypass                   ] Bypassing signal SIGCONT
+submitit WARNING (2026-05-10 04:55:29,547) - Bypassing signal SIGTERM
+[WARNING ][2026-05-10 04:55:29][submitit            ][bypass                   ] Bypassing signal SIGTERM
+submitit WARNING (2026-05-10 04:55:29,620) - Bypassing signal SIGCONT
+[WARNING ][2026-05-10 04:55:29][submitit            ][bypass                   ] Bypassing signal SIGCONT
+submitit INFO (2026-05-10 04:59:12,611) - Starting with JobEnvironment(job_id=22606429, hostname=gcn76.local.snellius.surf.nl, local_rank=0(4), node=2(4), global_rank=8(16))
+submitit INFO (2026-05-10 04:59:12,611) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/job_22606429/22606429_submitted.pkl
+INFO:root:loaded pretrain params...
+{   'app': 'vjepa',
+    'cpus_per_task': 16,
+    'data': {   'batch_size': 64,
+                'crop_size': 224,
+                'dataset_fpcs': [16, 16],
+                'dataset_type': 'VideoDataset',
+                'datasets': [   '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
+                                '/scratch-shared/dcanez/data/ssv2/train.csv'],
+                'datasets_weights': [0.65, 0.35],
+                'fps': 4,
+                'num_workers': 10,
+                'patch_size': 14,
+                'persistent_workers': True,
+                'pin_mem': True,
+                'stage': [   {   'dest': 'kinetics_240',
+                                 'format': 'targz_parts',
+                                 'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
+                             {   'dest': 'ssv2',
+                                 'format': 'multipart_tar',
+                                 'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
+                'tubelet_size': 1},
+    'data_aug': {   'auto_augment': False,
+                    'motion_shift': False,
+                    'random_resize_aspect_ratio': [0.75, 1.35],
+                    'random_resize_scale': [0.3, 1.0],
+                    'reprob': 0.0},
+    'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m',
+    'loss': {'loss_exp': 1.0},
+    'mask': [   {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 8,
+                    'spatial_scale': [0.15, 0.15],
+                    'temporal_scale': [1.0, 1.0]},
+                {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 2,
+                    'spatial_scale': [0.7, 0.7],
+                    'temporal_scale': [1.0, 1.0]}],
+    'mem_per_gpu': '180G',
+    'meta': {   'dtype': 'bfloat16',
+                'knn_eval_epoch0': False,
+                'knn_eval_freq': 5,
+                'knn_eval_presets': [   {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
+                                                          'eval_videos_per_class': 25,
+                                                          'num_workers': 8,
+                                                          'train_videos_per_class': 100},
+                                            'preset': 'ucf101'},
+                                        {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
+                                                          'eval_videos_per_class': 100,
+                                                          'linear_probe': True,
+                                                          'num_workers': 8,
+                                                          'pool_type': 'temporal_concat',
+                                                          'train_videos_per_class': 500},
+                                            'preset': 'ssv2_coarse10'}],
+                'load_checkpoint': True,
+                'read_checkpoint': None,
+                'save_every_freq': 5,
+                'seed': 239,
+                'use_sdpa': True,
+                'use_wandb': True,
+                'wandb_project': 'vjepa_ablation'},
+    'metrics': {'sigreg': {}, 'std': {}},
+    'model': {   'class_token': False,
+                 'freeze_backbone': True,
+                 'is_causal': False,
+                 'lora_alpha': 16,
+                 'lora_rank': 8,
+                 'lora_targets': ['attn.qkv', 'attn.proj'],
+                 'model_name': 'vit_large_patch14_capi_lvd1689m',
+                 'num_reg_tokens': 16,
+                 'pred_depth': 6,
+                 'pred_embed_dim': 384,
+                 'pred_num_heads': 12,
+                 'rope_type': 'capi2d',
+                 'stem_type': '2d',
+                 'target_kind': 'ema',
+                 'uniform_power': True,
+                 'use_activation_checkpointing': True,
+                 'use_lora': True,
+                 'use_mask_tokens': True,
+                 'use_rope': True,
+                 'use_sdpa': True,
+                 'zero_init_mask_tokens': True},
+    'nodes': 4,
+    'optimization': {   'ema': [0.99925, 0.99925],
+                        'epochs': 100,
+                        'final_lr': 0.0001,
+                        'final_weight_decay': 0.04,
+                        'ipe': 300,
+                        'ipe_scale': 1.0,
+                        'lr': 0.0005,
+                        'start_lr': 0.0001,
+                        'warmup': 10,
+                        'weight_decay': 0.04},
+    'tasks_per_node': 4}
+INFO:root:Running pre-training of app: vjepa
+[INFO    ][2026-05-10 04:59:17][app.vjepa.train     ][main                     ] which_dtype='bfloat16'
+[INFO    ][2026-05-10 04:59:17][app.vjepa.train     ][main                     ] Disabling persistent_workers (incompatible with KNN eval)
+[INFO    ][2026-05-10 04:59:17][app.vjepa.train     ][main                     ] NCCL_SOCKET_IFNAME=eno
+[INFO    ][2026-05-10 04:59:21][app.vjepa.train     ][main                     ] Initialized (rank/world-size) 8/16, tasks_per_node=4
+[INFO    ][2026-05-10 04:59:21][root                ][stage_datasets           ] [local_rank 0/4] Staging kinetics_240 (targz_parts)
+[INFO    ][2026-05-10 04:59:21][root                ][_stage_targz_parts       ] [rank 0] Extracting 26/103 tar.gz parts to /scratch-node/dcanez.22606429/kinetics_240
+[INFO    ][2026-05-10 04:59:36][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 2/26 parts
+[INFO    ][2026-05-10 04:59:50][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 4/26 parts
+[INFO    ][2026-05-10 05:00:04][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 6/26 parts
+[INFO    ][2026-05-10 05:00:18][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 8/26 parts
+[INFO    ][2026-05-10 05:00:32][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 10/26 parts
+[INFO    ][2026-05-10 05:00:46][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 12/26 parts
+[INFO    ][2026-05-10 05:01:00][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 14/26 parts
+[INFO    ][2026-05-10 05:01:15][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 16/26 parts
+[INFO    ][2026-05-10 05:01:29][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 18/26 parts
+[INFO    ][2026-05-10 05:01:43][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 20/26 parts
+[INFO    ][2026-05-10 05:01:57][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 22/26 parts
+[INFO    ][2026-05-10 05:02:12][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 24/26 parts
+[INFO    ][2026-05-10 05:02:26][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 26/26 parts
+[INFO    ][2026-05-10 05:02:26][root                ][stage_datasets           ] [local_rank 0/4] Staging ssv2 (multipart_tar)
+[INFO    ][2026-05-10 05:02:26][root                ][_stage_multipart_tar     ] [rank 0] Extracting multipart tar (2 files) to /scratch-node/dcanez.22606429/ssv2
+[INFO    ][2026-05-10 05:04:32][root                ][stage_datasets           ] Data staging completed in 310.9s (5.2min)
+[INFO    ][2026-05-10 05:04:32][root                ][_rewrite_csv             ] Wrote local CSV: /scratch-node/dcanez.22606429/kinetics_240/train.csv (239789 entries)
+[INFO    ][2026-05-10 05:04:33][root                ][_rewrite_csv             ] Wrote local CSV: /scratch-node/dcanez.22606429/ssv2/train.csv (168913 entries)
+[INFO    ][2026-05-10 05:04:33][app.vjepa.train     ][main                     ] Staged datasets: ['/scratch-node/dcanez.22606429/kinetics_240/train.csv', '/scratch-node/dcanez.22606429/ssv2/train.csv']
+[INFO    ][2026-05-10 05:04:47][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
+[INFO    ][2026-05-10 05:04:51][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA applied: 48 modules replaced (rank=8, alpha=16, dropout=0.0, targets=['attn.qkv', 'attn.proj'], scope=blocks)
+[INFO    ][2026-05-10 05:04:51][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA trainable params: 1,179,648
+[INFO    ][2026-05-10 05:04:52][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-10 05:04:52][root                ][init_video_model         ] VJEPAPredictorMultiSeqWrapper(
+  (backbone): VisionTransformerPredictor(
+    (predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
+    (mask_tokens): ParameterList(
+        (0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+    )
+    (predictor_blocks): ModuleList(
+      (0-5): 6 x Block(
+        (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (attn): RoPEAttention(
+          (qkv): Linear(in_features=384, out_features=1152, bias=True)
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): Linear(in_features=384, out_features=384, bias=True)
+          (proj_drop): Dropout(p=0.0, inplace=False)
+        )
+        (drop_path): Identity()
+        (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (mlp): MLP(
+          (fc1): Linear(in_features=384, out_features=1536, bias=True)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=1536, out_features=384, bias=True)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+    (predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
+  )
+)
+[INFO    ][2026-05-10 05:04:52][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-10 05:04:52][root                ][init_video_model         ] Encoder number of parameters: 1179648
+[INFO    ][2026-05-10 05:04:52][root                ][init_video_model         ] Predictor number of parameters: 11436928
+[INFO    ][2026-05-10 05:04:52][root                ][init_video_model         ] Target encoder number of parameters: 1179648
+[INFO    ][2026-05-10 05:04:53][root                ][make_videodataset        ] VideoDataset dataset created
+[INFO    ][2026-05-10 05:04:53][WeightedSampler     ][__init__                 ] Using DistributedWeightedSampler with rank 8 / 16
+[INFO    ][2026-05-10 05:04:53][root                ][make_videodataset        ] VideoDataset unsupervised data loader created
+[INFO    ][2026-05-10 05:04:53][app.vjepa.train     ][main                     ] iterations per epoch/dataset length: 300/399
+[INFO    ][2026-05-10 05:04:53][app.vjepa.train     ][main                     ] Wrapping models in DDP (rank 8)...
+[INFO    ][2026-05-10 05:06:08][root                ][load_checkpoint          ] Loading checkpoint from /scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/latest.pt
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] loaded pretrained encoder from epoch 33 with msg: <All keys matched successfully>
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] loaded pretrained predictor from epoch 33 with msg: <All keys matched successfully>
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] loaded pretrained target encoder from epoch 33 with msg: <All keys matched successfully>
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] loaded optimizers from epoch 33
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] read-path: /scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/latest.pt
+[INFO    ][2026-05-10 05:06:09][app.vjepa.train     ][main                     ] Initializing loader...
+submitit WARNING (2026-05-10 09:26:31,152) - Bypassing signal SIGTERM
+[WARNING ][2026-05-10 09:26:31][submitit            ][bypass                   ] Bypassing signal SIGTERM
+submitit WARNING (2026-05-10 09:26:31,152) - Bypassing signal SIGCONT
+[WARNING ][2026-05-10 09:26:31][submitit            ][bypass                   ] Bypassing signal SIGCONT

vjepa/inflated/d003_lora_ema_capilvd1689m/job_22606429/22606429_9_log.err

@@ -0,0 +1,295 @@
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
+  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
+[rank9]:[W509 17:02:20.584172765 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
+  scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+submitit WARNING (2026-05-10 04:55:29,391) - Caught signal SIGUSR2 on gcn76.local.snellius.surf.nl: this job is timed-out.
+submitit WARNING (2026-05-10 04:55:29,391) - Bypassing signal SIGCONT
+submitit WARNING (2026-05-10 04:55:29,547) - Bypassing signal SIGTERM
+submitit WARNING (2026-05-10 04:55:29,547) - Bypassing signal SIGCONT
+[rank9]:[W510 04:55:34.969330636 TCPStore.cpp:125] [c10d] recvValue failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:37972, remote=[gcn74.local.snellius.surf.nl]:37129): Failed to recv, got 0 bytes. Connection was likely closed. Did the remote server shutdown or crash?
+Exception raised from recvBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:682 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x148711b28fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a3325d (0x148755cb725d in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x273 (0x148755cb51f3 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x148712d97eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x148852301164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x148862d7319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x148862df8100 in /lib64/libc.so.6)
+
+[rank9]:[W510 04:55:34.973070510 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 9] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Failed to recv, got 0 bytes. Connection was likely closed. Did the remote server shutdown or crash?
+[rank9]:[W510 04:55:35.973233462 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:37972, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x148711b28fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x148755cb66d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x148755cb51cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x148712d97eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x148852301164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x148862d7319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x148862df8100 in /lib64/libc.so.6)
+
+[rank9]:[W510 04:55:35.976656168 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 9] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank9]:[W510 04:55:36.976801180 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:37972, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x148711b28fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x148755cb66d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x148755cb51cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x148712d97eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x148852301164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x148862d7319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x148862df8100 in /lib64/libc.so.6)
+
+[rank9]:[W510 04:55:36.979071111 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 9] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank9]:[W510 04:55:37.979232693 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:37972, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x148711b28fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x148755cb66d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x148755cb51cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x148712d97eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x148852301164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x148862d7319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x148862df8100 in /lib64/libc.so.6)
+
+[rank9]:[W510 04:55:37.981565593 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 9] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank9]:[W510 04:55:38.981707656 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:37972, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x148711b28fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x148755cb66d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x148755cb51cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x148712d97eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x148852301164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x148862d7319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x148862df8100 in /lib64/libc.so.6)
+
+[rank9]:[W510 04:55:38.984105786 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 9] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank9]:[W510 04:55:39.984253718 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:37972, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x148711b28fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x148755cb66d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x148755cb51cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x148712d97eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x148852301164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x148862d7319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x148862df8100 in /lib64/libc.so.6)
+
+[rank9]:[W510 04:55:39.986471249 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 9] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank9]:[W510 04:55:40.986614191 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:37972, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x148711b28fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x148755cb66d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x148755cb51cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x148712d97eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x148852301164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x148862d7319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x148862df8100 in /lib64/libc.so.6)
+
+[rank9]:[W510 04:55:40.988969611 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 9] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank9]:[W510 04:55:41.989130454 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:37972, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x148711b28fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x148755cb66d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x148755cb51cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x148712d97eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x148852301164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x148862d7319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x148862df8100 in /lib64/libc.so.6)
+
+[rank9]:[W510 04:55:41.991462654 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 9] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank9]:[W510 04:55:42.991615977 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:37972, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x148711b28fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x148755cb66d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x148755cb51cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x148712d97eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x148852301164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x148862d7319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x148862df8100 in /lib64/libc.so.6)
+
+[rank9]:[W510 04:55:42.993879907 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 9] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank9]:[W510 04:55:43.994064649 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:37972, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x148711b28fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x148755cb66d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x148755cb51cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x148712d97eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x148852301164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x148862d7319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x148862df8100 in /lib64/libc.so.6)
+
+[rank9]:[W510 04:55:43.996358380 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 9] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank9]:[W510 04:55:44.996511182 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:37972, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x148711b28fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x148755cb66d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x148755cb51cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x148712d97eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x148852301164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x148862d7319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x148862df8100 in /lib64/libc.so.6)
+
+[rank9]:[W510 04:55:44.998874632 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 9] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank9]:[W510 04:55:45.999045185 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:37972, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x148711b28fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x148755cb66d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x148755cb51cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x148712d97eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x148852301164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x148862d7319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x148862df8100 in /lib64/libc.so.6)
+
+[rank9]:[W510 04:55:45.001380155 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 9] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank9]:[W510 04:55:46.001525157 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:37972, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x148711b28fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x148755cb66d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x148755cb51cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x148712d97eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x148852301164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x148862d7319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x148862df8100 in /lib64/libc.so.6)
+
+[rank9]:[W510 04:55:46.003848987 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 9] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank9]:[W510 04:55:47.003988370 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:37972, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x148711b28fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x148755cb66d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x148755cb51cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x148712d97eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x148852301164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x148862d7319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x148862df8100 in /lib64/libc.so.6)
+
+[rank9]:[W510 04:55:47.006464820 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 9] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank9]:[W510 04:55:48.006601732 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:37972, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x148711b28fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x148755cb66d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x148755cb51cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x148712d97eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x148852301164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x148862d7319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x148862df8100 in /lib64/libc.so.6)
+
+[rank9]:[W510 04:55:48.009010512 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 9] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank9]:[W510 04:55:49.009163655 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:37972, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x148711b28fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x148755cb66d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x148755cb51cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x148712d97eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x148852301164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x148862d7319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x148862df8100 in /lib64/libc.so.6)
+
+[rank9]:[W510 04:55:49.011377475 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 9] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank9]:[W510 04:55:50.011541838 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:37972, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x148711b28fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x148755cb66d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x148755cb51cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x148712d97eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x148852301164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x148862d7319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x148862df8100 in /lib64/libc.so.6)
+
+[rank9]:[W510 04:55:50.014022677 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 9] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank9]:[W510 04:55:51.014161580 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:37972, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x148711b28fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x148755cb66d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x148755cb51cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x148712d97eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x148852301164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x148862d7319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x148862df8100 in /lib64/libc.so.6)
+
+[rank9]:[W510 04:55:51.016431400 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 9] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank9]:[W510 04:55:52.016587733 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:37972, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x148711b28fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x148755cb66d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x148755cb51cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x148712d97eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x148852301164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x148862d7319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x148862df8100 in /lib64/libc.so.6)
+
+[rank9]:[W510 04:55:52.018958332 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 9] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank9]:[W510 04:55:53.019102525 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:37972, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x148711b28fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x148755cb66d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x148755cb51cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x148712d97eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x148852301164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x148862d7319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x148862df8100 in /lib64/libc.so.6)
+
+[rank9]:[W510 04:55:53.021450415 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 9] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank9]:[W510 04:55:54.021593698 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:37972, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x148711b28fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x148755cb66d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x148755cb51cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x148712d97eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x148852301164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x148862d7319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x148862df8100 in /lib64/libc.so.6)
+
+[rank9]:[W510 04:55:54.023853568 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 9] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank9]:[W510 04:55:55.023980671 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:37972, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x148711b28fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x148755cb66d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x148755cb51cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x148712d97eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x148852301164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x148862d7319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x148862df8100 in /lib64/libc.so.6)
+
+[rank9]:[W510 04:55:55.026125862 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 9] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank9]:[W510 04:55:56.026261284 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:37972, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x148711b28fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x148755cb66d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x148755cb51cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x148712d97eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x148852301164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x148862d7319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x148862df8100 in /lib64/libc.so.6)
+
+[rank9]:[W510 04:55:56.028668174 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 9] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank9]:[W510 04:55:57.028815727 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:37972, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x148711b28fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x148755cb66d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x148755cb51cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x148712d97eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x148852301164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x148862d7319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x148862df8100 in /lib64/libc.so.6)
+
+[rank9]:[W510 04:55:57.031160617 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 9] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank9]:[W510 04:55:58.031306689 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=4, addr=[gcn76.local.snellius.surf.nl]:37972, remote=[gcn74.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x148711b28fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x148755cb66d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x148755cb51cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x148712d97eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x148852301164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x148862d7319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x148862df8100 in /lib64/libc.so.6)
+
+[rank9]:[W510 04:55:58.033582660 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 9] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
+  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
+[rank9]:[W510 05:02:23.167011408 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
+  scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+submitit WARNING (2026-05-10 09:26:31,151) - Bypassing signal SIGTERM
+submitit WARNING (2026-05-10 09:26:31,151) - Bypassing signal SIGCONT

vjepa/inflated/d003_lora_ema_capilvd1689m/job_22606429/22606429_9_log.out

@@ -0,0 +1,522 @@
+submitit INFO (2026-05-09 16:57:34,481) - Starting with JobEnvironment(job_id=22606429, hostname=gcn76.local.snellius.surf.nl, local_rank=1(4), node=2(4), global_rank=9(16))
+submitit INFO (2026-05-09 16:57:34,481) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/job_22606429/22606429_submitted.pkl
+INFO:root:loaded pretrain params...
+{   'app': 'vjepa',
+    'cpus_per_task': 16,
+    'data': {   'batch_size': 64,
+                'crop_size': 224,
+                'dataset_fpcs': [16, 16],
+                'dataset_type': 'VideoDataset',
+                'datasets': [   '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
+                                '/scratch-shared/dcanez/data/ssv2/train.csv'],
+                'datasets_weights': [0.65, 0.35],
+                'fps': 4,
+                'num_workers': 10,
+                'patch_size': 14,
+                'persistent_workers': True,
+                'pin_mem': True,
+                'stage': [   {   'dest': 'kinetics_240',
+                                 'format': 'targz_parts',
+                                 'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
+                             {   'dest': 'ssv2',
+                                 'format': 'multipart_tar',
+                                 'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
+                'tubelet_size': 1},
+    'data_aug': {   'auto_augment': False,
+                    'motion_shift': False,
+                    'random_resize_aspect_ratio': [0.75, 1.35],
+                    'random_resize_scale': [0.3, 1.0],
+                    'reprob': 0.0},
+    'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m',
+    'loss': {'loss_exp': 1.0},
+    'mask': [   {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 8,
+                    'spatial_scale': [0.15, 0.15],
+                    'temporal_scale': [1.0, 1.0]},
+                {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 2,
+                    'spatial_scale': [0.7, 0.7],
+                    'temporal_scale': [1.0, 1.0]}],
+    'mem_per_gpu': '180G',
+    'meta': {   'dtype': 'bfloat16',
+                'knn_eval_epoch0': False,
+                'knn_eval_freq': 5,
+                'knn_eval_presets': [   {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
+                                                          'eval_videos_per_class': 25,
+                                                          'num_workers': 8,
+                                                          'train_videos_per_class': 100},
+                                            'preset': 'ucf101'},
+                                        {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
+                                                          'eval_videos_per_class': 100,
+                                                          'linear_probe': True,
+                                                          'num_workers': 8,
+                                                          'pool_type': 'temporal_concat',
+                                                          'train_videos_per_class': 500},
+                                            'preset': 'ssv2_coarse10'}],
+                'load_checkpoint': True,
+                'read_checkpoint': None,
+                'save_every_freq': 5,
+                'seed': 239,
+                'use_sdpa': True,
+                'use_wandb': True,
+                'wandb_project': 'vjepa_ablation'},
+    'metrics': {'sigreg': {}, 'std': {}},
+    'model': {   'class_token': False,
+                 'freeze_backbone': True,
+                 'is_causal': False,
+                 'lora_alpha': 16,
+                 'lora_rank': 8,
+                 'lora_targets': ['attn.qkv', 'attn.proj'],
+                 'model_name': 'vit_large_patch14_capi_lvd1689m',
+                 'num_reg_tokens': 16,
+                 'pred_depth': 6,
+                 'pred_embed_dim': 384,
+                 'pred_num_heads': 12,
+                 'rope_type': 'capi2d',
+                 'stem_type': '2d',
+                 'target_kind': 'ema',
+                 'uniform_power': True,
+                 'use_activation_checkpointing': True,
+                 'use_lora': True,
+                 'use_mask_tokens': True,
+                 'use_rope': True,
+                 'use_sdpa': True,
+                 'zero_init_mask_tokens': True},
+    'nodes': 4,
+    'optimization': {   'ema': [0.99925, 0.99925],
+                        'epochs': 100,
+                        'final_lr': 0.0001,
+                        'final_weight_decay': 0.04,
+                        'ipe': 300,
+                        'ipe_scale': 1.0,
+                        'lr': 0.0005,
+                        'start_lr': 0.0001,
+                        'warmup': 10,
+                        'weight_decay': 0.04},
+    'tasks_per_node': 4}
+INFO:root:Running pre-training of app: vjepa
+[INFO    ][2026-05-09 16:57:51][app.vjepa.train     ][main                     ] which_dtype='bfloat16'
+[INFO    ][2026-05-09 16:57:51][app.vjepa.train     ][main                     ] Disabling persistent_workers (incompatible with KNN eval)
+[INFO    ][2026-05-09 16:57:51][app.vjepa.train     ][main                     ] NCCL_SOCKET_IFNAME=eno
+[INFO    ][2026-05-09 16:59:18][app.vjepa.train     ][main                     ] Initialized (rank/world-size) 9/16, tasks_per_node=4
+[INFO    ][2026-05-09 16:59:18][root                ][stage_datasets           ] [local_rank 1/4] Staging kinetics_240 (targz_parts)
+[INFO    ][2026-05-09 16:59:18][root                ][_stage_targz_parts       ] [rank 1] Extracting 26/103 tar.gz parts to /scratch-node/dcanez.22606429/kinetics_240
+[INFO    ][2026-05-09 16:59:30][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 2/26 parts
+[INFO    ][2026-05-09 16:59:44][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 4/26 parts
+[INFO    ][2026-05-09 16:59:58][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 6/26 parts
+[INFO    ][2026-05-09 17:00:12][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 8/26 parts
+[INFO    ][2026-05-09 17:00:26][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 10/26 parts
+[INFO    ][2026-05-09 17:00:41][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 12/26 parts
+[INFO    ][2026-05-09 17:00:54][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 14/26 parts
+[INFO    ][2026-05-09 17:01:08][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 16/26 parts
+[INFO    ][2026-05-09 17:01:22][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 18/26 parts
+[INFO    ][2026-05-09 17:01:37][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 20/26 parts
+[INFO    ][2026-05-09 17:01:51][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 22/26 parts
+[INFO    ][2026-05-09 17:02:05][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 24/26 parts
+[INFO    ][2026-05-09 17:02:20][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 26/26 parts
+[INFO    ][2026-05-09 17:02:20][root                ][stage_datasets           ] [local_rank 1/4] Staging ssv2 (multipart_tar)
+[INFO    ][2026-05-09 17:03:31][app.vjepa.train     ][main                     ] Staged datasets: ['/scratch-node/dcanez.22606429/kinetics_240/train.csv', '/scratch-node/dcanez.22606429/ssv2/train.csv']
+[INFO    ][2026-05-09 17:03:49][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
+[INFO    ][2026-05-09 17:03:53][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA applied: 48 modules replaced (rank=8, alpha=16, dropout=0.0, targets=['attn.qkv', 'attn.proj'], scope=blocks)
+[INFO    ][2026-05-09 17:03:53][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA trainable params: 1,179,648
+[INFO    ][2026-05-09 17:03:53][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-09 17:03:53][root                ][init_video_model         ] VJEPAPredictorMultiSeqWrapper(
+  (backbone): VisionTransformerPredictor(
+    (predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
+    (mask_tokens): ParameterList(
+        (0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+    )
+    (predictor_blocks): ModuleList(
+      (0-5): 6 x Block(
+        (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (attn): RoPEAttention(
+          (qkv): Linear(in_features=384, out_features=1152, bias=True)
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): Linear(in_features=384, out_features=384, bias=True)
+          (proj_drop): Dropout(p=0.0, inplace=False)
+        )
+        (drop_path): Identity()
+        (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (mlp): MLP(
+          (fc1): Linear(in_features=384, out_features=1536, bias=True)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=1536, out_features=384, bias=True)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+    (predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
+  )
+)
+[INFO    ][2026-05-09 17:03:53][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-09 17:03:53][root                ][init_video_model         ] Encoder number of parameters: 1179648
+[INFO    ][2026-05-09 17:03:53][root                ][init_video_model         ] Predictor number of parameters: 11436928
+[INFO    ][2026-05-09 17:03:53][root                ][init_video_model         ] Target encoder number of parameters: 1179648
+[INFO    ][2026-05-09 17:04:03][root                ][make_videodataset        ] VideoDataset dataset created
+[INFO    ][2026-05-09 17:04:03][WeightedSampler     ][__init__                 ] Using DistributedWeightedSampler with rank 9 / 16
+[INFO    ][2026-05-09 17:04:03][root                ][make_videodataset        ] VideoDataset unsupervised data loader created
+[INFO    ][2026-05-09 17:04:03][app.vjepa.train     ][main                     ] iterations per epoch/dataset length: 300/399
+[INFO    ][2026-05-09 17:04:03][app.vjepa.train     ][main                     ] Wrapping models in DDP (rank 9)...
+[INFO    ][2026-05-09 17:05:26][app.vjepa.train     ][main                     ] Initializing loader...
+submitit INFO (2026-05-10 04:55:29,390) - Job has timed out. Ran 718 minutes out of requested 720 minutes.
+[INFO    ][2026-05-10 04:55:29][submitit            ][has_timed_out            ] Job has timed out. Ran 718 minutes out of requested 720 minutes.
+submitit WARNING (2026-05-10 04:55:29,391) - Caught signal SIGUSR2 on gcn76.local.snellius.surf.nl: this job is timed-out.
+[WARNING ][2026-05-10 04:55:29][submitit            ][checkpoint_and_try_requeue] Caught signal SIGUSR2 on gcn76.local.snellius.surf.nl: this job is timed-out.
+submitit INFO (2026-05-10 04:55:29,391) - Not checkpointing nor requeuing since I am a slave (procid=9).
+[INFO    ][2026-05-10 04:55:29][submitit            ][checkpoint_and_try_requeue] Not checkpointing nor requeuing since I am a slave (procid=9).
+submitit WARNING (2026-05-10 04:55:29,391) - Bypassing signal SIGCONT
+[WARNING ][2026-05-10 04:55:29][submitit            ][bypass                   ] Bypassing signal SIGCONT
+submitit WARNING (2026-05-10 04:55:29,547) - Bypassing signal SIGTERM
+[WARNING ][2026-05-10 04:55:29][submitit            ][bypass                   ] Bypassing signal SIGTERM
+submitit WARNING (2026-05-10 04:55:29,547) - Bypassing signal SIGCONT
+[WARNING ][2026-05-10 04:55:29][submitit            ][bypass                   ] Bypassing signal SIGCONT
+submitit INFO (2026-05-10 04:59:12,611) - Starting with JobEnvironment(job_id=22606429, hostname=gcn76.local.snellius.surf.nl, local_rank=1(4), node=2(4), global_rank=9(16))
+submitit INFO (2026-05-10 04:59:12,611) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/job_22606429/22606429_submitted.pkl
+INFO:root:loaded pretrain params...
+{   'app': 'vjepa',
+    'cpus_per_task': 16,
+    'data': {   'batch_size': 64,
+                'crop_size': 224,
+                'dataset_fpcs': [16, 16],
+                'dataset_type': 'VideoDataset',
+                'datasets': [   '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
+                                '/scratch-shared/dcanez/data/ssv2/train.csv'],
+                'datasets_weights': [0.65, 0.35],
+                'fps': 4,
+                'num_workers': 10,
+                'patch_size': 14,
+                'persistent_workers': True,
+                'pin_mem': True,
+                'stage': [   {   'dest': 'kinetics_240',
+                                 'format': 'targz_parts',
+                                 'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
+                             {   'dest': 'ssv2',
+                                 'format': 'multipart_tar',
+                                 'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
+                'tubelet_size': 1},
+    'data_aug': {   'auto_augment': False,
+                    'motion_shift': False,
+                    'random_resize_aspect_ratio': [0.75, 1.35],
+                    'random_resize_scale': [0.3, 1.0],
+                    'reprob': 0.0},
+    'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m',
+    'loss': {'loss_exp': 1.0},
+    'mask': [   {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 8,
+                    'spatial_scale': [0.15, 0.15],
+                    'temporal_scale': [1.0, 1.0]},
+                {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 2,
+                    'spatial_scale': [0.7, 0.7],
+                    'temporal_scale': [1.0, 1.0]}],
+    'mem_per_gpu': '180G',
+    'meta': {   'dtype': 'bfloat16',
+                'knn_eval_epoch0': False,
+                'knn_eval_freq': 5,
+                'knn_eval_presets': [   {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
+                                                          'eval_videos_per_class': 25,
+                                                          'num_workers': 8,
+                                                          'train_videos_per_class': 100},
+                                            'preset': 'ucf101'},
+                                        {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
+                                                          'eval_videos_per_class': 100,
+                                                          'linear_probe': True,
+                                                          'num_workers': 8,
+                                                          'pool_type': 'temporal_concat',
+                                                          'train_videos_per_class': 500},
+                                            'preset': 'ssv2_coarse10'}],
+                'load_checkpoint': True,
+                'read_checkpoint': None,
+                'save_every_freq': 5,
+                'seed': 239,
+                'use_sdpa': True,
+                'use_wandb': True,
+                'wandb_project': 'vjepa_ablation'},
+    'metrics': {'sigreg': {}, 'std': {}},
+    'model': {   'class_token': False,
+                 'freeze_backbone': True,
+                 'is_causal': False,
+                 'lora_alpha': 16,
+                 'lora_rank': 8,
+                 'lora_targets': ['attn.qkv', 'attn.proj'],
+                 'model_name': 'vit_large_patch14_capi_lvd1689m',
+                 'num_reg_tokens': 16,
+                 'pred_depth': 6,
+                 'pred_embed_dim': 384,
+                 'pred_num_heads': 12,
+                 'rope_type': 'capi2d',
+                 'stem_type': '2d',
+                 'target_kind': 'ema',
+                 'uniform_power': True,
+                 'use_activation_checkpointing': True,
+                 'use_lora': True,
+                 'use_mask_tokens': True,
+                 'use_rope': True,
+                 'use_sdpa': True,
+                 'zero_init_mask_tokens': True},
+    'nodes': 4,
+    'optimization': {   'ema': [0.99925, 0.99925],
+                        'epochs': 100,
+                        'final_lr': 0.0001,
+                        'final_weight_decay': 0.04,
+                        'ipe': 300,
+                        'ipe_scale': 1.0,
+                        'lr': 0.0005,
+                        'start_lr': 0.0001,
+                        'warmup': 10,
+                        'weight_decay': 0.04},
+    'tasks_per_node': 4}
+INFO:root:Running pre-training of app: vjepa
+[INFO    ][2026-05-10 04:59:17][app.vjepa.train     ][main                     ] which_dtype='bfloat16'
+[INFO    ][2026-05-10 04:59:17][app.vjepa.train     ][main                     ] Disabling persistent_workers (incompatible with KNN eval)
+[INFO    ][2026-05-10 04:59:17][app.vjepa.train     ][main                     ] NCCL_SOCKET_IFNAME=eno
+[INFO    ][2026-05-10 04:59:21][app.vjepa.train     ][main                     ] Initialized (rank/world-size) 9/16, tasks_per_node=4
+[INFO    ][2026-05-10 04:59:21][root                ][stage_datasets           ] [local_rank 1/4] Staging kinetics_240 (targz_parts)
+[INFO    ][2026-05-10 04:59:21][root                ][_stage_targz_parts       ] [rank 1] Extracting 26/103 tar.gz parts to /scratch-node/dcanez.22606429/kinetics_240
+[INFO    ][2026-05-10 04:59:33][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 2/26 parts
+[INFO    ][2026-05-10 04:59:47][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 4/26 parts
+[INFO    ][2026-05-10 05:00:01][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 6/26 parts
+[INFO    ][2026-05-10 05:00:15][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 8/26 parts
+[INFO    ][2026-05-10 05:00:30][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 10/26 parts
+[INFO    ][2026-05-10 05:00:44][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 12/26 parts
+[INFO    ][2026-05-10 05:00:58][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 14/26 parts
+[INFO    ][2026-05-10 05:01:11][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 16/26 parts
+[INFO    ][2026-05-10 05:01:26][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 18/26 parts
+[INFO    ][2026-05-10 05:01:40][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 20/26 parts
+[INFO    ][2026-05-10 05:01:55][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 22/26 parts
+[INFO    ][2026-05-10 05:02:09][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 24/26 parts
+[INFO    ][2026-05-10 05:02:23][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 26/26 parts
+[INFO    ][2026-05-10 05:02:23][root                ][stage_datasets           ] [local_rank 1/4] Staging ssv2 (multipart_tar)
+[INFO    ][2026-05-10 05:04:33][app.vjepa.train     ][main                     ] Staged datasets: ['/scratch-node/dcanez.22606429/kinetics_240/train.csv', '/scratch-node/dcanez.22606429/ssv2/train.csv']
+[INFO    ][2026-05-10 05:04:47][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
+[INFO    ][2026-05-10 05:04:51][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA applied: 48 modules replaced (rank=8, alpha=16, dropout=0.0, targets=['attn.qkv', 'attn.proj'], scope=blocks)
+[INFO    ][2026-05-10 05:04:51][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA trainable params: 1,179,648
+[INFO    ][2026-05-10 05:04:52][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-10 05:04:52][root                ][init_video_model         ] VJEPAPredictorMultiSeqWrapper(
+  (backbone): VisionTransformerPredictor(
+    (predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
+    (mask_tokens): ParameterList(
+        (0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+    )
+    (predictor_blocks): ModuleList(
+      (0-5): 6 x Block(
+        (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (attn): RoPEAttention(
+          (qkv): Linear(in_features=384, out_features=1152, bias=True)
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): Linear(in_features=384, out_features=384, bias=True)
+          (proj_drop): Dropout(p=0.0, inplace=False)
+        )
+        (drop_path): Identity()
+        (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (mlp): MLP(
+          (fc1): Linear(in_features=384, out_features=1536, bias=True)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=1536, out_features=384, bias=True)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+    (predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
+  )
+)
+[INFO    ][2026-05-10 05:04:52][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-10 05:04:52][root                ][init_video_model         ] Encoder number of parameters: 1179648
+[INFO    ][2026-05-10 05:04:52][root                ][init_video_model         ] Predictor number of parameters: 11436928
+[INFO    ][2026-05-10 05:04:52][root                ][init_video_model         ] Target encoder number of parameters: 1179648
+[INFO    ][2026-05-10 05:04:53][root                ][make_videodataset        ] VideoDataset dataset created
+[INFO    ][2026-05-10 05:04:53][WeightedSampler     ][__init__                 ] Using DistributedWeightedSampler with rank 9 / 16
+[INFO    ][2026-05-10 05:04:53][root                ][make_videodataset        ] VideoDataset unsupervised data loader created
+[INFO    ][2026-05-10 05:04:53][app.vjepa.train     ][main                     ] iterations per epoch/dataset length: 300/399
+[INFO    ][2026-05-10 05:04:53][app.vjepa.train     ][main                     ] Wrapping models in DDP (rank 9)...
+[INFO    ][2026-05-10 05:06:08][root                ][load_checkpoint          ] Loading checkpoint from /scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/latest.pt
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] loaded pretrained encoder from epoch 33 with msg: <All keys matched successfully>
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] loaded pretrained predictor from epoch 33 with msg: <All keys matched successfully>
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] loaded pretrained target encoder from epoch 33 with msg: <All keys matched successfully>
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] loaded optimizers from epoch 33
+[INFO    ][2026-05-10 05:06:09][root                ][load_checkpoint          ] read-path: /scratch-shared/dcanez/runs/vjepa/inflated/d003_lora_ema_capilvd1689m/latest.pt
+[INFO    ][2026-05-10 05:06:09][app.vjepa.train     ][main                     ] Initializing loader...
+submitit WARNING (2026-05-10 09:26:31,151) - Bypassing signal SIGTERM
+[WARNING ][2026-05-10 09:26:31][submitit            ][bypass                   ] Bypassing signal SIGTERM
+submitit WARNING (2026-05-10 09:26:31,151) - Bypassing signal SIGCONT
+[WARNING ][2026-05-10 09:26:31][submitit            ][bypass                   ] Bypassing signal SIGCONT

vjepa/inflated/d003_lora_ema_capilvd1689m/latest.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fd4faecc49808185455a3d5d603607efad5ceb715685eef6f482cf66a60053bd
+size 2577781895