sync run artifacts: vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m

vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/best.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9512a4cf34b50ecb9239fc8591308a08a5b4308913a6de1d020380e2a2a03ded
+size 2573591501

vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/git-info.txt

@@ -0,0 +1,2 @@
+branch: main
+commit: 5180282dd49ee659e684550b98e06afd946a84ff

vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/job_22595640/22595640_0_log.err

@@ -0,0 +1,115 @@
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
+  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
+  return func(*args, **kwargs)
+[rank0]:[W508 20:23:12.158441144 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
+wandb: Currently logged in as: dgcnz (uvjepa) to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
+wandb: setting up run 2y6xyb8y
+wandb: Tracking run with wandb version 0.23.1
+wandb: Run data is saved locally in /scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/wandb/run-20260508_202320-2y6xyb8y
+wandb: Run `wandb offline` to turn off syncing.
+wandb: Syncing run d001_lora_frozen2d_bidir_capilvd1689m
+wandb: ⭐️ View project at https://wandb.ai/uvjepa/vjepa_ablation
+wandb: 🚀 View run at https://wandb.ai/uvjepa/vjepa_ablation/runs/2y6xyb8y
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
+  scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
+  return func(*args, **kwargs)
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
+  return func(*args, **kwargs)
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
+  return func(*args, **kwargs)
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
+  return func(*args, **kwargs)
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
+  return func(*args, **kwargs)
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
+  return func(*args, **kwargs)
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
+  return func(*args, **kwargs)
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
+  return func(*args, **kwargs)
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+submitit WARNING (2026-05-09 08:16:14,174) - Caught signal SIGUSR2 on gcn75.local.snellius.surf.nl: this job is timed-out.
+srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
+submitit WARNING (2026-05-09 08:16:14,296) - Bypassing signal SIGCONT
+[2026-05-09T08:16:14.297] error: *** JOB 22595640 ON gcn75 CANCELLED AT 2026-05-09T08:16:14 DUE TO JOB REQUEUE ***
+[2026-05-09T08:16:14.297] error: *** STEP 22595640.0 ON gcn75 CANCELLED AT 2026-05-09T08:16:14 DUE TO JOB REQUEUE ***
+submitit WARNING (2026-05-09 08:16:14,298) - Bypassing signal SIGCONT
+submitit WARNING (2026-05-09 08:16:14,300) - Bypassing signal SIGTERM
+[rank0]:[W509 08:16:16.066936918 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
+[2026-05-09T08:16:25.024] error: Failed to send MESSAGE_TASK_EXIT: Connection refused
+[2026-05-09T08:16:48.063] error: namespace_p_join: open failed for /slurm/22595640/.ns: No such file or directory
+[2026-05-09T08:16:48.064] error: namespace_g_join(JobId=22595640 SLUID=s8FXKR4CZ0G200): No such file or directory
+[2026-05-09T08:16:49.701] error: Failed to send MESSAGE_TASK_EXIT: Connection refused
+[2026-05-09T08:16:51.600] error: namespace_p_join: open failed for /slurm/22595640/.ns: No such file or directory
+[2026-05-09T08:16:51.602] error: namespace_g_join(JobId=22595640 SLUID=s8FXKR4CZ0G200): No such file or directory
+[2026-05-09T08:16:51.741] error: namespace_p_join: open failed for /slurm/22595640/.ns: No such file or directory
+[2026-05-09T08:16:51.741] error: namespace_g_join(JobId=22595640 SLUID=s8FXKR4CZ0G200): No such file or directory
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
+  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
+  return func(*args, **kwargs)
+[rank0]:[W509 08:25:29.573478230 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
+wandb: Currently logged in as: dgcnz (uvjepa) to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
+wandb: setting up run 2y6xyb8y
+wandb: Tracking run with wandb version 0.23.1
+wandb: Run data is saved locally in /scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/wandb/run-20260509_082534-2y6xyb8y
+wandb: Run `wandb offline` to turn off syncing.
+wandb: Resuming run d001_lora_frozen2d_bidir_capilvd1689m
+wandb: ⭐️ View project at https://wandb.ai/uvjepa/vjepa_ablation
+wandb: 🚀 View run at https://wandb.ai/uvjepa/vjepa_ablation/runs/2y6xyb8y
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
+  scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+wandb: WARNING Tried to log to step 11700 that is less than the current step 11831. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+wandb: WARNING Tried to log to step 11710 that is less than the current step 11831. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+wandb: WARNING Tried to log to step 11720 that is less than the current step 11831. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+wandb: WARNING Tried to log to step 11730 that is less than the current step 11831. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+wandb: WARNING Tried to log to step 11740 that is less than the current step 11831. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+wandb: WARNING Tried to log to step 11750 that is less than the current step 11831. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+wandb: WARNING Tried to log to step 11760 that is less than the current step 11831. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+wandb: WARNING Tried to log to step 11770 that is less than the current step 11831. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+wandb: WARNING Tried to log to step 11780 that is less than the current step 11831. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+wandb: WARNING Tried to log to step 11790 that is less than the current step 11831. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+wandb: WARNING Tried to log to step 11800 that is less than the current step 11831. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+wandb: WARNING Tried to log to step 11810 that is less than the current step 11831. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+wandb: WARNING Tried to log to step 11820 that is less than the current step 11831. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+wandb: WARNING Tried to log to step 11830 that is less than the current step 11831. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
+  return func(*args, **kwargs)
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
+  return func(*args, **kwargs)
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
+[2026-05-09T09:56:00.243] error: *** JOB 22595640 ON gcn73 CANCELLED AT 2026-05-09T09:56:00 DUE to SIGNAL Terminated ***
+[2026-05-09T09:56:00.244] error: *** STEP 22595640.0 ON gcn73 CANCELLED AT 2026-05-09T09:56:00 DUE to SIGNAL Terminated ***
+submitit WARNING (2026-05-09 09:56:00,820) - Bypassing signal SIGTERM
+submitit WARNING (2026-05-09 09:56:00,824) - Bypassing signal SIGCONT
+[2026-05-09T09:56:35.681] error: namespace_p_join: open failed for /slurm/22595640/.ns: No such file or directory
+[2026-05-09T09:56:35.681] error: namespace_g_join(JobId=22595640 SLUID=s8FXKR5P56S300): No such file or directory
+[2026-05-09T09:56:37.331] error: Failed to send MESSAGE_TASK_EXIT: Connection refused
+[2026-05-09T09:56:37.481] error: namespace_p_join: open failed for /slurm/22595640/.ns: No such file or directory
+[2026-05-09T09:56:37.481] error: namespace_g_join(JobId=22595640 SLUID=s8FXKR5P56S300): No such file or directory
+[2026-05-09T09:56:39.119] error: Failed to send MESSAGE_TASK_EXIT: Connection refused
+[2026-05-09T09:56:39.274] error: namespace_p_join: open failed for /slurm/22595640/.ns: No such file or directory
+[2026-05-09T09:56:39.274] error: namespace_g_join(JobId=22595640 SLUID=s8FXKR5P56S300): No such file or directory
+[2026-05-09T09:56:39.413] error: namespace_p_join: open failed for /slurm/22595640/.ns: No such file or directory
+[2026-05-09T09:56:39.413] error: namespace_g_join(JobId=22595640 SLUID=s8FXKR5P56S300): No such file or directory

vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/job_22595640/22595640_10_log.err

@@ -0,0 +1,305 @@
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
+  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
+[rank10]:[W508 20:22:19.099652594 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
+  scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+submitit WARNING (2026-05-09 08:16:14,377) - Caught signal SIGUSR2 on gcn81.local.snellius.surf.nl: this job is timed-out.
+submitit WARNING (2026-05-09 08:16:14,377) - Bypassing signal SIGTERM
+submitit WARNING (2026-05-09 08:16:14,377) - Bypassing signal SIGCONT
+[rank10]:[W509 08:16:18.190214277 TCPStore.cpp:125] [c10d] recvValue failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58904, remote=[gcn75.local.snellius.surf.nl]:37129): Failed to recv, got 0 bytes. Connection was likely closed. Did the remote server shutdown or crash?
+Exception raised from recvBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:682 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14b9cf770fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a3325d (0x14ba138ff25d in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x273 (0x14ba138fd1f3 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14b9d09dfeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14bb0fe49164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14bb208bb19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14bb20940100 in /lib64/libc.so.6)
+
+[rank10]:[W509 08:16:18.192608845 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 10] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Failed to recv, got 0 bytes. Connection was likely closed. Did the remote server shutdown or crash?
+[rank10]:[W509 08:16:19.192765622 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58904, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14b9cf770fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14ba138fe6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14ba138fd1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14b9d09dfeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14bb0fe49164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14bb208bb19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14bb20940100 in /lib64/libc.so.6)
+
+[rank10]:[W509 08:16:19.195840437 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 10] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank10]:[W509 08:16:20.195989163 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58904, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14b9cf770fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14ba138fe6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14ba138fd1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14b9d09dfeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14bb0fe49164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14bb208bb19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14bb20940100 in /lib64/libc.so.6)
+
+[rank10]:[W509 08:16:20.198324532 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 10] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank10]:[W509 08:16:21.198484488 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58904, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14b9cf770fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14ba138fe6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14ba138fd1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14b9d09dfeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14bb0fe49164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14bb208bb19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14bb20940100 in /lib64/libc.so.6)
+
+[rank10]:[W509 08:16:21.200759027 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 10] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank10]:[W509 08:16:22.200904204 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58904, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14b9cf770fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14ba138fe6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14ba138fd1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14b9d09dfeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14bb0fe49164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14bb208bb19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14bb20940100 in /lib64/libc.so.6)
+
+[rank10]:[W509 08:16:22.203256903 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 10] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank10]:[W509 08:16:23.203420589 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58904, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14b9cf770fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14ba138fe6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14ba138fd1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14b9d09dfeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14bb0fe49164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14bb208bb19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14bb20940100 in /lib64/libc.so.6)
+
+[rank10]:[W509 08:16:23.205728108 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 10] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank10]:[W509 08:16:24.205873834 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58904, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14b9cf770fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14ba138fe6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14ba138fd1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14b9d09dfeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14bb0fe49164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14bb208bb19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14bb20940100 in /lib64/libc.so.6)
+
+[rank10]:[W509 08:16:24.208119753 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 10] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank10]:[W509 08:16:25.208275179 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58904, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14b9cf770fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14ba138fe6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14ba138fd1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14b9d09dfeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14bb0fe49164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14bb208bb19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14bb20940100 in /lib64/libc.so.6)
+
+[rank10]:[W509 08:16:25.210694308 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 10] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank10]:[W509 08:16:26.210836864 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58904, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14b9cf770fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14ba138fe6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14ba138fd1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14b9d09dfeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14bb0fe49164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14bb208bb19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14bb20940100 in /lib64/libc.so.6)
+
+[rank10]:[W509 08:16:26.213194433 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 10] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank10]:[W509 08:16:27.213347739 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58904, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14b9cf770fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14ba138fe6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14ba138fd1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14b9d09dfeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14bb0fe49164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14bb208bb19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14bb20940100 in /lib64/libc.so.6)
+
+[rank10]:[W509 08:16:27.215624498 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 10] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank10]:[W509 08:16:28.215769314 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58904, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14b9cf770fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14ba138fe6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14ba138fd1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14b9d09dfeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14bb0fe49164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14bb208bb19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14bb20940100 in /lib64/libc.so.6)
+
+[rank10]:[W509 08:16:28.218072753 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 10] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank10]:[W509 08:16:29.218214340 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58904, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14b9cf770fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14ba138fe6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14ba138fd1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14b9d09dfeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14bb0fe49164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14bb208bb19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14bb20940100 in /lib64/libc.so.6)
+
+[rank10]:[W509 08:16:29.220497859 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 10] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank10]:[W509 08:16:30.220656185 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58904, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14b9cf770fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14ba138fe6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14ba138fd1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14b9d09dfeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14bb0fe49164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14bb208bb19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14bb20940100 in /lib64/libc.so.6)
+
+[rank10]:[W509 08:16:30.223009554 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 10] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank10]:[W509 08:16:31.223152250 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58904, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14b9cf770fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14ba138fe6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14ba138fd1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14b9d09dfeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14bb0fe49164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14bb208bb19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14bb20940100 in /lib64/libc.so.6)
+
+[rank10]:[W509 08:16:31.225402339 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 10] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank10]:[W509 08:16:32.225540655 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58904, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14b9cf770fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14ba138fe6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14ba138fd1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14b9d09dfeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14bb0fe49164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14bb208bb19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14bb20940100 in /lib64/libc.so.6)
+
+[rank10]:[W509 08:16:32.227823535 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 10] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank10]:[W509 08:16:33.227961631 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58904, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14b9cf770fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14ba138fe6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14ba138fd1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14b9d09dfeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14bb0fe49164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14bb208bb19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14bb20940100 in /lib64/libc.so.6)
+
+[rank10]:[W509 08:16:33.230267720 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 10] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank10]:[W509 08:16:34.230424106 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58904, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14b9cf770fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14ba138fe6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14ba138fd1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14b9d09dfeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14bb0fe49164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14bb208bb19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14bb20940100 in /lib64/libc.so.6)
+
+[rank10]:[W509 08:16:34.232738345 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 10] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank10]:[W509 08:16:35.232881771 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58904, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14b9cf770fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14ba138fe6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14ba138fd1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14b9d09dfeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14bb0fe49164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14bb208bb19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14bb20940100 in /lib64/libc.so.6)
+
+[rank10]:[W509 08:16:35.235183360 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 10] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank10]:[W509 08:16:36.235324527 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58904, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14b9cf770fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14ba138fe6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14ba138fd1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14b9d09dfeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14bb0fe49164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14bb208bb19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14bb20940100 in /lib64/libc.so.6)
+
+[rank10]:[W509 08:16:36.237600816 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 10] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank10]:[W509 08:16:37.237736982 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58904, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14b9cf770fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14ba138fe6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14ba138fd1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14b9d09dfeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14bb0fe49164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14bb208bb19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14bb20940100 in /lib64/libc.so.6)
+
+[rank10]:[W509 08:16:37.239941262 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 10] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank10]:[W509 08:16:38.240092498 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58904, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14b9cf770fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14ba138fe6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14ba138fd1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14b9d09dfeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14bb0fe49164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14bb208bb19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14bb20940100 in /lib64/libc.so.6)
+
+[rank10]:[W509 08:16:38.242462066 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 10] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank10]:[W509 08:16:39.242597053 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58904, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14b9cf770fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14ba138fe6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14ba138fd1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14b9d09dfeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14bb0fe49164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14bb208bb19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14bb20940100 in /lib64/libc.so.6)
+
+[rank10]:[W509 08:16:39.244895442 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 10] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank10]:[W509 08:16:40.245036628 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58904, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14b9cf770fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14ba138fe6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14ba138fd1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14b9d09dfeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14bb0fe49164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14bb208bb19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14bb20940100 in /lib64/libc.so.6)
+
+[rank10]:[W509 08:16:40.247323417 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 10] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank10]:[W509 08:16:41.247468203 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58904, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14b9cf770fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14ba138fe6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14ba138fd1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14b9d09dfeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14bb0fe49164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14bb208bb19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14bb20940100 in /lib64/libc.so.6)
+
+[rank10]:[W509 08:16:41.249752022 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 10] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank10]:[W509 08:16:42.249887749 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58904, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14b9cf770fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14ba138fe6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14ba138fd1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14b9d09dfeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14bb0fe49164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14bb208bb19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14bb20940100 in /lib64/libc.so.6)
+
+[rank10]:[W509 08:16:42.252208818 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 10] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank10]:[W509 08:16:43.252345474 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58904, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14b9cf770fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14ba138fe6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14ba138fd1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14b9d09dfeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14bb0fe49164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14bb208bb19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14bb20940100 in /lib64/libc.so.6)
+
+[rank10]:[W509 08:16:43.254634743 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 10] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
+  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
+[rank10]:[W509 08:24:02.866489868 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
+  scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+submitit WARNING (2026-05-09 09:56:00,842) - Bypassing signal SIGTERM
+submitit WARNING (2026-05-09 09:56:00,851) - Bypassing signal SIGCONT

vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/job_22595640/22595640_10_log.out

@@ -0,0 +1,530 @@
+submitit INFO (2026-05-08 20:18:11,513) - Starting with JobEnvironment(job_id=22595640, hostname=gcn81.local.snellius.surf.nl, local_rank=2(4), node=2(4), global_rank=10(16))
+submitit INFO (2026-05-08 20:18:11,513) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/job_22595640/22595640_submitted.pkl
+INFO:root:loaded pretrain params...
+{   'app': 'vjepa',
+    'cpus_per_task': 16,
+    'data': {   'batch_size': 64,
+                'crop_size': 224,
+                'dataset_fpcs': [16, 16],
+                'dataset_type': 'VideoDataset',
+                'datasets': [   '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
+                                '/scratch-shared/dcanez/data/ssv2/train.csv'],
+                'datasets_weights': [0.65, 0.35],
+                'fps': 4,
+                'num_workers': 10,
+                'patch_size': 14,
+                'persistent_workers': True,
+                'pin_mem': True,
+                'stage': [   {   'dest': 'kinetics_240',
+                                 'format': 'targz_parts',
+                                 'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
+                             {   'dest': 'ssv2',
+                                 'format': 'multipart_tar',
+                                 'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
+                'tubelet_size': 1},
+    'data_aug': {   'auto_augment': False,
+                    'motion_shift': False,
+                    'random_resize_aspect_ratio': [0.75, 1.35],
+                    'random_resize_scale': [0.3, 1.0],
+                    'reprob': 0.0},
+    'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m',
+    'loss': {'loss_exp': 1.0},
+    'mask': [   {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 8,
+                    'spatial_scale': [0.15, 0.15],
+                    'temporal_scale': [1.0, 1.0]},
+                {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 2,
+                    'spatial_scale': [0.7, 0.7],
+                    'temporal_scale': [1.0, 1.0]}],
+    'mem_per_gpu': '180G',
+    'meta': {   'dtype': 'bfloat16',
+                'knn_eval_epoch0': False,
+                'knn_eval_freq': 5,
+                'knn_eval_presets': [   {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
+                                                          'eval_videos_per_class': 25,
+                                                          'num_workers': 8,
+                                                          'train_videos_per_class': 100},
+                                            'preset': 'ucf101'},
+                                        {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
+                                                          'eval_videos_per_class': 100,
+                                                          'linear_probe': True,
+                                                          'num_workers': 8,
+                                                          'pool_type': 'temporal_concat',
+                                                          'train_videos_per_class': 500},
+                                            'preset': 'ssv2_coarse10'}],
+                'load_checkpoint': True,
+                'read_checkpoint': None,
+                'save_every_freq': 5,
+                'seed': 239,
+                'use_sdpa': True,
+                'use_wandb': True,
+                'wandb_project': 'vjepa_ablation'},
+    'metrics': {'sigreg': {}, 'std': {}},
+    'model': {   'class_token': False,
+                 'freeze_backbone': True,
+                 'is_causal': False,
+                 'lora_alpha': 16,
+                 'lora_rank': 8,
+                 'lora_targets': ['attn.qkv', 'attn.proj'],
+                 'model_name': 'vit_large_patch14_capi_lvd1689m',
+                 'num_reg_tokens': 16,
+                 'pred_depth': 6,
+                 'pred_embed_dim': 384,
+                 'pred_num_heads': 12,
+                 'rope_type': 'capi2d',
+                 'stem_type': '2d',
+                 'target_kind': 'frozen_2d',
+                 'target_type': 'vit_large_patch14_capi.lvd1689m',
+                 'uniform_power': True,
+                 'use_activation_checkpointing': True,
+                 'use_lora': True,
+                 'use_mask_tokens': True,
+                 'use_rope': True,
+                 'use_sdpa': True,
+                 'zero_init_mask_tokens': True},
+    'nodes': 4,
+    'optimization': {   'ema': [0.99925, 0.99925],
+                        'epochs': 100,
+                        'final_lr': 0.0001,
+                        'final_weight_decay': 0.04,
+                        'ipe': 300,
+                        'ipe_scale': 1.0,
+                        'lr': 0.0005,
+                        'start_lr': 0.0001,
+                        'warmup': 10,
+                        'weight_decay': 0.04},
+    'tasks_per_node': 4}
+INFO:root:Running pre-training of app: vjepa
+[INFO    ][2026-05-08 20:19:12][app.vjepa.train     ][main                     ] which_dtype='bfloat16'
+[INFO    ][2026-05-08 20:19:12][app.vjepa.train     ][main                     ] Disabling persistent_workers (incompatible with KNN eval)
+[INFO    ][2026-05-08 20:19:12][app.vjepa.train     ][main                     ] NCCL_SOCKET_IFNAME=eno
+[INFO    ][2026-05-08 20:19:15][app.vjepa.train     ][main                     ] Initialized (rank/world-size) 10/16, tasks_per_node=4
+[INFO    ][2026-05-08 20:19:15][root                ][stage_datasets           ] [local_rank 2/4] Staging kinetics_240 (targz_parts)
+[INFO    ][2026-05-08 20:19:15][root                ][_stage_targz_parts       ] [rank 2] Extracting 26/103 tar.gz parts to /scratch-node/dcanez.22595640/kinetics_240
+[INFO    ][2026-05-08 20:19:28][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 2/26 parts
+[INFO    ][2026-05-08 20:19:42][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 4/26 parts
+[INFO    ][2026-05-08 20:19:57][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 6/26 parts
+[INFO    ][2026-05-08 20:20:11][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 8/26 parts
+[INFO    ][2026-05-08 20:20:25][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 10/26 parts
+[INFO    ][2026-05-08 20:20:39][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 12/26 parts
+[INFO    ][2026-05-08 20:20:53][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 14/26 parts
+[INFO    ][2026-05-08 20:21:07][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 16/26 parts
+[INFO    ][2026-05-08 20:21:22][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 18/26 parts
+[INFO    ][2026-05-08 20:21:36][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 20/26 parts
+[INFO    ][2026-05-08 20:21:50][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 22/26 parts
+[INFO    ][2026-05-08 20:22:04][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 24/26 parts
+[INFO    ][2026-05-08 20:22:19][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 26/26 parts
+[INFO    ][2026-05-08 20:22:19][root                ][stage_datasets           ] [local_rank 2/4] Staging ssv2 (multipart_tar)
+[INFO    ][2026-05-08 20:23:17][app.vjepa.train     ][main                     ] Staged datasets: ['/scratch-node/dcanez.22595640/kinetics_240/train.csv', '/scratch-node/dcanez.22595640/ssv2/train.csv']
+[INFO    ][2026-05-08 20:24:23][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
+[INFO    ][2026-05-08 20:24:25][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA applied: 48 modules replaced (rank=8, alpha=16, dropout=0.0, targets=['attn.qkv', 'attn.proj'], scope=blocks)
+[INFO    ][2026-05-08 20:24:25][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA trainable params: 1,179,648
+[INFO    ][2026-05-08 20:24:27][root                ][_build_frozen_2d_target  ] Loaded frozen_2d target from timm: vit_large_patch14_capi.lvd1689m
+[INFO    ][2026-05-08 20:24:28][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-08 20:24:28][root                ][init_video_model         ] VJEPAPredictorMultiSeqWrapper(
+  (backbone): VisionTransformerPredictor(
+    (predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
+    (mask_tokens): ParameterList(
+        (0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+    )
+    (predictor_blocks): ModuleList(
+      (0-5): 6 x Block(
+        (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (attn): RoPEAttention(
+          (qkv): Linear(in_features=384, out_features=1152, bias=True)
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): Linear(in_features=384, out_features=384, bias=True)
+          (proj_drop): Dropout(p=0.0, inplace=False)
+        )
+        (drop_path): Identity()
+        (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (mlp): MLP(
+          (fc1): Linear(in_features=384, out_features=1536, bias=True)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=1536, out_features=384, bias=True)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+    (predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
+  )
+)
+[INFO    ][2026-05-08 20:24:28][root                ][init_video_model         ] MultiSeqWrapper(
+  (backbone): Frozen2DTargetWrapper(
+    (backbone): Eva(
+      (patch_embed): PatchEmbed(
+        (proj): Conv2d(3, 1024, kernel_size=(14, 14), stride=(14, 14))
+        (norm): Identity()
+      )
+      (pos_drop): Dropout(p=0.0, inplace=False)
+      (norm_pre): Identity()
+      (blocks): ModuleList(
+        (0-23): 24 x EvaBlock(
+          (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+          (attn): EvaAttention(
+            (qkv): Linear(in_features=1024, out_features=3072, bias=False)
+            (q_norm): Identity()
+            (k_norm): Identity()
+            (attn_drop): Dropout(p=0.0, inplace=False)
+            (norm): Identity()
+            (proj): Linear(in_features=1024, out_features=1024, bias=True)
+            (proj_drop): Dropout(p=0.0, inplace=False)
+          )
+          (drop_path1): Identity()
+          (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+          (mlp): Mlp(
+            (fc1): Linear(in_features=1024, out_features=4096, bias=True)
+            (act): GELU(approximate='none')
+            (drop1): Dropout(p=0.0, inplace=False)
+            (norm): Identity()
+            (fc2): Linear(in_features=4096, out_features=1024, bias=True)
+            (drop2): Dropout(p=0.0, inplace=False)
+          )
+          (drop_path2): Identity()
+        )
+      )
+      (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+      (fc_norm): Identity()
+      (head_drop): Dropout(p=0.0, inplace=False)
+      (head): Identity()
+      (rope): _CapiPatchRoPE()
+    )
+  )
+)
+[INFO    ][2026-05-08 20:24:28][root                ][init_video_model         ] Encoder number of parameters: 1179648
+[INFO    ][2026-05-08 20:24:28][root                ][init_video_model         ] Predictor number of parameters: 11436928
+[INFO    ][2026-05-08 20:24:28][root                ][init_video_model         ] Target encoder number of parameters: 0
+[INFO    ][2026-05-08 20:24:29][root                ][make_videodataset        ] VideoDataset dataset created
+[INFO    ][2026-05-08 20:24:29][WeightedSampler     ][__init__                 ] Using DistributedWeightedSampler with rank 10 / 16
+[INFO    ][2026-05-08 20:24:29][root                ][make_videodataset        ] VideoDataset unsupervised data loader created
+[INFO    ][2026-05-08 20:24:29][app.vjepa.train     ][main                     ] iterations per epoch/dataset length: 300/399
+[INFO    ][2026-05-08 20:24:29][app.vjepa.train     ][main                     ] Wrapping models in DDP (rank 10)...
+[INFO    ][2026-05-08 20:24:41][app.vjepa.train     ][main                     ] Initializing loader...
+submitit INFO (2026-05-09 08:16:14,358) - Job has timed out. Ran 718 minutes out of requested 720 minutes.
+[INFO    ][2026-05-09 08:16:14][submitit            ][has_timed_out            ] Job has timed out. Ran 718 minutes out of requested 720 minutes.
+submitit WARNING (2026-05-09 08:16:14,377) - Caught signal SIGUSR2 on gcn81.local.snellius.surf.nl: this job is timed-out.
+[WARNING ][2026-05-09 08:16:14][submitit            ][checkpoint_and_try_requeue] Caught signal SIGUSR2 on gcn81.local.snellius.surf.nl: this job is timed-out.
+submitit INFO (2026-05-09 08:16:14,377) - Not checkpointing nor requeuing since I am a slave (procid=10).
+[INFO    ][2026-05-09 08:16:14][submitit            ][checkpoint_and_try_requeue] Not checkpointing nor requeuing since I am a slave (procid=10).
+submitit WARNING (2026-05-09 08:16:14,377) - Bypassing signal SIGTERM
+[WARNING ][2026-05-09 08:16:14][submitit            ][bypass                   ] Bypassing signal SIGTERM
+submitit WARNING (2026-05-09 08:16:14,377) - Bypassing signal SIGCONT
+[WARNING ][2026-05-09 08:16:14][submitit            ][bypass                   ] Bypassing signal SIGCONT
+submitit INFO (2026-05-09 08:19:43,932) - Starting with JobEnvironment(job_id=22595640, hostname=gcn76.local.snellius.surf.nl, local_rank=2(4), node=2(4), global_rank=10(16))
+submitit INFO (2026-05-09 08:19:44,740) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/job_22595640/22595640_submitted.pkl
+INFO:root:loaded pretrain params...
+{   'app': 'vjepa',
+    'cpus_per_task': 16,
+    'data': {   'batch_size': 64,
+                'crop_size': 224,
+                'dataset_fpcs': [16, 16],
+                'dataset_type': 'VideoDataset',
+                'datasets': [   '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
+                                '/scratch-shared/dcanez/data/ssv2/train.csv'],
+                'datasets_weights': [0.65, 0.35],
+                'fps': 4,
+                'num_workers': 10,
+                'patch_size': 14,
+                'persistent_workers': True,
+                'pin_mem': True,
+                'stage': [   {   'dest': 'kinetics_240',
+                                 'format': 'targz_parts',
+                                 'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
+                             {   'dest': 'ssv2',
+                                 'format': 'multipart_tar',
+                                 'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
+                'tubelet_size': 1},
+    'data_aug': {   'auto_augment': False,
+                    'motion_shift': False,
+                    'random_resize_aspect_ratio': [0.75, 1.35],
+                    'random_resize_scale': [0.3, 1.0],
+                    'reprob': 0.0},
+    'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m',
+    'loss': {'loss_exp': 1.0},
+    'mask': [   {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 8,
+                    'spatial_scale': [0.15, 0.15],
+                    'temporal_scale': [1.0, 1.0]},
+                {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 2,
+                    'spatial_scale': [0.7, 0.7],
+                    'temporal_scale': [1.0, 1.0]}],
+    'mem_per_gpu': '180G',
+    'meta': {   'dtype': 'bfloat16',
+                'knn_eval_epoch0': False,
+                'knn_eval_freq': 5,
+                'knn_eval_presets': [   {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
+                                                          'eval_videos_per_class': 25,
+                                                          'num_workers': 8,
+                                                          'train_videos_per_class': 100},
+                                            'preset': 'ucf101'},
+                                        {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
+                                                          'eval_videos_per_class': 100,
+                                                          'linear_probe': True,
+                                                          'num_workers': 8,
+                                                          'pool_type': 'temporal_concat',
+                                                          'train_videos_per_class': 500},
+                                            'preset': 'ssv2_coarse10'}],
+                'load_checkpoint': True,
+                'read_checkpoint': None,
+                'save_every_freq': 5,
+                'seed': 239,
+                'use_sdpa': True,
+                'use_wandb': True,
+                'wandb_project': 'vjepa_ablation'},
+    'metrics': {'sigreg': {}, 'std': {}},
+    'model': {   'class_token': False,
+                 'freeze_backbone': True,
+                 'is_causal': False,
+                 'lora_alpha': 16,
+                 'lora_rank': 8,
+                 'lora_targets': ['attn.qkv', 'attn.proj'],
+                 'model_name': 'vit_large_patch14_capi_lvd1689m',
+                 'num_reg_tokens': 16,
+                 'pred_depth': 6,
+                 'pred_embed_dim': 384,
+                 'pred_num_heads': 12,
+                 'rope_type': 'capi2d',
+                 'stem_type': '2d',
+                 'target_kind': 'frozen_2d',
+                 'target_type': 'vit_large_patch14_capi.lvd1689m',
+                 'uniform_power': True,
+                 'use_activation_checkpointing': True,
+                 'use_lora': True,
+                 'use_mask_tokens': True,
+                 'use_rope': True,
+                 'use_sdpa': True,
+                 'zero_init_mask_tokens': True},
+    'nodes': 4,
+    'optimization': {   'ema': [0.99925, 0.99925],
+                        'epochs': 100,
+                        'final_lr': 0.0001,
+                        'final_weight_decay': 0.04,
+                        'ipe': 300,
+                        'ipe_scale': 1.0,
+                        'lr': 0.0005,
+                        'start_lr': 0.0001,
+                        'warmup': 10,
+                        'weight_decay': 0.04},
+    'tasks_per_node': 4}
+INFO:root:Running pre-training of app: vjepa
+[INFO    ][2026-05-09 08:19:48][app.vjepa.train     ][main                     ] which_dtype='bfloat16'
+[INFO    ][2026-05-09 08:19:48][app.vjepa.train     ][main                     ] Disabling persistent_workers (incompatible with KNN eval)
+[INFO    ][2026-05-09 08:19:48][app.vjepa.train     ][main                     ] NCCL_SOCKET_IFNAME=eno
+[INFO    ][2026-05-09 08:20:57][app.vjepa.train     ][main                     ] Initialized (rank/world-size) 10/16, tasks_per_node=4
+[INFO    ][2026-05-09 08:20:57][root                ][stage_datasets           ] [local_rank 2/4] Staging kinetics_240 (targz_parts)
+[INFO    ][2026-05-09 08:20:57][root                ][_stage_targz_parts       ] [rank 2] Extracting 26/103 tar.gz parts to /scratch-node/dcanez.22595640/kinetics_240
+[INFO    ][2026-05-09 08:21:11][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 2/26 parts
+[INFO    ][2026-05-09 08:21:25][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 4/26 parts
+[INFO    ][2026-05-09 08:21:39][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 6/26 parts
+[INFO    ][2026-05-09 08:21:54][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 8/26 parts
+[INFO    ][2026-05-09 08:22:08][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 10/26 parts
+[INFO    ][2026-05-09 08:22:22][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 12/26 parts
+[INFO    ][2026-05-09 08:22:36][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 14/26 parts
+[INFO    ][2026-05-09 08:22:50][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 16/26 parts
+[INFO    ][2026-05-09 08:23:05][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 18/26 parts
+[INFO    ][2026-05-09 08:23:19][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 20/26 parts
+[INFO    ][2026-05-09 08:23:33][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 22/26 parts
+[INFO    ][2026-05-09 08:23:47][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 24/26 parts
+[INFO    ][2026-05-09 08:24:02][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 26/26 parts
+[INFO    ][2026-05-09 08:24:02][root                ][stage_datasets           ] [local_rank 2/4] Staging ssv2 (multipart_tar)
+[INFO    ][2026-05-09 08:25:33][app.vjepa.train     ][main                     ] Staged datasets: ['/scratch-node/dcanez.22595640/kinetics_240/train.csv', '/scratch-node/dcanez.22595640/ssv2/train.csv']
+[INFO    ][2026-05-09 08:26:26][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
+[INFO    ][2026-05-09 08:26:30][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA applied: 48 modules replaced (rank=8, alpha=16, dropout=0.0, targets=['attn.qkv', 'attn.proj'], scope=blocks)
+[INFO    ][2026-05-09 08:26:30][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA trainable params: 1,179,648
+[INFO    ][2026-05-09 08:26:33][root                ][_build_frozen_2d_target  ] Loaded frozen_2d target from timm: vit_large_patch14_capi.lvd1689m
+[INFO    ][2026-05-09 08:26:33][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-09 08:26:33][root                ][init_video_model         ] VJEPAPredictorMultiSeqWrapper(
+  (backbone): VisionTransformerPredictor(
+    (predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
+    (mask_tokens): ParameterList(
+        (0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+    )
+    (predictor_blocks): ModuleList(
+      (0-5): 6 x Block(
+        (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (attn): RoPEAttention(
+          (qkv): Linear(in_features=384, out_features=1152, bias=True)
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): Linear(in_features=384, out_features=384, bias=True)
+          (proj_drop): Dropout(p=0.0, inplace=False)
+        )
+        (drop_path): Identity()
+        (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (mlp): MLP(
+          (fc1): Linear(in_features=384, out_features=1536, bias=True)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=1536, out_features=384, bias=True)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+    (predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
+  )
+)
+[INFO    ][2026-05-09 08:26:33][root                ][init_video_model         ] MultiSeqWrapper(
+  (backbone): Frozen2DTargetWrapper(
+    (backbone): Eva(
+      (patch_embed): PatchEmbed(
+        (proj): Conv2d(3, 1024, kernel_size=(14, 14), stride=(14, 14))
+        (norm): Identity()
+      )
+      (pos_drop): Dropout(p=0.0, inplace=False)
+      (norm_pre): Identity()
+      (blocks): ModuleList(
+        (0-23): 24 x EvaBlock(
+          (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+          (attn): EvaAttention(
+            (qkv): Linear(in_features=1024, out_features=3072, bias=False)
+            (q_norm): Identity()
+            (k_norm): Identity()
+            (attn_drop): Dropout(p=0.0, inplace=False)
+            (norm): Identity()
+            (proj): Linear(in_features=1024, out_features=1024, bias=True)
+            (proj_drop): Dropout(p=0.0, inplace=False)
+          )
+          (drop_path1): Identity()
+          (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+          (mlp): Mlp(
+            (fc1): Linear(in_features=1024, out_features=4096, bias=True)
+            (act): GELU(approximate='none')
+            (drop1): Dropout(p=0.0, inplace=False)
+            (norm): Identity()
+            (fc2): Linear(in_features=4096, out_features=1024, bias=True)
+            (drop2): Dropout(p=0.0, inplace=False)
+          )
+          (drop_path2): Identity()
+        )
+      )
+      (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+      (fc_norm): Identity()
+      (head_drop): Dropout(p=0.0, inplace=False)
+      (head): Identity()
+      (rope): _CapiPatchRoPE()
+    )
+  )
+)
+[INFO    ][2026-05-09 08:26:33][root                ][init_video_model         ] Encoder number of parameters: 1179648
+[INFO    ][2026-05-09 08:26:33][root                ][init_video_model         ] Predictor number of parameters: 11436928
+[INFO    ][2026-05-09 08:26:33][root                ][init_video_model         ] Target encoder number of parameters: 0
+[INFO    ][2026-05-09 08:26:34][root                ][make_videodataset        ] VideoDataset dataset created
+[INFO    ][2026-05-09 08:26:34][WeightedSampler     ][__init__                 ] Using DistributedWeightedSampler with rank 10 / 16
+[INFO    ][2026-05-09 08:26:34][root                ][make_videodataset        ] VideoDataset unsupervised data loader created
+[INFO    ][2026-05-09 08:26:34][app.vjepa.train     ][main                     ] iterations per epoch/dataset length: 300/399
+[INFO    ][2026-05-09 08:26:37][app.vjepa.train     ][main                     ] Wrapping models in DDP (rank 10)...
+[INFO    ][2026-05-09 08:26:37][root                ][load_checkpoint          ] Loading checkpoint from /scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/latest.pt
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] loaded pretrained encoder from epoch 39 with msg: <All keys matched successfully>
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] loaded pretrained predictor from epoch 39 with msg: <All keys matched successfully>
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] loaded pretrained target encoder from epoch 39 with msg: <All keys matched successfully>
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] loaded optimizers from epoch 39
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] read-path: /scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/latest.pt
+[INFO    ][2026-05-09 08:26:39][app.vjepa.train     ][main                     ] Initializing loader...
+submitit WARNING (2026-05-09 09:56:00,842) - Bypassing signal SIGTERM
+[WARNING ][2026-05-09 09:56:00][submitit            ][bypass                   ] Bypassing signal SIGTERM
+submitit WARNING (2026-05-09 09:56:00,851) - Bypassing signal SIGCONT
+[WARNING ][2026-05-09 09:56:00][submitit            ][bypass                   ] Bypassing signal SIGCONT

vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/job_22595640/22595640_11_log.err

@@ -0,0 +1,305 @@
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
+  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
+[rank11]:[W508 20:22:14.270362429 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
+  scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+submitit WARNING (2026-05-09 08:16:14,392) - Caught signal SIGUSR2 on gcn81.local.snellius.surf.nl: this job is timed-out.
+submitit WARNING (2026-05-09 08:16:14,392) - Bypassing signal SIGTERM
+submitit WARNING (2026-05-09 08:16:14,392) - Bypassing signal SIGCONT
+[rank11]:[W509 08:16:18.937938133 TCPStore.cpp:125] [c10d] recvValue failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58932, remote=[gcn75.local.snellius.surf.nl]:37129): Failed to recv, got 0 bytes. Connection was likely closed. Did the remote server shutdown or crash?
+Exception raised from recvBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:682 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15209e714fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a3325d (0x1520e28a325d in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x273 (0x1520e28a11f3 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15209f983eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1521deded164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1521ef85f19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1521ef8e4100 in /lib64/libc.so.6)
+
+[rank11]:[W509 08:16:18.943204498 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 11] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Failed to recv, got 0 bytes. Connection was likely closed. Did the remote server shutdown or crash?
+[rank11]:[W509 08:16:19.943355155 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58932, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15209e714fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1520e28a26d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1520e28a11cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15209f983eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1521deded164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1521ef85f19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1521ef8e4100 in /lib64/libc.so.6)
+
+[rank11]:[W509 08:16:19.946439940 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 11] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank11]:[W509 08:16:20.946596456 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58932, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15209e714fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1520e28a26d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1520e28a11cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15209f983eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1521deded164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1521ef85f19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1521ef8e4100 in /lib64/libc.so.6)
+
+[rank11]:[W509 08:16:20.948891955 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 11] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank11]:[W509 08:16:21.949065711 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58932, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15209e714fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1520e28a26d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1520e28a11cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15209f983eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1521deded164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1521ef85f19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1521ef8e4100 in /lib64/libc.so.6)
+
+[rank11]:[W509 08:16:21.951448050 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 11] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank11]:[W509 08:16:22.951588436 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58932, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15209e714fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1520e28a26d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1520e28a11cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15209f983eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1521deded164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1521ef85f19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1521ef8e4100 in /lib64/libc.so.6)
+
+[rank11]:[W509 08:16:22.953910155 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 11] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank11]:[W509 08:16:23.954062121 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58932, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15209e714fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1520e28a26d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1520e28a11cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15209f983eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1521deded164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1521ef85f19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1521ef8e4100 in /lib64/libc.so.6)
+
+[rank11]:[W509 08:16:23.956357800 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 11] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank11]:[W509 08:16:24.956493007 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58932, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15209e714fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1520e28a26d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1520e28a11cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15209f983eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1521deded164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1521ef85f19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1521ef8e4100 in /lib64/libc.so.6)
+
+[rank11]:[W509 08:16:24.958792716 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 11] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank11]:[W509 08:16:25.958936552 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58932, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15209e714fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1520e28a26d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1520e28a11cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15209f983eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1521deded164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1521ef85f19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1521ef8e4100 in /lib64/libc.so.6)
+
+[rank11]:[W509 08:16:25.961230731 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 11] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank11]:[W509 08:16:26.961381347 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58932, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15209e714fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1520e28a26d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1520e28a11cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15209f983eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1521deded164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1521ef85f19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1521ef8e4100 in /lib64/libc.so.6)
+
+[rank11]:[W509 08:16:26.963623507 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 11] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank11]:[W509 08:16:27.963768303 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58932, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15209e714fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1520e28a26d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1520e28a11cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15209f983eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1521deded164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1521ef85f19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1521ef8e4100 in /lib64/libc.so.6)
+
+[rank11]:[W509 08:16:27.966125662 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 11] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank11]:[W509 08:16:28.966256708 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58932, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15209e714fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1520e28a26d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1520e28a11cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15209f983eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1521deded164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1521ef85f19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1521ef8e4100 in /lib64/libc.so.6)
+
+[rank11]:[W509 08:16:28.968565207 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 11] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank11]:[W509 08:16:29.968698933 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58932, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15209e714fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1520e28a26d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1520e28a11cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15209f983eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1521deded164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1521ef85f19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1521ef8e4100 in /lib64/libc.so.6)
+
+[rank11]:[W509 08:16:29.971042632 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 11] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank11]:[W509 08:16:30.971172618 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58932, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15209e714fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1520e28a26d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1520e28a11cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15209f983eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1521deded164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1521ef85f19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1521ef8e4100 in /lib64/libc.so.6)
+
+[rank11]:[W509 08:16:30.973518657 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 11] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank11]:[W509 08:16:31.973652733 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58932, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15209e714fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1520e28a26d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1520e28a11cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15209f983eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1521deded164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1521ef85f19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1521ef8e4100 in /lib64/libc.so.6)
+
+[rank11]:[W509 08:16:31.976014082 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 11] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank11]:[W509 08:16:32.976153128 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58932, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15209e714fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1520e28a26d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1520e28a11cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15209f983eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1521deded164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1521ef85f19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1521ef8e4100 in /lib64/libc.so.6)
+
+[rank11]:[W509 08:16:32.978447397 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 11] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank11]:[W509 08:16:33.978585684 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58932, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15209e714fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1520e28a26d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1520e28a11cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15209f983eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1521deded164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1521ef85f19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1521ef8e4100 in /lib64/libc.so.6)
+
+[rank11]:[W509 08:16:33.980900193 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 11] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank11]:[W509 08:16:34.981044309 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58932, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15209e714fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1520e28a26d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1520e28a11cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15209f983eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1521deded164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1521ef85f19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1521ef8e4100 in /lib64/libc.so.6)
+
+[rank11]:[W509 08:16:34.983336098 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 11] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank11]:[W509 08:16:35.983471714 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58932, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15209e714fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1520e28a26d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1520e28a11cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15209f983eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1521deded164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1521ef85f19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1521ef8e4100 in /lib64/libc.so.6)
+
+[rank11]:[W509 08:16:35.985815153 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 11] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank11]:[W509 08:16:36.985959629 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58932, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15209e714fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1520e28a26d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1520e28a11cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15209f983eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1521deded164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1521ef85f19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1521ef8e4100 in /lib64/libc.so.6)
+
+[rank11]:[W509 08:16:36.988294928 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 11] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank11]:[W509 08:16:37.988431174 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58932, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15209e714fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1520e28a26d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1520e28a11cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15209f983eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1521deded164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1521ef85f19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1521ef8e4100 in /lib64/libc.so.6)
+
+[rank11]:[W509 08:16:37.990791283 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 11] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank11]:[W509 08:16:38.990923969 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58932, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15209e714fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1520e28a26d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1520e28a11cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15209f983eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1521deded164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1521ef85f19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1521ef8e4100 in /lib64/libc.so.6)
+
+[rank11]:[W509 08:16:38.993244978 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 11] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank11]:[W509 08:16:39.993374085 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58932, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15209e714fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1520e28a26d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1520e28a11cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15209f983eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1521deded164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1521ef85f19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1521ef8e4100 in /lib64/libc.so.6)
+
+[rank11]:[W509 08:16:39.995688144 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 11] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank11]:[W509 08:16:40.995821050 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58932, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15209e714fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1520e28a26d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1520e28a11cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15209f983eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1521deded164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1521ef85f19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1521ef8e4100 in /lib64/libc.so.6)
+
+[rank11]:[W509 08:16:40.998117449 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 11] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank11]:[W509 08:16:41.998258975 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58932, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15209e714fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1520e28a26d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1520e28a11cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15209f983eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1521deded164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1521ef85f19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1521ef8e4100 in /lib64/libc.so.6)
+
+[rank11]:[W509 08:16:41.000593744 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 11] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank11]:[W509 08:16:42.000729100 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58932, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15209e714fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1520e28a26d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1520e28a11cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15209f983eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1521deded164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1521ef85f19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1521ef8e4100 in /lib64/libc.so.6)
+
+[rank11]:[W509 08:16:42.003103289 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 11] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank11]:[W509 08:16:43.003238915 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58932, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15209e714fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1520e28a26d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1520e28a11cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x15209f983eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1521deded164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1521ef85f19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1521ef8e4100 in /lib64/libc.so.6)
+
+[rank11]:[W509 08:16:43.005591054 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 11] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
+  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
+[rank11]:[W509 08:24:17.098899958 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
+  scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+submitit WARNING (2026-05-09 09:56:00,859) - Bypassing signal SIGTERM
+submitit WARNING (2026-05-09 09:56:00,860) - Bypassing signal SIGCONT

vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/job_22595640/22595640_11_log.out

@@ -0,0 +1,530 @@
+submitit INFO (2026-05-08 20:18:11,513) - Starting with JobEnvironment(job_id=22595640, hostname=gcn81.local.snellius.surf.nl, local_rank=3(4), node=2(4), global_rank=11(16))
+submitit INFO (2026-05-08 20:18:11,513) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/job_22595640/22595640_submitted.pkl
+INFO:root:loaded pretrain params...
+{   'app': 'vjepa',
+    'cpus_per_task': 16,
+    'data': {   'batch_size': 64,
+                'crop_size': 224,
+                'dataset_fpcs': [16, 16],
+                'dataset_type': 'VideoDataset',
+                'datasets': [   '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
+                                '/scratch-shared/dcanez/data/ssv2/train.csv'],
+                'datasets_weights': [0.65, 0.35],
+                'fps': 4,
+                'num_workers': 10,
+                'patch_size': 14,
+                'persistent_workers': True,
+                'pin_mem': True,
+                'stage': [   {   'dest': 'kinetics_240',
+                                 'format': 'targz_parts',
+                                 'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
+                             {   'dest': 'ssv2',
+                                 'format': 'multipart_tar',
+                                 'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
+                'tubelet_size': 1},
+    'data_aug': {   'auto_augment': False,
+                    'motion_shift': False,
+                    'random_resize_aspect_ratio': [0.75, 1.35],
+                    'random_resize_scale': [0.3, 1.0],
+                    'reprob': 0.0},
+    'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m',
+    'loss': {'loss_exp': 1.0},
+    'mask': [   {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 8,
+                    'spatial_scale': [0.15, 0.15],
+                    'temporal_scale': [1.0, 1.0]},
+                {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 2,
+                    'spatial_scale': [0.7, 0.7],
+                    'temporal_scale': [1.0, 1.0]}],
+    'mem_per_gpu': '180G',
+    'meta': {   'dtype': 'bfloat16',
+                'knn_eval_epoch0': False,
+                'knn_eval_freq': 5,
+                'knn_eval_presets': [   {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
+                                                          'eval_videos_per_class': 25,
+                                                          'num_workers': 8,
+                                                          'train_videos_per_class': 100},
+                                            'preset': 'ucf101'},
+                                        {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
+                                                          'eval_videos_per_class': 100,
+                                                          'linear_probe': True,
+                                                          'num_workers': 8,
+                                                          'pool_type': 'temporal_concat',
+                                                          'train_videos_per_class': 500},
+                                            'preset': 'ssv2_coarse10'}],
+                'load_checkpoint': True,
+                'read_checkpoint': None,
+                'save_every_freq': 5,
+                'seed': 239,
+                'use_sdpa': True,
+                'use_wandb': True,
+                'wandb_project': 'vjepa_ablation'},
+    'metrics': {'sigreg': {}, 'std': {}},
+    'model': {   'class_token': False,
+                 'freeze_backbone': True,
+                 'is_causal': False,
+                 'lora_alpha': 16,
+                 'lora_rank': 8,
+                 'lora_targets': ['attn.qkv', 'attn.proj'],
+                 'model_name': 'vit_large_patch14_capi_lvd1689m',
+                 'num_reg_tokens': 16,
+                 'pred_depth': 6,
+                 'pred_embed_dim': 384,
+                 'pred_num_heads': 12,
+                 'rope_type': 'capi2d',
+                 'stem_type': '2d',
+                 'target_kind': 'frozen_2d',
+                 'target_type': 'vit_large_patch14_capi.lvd1689m',
+                 'uniform_power': True,
+                 'use_activation_checkpointing': True,
+                 'use_lora': True,
+                 'use_mask_tokens': True,
+                 'use_rope': True,
+                 'use_sdpa': True,
+                 'zero_init_mask_tokens': True},
+    'nodes': 4,
+    'optimization': {   'ema': [0.99925, 0.99925],
+                        'epochs': 100,
+                        'final_lr': 0.0001,
+                        'final_weight_decay': 0.04,
+                        'ipe': 300,
+                        'ipe_scale': 1.0,
+                        'lr': 0.0005,
+                        'start_lr': 0.0001,
+                        'warmup': 10,
+                        'weight_decay': 0.04},
+    'tasks_per_node': 4}
+INFO:root:Running pre-training of app: vjepa
+[INFO    ][2026-05-08 20:19:12][app.vjepa.train     ][main                     ] which_dtype='bfloat16'
+[INFO    ][2026-05-08 20:19:12][app.vjepa.train     ][main                     ] Disabling persistent_workers (incompatible with KNN eval)
+[INFO    ][2026-05-08 20:19:12][app.vjepa.train     ][main                     ] NCCL_SOCKET_IFNAME=eno
+[INFO    ][2026-05-08 20:19:16][app.vjepa.train     ][main                     ] Initialized (rank/world-size) 11/16, tasks_per_node=4
+[INFO    ][2026-05-08 20:19:16][root                ][stage_datasets           ] [local_rank 3/4] Staging kinetics_240 (targz_parts)
+[INFO    ][2026-05-08 20:19:16][root                ][_stage_targz_parts       ] [rank 3] Extracting 25/103 tar.gz parts to /scratch-node/dcanez.22595640/kinetics_240
+[INFO    ][2026-05-08 20:19:30][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 2/25 parts
+[INFO    ][2026-05-08 20:19:45][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 4/25 parts
+[INFO    ][2026-05-08 20:19:58][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 6/25 parts
+[INFO    ][2026-05-08 20:20:13][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 8/25 parts
+[INFO    ][2026-05-08 20:20:27][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 10/25 parts
+[INFO    ][2026-05-08 20:20:41][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 12/25 parts
+[INFO    ][2026-05-08 20:20:56][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 14/25 parts
+[INFO    ][2026-05-08 20:21:10][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 16/25 parts
+[INFO    ][2026-05-08 20:21:23][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 18/25 parts
+[INFO    ][2026-05-08 20:21:37][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 20/25 parts
+[INFO    ][2026-05-08 20:21:52][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 22/25 parts
+[INFO    ][2026-05-08 20:22:05][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 24/25 parts
+[INFO    ][2026-05-08 20:22:12][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 25/25 parts
+[INFO    ][2026-05-08 20:22:12][root                ][stage_datasets           ] [local_rank 3/4] Staging ssv2 (multipart_tar)
+[INFO    ][2026-05-08 20:23:17][app.vjepa.train     ][main                     ] Staged datasets: ['/scratch-node/dcanez.22595640/kinetics_240/train.csv', '/scratch-node/dcanez.22595640/ssv2/train.csv']
+[INFO    ][2026-05-08 20:24:23][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
+[INFO    ][2026-05-08 20:24:25][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA applied: 48 modules replaced (rank=8, alpha=16, dropout=0.0, targets=['attn.qkv', 'attn.proj'], scope=blocks)
+[INFO    ][2026-05-08 20:24:25][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA trainable params: 1,179,648
+[INFO    ][2026-05-08 20:24:27][root                ][_build_frozen_2d_target  ] Loaded frozen_2d target from timm: vit_large_patch14_capi.lvd1689m
+[INFO    ][2026-05-08 20:24:28][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-08 20:24:28][root                ][init_video_model         ] VJEPAPredictorMultiSeqWrapper(
+  (backbone): VisionTransformerPredictor(
+    (predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
+    (mask_tokens): ParameterList(
+        (0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+    )
+    (predictor_blocks): ModuleList(
+      (0-5): 6 x Block(
+        (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (attn): RoPEAttention(
+          (qkv): Linear(in_features=384, out_features=1152, bias=True)
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): Linear(in_features=384, out_features=384, bias=True)
+          (proj_drop): Dropout(p=0.0, inplace=False)
+        )
+        (drop_path): Identity()
+        (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (mlp): MLP(
+          (fc1): Linear(in_features=384, out_features=1536, bias=True)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=1536, out_features=384, bias=True)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+    (predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
+  )
+)
+[INFO    ][2026-05-08 20:24:28][root                ][init_video_model         ] MultiSeqWrapper(
+  (backbone): Frozen2DTargetWrapper(
+    (backbone): Eva(
+      (patch_embed): PatchEmbed(
+        (proj): Conv2d(3, 1024, kernel_size=(14, 14), stride=(14, 14))
+        (norm): Identity()
+      )
+      (pos_drop): Dropout(p=0.0, inplace=False)
+      (norm_pre): Identity()
+      (blocks): ModuleList(
+        (0-23): 24 x EvaBlock(
+          (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+          (attn): EvaAttention(
+            (qkv): Linear(in_features=1024, out_features=3072, bias=False)
+            (q_norm): Identity()
+            (k_norm): Identity()
+            (attn_drop): Dropout(p=0.0, inplace=False)
+            (norm): Identity()
+            (proj): Linear(in_features=1024, out_features=1024, bias=True)
+            (proj_drop): Dropout(p=0.0, inplace=False)
+          )
+          (drop_path1): Identity()
+          (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+          (mlp): Mlp(
+            (fc1): Linear(in_features=1024, out_features=4096, bias=True)
+            (act): GELU(approximate='none')
+            (drop1): Dropout(p=0.0, inplace=False)
+            (norm): Identity()
+            (fc2): Linear(in_features=4096, out_features=1024, bias=True)
+            (drop2): Dropout(p=0.0, inplace=False)
+          )
+          (drop_path2): Identity()
+        )
+      )
+      (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+      (fc_norm): Identity()
+      (head_drop): Dropout(p=0.0, inplace=False)
+      (head): Identity()
+      (rope): _CapiPatchRoPE()
+    )
+  )
+)
+[INFO    ][2026-05-08 20:24:28][root                ][init_video_model         ] Encoder number of parameters: 1179648
+[INFO    ][2026-05-08 20:24:28][root                ][init_video_model         ] Predictor number of parameters: 11436928
+[INFO    ][2026-05-08 20:24:28][root                ][init_video_model         ] Target encoder number of parameters: 0
+[INFO    ][2026-05-08 20:24:29][root                ][make_videodataset        ] VideoDataset dataset created
+[INFO    ][2026-05-08 20:24:29][WeightedSampler     ][__init__                 ] Using DistributedWeightedSampler with rank 11 / 16
+[INFO    ][2026-05-08 20:24:29][root                ][make_videodataset        ] VideoDataset unsupervised data loader created
+[INFO    ][2026-05-08 20:24:29][app.vjepa.train     ][main                     ] iterations per epoch/dataset length: 300/399
+[INFO    ][2026-05-08 20:24:29][app.vjepa.train     ][main                     ] Wrapping models in DDP (rank 11)...
+[INFO    ][2026-05-08 20:24:41][app.vjepa.train     ][main                     ] Initializing loader...
+submitit INFO (2026-05-09 08:16:14,373) - Job has timed out. Ran 718 minutes out of requested 720 minutes.
+[INFO    ][2026-05-09 08:16:14][submitit            ][has_timed_out            ] Job has timed out. Ran 718 minutes out of requested 720 minutes.
+submitit WARNING (2026-05-09 08:16:14,392) - Caught signal SIGUSR2 on gcn81.local.snellius.surf.nl: this job is timed-out.
+[WARNING ][2026-05-09 08:16:14][submitit            ][checkpoint_and_try_requeue] Caught signal SIGUSR2 on gcn81.local.snellius.surf.nl: this job is timed-out.
+submitit INFO (2026-05-09 08:16:14,392) - Not checkpointing nor requeuing since I am a slave (procid=11).
+[INFO    ][2026-05-09 08:16:14][submitit            ][checkpoint_and_try_requeue] Not checkpointing nor requeuing since I am a slave (procid=11).
+submitit WARNING (2026-05-09 08:16:14,392) - Bypassing signal SIGTERM
+[WARNING ][2026-05-09 08:16:14][submitit            ][bypass                   ] Bypassing signal SIGTERM
+submitit WARNING (2026-05-09 08:16:14,392) - Bypassing signal SIGCONT
+[WARNING ][2026-05-09 08:16:14][submitit            ][bypass                   ] Bypassing signal SIGCONT
+submitit INFO (2026-05-09 08:19:43,932) - Starting with JobEnvironment(job_id=22595640, hostname=gcn76.local.snellius.surf.nl, local_rank=3(4), node=2(4), global_rank=11(16))
+submitit INFO (2026-05-09 08:19:44,810) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/job_22595640/22595640_submitted.pkl
+INFO:root:loaded pretrain params...
+{   'app': 'vjepa',
+    'cpus_per_task': 16,
+    'data': {   'batch_size': 64,
+                'crop_size': 224,
+                'dataset_fpcs': [16, 16],
+                'dataset_type': 'VideoDataset',
+                'datasets': [   '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
+                                '/scratch-shared/dcanez/data/ssv2/train.csv'],
+                'datasets_weights': [0.65, 0.35],
+                'fps': 4,
+                'num_workers': 10,
+                'patch_size': 14,
+                'persistent_workers': True,
+                'pin_mem': True,
+                'stage': [   {   'dest': 'kinetics_240',
+                                 'format': 'targz_parts',
+                                 'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
+                             {   'dest': 'ssv2',
+                                 'format': 'multipart_tar',
+                                 'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
+                'tubelet_size': 1},
+    'data_aug': {   'auto_augment': False,
+                    'motion_shift': False,
+                    'random_resize_aspect_ratio': [0.75, 1.35],
+                    'random_resize_scale': [0.3, 1.0],
+                    'reprob': 0.0},
+    'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m',
+    'loss': {'loss_exp': 1.0},
+    'mask': [   {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 8,
+                    'spatial_scale': [0.15, 0.15],
+                    'temporal_scale': [1.0, 1.0]},
+                {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 2,
+                    'spatial_scale': [0.7, 0.7],
+                    'temporal_scale': [1.0, 1.0]}],
+    'mem_per_gpu': '180G',
+    'meta': {   'dtype': 'bfloat16',
+                'knn_eval_epoch0': False,
+                'knn_eval_freq': 5,
+                'knn_eval_presets': [   {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
+                                                          'eval_videos_per_class': 25,
+                                                          'num_workers': 8,
+                                                          'train_videos_per_class': 100},
+                                            'preset': 'ucf101'},
+                                        {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
+                                                          'eval_videos_per_class': 100,
+                                                          'linear_probe': True,
+                                                          'num_workers': 8,
+                                                          'pool_type': 'temporal_concat',
+                                                          'train_videos_per_class': 500},
+                                            'preset': 'ssv2_coarse10'}],
+                'load_checkpoint': True,
+                'read_checkpoint': None,
+                'save_every_freq': 5,
+                'seed': 239,
+                'use_sdpa': True,
+                'use_wandb': True,
+                'wandb_project': 'vjepa_ablation'},
+    'metrics': {'sigreg': {}, 'std': {}},
+    'model': {   'class_token': False,
+                 'freeze_backbone': True,
+                 'is_causal': False,
+                 'lora_alpha': 16,
+                 'lora_rank': 8,
+                 'lora_targets': ['attn.qkv', 'attn.proj'],
+                 'model_name': 'vit_large_patch14_capi_lvd1689m',
+                 'num_reg_tokens': 16,
+                 'pred_depth': 6,
+                 'pred_embed_dim': 384,
+                 'pred_num_heads': 12,
+                 'rope_type': 'capi2d',
+                 'stem_type': '2d',
+                 'target_kind': 'frozen_2d',
+                 'target_type': 'vit_large_patch14_capi.lvd1689m',
+                 'uniform_power': True,
+                 'use_activation_checkpointing': True,
+                 'use_lora': True,
+                 'use_mask_tokens': True,
+                 'use_rope': True,
+                 'use_sdpa': True,
+                 'zero_init_mask_tokens': True},
+    'nodes': 4,
+    'optimization': {   'ema': [0.99925, 0.99925],
+                        'epochs': 100,
+                        'final_lr': 0.0001,
+                        'final_weight_decay': 0.04,
+                        'ipe': 300,
+                        'ipe_scale': 1.0,
+                        'lr': 0.0005,
+                        'start_lr': 0.0001,
+                        'warmup': 10,
+                        'weight_decay': 0.04},
+    'tasks_per_node': 4}
+INFO:root:Running pre-training of app: vjepa
+[INFO    ][2026-05-09 08:19:48][app.vjepa.train     ][main                     ] which_dtype='bfloat16'
+[INFO    ][2026-05-09 08:19:48][app.vjepa.train     ][main                     ] Disabling persistent_workers (incompatible with KNN eval)
+[INFO    ][2026-05-09 08:19:48][app.vjepa.train     ][main                     ] NCCL_SOCKET_IFNAME=eno
+[INFO    ][2026-05-09 08:21:19][app.vjepa.train     ][main                     ] Initialized (rank/world-size) 11/16, tasks_per_node=4
+[INFO    ][2026-05-09 08:21:19][root                ][stage_datasets           ] [local_rank 3/4] Staging kinetics_240 (targz_parts)
+[INFO    ][2026-05-09 08:21:19][root                ][_stage_targz_parts       ] [rank 3] Extracting 25/103 tar.gz parts to /scratch-node/dcanez.22595640/kinetics_240
+[INFO    ][2026-05-09 08:21:33][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 2/25 parts
+[INFO    ][2026-05-09 08:21:48][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 4/25 parts
+[INFO    ][2026-05-09 08:22:02][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 6/25 parts
+[INFO    ][2026-05-09 08:22:16][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 8/25 parts
+[INFO    ][2026-05-09 08:22:31][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 10/25 parts
+[INFO    ][2026-05-09 08:22:45][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 12/25 parts
+[INFO    ][2026-05-09 08:22:59][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 14/25 parts
+[INFO    ][2026-05-09 08:23:13][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 16/25 parts
+[INFO    ][2026-05-09 08:23:27][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 18/25 parts
+[INFO    ][2026-05-09 08:23:42][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 20/25 parts
+[INFO    ][2026-05-09 08:23:56][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 22/25 parts
+[INFO    ][2026-05-09 08:24:10][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 24/25 parts
+[INFO    ][2026-05-09 08:24:17][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 25/25 parts
+[INFO    ][2026-05-09 08:24:17][root                ][stage_datasets           ] [local_rank 3/4] Staging ssv2 (multipart_tar)
+[INFO    ][2026-05-09 08:25:33][app.vjepa.train     ][main                     ] Staged datasets: ['/scratch-node/dcanez.22595640/kinetics_240/train.csv', '/scratch-node/dcanez.22595640/ssv2/train.csv']
+[INFO    ][2026-05-09 08:26:26][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
+[INFO    ][2026-05-09 08:26:30][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA applied: 48 modules replaced (rank=8, alpha=16, dropout=0.0, targets=['attn.qkv', 'attn.proj'], scope=blocks)
+[INFO    ][2026-05-09 08:26:30][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA trainable params: 1,179,648
+[INFO    ][2026-05-09 08:26:33][root                ][_build_frozen_2d_target  ] Loaded frozen_2d target from timm: vit_large_patch14_capi.lvd1689m
+[INFO    ][2026-05-09 08:26:33][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-09 08:26:33][root                ][init_video_model         ] VJEPAPredictorMultiSeqWrapper(
+  (backbone): VisionTransformerPredictor(
+    (predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
+    (mask_tokens): ParameterList(
+        (0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+    )
+    (predictor_blocks): ModuleList(
+      (0-5): 6 x Block(
+        (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (attn): RoPEAttention(
+          (qkv): Linear(in_features=384, out_features=1152, bias=True)
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): Linear(in_features=384, out_features=384, bias=True)
+          (proj_drop): Dropout(p=0.0, inplace=False)
+        )
+        (drop_path): Identity()
+        (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (mlp): MLP(
+          (fc1): Linear(in_features=384, out_features=1536, bias=True)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=1536, out_features=384, bias=True)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+    (predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
+  )
+)
+[INFO    ][2026-05-09 08:26:33][root                ][init_video_model         ] MultiSeqWrapper(
+  (backbone): Frozen2DTargetWrapper(
+    (backbone): Eva(
+      (patch_embed): PatchEmbed(
+        (proj): Conv2d(3, 1024, kernel_size=(14, 14), stride=(14, 14))
+        (norm): Identity()
+      )
+      (pos_drop): Dropout(p=0.0, inplace=False)
+      (norm_pre): Identity()
+      (blocks): ModuleList(
+        (0-23): 24 x EvaBlock(
+          (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+          (attn): EvaAttention(
+            (qkv): Linear(in_features=1024, out_features=3072, bias=False)
+            (q_norm): Identity()
+            (k_norm): Identity()
+            (attn_drop): Dropout(p=0.0, inplace=False)
+            (norm): Identity()
+            (proj): Linear(in_features=1024, out_features=1024, bias=True)
+            (proj_drop): Dropout(p=0.0, inplace=False)
+          )
+          (drop_path1): Identity()
+          (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+          (mlp): Mlp(
+            (fc1): Linear(in_features=1024, out_features=4096, bias=True)
+            (act): GELU(approximate='none')
+            (drop1): Dropout(p=0.0, inplace=False)
+            (norm): Identity()
+            (fc2): Linear(in_features=4096, out_features=1024, bias=True)
+            (drop2): Dropout(p=0.0, inplace=False)
+          )
+          (drop_path2): Identity()
+        )
+      )
+      (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+      (fc_norm): Identity()
+      (head_drop): Dropout(p=0.0, inplace=False)
+      (head): Identity()
+      (rope): _CapiPatchRoPE()
+    )
+  )
+)
+[INFO    ][2026-05-09 08:26:33][root                ][init_video_model         ] Encoder number of parameters: 1179648
+[INFO    ][2026-05-09 08:26:33][root                ][init_video_model         ] Predictor number of parameters: 11436928
+[INFO    ][2026-05-09 08:26:33][root                ][init_video_model         ] Target encoder number of parameters: 0
+[INFO    ][2026-05-09 08:26:34][root                ][make_videodataset        ] VideoDataset dataset created
+[INFO    ][2026-05-09 08:26:34][WeightedSampler     ][__init__                 ] Using DistributedWeightedSampler with rank 11 / 16
+[INFO    ][2026-05-09 08:26:34][root                ][make_videodataset        ] VideoDataset unsupervised data loader created
+[INFO    ][2026-05-09 08:26:34][app.vjepa.train     ][main                     ] iterations per epoch/dataset length: 300/399
+[INFO    ][2026-05-09 08:26:35][app.vjepa.train     ][main                     ] Wrapping models in DDP (rank 11)...
+[INFO    ][2026-05-09 08:26:37][root                ][load_checkpoint          ] Loading checkpoint from /scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/latest.pt
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] loaded pretrained encoder from epoch 39 with msg: <All keys matched successfully>
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] loaded pretrained predictor from epoch 39 with msg: <All keys matched successfully>
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] loaded pretrained target encoder from epoch 39 with msg: <All keys matched successfully>
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] loaded optimizers from epoch 39
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] read-path: /scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/latest.pt
+[INFO    ][2026-05-09 08:26:39][app.vjepa.train     ][main                     ] Initializing loader...
+submitit WARNING (2026-05-09 09:56:00,859) - Bypassing signal SIGTERM
+[WARNING ][2026-05-09 09:56:00][submitit            ][bypass                   ] Bypassing signal SIGTERM
+submitit WARNING (2026-05-09 09:56:00,860) - Bypassing signal SIGCONT
+[WARNING ][2026-05-09 09:56:00][submitit            ][bypass                   ] Bypassing signal SIGCONT

vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/job_22595640/22595640_12_log.err

@@ -0,0 +1,323 @@
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
+  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
+[rank12]:[W508 20:23:13.639260124 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
+  scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+submitit WARNING (2026-05-09 08:16:14,384) - Caught signal SIGUSR2 on gcn84.local.snellius.surf.nl: this job is timed-out.
+submitit WARNING (2026-05-09 08:16:14,384) - Bypassing signal SIGTERM
+submitit WARNING (2026-05-09 08:16:14,384) - Bypassing signal SIGCONT
+[rank12]:[W509 08:16:18.167926540 TCPStore.cpp:125] [c10d] recvValue failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37368, remote=[gcn75.local.snellius.surf.nl]:37129): Failed to recv, got 0 bytes. Connection was likely closed. Did the remote server shutdown or crash?
+Exception raised from recvBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:682 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14617ccc7fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a3325d (0x1461c0e5625d in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x273 (0x1461c0e541f3 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14617df36eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1462bd3a0164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1462cde1219a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1462cde97100 in /lib64/libc.so.6)
+
+[rank12]:[W509 08:16:18.170439633 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 12] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Failed to recv, got 0 bytes. Connection was likely closed. Did the remote server shutdown or crash?
+[rank12]:[W509 08:16:19.170595894 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37368, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14617ccc7fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1461c0e556d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1461c0e541cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14617df36eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1462bd3a0164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1462cde1219a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1462cde97100 in /lib64/libc.so.6)
+
+[rank12]:[W509 08:16:19.173877952 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 12] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank12]:[W509 08:16:20.174043204 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37368, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14617ccc7fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1461c0e556d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1461c0e541cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14617df36eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1462bd3a0164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1462cde1219a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1462cde97100 in /lib64/libc.so.6)
+
+[rank12]:[W509 08:16:20.176437848 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 12] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank12]:[W509 08:16:21.176563230 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37368, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14617ccc7fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1461c0e556d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1461c0e541cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14617df36eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1462bd3a0164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1462cde1219a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1462cde97100 in /lib64/libc.so.6)
+
+[rank12]:[W509 08:16:21.178899024 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 12] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank12]:[W509 08:16:22.179066156 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37368, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14617ccc7fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1461c0e556d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1461c0e541cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14617df36eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1462bd3a0164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1462cde1219a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1462cde97100 in /lib64/libc.so.6)
+
+[rank12]:[W509 08:16:22.181458420 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 12] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank12]:[W509 08:16:23.181619041 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37368, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14617ccc7fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1461c0e556d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1461c0e541cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14617df36eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1462bd3a0164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1462cde1219a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1462cde97100 in /lib64/libc.so.6)
+
+[rank12]:[W509 08:16:23.183989856 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 12] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank12]:[W509 08:16:24.184146957 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37368, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14617ccc7fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1461c0e556d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1461c0e541cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14617df36eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1462bd3a0164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1462cde1219a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1462cde97100 in /lib64/libc.so.6)
+
+[rank12]:[W509 08:16:24.186601541 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 12] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank12]:[W509 08:16:25.186747662 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37368, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14617ccc7fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1461c0e556d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1461c0e541cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14617df36eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1462bd3a0164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1462cde1219a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1462cde97100 in /lib64/libc.so.6)
+
+[rank12]:[W509 08:16:25.189105197 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 12] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank12]:[W509 08:16:26.189229708 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37368, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14617ccc7fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1461c0e556d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1461c0e541cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14617df36eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1462bd3a0164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1462cde1219a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1462cde97100 in /lib64/libc.so.6)
+
+[rank12]:[W509 08:16:26.191568593 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 12] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank12]:[W509 08:16:27.191728465 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37368, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14617ccc7fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1461c0e556d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1461c0e541cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14617df36eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1462bd3a0164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1462cde1219a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1462cde97100 in /lib64/libc.so.6)
+
+[rank12]:[W509 08:16:27.194112208 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 12] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank12]:[W509 08:16:28.194242640 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37368, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14617ccc7fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1461c0e556d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1461c0e541cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14617df36eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1462bd3a0164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1462cde1219a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1462cde97100 in /lib64/libc.so.6)
+
+[rank12]:[W509 08:16:28.196487165 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 12] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank12]:[W509 08:16:29.196617597 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37368, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14617ccc7fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1461c0e556d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1461c0e541cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14617df36eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1462bd3a0164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1462cde1219a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1462cde97100 in /lib64/libc.so.6)
+
+[rank12]:[W509 08:16:29.198911052 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 12] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank12]:[W509 08:16:30.199069103 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37368, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14617ccc7fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1461c0e556d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1461c0e541cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14617df36eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1462bd3a0164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1462cde1219a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1462cde97100 in /lib64/libc.so.6)
+
+[rank12]:[W509 08:16:30.201486147 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 12] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank12]:[W509 08:16:31.201638069 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37368, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14617ccc7fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1461c0e556d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1461c0e541cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14617df36eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1462bd3a0164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1462cde1219a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1462cde97100 in /lib64/libc.so.6)
+
+[rank12]:[W509 08:16:31.203930493 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 12] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank12]:[W509 08:16:32.204079585 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37368, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14617ccc7fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1461c0e556d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1461c0e541cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14617df36eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1462bd3a0164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1462cde1219a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1462cde97100 in /lib64/libc.so.6)
+
+[rank12]:[W509 08:16:32.206314240 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 12] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank12]:[W509 08:16:33.206446262 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37368, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14617ccc7fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1461c0e556d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1461c0e541cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14617df36eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1462bd3a0164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1462cde1219a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1462cde97100 in /lib64/libc.so.6)
+
+[rank12]:[W509 08:16:33.208793256 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 12] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank12]:[W509 08:16:34.208924558 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37368, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14617ccc7fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1461c0e556d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1461c0e541cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14617df36eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1462bd3a0164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1462cde1219a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1462cde97100 in /lib64/libc.so.6)
+
+[rank12]:[W509 08:16:34.211177873 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 12] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank12]:[W509 08:16:35.211354094 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37368, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14617ccc7fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1461c0e556d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1461c0e541cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14617df36eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1462bd3a0164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1462cde1219a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1462cde97100 in /lib64/libc.so.6)
+
+[rank12]:[W509 08:16:35.213701259 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 12] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank12]:[W509 08:16:36.213832451 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37368, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14617ccc7fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1461c0e556d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1461c0e541cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14617df36eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1462bd3a0164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1462cde1219a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1462cde97100 in /lib64/libc.so.6)
+
+[rank12]:[W509 08:16:36.216137985 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 12] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank12]:[W509 08:16:37.216264787 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37368, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14617ccc7fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1461c0e556d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1461c0e541cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14617df36eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1462bd3a0164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1462cde1219a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1462cde97100 in /lib64/libc.so.6)
+
+[rank12]:[W509 08:16:37.218513472 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 12] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank12]:[W509 08:16:38.218649954 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37368, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14617ccc7fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1461c0e556d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1461c0e541cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14617df36eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1462bd3a0164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1462cde1219a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1462cde97100 in /lib64/libc.so.6)
+
+[rank12]:[W509 08:16:38.220953678 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 12] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank12]:[W509 08:16:39.221081760 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37368, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14617ccc7fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1461c0e556d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1461c0e541cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14617df36eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1462bd3a0164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1462cde1219a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1462cde97100 in /lib64/libc.so.6)
+
+[rank12]:[W509 08:16:39.223495754 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 12] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank12]:[W509 08:16:40.223632746 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37368, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14617ccc7fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1461c0e556d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1461c0e541cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14617df36eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1462bd3a0164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1462cde1219a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1462cde97100 in /lib64/libc.so.6)
+
+[rank12]:[W509 08:16:40.225950870 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 12] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank12]:[W509 08:16:41.226078332 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37368, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14617ccc7fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1461c0e556d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1461c0e541cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14617df36eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1462bd3a0164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1462cde1219a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1462cde97100 in /lib64/libc.so.6)
+
+[rank12]:[W509 08:16:41.228365777 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 12] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank12]:[W509 08:16:42.228505089 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37368, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14617ccc7fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1461c0e556d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1461c0e541cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14617df36eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1462bd3a0164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1462cde1219a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1462cde97100 in /lib64/libc.so.6)
+
+[rank12]:[W509 08:16:42.230965322 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 12] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank12]:[W509 08:16:43.231092574 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37368, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14617ccc7fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1461c0e556d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1461c0e541cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14617df36eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1462bd3a0164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1462cde1219a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1462cde97100 in /lib64/libc.so.6)
+
+[rank12]:[W509 08:16:43.233411518 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 12] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[2026-05-09T08:16:46.799] error: namespace_p_join: open failed for /slurm/22595640/.ns: No such file or directory
+[2026-05-09T08:16:46.799] error: namespace_g_join(JobId=22595640 SLUID=s8FXKR4CZ0G200): No such file or directory
+[2026-05-09T08:16:48.444] error: Failed to send MESSAGE_TASK_EXIT: Connection refused
+[2026-05-09T08:16:50.472] error: namespace_p_join: open failed for /slurm/22595640/.ns: No such file or directory
+[2026-05-09T08:16:50.472] error: namespace_g_join(JobId=22595640 SLUID=s8FXKR4CZ0G200): No such file or directory
+[2026-05-09T08:16:50.611] error: namespace_p_join: open failed for /slurm/22595640/.ns: No such file or directory
+[2026-05-09T08:16:50.611] error: namespace_g_join(JobId=22595640 SLUID=s8FXKR4CZ0G200): No such file or directory
+[2026-05-09T08:16:50.650] error: namespace_p_join: open failed for /slurm/22595640/.ns: No such file or directory
+[2026-05-09T08:16:50.650] error: namespace_g_join(JobId=22595640 SLUID=s8FXKR4CZ0G200): No such file or directory
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
+  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
+[rank12]:[W509 08:24:51.130783001 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
+  scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+submitit WARNING (2026-05-09 09:56:00,945) - Bypassing signal SIGTERM
+submitit WARNING (2026-05-09 09:56:00,946) - Bypassing signal SIGCONT
+[2026-05-09T09:56:34.637] error: namespace_p_join: open failed for /slurm/22595640/.ns: No such file or directory
+[2026-05-09T09:56:34.637] error: namespace_g_join(JobId=22595640 SLUID=s8FXKR5P56S300): No such file or directory
+[2026-05-09T09:56:36.280] error: Failed to send MESSAGE_TASK_EXIT: Connection refused
+[2026-05-09T09:56:37.498] error: namespace_p_join: open failed for /slurm/22595640/.ns: No such file or directory
+[2026-05-09T09:56:37.498] error: namespace_g_join(JobId=22595640 SLUID=s8FXKR5P56S300): No such file or directory
+[2026-05-09T09:56:37.637] error: namespace_p_join: open failed for /slurm/22595640/.ns: No such file or directory
+[2026-05-09T09:56:37.638] error: namespace_g_join(JobId=22595640 SLUID=s8FXKR5P56S300): No such file or directory
+[2026-05-09T09:56:37.676] error: namespace_p_join: open failed for /slurm/22595640/.ns: No such file or directory
+[2026-05-09T09:56:37.676] error: namespace_g_join(JobId=22595640 SLUID=s8FXKR5P56S300): No such file or directory

vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/job_22595640/22595640_12_log.out

@@ -0,0 +1,538 @@
+submitit INFO (2026-05-08 20:18:11,518) - Starting with JobEnvironment(job_id=22595640, hostname=gcn84.local.snellius.surf.nl, local_rank=0(4), node=3(4), global_rank=12(16))
+submitit INFO (2026-05-08 20:18:11,518) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/job_22595640/22595640_submitted.pkl
+INFO:root:loaded pretrain params...
+{   'app': 'vjepa',
+    'cpus_per_task': 16,
+    'data': {   'batch_size': 64,
+                'crop_size': 224,
+                'dataset_fpcs': [16, 16],
+                'dataset_type': 'VideoDataset',
+                'datasets': [   '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
+                                '/scratch-shared/dcanez/data/ssv2/train.csv'],
+                'datasets_weights': [0.65, 0.35],
+                'fps': 4,
+                'num_workers': 10,
+                'patch_size': 14,
+                'persistent_workers': True,
+                'pin_mem': True,
+                'stage': [   {   'dest': 'kinetics_240',
+                                 'format': 'targz_parts',
+                                 'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
+                             {   'dest': 'ssv2',
+                                 'format': 'multipart_tar',
+                                 'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
+                'tubelet_size': 1},
+    'data_aug': {   'auto_augment': False,
+                    'motion_shift': False,
+                    'random_resize_aspect_ratio': [0.75, 1.35],
+                    'random_resize_scale': [0.3, 1.0],
+                    'reprob': 0.0},
+    'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m',
+    'loss': {'loss_exp': 1.0},
+    'mask': [   {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 8,
+                    'spatial_scale': [0.15, 0.15],
+                    'temporal_scale': [1.0, 1.0]},
+                {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 2,
+                    'spatial_scale': [0.7, 0.7],
+                    'temporal_scale': [1.0, 1.0]}],
+    'mem_per_gpu': '180G',
+    'meta': {   'dtype': 'bfloat16',
+                'knn_eval_epoch0': False,
+                'knn_eval_freq': 5,
+                'knn_eval_presets': [   {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
+                                                          'eval_videos_per_class': 25,
+                                                          'num_workers': 8,
+                                                          'train_videos_per_class': 100},
+                                            'preset': 'ucf101'},
+                                        {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
+                                                          'eval_videos_per_class': 100,
+                                                          'linear_probe': True,
+                                                          'num_workers': 8,
+                                                          'pool_type': 'temporal_concat',
+                                                          'train_videos_per_class': 500},
+                                            'preset': 'ssv2_coarse10'}],
+                'load_checkpoint': True,
+                'read_checkpoint': None,
+                'save_every_freq': 5,
+                'seed': 239,
+                'use_sdpa': True,
+                'use_wandb': True,
+                'wandb_project': 'vjepa_ablation'},
+    'metrics': {'sigreg': {}, 'std': {}},
+    'model': {   'class_token': False,
+                 'freeze_backbone': True,
+                 'is_causal': False,
+                 'lora_alpha': 16,
+                 'lora_rank': 8,
+                 'lora_targets': ['attn.qkv', 'attn.proj'],
+                 'model_name': 'vit_large_patch14_capi_lvd1689m',
+                 'num_reg_tokens': 16,
+                 'pred_depth': 6,
+                 'pred_embed_dim': 384,
+                 'pred_num_heads': 12,
+                 'rope_type': 'capi2d',
+                 'stem_type': '2d',
+                 'target_kind': 'frozen_2d',
+                 'target_type': 'vit_large_patch14_capi.lvd1689m',
+                 'uniform_power': True,
+                 'use_activation_checkpointing': True,
+                 'use_lora': True,
+                 'use_mask_tokens': True,
+                 'use_rope': True,
+                 'use_sdpa': True,
+                 'zero_init_mask_tokens': True},
+    'nodes': 4,
+    'optimization': {   'ema': [0.99925, 0.99925],
+                        'epochs': 100,
+                        'final_lr': 0.0001,
+                        'final_weight_decay': 0.04,
+                        'ipe': 300,
+                        'ipe_scale': 1.0,
+                        'lr': 0.0005,
+                        'start_lr': 0.0001,
+                        'warmup': 10,
+                        'weight_decay': 0.04},
+    'tasks_per_node': 4}
+INFO:root:Running pre-training of app: vjepa
+[INFO    ][2026-05-08 20:19:11][app.vjepa.train     ][main                     ] which_dtype='bfloat16'
+[INFO    ][2026-05-08 20:19:11][app.vjepa.train     ][main                     ] Disabling persistent_workers (incompatible with KNN eval)
+[INFO    ][2026-05-08 20:19:11][app.vjepa.train     ][main                     ] NCCL_SOCKET_IFNAME=eno
+[INFO    ][2026-05-08 20:19:13][app.vjepa.train     ][main                     ] Initialized (rank/world-size) 12/16, tasks_per_node=4
+[INFO    ][2026-05-08 20:19:13][root                ][stage_datasets           ] [local_rank 0/4] Staging kinetics_240 (targz_parts)
+[INFO    ][2026-05-08 20:19:13][root                ][_stage_targz_parts       ] [rank 0] Extracting 26/103 tar.gz parts to /scratch-node/dcanez.22595640/kinetics_240
+[INFO    ][2026-05-08 20:19:27][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 2/26 parts
+[INFO    ][2026-05-08 20:19:41][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 4/26 parts
+[INFO    ][2026-05-08 20:19:55][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 6/26 parts
+[INFO    ][2026-05-08 20:20:09][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 8/26 parts
+[INFO    ][2026-05-08 20:20:23][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 10/26 parts
+[INFO    ][2026-05-08 20:20:38][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 12/26 parts
+[INFO    ][2026-05-08 20:20:52][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 14/26 parts
+[INFO    ][2026-05-08 20:21:06][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 16/26 parts
+[INFO    ][2026-05-08 20:21:21][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 18/26 parts
+[INFO    ][2026-05-08 20:21:35][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 20/26 parts
+[INFO    ][2026-05-08 20:21:49][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 22/26 parts
+[INFO    ][2026-05-08 20:22:03][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 24/26 parts
+[INFO    ][2026-05-08 20:22:17][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 26/26 parts
+[INFO    ][2026-05-08 20:22:17][root                ][stage_datasets           ] [local_rank 0/4] Staging ssv2 (multipart_tar)
+[INFO    ][2026-05-08 20:22:17][root                ][_stage_multipart_tar     ] [rank 0] Extracting multipart tar (2 files) to /scratch-node/dcanez.22595640/ssv2
+[INFO    ][2026-05-08 20:23:16][root                ][stage_datasets           ] Data staging completed in 242.6s (4.0min)
+[INFO    ][2026-05-08 20:23:16][root                ][_rewrite_csv             ] Wrote local CSV: /scratch-node/dcanez.22595640/kinetics_240/train.csv (239789 entries)
+[INFO    ][2026-05-08 20:23:17][root                ][_rewrite_csv             ] Wrote local CSV: /scratch-node/dcanez.22595640/ssv2/train.csv (168913 entries)
+[INFO    ][2026-05-08 20:23:17][app.vjepa.train     ][main                     ] Staged datasets: ['/scratch-node/dcanez.22595640/kinetics_240/train.csv', '/scratch-node/dcanez.22595640/ssv2/train.csv']
+[INFO    ][2026-05-08 20:24:23][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
+[INFO    ][2026-05-08 20:24:25][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA applied: 48 modules replaced (rank=8, alpha=16, dropout=0.0, targets=['attn.qkv', 'attn.proj'], scope=blocks)
+[INFO    ][2026-05-08 20:24:25][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA trainable params: 1,179,648
+[INFO    ][2026-05-08 20:24:27][root                ][_build_frozen_2d_target  ] Loaded frozen_2d target from timm: vit_large_patch14_capi.lvd1689m
+[INFO    ][2026-05-08 20:24:28][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-08 20:24:28][root                ][init_video_model         ] VJEPAPredictorMultiSeqWrapper(
+  (backbone): VisionTransformerPredictor(
+    (predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
+    (mask_tokens): ParameterList(
+        (0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+    )
+    (predictor_blocks): ModuleList(
+      (0-5): 6 x Block(
+        (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (attn): RoPEAttention(
+          (qkv): Linear(in_features=384, out_features=1152, bias=True)
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): Linear(in_features=384, out_features=384, bias=True)
+          (proj_drop): Dropout(p=0.0, inplace=False)
+        )
+        (drop_path): Identity()
+        (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (mlp): MLP(
+          (fc1): Linear(in_features=384, out_features=1536, bias=True)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=1536, out_features=384, bias=True)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+    (predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
+  )
+)
+[INFO    ][2026-05-08 20:24:28][root                ][init_video_model         ] MultiSeqWrapper(
+  (backbone): Frozen2DTargetWrapper(
+    (backbone): Eva(
+      (patch_embed): PatchEmbed(
+        (proj): Conv2d(3, 1024, kernel_size=(14, 14), stride=(14, 14))
+        (norm): Identity()
+      )
+      (pos_drop): Dropout(p=0.0, inplace=False)
+      (norm_pre): Identity()
+      (blocks): ModuleList(
+        (0-23): 24 x EvaBlock(
+          (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+          (attn): EvaAttention(
+            (qkv): Linear(in_features=1024, out_features=3072, bias=False)
+            (q_norm): Identity()
+            (k_norm): Identity()
+            (attn_drop): Dropout(p=0.0, inplace=False)
+            (norm): Identity()
+            (proj): Linear(in_features=1024, out_features=1024, bias=True)
+            (proj_drop): Dropout(p=0.0, inplace=False)
+          )
+          (drop_path1): Identity()
+          (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+          (mlp): Mlp(
+            (fc1): Linear(in_features=1024, out_features=4096, bias=True)
+            (act): GELU(approximate='none')
+            (drop1): Dropout(p=0.0, inplace=False)
+            (norm): Identity()
+            (fc2): Linear(in_features=4096, out_features=1024, bias=True)
+            (drop2): Dropout(p=0.0, inplace=False)
+          )
+          (drop_path2): Identity()
+        )
+      )
+      (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+      (fc_norm): Identity()
+      (head_drop): Dropout(p=0.0, inplace=False)
+      (head): Identity()
+      (rope): _CapiPatchRoPE()
+    )
+  )
+)
+[INFO    ][2026-05-08 20:24:28][root                ][init_video_model         ] Encoder number of parameters: 1179648
+[INFO    ][2026-05-08 20:24:28][root                ][init_video_model         ] Predictor number of parameters: 11436928
+[INFO    ][2026-05-08 20:24:28][root                ][init_video_model         ] Target encoder number of parameters: 0
+[INFO    ][2026-05-08 20:24:29][root                ][make_videodataset        ] VideoDataset dataset created
+[INFO    ][2026-05-08 20:24:29][WeightedSampler     ][__init__                 ] Using DistributedWeightedSampler with rank 12 / 16
+[INFO    ][2026-05-08 20:24:29][root                ][make_videodataset        ] VideoDataset unsupervised data loader created
+[INFO    ][2026-05-08 20:24:29][app.vjepa.train     ][main                     ] iterations per epoch/dataset length: 300/399
+[INFO    ][2026-05-08 20:24:29][app.vjepa.train     ][main                     ] Wrapping models in DDP (rank 12)...
+[INFO    ][2026-05-08 20:24:41][app.vjepa.train     ][main                     ] Initializing loader...
+submitit INFO (2026-05-09 08:16:14,364) - Job has timed out. Ran 718 minutes out of requested 720 minutes.
+[INFO    ][2026-05-09 08:16:14][submitit            ][has_timed_out            ] Job has timed out. Ran 718 minutes out of requested 720 minutes.
+submitit WARNING (2026-05-09 08:16:14,384) - Caught signal SIGUSR2 on gcn84.local.snellius.surf.nl: this job is timed-out.
+[WARNING ][2026-05-09 08:16:14][submitit            ][checkpoint_and_try_requeue] Caught signal SIGUSR2 on gcn84.local.snellius.surf.nl: this job is timed-out.
+submitit INFO (2026-05-09 08:16:14,384) - Not checkpointing nor requeuing since I am a slave (procid=12).
+[INFO    ][2026-05-09 08:16:14][submitit            ][checkpoint_and_try_requeue] Not checkpointing nor requeuing since I am a slave (procid=12).
+submitit WARNING (2026-05-09 08:16:14,384) - Bypassing signal SIGTERM
+[WARNING ][2026-05-09 08:16:14][submitit            ][bypass                   ] Bypassing signal SIGTERM
+submitit WARNING (2026-05-09 08:16:14,384) - Bypassing signal SIGCONT
+[WARNING ][2026-05-09 08:16:14][submitit            ][bypass                   ] Bypassing signal SIGCONT
+submitit INFO (2026-05-09 08:19:47,092) - Starting with JobEnvironment(job_id=22595640, hostname=gcn78.local.snellius.surf.nl, local_rank=0(4), node=3(4), global_rank=12(16))
+submitit INFO (2026-05-09 08:19:47,108) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/job_22595640/22595640_submitted.pkl
+INFO:root:loaded pretrain params...
+{   'app': 'vjepa',
+    'cpus_per_task': 16,
+    'data': {   'batch_size': 64,
+                'crop_size': 224,
+                'dataset_fpcs': [16, 16],
+                'dataset_type': 'VideoDataset',
+                'datasets': [   '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
+                                '/scratch-shared/dcanez/data/ssv2/train.csv'],
+                'datasets_weights': [0.65, 0.35],
+                'fps': 4,
+                'num_workers': 10,
+                'patch_size': 14,
+                'persistent_workers': True,
+                'pin_mem': True,
+                'stage': [   {   'dest': 'kinetics_240',
+                                 'format': 'targz_parts',
+                                 'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
+                             {   'dest': 'ssv2',
+                                 'format': 'multipart_tar',
+                                 'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
+                'tubelet_size': 1},
+    'data_aug': {   'auto_augment': False,
+                    'motion_shift': False,
+                    'random_resize_aspect_ratio': [0.75, 1.35],
+                    'random_resize_scale': [0.3, 1.0],
+                    'reprob': 0.0},
+    'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m',
+    'loss': {'loss_exp': 1.0},
+    'mask': [   {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 8,
+                    'spatial_scale': [0.15, 0.15],
+                    'temporal_scale': [1.0, 1.0]},
+                {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 2,
+                    'spatial_scale': [0.7, 0.7],
+                    'temporal_scale': [1.0, 1.0]}],
+    'mem_per_gpu': '180G',
+    'meta': {   'dtype': 'bfloat16',
+                'knn_eval_epoch0': False,
+                'knn_eval_freq': 5,
+                'knn_eval_presets': [   {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
+                                                          'eval_videos_per_class': 25,
+                                                          'num_workers': 8,
+                                                          'train_videos_per_class': 100},
+                                            'preset': 'ucf101'},
+                                        {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
+                                                          'eval_videos_per_class': 100,
+                                                          'linear_probe': True,
+                                                          'num_workers': 8,
+                                                          'pool_type': 'temporal_concat',
+                                                          'train_videos_per_class': 500},
+                                            'preset': 'ssv2_coarse10'}],
+                'load_checkpoint': True,
+                'read_checkpoint': None,
+                'save_every_freq': 5,
+                'seed': 239,
+                'use_sdpa': True,
+                'use_wandb': True,
+                'wandb_project': 'vjepa_ablation'},
+    'metrics': {'sigreg': {}, 'std': {}},
+    'model': {   'class_token': False,
+                 'freeze_backbone': True,
+                 'is_causal': False,
+                 'lora_alpha': 16,
+                 'lora_rank': 8,
+                 'lora_targets': ['attn.qkv', 'attn.proj'],
+                 'model_name': 'vit_large_patch14_capi_lvd1689m',
+                 'num_reg_tokens': 16,
+                 'pred_depth': 6,
+                 'pred_embed_dim': 384,
+                 'pred_num_heads': 12,
+                 'rope_type': 'capi2d',
+                 'stem_type': '2d',
+                 'target_kind': 'frozen_2d',
+                 'target_type': 'vit_large_patch14_capi.lvd1689m',
+                 'uniform_power': True,
+                 'use_activation_checkpointing': True,
+                 'use_lora': True,
+                 'use_mask_tokens': True,
+                 'use_rope': True,
+                 'use_sdpa': True,
+                 'zero_init_mask_tokens': True},
+    'nodes': 4,
+    'optimization': {   'ema': [0.99925, 0.99925],
+                        'epochs': 100,
+                        'final_lr': 0.0001,
+                        'final_weight_decay': 0.04,
+                        'ipe': 300,
+                        'ipe_scale': 1.0,
+                        'lr': 0.0005,
+                        'start_lr': 0.0001,
+                        'warmup': 10,
+                        'weight_decay': 0.04},
+    'tasks_per_node': 4}
+INFO:root:Running pre-training of app: vjepa
+[INFO    ][2026-05-09 08:20:49][app.vjepa.train     ][main                     ] which_dtype='bfloat16'
+[INFO    ][2026-05-09 08:20:49][app.vjepa.train     ][main                     ] Disabling persistent_workers (incompatible with KNN eval)
+[INFO    ][2026-05-09 08:20:49][app.vjepa.train     ][main                     ] NCCL_SOCKET_IFNAME=eno
+[INFO    ][2026-05-09 08:20:51][app.vjepa.train     ][main                     ] Initialized (rank/world-size) 12/16, tasks_per_node=4
+[INFO    ][2026-05-09 08:20:51][root                ][stage_datasets           ] [local_rank 0/4] Staging kinetics_240 (targz_parts)
+[INFO    ][2026-05-09 08:20:51][root                ][_stage_targz_parts       ] [rank 0] Extracting 26/103 tar.gz parts to /scratch-node/dcanez.22595640/kinetics_240
+[INFO    ][2026-05-09 08:21:05][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 2/26 parts
+[INFO    ][2026-05-09 08:21:19][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 4/26 parts
+[INFO    ][2026-05-09 08:21:33][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 6/26 parts
+[INFO    ][2026-05-09 08:21:47][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 8/26 parts
+[INFO    ][2026-05-09 08:22:01][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 10/26 parts
+[INFO    ][2026-05-09 08:22:15][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 12/26 parts
+[INFO    ][2026-05-09 08:22:29][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 14/26 parts
+[INFO    ][2026-05-09 08:22:44][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 16/26 parts
+[INFO    ][2026-05-09 08:22:58][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 18/26 parts
+[INFO    ][2026-05-09 08:23:12][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 20/26 parts
+[INFO    ][2026-05-09 08:23:26][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 22/26 parts
+[INFO    ][2026-05-09 08:23:41][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 24/26 parts
+[INFO    ][2026-05-09 08:23:55][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 26/26 parts
+[INFO    ][2026-05-09 08:23:55][root                ][stage_datasets           ] [local_rank 0/4] Staging ssv2 (multipart_tar)
+[INFO    ][2026-05-09 08:23:55][root                ][_stage_multipart_tar     ] [rank 0] Extracting multipart tar (2 files) to /scratch-node/dcanez.22595640/ssv2
+[INFO    ][2026-05-09 08:25:32][root                ][stage_datasets           ] Data staging completed in 281.7s (4.7min)
+[INFO    ][2026-05-09 08:25:33][root                ][_rewrite_csv             ] Wrote local CSV: /scratch-node/dcanez.22595640/kinetics_240/train.csv (239789 entries)
+[INFO    ][2026-05-09 08:25:33][root                ][_rewrite_csv             ] Wrote local CSV: /scratch-node/dcanez.22595640/ssv2/train.csv (168913 entries)
+[INFO    ][2026-05-09 08:25:33][app.vjepa.train     ][main                     ] Staged datasets: ['/scratch-node/dcanez.22595640/kinetics_240/train.csv', '/scratch-node/dcanez.22595640/ssv2/train.csv']
+[INFO    ][2026-05-09 08:26:29][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
+[INFO    ][2026-05-09 08:26:30][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA applied: 48 modules replaced (rank=8, alpha=16, dropout=0.0, targets=['attn.qkv', 'attn.proj'], scope=blocks)
+[INFO    ][2026-05-09 08:26:30][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA trainable params: 1,179,648
+[INFO    ][2026-05-09 08:26:33][root                ][_build_frozen_2d_target  ] Loaded frozen_2d target from timm: vit_large_patch14_capi.lvd1689m
+[INFO    ][2026-05-09 08:26:33][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-09 08:26:33][root                ][init_video_model         ] VJEPAPredictorMultiSeqWrapper(
+  (backbone): VisionTransformerPredictor(
+    (predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
+    (mask_tokens): ParameterList(
+        (0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+    )
+    (predictor_blocks): ModuleList(
+      (0-5): 6 x Block(
+        (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (attn): RoPEAttention(
+          (qkv): Linear(in_features=384, out_features=1152, bias=True)
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): Linear(in_features=384, out_features=384, bias=True)
+          (proj_drop): Dropout(p=0.0, inplace=False)
+        )
+        (drop_path): Identity()
+        (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (mlp): MLP(
+          (fc1): Linear(in_features=384, out_features=1536, bias=True)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=1536, out_features=384, bias=True)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+    (predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
+  )
+)
+[INFO    ][2026-05-09 08:26:33][root                ][init_video_model         ] MultiSeqWrapper(
+  (backbone): Frozen2DTargetWrapper(
+    (backbone): Eva(
+      (patch_embed): PatchEmbed(
+        (proj): Conv2d(3, 1024, kernel_size=(14, 14), stride=(14, 14))
+        (norm): Identity()
+      )
+      (pos_drop): Dropout(p=0.0, inplace=False)
+      (norm_pre): Identity()
+      (blocks): ModuleList(
+        (0-23): 24 x EvaBlock(
+          (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+          (attn): EvaAttention(
+            (qkv): Linear(in_features=1024, out_features=3072, bias=False)
+            (q_norm): Identity()
+            (k_norm): Identity()
+            (attn_drop): Dropout(p=0.0, inplace=False)
+            (norm): Identity()
+            (proj): Linear(in_features=1024, out_features=1024, bias=True)
+            (proj_drop): Dropout(p=0.0, inplace=False)
+          )
+          (drop_path1): Identity()
+          (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+          (mlp): Mlp(
+            (fc1): Linear(in_features=1024, out_features=4096, bias=True)
+            (act): GELU(approximate='none')
+            (drop1): Dropout(p=0.0, inplace=False)
+            (norm): Identity()
+            (fc2): Linear(in_features=4096, out_features=1024, bias=True)
+            (drop2): Dropout(p=0.0, inplace=False)
+          )
+          (drop_path2): Identity()
+        )
+      )
+      (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+      (fc_norm): Identity()
+      (head_drop): Dropout(p=0.0, inplace=False)
+      (head): Identity()
+      (rope): _CapiPatchRoPE()
+    )
+  )
+)
+[INFO    ][2026-05-09 08:26:33][root                ][init_video_model         ] Encoder number of parameters: 1179648
+[INFO    ][2026-05-09 08:26:33][root                ][init_video_model         ] Predictor number of parameters: 11436928
+[INFO    ][2026-05-09 08:26:33][root                ][init_video_model         ] Target encoder number of parameters: 0
+[INFO    ][2026-05-09 08:26:34][root                ][make_videodataset        ] VideoDataset dataset created
+[INFO    ][2026-05-09 08:26:34][WeightedSampler     ][__init__                 ] Using DistributedWeightedSampler with rank 12 / 16
+[INFO    ][2026-05-09 08:26:34][root                ][make_videodataset        ] VideoDataset unsupervised data loader created
+[INFO    ][2026-05-09 08:26:34][app.vjepa.train     ][main                     ] iterations per epoch/dataset length: 300/399
+[INFO    ][2026-05-09 08:26:34][app.vjepa.train     ][main                     ] Wrapping models in DDP (rank 12)...
+[INFO    ][2026-05-09 08:26:37][root                ][load_checkpoint          ] Loading checkpoint from /scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/latest.pt
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] loaded pretrained encoder from epoch 39 with msg: <All keys matched successfully>
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] loaded pretrained predictor from epoch 39 with msg: <All keys matched successfully>
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] loaded pretrained target encoder from epoch 39 with msg: <All keys matched successfully>
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] loaded optimizers from epoch 39
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] read-path: /scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/latest.pt
+[INFO    ][2026-05-09 08:26:39][app.vjepa.train     ][main                     ] Initializing loader...
+submitit WARNING (2026-05-09 09:56:00,945) - Bypassing signal SIGTERM
+[WARNING ][2026-05-09 09:56:00][submitit            ][bypass                   ] Bypassing signal SIGTERM
+submitit WARNING (2026-05-09 09:56:00,946) - Bypassing signal SIGCONT
+[WARNING ][2026-05-09 09:56:00][submitit            ][bypass                   ] Bypassing signal SIGCONT

vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/job_22595640/22595640_13_log.err

@@ -0,0 +1,305 @@
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
+  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
+[rank13]:[W508 20:22:16.189583522 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
+  scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+submitit WARNING (2026-05-09 08:16:14,377) - Caught signal SIGUSR2 on gcn84.local.snellius.surf.nl: this job is timed-out.
+submitit WARNING (2026-05-09 08:16:14,377) - Bypassing signal SIGTERM
+submitit WARNING (2026-05-09 08:16:14,377) - Bypassing signal SIGCONT
+[rank13]:[W509 08:16:18.683249600 TCPStore.cpp:125] [c10d] recvValue failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37344, remote=[gcn75.local.snellius.surf.nl]:37129): Failed to recv, got 0 bytes. Connection was likely closed. Did the remote server shutdown or crash?
+Exception raised from recvBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:682 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x150a78b9afdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a3325d (0x150abcd2925d in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x273 (0x150abcd271f3 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x150a79e09eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x150bb9273164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x150bc9ce519a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x150bc9d6a100 in /lib64/libc.so.6)
+
+[rank13]:[W509 08:16:18.781879787 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 13] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Failed to recv, got 0 bytes. Connection was likely closed. Did the remote server shutdown or crash?
+[rank13]:[W509 08:16:19.782075188 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37344, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x150a78b9afdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x150abcd286d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x150abcd271cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x150a79e09eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x150bb9273164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x150bc9ce519a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x150bc9d6a100 in /lib64/libc.so.6)
+
+[rank13]:[W509 08:16:19.785310146 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 13] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank13]:[W509 08:16:20.785457198 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37344, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x150a78b9afdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x150abcd286d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x150abcd271cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x150a79e09eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x150bb9273164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x150bc9ce519a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x150bc9d6a100 in /lib64/libc.so.6)
+
+[rank13]:[W509 08:16:20.787796452 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 13] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank13]:[W509 08:16:21.787939944 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37344, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x150a78b9afdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x150abcd286d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x150abcd271cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x150a79e09eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x150bb9273164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x150bc9ce519a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x150bc9d6a100 in /lib64/libc.so.6)
+
+[rank13]:[W509 08:16:21.790404158 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 13] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank13]:[W509 08:16:22.790556390 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37344, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x150a78b9afdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x150abcd286d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x150abcd271cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x150a79e09eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x150bb9273164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x150bc9ce519a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x150bc9d6a100 in /lib64/libc.so.6)
+
+[rank13]:[W509 08:16:22.792813204 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 13] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank13]:[W509 08:16:23.792963156 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37344, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x150a78b9afdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x150abcd286d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x150abcd271cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x150a79e09eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x150bb9273164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x150bc9ce519a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x150bc9d6a100 in /lib64/libc.so.6)
+
+[rank13]:[W509 08:16:23.795170591 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 13] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank13]:[W509 08:16:24.795335373 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37344, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x150a78b9afdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x150abcd286d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x150abcd271cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x150a79e09eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x150bb9273164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x150bc9ce519a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x150bc9d6a100 in /lib64/libc.so.6)
+
+[rank13]:[W509 08:16:24.797737967 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 13] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank13]:[W509 08:16:25.797878558 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37344, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x150a78b9afdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x150abcd286d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x150abcd271cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x150a79e09eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x150bb9273164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x150bc9ce519a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x150bc9d6a100 in /lib64/libc.so.6)
+
+[rank13]:[W509 08:16:25.800155453 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 13] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank13]:[W509 08:16:26.800289815 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37344, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x150a78b9afdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x150abcd286d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x150abcd271cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x150a79e09eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x150bb9273164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x150bc9ce519a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x150bc9d6a100 in /lib64/libc.so.6)
+
+[rank13]:[W509 08:16:26.802569660 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 13] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank13]:[W509 08:16:27.802715722 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37344, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x150a78b9afdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x150abcd286d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x150abcd271cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x150a79e09eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x150bb9273164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x150bc9ce519a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x150bc9d6a100 in /lib64/libc.so.6)
+
+[rank13]:[W509 08:16:27.805144485 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 13] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank13]:[W509 08:16:28.805278747 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37344, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x150a78b9afdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x150abcd286d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x150abcd271cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x150a79e09eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x150bb9273164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x150bc9ce519a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x150bc9d6a100 in /lib64/libc.so.6)
+
+[rank13]:[W509 08:16:28.807568152 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 13] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank13]:[W509 08:16:29.807695784 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37344, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x150a78b9afdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x150abcd286d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x150abcd271cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x150a79e09eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x150bb9273164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x150bc9ce519a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x150bc9d6a100 in /lib64/libc.so.6)
+
+[rank13]:[W509 08:16:29.810039788 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 13] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank13]:[W509 08:16:30.810176100 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37344, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x150a78b9afdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x150abcd286d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x150abcd271cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x150a79e09eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x150bb9273164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x150bc9ce519a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x150bc9d6a100 in /lib64/libc.so.6)
+
+[rank13]:[W509 08:16:30.812578953 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 13] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank13]:[W509 08:16:31.812711165 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37344, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x150a78b9afdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x150abcd286d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x150abcd271cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x150a79e09eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x150bb9273164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x150bc9ce519a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x150bc9d6a100 in /lib64/libc.so.6)
+
+[rank13]:[W509 08:16:31.814958070 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 13] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank13]:[W509 08:16:32.815092492 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37344, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x150a78b9afdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x150abcd286d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x150abcd271cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x150a79e09eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x150bb9273164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x150bc9ce519a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x150bc9d6a100 in /lib64/libc.so.6)
+
+[rank13]:[W509 08:16:32.817433486 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 13] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank13]:[W509 08:16:33.817575348 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37344, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x150a78b9afdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x150abcd286d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x150abcd271cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x150a79e09eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x150bb9273164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x150bc9ce519a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x150bc9d6a100 in /lib64/libc.so.6)
+
+[rank13]:[W509 08:16:33.820043182 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 13] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank13]:[W509 08:16:34.820180263 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37344, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x150a78b9afdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x150abcd286d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x150abcd271cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x150a79e09eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x150bb9273164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x150bc9ce519a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x150bc9d6a100 in /lib64/libc.so.6)
+
+[rank13]:[W509 08:16:34.822465908 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 13] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank13]:[W509 08:16:35.822621570 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37344, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x150a78b9afdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x150abcd286d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x150abcd271cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x150a79e09eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x150bb9273164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x150bc9ce519a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x150bc9d6a100 in /lib64/libc.so.6)
+
+[rank13]:[W509 08:16:35.824925854 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 13] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank13]:[W509 08:16:36.825076586 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37344, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x150a78b9afdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x150abcd286d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x150abcd271cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x150a79e09eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x150bb9273164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x150bc9ce519a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x150bc9d6a100 in /lib64/libc.so.6)
+
+[rank13]:[W509 08:16:36.827423640 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 13] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank13]:[W509 08:16:37.827577662 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37344, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x150a78b9afdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x150abcd286d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x150abcd271cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x150a79e09eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x150bb9273164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x150bc9ce519a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x150bc9d6a100 in /lib64/libc.so.6)
+
+[rank13]:[W509 08:16:37.829877806 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 13] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank13]:[W509 08:16:38.830048138 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37344, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x150a78b9afdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x150abcd286d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x150abcd271cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x150a79e09eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x150bb9273164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x150bc9ce519a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x150bc9d6a100 in /lib64/libc.so.6)
+
+[rank13]:[W509 08:16:38.832339383 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 13] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank13]:[W509 08:16:39.832485474 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37344, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x150a78b9afdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x150abcd286d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x150abcd271cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x150a79e09eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x150bb9273164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x150bc9ce519a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x150bc9d6a100 in /lib64/libc.so.6)
+
+[rank13]:[W509 08:16:39.834812449 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 13] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank13]:[W509 08:16:40.834953900 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37344, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x150a78b9afdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x150abcd286d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x150abcd271cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x150a79e09eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x150bb9273164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x150bc9ce519a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x150bc9d6a100 in /lib64/libc.so.6)
+
+[rank13]:[W509 08:16:40.837155886 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 13] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank13]:[W509 08:16:41.837286218 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37344, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x150a78b9afdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x150abcd286d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x150abcd271cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x150a79e09eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x150bb9273164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x150bc9ce519a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x150bc9d6a100 in /lib64/libc.so.6)
+
+[rank13]:[W509 08:16:41.839584212 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 13] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank13]:[W509 08:16:42.839714504 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37344, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x150a78b9afdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x150abcd286d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x150abcd271cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x150a79e09eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x150bb9273164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x150bc9ce519a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x150bc9d6a100 in /lib64/libc.so.6)
+
+[rank13]:[W509 08:16:42.842101048 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 13] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank13]:[W509 08:16:43.842233940 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37344, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x150a78b9afdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x150abcd286d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x150abcd271cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x150a79e09eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x150bb9273164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x150bc9ce519a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x150bc9d6a100 in /lib64/libc.so.6)
+
+[rank13]:[W509 08:16:43.844520795 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 13] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
+  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
+[rank13]:[W509 08:23:53.217710675 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
+  scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+submitit WARNING (2026-05-09 09:56:00,939) - Bypassing signal SIGTERM
+submitit WARNING (2026-05-09 09:56:00,940) - Bypassing signal SIGCONT

vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/job_22595640/22595640_13_log.out

@@ -0,0 +1,530 @@
+submitit INFO (2026-05-08 20:18:11,518) - Starting with JobEnvironment(job_id=22595640, hostname=gcn84.local.snellius.surf.nl, local_rank=1(4), node=3(4), global_rank=13(16))
+submitit INFO (2026-05-08 20:18:11,518) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/job_22595640/22595640_submitted.pkl
+INFO:root:loaded pretrain params...
+{   'app': 'vjepa',
+    'cpus_per_task': 16,
+    'data': {   'batch_size': 64,
+                'crop_size': 224,
+                'dataset_fpcs': [16, 16],
+                'dataset_type': 'VideoDataset',
+                'datasets': [   '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
+                                '/scratch-shared/dcanez/data/ssv2/train.csv'],
+                'datasets_weights': [0.65, 0.35],
+                'fps': 4,
+                'num_workers': 10,
+                'patch_size': 14,
+                'persistent_workers': True,
+                'pin_mem': True,
+                'stage': [   {   'dest': 'kinetics_240',
+                                 'format': 'targz_parts',
+                                 'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
+                             {   'dest': 'ssv2',
+                                 'format': 'multipart_tar',
+                                 'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
+                'tubelet_size': 1},
+    'data_aug': {   'auto_augment': False,
+                    'motion_shift': False,
+                    'random_resize_aspect_ratio': [0.75, 1.35],
+                    'random_resize_scale': [0.3, 1.0],
+                    'reprob': 0.0},
+    'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m',
+    'loss': {'loss_exp': 1.0},
+    'mask': [   {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 8,
+                    'spatial_scale': [0.15, 0.15],
+                    'temporal_scale': [1.0, 1.0]},
+                {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 2,
+                    'spatial_scale': [0.7, 0.7],
+                    'temporal_scale': [1.0, 1.0]}],
+    'mem_per_gpu': '180G',
+    'meta': {   'dtype': 'bfloat16',
+                'knn_eval_epoch0': False,
+                'knn_eval_freq': 5,
+                'knn_eval_presets': [   {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
+                                                          'eval_videos_per_class': 25,
+                                                          'num_workers': 8,
+                                                          'train_videos_per_class': 100},
+                                            'preset': 'ucf101'},
+                                        {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
+                                                          'eval_videos_per_class': 100,
+                                                          'linear_probe': True,
+                                                          'num_workers': 8,
+                                                          'pool_type': 'temporal_concat',
+                                                          'train_videos_per_class': 500},
+                                            'preset': 'ssv2_coarse10'}],
+                'load_checkpoint': True,
+                'read_checkpoint': None,
+                'save_every_freq': 5,
+                'seed': 239,
+                'use_sdpa': True,
+                'use_wandb': True,
+                'wandb_project': 'vjepa_ablation'},
+    'metrics': {'sigreg': {}, 'std': {}},
+    'model': {   'class_token': False,
+                 'freeze_backbone': True,
+                 'is_causal': False,
+                 'lora_alpha': 16,
+                 'lora_rank': 8,
+                 'lora_targets': ['attn.qkv', 'attn.proj'],
+                 'model_name': 'vit_large_patch14_capi_lvd1689m',
+                 'num_reg_tokens': 16,
+                 'pred_depth': 6,
+                 'pred_embed_dim': 384,
+                 'pred_num_heads': 12,
+                 'rope_type': 'capi2d',
+                 'stem_type': '2d',
+                 'target_kind': 'frozen_2d',
+                 'target_type': 'vit_large_patch14_capi.lvd1689m',
+                 'uniform_power': True,
+                 'use_activation_checkpointing': True,
+                 'use_lora': True,
+                 'use_mask_tokens': True,
+                 'use_rope': True,
+                 'use_sdpa': True,
+                 'zero_init_mask_tokens': True},
+    'nodes': 4,
+    'optimization': {   'ema': [0.99925, 0.99925],
+                        'epochs': 100,
+                        'final_lr': 0.0001,
+                        'final_weight_decay': 0.04,
+                        'ipe': 300,
+                        'ipe_scale': 1.0,
+                        'lr': 0.0005,
+                        'start_lr': 0.0001,
+                        'warmup': 10,
+                        'weight_decay': 0.04},
+    'tasks_per_node': 4}
+INFO:root:Running pre-training of app: vjepa
+[INFO    ][2026-05-08 20:19:11][app.vjepa.train     ][main                     ] which_dtype='bfloat16'
+[INFO    ][2026-05-08 20:19:11][app.vjepa.train     ][main                     ] Disabling persistent_workers (incompatible with KNN eval)
+[INFO    ][2026-05-08 20:19:11][app.vjepa.train     ][main                     ] NCCL_SOCKET_IFNAME=eno
+[INFO    ][2026-05-08 20:19:15][app.vjepa.train     ][main                     ] Initialized (rank/world-size) 13/16, tasks_per_node=4
+[INFO    ][2026-05-08 20:19:15][root                ][stage_datasets           ] [local_rank 1/4] Staging kinetics_240 (targz_parts)
+[INFO    ][2026-05-08 20:19:15][root                ][_stage_targz_parts       ] [rank 1] Extracting 26/103 tar.gz parts to /scratch-node/dcanez.22595640/kinetics_240
+[INFO    ][2026-05-08 20:19:28][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 2/26 parts
+[INFO    ][2026-05-08 20:19:41][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 4/26 parts
+[INFO    ][2026-05-08 20:19:55][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 6/26 parts
+[INFO    ][2026-05-08 20:20:09][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 8/26 parts
+[INFO    ][2026-05-08 20:20:24][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 10/26 parts
+[INFO    ][2026-05-08 20:20:38][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 12/26 parts
+[INFO    ][2026-05-08 20:20:51][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 14/26 parts
+[INFO    ][2026-05-08 20:21:04][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 16/26 parts
+[INFO    ][2026-05-08 20:21:19][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 18/26 parts
+[INFO    ][2026-05-08 20:21:33][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 20/26 parts
+[INFO    ][2026-05-08 20:21:48][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 22/26 parts
+[INFO    ][2026-05-08 20:22:02][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 24/26 parts
+[INFO    ][2026-05-08 20:22:16][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 26/26 parts
+[INFO    ][2026-05-08 20:22:16][root                ][stage_datasets           ] [local_rank 1/4] Staging ssv2 (multipart_tar)
+[INFO    ][2026-05-08 20:23:17][app.vjepa.train     ][main                     ] Staged datasets: ['/scratch-node/dcanez.22595640/kinetics_240/train.csv', '/scratch-node/dcanez.22595640/ssv2/train.csv']
+[INFO    ][2026-05-08 20:24:23][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
+[INFO    ][2026-05-08 20:24:25][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA applied: 48 modules replaced (rank=8, alpha=16, dropout=0.0, targets=['attn.qkv', 'attn.proj'], scope=blocks)
+[INFO    ][2026-05-08 20:24:25][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA trainable params: 1,179,648
+[INFO    ][2026-05-08 20:24:28][root                ][_build_frozen_2d_target  ] Loaded frozen_2d target from timm: vit_large_patch14_capi.lvd1689m
+[INFO    ][2026-05-08 20:24:28][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-08 20:24:28][root                ][init_video_model         ] VJEPAPredictorMultiSeqWrapper(
+  (backbone): VisionTransformerPredictor(
+    (predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
+    (mask_tokens): ParameterList(
+        (0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+    )
+    (predictor_blocks): ModuleList(
+      (0-5): 6 x Block(
+        (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (attn): RoPEAttention(
+          (qkv): Linear(in_features=384, out_features=1152, bias=True)
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): Linear(in_features=384, out_features=384, bias=True)
+          (proj_drop): Dropout(p=0.0, inplace=False)
+        )
+        (drop_path): Identity()
+        (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (mlp): MLP(
+          (fc1): Linear(in_features=384, out_features=1536, bias=True)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=1536, out_features=384, bias=True)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+    (predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
+  )
+)
+[INFO    ][2026-05-08 20:24:28][root                ][init_video_model         ] MultiSeqWrapper(
+  (backbone): Frozen2DTargetWrapper(
+    (backbone): Eva(
+      (patch_embed): PatchEmbed(
+        (proj): Conv2d(3, 1024, kernel_size=(14, 14), stride=(14, 14))
+        (norm): Identity()
+      )
+      (pos_drop): Dropout(p=0.0, inplace=False)
+      (norm_pre): Identity()
+      (blocks): ModuleList(
+        (0-23): 24 x EvaBlock(
+          (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+          (attn): EvaAttention(
+            (qkv): Linear(in_features=1024, out_features=3072, bias=False)
+            (q_norm): Identity()
+            (k_norm): Identity()
+            (attn_drop): Dropout(p=0.0, inplace=False)
+            (norm): Identity()
+            (proj): Linear(in_features=1024, out_features=1024, bias=True)
+            (proj_drop): Dropout(p=0.0, inplace=False)
+          )
+          (drop_path1): Identity()
+          (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+          (mlp): Mlp(
+            (fc1): Linear(in_features=1024, out_features=4096, bias=True)
+            (act): GELU(approximate='none')
+            (drop1): Dropout(p=0.0, inplace=False)
+            (norm): Identity()
+            (fc2): Linear(in_features=4096, out_features=1024, bias=True)
+            (drop2): Dropout(p=0.0, inplace=False)
+          )
+          (drop_path2): Identity()
+        )
+      )
+      (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+      (fc_norm): Identity()
+      (head_drop): Dropout(p=0.0, inplace=False)
+      (head): Identity()
+      (rope): _CapiPatchRoPE()
+    )
+  )
+)
+[INFO    ][2026-05-08 20:24:28][root                ][init_video_model         ] Encoder number of parameters: 1179648
+[INFO    ][2026-05-08 20:24:28][root                ][init_video_model         ] Predictor number of parameters: 11436928
+[INFO    ][2026-05-08 20:24:28][root                ][init_video_model         ] Target encoder number of parameters: 0
+[INFO    ][2026-05-08 20:24:29][root                ][make_videodataset        ] VideoDataset dataset created
+[INFO    ][2026-05-08 20:24:29][WeightedSampler     ][__init__                 ] Using DistributedWeightedSampler with rank 13 / 16
+[INFO    ][2026-05-08 20:24:29][root                ][make_videodataset        ] VideoDataset unsupervised data loader created
+[INFO    ][2026-05-08 20:24:29][app.vjepa.train     ][main                     ] iterations per epoch/dataset length: 300/399
+[INFO    ][2026-05-08 20:24:29][app.vjepa.train     ][main                     ] Wrapping models in DDP (rank 13)...
+[INFO    ][2026-05-08 20:24:41][app.vjepa.train     ][main                     ] Initializing loader...
+submitit INFO (2026-05-09 08:16:14,356) - Job has timed out. Ran 718 minutes out of requested 720 minutes.
+[INFO    ][2026-05-09 08:16:14][submitit            ][has_timed_out            ] Job has timed out. Ran 718 minutes out of requested 720 minutes.
+submitit WARNING (2026-05-09 08:16:14,377) - Caught signal SIGUSR2 on gcn84.local.snellius.surf.nl: this job is timed-out.
+[WARNING ][2026-05-09 08:16:14][submitit            ][checkpoint_and_try_requeue] Caught signal SIGUSR2 on gcn84.local.snellius.surf.nl: this job is timed-out.
+submitit INFO (2026-05-09 08:16:14,377) - Not checkpointing nor requeuing since I am a slave (procid=13).
+[INFO    ][2026-05-09 08:16:14][submitit            ][checkpoint_and_try_requeue] Not checkpointing nor requeuing since I am a slave (procid=13).
+submitit WARNING (2026-05-09 08:16:14,377) - Bypassing signal SIGTERM
+[WARNING ][2026-05-09 08:16:14][submitit            ][bypass                   ] Bypassing signal SIGTERM
+submitit WARNING (2026-05-09 08:16:14,377) - Bypassing signal SIGCONT
+[WARNING ][2026-05-09 08:16:14][submitit            ][bypass                   ] Bypassing signal SIGCONT
+submitit INFO (2026-05-09 08:19:47,092) - Starting with JobEnvironment(job_id=22595640, hostname=gcn78.local.snellius.surf.nl, local_rank=1(4), node=3(4), global_rank=13(16))
+submitit INFO (2026-05-09 08:19:49,101) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/job_22595640/22595640_submitted.pkl
+INFO:root:loaded pretrain params...
+{   'app': 'vjepa',
+    'cpus_per_task': 16,
+    'data': {   'batch_size': 64,
+                'crop_size': 224,
+                'dataset_fpcs': [16, 16],
+                'dataset_type': 'VideoDataset',
+                'datasets': [   '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
+                                '/scratch-shared/dcanez/data/ssv2/train.csv'],
+                'datasets_weights': [0.65, 0.35],
+                'fps': 4,
+                'num_workers': 10,
+                'patch_size': 14,
+                'persistent_workers': True,
+                'pin_mem': True,
+                'stage': [   {   'dest': 'kinetics_240',
+                                 'format': 'targz_parts',
+                                 'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
+                             {   'dest': 'ssv2',
+                                 'format': 'multipart_tar',
+                                 'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
+                'tubelet_size': 1},
+    'data_aug': {   'auto_augment': False,
+                    'motion_shift': False,
+                    'random_resize_aspect_ratio': [0.75, 1.35],
+                    'random_resize_scale': [0.3, 1.0],
+                    'reprob': 0.0},
+    'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m',
+    'loss': {'loss_exp': 1.0},
+    'mask': [   {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 8,
+                    'spatial_scale': [0.15, 0.15],
+                    'temporal_scale': [1.0, 1.0]},
+                {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 2,
+                    'spatial_scale': [0.7, 0.7],
+                    'temporal_scale': [1.0, 1.0]}],
+    'mem_per_gpu': '180G',
+    'meta': {   'dtype': 'bfloat16',
+                'knn_eval_epoch0': False,
+                'knn_eval_freq': 5,
+                'knn_eval_presets': [   {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
+                                                          'eval_videos_per_class': 25,
+                                                          'num_workers': 8,
+                                                          'train_videos_per_class': 100},
+                                            'preset': 'ucf101'},
+                                        {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
+                                                          'eval_videos_per_class': 100,
+                                                          'linear_probe': True,
+                                                          'num_workers': 8,
+                                                          'pool_type': 'temporal_concat',
+                                                          'train_videos_per_class': 500},
+                                            'preset': 'ssv2_coarse10'}],
+                'load_checkpoint': True,
+                'read_checkpoint': None,
+                'save_every_freq': 5,
+                'seed': 239,
+                'use_sdpa': True,
+                'use_wandb': True,
+                'wandb_project': 'vjepa_ablation'},
+    'metrics': {'sigreg': {}, 'std': {}},
+    'model': {   'class_token': False,
+                 'freeze_backbone': True,
+                 'is_causal': False,
+                 'lora_alpha': 16,
+                 'lora_rank': 8,
+                 'lora_targets': ['attn.qkv', 'attn.proj'],
+                 'model_name': 'vit_large_patch14_capi_lvd1689m',
+                 'num_reg_tokens': 16,
+                 'pred_depth': 6,
+                 'pred_embed_dim': 384,
+                 'pred_num_heads': 12,
+                 'rope_type': 'capi2d',
+                 'stem_type': '2d',
+                 'target_kind': 'frozen_2d',
+                 'target_type': 'vit_large_patch14_capi.lvd1689m',
+                 'uniform_power': True,
+                 'use_activation_checkpointing': True,
+                 'use_lora': True,
+                 'use_mask_tokens': True,
+                 'use_rope': True,
+                 'use_sdpa': True,
+                 'zero_init_mask_tokens': True},
+    'nodes': 4,
+    'optimization': {   'ema': [0.99925, 0.99925],
+                        'epochs': 100,
+                        'final_lr': 0.0001,
+                        'final_weight_decay': 0.04,
+                        'ipe': 300,
+                        'ipe_scale': 1.0,
+                        'lr': 0.0005,
+                        'start_lr': 0.0001,
+                        'warmup': 10,
+                        'weight_decay': 0.04},
+    'tasks_per_node': 4}
+INFO:root:Running pre-training of app: vjepa
+[INFO    ][2026-05-09 08:20:49][app.vjepa.train     ][main                     ] which_dtype='bfloat16'
+[INFO    ][2026-05-09 08:20:49][app.vjepa.train     ][main                     ] Disabling persistent_workers (incompatible with KNN eval)
+[INFO    ][2026-05-09 08:20:49][app.vjepa.train     ][main                     ] NCCL_SOCKET_IFNAME=eno
+[INFO    ][2026-05-09 08:20:52][app.vjepa.train     ][main                     ] Initialized (rank/world-size) 13/16, tasks_per_node=4
+[INFO    ][2026-05-09 08:20:52][root                ][stage_datasets           ] [local_rank 1/4] Staging kinetics_240 (targz_parts)
+[INFO    ][2026-05-09 08:20:52][root                ][_stage_targz_parts       ] [rank 1] Extracting 26/103 tar.gz parts to /scratch-node/dcanez.22595640/kinetics_240
+[INFO    ][2026-05-09 08:21:05][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 2/26 parts
+[INFO    ][2026-05-09 08:21:18][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 4/26 parts
+[INFO    ][2026-05-09 08:21:32][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 6/26 parts
+[INFO    ][2026-05-09 08:21:46][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 8/26 parts
+[INFO    ][2026-05-09 08:22:00][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 10/26 parts
+[INFO    ][2026-05-09 08:22:14][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 12/26 parts
+[INFO    ][2026-05-09 08:22:27][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 14/26 parts
+[INFO    ][2026-05-09 08:22:41][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 16/26 parts
+[INFO    ][2026-05-09 08:22:56][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 18/26 parts
+[INFO    ][2026-05-09 08:23:10][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 20/26 parts
+[INFO    ][2026-05-09 08:23:24][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 22/26 parts
+[INFO    ][2026-05-09 08:23:38][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 24/26 parts
+[INFO    ][2026-05-09 08:23:52][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 26/26 parts
+[INFO    ][2026-05-09 08:23:52][root                ][stage_datasets           ] [local_rank 1/4] Staging ssv2 (multipart_tar)
+[INFO    ][2026-05-09 08:25:33][app.vjepa.train     ][main                     ] Staged datasets: ['/scratch-node/dcanez.22595640/kinetics_240/train.csv', '/scratch-node/dcanez.22595640/ssv2/train.csv']
+[INFO    ][2026-05-09 08:26:29][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
+[INFO    ][2026-05-09 08:26:30][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA applied: 48 modules replaced (rank=8, alpha=16, dropout=0.0, targets=['attn.qkv', 'attn.proj'], scope=blocks)
+[INFO    ][2026-05-09 08:26:30][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA trainable params: 1,179,648
+[INFO    ][2026-05-09 08:26:33][root                ][_build_frozen_2d_target  ] Loaded frozen_2d target from timm: vit_large_patch14_capi.lvd1689m
+[INFO    ][2026-05-09 08:26:33][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-09 08:26:33][root                ][init_video_model         ] VJEPAPredictorMultiSeqWrapper(
+  (backbone): VisionTransformerPredictor(
+    (predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
+    (mask_tokens): ParameterList(
+        (0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+    )
+    (predictor_blocks): ModuleList(
+      (0-5): 6 x Block(
+        (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (attn): RoPEAttention(
+          (qkv): Linear(in_features=384, out_features=1152, bias=True)
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): Linear(in_features=384, out_features=384, bias=True)
+          (proj_drop): Dropout(p=0.0, inplace=False)
+        )
+        (drop_path): Identity()
+        (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (mlp): MLP(
+          (fc1): Linear(in_features=384, out_features=1536, bias=True)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=1536, out_features=384, bias=True)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+    (predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
+  )
+)
+[INFO    ][2026-05-09 08:26:33][root                ][init_video_model         ] MultiSeqWrapper(
+  (backbone): Frozen2DTargetWrapper(
+    (backbone): Eva(
+      (patch_embed): PatchEmbed(
+        (proj): Conv2d(3, 1024, kernel_size=(14, 14), stride=(14, 14))
+        (norm): Identity()
+      )
+      (pos_drop): Dropout(p=0.0, inplace=False)
+      (norm_pre): Identity()
+      (blocks): ModuleList(
+        (0-23): 24 x EvaBlock(
+          (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+          (attn): EvaAttention(
+            (qkv): Linear(in_features=1024, out_features=3072, bias=False)
+            (q_norm): Identity()
+            (k_norm): Identity()
+            (attn_drop): Dropout(p=0.0, inplace=False)
+            (norm): Identity()
+            (proj): Linear(in_features=1024, out_features=1024, bias=True)
+            (proj_drop): Dropout(p=0.0, inplace=False)
+          )
+          (drop_path1): Identity()
+          (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+          (mlp): Mlp(
+            (fc1): Linear(in_features=1024, out_features=4096, bias=True)
+            (act): GELU(approximate='none')
+            (drop1): Dropout(p=0.0, inplace=False)
+            (norm): Identity()
+            (fc2): Linear(in_features=4096, out_features=1024, bias=True)
+            (drop2): Dropout(p=0.0, inplace=False)
+          )
+          (drop_path2): Identity()
+        )
+      )
+      (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+      (fc_norm): Identity()
+      (head_drop): Dropout(p=0.0, inplace=False)
+      (head): Identity()
+      (rope): _CapiPatchRoPE()
+    )
+  )
+)
+[INFO    ][2026-05-09 08:26:33][root                ][init_video_model         ] Encoder number of parameters: 1179648
+[INFO    ][2026-05-09 08:26:33][root                ][init_video_model         ] Predictor number of parameters: 11436928
+[INFO    ][2026-05-09 08:26:33][root                ][init_video_model         ] Target encoder number of parameters: 0
+[INFO    ][2026-05-09 08:26:34][root                ][make_videodataset        ] VideoDataset dataset created
+[INFO    ][2026-05-09 08:26:34][WeightedSampler     ][__init__                 ] Using DistributedWeightedSampler with rank 13 / 16
+[INFO    ][2026-05-09 08:26:34][root                ][make_videodataset        ] VideoDataset unsupervised data loader created
+[INFO    ][2026-05-09 08:26:34][app.vjepa.train     ][main                     ] iterations per epoch/dataset length: 300/399
+[INFO    ][2026-05-09 08:26:35][app.vjepa.train     ][main                     ] Wrapping models in DDP (rank 13)...
+[INFO    ][2026-05-09 08:26:37][root                ][load_checkpoint          ] Loading checkpoint from /scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/latest.pt
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] loaded pretrained encoder from epoch 39 with msg: <All keys matched successfully>
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] loaded pretrained predictor from epoch 39 with msg: <All keys matched successfully>
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] loaded pretrained target encoder from epoch 39 with msg: <All keys matched successfully>
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] loaded optimizers from epoch 39
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] read-path: /scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/latest.pt
+[INFO    ][2026-05-09 08:26:39][app.vjepa.train     ][main                     ] Initializing loader...
+submitit WARNING (2026-05-09 09:56:00,939) - Bypassing signal SIGTERM
+[WARNING ][2026-05-09 09:56:00][submitit            ][bypass                   ] Bypassing signal SIGTERM
+submitit WARNING (2026-05-09 09:56:00,940) - Bypassing signal SIGCONT
+[WARNING ][2026-05-09 09:56:00][submitit            ][bypass                   ] Bypassing signal SIGCONT

vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/job_22595640/22595640_14_log.err

@@ -0,0 +1,305 @@
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
+  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
+[rank14]:[W508 20:22:19.933161198 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
+  scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+submitit WARNING (2026-05-09 08:16:14,440) - Caught signal SIGUSR2 on gcn84.local.snellius.surf.nl: this job is timed-out.
+submitit WARNING (2026-05-09 08:16:14,441) - Bypassing signal SIGTERM
+submitit WARNING (2026-05-09 08:16:14,441) - Bypassing signal SIGCONT
+[rank14]:[W509 08:16:18.683251890 TCPStore.cpp:125] [c10d] recvValue failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37342, remote=[gcn75.local.snellius.surf.nl]:37129): Failed to recv, got 0 bytes. Connection was likely closed. Did the remote server shutdown or crash?
+Exception raised from recvBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:682 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14adef2d8fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a3325d (0x14ae3346725d in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x273 (0x14ae334651f3 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14adf0547eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14af2f9b1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14af4042319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14af404a8100 in /lib64/libc.so.6)
+
+[rank14]:[W509 08:16:18.781874247 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 14] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Failed to recv, got 0 bytes. Connection was likely closed. Did the remote server shutdown or crash?
+[rank14]:[W509 08:16:19.782054768 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37342, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14adef2d8fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14ae334666d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14ae334651cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14adf0547eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14af2f9b1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14af4042319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14af404a8100 in /lib64/libc.so.6)
+
+[rank14]:[W509 08:16:19.785104608 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 14] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank14]:[W509 08:16:20.785239630 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37342, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14adef2d8fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14ae334666d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14ae334651cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14adf0547eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14af2f9b1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14af4042319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14af404a8100 in /lib64/libc.so.6)
+
+[rank14]:[W509 08:16:20.787484355 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 14] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank14]:[W509 08:16:21.787605417 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37342, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14adef2d8fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14ae334666d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14ae334651cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14adf0547eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14af2f9b1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14af4042319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14af404a8100 in /lib64/libc.so.6)
+
+[rank14]:[W509 08:16:21.789883901 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 14] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank14]:[W509 08:16:22.790047513 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37342, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14adef2d8fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14ae334666d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14ae334651cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14adf0547eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14af2f9b1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14af4042319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14af404a8100 in /lib64/libc.so.6)
+
+[rank14]:[W509 08:16:22.792441817 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 14] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank14]:[W509 08:16:23.792579179 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37342, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14adef2d8fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14ae334666d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14ae334651cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14adf0547eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14af2f9b1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14af4042319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14af404a8100 in /lib64/libc.so.6)
+
+[rank14]:[W509 08:16:23.794873193 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 14] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank14]:[W509 08:16:24.795044205 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37342, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14adef2d8fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14ae334666d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14ae334651cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14adf0547eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14af2f9b1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14af4042319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14af404a8100 in /lib64/libc.so.6)
+
+[rank14]:[W509 08:16:24.797376889 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 14] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank14]:[W509 08:16:25.797511301 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37342, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14adef2d8fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14ae334666d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14ae334651cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14adf0547eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14af2f9b1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14af4042319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14af404a8100 in /lib64/libc.so.6)
+
+[rank14]:[W509 08:16:25.799840325 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 14] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank14]:[W509 08:16:26.799977067 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37342, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14adef2d8fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14ae334666d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14ae334651cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14adf0547eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14af2f9b1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14af4042319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14af404a8100 in /lib64/libc.so.6)
+
+[rank14]:[W509 08:16:26.802266842 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 14] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank14]:[W509 08:16:27.802413684 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37342, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14adef2d8fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14ae334666d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14ae334651cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14adf0547eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14af2f9b1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14af4042319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14af404a8100 in /lib64/libc.so.6)
+
+[rank14]:[W509 08:16:27.804755598 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 14] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank14]:[W509 08:16:28.804887250 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37342, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14adef2d8fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14ae334666d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14ae334651cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14adf0547eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14af2f9b1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14af4042319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14af404a8100 in /lib64/libc.so.6)
+
+[rank14]:[W509 08:16:28.807235854 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 14] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank14]:[W509 08:16:29.807363916 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37342, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14adef2d8fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14ae334666d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14ae334651cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14adf0547eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14af2f9b1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14af4042319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14af404a8100 in /lib64/libc.so.6)
+
+[rank14]:[W509 08:16:29.809658150 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 14] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank14]:[W509 08:16:30.809787942 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37342, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14adef2d8fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14ae334666d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14ae334651cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14adf0547eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14af2f9b1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14af4042319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14af404a8100 in /lib64/libc.so.6)
+
+[rank14]:[W509 08:16:30.812048937 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 14] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank14]:[W509 08:16:31.812180919 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37342, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14adef2d8fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14ae334666d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14ae334651cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14adf0547eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14af2f9b1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14af4042319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14af404a8100 in /lib64/libc.so.6)
+
+[rank14]:[W509 08:16:31.814582433 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 14] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank14]:[W509 08:16:32.814710915 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37342, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14adef2d8fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14ae334666d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14ae334651cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14adf0547eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14af2f9b1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14af4042319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14af404a8100 in /lib64/libc.so.6)
+
+[rank14]:[W509 08:16:32.816992269 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 14] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank14]:[W509 08:16:33.817129691 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37342, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14adef2d8fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14ae334666d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14ae334651cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14adf0547eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14af2f9b1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14af4042319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14af404a8100 in /lib64/libc.so.6)
+
+[rank14]:[W509 08:16:33.819463936 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 14] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank14]:[W509 08:16:34.819595097 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37342, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14adef2d8fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14ae334666d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14ae334651cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14adf0547eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14af2f9b1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14af4042319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14af404a8100 in /lib64/libc.so.6)
+
+[rank14]:[W509 08:16:34.821963051 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 14] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank14]:[W509 08:16:35.822114243 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37342, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14adef2d8fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14ae334666d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14ae334651cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14adf0547eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14af2f9b1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14af4042319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14af404a8100 in /lib64/libc.so.6)
+
+[rank14]:[W509 08:16:35.824432917 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 14] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank14]:[W509 08:16:36.824569809 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37342, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14adef2d8fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14ae334666d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14ae334651cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14adf0547eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14af2f9b1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14af4042319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14af404a8100 in /lib64/libc.so.6)
+
+[rank14]:[W509 08:16:36.826861074 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 14] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank14]:[W509 08:16:37.826997266 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37342, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14adef2d8fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14ae334666d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14ae334651cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14adf0547eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14af2f9b1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14af4042319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14af404a8100 in /lib64/libc.so.6)
+
+[rank14]:[W509 08:16:37.829381950 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 14] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank14]:[W509 08:16:38.829518912 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37342, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14adef2d8fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14ae334666d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14ae334651cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14adf0547eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14af2f9b1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14af4042319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14af404a8100 in /lib64/libc.so.6)
+
+[rank14]:[W509 08:16:38.831734817 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 14] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank14]:[W509 08:16:39.831870318 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37342, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14adef2d8fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14ae334666d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14ae334651cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14adf0547eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14af2f9b1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14af4042319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14af404a8100 in /lib64/libc.so.6)
+
+[rank14]:[W509 08:16:39.834140433 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 14] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank14]:[W509 08:16:40.834277435 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37342, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14adef2d8fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14ae334666d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14ae334651cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14adf0547eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14af2f9b1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14af4042319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14af404a8100 in /lib64/libc.so.6)
+
+[rank14]:[W509 08:16:40.836660399 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 14] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank14]:[W509 08:16:41.836795121 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37342, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14adef2d8fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14ae334666d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14ae334651cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14adf0547eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14af2f9b1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14af4042319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14af404a8100 in /lib64/libc.so.6)
+
+[rank14]:[W509 08:16:41.839084645 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 14] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank14]:[W509 08:16:42.839220367 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37342, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14adef2d8fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14ae334666d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14ae334651cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14adf0547eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14af2f9b1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14af4042319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14af404a8100 in /lib64/libc.so.6)
+
+[rank14]:[W509 08:16:42.841502542 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 14] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank14]:[W509 08:16:43.841651244 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37342, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14adef2d8fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14ae334666d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14ae334651cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14adf0547eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14af2f9b1164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14af4042319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14af404a8100 in /lib64/libc.so.6)
+
+[rank14]:[W509 08:16:43.844035488 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 14] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
+  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
+[rank14]:[W509 08:23:56.013909411 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
+  scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+submitit WARNING (2026-05-09 09:56:00,848) - Bypassing signal SIGTERM
+submitit WARNING (2026-05-09 09:56:00,856) - Bypassing signal SIGCONT

vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/job_22595640/22595640_14_log.out

@@ -0,0 +1,530 @@
+submitit INFO (2026-05-08 20:18:11,518) - Starting with JobEnvironment(job_id=22595640, hostname=gcn84.local.snellius.surf.nl, local_rank=2(4), node=3(4), global_rank=14(16))
+submitit INFO (2026-05-08 20:18:11,518) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/job_22595640/22595640_submitted.pkl
+INFO:root:loaded pretrain params...
+{   'app': 'vjepa',
+    'cpus_per_task': 16,
+    'data': {   'batch_size': 64,
+                'crop_size': 224,
+                'dataset_fpcs': [16, 16],
+                'dataset_type': 'VideoDataset',
+                'datasets': [   '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
+                                '/scratch-shared/dcanez/data/ssv2/train.csv'],
+                'datasets_weights': [0.65, 0.35],
+                'fps': 4,
+                'num_workers': 10,
+                'patch_size': 14,
+                'persistent_workers': True,
+                'pin_mem': True,
+                'stage': [   {   'dest': 'kinetics_240',
+                                 'format': 'targz_parts',
+                                 'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
+                             {   'dest': 'ssv2',
+                                 'format': 'multipart_tar',
+                                 'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
+                'tubelet_size': 1},
+    'data_aug': {   'auto_augment': False,
+                    'motion_shift': False,
+                    'random_resize_aspect_ratio': [0.75, 1.35],
+                    'random_resize_scale': [0.3, 1.0],
+                    'reprob': 0.0},
+    'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m',
+    'loss': {'loss_exp': 1.0},
+    'mask': [   {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 8,
+                    'spatial_scale': [0.15, 0.15],
+                    'temporal_scale': [1.0, 1.0]},
+                {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 2,
+                    'spatial_scale': [0.7, 0.7],
+                    'temporal_scale': [1.0, 1.0]}],
+    'mem_per_gpu': '180G',
+    'meta': {   'dtype': 'bfloat16',
+                'knn_eval_epoch0': False,
+                'knn_eval_freq': 5,
+                'knn_eval_presets': [   {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
+                                                          'eval_videos_per_class': 25,
+                                                          'num_workers': 8,
+                                                          'train_videos_per_class': 100},
+                                            'preset': 'ucf101'},
+                                        {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
+                                                          'eval_videos_per_class': 100,
+                                                          'linear_probe': True,
+                                                          'num_workers': 8,
+                                                          'pool_type': 'temporal_concat',
+                                                          'train_videos_per_class': 500},
+                                            'preset': 'ssv2_coarse10'}],
+                'load_checkpoint': True,
+                'read_checkpoint': None,
+                'save_every_freq': 5,
+                'seed': 239,
+                'use_sdpa': True,
+                'use_wandb': True,
+                'wandb_project': 'vjepa_ablation'},
+    'metrics': {'sigreg': {}, 'std': {}},
+    'model': {   'class_token': False,
+                 'freeze_backbone': True,
+                 'is_causal': False,
+                 'lora_alpha': 16,
+                 'lora_rank': 8,
+                 'lora_targets': ['attn.qkv', 'attn.proj'],
+                 'model_name': 'vit_large_patch14_capi_lvd1689m',
+                 'num_reg_tokens': 16,
+                 'pred_depth': 6,
+                 'pred_embed_dim': 384,
+                 'pred_num_heads': 12,
+                 'rope_type': 'capi2d',
+                 'stem_type': '2d',
+                 'target_kind': 'frozen_2d',
+                 'target_type': 'vit_large_patch14_capi.lvd1689m',
+                 'uniform_power': True,
+                 'use_activation_checkpointing': True,
+                 'use_lora': True,
+                 'use_mask_tokens': True,
+                 'use_rope': True,
+                 'use_sdpa': True,
+                 'zero_init_mask_tokens': True},
+    'nodes': 4,
+    'optimization': {   'ema': [0.99925, 0.99925],
+                        'epochs': 100,
+                        'final_lr': 0.0001,
+                        'final_weight_decay': 0.04,
+                        'ipe': 300,
+                        'ipe_scale': 1.0,
+                        'lr': 0.0005,
+                        'start_lr': 0.0001,
+                        'warmup': 10,
+                        'weight_decay': 0.04},
+    'tasks_per_node': 4}
+INFO:root:Running pre-training of app: vjepa
+[INFO    ][2026-05-08 20:19:11][app.vjepa.train     ][main                     ] which_dtype='bfloat16'
+[INFO    ][2026-05-08 20:19:11][app.vjepa.train     ][main                     ] Disabling persistent_workers (incompatible with KNN eval)
+[INFO    ][2026-05-08 20:19:11][app.vjepa.train     ][main                     ] NCCL_SOCKET_IFNAME=eno
+[INFO    ][2026-05-08 20:19:14][app.vjepa.train     ][main                     ] Initialized (rank/world-size) 14/16, tasks_per_node=4
+[INFO    ][2026-05-08 20:19:14][root                ][stage_datasets           ] [local_rank 2/4] Staging kinetics_240 (targz_parts)
+[INFO    ][2026-05-08 20:19:14][root                ][_stage_targz_parts       ] [rank 2] Extracting 26/103 tar.gz parts to /scratch-node/dcanez.22595640/kinetics_240
+[INFO    ][2026-05-08 20:19:28][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 2/26 parts
+[INFO    ][2026-05-08 20:19:41][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 4/26 parts
+[INFO    ][2026-05-08 20:19:56][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 6/26 parts
+[INFO    ][2026-05-08 20:20:11][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 8/26 parts
+[INFO    ][2026-05-08 20:20:25][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 10/26 parts
+[INFO    ][2026-05-08 20:20:39][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 12/26 parts
+[INFO    ][2026-05-08 20:20:53][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 14/26 parts
+[INFO    ][2026-05-08 20:21:07][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 16/26 parts
+[INFO    ][2026-05-08 20:21:22][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 18/26 parts
+[INFO    ][2026-05-08 20:21:36][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 20/26 parts
+[INFO    ][2026-05-08 20:21:50][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 22/26 parts
+[INFO    ][2026-05-08 20:22:04][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 24/26 parts
+[INFO    ][2026-05-08 20:22:19][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 26/26 parts
+[INFO    ][2026-05-08 20:22:19][root                ][stage_datasets           ] [local_rank 2/4] Staging ssv2 (multipart_tar)
+[INFO    ][2026-05-08 20:23:17][app.vjepa.train     ][main                     ] Staged datasets: ['/scratch-node/dcanez.22595640/kinetics_240/train.csv', '/scratch-node/dcanez.22595640/ssv2/train.csv']
+[INFO    ][2026-05-08 20:24:23][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
+[INFO    ][2026-05-08 20:24:25][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA applied: 48 modules replaced (rank=8, alpha=16, dropout=0.0, targets=['attn.qkv', 'attn.proj'], scope=blocks)
+[INFO    ][2026-05-08 20:24:25][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA trainable params: 1,179,648
+[INFO    ][2026-05-08 20:24:27][root                ][_build_frozen_2d_target  ] Loaded frozen_2d target from timm: vit_large_patch14_capi.lvd1689m
+[INFO    ][2026-05-08 20:24:28][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-08 20:24:28][root                ][init_video_model         ] VJEPAPredictorMultiSeqWrapper(
+  (backbone): VisionTransformerPredictor(
+    (predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
+    (mask_tokens): ParameterList(
+        (0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+    )
+    (predictor_blocks): ModuleList(
+      (0-5): 6 x Block(
+        (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (attn): RoPEAttention(
+          (qkv): Linear(in_features=384, out_features=1152, bias=True)
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): Linear(in_features=384, out_features=384, bias=True)
+          (proj_drop): Dropout(p=0.0, inplace=False)
+        )
+        (drop_path): Identity()
+        (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (mlp): MLP(
+          (fc1): Linear(in_features=384, out_features=1536, bias=True)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=1536, out_features=384, bias=True)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+    (predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
+  )
+)
+[INFO    ][2026-05-08 20:24:28][root                ][init_video_model         ] MultiSeqWrapper(
+  (backbone): Frozen2DTargetWrapper(
+    (backbone): Eva(
+      (patch_embed): PatchEmbed(
+        (proj): Conv2d(3, 1024, kernel_size=(14, 14), stride=(14, 14))
+        (norm): Identity()
+      )
+      (pos_drop): Dropout(p=0.0, inplace=False)
+      (norm_pre): Identity()
+      (blocks): ModuleList(
+        (0-23): 24 x EvaBlock(
+          (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+          (attn): EvaAttention(
+            (qkv): Linear(in_features=1024, out_features=3072, bias=False)
+            (q_norm): Identity()
+            (k_norm): Identity()
+            (attn_drop): Dropout(p=0.0, inplace=False)
+            (norm): Identity()
+            (proj): Linear(in_features=1024, out_features=1024, bias=True)
+            (proj_drop): Dropout(p=0.0, inplace=False)
+          )
+          (drop_path1): Identity()
+          (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+          (mlp): Mlp(
+            (fc1): Linear(in_features=1024, out_features=4096, bias=True)
+            (act): GELU(approximate='none')
+            (drop1): Dropout(p=0.0, inplace=False)
+            (norm): Identity()
+            (fc2): Linear(in_features=4096, out_features=1024, bias=True)
+            (drop2): Dropout(p=0.0, inplace=False)
+          )
+          (drop_path2): Identity()
+        )
+      )
+      (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+      (fc_norm): Identity()
+      (head_drop): Dropout(p=0.0, inplace=False)
+      (head): Identity()
+      (rope): _CapiPatchRoPE()
+    )
+  )
+)
+[INFO    ][2026-05-08 20:24:28][root                ][init_video_model         ] Encoder number of parameters: 1179648
+[INFO    ][2026-05-08 20:24:28][root                ][init_video_model         ] Predictor number of parameters: 11436928
+[INFO    ][2026-05-08 20:24:28][root                ][init_video_model         ] Target encoder number of parameters: 0
+[INFO    ][2026-05-08 20:24:29][root                ][make_videodataset        ] VideoDataset dataset created
+[INFO    ][2026-05-08 20:24:29][WeightedSampler     ][__init__                 ] Using DistributedWeightedSampler with rank 14 / 16
+[INFO    ][2026-05-08 20:24:29][root                ][make_videodataset        ] VideoDataset unsupervised data loader created
+[INFO    ][2026-05-08 20:24:29][app.vjepa.train     ][main                     ] iterations per epoch/dataset length: 300/399
+[INFO    ][2026-05-08 20:24:29][app.vjepa.train     ][main                     ] Wrapping models in DDP (rank 14)...
+[INFO    ][2026-05-08 20:24:41][app.vjepa.train     ][main                     ] Initializing loader...
+submitit INFO (2026-05-09 08:16:14,359) - Job has timed out. Ran 718 minutes out of requested 720 minutes.
+[INFO    ][2026-05-09 08:16:14][submitit            ][has_timed_out            ] Job has timed out. Ran 718 minutes out of requested 720 minutes.
+submitit WARNING (2026-05-09 08:16:14,440) - Caught signal SIGUSR2 on gcn84.local.snellius.surf.nl: this job is timed-out.
+[WARNING ][2026-05-09 08:16:14][submitit            ][checkpoint_and_try_requeue] Caught signal SIGUSR2 on gcn84.local.snellius.surf.nl: this job is timed-out.
+submitit INFO (2026-05-09 08:16:14,441) - Not checkpointing nor requeuing since I am a slave (procid=14).
+[INFO    ][2026-05-09 08:16:14][submitit            ][checkpoint_and_try_requeue] Not checkpointing nor requeuing since I am a slave (procid=14).
+submitit WARNING (2026-05-09 08:16:14,441) - Bypassing signal SIGTERM
+[WARNING ][2026-05-09 08:16:14][submitit            ][bypass                   ] Bypassing signal SIGTERM
+submitit WARNING (2026-05-09 08:16:14,441) - Bypassing signal SIGCONT
+[WARNING ][2026-05-09 08:16:14][submitit            ][bypass                   ] Bypassing signal SIGCONT
+submitit INFO (2026-05-09 08:19:47,092) - Starting with JobEnvironment(job_id=22595640, hostname=gcn78.local.snellius.surf.nl, local_rank=2(4), node=3(4), global_rank=14(16))
+submitit INFO (2026-05-09 08:19:49,101) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/job_22595640/22595640_submitted.pkl
+INFO:root:loaded pretrain params...
+{   'app': 'vjepa',
+    'cpus_per_task': 16,
+    'data': {   'batch_size': 64,
+                'crop_size': 224,
+                'dataset_fpcs': [16, 16],
+                'dataset_type': 'VideoDataset',
+                'datasets': [   '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
+                                '/scratch-shared/dcanez/data/ssv2/train.csv'],
+                'datasets_weights': [0.65, 0.35],
+                'fps': 4,
+                'num_workers': 10,
+                'patch_size': 14,
+                'persistent_workers': True,
+                'pin_mem': True,
+                'stage': [   {   'dest': 'kinetics_240',
+                                 'format': 'targz_parts',
+                                 'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
+                             {   'dest': 'ssv2',
+                                 'format': 'multipart_tar',
+                                 'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
+                'tubelet_size': 1},
+    'data_aug': {   'auto_augment': False,
+                    'motion_shift': False,
+                    'random_resize_aspect_ratio': [0.75, 1.35],
+                    'random_resize_scale': [0.3, 1.0],
+                    'reprob': 0.0},
+    'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m',
+    'loss': {'loss_exp': 1.0},
+    'mask': [   {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 8,
+                    'spatial_scale': [0.15, 0.15],
+                    'temporal_scale': [1.0, 1.0]},
+                {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 2,
+                    'spatial_scale': [0.7, 0.7],
+                    'temporal_scale': [1.0, 1.0]}],
+    'mem_per_gpu': '180G',
+    'meta': {   'dtype': 'bfloat16',
+                'knn_eval_epoch0': False,
+                'knn_eval_freq': 5,
+                'knn_eval_presets': [   {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
+                                                          'eval_videos_per_class': 25,
+                                                          'num_workers': 8,
+                                                          'train_videos_per_class': 100},
+                                            'preset': 'ucf101'},
+                                        {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
+                                                          'eval_videos_per_class': 100,
+                                                          'linear_probe': True,
+                                                          'num_workers': 8,
+                                                          'pool_type': 'temporal_concat',
+                                                          'train_videos_per_class': 500},
+                                            'preset': 'ssv2_coarse10'}],
+                'load_checkpoint': True,
+                'read_checkpoint': None,
+                'save_every_freq': 5,
+                'seed': 239,
+                'use_sdpa': True,
+                'use_wandb': True,
+                'wandb_project': 'vjepa_ablation'},
+    'metrics': {'sigreg': {}, 'std': {}},
+    'model': {   'class_token': False,
+                 'freeze_backbone': True,
+                 'is_causal': False,
+                 'lora_alpha': 16,
+                 'lora_rank': 8,
+                 'lora_targets': ['attn.qkv', 'attn.proj'],
+                 'model_name': 'vit_large_patch14_capi_lvd1689m',
+                 'num_reg_tokens': 16,
+                 'pred_depth': 6,
+                 'pred_embed_dim': 384,
+                 'pred_num_heads': 12,
+                 'rope_type': 'capi2d',
+                 'stem_type': '2d',
+                 'target_kind': 'frozen_2d',
+                 'target_type': 'vit_large_patch14_capi.lvd1689m',
+                 'uniform_power': True,
+                 'use_activation_checkpointing': True,
+                 'use_lora': True,
+                 'use_mask_tokens': True,
+                 'use_rope': True,
+                 'use_sdpa': True,
+                 'zero_init_mask_tokens': True},
+    'nodes': 4,
+    'optimization': {   'ema': [0.99925, 0.99925],
+                        'epochs': 100,
+                        'final_lr': 0.0001,
+                        'final_weight_decay': 0.04,
+                        'ipe': 300,
+                        'ipe_scale': 1.0,
+                        'lr': 0.0005,
+                        'start_lr': 0.0001,
+                        'warmup': 10,
+                        'weight_decay': 0.04},
+    'tasks_per_node': 4}
+INFO:root:Running pre-training of app: vjepa
+[INFO    ][2026-05-09 08:20:49][app.vjepa.train     ][main                     ] which_dtype='bfloat16'
+[INFO    ][2026-05-09 08:20:49][app.vjepa.train     ][main                     ] Disabling persistent_workers (incompatible with KNN eval)
+[INFO    ][2026-05-09 08:20:49][app.vjepa.train     ][main                     ] NCCL_SOCKET_IFNAME=eno
+[INFO    ][2026-05-09 08:20:52][app.vjepa.train     ][main                     ] Initialized (rank/world-size) 14/16, tasks_per_node=4
+[INFO    ][2026-05-09 08:20:52][root                ][stage_datasets           ] [local_rank 2/4] Staging kinetics_240 (targz_parts)
+[INFO    ][2026-05-09 08:20:52][root                ][_stage_targz_parts       ] [rank 2] Extracting 26/103 tar.gz parts to /scratch-node/dcanez.22595640/kinetics_240
+[INFO    ][2026-05-09 08:21:05][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 2/26 parts
+[INFO    ][2026-05-09 08:21:19][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 4/26 parts
+[INFO    ][2026-05-09 08:21:34][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 6/26 parts
+[INFO    ][2026-05-09 08:21:48][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 8/26 parts
+[INFO    ][2026-05-09 08:22:02][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 10/26 parts
+[INFO    ][2026-05-09 08:22:16][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 12/26 parts
+[INFO    ][2026-05-09 08:22:30][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 14/26 parts
+[INFO    ][2026-05-09 08:22:45][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 16/26 parts
+[INFO    ][2026-05-09 08:22:59][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 18/26 parts
+[INFO    ][2026-05-09 08:23:13][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 20/26 parts
+[INFO    ][2026-05-09 08:23:28][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 22/26 parts
+[INFO    ][2026-05-09 08:23:42][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 24/26 parts
+[INFO    ][2026-05-09 08:23:56][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 26/26 parts
+[INFO    ][2026-05-09 08:23:56][root                ][stage_datasets           ] [local_rank 2/4] Staging ssv2 (multipart_tar)
+[INFO    ][2026-05-09 08:25:33][app.vjepa.train     ][main                     ] Staged datasets: ['/scratch-node/dcanez.22595640/kinetics_240/train.csv', '/scratch-node/dcanez.22595640/ssv2/train.csv']
+[INFO    ][2026-05-09 08:26:29][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
+[INFO    ][2026-05-09 08:26:30][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA applied: 48 modules replaced (rank=8, alpha=16, dropout=0.0, targets=['attn.qkv', 'attn.proj'], scope=blocks)
+[INFO    ][2026-05-09 08:26:30][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA trainable params: 1,179,648
+[INFO    ][2026-05-09 08:26:33][root                ][_build_frozen_2d_target  ] Loaded frozen_2d target from timm: vit_large_patch14_capi.lvd1689m
+[INFO    ][2026-05-09 08:26:33][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-09 08:26:33][root                ][init_video_model         ] VJEPAPredictorMultiSeqWrapper(
+  (backbone): VisionTransformerPredictor(
+    (predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
+    (mask_tokens): ParameterList(
+        (0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+    )
+    (predictor_blocks): ModuleList(
+      (0-5): 6 x Block(
+        (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (attn): RoPEAttention(
+          (qkv): Linear(in_features=384, out_features=1152, bias=True)
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): Linear(in_features=384, out_features=384, bias=True)
+          (proj_drop): Dropout(p=0.0, inplace=False)
+        )
+        (drop_path): Identity()
+        (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (mlp): MLP(
+          (fc1): Linear(in_features=384, out_features=1536, bias=True)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=1536, out_features=384, bias=True)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+    (predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
+  )
+)
+[INFO    ][2026-05-09 08:26:33][root                ][init_video_model         ] MultiSeqWrapper(
+  (backbone): Frozen2DTargetWrapper(
+    (backbone): Eva(
+      (patch_embed): PatchEmbed(
+        (proj): Conv2d(3, 1024, kernel_size=(14, 14), stride=(14, 14))
+        (norm): Identity()
+      )
+      (pos_drop): Dropout(p=0.0, inplace=False)
+      (norm_pre): Identity()
+      (blocks): ModuleList(
+        (0-23): 24 x EvaBlock(
+          (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+          (attn): EvaAttention(
+            (qkv): Linear(in_features=1024, out_features=3072, bias=False)
+            (q_norm): Identity()
+            (k_norm): Identity()
+            (attn_drop): Dropout(p=0.0, inplace=False)
+            (norm): Identity()
+            (proj): Linear(in_features=1024, out_features=1024, bias=True)
+            (proj_drop): Dropout(p=0.0, inplace=False)
+          )
+          (drop_path1): Identity()
+          (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+          (mlp): Mlp(
+            (fc1): Linear(in_features=1024, out_features=4096, bias=True)
+            (act): GELU(approximate='none')
+            (drop1): Dropout(p=0.0, inplace=False)
+            (norm): Identity()
+            (fc2): Linear(in_features=4096, out_features=1024, bias=True)
+            (drop2): Dropout(p=0.0, inplace=False)
+          )
+          (drop_path2): Identity()
+        )
+      )
+      (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+      (fc_norm): Identity()
+      (head_drop): Dropout(p=0.0, inplace=False)
+      (head): Identity()
+      (rope): _CapiPatchRoPE()
+    )
+  )
+)
+[INFO    ][2026-05-09 08:26:33][root                ][init_video_model         ] Encoder number of parameters: 1179648
+[INFO    ][2026-05-09 08:26:33][root                ][init_video_model         ] Predictor number of parameters: 11436928
+[INFO    ][2026-05-09 08:26:33][root                ][init_video_model         ] Target encoder number of parameters: 0
+[INFO    ][2026-05-09 08:26:34][root                ][make_videodataset        ] VideoDataset dataset created
+[INFO    ][2026-05-09 08:26:34][WeightedSampler     ][__init__                 ] Using DistributedWeightedSampler with rank 14 / 16
+[INFO    ][2026-05-09 08:26:34][root                ][make_videodataset        ] VideoDataset unsupervised data loader created
+[INFO    ][2026-05-09 08:26:34][app.vjepa.train     ][main                     ] iterations per epoch/dataset length: 300/399
+[INFO    ][2026-05-09 08:26:35][app.vjepa.train     ][main                     ] Wrapping models in DDP (rank 14)...
+[INFO    ][2026-05-09 08:26:37][root                ][load_checkpoint          ] Loading checkpoint from /scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/latest.pt
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] loaded pretrained encoder from epoch 39 with msg: <All keys matched successfully>
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] loaded pretrained predictor from epoch 39 with msg: <All keys matched successfully>
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] loaded pretrained target encoder from epoch 39 with msg: <All keys matched successfully>
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] loaded optimizers from epoch 39
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] read-path: /scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/latest.pt
+[INFO    ][2026-05-09 08:26:39][app.vjepa.train     ][main                     ] Initializing loader...
+submitit WARNING (2026-05-09 09:56:00,848) - Bypassing signal SIGTERM
+[WARNING ][2026-05-09 09:56:00][submitit            ][bypass                   ] Bypassing signal SIGTERM
+submitit WARNING (2026-05-09 09:56:00,856) - Bypassing signal SIGCONT
+[WARNING ][2026-05-09 09:56:00][submitit            ][bypass                   ] Bypassing signal SIGCONT

vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/job_22595640/22595640_15_log.err

@@ -0,0 +1,305 @@
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
+  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
+[rank15]:[W508 20:22:14.139243687 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
+  scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+submitit WARNING (2026-05-09 08:16:14,375) - Caught signal SIGUSR2 on gcn84.local.snellius.surf.nl: this job is timed-out.
+submitit WARNING (2026-05-09 08:16:14,375) - Bypassing signal SIGTERM
+submitit WARNING (2026-05-09 08:16:14,375) - Bypassing signal SIGCONT
+[rank15]:[W509 08:16:18.683254710 TCPStore.cpp:125] [c10d] recvValue failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37330, remote=[gcn75.local.snellius.surf.nl]:37129): Failed to recv, got 0 bytes. Connection was likely closed. Did the remote server shutdown or crash?
+Exception raised from recvBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:682 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14beedfedfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a3325d (0x14bf3217c25d in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x273 (0x14bf3217a1f3 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14beef25ceec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14c02e6c6164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14c03f13819a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14c03f1bd100 in /lib64/libc.so.6)
+
+[rank15]:[W509 08:16:18.781874867 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 15] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Failed to recv, got 0 bytes. Connection was likely closed. Did the remote server shutdown or crash?
+[rank15]:[W509 08:16:19.782102928 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37330, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14beedfedfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14bf3217b6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14bf3217a1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14beef25ceec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14c02e6c6164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14c03f13819a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14c03f1bd100 in /lib64/libc.so.6)
+
+[rank15]:[W509 08:16:19.785167527 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 15] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank15]:[W509 08:16:20.785321289 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37330, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14beedfedfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14bf3217b6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14bf3217a1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14beef25ceec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14c02e6c6164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14c03f13819a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14c03f1bd100 in /lib64/libc.so.6)
+
+[rank15]:[W509 08:16:20.787613764 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 15] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank15]:[W509 08:16:21.787753296 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37330, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14beedfedfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14bf3217b6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14bf3217a1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14beef25ceec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14c02e6c6164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14c03f13819a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14c03f1bd100 in /lib64/libc.so.6)
+
+[rank15]:[W509 08:16:21.790045100 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 15] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank15]:[W509 08:16:22.790187732 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37330, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14beedfedfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14bf3217b6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14bf3217a1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14beef25ceec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14c02e6c6164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14c03f13819a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14c03f1bd100 in /lib64/libc.so.6)
+
+[rank15]:[W509 08:16:22.792571576 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 15] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank15]:[W509 08:16:23.792725548 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37330, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14beedfedfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14bf3217b6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14bf3217a1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14beef25ceec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14c02e6c6164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14c03f13819a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14c03f1bd100 in /lib64/libc.so.6)
+
+[rank15]:[W509 08:16:23.795035632 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 15] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank15]:[W509 08:16:24.795178714 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37330, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14beedfedfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14bf3217b6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14bf3217a1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14beef25ceec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14c02e6c6164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14c03f13819a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14c03f1bd100 in /lib64/libc.so.6)
+
+[rank15]:[W509 08:16:24.797473648 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 15] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank15]:[W509 08:16:25.797610750 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37330, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14beedfedfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14bf3217b6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14bf3217a1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14beef25ceec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14c02e6c6164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14c03f13819a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14c03f1bd100 in /lib64/libc.so.6)
+
+[rank15]:[W509 08:16:25.799945245 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 15] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank15]:[W509 08:16:26.800082896 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37330, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14beedfedfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14bf3217b6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14bf3217a1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14beef25ceec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14c02e6c6164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14c03f13819a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14c03f1bd100 in /lib64/libc.so.6)
+
+[rank15]:[W509 08:16:26.802394331 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 15] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank15]:[W509 08:16:27.802536603 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37330, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14beedfedfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14bf3217b6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14bf3217a1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14beef25ceec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14c02e6c6164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14c03f13819a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14c03f1bd100 in /lib64/libc.so.6)
+
+[rank15]:[W509 08:16:27.804759458 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 15] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank15]:[W509 08:16:28.804899860 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37330, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14beedfedfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14bf3217b6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14bf3217a1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14beef25ceec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14c02e6c6164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14c03f13819a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14c03f1bd100 in /lib64/libc.so.6)
+
+[rank15]:[W509 08:16:28.807236564 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 15] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank15]:[W509 08:16:29.807363766 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37330, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14beedfedfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14bf3217b6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14bf3217a1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14beef25ceec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14c02e6c6164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14c03f13819a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14c03f1bd100 in /lib64/libc.so.6)
+
+[rank15]:[W509 08:16:29.809656710 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 15] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank15]:[W509 08:16:30.809797432 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37330, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14beedfedfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14bf3217b6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14bf3217a1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14beef25ceec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14c02e6c6164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14c03f13819a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14c03f1bd100 in /lib64/libc.so.6)
+
+[rank15]:[W509 08:16:30.812090877 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 15] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank15]:[W509 08:16:31.812233408 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37330, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14beedfedfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14bf3217b6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14bf3217a1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14beef25ceec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14c02e6c6164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14c03f13819a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14c03f1bd100 in /lib64/libc.so.6)
+
+[rank15]:[W509 08:16:31.814630182 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 15] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank15]:[W509 08:16:32.814765834 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37330, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14beedfedfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14bf3217b6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14bf3217a1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14beef25ceec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14c02e6c6164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14c03f13819a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14c03f1bd100 in /lib64/libc.so.6)
+
+[rank15]:[W509 08:16:32.816980479 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 15] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank15]:[W509 08:16:33.817119041 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37330, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14beedfedfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14bf3217b6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14bf3217a1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14beef25ceec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14c02e6c6164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14c03f13819a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14c03f1bd100 in /lib64/libc.so.6)
+
+[rank15]:[W509 08:16:33.819475825 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 15] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank15]:[W509 08:16:34.819618347 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37330, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14beedfedfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14bf3217b6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14bf3217a1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14beef25ceec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14c02e6c6164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14c03f13819a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14c03f1bd100 in /lib64/libc.so.6)
+
+[rank15]:[W509 08:16:34.821968811 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 15] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank15]:[W509 08:16:35.822117523 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37330, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14beedfedfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14bf3217b6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14bf3217a1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14beef25ceec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14c02e6c6164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14c03f13819a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14c03f1bd100 in /lib64/libc.so.6)
+
+[rank15]:[W509 08:16:35.824414208 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 15] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank15]:[W509 08:16:36.824553079 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37330, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14beedfedfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14bf3217b6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14bf3217a1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14beef25ceec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14c02e6c6164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14c03f13819a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14c03f1bd100 in /lib64/libc.so.6)
+
+[rank15]:[W509 08:16:36.826902644 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 15] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank15]:[W509 08:16:37.827090945 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37330, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14beedfedfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14bf3217b6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14bf3217a1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14beef25ceec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14c02e6c6164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14c03f13819a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14c03f1bd100 in /lib64/libc.so.6)
+
+[rank15]:[W509 08:16:37.829371790 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 15] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank15]:[W509 08:16:38.829525811 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37330, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14beedfedfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14bf3217b6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14bf3217a1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14beef25ceec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14c02e6c6164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14c03f13819a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14c03f1bd100 in /lib64/libc.so.6)
+
+[rank15]:[W509 08:16:38.831835786 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 15] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank15]:[W509 08:16:39.831970968 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37330, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14beedfedfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14bf3217b6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14bf3217a1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14beef25ceec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14c02e6c6164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14c03f13819a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14c03f1bd100 in /lib64/libc.so.6)
+
+[rank15]:[W509 08:16:39.834307542 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 15] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank15]:[W509 08:16:40.834441414 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37330, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14beedfedfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14bf3217b6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14bf3217a1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14beef25ceec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14c02e6c6164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14c03f13819a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14c03f1bd100 in /lib64/libc.so.6)
+
+[rank15]:[W509 08:16:40.836781818 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 15] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank15]:[W509 08:16:41.836910690 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37330, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14beedfedfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14bf3217b6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14bf3217a1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14beef25ceec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14c02e6c6164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14c03f13819a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14c03f1bd100 in /lib64/libc.so.6)
+
+[rank15]:[W509 08:16:41.839186345 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 15] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank15]:[W509 08:16:42.839329907 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37330, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14beedfedfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14bf3217b6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14bf3217a1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14beef25ceec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14c02e6c6164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14c03f13819a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14c03f1bd100 in /lib64/libc.so.6)
+
+[rank15]:[W509 08:16:42.841505142 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 15] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank15]:[W509 08:16:43.841643854 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn84.local.snellius.surf.nl]:37330, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14beedfedfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14bf3217b6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14bf3217a1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14beef25ceec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14c02e6c6164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14c03f13819a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14c03f1bd100 in /lib64/libc.so.6)
+
+[rank15]:[W509 08:16:43.843957898 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 15] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
+  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
+[rank15]:[W509 08:23:51.242911616 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
+  scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+submitit WARNING (2026-05-09 09:56:00,840) - Bypassing signal SIGTERM
+submitit WARNING (2026-05-09 09:56:01,006) - Bypassing signal SIGCONT

vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/job_22595640/22595640_15_log.out

@@ -0,0 +1,530 @@
+submitit INFO (2026-05-08 20:18:11,518) - Starting with JobEnvironment(job_id=22595640, hostname=gcn84.local.snellius.surf.nl, local_rank=3(4), node=3(4), global_rank=15(16))
+submitit INFO (2026-05-08 20:18:11,518) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/job_22595640/22595640_submitted.pkl
+INFO:root:loaded pretrain params...
+{   'app': 'vjepa',
+    'cpus_per_task': 16,
+    'data': {   'batch_size': 64,
+                'crop_size': 224,
+                'dataset_fpcs': [16, 16],
+                'dataset_type': 'VideoDataset',
+                'datasets': [   '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
+                                '/scratch-shared/dcanez/data/ssv2/train.csv'],
+                'datasets_weights': [0.65, 0.35],
+                'fps': 4,
+                'num_workers': 10,
+                'patch_size': 14,
+                'persistent_workers': True,
+                'pin_mem': True,
+                'stage': [   {   'dest': 'kinetics_240',
+                                 'format': 'targz_parts',
+                                 'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
+                             {   'dest': 'ssv2',
+                                 'format': 'multipart_tar',
+                                 'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
+                'tubelet_size': 1},
+    'data_aug': {   'auto_augment': False,
+                    'motion_shift': False,
+                    'random_resize_aspect_ratio': [0.75, 1.35],
+                    'random_resize_scale': [0.3, 1.0],
+                    'reprob': 0.0},
+    'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m',
+    'loss': {'loss_exp': 1.0},
+    'mask': [   {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 8,
+                    'spatial_scale': [0.15, 0.15],
+                    'temporal_scale': [1.0, 1.0]},
+                {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 2,
+                    'spatial_scale': [0.7, 0.7],
+                    'temporal_scale': [1.0, 1.0]}],
+    'mem_per_gpu': '180G',
+    'meta': {   'dtype': 'bfloat16',
+                'knn_eval_epoch0': False,
+                'knn_eval_freq': 5,
+                'knn_eval_presets': [   {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
+                                                          'eval_videos_per_class': 25,
+                                                          'num_workers': 8,
+                                                          'train_videos_per_class': 100},
+                                            'preset': 'ucf101'},
+                                        {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
+                                                          'eval_videos_per_class': 100,
+                                                          'linear_probe': True,
+                                                          'num_workers': 8,
+                                                          'pool_type': 'temporal_concat',
+                                                          'train_videos_per_class': 500},
+                                            'preset': 'ssv2_coarse10'}],
+                'load_checkpoint': True,
+                'read_checkpoint': None,
+                'save_every_freq': 5,
+                'seed': 239,
+                'use_sdpa': True,
+                'use_wandb': True,
+                'wandb_project': 'vjepa_ablation'},
+    'metrics': {'sigreg': {}, 'std': {}},
+    'model': {   'class_token': False,
+                 'freeze_backbone': True,
+                 'is_causal': False,
+                 'lora_alpha': 16,
+                 'lora_rank': 8,
+                 'lora_targets': ['attn.qkv', 'attn.proj'],
+                 'model_name': 'vit_large_patch14_capi_lvd1689m',
+                 'num_reg_tokens': 16,
+                 'pred_depth': 6,
+                 'pred_embed_dim': 384,
+                 'pred_num_heads': 12,
+                 'rope_type': 'capi2d',
+                 'stem_type': '2d',
+                 'target_kind': 'frozen_2d',
+                 'target_type': 'vit_large_patch14_capi.lvd1689m',
+                 'uniform_power': True,
+                 'use_activation_checkpointing': True,
+                 'use_lora': True,
+                 'use_mask_tokens': True,
+                 'use_rope': True,
+                 'use_sdpa': True,
+                 'zero_init_mask_tokens': True},
+    'nodes': 4,
+    'optimization': {   'ema': [0.99925, 0.99925],
+                        'epochs': 100,
+                        'final_lr': 0.0001,
+                        'final_weight_decay': 0.04,
+                        'ipe': 300,
+                        'ipe_scale': 1.0,
+                        'lr': 0.0005,
+                        'start_lr': 0.0001,
+                        'warmup': 10,
+                        'weight_decay': 0.04},
+    'tasks_per_node': 4}
+INFO:root:Running pre-training of app: vjepa
+[INFO    ][2026-05-08 20:19:11][app.vjepa.train     ][main                     ] which_dtype='bfloat16'
+[INFO    ][2026-05-08 20:19:11][app.vjepa.train     ][main                     ] Disabling persistent_workers (incompatible with KNN eval)
+[INFO    ][2026-05-08 20:19:11][app.vjepa.train     ][main                     ] NCCL_SOCKET_IFNAME=eno
+[INFO    ][2026-05-08 20:19:14][app.vjepa.train     ][main                     ] Initialized (rank/world-size) 15/16, tasks_per_node=4
+[INFO    ][2026-05-08 20:19:14][root                ][stage_datasets           ] [local_rank 3/4] Staging kinetics_240 (targz_parts)
+[INFO    ][2026-05-08 20:19:14][root                ][_stage_targz_parts       ] [rank 3] Extracting 25/103 tar.gz parts to /scratch-node/dcanez.22595640/kinetics_240
+[INFO    ][2026-05-08 20:19:28][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 2/25 parts
+[INFO    ][2026-05-08 20:19:43][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 4/25 parts
+[INFO    ][2026-05-08 20:19:57][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 6/25 parts
+[INFO    ][2026-05-08 20:20:11][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 8/25 parts
+[INFO    ][2026-05-08 20:20:25][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 10/25 parts
+[INFO    ][2026-05-08 20:20:39][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 12/25 parts
+[INFO    ][2026-05-08 20:20:54][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 14/25 parts
+[INFO    ][2026-05-08 20:21:08][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 16/25 parts
+[INFO    ][2026-05-08 20:21:22][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 18/25 parts
+[INFO    ][2026-05-08 20:21:37][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 20/25 parts
+[INFO    ][2026-05-08 20:21:51][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 22/25 parts
+[INFO    ][2026-05-08 20:22:05][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 24/25 parts
+[INFO    ][2026-05-08 20:22:12][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 25/25 parts
+[INFO    ][2026-05-08 20:22:12][root                ][stage_datasets           ] [local_rank 3/4] Staging ssv2 (multipart_tar)
+[INFO    ][2026-05-08 20:23:17][app.vjepa.train     ][main                     ] Staged datasets: ['/scratch-node/dcanez.22595640/kinetics_240/train.csv', '/scratch-node/dcanez.22595640/ssv2/train.csv']
+[INFO    ][2026-05-08 20:24:23][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
+[INFO    ][2026-05-08 20:24:25][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA applied: 48 modules replaced (rank=8, alpha=16, dropout=0.0, targets=['attn.qkv', 'attn.proj'], scope=blocks)
+[INFO    ][2026-05-08 20:24:25][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA trainable params: 1,179,648
+[INFO    ][2026-05-08 20:24:27][root                ][_build_frozen_2d_target  ] Loaded frozen_2d target from timm: vit_large_patch14_capi.lvd1689m
+[INFO    ][2026-05-08 20:24:28][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-08 20:24:28][root                ][init_video_model         ] VJEPAPredictorMultiSeqWrapper(
+  (backbone): VisionTransformerPredictor(
+    (predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
+    (mask_tokens): ParameterList(
+        (0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+    )
+    (predictor_blocks): ModuleList(
+      (0-5): 6 x Block(
+        (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (attn): RoPEAttention(
+          (qkv): Linear(in_features=384, out_features=1152, bias=True)
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): Linear(in_features=384, out_features=384, bias=True)
+          (proj_drop): Dropout(p=0.0, inplace=False)
+        )
+        (drop_path): Identity()
+        (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (mlp): MLP(
+          (fc1): Linear(in_features=384, out_features=1536, bias=True)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=1536, out_features=384, bias=True)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+    (predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
+  )
+)
+[INFO    ][2026-05-08 20:24:28][root                ][init_video_model         ] MultiSeqWrapper(
+  (backbone): Frozen2DTargetWrapper(
+    (backbone): Eva(
+      (patch_embed): PatchEmbed(
+        (proj): Conv2d(3, 1024, kernel_size=(14, 14), stride=(14, 14))
+        (norm): Identity()
+      )
+      (pos_drop): Dropout(p=0.0, inplace=False)
+      (norm_pre): Identity()
+      (blocks): ModuleList(
+        (0-23): 24 x EvaBlock(
+          (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+          (attn): EvaAttention(
+            (qkv): Linear(in_features=1024, out_features=3072, bias=False)
+            (q_norm): Identity()
+            (k_norm): Identity()
+            (attn_drop): Dropout(p=0.0, inplace=False)
+            (norm): Identity()
+            (proj): Linear(in_features=1024, out_features=1024, bias=True)
+            (proj_drop): Dropout(p=0.0, inplace=False)
+          )
+          (drop_path1): Identity()
+          (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+          (mlp): Mlp(
+            (fc1): Linear(in_features=1024, out_features=4096, bias=True)
+            (act): GELU(approximate='none')
+            (drop1): Dropout(p=0.0, inplace=False)
+            (norm): Identity()
+            (fc2): Linear(in_features=4096, out_features=1024, bias=True)
+            (drop2): Dropout(p=0.0, inplace=False)
+          )
+          (drop_path2): Identity()
+        )
+      )
+      (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+      (fc_norm): Identity()
+      (head_drop): Dropout(p=0.0, inplace=False)
+      (head): Identity()
+      (rope): _CapiPatchRoPE()
+    )
+  )
+)
+[INFO    ][2026-05-08 20:24:28][root                ][init_video_model         ] Encoder number of parameters: 1179648
+[INFO    ][2026-05-08 20:24:28][root                ][init_video_model         ] Predictor number of parameters: 11436928
+[INFO    ][2026-05-08 20:24:28][root                ][init_video_model         ] Target encoder number of parameters: 0
+[INFO    ][2026-05-08 20:24:29][root                ][make_videodataset        ] VideoDataset dataset created
+[INFO    ][2026-05-08 20:24:29][WeightedSampler     ][__init__                 ] Using DistributedWeightedSampler with rank 15 / 16
+[INFO    ][2026-05-08 20:24:29][root                ][make_videodataset        ] VideoDataset unsupervised data loader created
+[INFO    ][2026-05-08 20:24:29][app.vjepa.train     ][main                     ] iterations per epoch/dataset length: 300/399
+[INFO    ][2026-05-08 20:24:29][app.vjepa.train     ][main                     ] Wrapping models in DDP (rank 15)...
+[INFO    ][2026-05-08 20:24:41][app.vjepa.train     ][main                     ] Initializing loader...
+submitit INFO (2026-05-09 08:16:14,354) - Job has timed out. Ran 718 minutes out of requested 720 minutes.
+[INFO    ][2026-05-09 08:16:14][submitit            ][has_timed_out            ] Job has timed out. Ran 718 minutes out of requested 720 minutes.
+submitit WARNING (2026-05-09 08:16:14,375) - Caught signal SIGUSR2 on gcn84.local.snellius.surf.nl: this job is timed-out.
+[WARNING ][2026-05-09 08:16:14][submitit            ][checkpoint_and_try_requeue] Caught signal SIGUSR2 on gcn84.local.snellius.surf.nl: this job is timed-out.
+submitit INFO (2026-05-09 08:16:14,375) - Not checkpointing nor requeuing since I am a slave (procid=15).
+[INFO    ][2026-05-09 08:16:14][submitit            ][checkpoint_and_try_requeue] Not checkpointing nor requeuing since I am a slave (procid=15).
+submitit WARNING (2026-05-09 08:16:14,375) - Bypassing signal SIGTERM
+[WARNING ][2026-05-09 08:16:14][submitit            ][bypass                   ] Bypassing signal SIGTERM
+submitit WARNING (2026-05-09 08:16:14,375) - Bypassing signal SIGCONT
+[WARNING ][2026-05-09 08:16:14][submitit            ][bypass                   ] Bypassing signal SIGCONT
+submitit INFO (2026-05-09 08:19:47,092) - Starting with JobEnvironment(job_id=22595640, hostname=gcn78.local.snellius.surf.nl, local_rank=3(4), node=3(4), global_rank=15(16))
+submitit INFO (2026-05-09 08:19:49,101) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/job_22595640/22595640_submitted.pkl
+INFO:root:loaded pretrain params...
+{   'app': 'vjepa',
+    'cpus_per_task': 16,
+    'data': {   'batch_size': 64,
+                'crop_size': 224,
+                'dataset_fpcs': [16, 16],
+                'dataset_type': 'VideoDataset',
+                'datasets': [   '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
+                                '/scratch-shared/dcanez/data/ssv2/train.csv'],
+                'datasets_weights': [0.65, 0.35],
+                'fps': 4,
+                'num_workers': 10,
+                'patch_size': 14,
+                'persistent_workers': True,
+                'pin_mem': True,
+                'stage': [   {   'dest': 'kinetics_240',
+                                 'format': 'targz_parts',
+                                 'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
+                             {   'dest': 'ssv2',
+                                 'format': 'multipart_tar',
+                                 'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
+                'tubelet_size': 1},
+    'data_aug': {   'auto_augment': False,
+                    'motion_shift': False,
+                    'random_resize_aspect_ratio': [0.75, 1.35],
+                    'random_resize_scale': [0.3, 1.0],
+                    'reprob': 0.0},
+    'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m',
+    'loss': {'loss_exp': 1.0},
+    'mask': [   {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 8,
+                    'spatial_scale': [0.15, 0.15],
+                    'temporal_scale': [1.0, 1.0]},
+                {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 2,
+                    'spatial_scale': [0.7, 0.7],
+                    'temporal_scale': [1.0, 1.0]}],
+    'mem_per_gpu': '180G',
+    'meta': {   'dtype': 'bfloat16',
+                'knn_eval_epoch0': False,
+                'knn_eval_freq': 5,
+                'knn_eval_presets': [   {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
+                                                          'eval_videos_per_class': 25,
+                                                          'num_workers': 8,
+                                                          'train_videos_per_class': 100},
+                                            'preset': 'ucf101'},
+                                        {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
+                                                          'eval_videos_per_class': 100,
+                                                          'linear_probe': True,
+                                                          'num_workers': 8,
+                                                          'pool_type': 'temporal_concat',
+                                                          'train_videos_per_class': 500},
+                                            'preset': 'ssv2_coarse10'}],
+                'load_checkpoint': True,
+                'read_checkpoint': None,
+                'save_every_freq': 5,
+                'seed': 239,
+                'use_sdpa': True,
+                'use_wandb': True,
+                'wandb_project': 'vjepa_ablation'},
+    'metrics': {'sigreg': {}, 'std': {}},
+    'model': {   'class_token': False,
+                 'freeze_backbone': True,
+                 'is_causal': False,
+                 'lora_alpha': 16,
+                 'lora_rank': 8,
+                 'lora_targets': ['attn.qkv', 'attn.proj'],
+                 'model_name': 'vit_large_patch14_capi_lvd1689m',
+                 'num_reg_tokens': 16,
+                 'pred_depth': 6,
+                 'pred_embed_dim': 384,
+                 'pred_num_heads': 12,
+                 'rope_type': 'capi2d',
+                 'stem_type': '2d',
+                 'target_kind': 'frozen_2d',
+                 'target_type': 'vit_large_patch14_capi.lvd1689m',
+                 'uniform_power': True,
+                 'use_activation_checkpointing': True,
+                 'use_lora': True,
+                 'use_mask_tokens': True,
+                 'use_rope': True,
+                 'use_sdpa': True,
+                 'zero_init_mask_tokens': True},
+    'nodes': 4,
+    'optimization': {   'ema': [0.99925, 0.99925],
+                        'epochs': 100,
+                        'final_lr': 0.0001,
+                        'final_weight_decay': 0.04,
+                        'ipe': 300,
+                        'ipe_scale': 1.0,
+                        'lr': 0.0005,
+                        'start_lr': 0.0001,
+                        'warmup': 10,
+                        'weight_decay': 0.04},
+    'tasks_per_node': 4}
+INFO:root:Running pre-training of app: vjepa
+[INFO    ][2026-05-09 08:20:49][app.vjepa.train     ][main                     ] which_dtype='bfloat16'
+[INFO    ][2026-05-09 08:20:49][app.vjepa.train     ][main                     ] Disabling persistent_workers (incompatible with KNN eval)
+[INFO    ][2026-05-09 08:20:49][app.vjepa.train     ][main                     ] NCCL_SOCKET_IFNAME=eno
+[INFO    ][2026-05-09 08:20:51][app.vjepa.train     ][main                     ] Initialized (rank/world-size) 15/16, tasks_per_node=4
+[INFO    ][2026-05-09 08:20:51][root                ][stage_datasets           ] [local_rank 3/4] Staging kinetics_240 (targz_parts)
+[INFO    ][2026-05-09 08:20:51][root                ][_stage_targz_parts       ] [rank 3] Extracting 25/103 tar.gz parts to /scratch-node/dcanez.22595640/kinetics_240
+[INFO    ][2026-05-09 08:21:05][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 2/25 parts
+[INFO    ][2026-05-09 08:21:19][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 4/25 parts
+[INFO    ][2026-05-09 08:21:33][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 6/25 parts
+[INFO    ][2026-05-09 08:21:48][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 8/25 parts
+[INFO    ][2026-05-09 08:22:02][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 10/25 parts
+[INFO    ][2026-05-09 08:22:17][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 12/25 parts
+[INFO    ][2026-05-09 08:22:31][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 14/25 parts
+[INFO    ][2026-05-09 08:22:45][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 16/25 parts
+[INFO    ][2026-05-09 08:22:59][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 18/25 parts
+[INFO    ][2026-05-09 08:23:13][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 20/25 parts
+[INFO    ][2026-05-09 08:23:28][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 22/25 parts
+[INFO    ][2026-05-09 08:23:42][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 24/25 parts
+[INFO    ][2026-05-09 08:23:49][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 25/25 parts
+[INFO    ][2026-05-09 08:23:49][root                ][stage_datasets           ] [local_rank 3/4] Staging ssv2 (multipart_tar)
+[INFO    ][2026-05-09 08:25:33][app.vjepa.train     ][main                     ] Staged datasets: ['/scratch-node/dcanez.22595640/kinetics_240/train.csv', '/scratch-node/dcanez.22595640/ssv2/train.csv']
+[INFO    ][2026-05-09 08:26:29][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
+[INFO    ][2026-05-09 08:26:30][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA applied: 48 modules replaced (rank=8, alpha=16, dropout=0.0, targets=['attn.qkv', 'attn.proj'], scope=blocks)
+[INFO    ][2026-05-09 08:26:30][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA trainable params: 1,179,648
+[INFO    ][2026-05-09 08:26:33][root                ][_build_frozen_2d_target  ] Loaded frozen_2d target from timm: vit_large_patch14_capi.lvd1689m
+[INFO    ][2026-05-09 08:26:33][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-09 08:26:33][root                ][init_video_model         ] VJEPAPredictorMultiSeqWrapper(
+  (backbone): VisionTransformerPredictor(
+    (predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
+    (mask_tokens): ParameterList(
+        (0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+    )
+    (predictor_blocks): ModuleList(
+      (0-5): 6 x Block(
+        (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (attn): RoPEAttention(
+          (qkv): Linear(in_features=384, out_features=1152, bias=True)
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): Linear(in_features=384, out_features=384, bias=True)
+          (proj_drop): Dropout(p=0.0, inplace=False)
+        )
+        (drop_path): Identity()
+        (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (mlp): MLP(
+          (fc1): Linear(in_features=384, out_features=1536, bias=True)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=1536, out_features=384, bias=True)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+    (predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
+  )
+)
+[INFO    ][2026-05-09 08:26:33][root                ][init_video_model         ] MultiSeqWrapper(
+  (backbone): Frozen2DTargetWrapper(
+    (backbone): Eva(
+      (patch_embed): PatchEmbed(
+        (proj): Conv2d(3, 1024, kernel_size=(14, 14), stride=(14, 14))
+        (norm): Identity()
+      )
+      (pos_drop): Dropout(p=0.0, inplace=False)
+      (norm_pre): Identity()
+      (blocks): ModuleList(
+        (0-23): 24 x EvaBlock(
+          (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+          (attn): EvaAttention(
+            (qkv): Linear(in_features=1024, out_features=3072, bias=False)
+            (q_norm): Identity()
+            (k_norm): Identity()
+            (attn_drop): Dropout(p=0.0, inplace=False)
+            (norm): Identity()
+            (proj): Linear(in_features=1024, out_features=1024, bias=True)
+            (proj_drop): Dropout(p=0.0, inplace=False)
+          )
+          (drop_path1): Identity()
+          (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+          (mlp): Mlp(
+            (fc1): Linear(in_features=1024, out_features=4096, bias=True)
+            (act): GELU(approximate='none')
+            (drop1): Dropout(p=0.0, inplace=False)
+            (norm): Identity()
+            (fc2): Linear(in_features=4096, out_features=1024, bias=True)
+            (drop2): Dropout(p=0.0, inplace=False)
+          )
+          (drop_path2): Identity()
+        )
+      )
+      (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+      (fc_norm): Identity()
+      (head_drop): Dropout(p=0.0, inplace=False)
+      (head): Identity()
+      (rope): _CapiPatchRoPE()
+    )
+  )
+)
+[INFO    ][2026-05-09 08:26:33][root                ][init_video_model         ] Encoder number of parameters: 1179648
+[INFO    ][2026-05-09 08:26:33][root                ][init_video_model         ] Predictor number of parameters: 11436928
+[INFO    ][2026-05-09 08:26:33][root                ][init_video_model         ] Target encoder number of parameters: 0
+[INFO    ][2026-05-09 08:26:34][root                ][make_videodataset        ] VideoDataset dataset created
+[INFO    ][2026-05-09 08:26:34][WeightedSampler     ][__init__                 ] Using DistributedWeightedSampler with rank 15 / 16
+[INFO    ][2026-05-09 08:26:34][root                ][make_videodataset        ] VideoDataset unsupervised data loader created
+[INFO    ][2026-05-09 08:26:34][app.vjepa.train     ][main                     ] iterations per epoch/dataset length: 300/399
+[INFO    ][2026-05-09 08:26:36][app.vjepa.train     ][main                     ] Wrapping models in DDP (rank 15)...
+[INFO    ][2026-05-09 08:26:37][root                ][load_checkpoint          ] Loading checkpoint from /scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/latest.pt
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] loaded pretrained encoder from epoch 39 with msg: <All keys matched successfully>
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] loaded pretrained predictor from epoch 39 with msg: <All keys matched successfully>
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] loaded pretrained target encoder from epoch 39 with msg: <All keys matched successfully>
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] loaded optimizers from epoch 39
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] read-path: /scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/latest.pt
+[INFO    ][2026-05-09 08:26:39][app.vjepa.train     ][main                     ] Initializing loader...
+submitit WARNING (2026-05-09 09:56:00,840) - Bypassing signal SIGTERM
+[WARNING ][2026-05-09 09:56:00][submitit            ][bypass                   ] Bypassing signal SIGTERM
+submitit WARNING (2026-05-09 09:56:01,006) - Bypassing signal SIGCONT
+[WARNING ][2026-05-09 09:56:01][submitit            ][bypass                   ] Bypassing signal SIGCONT

vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/job_22595640/22595640_1_log.err

@@ -0,0 +1,322 @@
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
+  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
+[rank1]:[W508 20:21:41.626845358 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
+  scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+submitit WARNING (2026-05-09 08:16:14,175) - Caught signal SIGUSR2 on gcn75.local.snellius.surf.nl: this job is timed-out.
+submitit WARNING (2026-05-09 08:16:14,175) - Bypassing signal SIGCONT
+submitit WARNING (2026-05-09 08:16:14,372) - Bypassing signal SIGTERM
+submitit WARNING (2026-05-09 08:16:14,372) - Bypassing signal SIGCONT
+[rank1]:[W509 08:16:18.542252772 TCPStore.cpp:125] [c10d] recvValue failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43532, remote=[gcn75.local.snellius.surf.nl]:37129): Failed to recv, got 0 bytes. Connection was likely closed. Did the remote server shutdown or crash?
+Exception raised from recvBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:682 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1521faf48fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a3325d (0x15223f0d725d in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x273 (0x15223f0d51f3 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1521fc1b7eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15233b721164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15234c19319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15234c218100 in /lib64/libc.so.6)
+
+[rank1]:[W509 08:16:18.545619215 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 1] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Failed to recv, got 0 bytes. Connection was likely closed. Did the remote server shutdown or crash?
+[rank1]:[W509 08:16:19.545771996 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43532, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1521faf48fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15223f0d66d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15223f0d51cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1521fc1b7eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15233b721164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15234c19319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15234c218100 in /lib64/libc.so.6)
+
+[rank1]:[W509 08:16:19.548050085 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 1] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank1]:[W509 08:16:20.548187756 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43532, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1521faf48fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15223f0d66d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15223f0d51cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1521fc1b7eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15233b721164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15234c19319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15234c218100 in /lib64/libc.so.6)
+
+[rank1]:[W509 08:16:20.550530484 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 1] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank1]:[W509 08:16:21.550665525 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43532, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1521faf48fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15223f0d66d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15223f0d51cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1521fc1b7eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15233b721164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15234c19319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15234c218100 in /lib64/libc.so.6)
+
+[rank1]:[W509 08:16:21.553021903 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 1] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank1]:[W509 08:16:22.553172244 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43532, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1521faf48fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15223f0d66d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15223f0d51cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1521fc1b7eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15233b721164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15234c19319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15234c218100 in /lib64/libc.so.6)
+
+[rank1]:[W509 08:16:22.555349623 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 1] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank1]:[W509 08:16:23.555480875 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43532, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1521faf48fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15223f0d66d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15223f0d51cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1521fc1b7eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15233b721164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15234c19319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15234c218100 in /lib64/libc.so.6)
+
+[rank1]:[W509 08:16:23.557908792 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 1] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank1]:[W509 08:16:24.558078193 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43532, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1521faf48fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15223f0d66d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15223f0d51cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1521fc1b7eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15233b721164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15234c19319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15234c218100 in /lib64/libc.so.6)
+
+[rank1]:[W509 08:16:24.560446361 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 1] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank1]:[W509 08:16:25.560594632 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43532, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1521faf48fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15223f0d66d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15223f0d51cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1521fc1b7eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15233b721164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15234c19319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15234c218100 in /lib64/libc.so.6)
+
+[rank1]:[W509 08:16:25.562939250 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 1] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank1]:[W509 08:16:26.563064812 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43532, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1521faf48fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15223f0d66d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15223f0d51cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1521fc1b7eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15233b721164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15234c19319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15234c218100 in /lib64/libc.so.6)
+
+[rank1]:[W509 08:16:26.565383320 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 1] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank1]:[W509 08:16:27.565540171 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43532, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1521faf48fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15223f0d66d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15223f0d51cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1521fc1b7eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15233b721164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15234c19319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15234c218100 in /lib64/libc.so.6)
+
+[rank1]:[W509 08:16:27.567938379 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 1] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank1]:[W509 08:16:28.568097780 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43532, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1521faf48fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15223f0d66d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15223f0d51cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1521fc1b7eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15233b721164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15234c19319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15234c218100 in /lib64/libc.so.6)
+
+[rank1]:[W509 08:16:28.570473728 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 1] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank1]:[W509 08:16:29.570590499 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43532, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1521faf48fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15223f0d66d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15223f0d51cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1521fc1b7eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15233b721164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15234c19319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15234c218100 in /lib64/libc.so.6)
+
+[rank1]:[W509 08:16:29.572904187 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 1] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank1]:[W509 08:16:30.573073389 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43532, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1521faf48fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15223f0d66d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15223f0d51cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1521fc1b7eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15233b721164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15234c19319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15234c218100 in /lib64/libc.so.6)
+
+[rank1]:[W509 08:16:30.575512166 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 1] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank1]:[W509 08:16:31.575655447 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43532, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1521faf48fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15223f0d66d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15223f0d51cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1521fc1b7eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15233b721164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15234c19319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15234c218100 in /lib64/libc.so.6)
+
+[rank1]:[W509 08:16:31.577974655 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 1] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank1]:[W509 08:16:32.578136417 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43532, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1521faf48fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15223f0d66d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15223f0d51cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1521fc1b7eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15233b721164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15234c19319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15234c218100 in /lib64/libc.so.6)
+
+[rank1]:[W509 08:16:32.580662024 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 1] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank1]:[W509 08:16:33.580820315 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43532, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1521faf48fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15223f0d66d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15223f0d51cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1521fc1b7eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15233b721164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15234c19319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15234c218100 in /lib64/libc.so.6)
+
+[rank1]:[W509 08:16:33.583186743 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 1] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank1]:[W509 08:16:34.583319824 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43532, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1521faf48fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15223f0d66d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15223f0d51cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1521fc1b7eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15233b721164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15234c19319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15234c218100 in /lib64/libc.so.6)
+
+[rank1]:[W509 08:16:34.585688012 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 1] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank1]:[W509 08:16:35.585813384 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43532, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1521faf48fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15223f0d66d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15223f0d51cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1521fc1b7eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15233b721164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15234c19319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15234c218100 in /lib64/libc.so.6)
+
+[rank1]:[W509 08:16:35.588052462 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 1] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank1]:[W509 08:16:36.588199683 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43532, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1521faf48fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15223f0d66d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15223f0d51cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1521fc1b7eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15233b721164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15234c19319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15234c218100 in /lib64/libc.so.6)
+
+[rank1]:[W509 08:16:36.590536281 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 1] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank1]:[W509 08:16:37.590691303 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43532, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1521faf48fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15223f0d66d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15223f0d51cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1521fc1b7eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15233b721164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15234c19319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15234c218100 in /lib64/libc.so.6)
+
+[rank1]:[W509 08:16:37.592959341 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 1] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank1]:[W509 08:16:38.593132482 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43532, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1521faf48fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15223f0d66d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15223f0d51cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1521fc1b7eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15233b721164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15234c19319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15234c218100 in /lib64/libc.so.6)
+
+[rank1]:[W509 08:16:38.595352531 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 1] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank1]:[W509 08:16:39.595471322 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43532, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1521faf48fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15223f0d66d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15223f0d51cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1521fc1b7eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15233b721164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15234c19319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15234c218100 in /lib64/libc.so.6)
+
+[rank1]:[W509 08:16:39.597837570 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 1] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank1]:[W509 08:16:40.597975211 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43532, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1521faf48fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15223f0d66d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15223f0d51cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1521fc1b7eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15233b721164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15234c19319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15234c218100 in /lib64/libc.so.6)
+
+[rank1]:[W509 08:16:40.600333529 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 1] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank1]:[W509 08:16:41.600487171 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43532, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1521faf48fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15223f0d66d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15223f0d51cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1521fc1b7eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15233b721164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15234c19319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15234c218100 in /lib64/libc.so.6)
+
+[rank1]:[W509 08:16:41.602797689 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 1] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank1]:[W509 08:16:42.602926010 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43532, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1521faf48fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15223f0d66d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15223f0d51cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1521fc1b7eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15233b721164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15234c19319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15234c218100 in /lib64/libc.so.6)
+
+[rank1]:[W509 08:16:42.605264278 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 1] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank1]:[W509 08:16:43.605395789 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43532, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1521faf48fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15223f0d66d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15223f0d51cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1521fc1b7eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x15233b721164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x15234c19319a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15234c218100 in /lib64/libc.so.6)
+
+[rank1]:[W509 08:16:43.607585348 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 1] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
+  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
+[rank1]:[W509 08:23:53.411896856 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
+  scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+submitit WARNING (2026-05-09 09:56:00,845) - Bypassing signal SIGTERM
+submitit WARNING (2026-05-09 09:56:00,869) - Bypassing signal SIGCONT

vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/job_22595640/22595640_1_log.out

@@ -0,0 +1,532 @@
+submitit INFO (2026-05-08 20:18:04,102) - Starting with JobEnvironment(job_id=22595640, hostname=gcn75.local.snellius.surf.nl, local_rank=1(4), node=0(4), global_rank=1(16))
+submitit INFO (2026-05-08 20:18:04,103) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/job_22595640/22595640_submitted.pkl
+INFO:root:loaded pretrain params...
+{   'app': 'vjepa',
+    'cpus_per_task': 16,
+    'data': {   'batch_size': 64,
+                'crop_size': 224,
+                'dataset_fpcs': [16, 16],
+                'dataset_type': 'VideoDataset',
+                'datasets': [   '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
+                                '/scratch-shared/dcanez/data/ssv2/train.csv'],
+                'datasets_weights': [0.65, 0.35],
+                'fps': 4,
+                'num_workers': 10,
+                'patch_size': 14,
+                'persistent_workers': True,
+                'pin_mem': True,
+                'stage': [   {   'dest': 'kinetics_240',
+                                 'format': 'targz_parts',
+                                 'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
+                             {   'dest': 'ssv2',
+                                 'format': 'multipart_tar',
+                                 'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
+                'tubelet_size': 1},
+    'data_aug': {   'auto_augment': False,
+                    'motion_shift': False,
+                    'random_resize_aspect_ratio': [0.75, 1.35],
+                    'random_resize_scale': [0.3, 1.0],
+                    'reprob': 0.0},
+    'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m',
+    'loss': {'loss_exp': 1.0},
+    'mask': [   {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 8,
+                    'spatial_scale': [0.15, 0.15],
+                    'temporal_scale': [1.0, 1.0]},
+                {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 2,
+                    'spatial_scale': [0.7, 0.7],
+                    'temporal_scale': [1.0, 1.0]}],
+    'mem_per_gpu': '180G',
+    'meta': {   'dtype': 'bfloat16',
+                'knn_eval_epoch0': False,
+                'knn_eval_freq': 5,
+                'knn_eval_presets': [   {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
+                                                          'eval_videos_per_class': 25,
+                                                          'num_workers': 8,
+                                                          'train_videos_per_class': 100},
+                                            'preset': 'ucf101'},
+                                        {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
+                                                          'eval_videos_per_class': 100,
+                                                          'linear_probe': True,
+                                                          'num_workers': 8,
+                                                          'pool_type': 'temporal_concat',
+                                                          'train_videos_per_class': 500},
+                                            'preset': 'ssv2_coarse10'}],
+                'load_checkpoint': True,
+                'read_checkpoint': None,
+                'save_every_freq': 5,
+                'seed': 239,
+                'use_sdpa': True,
+                'use_wandb': True,
+                'wandb_project': 'vjepa_ablation'},
+    'metrics': {'sigreg': {}, 'std': {}},
+    'model': {   'class_token': False,
+                 'freeze_backbone': True,
+                 'is_causal': False,
+                 'lora_alpha': 16,
+                 'lora_rank': 8,
+                 'lora_targets': ['attn.qkv', 'attn.proj'],
+                 'model_name': 'vit_large_patch14_capi_lvd1689m',
+                 'num_reg_tokens': 16,
+                 'pred_depth': 6,
+                 'pred_embed_dim': 384,
+                 'pred_num_heads': 12,
+                 'rope_type': 'capi2d',
+                 'stem_type': '2d',
+                 'target_kind': 'frozen_2d',
+                 'target_type': 'vit_large_patch14_capi.lvd1689m',
+                 'uniform_power': True,
+                 'use_activation_checkpointing': True,
+                 'use_lora': True,
+                 'use_mask_tokens': True,
+                 'use_rope': True,
+                 'use_sdpa': True,
+                 'zero_init_mask_tokens': True},
+    'nodes': 4,
+    'optimization': {   'ema': [0.99925, 0.99925],
+                        'epochs': 100,
+                        'final_lr': 0.0001,
+                        'final_weight_decay': 0.04,
+                        'ipe': 300,
+                        'ipe_scale': 1.0,
+                        'lr': 0.0005,
+                        'start_lr': 0.0001,
+                        'warmup': 10,
+                        'weight_decay': 0.04},
+    'tasks_per_node': 4}
+INFO:root:Running pre-training of app: vjepa
+[INFO    ][2026-05-08 20:18:35][app.vjepa.train     ][main                     ] which_dtype='bfloat16'
+[INFO    ][2026-05-08 20:18:35][app.vjepa.train     ][main                     ] Disabling persistent_workers (incompatible with KNN eval)
+[INFO    ][2026-05-08 20:18:35][app.vjepa.train     ][main                     ] NCCL_SOCKET_IFNAME=eno
+[INFO    ][2026-05-08 20:18:37][app.vjepa.train     ][main                     ] Initialized (rank/world-size) 1/16, tasks_per_node=4
+[INFO    ][2026-05-08 20:18:37][root                ][stage_datasets           ] [local_rank 1/4] Staging kinetics_240 (targz_parts)
+[INFO    ][2026-05-08 20:18:37][root                ][_stage_targz_parts       ] [rank 1] Extracting 26/103 tar.gz parts to /scratch-node/dcanez.22595640/kinetics_240
+[INFO    ][2026-05-08 20:18:50][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 2/26 parts
+[INFO    ][2026-05-08 20:19:03][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 4/26 parts
+[INFO    ][2026-05-08 20:19:18][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 6/26 parts
+[INFO    ][2026-05-08 20:19:32][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 8/26 parts
+[INFO    ][2026-05-08 20:19:46][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 10/26 parts
+[INFO    ][2026-05-08 20:20:01][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 12/26 parts
+[INFO    ][2026-05-08 20:20:14][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 14/26 parts
+[INFO    ][2026-05-08 20:20:29][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 16/26 parts
+[INFO    ][2026-05-08 20:20:43][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 18/26 parts
+[INFO    ][2026-05-08 20:20:58][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 20/26 parts
+[INFO    ][2026-05-08 20:21:12][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 22/26 parts
+[INFO    ][2026-05-08 20:21:27][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 24/26 parts
+[INFO    ][2026-05-08 20:21:41][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 26/26 parts
+[INFO    ][2026-05-08 20:21:41][root                ][stage_datasets           ] [local_rank 1/4] Staging ssv2 (multipart_tar)
+[INFO    ][2026-05-08 20:23:17][app.vjepa.train     ][main                     ] Staged datasets: ['/scratch-node/dcanez.22595640/kinetics_240/train.csv', '/scratch-node/dcanez.22595640/ssv2/train.csv']
+[INFO    ][2026-05-08 20:23:59][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
+[INFO    ][2026-05-08 20:24:03][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA applied: 48 modules replaced (rank=8, alpha=16, dropout=0.0, targets=['attn.qkv', 'attn.proj'], scope=blocks)
+[INFO    ][2026-05-08 20:24:03][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA trainable params: 1,179,648
+[INFO    ][2026-05-08 20:24:06][root                ][_build_frozen_2d_target  ] Loaded frozen_2d target from timm: vit_large_patch14_capi.lvd1689m
+[INFO    ][2026-05-08 20:24:06][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-08 20:24:06][root                ][init_video_model         ] VJEPAPredictorMultiSeqWrapper(
+  (backbone): VisionTransformerPredictor(
+    (predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
+    (mask_tokens): ParameterList(
+        (0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+    )
+    (predictor_blocks): ModuleList(
+      (0-5): 6 x Block(
+        (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (attn): RoPEAttention(
+          (qkv): Linear(in_features=384, out_features=1152, bias=True)
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): Linear(in_features=384, out_features=384, bias=True)
+          (proj_drop): Dropout(p=0.0, inplace=False)
+        )
+        (drop_path): Identity()
+        (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (mlp): MLP(
+          (fc1): Linear(in_features=384, out_features=1536, bias=True)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=1536, out_features=384, bias=True)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+    (predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
+  )
+)
+[INFO    ][2026-05-08 20:24:06][root                ][init_video_model         ] MultiSeqWrapper(
+  (backbone): Frozen2DTargetWrapper(
+    (backbone): Eva(
+      (patch_embed): PatchEmbed(
+        (proj): Conv2d(3, 1024, kernel_size=(14, 14), stride=(14, 14))
+        (norm): Identity()
+      )
+      (pos_drop): Dropout(p=0.0, inplace=False)
+      (norm_pre): Identity()
+      (blocks): ModuleList(
+        (0-23): 24 x EvaBlock(
+          (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+          (attn): EvaAttention(
+            (qkv): Linear(in_features=1024, out_features=3072, bias=False)
+            (q_norm): Identity()
+            (k_norm): Identity()
+            (attn_drop): Dropout(p=0.0, inplace=False)
+            (norm): Identity()
+            (proj): Linear(in_features=1024, out_features=1024, bias=True)
+            (proj_drop): Dropout(p=0.0, inplace=False)
+          )
+          (drop_path1): Identity()
+          (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+          (mlp): Mlp(
+            (fc1): Linear(in_features=1024, out_features=4096, bias=True)
+            (act): GELU(approximate='none')
+            (drop1): Dropout(p=0.0, inplace=False)
+            (norm): Identity()
+            (fc2): Linear(in_features=4096, out_features=1024, bias=True)
+            (drop2): Dropout(p=0.0, inplace=False)
+          )
+          (drop_path2): Identity()
+        )
+      )
+      (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+      (fc_norm): Identity()
+      (head_drop): Dropout(p=0.0, inplace=False)
+      (head): Identity()
+      (rope): _CapiPatchRoPE()
+    )
+  )
+)
+[INFO    ][2026-05-08 20:24:06][root                ][init_video_model         ] Encoder number of parameters: 1179648
+[INFO    ][2026-05-08 20:24:06][root                ][init_video_model         ] Predictor number of parameters: 11436928
+[INFO    ][2026-05-08 20:24:06][root                ][init_video_model         ] Target encoder number of parameters: 0
+[INFO    ][2026-05-08 20:24:08][root                ][make_videodataset        ] VideoDataset dataset created
+[INFO    ][2026-05-08 20:24:08][WeightedSampler     ][__init__                 ] Using DistributedWeightedSampler with rank 1 / 16
+[INFO    ][2026-05-08 20:24:08][root                ][make_videodataset        ] VideoDataset unsupervised data loader created
+[INFO    ][2026-05-08 20:24:08][app.vjepa.train     ][main                     ] iterations per epoch/dataset length: 300/399
+[INFO    ][2026-05-08 20:24:08][app.vjepa.train     ][main                     ] Wrapping models in DDP (rank 1)...
+[INFO    ][2026-05-08 20:24:41][app.vjepa.train     ][main                     ] Initializing loader...
+submitit INFO (2026-05-09 08:16:14,160) - Job has timed out. Ran 718 minutes out of requested 720 minutes.
+[INFO    ][2026-05-09 08:16:14][submitit            ][has_timed_out            ] Job has timed out. Ran 718 minutes out of requested 720 minutes.
+submitit WARNING (2026-05-09 08:16:14,175) - Caught signal SIGUSR2 on gcn75.local.snellius.surf.nl: this job is timed-out.
+[WARNING ][2026-05-09 08:16:14][submitit            ][checkpoint_and_try_requeue] Caught signal SIGUSR2 on gcn75.local.snellius.surf.nl: this job is timed-out.
+submitit INFO (2026-05-09 08:16:14,175) - Not checkpointing nor requeuing since I am a slave (procid=1).
+[INFO    ][2026-05-09 08:16:14][submitit            ][checkpoint_and_try_requeue] Not checkpointing nor requeuing since I am a slave (procid=1).
+submitit WARNING (2026-05-09 08:16:14,175) - Bypassing signal SIGCONT
+[WARNING ][2026-05-09 08:16:14][submitit            ][bypass                   ] Bypassing signal SIGCONT
+submitit WARNING (2026-05-09 08:16:14,372) - Bypassing signal SIGTERM
+[WARNING ][2026-05-09 08:16:14][submitit            ][bypass                   ] Bypassing signal SIGTERM
+submitit WARNING (2026-05-09 08:16:14,372) - Bypassing signal SIGCONT
+[WARNING ][2026-05-09 08:16:14][submitit            ][bypass                   ] Bypassing signal SIGCONT
+submitit INFO (2026-05-09 08:19:47,093) - Starting with JobEnvironment(job_id=22595640, hostname=gcn73.local.snellius.surf.nl, local_rank=1(4), node=0(4), global_rank=1(16))
+submitit INFO (2026-05-09 08:19:47,129) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/job_22595640/22595640_submitted.pkl
+INFO:root:loaded pretrain params...
+{   'app': 'vjepa',
+    'cpus_per_task': 16,
+    'data': {   'batch_size': 64,
+                'crop_size': 224,
+                'dataset_fpcs': [16, 16],
+                'dataset_type': 'VideoDataset',
+                'datasets': [   '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
+                                '/scratch-shared/dcanez/data/ssv2/train.csv'],
+                'datasets_weights': [0.65, 0.35],
+                'fps': 4,
+                'num_workers': 10,
+                'patch_size': 14,
+                'persistent_workers': True,
+                'pin_mem': True,
+                'stage': [   {   'dest': 'kinetics_240',
+                                 'format': 'targz_parts',
+                                 'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
+                             {   'dest': 'ssv2',
+                                 'format': 'multipart_tar',
+                                 'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
+                'tubelet_size': 1},
+    'data_aug': {   'auto_augment': False,
+                    'motion_shift': False,
+                    'random_resize_aspect_ratio': [0.75, 1.35],
+                    'random_resize_scale': [0.3, 1.0],
+                    'reprob': 0.0},
+    'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m',
+    'loss': {'loss_exp': 1.0},
+    'mask': [   {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 8,
+                    'spatial_scale': [0.15, 0.15],
+                    'temporal_scale': [1.0, 1.0]},
+                {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 2,
+                    'spatial_scale': [0.7, 0.7],
+                    'temporal_scale': [1.0, 1.0]}],
+    'mem_per_gpu': '180G',
+    'meta': {   'dtype': 'bfloat16',
+                'knn_eval_epoch0': False,
+                'knn_eval_freq': 5,
+                'knn_eval_presets': [   {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
+                                                          'eval_videos_per_class': 25,
+                                                          'num_workers': 8,
+                                                          'train_videos_per_class': 100},
+                                            'preset': 'ucf101'},
+                                        {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
+                                                          'eval_videos_per_class': 100,
+                                                          'linear_probe': True,
+                                                          'num_workers': 8,
+                                                          'pool_type': 'temporal_concat',
+                                                          'train_videos_per_class': 500},
+                                            'preset': 'ssv2_coarse10'}],
+                'load_checkpoint': True,
+                'read_checkpoint': None,
+                'save_every_freq': 5,
+                'seed': 239,
+                'use_sdpa': True,
+                'use_wandb': True,
+                'wandb_project': 'vjepa_ablation'},
+    'metrics': {'sigreg': {}, 'std': {}},
+    'model': {   'class_token': False,
+                 'freeze_backbone': True,
+                 'is_causal': False,
+                 'lora_alpha': 16,
+                 'lora_rank': 8,
+                 'lora_targets': ['attn.qkv', 'attn.proj'],
+                 'model_name': 'vit_large_patch14_capi_lvd1689m',
+                 'num_reg_tokens': 16,
+                 'pred_depth': 6,
+                 'pred_embed_dim': 384,
+                 'pred_num_heads': 12,
+                 'rope_type': 'capi2d',
+                 'stem_type': '2d',
+                 'target_kind': 'frozen_2d',
+                 'target_type': 'vit_large_patch14_capi.lvd1689m',
+                 'uniform_power': True,
+                 'use_activation_checkpointing': True,
+                 'use_lora': True,
+                 'use_mask_tokens': True,
+                 'use_rope': True,
+                 'use_sdpa': True,
+                 'zero_init_mask_tokens': True},
+    'nodes': 4,
+    'optimization': {   'ema': [0.99925, 0.99925],
+                        'epochs': 100,
+                        'final_lr': 0.0001,
+                        'final_weight_decay': 0.04,
+                        'ipe': 300,
+                        'ipe_scale': 1.0,
+                        'lr': 0.0005,
+                        'start_lr': 0.0001,
+                        'warmup': 10,
+                        'weight_decay': 0.04},
+    'tasks_per_node': 4}
+INFO:root:Running pre-training of app: vjepa
+[INFO    ][2026-05-09 08:20:49][app.vjepa.train     ][main                     ] which_dtype='bfloat16'
+[INFO    ][2026-05-09 08:20:49][app.vjepa.train     ][main                     ] Disabling persistent_workers (incompatible with KNN eval)
+[INFO    ][2026-05-09 08:20:49][app.vjepa.train     ][main                     ] NCCL_SOCKET_IFNAME=eno
+[INFO    ][2026-05-09 08:20:50][app.vjepa.train     ][main                     ] Initialized (rank/world-size) 1/16, tasks_per_node=4
+[INFO    ][2026-05-09 08:20:50][root                ][stage_datasets           ] [local_rank 1/4] Staging kinetics_240 (targz_parts)
+[INFO    ][2026-05-09 08:20:50][root                ][_stage_targz_parts       ] [rank 1] Extracting 26/103 tar.gz parts to /scratch-node/dcanez.22595640/kinetics_240
+[INFO    ][2026-05-09 08:21:03][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 2/26 parts
+[INFO    ][2026-05-09 08:21:17][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 4/26 parts
+[INFO    ][2026-05-09 08:21:31][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 6/26 parts
+[INFO    ][2026-05-09 08:21:45][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 8/26 parts
+[INFO    ][2026-05-09 08:21:59][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 10/26 parts
+[INFO    ][2026-05-09 08:22:13][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 12/26 parts
+[INFO    ][2026-05-09 08:22:27][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 14/26 parts
+[INFO    ][2026-05-09 08:22:41][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 16/26 parts
+[INFO    ][2026-05-09 08:22:55][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 18/26 parts
+[INFO    ][2026-05-09 08:23:09][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 20/26 parts
+[INFO    ][2026-05-09 08:23:24][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 22/26 parts
+[INFO    ][2026-05-09 08:23:38][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 24/26 parts
+[INFO    ][2026-05-09 08:23:53][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 26/26 parts
+[INFO    ][2026-05-09 08:23:53][root                ][stage_datasets           ] [local_rank 1/4] Staging ssv2 (multipart_tar)
+[INFO    ][2026-05-09 08:25:33][app.vjepa.train     ][main                     ] Staged datasets: ['/scratch-node/dcanez.22595640/kinetics_240/train.csv', '/scratch-node/dcanez.22595640/ssv2/train.csv']
+[INFO    ][2026-05-09 08:26:29][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
+[INFO    ][2026-05-09 08:26:30][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA applied: 48 modules replaced (rank=8, alpha=16, dropout=0.0, targets=['attn.qkv', 'attn.proj'], scope=blocks)
+[INFO    ][2026-05-09 08:26:30][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA trainable params: 1,179,648
+[INFO    ][2026-05-09 08:26:33][root                ][_build_frozen_2d_target  ] Loaded frozen_2d target from timm: vit_large_patch14_capi.lvd1689m
+[INFO    ][2026-05-09 08:26:33][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-09 08:26:33][root                ][init_video_model         ] VJEPAPredictorMultiSeqWrapper(
+  (backbone): VisionTransformerPredictor(
+    (predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
+    (mask_tokens): ParameterList(
+        (0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+    )
+    (predictor_blocks): ModuleList(
+      (0-5): 6 x Block(
+        (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (attn): RoPEAttention(
+          (qkv): Linear(in_features=384, out_features=1152, bias=True)
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): Linear(in_features=384, out_features=384, bias=True)
+          (proj_drop): Dropout(p=0.0, inplace=False)
+        )
+        (drop_path): Identity()
+        (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (mlp): MLP(
+          (fc1): Linear(in_features=384, out_features=1536, bias=True)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=1536, out_features=384, bias=True)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+    (predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
+  )
+)
+[INFO    ][2026-05-09 08:26:33][root                ][init_video_model         ] MultiSeqWrapper(
+  (backbone): Frozen2DTargetWrapper(
+    (backbone): Eva(
+      (patch_embed): PatchEmbed(
+        (proj): Conv2d(3, 1024, kernel_size=(14, 14), stride=(14, 14))
+        (norm): Identity()
+      )
+      (pos_drop): Dropout(p=0.0, inplace=False)
+      (norm_pre): Identity()
+      (blocks): ModuleList(
+        (0-23): 24 x EvaBlock(
+          (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+          (attn): EvaAttention(
+            (qkv): Linear(in_features=1024, out_features=3072, bias=False)
+            (q_norm): Identity()
+            (k_norm): Identity()
+            (attn_drop): Dropout(p=0.0, inplace=False)
+            (norm): Identity()
+            (proj): Linear(in_features=1024, out_features=1024, bias=True)
+            (proj_drop): Dropout(p=0.0, inplace=False)
+          )
+          (drop_path1): Identity()
+          (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+          (mlp): Mlp(
+            (fc1): Linear(in_features=1024, out_features=4096, bias=True)
+            (act): GELU(approximate='none')
+            (drop1): Dropout(p=0.0, inplace=False)
+            (norm): Identity()
+            (fc2): Linear(in_features=4096, out_features=1024, bias=True)
+            (drop2): Dropout(p=0.0, inplace=False)
+          )
+          (drop_path2): Identity()
+        )
+      )
+      (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+      (fc_norm): Identity()
+      (head_drop): Dropout(p=0.0, inplace=False)
+      (head): Identity()
+      (rope): _CapiPatchRoPE()
+    )
+  )
+)
+[INFO    ][2026-05-09 08:26:33][root                ][init_video_model         ] Encoder number of parameters: 1179648
+[INFO    ][2026-05-09 08:26:33][root                ][init_video_model         ] Predictor number of parameters: 11436928
+[INFO    ][2026-05-09 08:26:33][root                ][init_video_model         ] Target encoder number of parameters: 0
+[INFO    ][2026-05-09 08:26:34][root                ][make_videodataset        ] VideoDataset dataset created
+[INFO    ][2026-05-09 08:26:34][WeightedSampler     ][__init__                 ] Using DistributedWeightedSampler with rank 1 / 16
+[INFO    ][2026-05-09 08:26:34][root                ][make_videodataset        ] VideoDataset unsupervised data loader created
+[INFO    ][2026-05-09 08:26:34][app.vjepa.train     ][main                     ] iterations per epoch/dataset length: 300/399
+[INFO    ][2026-05-09 08:26:34][app.vjepa.train     ][main                     ] Wrapping models in DDP (rank 1)...
+[INFO    ][2026-05-09 08:26:37][root                ][load_checkpoint          ] Loading checkpoint from /scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/latest.pt
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] loaded pretrained encoder from epoch 39 with msg: <All keys matched successfully>
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] loaded pretrained predictor from epoch 39 with msg: <All keys matched successfully>
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] loaded pretrained target encoder from epoch 39 with msg: <All keys matched successfully>
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] loaded optimizers from epoch 39
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] read-path: /scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/latest.pt
+[INFO    ][2026-05-09 08:26:39][app.vjepa.train     ][main                     ] Initializing loader...
+submitit WARNING (2026-05-09 09:56:00,845) - Bypassing signal SIGTERM
+[WARNING ][2026-05-09 09:56:00][submitit            ][bypass                   ] Bypassing signal SIGTERM
+submitit WARNING (2026-05-09 09:56:00,869) - Bypassing signal SIGCONT
+[WARNING ][2026-05-09 09:56:00][submitit            ][bypass                   ] Bypassing signal SIGCONT

vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/job_22595640/22595640_2_log.err

@@ -0,0 +1,322 @@
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
+  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
+[rank2]:[W508 20:21:43.769860827 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
+  scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+submitit WARNING (2026-05-09 08:16:14,171) - Caught signal SIGUSR2 on gcn75.local.snellius.surf.nl: this job is timed-out.
+submitit WARNING (2026-05-09 08:16:14,171) - Bypassing signal SIGCONT
+submitit WARNING (2026-05-09 08:16:14,372) - Bypassing signal SIGTERM
+submitit WARNING (2026-05-09 08:16:14,372) - Bypassing signal SIGCONT
+[rank2]:[W509 08:16:18.841505693 TCPStore.cpp:125] [c10d] recvValue failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43534, remote=[gcn75.local.snellius.surf.nl]:37129): Failed to recv, got 0 bytes. Connection was likely closed. Did the remote server shutdown or crash?
+Exception raised from recvBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:682 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14c6aea03fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a3325d (0x14c6f2b9225d in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x273 (0x14c6f2b901f3 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14c6afc72eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14c7ef1dc164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14c7ffc4e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14c7ffcd3100 in /lib64/libc.so.6)
+
+[rank2]:[W509 08:16:18.844897926 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 2] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Failed to recv, got 0 bytes. Connection was likely closed. Did the remote server shutdown or crash?
+[rank2]:[W509 08:16:19.845066067 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43534, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14c6aea03fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14c6f2b916d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14c6f2b901cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14c6afc72eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14c7ef1dc164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14c7ffc4e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14c7ffcd3100 in /lib64/libc.so.6)
+
+[rank2]:[W509 08:16:19.847430865 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 2] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank2]:[W509 08:16:20.847565357 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43534, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14c6aea03fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14c6f2b916d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14c6f2b901cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14c6afc72eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14c7ef1dc164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14c7ffc4e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14c7ffcd3100 in /lib64/libc.so.6)
+
+[rank2]:[W509 08:16:20.849975544 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 2] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank2]:[W509 08:16:21.850102856 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43534, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14c6aea03fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14c6f2b916d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14c6f2b901cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14c6afc72eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14c7ef1dc164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14c7ffc4e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14c7ffcd3100 in /lib64/libc.so.6)
+
+[rank2]:[W509 08:16:21.852511123 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 2] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank2]:[W509 08:16:22.852669914 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43534, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14c6aea03fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14c6f2b916d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14c6f2b901cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14c6afc72eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14c7ef1dc164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14c7ffc4e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14c7ffcd3100 in /lib64/libc.so.6)
+
+[rank2]:[W509 08:16:22.855059652 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 2] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank2]:[W509 08:16:23.855180004 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43534, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14c6aea03fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14c6f2b916d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14c6f2b901cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14c6afc72eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14c7ef1dc164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14c7ffc4e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14c7ffcd3100 in /lib64/libc.so.6)
+
+[rank2]:[W509 08:16:23.857600321 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 2] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank2]:[W509 08:16:24.857712833 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43534, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14c6aea03fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14c6f2b916d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14c6f2b901cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14c6afc72eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14c7ef1dc164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14c7ffc4e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14c7ffcd3100 in /lib64/libc.so.6)
+
+[rank2]:[W509 08:16:24.860060641 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 2] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank2]:[W509 08:16:25.860202282 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43534, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14c6aea03fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14c6f2b916d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14c6f2b901cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14c6afc72eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14c7ef1dc164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14c7ffc4e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14c7ffcd3100 in /lib64/libc.so.6)
+
+[rank2]:[W509 08:16:25.862591520 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 2] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank2]:[W509 08:16:26.862705261 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43534, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14c6aea03fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14c6f2b916d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14c6f2b901cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14c6afc72eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14c7ef1dc164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14c7ffc4e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14c7ffcd3100 in /lib64/libc.so.6)
+
+[rank2]:[W509 08:16:26.865109419 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 2] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank2]:[W509 08:16:27.865218980 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43534, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14c6aea03fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14c6f2b916d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14c6f2b901cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14c6afc72eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14c7ef1dc164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14c7ffc4e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14c7ffcd3100 in /lib64/libc.so.6)
+
+[rank2]:[W509 08:16:27.867460799 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 2] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank2]:[W509 08:16:28.867579140 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43534, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14c6aea03fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14c6f2b916d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14c6f2b901cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14c6afc72eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14c7ef1dc164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14c7ffc4e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14c7ffcd3100 in /lib64/libc.so.6)
+
+[rank2]:[W509 08:16:28.869928478 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 2] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank2]:[W509 08:16:29.870074569 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43534, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14c6aea03fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14c6f2b916d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14c6f2b901cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14c6afc72eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14c7ef1dc164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14c7ffc4e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14c7ffcd3100 in /lib64/libc.so.6)
+
+[rank2]:[W509 08:16:29.872529697 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 2] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank2]:[W509 08:16:30.872652188 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43534, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14c6aea03fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14c6f2b916d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14c6f2b901cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14c6afc72eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14c7ef1dc164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14c7ffc4e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14c7ffcd3100 in /lib64/libc.so.6)
+
+[rank2]:[W509 08:16:30.875039496 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 2] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank2]:[W509 08:16:31.875158308 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43534, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14c6aea03fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14c6f2b916d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14c6f2b901cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14c6afc72eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14c7ef1dc164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14c7ffc4e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14c7ffcd3100 in /lib64/libc.so.6)
+
+[rank2]:[W509 08:16:31.877404446 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 2] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank2]:[W509 08:16:32.877512347 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43534, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14c6aea03fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14c6f2b916d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14c6f2b901cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14c6afc72eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14c7ef1dc164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14c7ffc4e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14c7ffcd3100 in /lib64/libc.so.6)
+
+[rank2]:[W509 08:16:32.879846036 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 2] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank2]:[W509 08:16:33.879955987 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43534, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14c6aea03fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14c6f2b916d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14c6f2b901cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14c6afc72eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14c7ef1dc164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14c7ffc4e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14c7ffcd3100 in /lib64/libc.so.6)
+
+[rank2]:[W509 08:16:33.882258905 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 2] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank2]:[W509 08:16:34.882372397 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43534, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14c6aea03fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14c6f2b916d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14c6f2b901cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14c6afc72eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14c7ef1dc164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14c7ffc4e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14c7ffcd3100 in /lib64/libc.so.6)
+
+[rank2]:[W509 08:16:34.884734785 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 2] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank2]:[W509 08:16:35.884843346 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43534, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14c6aea03fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14c6f2b916d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14c6f2b901cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14c6afc72eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14c7ef1dc164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14c7ffc4e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14c7ffcd3100 in /lib64/libc.so.6)
+
+[rank2]:[W509 08:16:35.887221914 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 2] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank2]:[W509 08:16:36.887330375 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43534, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14c6aea03fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14c6f2b916d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14c6f2b901cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14c6afc72eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14c7ef1dc164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14c7ffc4e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14c7ffcd3100 in /lib64/libc.so.6)
+
+[rank2]:[W509 08:16:36.889648383 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 2] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank2]:[W509 08:16:37.889749095 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43534, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14c6aea03fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14c6f2b916d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14c6f2b901cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14c6afc72eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14c7ef1dc164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14c7ffc4e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14c7ffcd3100 in /lib64/libc.so.6)
+
+[rank2]:[W509 08:16:37.892055013 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 2] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank2]:[W509 08:16:38.892184135 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43534, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14c6aea03fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14c6f2b916d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14c6f2b901cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14c6afc72eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14c7ef1dc164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14c7ffc4e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14c7ffcd3100 in /lib64/libc.so.6)
+
+[rank2]:[W509 08:16:38.894608142 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 2] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank2]:[W509 08:16:39.894725224 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43534, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14c6aea03fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14c6f2b916d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14c6f2b901cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14c6afc72eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14c7ef1dc164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14c7ffc4e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14c7ffcd3100 in /lib64/libc.so.6)
+
+[rank2]:[W509 08:16:39.897088362 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 2] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank2]:[W509 08:16:40.897204313 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43534, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14c6aea03fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14c6f2b916d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14c6f2b901cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14c6afc72eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14c7ef1dc164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14c7ffc4e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14c7ffcd3100 in /lib64/libc.so.6)
+
+[rank2]:[W509 08:16:40.899506631 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 2] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank2]:[W509 08:16:41.899630222 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43534, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14c6aea03fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14c6f2b916d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14c6f2b901cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14c6afc72eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14c7ef1dc164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14c7ffc4e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14c7ffcd3100 in /lib64/libc.so.6)
+
+[rank2]:[W509 08:16:41.901983880 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 2] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank2]:[W509 08:16:42.902110432 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43534, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14c6aea03fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14c6f2b916d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14c6f2b901cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14c6afc72eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14c7ef1dc164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14c7ffc4e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14c7ffcd3100 in /lib64/libc.so.6)
+
+[rank2]:[W509 08:16:42.904458620 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 2] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank2]:[W509 08:16:43.904582471 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43534, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14c6aea03fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14c6f2b916d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14c6f2b901cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14c6afc72eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14c7ef1dc164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14c7ffc4e19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14c7ffcd3100 in /lib64/libc.so.6)
+
+[rank2]:[W509 08:16:43.906890999 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 2] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
+  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
+[rank2]:[W509 08:23:56.124194846 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
+  scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+submitit WARNING (2026-05-09 09:56:00,827) - Bypassing signal SIGTERM
+submitit WARNING (2026-05-09 09:56:00,852) - Bypassing signal SIGCONT

vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/job_22595640/22595640_2_log.out

@@ -0,0 +1,532 @@
+submitit INFO (2026-05-08 20:18:04,102) - Starting with JobEnvironment(job_id=22595640, hostname=gcn75.local.snellius.surf.nl, local_rank=2(4), node=0(4), global_rank=2(16))
+submitit INFO (2026-05-08 20:18:04,102) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/job_22595640/22595640_submitted.pkl
+INFO:root:loaded pretrain params...
+{   'app': 'vjepa',
+    'cpus_per_task': 16,
+    'data': {   'batch_size': 64,
+                'crop_size': 224,
+                'dataset_fpcs': [16, 16],
+                'dataset_type': 'VideoDataset',
+                'datasets': [   '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
+                                '/scratch-shared/dcanez/data/ssv2/train.csv'],
+                'datasets_weights': [0.65, 0.35],
+                'fps': 4,
+                'num_workers': 10,
+                'patch_size': 14,
+                'persistent_workers': True,
+                'pin_mem': True,
+                'stage': [   {   'dest': 'kinetics_240',
+                                 'format': 'targz_parts',
+                                 'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
+                             {   'dest': 'ssv2',
+                                 'format': 'multipart_tar',
+                                 'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
+                'tubelet_size': 1},
+    'data_aug': {   'auto_augment': False,
+                    'motion_shift': False,
+                    'random_resize_aspect_ratio': [0.75, 1.35],
+                    'random_resize_scale': [0.3, 1.0],
+                    'reprob': 0.0},
+    'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m',
+    'loss': {'loss_exp': 1.0},
+    'mask': [   {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 8,
+                    'spatial_scale': [0.15, 0.15],
+                    'temporal_scale': [1.0, 1.0]},
+                {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 2,
+                    'spatial_scale': [0.7, 0.7],
+                    'temporal_scale': [1.0, 1.0]}],
+    'mem_per_gpu': '180G',
+    'meta': {   'dtype': 'bfloat16',
+                'knn_eval_epoch0': False,
+                'knn_eval_freq': 5,
+                'knn_eval_presets': [   {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
+                                                          'eval_videos_per_class': 25,
+                                                          'num_workers': 8,
+                                                          'train_videos_per_class': 100},
+                                            'preset': 'ucf101'},
+                                        {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
+                                                          'eval_videos_per_class': 100,
+                                                          'linear_probe': True,
+                                                          'num_workers': 8,
+                                                          'pool_type': 'temporal_concat',
+                                                          'train_videos_per_class': 500},
+                                            'preset': 'ssv2_coarse10'}],
+                'load_checkpoint': True,
+                'read_checkpoint': None,
+                'save_every_freq': 5,
+                'seed': 239,
+                'use_sdpa': True,
+                'use_wandb': True,
+                'wandb_project': 'vjepa_ablation'},
+    'metrics': {'sigreg': {}, 'std': {}},
+    'model': {   'class_token': False,
+                 'freeze_backbone': True,
+                 'is_causal': False,
+                 'lora_alpha': 16,
+                 'lora_rank': 8,
+                 'lora_targets': ['attn.qkv', 'attn.proj'],
+                 'model_name': 'vit_large_patch14_capi_lvd1689m',
+                 'num_reg_tokens': 16,
+                 'pred_depth': 6,
+                 'pred_embed_dim': 384,
+                 'pred_num_heads': 12,
+                 'rope_type': 'capi2d',
+                 'stem_type': '2d',
+                 'target_kind': 'frozen_2d',
+                 'target_type': 'vit_large_patch14_capi.lvd1689m',
+                 'uniform_power': True,
+                 'use_activation_checkpointing': True,
+                 'use_lora': True,
+                 'use_mask_tokens': True,
+                 'use_rope': True,
+                 'use_sdpa': True,
+                 'zero_init_mask_tokens': True},
+    'nodes': 4,
+    'optimization': {   'ema': [0.99925, 0.99925],
+                        'epochs': 100,
+                        'final_lr': 0.0001,
+                        'final_weight_decay': 0.04,
+                        'ipe': 300,
+                        'ipe_scale': 1.0,
+                        'lr': 0.0005,
+                        'start_lr': 0.0001,
+                        'warmup': 10,
+                        'weight_decay': 0.04},
+    'tasks_per_node': 4}
+INFO:root:Running pre-training of app: vjepa
+[INFO    ][2026-05-08 20:18:35][app.vjepa.train     ][main                     ] which_dtype='bfloat16'
+[INFO    ][2026-05-08 20:18:35][app.vjepa.train     ][main                     ] Disabling persistent_workers (incompatible with KNN eval)
+[INFO    ][2026-05-08 20:18:35][app.vjepa.train     ][main                     ] NCCL_SOCKET_IFNAME=eno
+[INFO    ][2026-05-08 20:18:38][app.vjepa.train     ][main                     ] Initialized (rank/world-size) 2/16, tasks_per_node=4
+[INFO    ][2026-05-08 20:18:38][root                ][stage_datasets           ] [local_rank 2/4] Staging kinetics_240 (targz_parts)
+[INFO    ][2026-05-08 20:18:38][root                ][_stage_targz_parts       ] [rank 2] Extracting 26/103 tar.gz parts to /scratch-node/dcanez.22595640/kinetics_240
+[INFO    ][2026-05-08 20:18:52][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 2/26 parts
+[INFO    ][2026-05-08 20:19:06][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 4/26 parts
+[INFO    ][2026-05-08 20:19:20][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 6/26 parts
+[INFO    ][2026-05-08 20:19:34][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 8/26 parts
+[INFO    ][2026-05-08 20:19:49][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 10/26 parts
+[INFO    ][2026-05-08 20:20:03][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 12/26 parts
+[INFO    ][2026-05-08 20:20:17][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 14/26 parts
+[INFO    ][2026-05-08 20:20:31][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 16/26 parts
+[INFO    ][2026-05-08 20:20:46][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 18/26 parts
+[INFO    ][2026-05-08 20:21:00][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 20/26 parts
+[INFO    ][2026-05-08 20:21:14][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 22/26 parts
+[INFO    ][2026-05-08 20:21:29][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 24/26 parts
+[INFO    ][2026-05-08 20:21:43][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 26/26 parts
+[INFO    ][2026-05-08 20:21:43][root                ][stage_datasets           ] [local_rank 2/4] Staging ssv2 (multipart_tar)
+[INFO    ][2026-05-08 20:23:17][app.vjepa.train     ][main                     ] Staged datasets: ['/scratch-node/dcanez.22595640/kinetics_240/train.csv', '/scratch-node/dcanez.22595640/ssv2/train.csv']
+[INFO    ][2026-05-08 20:23:59][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
+[INFO    ][2026-05-08 20:24:03][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA applied: 48 modules replaced (rank=8, alpha=16, dropout=0.0, targets=['attn.qkv', 'attn.proj'], scope=blocks)
+[INFO    ][2026-05-08 20:24:03][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA trainable params: 1,179,648
+[INFO    ][2026-05-08 20:24:06][root                ][_build_frozen_2d_target  ] Loaded frozen_2d target from timm: vit_large_patch14_capi.lvd1689m
+[INFO    ][2026-05-08 20:24:06][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-08 20:24:06][root                ][init_video_model         ] VJEPAPredictorMultiSeqWrapper(
+  (backbone): VisionTransformerPredictor(
+    (predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
+    (mask_tokens): ParameterList(
+        (0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+    )
+    (predictor_blocks): ModuleList(
+      (0-5): 6 x Block(
+        (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (attn): RoPEAttention(
+          (qkv): Linear(in_features=384, out_features=1152, bias=True)
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): Linear(in_features=384, out_features=384, bias=True)
+          (proj_drop): Dropout(p=0.0, inplace=False)
+        )
+        (drop_path): Identity()
+        (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (mlp): MLP(
+          (fc1): Linear(in_features=384, out_features=1536, bias=True)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=1536, out_features=384, bias=True)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+    (predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
+  )
+)
+[INFO    ][2026-05-08 20:24:06][root                ][init_video_model         ] MultiSeqWrapper(
+  (backbone): Frozen2DTargetWrapper(
+    (backbone): Eva(
+      (patch_embed): PatchEmbed(
+        (proj): Conv2d(3, 1024, kernel_size=(14, 14), stride=(14, 14))
+        (norm): Identity()
+      )
+      (pos_drop): Dropout(p=0.0, inplace=False)
+      (norm_pre): Identity()
+      (blocks): ModuleList(
+        (0-23): 24 x EvaBlock(
+          (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+          (attn): EvaAttention(
+            (qkv): Linear(in_features=1024, out_features=3072, bias=False)
+            (q_norm): Identity()
+            (k_norm): Identity()
+            (attn_drop): Dropout(p=0.0, inplace=False)
+            (norm): Identity()
+            (proj): Linear(in_features=1024, out_features=1024, bias=True)
+            (proj_drop): Dropout(p=0.0, inplace=False)
+          )
+          (drop_path1): Identity()
+          (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+          (mlp): Mlp(
+            (fc1): Linear(in_features=1024, out_features=4096, bias=True)
+            (act): GELU(approximate='none')
+            (drop1): Dropout(p=0.0, inplace=False)
+            (norm): Identity()
+            (fc2): Linear(in_features=4096, out_features=1024, bias=True)
+            (drop2): Dropout(p=0.0, inplace=False)
+          )
+          (drop_path2): Identity()
+        )
+      )
+      (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+      (fc_norm): Identity()
+      (head_drop): Dropout(p=0.0, inplace=False)
+      (head): Identity()
+      (rope): _CapiPatchRoPE()
+    )
+  )
+)
+[INFO    ][2026-05-08 20:24:06][root                ][init_video_model         ] Encoder number of parameters: 1179648
+[INFO    ][2026-05-08 20:24:06][root                ][init_video_model         ] Predictor number of parameters: 11436928
+[INFO    ][2026-05-08 20:24:06][root                ][init_video_model         ] Target encoder number of parameters: 0
+[INFO    ][2026-05-08 20:24:08][root                ][make_videodataset        ] VideoDataset dataset created
+[INFO    ][2026-05-08 20:24:08][WeightedSampler     ][__init__                 ] Using DistributedWeightedSampler with rank 2 / 16
+[INFO    ][2026-05-08 20:24:08][root                ][make_videodataset        ] VideoDataset unsupervised data loader created
+[INFO    ][2026-05-08 20:24:08][app.vjepa.train     ][main                     ] iterations per epoch/dataset length: 300/399
+[INFO    ][2026-05-08 20:24:08][app.vjepa.train     ][main                     ] Wrapping models in DDP (rank 2)...
+[INFO    ][2026-05-08 20:24:41][app.vjepa.train     ][main                     ] Initializing loader...
+submitit INFO (2026-05-09 08:16:14,146) - Job has timed out. Ran 718 minutes out of requested 720 minutes.
+[INFO    ][2026-05-09 08:16:14][submitit            ][has_timed_out            ] Job has timed out. Ran 718 minutes out of requested 720 minutes.
+submitit WARNING (2026-05-09 08:16:14,171) - Caught signal SIGUSR2 on gcn75.local.snellius.surf.nl: this job is timed-out.
+[WARNING ][2026-05-09 08:16:14][submitit            ][checkpoint_and_try_requeue] Caught signal SIGUSR2 on gcn75.local.snellius.surf.nl: this job is timed-out.
+submitit INFO (2026-05-09 08:16:14,171) - Not checkpointing nor requeuing since I am a slave (procid=2).
+[INFO    ][2026-05-09 08:16:14][submitit            ][checkpoint_and_try_requeue] Not checkpointing nor requeuing since I am a slave (procid=2).
+submitit WARNING (2026-05-09 08:16:14,171) - Bypassing signal SIGCONT
+[WARNING ][2026-05-09 08:16:14][submitit            ][bypass                   ] Bypassing signal SIGCONT
+submitit WARNING (2026-05-09 08:16:14,372) - Bypassing signal SIGTERM
+[WARNING ][2026-05-09 08:16:14][submitit            ][bypass                   ] Bypassing signal SIGTERM
+submitit WARNING (2026-05-09 08:16:14,372) - Bypassing signal SIGCONT
+[WARNING ][2026-05-09 08:16:14][submitit            ][bypass                   ] Bypassing signal SIGCONT
+submitit INFO (2026-05-09 08:19:47,093) - Starting with JobEnvironment(job_id=22595640, hostname=gcn73.local.snellius.surf.nl, local_rank=2(4), node=0(4), global_rank=2(16))
+submitit INFO (2026-05-09 08:19:47,106) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/job_22595640/22595640_submitted.pkl
+INFO:root:loaded pretrain params...
+{   'app': 'vjepa',
+    'cpus_per_task': 16,
+    'data': {   'batch_size': 64,
+                'crop_size': 224,
+                'dataset_fpcs': [16, 16],
+                'dataset_type': 'VideoDataset',
+                'datasets': [   '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
+                                '/scratch-shared/dcanez/data/ssv2/train.csv'],
+                'datasets_weights': [0.65, 0.35],
+                'fps': 4,
+                'num_workers': 10,
+                'patch_size': 14,
+                'persistent_workers': True,
+                'pin_mem': True,
+                'stage': [   {   'dest': 'kinetics_240',
+                                 'format': 'targz_parts',
+                                 'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
+                             {   'dest': 'ssv2',
+                                 'format': 'multipart_tar',
+                                 'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
+                'tubelet_size': 1},
+    'data_aug': {   'auto_augment': False,
+                    'motion_shift': False,
+                    'random_resize_aspect_ratio': [0.75, 1.35],
+                    'random_resize_scale': [0.3, 1.0],
+                    'reprob': 0.0},
+    'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m',
+    'loss': {'loss_exp': 1.0},
+    'mask': [   {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 8,
+                    'spatial_scale': [0.15, 0.15],
+                    'temporal_scale': [1.0, 1.0]},
+                {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 2,
+                    'spatial_scale': [0.7, 0.7],
+                    'temporal_scale': [1.0, 1.0]}],
+    'mem_per_gpu': '180G',
+    'meta': {   'dtype': 'bfloat16',
+                'knn_eval_epoch0': False,
+                'knn_eval_freq': 5,
+                'knn_eval_presets': [   {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
+                                                          'eval_videos_per_class': 25,
+                                                          'num_workers': 8,
+                                                          'train_videos_per_class': 100},
+                                            'preset': 'ucf101'},
+                                        {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
+                                                          'eval_videos_per_class': 100,
+                                                          'linear_probe': True,
+                                                          'num_workers': 8,
+                                                          'pool_type': 'temporal_concat',
+                                                          'train_videos_per_class': 500},
+                                            'preset': 'ssv2_coarse10'}],
+                'load_checkpoint': True,
+                'read_checkpoint': None,
+                'save_every_freq': 5,
+                'seed': 239,
+                'use_sdpa': True,
+                'use_wandb': True,
+                'wandb_project': 'vjepa_ablation'},
+    'metrics': {'sigreg': {}, 'std': {}},
+    'model': {   'class_token': False,
+                 'freeze_backbone': True,
+                 'is_causal': False,
+                 'lora_alpha': 16,
+                 'lora_rank': 8,
+                 'lora_targets': ['attn.qkv', 'attn.proj'],
+                 'model_name': 'vit_large_patch14_capi_lvd1689m',
+                 'num_reg_tokens': 16,
+                 'pred_depth': 6,
+                 'pred_embed_dim': 384,
+                 'pred_num_heads': 12,
+                 'rope_type': 'capi2d',
+                 'stem_type': '2d',
+                 'target_kind': 'frozen_2d',
+                 'target_type': 'vit_large_patch14_capi.lvd1689m',
+                 'uniform_power': True,
+                 'use_activation_checkpointing': True,
+                 'use_lora': True,
+                 'use_mask_tokens': True,
+                 'use_rope': True,
+                 'use_sdpa': True,
+                 'zero_init_mask_tokens': True},
+    'nodes': 4,
+    'optimization': {   'ema': [0.99925, 0.99925],
+                        'epochs': 100,
+                        'final_lr': 0.0001,
+                        'final_weight_decay': 0.04,
+                        'ipe': 300,
+                        'ipe_scale': 1.0,
+                        'lr': 0.0005,
+                        'start_lr': 0.0001,
+                        'warmup': 10,
+                        'weight_decay': 0.04},
+    'tasks_per_node': 4}
+INFO:root:Running pre-training of app: vjepa
+[INFO    ][2026-05-09 08:20:49][app.vjepa.train     ][main                     ] which_dtype='bfloat16'
+[INFO    ][2026-05-09 08:20:49][app.vjepa.train     ][main                     ] Disabling persistent_workers (incompatible with KNN eval)
+[INFO    ][2026-05-09 08:20:49][app.vjepa.train     ][main                     ] NCCL_SOCKET_IFNAME=eno
+[INFO    ][2026-05-09 08:20:51][app.vjepa.train     ][main                     ] Initialized (rank/world-size) 2/16, tasks_per_node=4
+[INFO    ][2026-05-09 08:20:51][root                ][stage_datasets           ] [local_rank 2/4] Staging kinetics_240 (targz_parts)
+[INFO    ][2026-05-09 08:20:51][root                ][_stage_targz_parts       ] [rank 2] Extracting 26/103 tar.gz parts to /scratch-node/dcanez.22595640/kinetics_240
+[INFO    ][2026-05-09 08:21:05][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 2/26 parts
+[INFO    ][2026-05-09 08:21:19][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 4/26 parts
+[INFO    ][2026-05-09 08:21:33][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 6/26 parts
+[INFO    ][2026-05-09 08:21:48][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 8/26 parts
+[INFO    ][2026-05-09 08:22:02][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 10/26 parts
+[INFO    ][2026-05-09 08:22:16][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 12/26 parts
+[INFO    ][2026-05-09 08:22:30][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 14/26 parts
+[INFO    ][2026-05-09 08:22:45][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 16/26 parts
+[INFO    ][2026-05-09 08:22:59][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 18/26 parts
+[INFO    ][2026-05-09 08:23:13][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 20/26 parts
+[INFO    ][2026-05-09 08:23:28][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 22/26 parts
+[INFO    ][2026-05-09 08:23:42][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 24/26 parts
+[INFO    ][2026-05-09 08:23:56][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 26/26 parts
+[INFO    ][2026-05-09 08:23:56][root                ][stage_datasets           ] [local_rank 2/4] Staging ssv2 (multipart_tar)
+[INFO    ][2026-05-09 08:25:33][app.vjepa.train     ][main                     ] Staged datasets: ['/scratch-node/dcanez.22595640/kinetics_240/train.csv', '/scratch-node/dcanez.22595640/ssv2/train.csv']
+[INFO    ][2026-05-09 08:26:29][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
+[INFO    ][2026-05-09 08:26:30][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA applied: 48 modules replaced (rank=8, alpha=16, dropout=0.0, targets=['attn.qkv', 'attn.proj'], scope=blocks)
+[INFO    ][2026-05-09 08:26:30][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA trainable params: 1,179,648
+[INFO    ][2026-05-09 08:26:33][root                ][_build_frozen_2d_target  ] Loaded frozen_2d target from timm: vit_large_patch14_capi.lvd1689m
+[INFO    ][2026-05-09 08:26:33][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-09 08:26:33][root                ][init_video_model         ] VJEPAPredictorMultiSeqWrapper(
+  (backbone): VisionTransformerPredictor(
+    (predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
+    (mask_tokens): ParameterList(
+        (0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+    )
+    (predictor_blocks): ModuleList(
+      (0-5): 6 x Block(
+        (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (attn): RoPEAttention(
+          (qkv): Linear(in_features=384, out_features=1152, bias=True)
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): Linear(in_features=384, out_features=384, bias=True)
+          (proj_drop): Dropout(p=0.0, inplace=False)
+        )
+        (drop_path): Identity()
+        (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (mlp): MLP(
+          (fc1): Linear(in_features=384, out_features=1536, bias=True)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=1536, out_features=384, bias=True)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+    (predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
+  )
+)
+[INFO    ][2026-05-09 08:26:33][root                ][init_video_model         ] MultiSeqWrapper(
+  (backbone): Frozen2DTargetWrapper(
+    (backbone): Eva(
+      (patch_embed): PatchEmbed(
+        (proj): Conv2d(3, 1024, kernel_size=(14, 14), stride=(14, 14))
+        (norm): Identity()
+      )
+      (pos_drop): Dropout(p=0.0, inplace=False)
+      (norm_pre): Identity()
+      (blocks): ModuleList(
+        (0-23): 24 x EvaBlock(
+          (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+          (attn): EvaAttention(
+            (qkv): Linear(in_features=1024, out_features=3072, bias=False)
+            (q_norm): Identity()
+            (k_norm): Identity()
+            (attn_drop): Dropout(p=0.0, inplace=False)
+            (norm): Identity()
+            (proj): Linear(in_features=1024, out_features=1024, bias=True)
+            (proj_drop): Dropout(p=0.0, inplace=False)
+          )
+          (drop_path1): Identity()
+          (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+          (mlp): Mlp(
+            (fc1): Linear(in_features=1024, out_features=4096, bias=True)
+            (act): GELU(approximate='none')
+            (drop1): Dropout(p=0.0, inplace=False)
+            (norm): Identity()
+            (fc2): Linear(in_features=4096, out_features=1024, bias=True)
+            (drop2): Dropout(p=0.0, inplace=False)
+          )
+          (drop_path2): Identity()
+        )
+      )
+      (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+      (fc_norm): Identity()
+      (head_drop): Dropout(p=0.0, inplace=False)
+      (head): Identity()
+      (rope): _CapiPatchRoPE()
+    )
+  )
+)
+[INFO    ][2026-05-09 08:26:33][root                ][init_video_model         ] Encoder number of parameters: 1179648
+[INFO    ][2026-05-09 08:26:33][root                ][init_video_model         ] Predictor number of parameters: 11436928
+[INFO    ][2026-05-09 08:26:33][root                ][init_video_model         ] Target encoder number of parameters: 0
+[INFO    ][2026-05-09 08:26:34][root                ][make_videodataset        ] VideoDataset dataset created
+[INFO    ][2026-05-09 08:26:34][WeightedSampler     ][__init__                 ] Using DistributedWeightedSampler with rank 2 / 16
+[INFO    ][2026-05-09 08:26:34][root                ][make_videodataset        ] VideoDataset unsupervised data loader created
+[INFO    ][2026-05-09 08:26:34][app.vjepa.train     ][main                     ] iterations per epoch/dataset length: 300/399
+[INFO    ][2026-05-09 08:26:34][app.vjepa.train     ][main                     ] Wrapping models in DDP (rank 2)...
+[INFO    ][2026-05-09 08:26:37][root                ][load_checkpoint          ] Loading checkpoint from /scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/latest.pt
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] loaded pretrained encoder from epoch 39 with msg: <All keys matched successfully>
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] loaded pretrained predictor from epoch 39 with msg: <All keys matched successfully>
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] loaded pretrained target encoder from epoch 39 with msg: <All keys matched successfully>
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] loaded optimizers from epoch 39
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] read-path: /scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/latest.pt
+[INFO    ][2026-05-09 08:26:39][app.vjepa.train     ][main                     ] Initializing loader...
+submitit WARNING (2026-05-09 09:56:00,827) - Bypassing signal SIGTERM
+[WARNING ][2026-05-09 09:56:00][submitit            ][bypass                   ] Bypassing signal SIGTERM
+submitit WARNING (2026-05-09 09:56:00,852) - Bypassing signal SIGCONT
+[WARNING ][2026-05-09 09:56:00][submitit            ][bypass                   ] Bypassing signal SIGCONT

vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/job_22595640/22595640_3_log.err

@@ -0,0 +1,322 @@
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
+  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
+[rank3]:[W508 20:21:37.872149564 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
+  scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+submitit WARNING (2026-05-09 08:16:14,207) - Caught signal SIGUSR2 on gcn75.local.snellius.surf.nl: this job is timed-out.
+submitit WARNING (2026-05-09 08:16:14,207) - Bypassing signal SIGCONT
+submitit WARNING (2026-05-09 08:16:14,372) - Bypassing signal SIGTERM
+submitit WARNING (2026-05-09 08:16:14,372) - Bypassing signal SIGCONT
+[rank3]:[W509 08:16:18.841287615 TCPStore.cpp:125] [c10d] recvValue failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43548, remote=[gcn75.local.snellius.surf.nl]:37129): Failed to recv, got 0 bytes. Connection was likely closed. Did the remote server shutdown or crash?
+Exception raised from recvBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:682 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14683221ffdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a3325d (0x1468763ae25d in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x273 (0x1468763ac1f3 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14683348eeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1469729f8164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14698346a19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1469834ef100 in /lib64/libc.so.6)
+
+[rank3]:[W509 08:16:18.844448178 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 3] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Failed to recv, got 0 bytes. Connection was likely closed. Did the remote server shutdown or crash?
+[rank3]:[W509 08:16:19.844603130 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43548, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14683221ffdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1468763ad6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1468763ac1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14683348eeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1469729f8164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14698346a19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1469834ef100 in /lib64/libc.so.6)
+
+[rank3]:[W509 08:16:19.846984828 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 3] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank3]:[W509 08:16:20.847133699 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43548, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14683221ffdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1468763ad6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1468763ac1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14683348eeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1469729f8164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14698346a19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1469834ef100 in /lib64/libc.so.6)
+
+[rank3]:[W509 08:16:20.849578106 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 3] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank3]:[W509 08:16:21.849715018 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43548, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14683221ffdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1468763ad6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1468763ac1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14683348eeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1469729f8164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14698346a19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1469834ef100 in /lib64/libc.so.6)
+
+[rank3]:[W509 08:16:21.852195325 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 3] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank3]:[W509 08:16:22.852358416 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43548, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14683221ffdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1468763ad6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1468763ac1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14683348eeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1469729f8164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14698346a19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1469834ef100 in /lib64/libc.so.6)
+
+[rank3]:[W509 08:16:22.854762384 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 3] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank3]:[W509 08:16:23.854896915 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43548, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14683221ffdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1468763ad6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1468763ac1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14683348eeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1469729f8164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14698346a19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1469834ef100 in /lib64/libc.so.6)
+
+[rank3]:[W509 08:16:23.857356733 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 3] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank3]:[W509 08:16:24.857486454 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43548, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14683221ffdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1468763ad6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1468763ac1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14683348eeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1469729f8164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14698346a19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1469834ef100 in /lib64/libc.so.6)
+
+[rank3]:[W509 08:16:24.859858322 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 3] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank3]:[W509 08:16:25.860029583 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43548, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14683221ffdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1468763ad6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1468763ac1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14683348eeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1469729f8164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14698346a19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1469834ef100 in /lib64/libc.so.6)
+
+[rank3]:[W509 08:16:25.862431421 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 3] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank3]:[W509 08:16:26.862567912 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43548, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14683221ffdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1468763ad6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1468763ac1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14683348eeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1469729f8164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14698346a19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1469834ef100 in /lib64/libc.so.6)
+
+[rank3]:[W509 08:16:26.864993050 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 3] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank3]:[W509 08:16:27.865132641 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43548, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14683221ffdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1468763ad6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1468763ac1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14683348eeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1469729f8164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14698346a19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1469834ef100 in /lib64/libc.so.6)
+
+[rank3]:[W509 08:16:27.867392339 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 3] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank3]:[W509 08:16:28.867534811 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43548, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14683221ffdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1468763ad6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1468763ac1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14683348eeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1469729f8164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14698346a19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1469834ef100 in /lib64/libc.so.6)
+
+[rank3]:[W509 08:16:28.869883898 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 3] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank3]:[W509 08:16:29.870050040 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43548, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14683221ffdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1468763ad6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1468763ac1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14683348eeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1469729f8164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14698346a19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1469834ef100 in /lib64/libc.so.6)
+
+[rank3]:[W509 08:16:29.872483367 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 3] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank3]:[W509 08:16:30.872616349 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43548, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14683221ffdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1468763ad6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1468763ac1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14683348eeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1469729f8164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14698346a19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1469834ef100 in /lib64/libc.so.6)
+
+[rank3]:[W509 08:16:30.875039176 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 3] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank3]:[W509 08:16:31.875187737 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43548, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14683221ffdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1468763ad6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1468763ac1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14683348eeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1469729f8164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14698346a19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1469834ef100 in /lib64/libc.so.6)
+
+[rank3]:[W509 08:16:31.877538925 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 3] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank3]:[W509 08:16:32.877671807 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43548, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14683221ffdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1468763ad6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1468763ac1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14683348eeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1469729f8164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14698346a19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1469834ef100 in /lib64/libc.so.6)
+
+[rank3]:[W509 08:16:32.880098994 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 3] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank3]:[W509 08:16:33.880231946 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43548, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14683221ffdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1468763ad6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1468763ac1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14683348eeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1469729f8164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14698346a19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1469834ef100 in /lib64/libc.so.6)
+
+[rank3]:[W509 08:16:33.882599434 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 3] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank3]:[W509 08:16:34.882746635 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43548, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14683221ffdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1468763ad6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1468763ac1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14683348eeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1469729f8164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14698346a19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1469834ef100 in /lib64/libc.so.6)
+
+[rank3]:[W509 08:16:34.885135383 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 3] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank3]:[W509 08:16:35.885269394 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43548, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14683221ffdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1468763ad6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1468763ac1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14683348eeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1469729f8164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14698346a19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1469834ef100 in /lib64/libc.so.6)
+
+[rank3]:[W509 08:16:35.888289619 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 3] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank3]:[W509 08:16:36.888434590 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43548, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14683221ffdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1468763ad6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1468763ac1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14683348eeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1469729f8164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14698346a19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1469834ef100 in /lib64/libc.so.6)
+
+[rank3]:[W509 08:16:36.890799568 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 3] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank3]:[W509 08:16:37.890931759 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43548, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14683221ffdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1468763ad6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1468763ac1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14683348eeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1469729f8164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14698346a19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1469834ef100 in /lib64/libc.so.6)
+
+[rank3]:[W509 08:16:37.893241957 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 3] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank3]:[W509 08:16:38.893370809 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43548, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14683221ffdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1468763ad6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1468763ac1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14683348eeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1469729f8164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14698346a19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1469834ef100 in /lib64/libc.so.6)
+
+[rank3]:[W509 08:16:38.895786106 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 3] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank3]:[W509 08:16:39.895925608 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43548, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14683221ffdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1468763ad6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1468763ac1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14683348eeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1469729f8164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14698346a19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1469834ef100 in /lib64/libc.so.6)
+
+[rank3]:[W509 08:16:39.898186576 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 3] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank3]:[W509 08:16:40.898312837 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43548, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14683221ffdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1468763ad6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1468763ac1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14683348eeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1469729f8164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14698346a19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1469834ef100 in /lib64/libc.so.6)
+
+[rank3]:[W509 08:16:40.900601146 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 3] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank3]:[W509 08:16:41.900726747 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43548, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14683221ffdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1468763ad6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1468763ac1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14683348eeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1469729f8164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14698346a19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1469834ef100 in /lib64/libc.so.6)
+
+[rank3]:[W509 08:16:41.903076085 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 3] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank3]:[W509 08:16:42.903205836 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43548, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14683221ffdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1468763ad6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1468763ac1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14683348eeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1469729f8164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14698346a19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1469834ef100 in /lib64/libc.so.6)
+
+[rank3]:[W509 08:16:42.905599304 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 3] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank3]:[W509 08:16:43.905743045 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn75.local.snellius.surf.nl]:43548, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14683221ffdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1468763ad6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1468763ac1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14683348eeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x1469729f8164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14698346a19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1469834ef100 in /lib64/libc.so.6)
+
+[rank3]:[W509 08:16:43.907987184 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 3] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
+  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
+[rank3]:[W509 08:23:51.348379780 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
+  scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+submitit WARNING (2026-05-09 09:56:00,846) - Bypassing signal SIGTERM
+submitit WARNING (2026-05-09 09:56:00,992) - Bypassing signal SIGCONT

vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/job_22595640/22595640_3_log.out

@@ -0,0 +1,532 @@
+submitit INFO (2026-05-08 20:18:04,102) - Starting with JobEnvironment(job_id=22595640, hostname=gcn75.local.snellius.surf.nl, local_rank=3(4), node=0(4), global_rank=3(16))
+submitit INFO (2026-05-08 20:18:04,103) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/job_22595640/22595640_submitted.pkl
+INFO:root:loaded pretrain params...
+{   'app': 'vjepa',
+    'cpus_per_task': 16,
+    'data': {   'batch_size': 64,
+                'crop_size': 224,
+                'dataset_fpcs': [16, 16],
+                'dataset_type': 'VideoDataset',
+                'datasets': [   '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
+                                '/scratch-shared/dcanez/data/ssv2/train.csv'],
+                'datasets_weights': [0.65, 0.35],
+                'fps': 4,
+                'num_workers': 10,
+                'patch_size': 14,
+                'persistent_workers': True,
+                'pin_mem': True,
+                'stage': [   {   'dest': 'kinetics_240',
+                                 'format': 'targz_parts',
+                                 'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
+                             {   'dest': 'ssv2',
+                                 'format': 'multipart_tar',
+                                 'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
+                'tubelet_size': 1},
+    'data_aug': {   'auto_augment': False,
+                    'motion_shift': False,
+                    'random_resize_aspect_ratio': [0.75, 1.35],
+                    'random_resize_scale': [0.3, 1.0],
+                    'reprob': 0.0},
+    'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m',
+    'loss': {'loss_exp': 1.0},
+    'mask': [   {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 8,
+                    'spatial_scale': [0.15, 0.15],
+                    'temporal_scale': [1.0, 1.0]},
+                {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 2,
+                    'spatial_scale': [0.7, 0.7],
+                    'temporal_scale': [1.0, 1.0]}],
+    'mem_per_gpu': '180G',
+    'meta': {   'dtype': 'bfloat16',
+                'knn_eval_epoch0': False,
+                'knn_eval_freq': 5,
+                'knn_eval_presets': [   {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
+                                                          'eval_videos_per_class': 25,
+                                                          'num_workers': 8,
+                                                          'train_videos_per_class': 100},
+                                            'preset': 'ucf101'},
+                                        {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
+                                                          'eval_videos_per_class': 100,
+                                                          'linear_probe': True,
+                                                          'num_workers': 8,
+                                                          'pool_type': 'temporal_concat',
+                                                          'train_videos_per_class': 500},
+                                            'preset': 'ssv2_coarse10'}],
+                'load_checkpoint': True,
+                'read_checkpoint': None,
+                'save_every_freq': 5,
+                'seed': 239,
+                'use_sdpa': True,
+                'use_wandb': True,
+                'wandb_project': 'vjepa_ablation'},
+    'metrics': {'sigreg': {}, 'std': {}},
+    'model': {   'class_token': False,
+                 'freeze_backbone': True,
+                 'is_causal': False,
+                 'lora_alpha': 16,
+                 'lora_rank': 8,
+                 'lora_targets': ['attn.qkv', 'attn.proj'],
+                 'model_name': 'vit_large_patch14_capi_lvd1689m',
+                 'num_reg_tokens': 16,
+                 'pred_depth': 6,
+                 'pred_embed_dim': 384,
+                 'pred_num_heads': 12,
+                 'rope_type': 'capi2d',
+                 'stem_type': '2d',
+                 'target_kind': 'frozen_2d',
+                 'target_type': 'vit_large_patch14_capi.lvd1689m',
+                 'uniform_power': True,
+                 'use_activation_checkpointing': True,
+                 'use_lora': True,
+                 'use_mask_tokens': True,
+                 'use_rope': True,
+                 'use_sdpa': True,
+                 'zero_init_mask_tokens': True},
+    'nodes': 4,
+    'optimization': {   'ema': [0.99925, 0.99925],
+                        'epochs': 100,
+                        'final_lr': 0.0001,
+                        'final_weight_decay': 0.04,
+                        'ipe': 300,
+                        'ipe_scale': 1.0,
+                        'lr': 0.0005,
+                        'start_lr': 0.0001,
+                        'warmup': 10,
+                        'weight_decay': 0.04},
+    'tasks_per_node': 4}
+INFO:root:Running pre-training of app: vjepa
+[INFO    ][2026-05-08 20:18:35][app.vjepa.train     ][main                     ] which_dtype='bfloat16'
+[INFO    ][2026-05-08 20:18:35][app.vjepa.train     ][main                     ] Disabling persistent_workers (incompatible with KNN eval)
+[INFO    ][2026-05-08 20:18:35][app.vjepa.train     ][main                     ] NCCL_SOCKET_IFNAME=eno
+[INFO    ][2026-05-08 20:18:37][app.vjepa.train     ][main                     ] Initialized (rank/world-size) 3/16, tasks_per_node=4
+[INFO    ][2026-05-08 20:18:37][root                ][stage_datasets           ] [local_rank 3/4] Staging kinetics_240 (targz_parts)
+[INFO    ][2026-05-08 20:18:37][root                ][_stage_targz_parts       ] [rank 3] Extracting 25/103 tar.gz parts to /scratch-node/dcanez.22595640/kinetics_240
+[INFO    ][2026-05-08 20:18:51][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 2/25 parts
+[INFO    ][2026-05-08 20:19:06][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 4/25 parts
+[INFO    ][2026-05-08 20:19:20][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 6/25 parts
+[INFO    ][2026-05-08 20:19:34][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 8/25 parts
+[INFO    ][2026-05-08 20:19:49][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 10/25 parts
+[INFO    ][2026-05-08 20:20:03][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 12/25 parts
+[INFO    ][2026-05-08 20:20:18][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 14/25 parts
+[INFO    ][2026-05-08 20:20:32][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 16/25 parts
+[INFO    ][2026-05-08 20:20:46][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 18/25 parts
+[INFO    ][2026-05-08 20:21:01][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 20/25 parts
+[INFO    ][2026-05-08 20:21:16][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 22/25 parts
+[INFO    ][2026-05-08 20:21:30][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 24/25 parts
+[INFO    ][2026-05-08 20:21:37][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 25/25 parts
+[INFO    ][2026-05-08 20:21:37][root                ][stage_datasets           ] [local_rank 3/4] Staging ssv2 (multipart_tar)
+[INFO    ][2026-05-08 20:23:17][app.vjepa.train     ][main                     ] Staged datasets: ['/scratch-node/dcanez.22595640/kinetics_240/train.csv', '/scratch-node/dcanez.22595640/ssv2/train.csv']
+[INFO    ][2026-05-08 20:23:59][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
+[INFO    ][2026-05-08 20:24:03][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA applied: 48 modules replaced (rank=8, alpha=16, dropout=0.0, targets=['attn.qkv', 'attn.proj'], scope=blocks)
+[INFO    ][2026-05-08 20:24:03][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA trainable params: 1,179,648
+[INFO    ][2026-05-08 20:24:06][root                ][_build_frozen_2d_target  ] Loaded frozen_2d target from timm: vit_large_patch14_capi.lvd1689m
+[INFO    ][2026-05-08 20:24:06][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-08 20:24:06][root                ][init_video_model         ] VJEPAPredictorMultiSeqWrapper(
+  (backbone): VisionTransformerPredictor(
+    (predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
+    (mask_tokens): ParameterList(
+        (0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+    )
+    (predictor_blocks): ModuleList(
+      (0-5): 6 x Block(
+        (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (attn): RoPEAttention(
+          (qkv): Linear(in_features=384, out_features=1152, bias=True)
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): Linear(in_features=384, out_features=384, bias=True)
+          (proj_drop): Dropout(p=0.0, inplace=False)
+        )
+        (drop_path): Identity()
+        (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (mlp): MLP(
+          (fc1): Linear(in_features=384, out_features=1536, bias=True)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=1536, out_features=384, bias=True)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+    (predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
+  )
+)
+[INFO    ][2026-05-08 20:24:06][root                ][init_video_model         ] MultiSeqWrapper(
+  (backbone): Frozen2DTargetWrapper(
+    (backbone): Eva(
+      (patch_embed): PatchEmbed(
+        (proj): Conv2d(3, 1024, kernel_size=(14, 14), stride=(14, 14))
+        (norm): Identity()
+      )
+      (pos_drop): Dropout(p=0.0, inplace=False)
+      (norm_pre): Identity()
+      (blocks): ModuleList(
+        (0-23): 24 x EvaBlock(
+          (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+          (attn): EvaAttention(
+            (qkv): Linear(in_features=1024, out_features=3072, bias=False)
+            (q_norm): Identity()
+            (k_norm): Identity()
+            (attn_drop): Dropout(p=0.0, inplace=False)
+            (norm): Identity()
+            (proj): Linear(in_features=1024, out_features=1024, bias=True)
+            (proj_drop): Dropout(p=0.0, inplace=False)
+          )
+          (drop_path1): Identity()
+          (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+          (mlp): Mlp(
+            (fc1): Linear(in_features=1024, out_features=4096, bias=True)
+            (act): GELU(approximate='none')
+            (drop1): Dropout(p=0.0, inplace=False)
+            (norm): Identity()
+            (fc2): Linear(in_features=4096, out_features=1024, bias=True)
+            (drop2): Dropout(p=0.0, inplace=False)
+          )
+          (drop_path2): Identity()
+        )
+      )
+      (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+      (fc_norm): Identity()
+      (head_drop): Dropout(p=0.0, inplace=False)
+      (head): Identity()
+      (rope): _CapiPatchRoPE()
+    )
+  )
+)
+[INFO    ][2026-05-08 20:24:06][root                ][init_video_model         ] Encoder number of parameters: 1179648
+[INFO    ][2026-05-08 20:24:06][root                ][init_video_model         ] Predictor number of parameters: 11436928
+[INFO    ][2026-05-08 20:24:06][root                ][init_video_model         ] Target encoder number of parameters: 0
+[INFO    ][2026-05-08 20:24:08][root                ][make_videodataset        ] VideoDataset dataset created
+[INFO    ][2026-05-08 20:24:08][WeightedSampler     ][__init__                 ] Using DistributedWeightedSampler with rank 3 / 16
+[INFO    ][2026-05-08 20:24:08][root                ][make_videodataset        ] VideoDataset unsupervised data loader created
+[INFO    ][2026-05-08 20:24:08][app.vjepa.train     ][main                     ] iterations per epoch/dataset length: 300/399
+[INFO    ][2026-05-08 20:24:08][app.vjepa.train     ][main                     ] Wrapping models in DDP (rank 3)...
+[INFO    ][2026-05-08 20:24:41][app.vjepa.train     ][main                     ] Initializing loader...
+submitit INFO (2026-05-09 08:16:14,184) - Job has timed out. Ran 718 minutes out of requested 720 minutes.
+[INFO    ][2026-05-09 08:16:14][submitit            ][has_timed_out            ] Job has timed out. Ran 718 minutes out of requested 720 minutes.
+submitit WARNING (2026-05-09 08:16:14,207) - Caught signal SIGUSR2 on gcn75.local.snellius.surf.nl: this job is timed-out.
+[WARNING ][2026-05-09 08:16:14][submitit            ][checkpoint_and_try_requeue] Caught signal SIGUSR2 on gcn75.local.snellius.surf.nl: this job is timed-out.
+submitit INFO (2026-05-09 08:16:14,207) - Not checkpointing nor requeuing since I am a slave (procid=3).
+[INFO    ][2026-05-09 08:16:14][submitit            ][checkpoint_and_try_requeue] Not checkpointing nor requeuing since I am a slave (procid=3).
+submitit WARNING (2026-05-09 08:16:14,207) - Bypassing signal SIGCONT
+[WARNING ][2026-05-09 08:16:14][submitit            ][bypass                   ] Bypassing signal SIGCONT
+submitit WARNING (2026-05-09 08:16:14,372) - Bypassing signal SIGTERM
+[WARNING ][2026-05-09 08:16:14][submitit            ][bypass                   ] Bypassing signal SIGTERM
+submitit WARNING (2026-05-09 08:16:14,372) - Bypassing signal SIGCONT
+[WARNING ][2026-05-09 08:16:14][submitit            ][bypass                   ] Bypassing signal SIGCONT
+submitit INFO (2026-05-09 08:19:47,093) - Starting with JobEnvironment(job_id=22595640, hostname=gcn73.local.snellius.surf.nl, local_rank=3(4), node=0(4), global_rank=3(16))
+submitit INFO (2026-05-09 08:19:47,106) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/job_22595640/22595640_submitted.pkl
+INFO:root:loaded pretrain params...
+{   'app': 'vjepa',
+    'cpus_per_task': 16,
+    'data': {   'batch_size': 64,
+                'crop_size': 224,
+                'dataset_fpcs': [16, 16],
+                'dataset_type': 'VideoDataset',
+                'datasets': [   '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
+                                '/scratch-shared/dcanez/data/ssv2/train.csv'],
+                'datasets_weights': [0.65, 0.35],
+                'fps': 4,
+                'num_workers': 10,
+                'patch_size': 14,
+                'persistent_workers': True,
+                'pin_mem': True,
+                'stage': [   {   'dest': 'kinetics_240',
+                                 'format': 'targz_parts',
+                                 'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
+                             {   'dest': 'ssv2',
+                                 'format': 'multipart_tar',
+                                 'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
+                'tubelet_size': 1},
+    'data_aug': {   'auto_augment': False,
+                    'motion_shift': False,
+                    'random_resize_aspect_ratio': [0.75, 1.35],
+                    'random_resize_scale': [0.3, 1.0],
+                    'reprob': 0.0},
+    'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m',
+    'loss': {'loss_exp': 1.0},
+    'mask': [   {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 8,
+                    'spatial_scale': [0.15, 0.15],
+                    'temporal_scale': [1.0, 1.0]},
+                {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 2,
+                    'spatial_scale': [0.7, 0.7],
+                    'temporal_scale': [1.0, 1.0]}],
+    'mem_per_gpu': '180G',
+    'meta': {   'dtype': 'bfloat16',
+                'knn_eval_epoch0': False,
+                'knn_eval_freq': 5,
+                'knn_eval_presets': [   {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
+                                                          'eval_videos_per_class': 25,
+                                                          'num_workers': 8,
+                                                          'train_videos_per_class': 100},
+                                            'preset': 'ucf101'},
+                                        {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
+                                                          'eval_videos_per_class': 100,
+                                                          'linear_probe': True,
+                                                          'num_workers': 8,
+                                                          'pool_type': 'temporal_concat',
+                                                          'train_videos_per_class': 500},
+                                            'preset': 'ssv2_coarse10'}],
+                'load_checkpoint': True,
+                'read_checkpoint': None,
+                'save_every_freq': 5,
+                'seed': 239,
+                'use_sdpa': True,
+                'use_wandb': True,
+                'wandb_project': 'vjepa_ablation'},
+    'metrics': {'sigreg': {}, 'std': {}},
+    'model': {   'class_token': False,
+                 'freeze_backbone': True,
+                 'is_causal': False,
+                 'lora_alpha': 16,
+                 'lora_rank': 8,
+                 'lora_targets': ['attn.qkv', 'attn.proj'],
+                 'model_name': 'vit_large_patch14_capi_lvd1689m',
+                 'num_reg_tokens': 16,
+                 'pred_depth': 6,
+                 'pred_embed_dim': 384,
+                 'pred_num_heads': 12,
+                 'rope_type': 'capi2d',
+                 'stem_type': '2d',
+                 'target_kind': 'frozen_2d',
+                 'target_type': 'vit_large_patch14_capi.lvd1689m',
+                 'uniform_power': True,
+                 'use_activation_checkpointing': True,
+                 'use_lora': True,
+                 'use_mask_tokens': True,
+                 'use_rope': True,
+                 'use_sdpa': True,
+                 'zero_init_mask_tokens': True},
+    'nodes': 4,
+    'optimization': {   'ema': [0.99925, 0.99925],
+                        'epochs': 100,
+                        'final_lr': 0.0001,
+                        'final_weight_decay': 0.04,
+                        'ipe': 300,
+                        'ipe_scale': 1.0,
+                        'lr': 0.0005,
+                        'start_lr': 0.0001,
+                        'warmup': 10,
+                        'weight_decay': 0.04},
+    'tasks_per_node': 4}
+INFO:root:Running pre-training of app: vjepa
+[INFO    ][2026-05-09 08:20:49][app.vjepa.train     ][main                     ] which_dtype='bfloat16'
+[INFO    ][2026-05-09 08:20:49][app.vjepa.train     ][main                     ] Disabling persistent_workers (incompatible with KNN eval)
+[INFO    ][2026-05-09 08:20:49][app.vjepa.train     ][main                     ] NCCL_SOCKET_IFNAME=eno
+[INFO    ][2026-05-09 08:20:51][app.vjepa.train     ][main                     ] Initialized (rank/world-size) 3/16, tasks_per_node=4
+[INFO    ][2026-05-09 08:20:51][root                ][stage_datasets           ] [local_rank 3/4] Staging kinetics_240 (targz_parts)
+[INFO    ][2026-05-09 08:20:51][root                ][_stage_targz_parts       ] [rank 3] Extracting 25/103 tar.gz parts to /scratch-node/dcanez.22595640/kinetics_240
+[INFO    ][2026-05-09 08:21:05][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 2/25 parts
+[INFO    ][2026-05-09 08:21:19][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 4/25 parts
+[INFO    ][2026-05-09 08:21:33][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 6/25 parts
+[INFO    ][2026-05-09 08:21:48][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 8/25 parts
+[INFO    ][2026-05-09 08:22:02][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 10/25 parts
+[INFO    ][2026-05-09 08:22:17][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 12/25 parts
+[INFO    ][2026-05-09 08:22:31][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 14/25 parts
+[INFO    ][2026-05-09 08:22:45][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 16/25 parts
+[INFO    ][2026-05-09 08:22:59][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 18/25 parts
+[INFO    ][2026-05-09 08:23:13][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 20/25 parts
+[INFO    ][2026-05-09 08:23:28][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 22/25 parts
+[INFO    ][2026-05-09 08:23:42][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 24/25 parts
+[INFO    ][2026-05-09 08:23:49][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 25/25 parts
+[INFO    ][2026-05-09 08:23:49][root                ][stage_datasets           ] [local_rank 3/4] Staging ssv2 (multipart_tar)
+[INFO    ][2026-05-09 08:25:33][app.vjepa.train     ][main                     ] Staged datasets: ['/scratch-node/dcanez.22595640/kinetics_240/train.csv', '/scratch-node/dcanez.22595640/ssv2/train.csv']
+[INFO    ][2026-05-09 08:26:29][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
+[INFO    ][2026-05-09 08:26:30][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA applied: 48 modules replaced (rank=8, alpha=16, dropout=0.0, targets=['attn.qkv', 'attn.proj'], scope=blocks)
+[INFO    ][2026-05-09 08:26:30][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA trainable params: 1,179,648
+[INFO    ][2026-05-09 08:26:33][root                ][_build_frozen_2d_target  ] Loaded frozen_2d target from timm: vit_large_patch14_capi.lvd1689m
+[INFO    ][2026-05-09 08:26:33][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-09 08:26:33][root                ][init_video_model         ] VJEPAPredictorMultiSeqWrapper(
+  (backbone): VisionTransformerPredictor(
+    (predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
+    (mask_tokens): ParameterList(
+        (0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+    )
+    (predictor_blocks): ModuleList(
+      (0-5): 6 x Block(
+        (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (attn): RoPEAttention(
+          (qkv): Linear(in_features=384, out_features=1152, bias=True)
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): Linear(in_features=384, out_features=384, bias=True)
+          (proj_drop): Dropout(p=0.0, inplace=False)
+        )
+        (drop_path): Identity()
+        (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (mlp): MLP(
+          (fc1): Linear(in_features=384, out_features=1536, bias=True)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=1536, out_features=384, bias=True)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+    (predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
+  )
+)
+[INFO    ][2026-05-09 08:26:33][root                ][init_video_model         ] MultiSeqWrapper(
+  (backbone): Frozen2DTargetWrapper(
+    (backbone): Eva(
+      (patch_embed): PatchEmbed(
+        (proj): Conv2d(3, 1024, kernel_size=(14, 14), stride=(14, 14))
+        (norm): Identity()
+      )
+      (pos_drop): Dropout(p=0.0, inplace=False)
+      (norm_pre): Identity()
+      (blocks): ModuleList(
+        (0-23): 24 x EvaBlock(
+          (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+          (attn): EvaAttention(
+            (qkv): Linear(in_features=1024, out_features=3072, bias=False)
+            (q_norm): Identity()
+            (k_norm): Identity()
+            (attn_drop): Dropout(p=0.0, inplace=False)
+            (norm): Identity()
+            (proj): Linear(in_features=1024, out_features=1024, bias=True)
+            (proj_drop): Dropout(p=0.0, inplace=False)
+          )
+          (drop_path1): Identity()
+          (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+          (mlp): Mlp(
+            (fc1): Linear(in_features=1024, out_features=4096, bias=True)
+            (act): GELU(approximate='none')
+            (drop1): Dropout(p=0.0, inplace=False)
+            (norm): Identity()
+            (fc2): Linear(in_features=4096, out_features=1024, bias=True)
+            (drop2): Dropout(p=0.0, inplace=False)
+          )
+          (drop_path2): Identity()
+        )
+      )
+      (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+      (fc_norm): Identity()
+      (head_drop): Dropout(p=0.0, inplace=False)
+      (head): Identity()
+      (rope): _CapiPatchRoPE()
+    )
+  )
+)
+[INFO    ][2026-05-09 08:26:33][root                ][init_video_model         ] Encoder number of parameters: 1179648
+[INFO    ][2026-05-09 08:26:33][root                ][init_video_model         ] Predictor number of parameters: 11436928
+[INFO    ][2026-05-09 08:26:33][root                ][init_video_model         ] Target encoder number of parameters: 0
+[INFO    ][2026-05-09 08:26:34][root                ][make_videodataset        ] VideoDataset dataset created
+[INFO    ][2026-05-09 08:26:34][WeightedSampler     ][__init__                 ] Using DistributedWeightedSampler with rank 3 / 16
+[INFO    ][2026-05-09 08:26:34][root                ][make_videodataset        ] VideoDataset unsupervised data loader created
+[INFO    ][2026-05-09 08:26:34][app.vjepa.train     ][main                     ] iterations per epoch/dataset length: 300/399
+[INFO    ][2026-05-09 08:26:34][app.vjepa.train     ][main                     ] Wrapping models in DDP (rank 3)...
+[INFO    ][2026-05-09 08:26:37][root                ][load_checkpoint          ] Loading checkpoint from /scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/latest.pt
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] loaded pretrained encoder from epoch 39 with msg: <All keys matched successfully>
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] loaded pretrained predictor from epoch 39 with msg: <All keys matched successfully>
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] loaded pretrained target encoder from epoch 39 with msg: <All keys matched successfully>
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] loaded optimizers from epoch 39
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] read-path: /scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/latest.pt
+[INFO    ][2026-05-09 08:26:39][app.vjepa.train     ][main                     ] Initializing loader...
+submitit WARNING (2026-05-09 09:56:00,846) - Bypassing signal SIGTERM
+[WARNING ][2026-05-09 09:56:00][submitit            ][bypass                   ] Bypassing signal SIGTERM
+submitit WARNING (2026-05-09 09:56:00,992) - Bypassing signal SIGCONT
+[WARNING ][2026-05-09 09:56:00][submitit            ][bypass                   ] Bypassing signal SIGCONT

vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/job_22595640/22595640_4_log.err

@@ -0,0 +1,324 @@
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
+  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
+[rank4]:[W508 20:23:12.235487498 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
+  scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+submitit WARNING (2026-05-09 08:16:14,369) - Caught signal SIGUSR2 on gcn76.local.snellius.surf.nl: this job is timed-out.
+submitit WARNING (2026-05-09 08:16:14,369) - Bypassing signal SIGTERM
+submitit WARNING (2026-05-09 08:16:14,369) - Bypassing signal SIGCONT
+[rank4]:[W509 08:16:18.962089687 TCPStore.cpp:125] [c10d] recvValue failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50450, remote=[gcn75.local.snellius.surf.nl]:37129): Failed to recv, got 0 bytes. Connection was likely closed. Did the remote server shutdown or crash?
+Exception raised from recvBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:682 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15101277efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a3325d (0x15105690d25d in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x273 (0x15105690b1f3 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1510139edeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x151152e57164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1511638c919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15116394e100 in /lib64/libc.so.6)
+
+[rank4]:[W509 08:16:18.964851095 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 4] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Failed to recv, got 0 bytes. Connection was likely closed. Did the remote server shutdown or crash?
+[rank4]:[W509 08:16:19.965046521 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50450, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15101277efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15105690c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15105690b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1510139edeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x151152e57164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1511638c919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15116394e100 in /lib64/libc.so.6)
+
+[rank4]:[W509 08:16:19.968131158 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 4] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank4]:[W509 08:16:20.968287754 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50450, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15101277efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15105690c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15105690b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1510139edeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x151152e57164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1511638c919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15116394e100 in /lib64/libc.so.6)
+
+[rank4]:[W509 08:16:20.970660254 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 4] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank4]:[W509 08:16:21.970804010 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50450, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15101277efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15105690c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15105690b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1510139edeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x151152e57164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1511638c919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15116394e100 in /lib64/libc.so.6)
+
+[rank4]:[W509 08:16:21.973181000 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 4] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank4]:[W509 08:16:22.973326786 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50450, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15101277efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15105690c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15105690b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1510139edeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x151152e57164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1511638c919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15116394e100 in /lib64/libc.so.6)
+
+[rank4]:[W509 08:16:22.975688926 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 4] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank4]:[W509 08:16:23.975840262 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50450, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15101277efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15105690c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15105690b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1510139edeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x151152e57164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1511638c919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15116394e100 in /lib64/libc.so.6)
+
+[rank4]:[W509 08:16:23.978164852 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 4] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank4]:[W509 08:16:24.978298429 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50450, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15101277efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15105690c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15105690b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1510139edeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x151152e57164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1511638c919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15116394e100 in /lib64/libc.so.6)
+
+[rank4]:[W509 08:16:24.980566629 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 4] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank4]:[W509 08:16:25.980707175 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50450, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15101277efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15105690c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15105690b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1510139edeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x151152e57164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1511638c919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15116394e100 in /lib64/libc.so.6)
+
+[rank4]:[W509 08:16:25.983019365 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 4] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank4]:[W509 08:16:26.983151371 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50450, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15101277efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15105690c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15105690b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1510139edeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x151152e57164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1511638c919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15116394e100 in /lib64/libc.so.6)
+
+[rank4]:[W509 08:16:26.985485471 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 4] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank4]:[W509 08:16:27.985616868 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50450, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15101277efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15105690c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15105690b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1510139edeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x151152e57164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1511638c919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15116394e100 in /lib64/libc.so.6)
+
+[rank4]:[W509 08:16:27.987927888 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 4] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank4]:[W509 08:16:28.988066954 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50450, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15101277efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15105690c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15105690b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1510139edeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x151152e57164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1511638c919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15116394e100 in /lib64/libc.so.6)
+
+[rank4]:[W509 08:16:28.990335844 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 4] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank4]:[W509 08:16:29.990455781 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50450, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15101277efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15105690c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15105690b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1510139edeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x151152e57164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1511638c919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15116394e100 in /lib64/libc.so.6)
+
+[rank4]:[W509 08:16:29.992722291 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 4] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank4]:[W509 08:16:30.992858057 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50450, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15101277efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15105690c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15105690b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1510139edeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x151152e57164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1511638c919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15116394e100 in /lib64/libc.so.6)
+
+[rank4]:[W509 08:16:30.995130157 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 4] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank4]:[W509 08:16:31.995263324 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50450, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15101277efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15105690c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15105690b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1510139edeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x151152e57164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1511638c919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15116394e100 in /lib64/libc.so.6)
+
+[rank4]:[W509 08:16:31.997517634 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 4] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank4]:[W509 08:16:32.997650010 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50450, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15101277efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15105690c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15105690b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1510139edeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x151152e57164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1511638c919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15116394e100 in /lib64/libc.so.6)
+
+[rank4]:[W509 08:16:32.999955490 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 4] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank4]:[W509 08:16:33.000090417 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50450, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15101277efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15105690c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15105690b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1510139edeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x151152e57164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1511638c919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15116394e100 in /lib64/libc.so.6)
+
+[rank4]:[W509 08:16:33.002401937 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 4] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank4]:[W509 08:16:34.002537493 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50450, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15101277efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15105690c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15105690b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1510139edeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x151152e57164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1511638c919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15116394e100 in /lib64/libc.so.6)
+
+[rank4]:[W509 08:16:34.004798643 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 4] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank4]:[W509 08:16:35.004930570 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50450, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15101277efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15105690c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15105690b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1510139edeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x151152e57164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1511638c919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15116394e100 in /lib64/libc.so.6)
+
+[rank4]:[W509 08:16:35.007293580 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 4] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank4]:[W509 08:16:36.007424296 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50450, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15101277efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15105690c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15105690b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1510139edeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x151152e57164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1511638c919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15116394e100 in /lib64/libc.so.6)
+
+[rank4]:[W509 08:16:36.009600077 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 4] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank4]:[W509 08:16:37.009732743 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50450, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15101277efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15105690c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15105690b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1510139edeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x151152e57164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1511638c919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15116394e100 in /lib64/libc.so.6)
+
+[rank4]:[W509 08:16:37.011944783 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 4] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank4]:[W509 08:16:38.012080120 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50450, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15101277efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15105690c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15105690b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1510139edeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x151152e57164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1511638c919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15116394e100 in /lib64/libc.so.6)
+
+[rank4]:[W509 08:16:38.014400200 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 4] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank4]:[W509 08:16:39.014524276 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50450, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15101277efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15105690c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15105690b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1510139edeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x151152e57164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1511638c919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15116394e100 in /lib64/libc.so.6)
+
+[rank4]:[W509 08:16:39.016795596 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 4] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank4]:[W509 08:16:40.016914713 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50450, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15101277efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15105690c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15105690b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1510139edeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x151152e57164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1511638c919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15116394e100 in /lib64/libc.so.6)
+
+[rank4]:[W509 08:16:40.019141273 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 4] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank4]:[W509 08:16:41.019262089 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50450, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15101277efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15105690c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15105690b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1510139edeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x151152e57164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1511638c919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15116394e100 in /lib64/libc.so.6)
+
+[rank4]:[W509 08:16:41.021521330 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 4] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank4]:[W509 08:16:42.021647266 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50450, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15101277efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15105690c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15105690b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1510139edeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x151152e57164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1511638c919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15116394e100 in /lib64/libc.so.6)
+
+[rank4]:[W509 08:16:42.023943006 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 4] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank4]:[W509 08:16:43.024069633 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50450, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x15101277efdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x15105690c6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x15105690b1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1510139edeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x151152e57164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1511638c919a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x15116394e100 in /lib64/libc.so.6)
+
+[rank4]:[W509 08:16:43.026263813 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 4] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[2026-05-09T08:16:46.538] error: namespace_p_join: open failed for /slurm/22595640/.ns: No such file or directory
+[2026-05-09T08:16:46.538] error: namespace_g_join(JobId=22595640 SLUID=s8FXKR4CZ0G200): No such file or directory
+[2026-05-09T08:16:48.184] error: Failed to send MESSAGE_TASK_EXIT: Connection refused
+[2026-05-09T08:16:49.917] error: namespace_p_join: open failed for /slurm/22595640/.ns: No such file or directory
+[2026-05-09T08:16:49.917] error: namespace_g_join(JobId=22595640 SLUID=s8FXKR4CZ0G200): No such file or directory
+[2026-05-09T08:16:50.056] error: namespace_p_join: open failed for /slurm/22595640/.ns: No such file or directory
+[2026-05-09T08:16:50.056] error: namespace_g_join(JobId=22595640 SLUID=s8FXKR4CZ0G200): No such file or directory
+[2026-05-09T08:16:50.095] error: namespace_p_join: open failed for /slurm/22595640/.ns: No such file or directory
+[2026-05-09T08:16:50.095] error: namespace_g_join(JobId=22595640 SLUID=s8FXKR4CZ0G200): No such file or directory
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
+  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
+[rank4]:[W509 08:25:24.288734903 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
+  scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+submitit WARNING (2026-05-09 09:56:00,866) - Bypassing signal SIGTERM
+submitit WARNING (2026-05-09 09:56:00,866) - Bypassing signal SIGCONT
+[2026-05-09T09:56:32.515] error: namespace_p_join: open failed for /slurm/22595640/.ns: No such file or directory
+[2026-05-09T09:56:32.515] error: namespace_g_join(JobId=22595640 SLUID=s8FXKR5P56S300): No such file or directory
+[2026-05-09T09:56:34.167] error: Failed to send MESSAGE_TASK_EXIT: Connection refused
+[2026-05-09T09:56:34.169] error: namespace_p_join: open failed for /slurm/22595640/.ns: No such file or directory
+[2026-05-09T09:56:34.169] error: namespace_g_join(JobId=22595640 SLUID=s8FXKR5P56S300): No such file or directory
+[2026-05-09T09:56:35.807] error: Failed to send MESSAGE_TASK_EXIT: Connection refused
+[2026-05-09T09:56:35.809] error: namespace_p_join: open failed for /slurm/22595640/.ns: No such file or directory
+[2026-05-09T09:56:35.809] error: namespace_g_join(JobId=22595640 SLUID=s8FXKR5P56S300): No such file or directory
+[2026-05-09T09:56:35.948] error: namespace_p_join: open failed for /slurm/22595640/.ns: No such file or directory
+[2026-05-09T09:56:35.948] error: namespace_g_join(JobId=22595640 SLUID=s8FXKR5P56S300): No such file or directory

vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/job_22595640/22595640_4_log.out

@@ -0,0 +1,538 @@
+submitit INFO (2026-05-08 20:18:11,423) - Starting with JobEnvironment(job_id=22595640, hostname=gcn76.local.snellius.surf.nl, local_rank=0(4), node=1(4), global_rank=4(16))
+submitit INFO (2026-05-08 20:18:11,423) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/job_22595640/22595640_submitted.pkl
+INFO:root:loaded pretrain params...
+{   'app': 'vjepa',
+    'cpus_per_task': 16,
+    'data': {   'batch_size': 64,
+                'crop_size': 224,
+                'dataset_fpcs': [16, 16],
+                'dataset_type': 'VideoDataset',
+                'datasets': [   '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
+                                '/scratch-shared/dcanez/data/ssv2/train.csv'],
+                'datasets_weights': [0.65, 0.35],
+                'fps': 4,
+                'num_workers': 10,
+                'patch_size': 14,
+                'persistent_workers': True,
+                'pin_mem': True,
+                'stage': [   {   'dest': 'kinetics_240',
+                                 'format': 'targz_parts',
+                                 'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
+                             {   'dest': 'ssv2',
+                                 'format': 'multipart_tar',
+                                 'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
+                'tubelet_size': 1},
+    'data_aug': {   'auto_augment': False,
+                    'motion_shift': False,
+                    'random_resize_aspect_ratio': [0.75, 1.35],
+                    'random_resize_scale': [0.3, 1.0],
+                    'reprob': 0.0},
+    'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m',
+    'loss': {'loss_exp': 1.0},
+    'mask': [   {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 8,
+                    'spatial_scale': [0.15, 0.15],
+                    'temporal_scale': [1.0, 1.0]},
+                {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 2,
+                    'spatial_scale': [0.7, 0.7],
+                    'temporal_scale': [1.0, 1.0]}],
+    'mem_per_gpu': '180G',
+    'meta': {   'dtype': 'bfloat16',
+                'knn_eval_epoch0': False,
+                'knn_eval_freq': 5,
+                'knn_eval_presets': [   {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
+                                                          'eval_videos_per_class': 25,
+                                                          'num_workers': 8,
+                                                          'train_videos_per_class': 100},
+                                            'preset': 'ucf101'},
+                                        {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
+                                                          'eval_videos_per_class': 100,
+                                                          'linear_probe': True,
+                                                          'num_workers': 8,
+                                                          'pool_type': 'temporal_concat',
+                                                          'train_videos_per_class': 500},
+                                            'preset': 'ssv2_coarse10'}],
+                'load_checkpoint': True,
+                'read_checkpoint': None,
+                'save_every_freq': 5,
+                'seed': 239,
+                'use_sdpa': True,
+                'use_wandb': True,
+                'wandb_project': 'vjepa_ablation'},
+    'metrics': {'sigreg': {}, 'std': {}},
+    'model': {   'class_token': False,
+                 'freeze_backbone': True,
+                 'is_causal': False,
+                 'lora_alpha': 16,
+                 'lora_rank': 8,
+                 'lora_targets': ['attn.qkv', 'attn.proj'],
+                 'model_name': 'vit_large_patch14_capi_lvd1689m',
+                 'num_reg_tokens': 16,
+                 'pred_depth': 6,
+                 'pred_embed_dim': 384,
+                 'pred_num_heads': 12,
+                 'rope_type': 'capi2d',
+                 'stem_type': '2d',
+                 'target_kind': 'frozen_2d',
+                 'target_type': 'vit_large_patch14_capi.lvd1689m',
+                 'uniform_power': True,
+                 'use_activation_checkpointing': True,
+                 'use_lora': True,
+                 'use_mask_tokens': True,
+                 'use_rope': True,
+                 'use_sdpa': True,
+                 'zero_init_mask_tokens': True},
+    'nodes': 4,
+    'optimization': {   'ema': [0.99925, 0.99925],
+                        'epochs': 100,
+                        'final_lr': 0.0001,
+                        'final_weight_decay': 0.04,
+                        'ipe': 300,
+                        'ipe_scale': 1.0,
+                        'lr': 0.0005,
+                        'start_lr': 0.0001,
+                        'warmup': 10,
+                        'weight_decay': 0.04},
+    'tasks_per_node': 4}
+INFO:root:Running pre-training of app: vjepa
+[INFO    ][2026-05-08 20:19:11][app.vjepa.train     ][main                     ] which_dtype='bfloat16'
+[INFO    ][2026-05-08 20:19:11][app.vjepa.train     ][main                     ] Disabling persistent_workers (incompatible with KNN eval)
+[INFO    ][2026-05-08 20:19:11][app.vjepa.train     ][main                     ] NCCL_SOCKET_IFNAME=eno
+[INFO    ][2026-05-08 20:19:14][app.vjepa.train     ][main                     ] Initialized (rank/world-size) 4/16, tasks_per_node=4
+[INFO    ][2026-05-08 20:19:14][root                ][stage_datasets           ] [local_rank 0/4] Staging kinetics_240 (targz_parts)
+[INFO    ][2026-05-08 20:19:14][root                ][_stage_targz_parts       ] [rank 0] Extracting 26/103 tar.gz parts to /scratch-node/dcanez.22595640/kinetics_240
+[INFO    ][2026-05-08 20:19:28][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 2/26 parts
+[INFO    ][2026-05-08 20:19:42][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 4/26 parts
+[INFO    ][2026-05-08 20:19:56][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 6/26 parts
+[INFO    ][2026-05-08 20:20:10][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 8/26 parts
+[INFO    ][2026-05-08 20:20:24][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 10/26 parts
+[INFO    ][2026-05-08 20:20:38][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 12/26 parts
+[INFO    ][2026-05-08 20:20:52][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 14/26 parts
+[INFO    ][2026-05-08 20:21:07][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 16/26 parts
+[INFO    ][2026-05-08 20:21:21][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 18/26 parts
+[INFO    ][2026-05-08 20:21:35][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 20/26 parts
+[INFO    ][2026-05-08 20:21:49][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 22/26 parts
+[INFO    ][2026-05-08 20:22:03][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 24/26 parts
+[INFO    ][2026-05-08 20:22:17][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 26/26 parts
+[INFO    ][2026-05-08 20:22:18][root                ][stage_datasets           ] [local_rank 0/4] Staging ssv2 (multipart_tar)
+[INFO    ][2026-05-08 20:22:18][root                ][_stage_multipart_tar     ] [rank 0] Extracting multipart tar (2 files) to /scratch-node/dcanez.22595640/ssv2
+[INFO    ][2026-05-08 20:23:16][root                ][stage_datasets           ] Data staging completed in 241.8s (4.0min)
+[INFO    ][2026-05-08 20:23:16][root                ][_rewrite_csv             ] Wrote local CSV: /scratch-node/dcanez.22595640/kinetics_240/train.csv (239789 entries)
+[INFO    ][2026-05-08 20:23:17][root                ][_rewrite_csv             ] Wrote local CSV: /scratch-node/dcanez.22595640/ssv2/train.csv (168913 entries)
+[INFO    ][2026-05-08 20:23:17][app.vjepa.train     ][main                     ] Staged datasets: ['/scratch-node/dcanez.22595640/kinetics_240/train.csv', '/scratch-node/dcanez.22595640/ssv2/train.csv']
+[INFO    ][2026-05-08 20:24:16][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
+[INFO    ][2026-05-08 20:24:20][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA applied: 48 modules replaced (rank=8, alpha=16, dropout=0.0, targets=['attn.qkv', 'attn.proj'], scope=blocks)
+[INFO    ][2026-05-08 20:24:20][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA trainable params: 1,179,648
+[INFO    ][2026-05-08 20:24:26][root                ][_build_frozen_2d_target  ] Loaded frozen_2d target from timm: vit_large_patch14_capi.lvd1689m
+[INFO    ][2026-05-08 20:24:26][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-08 20:24:26][root                ][init_video_model         ] VJEPAPredictorMultiSeqWrapper(
+  (backbone): VisionTransformerPredictor(
+    (predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
+    (mask_tokens): ParameterList(
+        (0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+    )
+    (predictor_blocks): ModuleList(
+      (0-5): 6 x Block(
+        (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (attn): RoPEAttention(
+          (qkv): Linear(in_features=384, out_features=1152, bias=True)
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): Linear(in_features=384, out_features=384, bias=True)
+          (proj_drop): Dropout(p=0.0, inplace=False)
+        )
+        (drop_path): Identity()
+        (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (mlp): MLP(
+          (fc1): Linear(in_features=384, out_features=1536, bias=True)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=1536, out_features=384, bias=True)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+    (predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
+  )
+)
+[INFO    ][2026-05-08 20:24:26][root                ][init_video_model         ] MultiSeqWrapper(
+  (backbone): Frozen2DTargetWrapper(
+    (backbone): Eva(
+      (patch_embed): PatchEmbed(
+        (proj): Conv2d(3, 1024, kernel_size=(14, 14), stride=(14, 14))
+        (norm): Identity()
+      )
+      (pos_drop): Dropout(p=0.0, inplace=False)
+      (norm_pre): Identity()
+      (blocks): ModuleList(
+        (0-23): 24 x EvaBlock(
+          (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+          (attn): EvaAttention(
+            (qkv): Linear(in_features=1024, out_features=3072, bias=False)
+            (q_norm): Identity()
+            (k_norm): Identity()
+            (attn_drop): Dropout(p=0.0, inplace=False)
+            (norm): Identity()
+            (proj): Linear(in_features=1024, out_features=1024, bias=True)
+            (proj_drop): Dropout(p=0.0, inplace=False)
+          )
+          (drop_path1): Identity()
+          (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+          (mlp): Mlp(
+            (fc1): Linear(in_features=1024, out_features=4096, bias=True)
+            (act): GELU(approximate='none')
+            (drop1): Dropout(p=0.0, inplace=False)
+            (norm): Identity()
+            (fc2): Linear(in_features=4096, out_features=1024, bias=True)
+            (drop2): Dropout(p=0.0, inplace=False)
+          )
+          (drop_path2): Identity()
+        )
+      )
+      (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+      (fc_norm): Identity()
+      (head_drop): Dropout(p=0.0, inplace=False)
+      (head): Identity()
+      (rope): _CapiPatchRoPE()
+    )
+  )
+)
+[INFO    ][2026-05-08 20:24:26][root                ][init_video_model         ] Encoder number of parameters: 1179648
+[INFO    ][2026-05-08 20:24:26][root                ][init_video_model         ] Predictor number of parameters: 11436928
+[INFO    ][2026-05-08 20:24:26][root                ][init_video_model         ] Target encoder number of parameters: 0
+[INFO    ][2026-05-08 20:24:29][root                ][make_videodataset        ] VideoDataset dataset created
+[INFO    ][2026-05-08 20:24:29][WeightedSampler     ][__init__                 ] Using DistributedWeightedSampler with rank 4 / 16
+[INFO    ][2026-05-08 20:24:29][root                ][make_videodataset        ] VideoDataset unsupervised data loader created
+[INFO    ][2026-05-08 20:24:29][app.vjepa.train     ][main                     ] iterations per epoch/dataset length: 300/399
+[INFO    ][2026-05-08 20:24:29][app.vjepa.train     ][main                     ] Wrapping models in DDP (rank 4)...
+[INFO    ][2026-05-08 20:24:41][app.vjepa.train     ][main                     ] Initializing loader...
+submitit INFO (2026-05-09 08:16:14,354) - Job has timed out. Ran 718 minutes out of requested 720 minutes.
+[INFO    ][2026-05-09 08:16:14][submitit            ][has_timed_out            ] Job has timed out. Ran 718 minutes out of requested 720 minutes.
+submitit WARNING (2026-05-09 08:16:14,369) - Caught signal SIGUSR2 on gcn76.local.snellius.surf.nl: this job is timed-out.
+[WARNING ][2026-05-09 08:16:14][submitit            ][checkpoint_and_try_requeue] Caught signal SIGUSR2 on gcn76.local.snellius.surf.nl: this job is timed-out.
+submitit INFO (2026-05-09 08:16:14,369) - Not checkpointing nor requeuing since I am a slave (procid=4).
+[INFO    ][2026-05-09 08:16:14][submitit            ][checkpoint_and_try_requeue] Not checkpointing nor requeuing since I am a slave (procid=4).
+submitit WARNING (2026-05-09 08:16:14,369) - Bypassing signal SIGTERM
+[WARNING ][2026-05-09 08:16:14][submitit            ][bypass                   ] Bypassing signal SIGTERM
+submitit WARNING (2026-05-09 08:16:14,369) - Bypassing signal SIGCONT
+[WARNING ][2026-05-09 08:16:14][submitit            ][bypass                   ] Bypassing signal SIGCONT
+submitit INFO (2026-05-09 08:19:43,911) - Starting with JobEnvironment(job_id=22595640, hostname=gcn75.local.snellius.surf.nl, local_rank=0(4), node=1(4), global_rank=4(16))
+submitit INFO (2026-05-09 08:19:43,941) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/job_22595640/22595640_submitted.pkl
+INFO:root:loaded pretrain params...
+{   'app': 'vjepa',
+    'cpus_per_task': 16,
+    'data': {   'batch_size': 64,
+                'crop_size': 224,
+                'dataset_fpcs': [16, 16],
+                'dataset_type': 'VideoDataset',
+                'datasets': [   '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
+                                '/scratch-shared/dcanez/data/ssv2/train.csv'],
+                'datasets_weights': [0.65, 0.35],
+                'fps': 4,
+                'num_workers': 10,
+                'patch_size': 14,
+                'persistent_workers': True,
+                'pin_mem': True,
+                'stage': [   {   'dest': 'kinetics_240',
+                                 'format': 'targz_parts',
+                                 'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
+                             {   'dest': 'ssv2',
+                                 'format': 'multipart_tar',
+                                 'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
+                'tubelet_size': 1},
+    'data_aug': {   'auto_augment': False,
+                    'motion_shift': False,
+                    'random_resize_aspect_ratio': [0.75, 1.35],
+                    'random_resize_scale': [0.3, 1.0],
+                    'reprob': 0.0},
+    'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m',
+    'loss': {'loss_exp': 1.0},
+    'mask': [   {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 8,
+                    'spatial_scale': [0.15, 0.15],
+                    'temporal_scale': [1.0, 1.0]},
+                {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 2,
+                    'spatial_scale': [0.7, 0.7],
+                    'temporal_scale': [1.0, 1.0]}],
+    'mem_per_gpu': '180G',
+    'meta': {   'dtype': 'bfloat16',
+                'knn_eval_epoch0': False,
+                'knn_eval_freq': 5,
+                'knn_eval_presets': [   {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
+                                                          'eval_videos_per_class': 25,
+                                                          'num_workers': 8,
+                                                          'train_videos_per_class': 100},
+                                            'preset': 'ucf101'},
+                                        {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
+                                                          'eval_videos_per_class': 100,
+                                                          'linear_probe': True,
+                                                          'num_workers': 8,
+                                                          'pool_type': 'temporal_concat',
+                                                          'train_videos_per_class': 500},
+                                            'preset': 'ssv2_coarse10'}],
+                'load_checkpoint': True,
+                'read_checkpoint': None,
+                'save_every_freq': 5,
+                'seed': 239,
+                'use_sdpa': True,
+                'use_wandb': True,
+                'wandb_project': 'vjepa_ablation'},
+    'metrics': {'sigreg': {}, 'std': {}},
+    'model': {   'class_token': False,
+                 'freeze_backbone': True,
+                 'is_causal': False,
+                 'lora_alpha': 16,
+                 'lora_rank': 8,
+                 'lora_targets': ['attn.qkv', 'attn.proj'],
+                 'model_name': 'vit_large_patch14_capi_lvd1689m',
+                 'num_reg_tokens': 16,
+                 'pred_depth': 6,
+                 'pred_embed_dim': 384,
+                 'pred_num_heads': 12,
+                 'rope_type': 'capi2d',
+                 'stem_type': '2d',
+                 'target_kind': 'frozen_2d',
+                 'target_type': 'vit_large_patch14_capi.lvd1689m',
+                 'uniform_power': True,
+                 'use_activation_checkpointing': True,
+                 'use_lora': True,
+                 'use_mask_tokens': True,
+                 'use_rope': True,
+                 'use_sdpa': True,
+                 'zero_init_mask_tokens': True},
+    'nodes': 4,
+    'optimization': {   'ema': [0.99925, 0.99925],
+                        'epochs': 100,
+                        'final_lr': 0.0001,
+                        'final_weight_decay': 0.04,
+                        'ipe': 300,
+                        'ipe_scale': 1.0,
+                        'lr': 0.0005,
+                        'start_lr': 0.0001,
+                        'warmup': 10,
+                        'weight_decay': 0.04},
+    'tasks_per_node': 4}
+INFO:root:Running pre-training of app: vjepa
+[INFO    ][2026-05-09 08:19:53][app.vjepa.train     ][main                     ] which_dtype='bfloat16'
+[INFO    ][2026-05-09 08:19:53][app.vjepa.train     ][main                     ] Disabling persistent_workers (incompatible with KNN eval)
+[INFO    ][2026-05-09 08:19:53][app.vjepa.train     ][main                     ] NCCL_SOCKET_IFNAME=eno
+[INFO    ][2026-05-09 08:21:24][app.vjepa.train     ][main                     ] Initialized (rank/world-size) 4/16, tasks_per_node=4
+[INFO    ][2026-05-09 08:21:24][root                ][stage_datasets           ] [local_rank 0/4] Staging kinetics_240 (targz_parts)
+[INFO    ][2026-05-09 08:21:24][root                ][_stage_targz_parts       ] [rank 0] Extracting 26/103 tar.gz parts to /scratch-node/dcanez.22595640/kinetics_240
+[INFO    ][2026-05-09 08:21:38][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 2/26 parts
+[INFO    ][2026-05-09 08:21:53][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 4/26 parts
+[INFO    ][2026-05-09 08:22:07][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 6/26 parts
+[INFO    ][2026-05-09 08:22:21][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 8/26 parts
+[INFO    ][2026-05-09 08:22:35][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 10/26 parts
+[INFO    ][2026-05-09 08:22:50][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 12/26 parts
+[INFO    ][2026-05-09 08:23:04][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 14/26 parts
+[INFO    ][2026-05-09 08:23:18][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 16/26 parts
+[INFO    ][2026-05-09 08:23:33][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 18/26 parts
+[INFO    ][2026-05-09 08:23:47][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 20/26 parts
+[INFO    ][2026-05-09 08:24:01][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 22/26 parts
+[INFO    ][2026-05-09 08:24:16][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 24/26 parts
+[INFO    ][2026-05-09 08:24:30][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 26/26 parts
+[INFO    ][2026-05-09 08:24:30][root                ][stage_datasets           ] [local_rank 0/4] Staging ssv2 (multipart_tar)
+[INFO    ][2026-05-09 08:24:30][root                ][_stage_multipart_tar     ] [rank 0] Extracting multipart tar (2 files) to /scratch-node/dcanez.22595640/ssv2
+[INFO    ][2026-05-09 08:25:32][root                ][stage_datasets           ] Data staging completed in 247.8s (4.1min)
+[INFO    ][2026-05-09 08:25:33][root                ][_rewrite_csv             ] Wrote local CSV: /scratch-node/dcanez.22595640/kinetics_240/train.csv (239789 entries)
+[INFO    ][2026-05-09 08:25:33][root                ][_rewrite_csv             ] Wrote local CSV: /scratch-node/dcanez.22595640/ssv2/train.csv (168913 entries)
+[INFO    ][2026-05-09 08:25:33][app.vjepa.train     ][main                     ] Staged datasets: ['/scratch-node/dcanez.22595640/kinetics_240/train.csv', '/scratch-node/dcanez.22595640/ssv2/train.csv']
+[INFO    ][2026-05-09 08:25:57][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
+[INFO    ][2026-05-09 08:26:02][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA applied: 48 modules replaced (rank=8, alpha=16, dropout=0.0, targets=['attn.qkv', 'attn.proj'], scope=blocks)
+[INFO    ][2026-05-09 08:26:02][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA trainable params: 1,179,648
+[INFO    ][2026-05-09 08:26:05][root                ][_build_frozen_2d_target  ] Loaded frozen_2d target from timm: vit_large_patch14_capi.lvd1689m
+[INFO    ][2026-05-09 08:26:05][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-09 08:26:05][root                ][init_video_model         ] VJEPAPredictorMultiSeqWrapper(
+  (backbone): VisionTransformerPredictor(
+    (predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
+    (mask_tokens): ParameterList(
+        (0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+    )
+    (predictor_blocks): ModuleList(
+      (0-5): 6 x Block(
+        (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (attn): RoPEAttention(
+          (qkv): Linear(in_features=384, out_features=1152, bias=True)
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): Linear(in_features=384, out_features=384, bias=True)
+          (proj_drop): Dropout(p=0.0, inplace=False)
+        )
+        (drop_path): Identity()
+        (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (mlp): MLP(
+          (fc1): Linear(in_features=384, out_features=1536, bias=True)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=1536, out_features=384, bias=True)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+    (predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
+  )
+)
+[INFO    ][2026-05-09 08:26:05][root                ][init_video_model         ] MultiSeqWrapper(
+  (backbone): Frozen2DTargetWrapper(
+    (backbone): Eva(
+      (patch_embed): PatchEmbed(
+        (proj): Conv2d(3, 1024, kernel_size=(14, 14), stride=(14, 14))
+        (norm): Identity()
+      )
+      (pos_drop): Dropout(p=0.0, inplace=False)
+      (norm_pre): Identity()
+      (blocks): ModuleList(
+        (0-23): 24 x EvaBlock(
+          (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+          (attn): EvaAttention(
+            (qkv): Linear(in_features=1024, out_features=3072, bias=False)
+            (q_norm): Identity()
+            (k_norm): Identity()
+            (attn_drop): Dropout(p=0.0, inplace=False)
+            (norm): Identity()
+            (proj): Linear(in_features=1024, out_features=1024, bias=True)
+            (proj_drop): Dropout(p=0.0, inplace=False)
+          )
+          (drop_path1): Identity()
+          (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+          (mlp): Mlp(
+            (fc1): Linear(in_features=1024, out_features=4096, bias=True)
+            (act): GELU(approximate='none')
+            (drop1): Dropout(p=0.0, inplace=False)
+            (norm): Identity()
+            (fc2): Linear(in_features=4096, out_features=1024, bias=True)
+            (drop2): Dropout(p=0.0, inplace=False)
+          )
+          (drop_path2): Identity()
+        )
+      )
+      (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+      (fc_norm): Identity()
+      (head_drop): Dropout(p=0.0, inplace=False)
+      (head): Identity()
+      (rope): _CapiPatchRoPE()
+    )
+  )
+)
+[INFO    ][2026-05-09 08:26:05][root                ][init_video_model         ] Encoder number of parameters: 1179648
+[INFO    ][2026-05-09 08:26:05][root                ][init_video_model         ] Predictor number of parameters: 11436928
+[INFO    ][2026-05-09 08:26:05][root                ][init_video_model         ] Target encoder number of parameters: 0
+[INFO    ][2026-05-09 08:26:06][root                ][make_videodataset        ] VideoDataset dataset created
+[INFO    ][2026-05-09 08:26:06][WeightedSampler     ][__init__                 ] Using DistributedWeightedSampler with rank 4 / 16
+[INFO    ][2026-05-09 08:26:06][root                ][make_videodataset        ] VideoDataset unsupervised data loader created
+[INFO    ][2026-05-09 08:26:06][app.vjepa.train     ][main                     ] iterations per epoch/dataset length: 300/399
+[INFO    ][2026-05-09 08:26:06][app.vjepa.train     ][main                     ] Wrapping models in DDP (rank 4)...
+[INFO    ][2026-05-09 08:26:37][root                ][load_checkpoint          ] Loading checkpoint from /scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/latest.pt
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] loaded pretrained encoder from epoch 39 with msg: <All keys matched successfully>
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] loaded pretrained predictor from epoch 39 with msg: <All keys matched successfully>
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] loaded pretrained target encoder from epoch 39 with msg: <All keys matched successfully>
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] loaded optimizers from epoch 39
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] read-path: /scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/latest.pt
+[INFO    ][2026-05-09 08:26:39][app.vjepa.train     ][main                     ] Initializing loader...
+submitit WARNING (2026-05-09 09:56:00,866) - Bypassing signal SIGTERM
+[WARNING ][2026-05-09 09:56:00][submitit            ][bypass                   ] Bypassing signal SIGTERM
+submitit WARNING (2026-05-09 09:56:00,866) - Bypassing signal SIGCONT
+[WARNING ][2026-05-09 09:56:00][submitit            ][bypass                   ] Bypassing signal SIGCONT

vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/job_22595640/22595640_5_log.err

@@ -0,0 +1,305 @@
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
+  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
+[rank5]:[W508 20:22:16.125638464 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
+  scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+submitit WARNING (2026-05-09 08:16:14,374) - Caught signal SIGUSR2 on gcn76.local.snellius.surf.nl: this job is timed-out.
+submitit WARNING (2026-05-09 08:16:14,374) - Bypassing signal SIGTERM
+submitit WARNING (2026-05-09 08:16:14,375) - Bypassing signal SIGCONT
+[rank5]:[W509 08:16:18.961869388 TCPStore.cpp:125] [c10d] recvValue failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50474, remote=[gcn75.local.snellius.surf.nl]:37129): Failed to recv, got 0 bytes. Connection was likely closed. Did the remote server shutdown or crash?
+Exception raised from recvBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:682 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x148cf5dbcfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a3325d (0x148d39f4b25d in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x273 (0x148d39f491f3 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x148cf702beec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x148e36495164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x148e46f0719a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x148e46f8c100 in /lib64/libc.so.6)
+
+[rank5]:[W509 08:16:18.964654256 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 5] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Failed to recv, got 0 bytes. Connection was likely closed. Did the remote server shutdown or crash?
+[rank5]:[W509 08:16:19.964823682 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50474, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x148cf5dbcfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x148d39f4a6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x148d39f491cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x148cf702beec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x148e36495164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x148e46f0719a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x148e46f8c100 in /lib64/libc.so.6)
+
+[rank5]:[W509 08:16:19.967888789 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 5] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank5]:[W509 08:16:20.968044065 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50474, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x148cf5dbcfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x148d39f4a6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x148d39f491cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x148cf702beec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x148e36495164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x148e46f0719a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x148e46f8c100 in /lib64/libc.so.6)
+
+[rank5]:[W509 08:16:20.970252576 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 5] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank5]:[W509 08:16:21.970396192 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50474, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x148cf5dbcfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x148d39f4a6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x148d39f491cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x148cf702beec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x148e36495164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x148e46f0719a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x148e46f8c100 in /lib64/libc.so.6)
+
+[rank5]:[W509 08:16:21.972790971 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 5] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank5]:[W509 08:16:22.972919838 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50474, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x148cf5dbcfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x148d39f4a6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x148d39f491cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x148cf702beec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x148e36495164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x148e46f0719a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x148e46f8c100 in /lib64/libc.so.6)
+
+[rank5]:[W509 08:16:22.975225848 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 5] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank5]:[W509 08:16:23.975351764 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50474, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x148cf5dbcfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x148d39f4a6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x148d39f491cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x148cf702beec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x148e36495164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x148e46f0719a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x148e46f8c100 in /lib64/libc.so.6)
+
+[rank5]:[W509 08:16:23.977572185 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 5] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank5]:[W509 08:16:24.977700631 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50474, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x148cf5dbcfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x148d39f4a6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x148d39f491cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x148cf702beec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x148e36495164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x148e46f0719a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x148e46f8c100 in /lib64/libc.so.6)
+
+[rank5]:[W509 08:16:24.980040811 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 5] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank5]:[W509 08:16:25.980167357 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50474, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x148cf5dbcfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x148d39f4a6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x148d39f491cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x148cf702beec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x148e36495164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x148e46f0719a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x148e46f8c100 in /lib64/libc.so.6)
+
+[rank5]:[W509 08:16:25.982434178 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 5] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank5]:[W509 08:16:26.982568684 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50474, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x148cf5dbcfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x148d39f4a6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x148d39f491cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x148cf702beec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x148e36495164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x148e46f0719a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x148e46f8c100 in /lib64/libc.so.6)
+
+[rank5]:[W509 08:16:26.984882904 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 5] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank5]:[W509 08:16:27.985015290 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50474, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x148cf5dbcfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x148d39f4a6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x148d39f491cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x148cf702beec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x148e36495164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x148e46f0719a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x148e46f8c100 in /lib64/libc.so.6)
+
+[rank5]:[W509 08:16:27.987338590 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 5] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank5]:[W509 08:16:28.987469217 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50474, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x148cf5dbcfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x148d39f4a6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x148d39f491cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x148cf702beec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x148e36495164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x148e46f0719a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x148e46f8c100 in /lib64/libc.so.6)
+
+[rank5]:[W509 08:16:28.989799666 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 5] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank5]:[W509 08:16:29.989925653 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50474, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x148cf5dbcfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x148d39f4a6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x148d39f491cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x148cf702beec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x148e36495164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x148e46f0719a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x148e46f8c100 in /lib64/libc.so.6)
+
+[rank5]:[W509 08:16:29.992147583 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 5] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank5]:[W509 08:16:30.992289470 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50474, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x148cf5dbcfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x148d39f4a6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x148d39f491cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x148cf702beec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x148e36495164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x148e46f0719a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x148e46f8c100 in /lib64/libc.so.6)
+
+[rank5]:[W509 08:16:30.994571090 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 5] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank5]:[W509 08:16:31.994705266 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50474, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x148cf5dbcfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x148d39f4a6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x148d39f491cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x148cf702beec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x148e36495164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x148e46f0719a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x148e46f8c100 in /lib64/libc.so.6)
+
+[rank5]:[W509 08:16:31.996950096 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 5] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank5]:[W509 08:16:32.997086463 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50474, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x148cf5dbcfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x148d39f4a6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x148d39f491cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x148cf702beec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x148e36495164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x148e46f0719a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x148e46f8c100 in /lib64/libc.so.6)
+
+[rank5]:[W509 08:16:32.999243373 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 5] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank5]:[W509 08:16:33.999377320 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50474, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x148cf5dbcfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x148d39f4a6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x148d39f491cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x148cf702beec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x148e36495164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x148e46f0719a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x148e46f8c100 in /lib64/libc.so.6)
+
+[rank5]:[W509 08:16:33.001698050 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 5] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank5]:[W509 08:16:34.001835516 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50474, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x148cf5dbcfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x148d39f4a6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x148d39f491cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x148cf702beec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x148e36495164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x148e46f0719a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x148e46f8c100 in /lib64/libc.so.6)
+
+[rank5]:[W509 08:16:34.004098336 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 5] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank5]:[W509 08:16:35.004248033 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50474, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x148cf5dbcfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x148d39f4a6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x148d39f491cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x148cf702beec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x148e36495164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x148e46f0719a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x148e46f8c100 in /lib64/libc.so.6)
+
+[rank5]:[W509 08:16:35.006486873 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 5] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank5]:[W509 08:16:36.006630659 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50474, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x148cf5dbcfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x148d39f4a6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x148d39f491cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x148cf702beec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x148e36495164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x148e46f0719a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x148e46f8c100 in /lib64/libc.so.6)
+
+[rank5]:[W509 08:16:36.008996709 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 5] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank5]:[W509 08:16:37.009142995 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50474, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x148cf5dbcfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x148d39f4a6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x148d39f491cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x148cf702beec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x148e36495164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x148e46f0719a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x148e46f8c100 in /lib64/libc.so.6)
+
+[rank5]:[W509 08:16:37.011331266 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 5] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank5]:[W509 08:16:38.011464742 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50474, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x148cf5dbcfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x148d39f4a6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x148d39f491cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x148cf702beec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x148e36495164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x148e46f0719a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x148e46f8c100 in /lib64/libc.so.6)
+
+[rank5]:[W509 08:16:38.013666623 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 5] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank5]:[W509 08:16:39.013797199 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50474, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x148cf5dbcfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x148d39f4a6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x148d39f491cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x148cf702beec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x148e36495164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x148e46f0719a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x148e46f8c100 in /lib64/libc.so.6)
+
+[rank5]:[W509 08:16:39.016045109 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 5] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank5]:[W509 08:16:40.016183006 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50474, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x148cf5dbcfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x148d39f4a6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x148d39f491cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x148cf702beec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x148e36495164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x148e46f0719a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x148e46f8c100 in /lib64/libc.so.6)
+
+[rank5]:[W509 08:16:40.018446746 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 5] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank5]:[W509 08:16:41.018591442 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50474, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x148cf5dbcfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x148d39f4a6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x148d39f491cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x148cf702beec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x148e36495164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x148e46f0719a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x148e46f8c100 in /lib64/libc.so.6)
+
+[rank5]:[W509 08:16:41.020853243 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 5] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank5]:[W509 08:16:42.020997399 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50474, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x148cf5dbcfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x148d39f4a6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x148d39f491cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x148cf702beec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x148e36495164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x148e46f0719a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x148e46f8c100 in /lib64/libc.so.6)
+
+[rank5]:[W509 08:16:42.023364159 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 5] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank5]:[W509 08:16:43.023502455 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50474, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x148cf5dbcfdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x148d39f4a6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x148d39f491cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x148cf702beec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x148e36495164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x148e46f0719a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x148e46f8c100 in /lib64/libc.so.6)
+
+[rank5]:[W509 08:16:43.025819015 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 5] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
+  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
+[rank5]:[W509 08:24:23.406352196 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
+  scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+submitit WARNING (2026-05-09 09:56:00,984) - Bypassing signal SIGTERM
+submitit WARNING (2026-05-09 09:56:00,984) - Bypassing signal SIGCONT

vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/job_22595640/22595640_5_log.out

@@ -0,0 +1,530 @@
+submitit INFO (2026-05-08 20:18:11,423) - Starting with JobEnvironment(job_id=22595640, hostname=gcn76.local.snellius.surf.nl, local_rank=1(4), node=1(4), global_rank=5(16))
+submitit INFO (2026-05-08 20:18:11,423) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/job_22595640/22595640_submitted.pkl
+INFO:root:loaded pretrain params...
+{   'app': 'vjepa',
+    'cpus_per_task': 16,
+    'data': {   'batch_size': 64,
+                'crop_size': 224,
+                'dataset_fpcs': [16, 16],
+                'dataset_type': 'VideoDataset',
+                'datasets': [   '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
+                                '/scratch-shared/dcanez/data/ssv2/train.csv'],
+                'datasets_weights': [0.65, 0.35],
+                'fps': 4,
+                'num_workers': 10,
+                'patch_size': 14,
+                'persistent_workers': True,
+                'pin_mem': True,
+                'stage': [   {   'dest': 'kinetics_240',
+                                 'format': 'targz_parts',
+                                 'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
+                             {   'dest': 'ssv2',
+                                 'format': 'multipart_tar',
+                                 'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
+                'tubelet_size': 1},
+    'data_aug': {   'auto_augment': False,
+                    'motion_shift': False,
+                    'random_resize_aspect_ratio': [0.75, 1.35],
+                    'random_resize_scale': [0.3, 1.0],
+                    'reprob': 0.0},
+    'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m',
+    'loss': {'loss_exp': 1.0},
+    'mask': [   {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 8,
+                    'spatial_scale': [0.15, 0.15],
+                    'temporal_scale': [1.0, 1.0]},
+                {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 2,
+                    'spatial_scale': [0.7, 0.7],
+                    'temporal_scale': [1.0, 1.0]}],
+    'mem_per_gpu': '180G',
+    'meta': {   'dtype': 'bfloat16',
+                'knn_eval_epoch0': False,
+                'knn_eval_freq': 5,
+                'knn_eval_presets': [   {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
+                                                          'eval_videos_per_class': 25,
+                                                          'num_workers': 8,
+                                                          'train_videos_per_class': 100},
+                                            'preset': 'ucf101'},
+                                        {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
+                                                          'eval_videos_per_class': 100,
+                                                          'linear_probe': True,
+                                                          'num_workers': 8,
+                                                          'pool_type': 'temporal_concat',
+                                                          'train_videos_per_class': 500},
+                                            'preset': 'ssv2_coarse10'}],
+                'load_checkpoint': True,
+                'read_checkpoint': None,
+                'save_every_freq': 5,
+                'seed': 239,
+                'use_sdpa': True,
+                'use_wandb': True,
+                'wandb_project': 'vjepa_ablation'},
+    'metrics': {'sigreg': {}, 'std': {}},
+    'model': {   'class_token': False,
+                 'freeze_backbone': True,
+                 'is_causal': False,
+                 'lora_alpha': 16,
+                 'lora_rank': 8,
+                 'lora_targets': ['attn.qkv', 'attn.proj'],
+                 'model_name': 'vit_large_patch14_capi_lvd1689m',
+                 'num_reg_tokens': 16,
+                 'pred_depth': 6,
+                 'pred_embed_dim': 384,
+                 'pred_num_heads': 12,
+                 'rope_type': 'capi2d',
+                 'stem_type': '2d',
+                 'target_kind': 'frozen_2d',
+                 'target_type': 'vit_large_patch14_capi.lvd1689m',
+                 'uniform_power': True,
+                 'use_activation_checkpointing': True,
+                 'use_lora': True,
+                 'use_mask_tokens': True,
+                 'use_rope': True,
+                 'use_sdpa': True,
+                 'zero_init_mask_tokens': True},
+    'nodes': 4,
+    'optimization': {   'ema': [0.99925, 0.99925],
+                        'epochs': 100,
+                        'final_lr': 0.0001,
+                        'final_weight_decay': 0.04,
+                        'ipe': 300,
+                        'ipe_scale': 1.0,
+                        'lr': 0.0005,
+                        'start_lr': 0.0001,
+                        'warmup': 10,
+                        'weight_decay': 0.04},
+    'tasks_per_node': 4}
+INFO:root:Running pre-training of app: vjepa
+[INFO    ][2026-05-08 20:19:11][app.vjepa.train     ][main                     ] which_dtype='bfloat16'
+[INFO    ][2026-05-08 20:19:11][app.vjepa.train     ][main                     ] Disabling persistent_workers (incompatible with KNN eval)
+[INFO    ][2026-05-08 20:19:11][app.vjepa.train     ][main                     ] NCCL_SOCKET_IFNAME=eno
+[INFO    ][2026-05-08 20:19:13][app.vjepa.train     ][main                     ] Initialized (rank/world-size) 5/16, tasks_per_node=4
+[INFO    ][2026-05-08 20:19:13][root                ][stage_datasets           ] [local_rank 1/4] Staging kinetics_240 (targz_parts)
+[INFO    ][2026-05-08 20:19:13][root                ][_stage_targz_parts       ] [rank 1] Extracting 26/103 tar.gz parts to /scratch-node/dcanez.22595640/kinetics_240
+[INFO    ][2026-05-08 20:19:26][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 2/26 parts
+[INFO    ][2026-05-08 20:19:40][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 4/26 parts
+[INFO    ][2026-05-08 20:19:54][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 6/26 parts
+[INFO    ][2026-05-08 20:20:08][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 8/26 parts
+[INFO    ][2026-05-08 20:20:22][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 10/26 parts
+[INFO    ][2026-05-08 20:20:37][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 12/26 parts
+[INFO    ][2026-05-08 20:20:50][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 14/26 parts
+[INFO    ][2026-05-08 20:21:05][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 16/26 parts
+[INFO    ][2026-05-08 20:21:19][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 18/26 parts
+[INFO    ][2026-05-08 20:21:33][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 20/26 parts
+[INFO    ][2026-05-08 20:21:47][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 22/26 parts
+[INFO    ][2026-05-08 20:22:02][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 24/26 parts
+[INFO    ][2026-05-08 20:22:16][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 26/26 parts
+[INFO    ][2026-05-08 20:22:16][root                ][stage_datasets           ] [local_rank 1/4] Staging ssv2 (multipart_tar)
+[INFO    ][2026-05-08 20:23:17][app.vjepa.train     ][main                     ] Staged datasets: ['/scratch-node/dcanez.22595640/kinetics_240/train.csv', '/scratch-node/dcanez.22595640/ssv2/train.csv']
+[INFO    ][2026-05-08 20:24:16][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
+[INFO    ][2026-05-08 20:24:20][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA applied: 48 modules replaced (rank=8, alpha=16, dropout=0.0, targets=['attn.qkv', 'attn.proj'], scope=blocks)
+[INFO    ][2026-05-08 20:24:20][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA trainable params: 1,179,648
+[INFO    ][2026-05-08 20:24:26][root                ][_build_frozen_2d_target  ] Loaded frozen_2d target from timm: vit_large_patch14_capi.lvd1689m
+[INFO    ][2026-05-08 20:24:26][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-08 20:24:26][root                ][init_video_model         ] VJEPAPredictorMultiSeqWrapper(
+  (backbone): VisionTransformerPredictor(
+    (predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
+    (mask_tokens): ParameterList(
+        (0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+    )
+    (predictor_blocks): ModuleList(
+      (0-5): 6 x Block(
+        (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (attn): RoPEAttention(
+          (qkv): Linear(in_features=384, out_features=1152, bias=True)
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): Linear(in_features=384, out_features=384, bias=True)
+          (proj_drop): Dropout(p=0.0, inplace=False)
+        )
+        (drop_path): Identity()
+        (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (mlp): MLP(
+          (fc1): Linear(in_features=384, out_features=1536, bias=True)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=1536, out_features=384, bias=True)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+    (predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
+  )
+)
+[INFO    ][2026-05-08 20:24:26][root                ][init_video_model         ] MultiSeqWrapper(
+  (backbone): Frozen2DTargetWrapper(
+    (backbone): Eva(
+      (patch_embed): PatchEmbed(
+        (proj): Conv2d(3, 1024, kernel_size=(14, 14), stride=(14, 14))
+        (norm): Identity()
+      )
+      (pos_drop): Dropout(p=0.0, inplace=False)
+      (norm_pre): Identity()
+      (blocks): ModuleList(
+        (0-23): 24 x EvaBlock(
+          (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+          (attn): EvaAttention(
+            (qkv): Linear(in_features=1024, out_features=3072, bias=False)
+            (q_norm): Identity()
+            (k_norm): Identity()
+            (attn_drop): Dropout(p=0.0, inplace=False)
+            (norm): Identity()
+            (proj): Linear(in_features=1024, out_features=1024, bias=True)
+            (proj_drop): Dropout(p=0.0, inplace=False)
+          )
+          (drop_path1): Identity()
+          (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+          (mlp): Mlp(
+            (fc1): Linear(in_features=1024, out_features=4096, bias=True)
+            (act): GELU(approximate='none')
+            (drop1): Dropout(p=0.0, inplace=False)
+            (norm): Identity()
+            (fc2): Linear(in_features=4096, out_features=1024, bias=True)
+            (drop2): Dropout(p=0.0, inplace=False)
+          )
+          (drop_path2): Identity()
+        )
+      )
+      (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+      (fc_norm): Identity()
+      (head_drop): Dropout(p=0.0, inplace=False)
+      (head): Identity()
+      (rope): _CapiPatchRoPE()
+    )
+  )
+)
+[INFO    ][2026-05-08 20:24:26][root                ][init_video_model         ] Encoder number of parameters: 1179648
+[INFO    ][2026-05-08 20:24:26][root                ][init_video_model         ] Predictor number of parameters: 11436928
+[INFO    ][2026-05-08 20:24:26][root                ][init_video_model         ] Target encoder number of parameters: 0
+[INFO    ][2026-05-08 20:24:29][root                ][make_videodataset        ] VideoDataset dataset created
+[INFO    ][2026-05-08 20:24:29][WeightedSampler     ][__init__                 ] Using DistributedWeightedSampler with rank 5 / 16
+[INFO    ][2026-05-08 20:24:29][root                ][make_videodataset        ] VideoDataset unsupervised data loader created
+[INFO    ][2026-05-08 20:24:29][app.vjepa.train     ][main                     ] iterations per epoch/dataset length: 300/399
+[INFO    ][2026-05-08 20:24:29][app.vjepa.train     ][main                     ] Wrapping models in DDP (rank 5)...
+[INFO    ][2026-05-08 20:24:41][app.vjepa.train     ][main                     ] Initializing loader...
+submitit INFO (2026-05-09 08:16:14,354) - Job has timed out. Ran 718 minutes out of requested 720 minutes.
+[INFO    ][2026-05-09 08:16:14][submitit            ][has_timed_out            ] Job has timed out. Ran 718 minutes out of requested 720 minutes.
+submitit WARNING (2026-05-09 08:16:14,374) - Caught signal SIGUSR2 on gcn76.local.snellius.surf.nl: this job is timed-out.
+[WARNING ][2026-05-09 08:16:14][submitit            ][checkpoint_and_try_requeue] Caught signal SIGUSR2 on gcn76.local.snellius.surf.nl: this job is timed-out.
+submitit INFO (2026-05-09 08:16:14,374) - Not checkpointing nor requeuing since I am a slave (procid=5).
+[INFO    ][2026-05-09 08:16:14][submitit            ][checkpoint_and_try_requeue] Not checkpointing nor requeuing since I am a slave (procid=5).
+submitit WARNING (2026-05-09 08:16:14,374) - Bypassing signal SIGTERM
+[WARNING ][2026-05-09 08:16:14][submitit            ][bypass                   ] Bypassing signal SIGTERM
+submitit WARNING (2026-05-09 08:16:14,375) - Bypassing signal SIGCONT
+[WARNING ][2026-05-09 08:16:14][submitit            ][bypass                   ] Bypassing signal SIGCONT
+submitit INFO (2026-05-09 08:19:43,911) - Starting with JobEnvironment(job_id=22595640, hostname=gcn75.local.snellius.surf.nl, local_rank=1(4), node=1(4), global_rank=5(16))
+submitit INFO (2026-05-09 08:19:44,474) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/job_22595640/22595640_submitted.pkl
+INFO:root:loaded pretrain params...
+{   'app': 'vjepa',
+    'cpus_per_task': 16,
+    'data': {   'batch_size': 64,
+                'crop_size': 224,
+                'dataset_fpcs': [16, 16],
+                'dataset_type': 'VideoDataset',
+                'datasets': [   '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
+                                '/scratch-shared/dcanez/data/ssv2/train.csv'],
+                'datasets_weights': [0.65, 0.35],
+                'fps': 4,
+                'num_workers': 10,
+                'patch_size': 14,
+                'persistent_workers': True,
+                'pin_mem': True,
+                'stage': [   {   'dest': 'kinetics_240',
+                                 'format': 'targz_parts',
+                                 'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
+                             {   'dest': 'ssv2',
+                                 'format': 'multipart_tar',
+                                 'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
+                'tubelet_size': 1},
+    'data_aug': {   'auto_augment': False,
+                    'motion_shift': False,
+                    'random_resize_aspect_ratio': [0.75, 1.35],
+                    'random_resize_scale': [0.3, 1.0],
+                    'reprob': 0.0},
+    'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m',
+    'loss': {'loss_exp': 1.0},
+    'mask': [   {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 8,
+                    'spatial_scale': [0.15, 0.15],
+                    'temporal_scale': [1.0, 1.0]},
+                {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 2,
+                    'spatial_scale': [0.7, 0.7],
+                    'temporal_scale': [1.0, 1.0]}],
+    'mem_per_gpu': '180G',
+    'meta': {   'dtype': 'bfloat16',
+                'knn_eval_epoch0': False,
+                'knn_eval_freq': 5,
+                'knn_eval_presets': [   {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
+                                                          'eval_videos_per_class': 25,
+                                                          'num_workers': 8,
+                                                          'train_videos_per_class': 100},
+                                            'preset': 'ucf101'},
+                                        {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
+                                                          'eval_videos_per_class': 100,
+                                                          'linear_probe': True,
+                                                          'num_workers': 8,
+                                                          'pool_type': 'temporal_concat',
+                                                          'train_videos_per_class': 500},
+                                            'preset': 'ssv2_coarse10'}],
+                'load_checkpoint': True,
+                'read_checkpoint': None,
+                'save_every_freq': 5,
+                'seed': 239,
+                'use_sdpa': True,
+                'use_wandb': True,
+                'wandb_project': 'vjepa_ablation'},
+    'metrics': {'sigreg': {}, 'std': {}},
+    'model': {   'class_token': False,
+                 'freeze_backbone': True,
+                 'is_causal': False,
+                 'lora_alpha': 16,
+                 'lora_rank': 8,
+                 'lora_targets': ['attn.qkv', 'attn.proj'],
+                 'model_name': 'vit_large_patch14_capi_lvd1689m',
+                 'num_reg_tokens': 16,
+                 'pred_depth': 6,
+                 'pred_embed_dim': 384,
+                 'pred_num_heads': 12,
+                 'rope_type': 'capi2d',
+                 'stem_type': '2d',
+                 'target_kind': 'frozen_2d',
+                 'target_type': 'vit_large_patch14_capi.lvd1689m',
+                 'uniform_power': True,
+                 'use_activation_checkpointing': True,
+                 'use_lora': True,
+                 'use_mask_tokens': True,
+                 'use_rope': True,
+                 'use_sdpa': True,
+                 'zero_init_mask_tokens': True},
+    'nodes': 4,
+    'optimization': {   'ema': [0.99925, 0.99925],
+                        'epochs': 100,
+                        'final_lr': 0.0001,
+                        'final_weight_decay': 0.04,
+                        'ipe': 300,
+                        'ipe_scale': 1.0,
+                        'lr': 0.0005,
+                        'start_lr': 0.0001,
+                        'warmup': 10,
+                        'weight_decay': 0.04},
+    'tasks_per_node': 4}
+INFO:root:Running pre-training of app: vjepa
+[INFO    ][2026-05-09 08:19:53][app.vjepa.train     ][main                     ] which_dtype='bfloat16'
+[INFO    ][2026-05-09 08:19:53][app.vjepa.train     ][main                     ] Disabling persistent_workers (incompatible with KNN eval)
+[INFO    ][2026-05-09 08:19:53][app.vjepa.train     ][main                     ] NCCL_SOCKET_IFNAME=eno
+[INFO    ][2026-05-09 08:21:20][app.vjepa.train     ][main                     ] Initialized (rank/world-size) 5/16, tasks_per_node=4
+[INFO    ][2026-05-09 08:21:20][root                ][stage_datasets           ] [local_rank 1/4] Staging kinetics_240 (targz_parts)
+[INFO    ][2026-05-09 08:21:20][root                ][_stage_targz_parts       ] [rank 1] Extracting 26/103 tar.gz parts to /scratch-node/dcanez.22595640/kinetics_240
+[INFO    ][2026-05-09 08:21:32][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 2/26 parts
+[INFO    ][2026-05-09 08:21:46][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 4/26 parts
+[INFO    ][2026-05-09 08:22:00][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 6/26 parts
+[INFO    ][2026-05-09 08:22:14][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 8/26 parts
+[INFO    ][2026-05-09 08:22:29][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 10/26 parts
+[INFO    ][2026-05-09 08:22:43][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 12/26 parts
+[INFO    ][2026-05-09 08:22:56][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 14/26 parts
+[INFO    ][2026-05-09 08:23:10][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 16/26 parts
+[INFO    ][2026-05-09 08:23:25][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 18/26 parts
+[INFO    ][2026-05-09 08:23:39][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 20/26 parts
+[INFO    ][2026-05-09 08:23:54][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 22/26 parts
+[INFO    ][2026-05-09 08:24:08][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 24/26 parts
+[INFO    ][2026-05-09 08:24:22][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 26/26 parts
+[INFO    ][2026-05-09 08:24:22][root                ][stage_datasets           ] [local_rank 1/4] Staging ssv2 (multipart_tar)
+[INFO    ][2026-05-09 08:25:33][app.vjepa.train     ][main                     ] Staged datasets: ['/scratch-node/dcanez.22595640/kinetics_240/train.csv', '/scratch-node/dcanez.22595640/ssv2/train.csv']
+[INFO    ][2026-05-09 08:25:57][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
+[INFO    ][2026-05-09 08:26:02][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA applied: 48 modules replaced (rank=8, alpha=16, dropout=0.0, targets=['attn.qkv', 'attn.proj'], scope=blocks)
+[INFO    ][2026-05-09 08:26:02][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA trainable params: 1,179,648
+[INFO    ][2026-05-09 08:26:05][root                ][_build_frozen_2d_target  ] Loaded frozen_2d target from timm: vit_large_patch14_capi.lvd1689m
+[INFO    ][2026-05-09 08:26:05][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-09 08:26:05][root                ][init_video_model         ] VJEPAPredictorMultiSeqWrapper(
+  (backbone): VisionTransformerPredictor(
+    (predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
+    (mask_tokens): ParameterList(
+        (0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+    )
+    (predictor_blocks): ModuleList(
+      (0-5): 6 x Block(
+        (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (attn): RoPEAttention(
+          (qkv): Linear(in_features=384, out_features=1152, bias=True)
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): Linear(in_features=384, out_features=384, bias=True)
+          (proj_drop): Dropout(p=0.0, inplace=False)
+        )
+        (drop_path): Identity()
+        (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (mlp): MLP(
+          (fc1): Linear(in_features=384, out_features=1536, bias=True)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=1536, out_features=384, bias=True)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+    (predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
+  )
+)
+[INFO    ][2026-05-09 08:26:05][root                ][init_video_model         ] MultiSeqWrapper(
+  (backbone): Frozen2DTargetWrapper(
+    (backbone): Eva(
+      (patch_embed): PatchEmbed(
+        (proj): Conv2d(3, 1024, kernel_size=(14, 14), stride=(14, 14))
+        (norm): Identity()
+      )
+      (pos_drop): Dropout(p=0.0, inplace=False)
+      (norm_pre): Identity()
+      (blocks): ModuleList(
+        (0-23): 24 x EvaBlock(
+          (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+          (attn): EvaAttention(
+            (qkv): Linear(in_features=1024, out_features=3072, bias=False)
+            (q_norm): Identity()
+            (k_norm): Identity()
+            (attn_drop): Dropout(p=0.0, inplace=False)
+            (norm): Identity()
+            (proj): Linear(in_features=1024, out_features=1024, bias=True)
+            (proj_drop): Dropout(p=0.0, inplace=False)
+          )
+          (drop_path1): Identity()
+          (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+          (mlp): Mlp(
+            (fc1): Linear(in_features=1024, out_features=4096, bias=True)
+            (act): GELU(approximate='none')
+            (drop1): Dropout(p=0.0, inplace=False)
+            (norm): Identity()
+            (fc2): Linear(in_features=4096, out_features=1024, bias=True)
+            (drop2): Dropout(p=0.0, inplace=False)
+          )
+          (drop_path2): Identity()
+        )
+      )
+      (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+      (fc_norm): Identity()
+      (head_drop): Dropout(p=0.0, inplace=False)
+      (head): Identity()
+      (rope): _CapiPatchRoPE()
+    )
+  )
+)
+[INFO    ][2026-05-09 08:26:05][root                ][init_video_model         ] Encoder number of parameters: 1179648
+[INFO    ][2026-05-09 08:26:05][root                ][init_video_model         ] Predictor number of parameters: 11436928
+[INFO    ][2026-05-09 08:26:05][root                ][init_video_model         ] Target encoder number of parameters: 0
+[INFO    ][2026-05-09 08:26:06][root                ][make_videodataset        ] VideoDataset dataset created
+[INFO    ][2026-05-09 08:26:06][WeightedSampler     ][__init__                 ] Using DistributedWeightedSampler with rank 5 / 16
+[INFO    ][2026-05-09 08:26:06][root                ][make_videodataset        ] VideoDataset unsupervised data loader created
+[INFO    ][2026-05-09 08:26:06][app.vjepa.train     ][main                     ] iterations per epoch/dataset length: 300/399
+[INFO    ][2026-05-09 08:26:06][app.vjepa.train     ][main                     ] Wrapping models in DDP (rank 5)...
+[INFO    ][2026-05-09 08:26:37][root                ][load_checkpoint          ] Loading checkpoint from /scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/latest.pt
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] loaded pretrained encoder from epoch 39 with msg: <All keys matched successfully>
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] loaded pretrained predictor from epoch 39 with msg: <All keys matched successfully>
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] loaded pretrained target encoder from epoch 39 with msg: <All keys matched successfully>
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] loaded optimizers from epoch 39
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] read-path: /scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/latest.pt
+[INFO    ][2026-05-09 08:26:39][app.vjepa.train     ][main                     ] Initializing loader...
+submitit WARNING (2026-05-09 09:56:00,984) - Bypassing signal SIGTERM
+[WARNING ][2026-05-09 09:56:00][submitit            ][bypass                   ] Bypassing signal SIGTERM
+submitit WARNING (2026-05-09 09:56:00,984) - Bypassing signal SIGCONT
+[WARNING ][2026-05-09 09:56:00][submitit            ][bypass                   ] Bypassing signal SIGCONT

vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/job_22595640/22595640_6_log.err

@@ -0,0 +1,305 @@
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
+  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
+[rank6]:[W508 20:22:19.668989392 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
+  scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+submitit WARNING (2026-05-09 08:16:14,388) - Caught signal SIGUSR2 on gcn76.local.snellius.surf.nl: this job is timed-out.
+submitit WARNING (2026-05-09 08:16:14,389) - Bypassing signal SIGTERM
+submitit WARNING (2026-05-09 08:16:14,389) - Bypassing signal SIGCONT
+[rank6]:[W509 08:16:18.714909170 TCPStore.cpp:125] [c10d] recvValue failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50470, remote=[gcn75.local.snellius.surf.nl]:37129): Failed to recv, got 0 bytes. Connection was likely closed. Did the remote server shutdown or crash?
+Exception raised from recvBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:682 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x153973362fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a3325d (0x1539b74f125d in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x273 (0x1539b74ef1f3 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1539745d1eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153ab3a3b164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x153ac44ad19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x153ac4532100 in /lib64/libc.so.6)
+
+[rank6]:[W509 08:16:18.718030307 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 6] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Failed to recv, got 0 bytes. Connection was likely closed. Did the remote server shutdown or crash?
+[rank6]:[W509 08:16:19.718221413 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50470, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x153973362fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1539b74f06d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1539b74ef1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1539745d1eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153ab3a3b164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x153ac44ad19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x153ac4532100 in /lib64/libc.so.6)
+
+[rank6]:[W509 08:16:19.721332840 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 6] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank6]:[W509 08:16:20.721486096 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50470, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x153973362fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1539b74f06d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1539b74ef1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1539745d1eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153ab3a3b164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x153ac44ad19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x153ac4532100 in /lib64/libc.so.6)
+
+[rank6]:[W509 08:16:20.723995495 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 6] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank6]:[W509 08:16:21.724141691 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50470, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x153973362fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1539b74f06d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1539b74ef1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1539745d1eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153ab3a3b164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x153ac44ad19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x153ac4532100 in /lib64/libc.so.6)
+
+[rank6]:[W509 08:16:21.726483451 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 6] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank6]:[W509 08:16:22.726619148 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50470, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x153973362fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1539b74f06d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1539b74ef1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1539745d1eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153ab3a3b164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x153ac44ad19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x153ac4532100 in /lib64/libc.so.6)
+
+[rank6]:[W509 08:16:22.728968508 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 6] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank6]:[W509 08:16:23.729097834 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50470, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x153973362fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1539b74f06d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1539b74ef1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1539745d1eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153ab3a3b164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x153ac44ad19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x153ac4532100 in /lib64/libc.so.6)
+
+[rank6]:[W509 08:16:23.731545153 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 6] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank6]:[W509 08:16:24.731676420 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50470, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x153973362fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1539b74f06d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1539b74ef1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1539745d1eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153ab3a3b164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x153ac44ad19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x153ac4532100 in /lib64/libc.so.6)
+
+[rank6]:[W509 08:16:24.733985050 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 6] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank6]:[W509 08:16:25.734126286 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50470, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x153973362fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1539b74f06d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1539b74ef1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1539745d1eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153ab3a3b164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x153ac44ad19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x153ac4532100 in /lib64/libc.so.6)
+
+[rank6]:[W509 08:16:25.736447166 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 6] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank6]:[W509 08:16:26.736582722 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50470, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x153973362fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1539b74f06d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1539b74ef1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1539745d1eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153ab3a3b164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x153ac44ad19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x153ac4532100 in /lib64/libc.so.6)
+
+[rank6]:[W509 08:16:26.738963682 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 6] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank6]:[W509 08:16:27.739094908 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50470, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x153973362fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1539b74f06d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1539b74ef1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1539745d1eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153ab3a3b164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x153ac44ad19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x153ac4532100 in /lib64/libc.so.6)
+
+[rank6]:[W509 08:16:27.741630758 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 6] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank6]:[W509 08:16:28.741750564 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50470, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x153973362fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1539b74f06d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1539b74ef1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1539745d1eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153ab3a3b164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x153ac44ad19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x153ac4532100 in /lib64/libc.so.6)
+
+[rank6]:[W509 08:16:28.743962544 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 6] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank6]:[W509 08:16:29.744105511 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50470, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x153973362fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1539b74f06d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1539b74ef1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1539745d1eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153ab3a3b164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x153ac44ad19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x153ac4532100 in /lib64/libc.so.6)
+
+[rank6]:[W509 08:16:29.746527990 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 6] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank6]:[W509 08:16:30.746658247 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50470, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x153973362fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1539b74f06d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1539b74ef1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1539745d1eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153ab3a3b164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x153ac44ad19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x153ac4532100 in /lib64/libc.so.6)
+
+[rank6]:[W509 08:16:30.748945017 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 6] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank6]:[W509 08:16:31.749067983 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50470, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x153973362fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1539b74f06d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1539b74ef1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1539745d1eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153ab3a3b164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x153ac44ad19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x153ac4532100 in /lib64/libc.so.6)
+
+[rank6]:[W509 08:16:31.751392483 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 6] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank6]:[W509 08:16:32.751522900 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50470, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x153973362fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1539b74f06d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1539b74ef1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1539745d1eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153ab3a3b164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x153ac44ad19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x153ac4532100 in /lib64/libc.so.6)
+
+[rank6]:[W509 08:16:32.753956939 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 6] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank6]:[W509 08:16:33.754088425 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50470, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x153973362fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1539b74f06d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1539b74ef1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1539745d1eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153ab3a3b164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x153ac44ad19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x153ac4532100 in /lib64/libc.so.6)
+
+[rank6]:[W509 08:16:33.756381645 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 6] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank6]:[W509 08:16:34.756504312 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50470, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x153973362fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1539b74f06d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1539b74ef1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1539745d1eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153ab3a3b164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x153ac44ad19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x153ac4532100 in /lib64/libc.so.6)
+
+[rank6]:[W509 08:16:34.758753252 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 6] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank6]:[W509 08:16:35.758874668 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50470, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x153973362fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1539b74f06d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1539b74ef1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1539745d1eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153ab3a3b164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x153ac44ad19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x153ac4532100 in /lib64/libc.so.6)
+
+[rank6]:[W509 08:16:35.761267028 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 6] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank6]:[W509 08:16:36.761391775 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50470, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x153973362fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1539b74f06d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1539b74ef1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1539745d1eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153ab3a3b164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x153ac44ad19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x153ac4532100 in /lib64/libc.so.6)
+
+[rank6]:[W509 08:16:36.763628445 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 6] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank6]:[W509 08:16:37.763752141 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50470, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x153973362fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1539b74f06d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1539b74ef1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1539745d1eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153ab3a3b164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x153ac44ad19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x153ac4532100 in /lib64/libc.so.6)
+
+[rank6]:[W509 08:16:37.766075831 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 6] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank6]:[W509 08:16:38.766205268 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50470, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x153973362fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1539b74f06d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1539b74ef1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1539745d1eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153ab3a3b164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x153ac44ad19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x153ac4532100 in /lib64/libc.so.6)
+
+[rank6]:[W509 08:16:38.768663127 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 6] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank6]:[W509 08:16:39.768799823 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50470, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x153973362fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1539b74f06d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1539b74ef1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1539745d1eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153ab3a3b164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x153ac44ad19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x153ac4532100 in /lib64/libc.so.6)
+
+[rank6]:[W509 08:16:39.771104254 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 6] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank6]:[W509 08:16:40.771248440 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50470, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x153973362fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1539b74f06d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1539b74ef1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1539745d1eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153ab3a3b164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x153ac44ad19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x153ac4532100 in /lib64/libc.so.6)
+
+[rank6]:[W509 08:16:40.773557200 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 6] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank6]:[W509 08:16:41.773681766 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50470, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x153973362fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1539b74f06d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1539b74ef1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1539745d1eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153ab3a3b164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x153ac44ad19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x153ac4532100 in /lib64/libc.so.6)
+
+[rank6]:[W509 08:16:41.776069886 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 6] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank6]:[W509 08:16:42.776191992 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50470, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x153973362fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1539b74f06d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1539b74ef1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1539745d1eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153ab3a3b164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x153ac44ad19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x153ac4532100 in /lib64/libc.so.6)
+
+[rank6]:[W509 08:16:42.778484153 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 6] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank6]:[W509 08:16:43.778603369 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50470, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x153973362fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x1539b74f06d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x1539b74ef1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1539745d1eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x153ab3a3b164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x153ac44ad19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x153ac4532100 in /lib64/libc.so.6)
+
+[rank6]:[W509 08:16:43.780814549 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 6] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
+  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
+[rank6]:[W509 08:24:27.514416053 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
+  scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+submitit WARNING (2026-05-09 09:56:00,971) - Bypassing signal SIGTERM
+submitit WARNING (2026-05-09 09:56:00,972) - Bypassing signal SIGCONT

vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/job_22595640/22595640_6_log.out

@@ -0,0 +1,530 @@
+submitit INFO (2026-05-08 20:18:11,423) - Starting with JobEnvironment(job_id=22595640, hostname=gcn76.local.snellius.surf.nl, local_rank=2(4), node=1(4), global_rank=6(16))
+submitit INFO (2026-05-08 20:18:11,423) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/job_22595640/22595640_submitted.pkl
+INFO:root:loaded pretrain params...
+{   'app': 'vjepa',
+    'cpus_per_task': 16,
+    'data': {   'batch_size': 64,
+                'crop_size': 224,
+                'dataset_fpcs': [16, 16],
+                'dataset_type': 'VideoDataset',
+                'datasets': [   '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
+                                '/scratch-shared/dcanez/data/ssv2/train.csv'],
+                'datasets_weights': [0.65, 0.35],
+                'fps': 4,
+                'num_workers': 10,
+                'patch_size': 14,
+                'persistent_workers': True,
+                'pin_mem': True,
+                'stage': [   {   'dest': 'kinetics_240',
+                                 'format': 'targz_parts',
+                                 'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
+                             {   'dest': 'ssv2',
+                                 'format': 'multipart_tar',
+                                 'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
+                'tubelet_size': 1},
+    'data_aug': {   'auto_augment': False,
+                    'motion_shift': False,
+                    'random_resize_aspect_ratio': [0.75, 1.35],
+                    'random_resize_scale': [0.3, 1.0],
+                    'reprob': 0.0},
+    'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m',
+    'loss': {'loss_exp': 1.0},
+    'mask': [   {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 8,
+                    'spatial_scale': [0.15, 0.15],
+                    'temporal_scale': [1.0, 1.0]},
+                {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 2,
+                    'spatial_scale': [0.7, 0.7],
+                    'temporal_scale': [1.0, 1.0]}],
+    'mem_per_gpu': '180G',
+    'meta': {   'dtype': 'bfloat16',
+                'knn_eval_epoch0': False,
+                'knn_eval_freq': 5,
+                'knn_eval_presets': [   {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
+                                                          'eval_videos_per_class': 25,
+                                                          'num_workers': 8,
+                                                          'train_videos_per_class': 100},
+                                            'preset': 'ucf101'},
+                                        {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
+                                                          'eval_videos_per_class': 100,
+                                                          'linear_probe': True,
+                                                          'num_workers': 8,
+                                                          'pool_type': 'temporal_concat',
+                                                          'train_videos_per_class': 500},
+                                            'preset': 'ssv2_coarse10'}],
+                'load_checkpoint': True,
+                'read_checkpoint': None,
+                'save_every_freq': 5,
+                'seed': 239,
+                'use_sdpa': True,
+                'use_wandb': True,
+                'wandb_project': 'vjepa_ablation'},
+    'metrics': {'sigreg': {}, 'std': {}},
+    'model': {   'class_token': False,
+                 'freeze_backbone': True,
+                 'is_causal': False,
+                 'lora_alpha': 16,
+                 'lora_rank': 8,
+                 'lora_targets': ['attn.qkv', 'attn.proj'],
+                 'model_name': 'vit_large_patch14_capi_lvd1689m',
+                 'num_reg_tokens': 16,
+                 'pred_depth': 6,
+                 'pred_embed_dim': 384,
+                 'pred_num_heads': 12,
+                 'rope_type': 'capi2d',
+                 'stem_type': '2d',
+                 'target_kind': 'frozen_2d',
+                 'target_type': 'vit_large_patch14_capi.lvd1689m',
+                 'uniform_power': True,
+                 'use_activation_checkpointing': True,
+                 'use_lora': True,
+                 'use_mask_tokens': True,
+                 'use_rope': True,
+                 'use_sdpa': True,
+                 'zero_init_mask_tokens': True},
+    'nodes': 4,
+    'optimization': {   'ema': [0.99925, 0.99925],
+                        'epochs': 100,
+                        'final_lr': 0.0001,
+                        'final_weight_decay': 0.04,
+                        'ipe': 300,
+                        'ipe_scale': 1.0,
+                        'lr': 0.0005,
+                        'start_lr': 0.0001,
+                        'warmup': 10,
+                        'weight_decay': 0.04},
+    'tasks_per_node': 4}
+INFO:root:Running pre-training of app: vjepa
+[INFO    ][2026-05-08 20:19:11][app.vjepa.train     ][main                     ] which_dtype='bfloat16'
+[INFO    ][2026-05-08 20:19:11][app.vjepa.train     ][main                     ] Disabling persistent_workers (incompatible with KNN eval)
+[INFO    ][2026-05-08 20:19:11][app.vjepa.train     ][main                     ] NCCL_SOCKET_IFNAME=eno
+[INFO    ][2026-05-08 20:19:15][app.vjepa.train     ][main                     ] Initialized (rank/world-size) 6/16, tasks_per_node=4
+[INFO    ][2026-05-08 20:19:15][root                ][stage_datasets           ] [local_rank 2/4] Staging kinetics_240 (targz_parts)
+[INFO    ][2026-05-08 20:19:15][root                ][_stage_targz_parts       ] [rank 2] Extracting 26/103 tar.gz parts to /scratch-node/dcanez.22595640/kinetics_240
+[INFO    ][2026-05-08 20:19:29][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 2/26 parts
+[INFO    ][2026-05-08 20:19:42][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 4/26 parts
+[INFO    ][2026-05-08 20:19:57][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 6/26 parts
+[INFO    ][2026-05-08 20:20:11][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 8/26 parts
+[INFO    ][2026-05-08 20:20:25][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 10/26 parts
+[INFO    ][2026-05-08 20:20:39][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 12/26 parts
+[INFO    ][2026-05-08 20:20:53][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 14/26 parts
+[INFO    ][2026-05-08 20:21:07][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 16/26 parts
+[INFO    ][2026-05-08 20:21:22][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 18/26 parts
+[INFO    ][2026-05-08 20:21:36][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 20/26 parts
+[INFO    ][2026-05-08 20:21:50][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 22/26 parts
+[INFO    ][2026-05-08 20:22:04][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 24/26 parts
+[INFO    ][2026-05-08 20:22:19][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 26/26 parts
+[INFO    ][2026-05-08 20:22:19][root                ][stage_datasets           ] [local_rank 2/4] Staging ssv2 (multipart_tar)
+[INFO    ][2026-05-08 20:23:17][app.vjepa.train     ][main                     ] Staged datasets: ['/scratch-node/dcanez.22595640/kinetics_240/train.csv', '/scratch-node/dcanez.22595640/ssv2/train.csv']
+[INFO    ][2026-05-08 20:24:16][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
+[INFO    ][2026-05-08 20:24:20][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA applied: 48 modules replaced (rank=8, alpha=16, dropout=0.0, targets=['attn.qkv', 'attn.proj'], scope=blocks)
+[INFO    ][2026-05-08 20:24:20][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA trainable params: 1,179,648
+[INFO    ][2026-05-08 20:24:26][root                ][_build_frozen_2d_target  ] Loaded frozen_2d target from timm: vit_large_patch14_capi.lvd1689m
+[INFO    ][2026-05-08 20:24:26][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-08 20:24:26][root                ][init_video_model         ] VJEPAPredictorMultiSeqWrapper(
+  (backbone): VisionTransformerPredictor(
+    (predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
+    (mask_tokens): ParameterList(
+        (0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+    )
+    (predictor_blocks): ModuleList(
+      (0-5): 6 x Block(
+        (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (attn): RoPEAttention(
+          (qkv): Linear(in_features=384, out_features=1152, bias=True)
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): Linear(in_features=384, out_features=384, bias=True)
+          (proj_drop): Dropout(p=0.0, inplace=False)
+        )
+        (drop_path): Identity()
+        (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (mlp): MLP(
+          (fc1): Linear(in_features=384, out_features=1536, bias=True)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=1536, out_features=384, bias=True)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+    (predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
+  )
+)
+[INFO    ][2026-05-08 20:24:26][root                ][init_video_model         ] MultiSeqWrapper(
+  (backbone): Frozen2DTargetWrapper(
+    (backbone): Eva(
+      (patch_embed): PatchEmbed(
+        (proj): Conv2d(3, 1024, kernel_size=(14, 14), stride=(14, 14))
+        (norm): Identity()
+      )
+      (pos_drop): Dropout(p=0.0, inplace=False)
+      (norm_pre): Identity()
+      (blocks): ModuleList(
+        (0-23): 24 x EvaBlock(
+          (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+          (attn): EvaAttention(
+            (qkv): Linear(in_features=1024, out_features=3072, bias=False)
+            (q_norm): Identity()
+            (k_norm): Identity()
+            (attn_drop): Dropout(p=0.0, inplace=False)
+            (norm): Identity()
+            (proj): Linear(in_features=1024, out_features=1024, bias=True)
+            (proj_drop): Dropout(p=0.0, inplace=False)
+          )
+          (drop_path1): Identity()
+          (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+          (mlp): Mlp(
+            (fc1): Linear(in_features=1024, out_features=4096, bias=True)
+            (act): GELU(approximate='none')
+            (drop1): Dropout(p=0.0, inplace=False)
+            (norm): Identity()
+            (fc2): Linear(in_features=4096, out_features=1024, bias=True)
+            (drop2): Dropout(p=0.0, inplace=False)
+          )
+          (drop_path2): Identity()
+        )
+      )
+      (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+      (fc_norm): Identity()
+      (head_drop): Dropout(p=0.0, inplace=False)
+      (head): Identity()
+      (rope): _CapiPatchRoPE()
+    )
+  )
+)
+[INFO    ][2026-05-08 20:24:26][root                ][init_video_model         ] Encoder number of parameters: 1179648
+[INFO    ][2026-05-08 20:24:26][root                ][init_video_model         ] Predictor number of parameters: 11436928
+[INFO    ][2026-05-08 20:24:26][root                ][init_video_model         ] Target encoder number of parameters: 0
+[INFO    ][2026-05-08 20:24:29][root                ][make_videodataset        ] VideoDataset dataset created
+[INFO    ][2026-05-08 20:24:29][WeightedSampler     ][__init__                 ] Using DistributedWeightedSampler with rank 6 / 16
+[INFO    ][2026-05-08 20:24:29][root                ][make_videodataset        ] VideoDataset unsupervised data loader created
+[INFO    ][2026-05-08 20:24:29][app.vjepa.train     ][main                     ] iterations per epoch/dataset length: 300/399
+[INFO    ][2026-05-08 20:24:29][app.vjepa.train     ][main                     ] Wrapping models in DDP (rank 6)...
+[INFO    ][2026-05-08 20:24:41][app.vjepa.train     ][main                     ] Initializing loader...
+submitit INFO (2026-05-09 08:16:14,364) - Job has timed out. Ran 718 minutes out of requested 720 minutes.
+[INFO    ][2026-05-09 08:16:14][submitit            ][has_timed_out            ] Job has timed out. Ran 718 minutes out of requested 720 minutes.
+submitit WARNING (2026-05-09 08:16:14,388) - Caught signal SIGUSR2 on gcn76.local.snellius.surf.nl: this job is timed-out.
+[WARNING ][2026-05-09 08:16:14][submitit            ][checkpoint_and_try_requeue] Caught signal SIGUSR2 on gcn76.local.snellius.surf.nl: this job is timed-out.
+submitit INFO (2026-05-09 08:16:14,389) - Not checkpointing nor requeuing since I am a slave (procid=6).
+[INFO    ][2026-05-09 08:16:14][submitit            ][checkpoint_and_try_requeue] Not checkpointing nor requeuing since I am a slave (procid=6).
+submitit WARNING (2026-05-09 08:16:14,389) - Bypassing signal SIGTERM
+[WARNING ][2026-05-09 08:16:14][submitit            ][bypass                   ] Bypassing signal SIGTERM
+submitit WARNING (2026-05-09 08:16:14,389) - Bypassing signal SIGCONT
+[WARNING ][2026-05-09 08:16:14][submitit            ][bypass                   ] Bypassing signal SIGCONT
+submitit INFO (2026-05-09 08:19:43,911) - Starting with JobEnvironment(job_id=22595640, hostname=gcn75.local.snellius.surf.nl, local_rank=2(4), node=1(4), global_rank=6(16))
+submitit INFO (2026-05-09 08:19:44,682) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/job_22595640/22595640_submitted.pkl
+INFO:root:loaded pretrain params...
+{   'app': 'vjepa',
+    'cpus_per_task': 16,
+    'data': {   'batch_size': 64,
+                'crop_size': 224,
+                'dataset_fpcs': [16, 16],
+                'dataset_type': 'VideoDataset',
+                'datasets': [   '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
+                                '/scratch-shared/dcanez/data/ssv2/train.csv'],
+                'datasets_weights': [0.65, 0.35],
+                'fps': 4,
+                'num_workers': 10,
+                'patch_size': 14,
+                'persistent_workers': True,
+                'pin_mem': True,
+                'stage': [   {   'dest': 'kinetics_240',
+                                 'format': 'targz_parts',
+                                 'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
+                             {   'dest': 'ssv2',
+                                 'format': 'multipart_tar',
+                                 'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
+                'tubelet_size': 1},
+    'data_aug': {   'auto_augment': False,
+                    'motion_shift': False,
+                    'random_resize_aspect_ratio': [0.75, 1.35],
+                    'random_resize_scale': [0.3, 1.0],
+                    'reprob': 0.0},
+    'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m',
+    'loss': {'loss_exp': 1.0},
+    'mask': [   {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 8,
+                    'spatial_scale': [0.15, 0.15],
+                    'temporal_scale': [1.0, 1.0]},
+                {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 2,
+                    'spatial_scale': [0.7, 0.7],
+                    'temporal_scale': [1.0, 1.0]}],
+    'mem_per_gpu': '180G',
+    'meta': {   'dtype': 'bfloat16',
+                'knn_eval_epoch0': False,
+                'knn_eval_freq': 5,
+                'knn_eval_presets': [   {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
+                                                          'eval_videos_per_class': 25,
+                                                          'num_workers': 8,
+                                                          'train_videos_per_class': 100},
+                                            'preset': 'ucf101'},
+                                        {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
+                                                          'eval_videos_per_class': 100,
+                                                          'linear_probe': True,
+                                                          'num_workers': 8,
+                                                          'pool_type': 'temporal_concat',
+                                                          'train_videos_per_class': 500},
+                                            'preset': 'ssv2_coarse10'}],
+                'load_checkpoint': True,
+                'read_checkpoint': None,
+                'save_every_freq': 5,
+                'seed': 239,
+                'use_sdpa': True,
+                'use_wandb': True,
+                'wandb_project': 'vjepa_ablation'},
+    'metrics': {'sigreg': {}, 'std': {}},
+    'model': {   'class_token': False,
+                 'freeze_backbone': True,
+                 'is_causal': False,
+                 'lora_alpha': 16,
+                 'lora_rank': 8,
+                 'lora_targets': ['attn.qkv', 'attn.proj'],
+                 'model_name': 'vit_large_patch14_capi_lvd1689m',
+                 'num_reg_tokens': 16,
+                 'pred_depth': 6,
+                 'pred_embed_dim': 384,
+                 'pred_num_heads': 12,
+                 'rope_type': 'capi2d',
+                 'stem_type': '2d',
+                 'target_kind': 'frozen_2d',
+                 'target_type': 'vit_large_patch14_capi.lvd1689m',
+                 'uniform_power': True,
+                 'use_activation_checkpointing': True,
+                 'use_lora': True,
+                 'use_mask_tokens': True,
+                 'use_rope': True,
+                 'use_sdpa': True,
+                 'zero_init_mask_tokens': True},
+    'nodes': 4,
+    'optimization': {   'ema': [0.99925, 0.99925],
+                        'epochs': 100,
+                        'final_lr': 0.0001,
+                        'final_weight_decay': 0.04,
+                        'ipe': 300,
+                        'ipe_scale': 1.0,
+                        'lr': 0.0005,
+                        'start_lr': 0.0001,
+                        'warmup': 10,
+                        'weight_decay': 0.04},
+    'tasks_per_node': 4}
+INFO:root:Running pre-training of app: vjepa
+[INFO    ][2026-05-09 08:19:53][app.vjepa.train     ][main                     ] which_dtype='bfloat16'
+[INFO    ][2026-05-09 08:19:53][app.vjepa.train     ][main                     ] Disabling persistent_workers (incompatible with KNN eval)
+[INFO    ][2026-05-09 08:19:53][app.vjepa.train     ][main                     ] NCCL_SOCKET_IFNAME=eno
+[INFO    ][2026-05-09 08:21:21][app.vjepa.train     ][main                     ] Initialized (rank/world-size) 6/16, tasks_per_node=4
+[INFO    ][2026-05-09 08:21:21][root                ][stage_datasets           ] [local_rank 2/4] Staging kinetics_240 (targz_parts)
+[INFO    ][2026-05-09 08:21:21][root                ][_stage_targz_parts       ] [rank 2] Extracting 26/103 tar.gz parts to /scratch-node/dcanez.22595640/kinetics_240
+[INFO    ][2026-05-09 08:21:35][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 2/26 parts
+[INFO    ][2026-05-09 08:21:48][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 4/26 parts
+[INFO    ][2026-05-09 08:22:03][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 6/26 parts
+[INFO    ][2026-05-09 08:22:18][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 8/26 parts
+[INFO    ][2026-05-09 08:22:32][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 10/26 parts
+[INFO    ][2026-05-09 08:22:46][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 12/26 parts
+[INFO    ][2026-05-09 08:23:00][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 14/26 parts
+[INFO    ][2026-05-09 08:23:15][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 16/26 parts
+[INFO    ][2026-05-09 08:23:29][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 18/26 parts
+[INFO    ][2026-05-09 08:23:43][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 20/26 parts
+[INFO    ][2026-05-09 08:23:58][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 22/26 parts
+[INFO    ][2026-05-09 08:24:12][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 24/26 parts
+[INFO    ][2026-05-09 08:24:27][root                ][_stage_targz_parts       ] [local_rank 2] Extracted 26/26 parts
+[INFO    ][2026-05-09 08:24:27][root                ][stage_datasets           ] [local_rank 2/4] Staging ssv2 (multipart_tar)
+[INFO    ][2026-05-09 08:25:33][app.vjepa.train     ][main                     ] Staged datasets: ['/scratch-node/dcanez.22595640/kinetics_240/train.csv', '/scratch-node/dcanez.22595640/ssv2/train.csv']
+[INFO    ][2026-05-09 08:25:58][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
+[INFO    ][2026-05-09 08:26:02][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA applied: 48 modules replaced (rank=8, alpha=16, dropout=0.0, targets=['attn.qkv', 'attn.proj'], scope=blocks)
+[INFO    ][2026-05-09 08:26:02][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA trainable params: 1,179,648
+[INFO    ][2026-05-09 08:26:05][root                ][_build_frozen_2d_target  ] Loaded frozen_2d target from timm: vit_large_patch14_capi.lvd1689m
+[INFO    ][2026-05-09 08:26:05][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-09 08:26:05][root                ][init_video_model         ] VJEPAPredictorMultiSeqWrapper(
+  (backbone): VisionTransformerPredictor(
+    (predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
+    (mask_tokens): ParameterList(
+        (0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+    )
+    (predictor_blocks): ModuleList(
+      (0-5): 6 x Block(
+        (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (attn): RoPEAttention(
+          (qkv): Linear(in_features=384, out_features=1152, bias=True)
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): Linear(in_features=384, out_features=384, bias=True)
+          (proj_drop): Dropout(p=0.0, inplace=False)
+        )
+        (drop_path): Identity()
+        (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (mlp): MLP(
+          (fc1): Linear(in_features=384, out_features=1536, bias=True)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=1536, out_features=384, bias=True)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+    (predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
+  )
+)
+[INFO    ][2026-05-09 08:26:05][root                ][init_video_model         ] MultiSeqWrapper(
+  (backbone): Frozen2DTargetWrapper(
+    (backbone): Eva(
+      (patch_embed): PatchEmbed(
+        (proj): Conv2d(3, 1024, kernel_size=(14, 14), stride=(14, 14))
+        (norm): Identity()
+      )
+      (pos_drop): Dropout(p=0.0, inplace=False)
+      (norm_pre): Identity()
+      (blocks): ModuleList(
+        (0-23): 24 x EvaBlock(
+          (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+          (attn): EvaAttention(
+            (qkv): Linear(in_features=1024, out_features=3072, bias=False)
+            (q_norm): Identity()
+            (k_norm): Identity()
+            (attn_drop): Dropout(p=0.0, inplace=False)
+            (norm): Identity()
+            (proj): Linear(in_features=1024, out_features=1024, bias=True)
+            (proj_drop): Dropout(p=0.0, inplace=False)
+          )
+          (drop_path1): Identity()
+          (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+          (mlp): Mlp(
+            (fc1): Linear(in_features=1024, out_features=4096, bias=True)
+            (act): GELU(approximate='none')
+            (drop1): Dropout(p=0.0, inplace=False)
+            (norm): Identity()
+            (fc2): Linear(in_features=4096, out_features=1024, bias=True)
+            (drop2): Dropout(p=0.0, inplace=False)
+          )
+          (drop_path2): Identity()
+        )
+      )
+      (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+      (fc_norm): Identity()
+      (head_drop): Dropout(p=0.0, inplace=False)
+      (head): Identity()
+      (rope): _CapiPatchRoPE()
+    )
+  )
+)
+[INFO    ][2026-05-09 08:26:05][root                ][init_video_model         ] Encoder number of parameters: 1179648
+[INFO    ][2026-05-09 08:26:05][root                ][init_video_model         ] Predictor number of parameters: 11436928
+[INFO    ][2026-05-09 08:26:05][root                ][init_video_model         ] Target encoder number of parameters: 0
+[INFO    ][2026-05-09 08:26:06][root                ][make_videodataset        ] VideoDataset dataset created
+[INFO    ][2026-05-09 08:26:06][WeightedSampler     ][__init__                 ] Using DistributedWeightedSampler with rank 6 / 16
+[INFO    ][2026-05-09 08:26:06][root                ][make_videodataset        ] VideoDataset unsupervised data loader created
+[INFO    ][2026-05-09 08:26:06][app.vjepa.train     ][main                     ] iterations per epoch/dataset length: 300/399
+[INFO    ][2026-05-09 08:26:06][app.vjepa.train     ][main                     ] Wrapping models in DDP (rank 6)...
+[INFO    ][2026-05-09 08:26:37][root                ][load_checkpoint          ] Loading checkpoint from /scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/latest.pt
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] loaded pretrained encoder from epoch 39 with msg: <All keys matched successfully>
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] loaded pretrained predictor from epoch 39 with msg: <All keys matched successfully>
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] loaded pretrained target encoder from epoch 39 with msg: <All keys matched successfully>
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] loaded optimizers from epoch 39
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] read-path: /scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/latest.pt
+[INFO    ][2026-05-09 08:26:39][app.vjepa.train     ][main                     ] Initializing loader...
+submitit WARNING (2026-05-09 09:56:00,971) - Bypassing signal SIGTERM
+[WARNING ][2026-05-09 09:56:00][submitit            ][bypass                   ] Bypassing signal SIGTERM
+submitit WARNING (2026-05-09 09:56:00,972) - Bypassing signal SIGCONT
+[WARNING ][2026-05-09 09:56:00][submitit            ][bypass                   ] Bypassing signal SIGCONT

vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/job_22595640/22595640_7_log.err

@@ -0,0 +1,305 @@
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
+  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
+[rank7]:[W508 20:22:12.287469402 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
+  scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+submitit WARNING (2026-05-09 08:16:14,423) - Caught signal SIGUSR2 on gcn76.local.snellius.surf.nl: this job is timed-out.
+submitit WARNING (2026-05-09 08:16:14,424) - Bypassing signal SIGTERM
+submitit WARNING (2026-05-09 08:16:14,424) - Bypassing signal SIGCONT
+[rank7]:[W509 08:16:18.714893930 TCPStore.cpp:125] [c10d] recvValue failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50456, remote=[gcn75.local.snellius.surf.nl]:37129): Failed to recv, got 0 bytes. Connection was likely closed. Did the remote server shutdown or crash?
+Exception raised from recvBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:682 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1548edb70fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a3325d (0x154931cff25d in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x273 (0x154931cfd1f3 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1548eeddfeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x154a2e249164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x154a3ecbb19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x154a3ed40100 in /lib64/libc.so.6)
+
+[rank7]:[W509 08:16:18.718026047 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 7] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Failed to recv, got 0 bytes. Connection was likely closed. Did the remote server shutdown or crash?
+[rank7]:[W509 08:16:19.718218153 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50456, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1548edb70fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x154931cfe6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x154931cfd1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1548eeddfeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x154a2e249164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x154a3ecbb19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x154a3ed40100 in /lib64/libc.so.6)
+
+[rank7]:[W509 08:16:19.721606189 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 7] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank7]:[W509 08:16:20.721747885 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50456, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1548edb70fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x154931cfe6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x154931cfd1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1548eeddfeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x154a2e249164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x154a3ecbb19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x154a3ed40100 in /lib64/libc.so.6)
+
+[rank7]:[W509 08:16:20.724071235 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 7] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank7]:[W509 08:16:21.724200881 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50456, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1548edb70fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x154931cfe6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x154931cfd1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1548eeddfeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x154a2e249164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x154a3ecbb19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x154a3ed40100 in /lib64/libc.so.6)
+
+[rank7]:[W509 08:16:21.726500141 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 7] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank7]:[W509 08:16:22.726638458 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50456, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1548edb70fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x154931cfe6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x154931cfd1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1548eeddfeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x154a2e249164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x154a3ecbb19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x154a3ed40100 in /lib64/libc.so.6)
+
+[rank7]:[W509 08:16:22.728988118 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 7] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank7]:[W509 08:16:23.729132614 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50456, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1548edb70fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x154931cfe6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x154931cfd1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1548eeddfeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x154a2e249164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x154a3ecbb19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x154a3ed40100 in /lib64/libc.so.6)
+
+[rank7]:[W509 08:16:23.731609913 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 7] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank7]:[W509 08:16:24.731742589 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50456, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1548edb70fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x154931cfe6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x154931cfd1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1548eeddfeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x154a2e249164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x154a3ecbb19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x154a3ed40100 in /lib64/libc.so.6)
+
+[rank7]:[W509 08:16:24.733968170 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 7] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank7]:[W509 08:16:25.734101836 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50456, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1548edb70fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x154931cfe6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x154931cfd1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1548eeddfeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x154a2e249164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x154a3ecbb19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x154a3ed40100 in /lib64/libc.so.6)
+
+[rank7]:[W509 08:16:25.736400076 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 7] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank7]:[W509 08:16:26.736536933 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50456, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1548edb70fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x154931cfe6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x154931cfd1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1548eeddfeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x154a2e249164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x154a3ecbb19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x154a3ed40100 in /lib64/libc.so.6)
+
+[rank7]:[W509 08:16:26.738951302 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 7] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank7]:[W509 08:16:27.739088769 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50456, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1548edb70fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x154931cfe6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x154931cfd1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1548eeddfeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x154a2e249164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x154a3ecbb19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x154a3ed40100 in /lib64/libc.so.6)
+
+[rank7]:[W509 08:16:27.741727867 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 7] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank7]:[W509 08:16:28.741866553 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50456, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1548edb70fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x154931cfe6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x154931cfd1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1548eeddfeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x154a2e249164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x154a3ecbb19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x154a3ed40100 in /lib64/libc.so.6)
+
+[rank7]:[W509 08:16:28.744120914 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 7] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank7]:[W509 08:16:29.744272440 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50456, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1548edb70fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x154931cfe6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x154931cfd1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1548eeddfeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x154a2e249164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x154a3ecbb19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x154a3ed40100 in /lib64/libc.so.6)
+
+[rank7]:[W509 08:16:29.746710739 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 7] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank7]:[W509 08:16:30.746840386 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50456, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1548edb70fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x154931cfe6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x154931cfd1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1548eeddfeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x154a2e249164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x154a3ecbb19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x154a3ed40100 in /lib64/libc.so.6)
+
+[rank7]:[W509 08:16:30.749093996 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 7] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank7]:[W509 08:16:31.749230593 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50456, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1548edb70fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x154931cfe6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x154931cfd1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1548eeddfeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x154a2e249164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x154a3ecbb19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x154a3ed40100 in /lib64/libc.so.6)
+
+[rank7]:[W509 08:16:31.751514193 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 7] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank7]:[W509 08:16:32.751651159 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50456, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1548edb70fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x154931cfe6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x154931cfd1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1548eeddfeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x154a2e249164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x154a3ecbb19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x154a3ed40100 in /lib64/libc.so.6)
+
+[rank7]:[W509 08:16:32.753963899 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 7] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank7]:[W509 08:16:33.754103135 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50456, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1548edb70fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x154931cfe6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x154931cfd1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1548eeddfeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x154a2e249164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x154a3ecbb19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x154a3ed40100 in /lib64/libc.so.6)
+
+[rank7]:[W509 08:16:33.756388125 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 7] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank7]:[W509 08:16:34.756516592 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50456, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1548edb70fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x154931cfe6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x154931cfd1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1548eeddfeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x154a2e249164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x154a3ecbb19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x154a3ed40100 in /lib64/libc.so.6)
+
+[rank7]:[W509 08:16:34.758743222 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 7] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank7]:[W509 08:16:35.758893318 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50456, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1548edb70fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x154931cfe6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x154931cfd1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1548eeddfeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x154a2e249164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x154a3ecbb19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x154a3ed40100 in /lib64/libc.so.6)
+
+[rank7]:[W509 08:16:35.761319438 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 7] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank7]:[W509 08:16:36.761449914 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50456, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1548edb70fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x154931cfe6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x154931cfd1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1548eeddfeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x154a2e249164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x154a3ecbb19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x154a3ed40100 in /lib64/libc.so.6)
+
+[rank7]:[W509 08:16:36.763693225 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 7] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank7]:[W509 08:16:37.763825701 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50456, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1548edb70fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x154931cfe6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x154931cfd1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1548eeddfeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x154a2e249164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x154a3ecbb19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x154a3ed40100 in /lib64/libc.so.6)
+
+[rank7]:[W509 08:16:37.766088841 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 7] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank7]:[W509 08:16:38.766219728 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50456, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1548edb70fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x154931cfe6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x154931cfd1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1548eeddfeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x154a2e249164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x154a3ecbb19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x154a3ed40100 in /lib64/libc.so.6)
+
+[rank7]:[W509 08:16:38.768699087 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 7] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank7]:[W509 08:16:39.768830133 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50456, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1548edb70fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x154931cfe6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x154931cfd1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1548eeddfeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x154a2e249164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x154a3ecbb19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x154a3ed40100 in /lib64/libc.so.6)
+
+[rank7]:[W509 08:16:39.771112043 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 7] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank7]:[W509 08:16:40.771238920 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50456, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1548edb70fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x154931cfe6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x154931cfd1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1548eeddfeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x154a2e249164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x154a3ecbb19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x154a3ed40100 in /lib64/libc.so.6)
+
+[rank7]:[W509 08:16:40.773511630 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 7] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank7]:[W509 08:16:41.773638966 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50456, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1548edb70fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x154931cfe6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x154931cfd1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1548eeddfeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x154a2e249164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x154a3ecbb19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x154a3ed40100 in /lib64/libc.so.6)
+
+[rank7]:[W509 08:16:41.775952366 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 7] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank7]:[W509 08:16:42.776079343 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50456, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1548edb70fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x154931cfe6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x154931cfd1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1548eeddfeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x154a2e249164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x154a3ecbb19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x154a3ed40100 in /lib64/libc.so.6)
+
+[rank7]:[W509 08:16:42.778340343 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 7] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank7]:[W509 08:16:43.778476759 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn76.local.snellius.surf.nl]:50456, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1548edb70fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x154931cfe6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x154931cfd1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x1548eeddfeec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x154a2e249164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x154a3ecbb19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x154a3ed40100 in /lib64/libc.so.6)
+
+[rank7]:[W509 08:16:43.780761910 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 7] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
+  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
+[rank7]:[W509 08:24:29.185901301 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
+  scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+submitit WARNING (2026-05-09 09:56:00,858) - Bypassing signal SIGTERM
+submitit WARNING (2026-05-09 09:56:00,858) - Bypassing signal SIGCONT

vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/job_22595640/22595640_7_log.out

@@ -0,0 +1,530 @@
+submitit INFO (2026-05-08 20:18:11,423) - Starting with JobEnvironment(job_id=22595640, hostname=gcn76.local.snellius.surf.nl, local_rank=3(4), node=1(4), global_rank=7(16))
+submitit INFO (2026-05-08 20:18:11,423) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/job_22595640/22595640_submitted.pkl
+INFO:root:loaded pretrain params...
+{   'app': 'vjepa',
+    'cpus_per_task': 16,
+    'data': {   'batch_size': 64,
+                'crop_size': 224,
+                'dataset_fpcs': [16, 16],
+                'dataset_type': 'VideoDataset',
+                'datasets': [   '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
+                                '/scratch-shared/dcanez/data/ssv2/train.csv'],
+                'datasets_weights': [0.65, 0.35],
+                'fps': 4,
+                'num_workers': 10,
+                'patch_size': 14,
+                'persistent_workers': True,
+                'pin_mem': True,
+                'stage': [   {   'dest': 'kinetics_240',
+                                 'format': 'targz_parts',
+                                 'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
+                             {   'dest': 'ssv2',
+                                 'format': 'multipart_tar',
+                                 'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
+                'tubelet_size': 1},
+    'data_aug': {   'auto_augment': False,
+                    'motion_shift': False,
+                    'random_resize_aspect_ratio': [0.75, 1.35],
+                    'random_resize_scale': [0.3, 1.0],
+                    'reprob': 0.0},
+    'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m',
+    'loss': {'loss_exp': 1.0},
+    'mask': [   {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 8,
+                    'spatial_scale': [0.15, 0.15],
+                    'temporal_scale': [1.0, 1.0]},
+                {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 2,
+                    'spatial_scale': [0.7, 0.7],
+                    'temporal_scale': [1.0, 1.0]}],
+    'mem_per_gpu': '180G',
+    'meta': {   'dtype': 'bfloat16',
+                'knn_eval_epoch0': False,
+                'knn_eval_freq': 5,
+                'knn_eval_presets': [   {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
+                                                          'eval_videos_per_class': 25,
+                                                          'num_workers': 8,
+                                                          'train_videos_per_class': 100},
+                                            'preset': 'ucf101'},
+                                        {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
+                                                          'eval_videos_per_class': 100,
+                                                          'linear_probe': True,
+                                                          'num_workers': 8,
+                                                          'pool_type': 'temporal_concat',
+                                                          'train_videos_per_class': 500},
+                                            'preset': 'ssv2_coarse10'}],
+                'load_checkpoint': True,
+                'read_checkpoint': None,
+                'save_every_freq': 5,
+                'seed': 239,
+                'use_sdpa': True,
+                'use_wandb': True,
+                'wandb_project': 'vjepa_ablation'},
+    'metrics': {'sigreg': {}, 'std': {}},
+    'model': {   'class_token': False,
+                 'freeze_backbone': True,
+                 'is_causal': False,
+                 'lora_alpha': 16,
+                 'lora_rank': 8,
+                 'lora_targets': ['attn.qkv', 'attn.proj'],
+                 'model_name': 'vit_large_patch14_capi_lvd1689m',
+                 'num_reg_tokens': 16,
+                 'pred_depth': 6,
+                 'pred_embed_dim': 384,
+                 'pred_num_heads': 12,
+                 'rope_type': 'capi2d',
+                 'stem_type': '2d',
+                 'target_kind': 'frozen_2d',
+                 'target_type': 'vit_large_patch14_capi.lvd1689m',
+                 'uniform_power': True,
+                 'use_activation_checkpointing': True,
+                 'use_lora': True,
+                 'use_mask_tokens': True,
+                 'use_rope': True,
+                 'use_sdpa': True,
+                 'zero_init_mask_tokens': True},
+    'nodes': 4,
+    'optimization': {   'ema': [0.99925, 0.99925],
+                        'epochs': 100,
+                        'final_lr': 0.0001,
+                        'final_weight_decay': 0.04,
+                        'ipe': 300,
+                        'ipe_scale': 1.0,
+                        'lr': 0.0005,
+                        'start_lr': 0.0001,
+                        'warmup': 10,
+                        'weight_decay': 0.04},
+    'tasks_per_node': 4}
+INFO:root:Running pre-training of app: vjepa
+[INFO    ][2026-05-08 20:19:11][app.vjepa.train     ][main                     ] which_dtype='bfloat16'
+[INFO    ][2026-05-08 20:19:11][app.vjepa.train     ][main                     ] Disabling persistent_workers (incompatible with KNN eval)
+[INFO    ][2026-05-08 20:19:11][app.vjepa.train     ][main                     ] NCCL_SOCKET_IFNAME=eno
+[INFO    ][2026-05-08 20:19:14][app.vjepa.train     ][main                     ] Initialized (rank/world-size) 7/16, tasks_per_node=4
+[INFO    ][2026-05-08 20:19:14][root                ][stage_datasets           ] [local_rank 3/4] Staging kinetics_240 (targz_parts)
+[INFO    ][2026-05-08 20:19:14][root                ][_stage_targz_parts       ] [rank 3] Extracting 25/103 tar.gz parts to /scratch-node/dcanez.22595640/kinetics_240
+[INFO    ][2026-05-08 20:19:28][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 2/25 parts
+[INFO    ][2026-05-08 20:19:43][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 4/25 parts
+[INFO    ][2026-05-08 20:19:57][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 6/25 parts
+[INFO    ][2026-05-08 20:20:11][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 8/25 parts
+[INFO    ][2026-05-08 20:20:25][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 10/25 parts
+[INFO    ][2026-05-08 20:20:40][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 12/25 parts
+[INFO    ][2026-05-08 20:20:54][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 14/25 parts
+[INFO    ][2026-05-08 20:21:08][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 16/25 parts
+[INFO    ][2026-05-08 20:21:22][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 18/25 parts
+[INFO    ][2026-05-08 20:21:37][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 20/25 parts
+[INFO    ][2026-05-08 20:21:51][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 22/25 parts
+[INFO    ][2026-05-08 20:22:05][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 24/25 parts
+[INFO    ][2026-05-08 20:22:12][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 25/25 parts
+[INFO    ][2026-05-08 20:22:12][root                ][stage_datasets           ] [local_rank 3/4] Staging ssv2 (multipart_tar)
+[INFO    ][2026-05-08 20:23:17][app.vjepa.train     ][main                     ] Staged datasets: ['/scratch-node/dcanez.22595640/kinetics_240/train.csv', '/scratch-node/dcanez.22595640/ssv2/train.csv']
+[INFO    ][2026-05-08 20:24:16][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
+[INFO    ][2026-05-08 20:24:20][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA applied: 48 modules replaced (rank=8, alpha=16, dropout=0.0, targets=['attn.qkv', 'attn.proj'], scope=blocks)
+[INFO    ][2026-05-08 20:24:20][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA trainable params: 1,179,648
+[INFO    ][2026-05-08 20:24:26][root                ][_build_frozen_2d_target  ] Loaded frozen_2d target from timm: vit_large_patch14_capi.lvd1689m
+[INFO    ][2026-05-08 20:24:26][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-08 20:24:26][root                ][init_video_model         ] VJEPAPredictorMultiSeqWrapper(
+  (backbone): VisionTransformerPredictor(
+    (predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
+    (mask_tokens): ParameterList(
+        (0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+    )
+    (predictor_blocks): ModuleList(
+      (0-5): 6 x Block(
+        (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (attn): RoPEAttention(
+          (qkv): Linear(in_features=384, out_features=1152, bias=True)
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): Linear(in_features=384, out_features=384, bias=True)
+          (proj_drop): Dropout(p=0.0, inplace=False)
+        )
+        (drop_path): Identity()
+        (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (mlp): MLP(
+          (fc1): Linear(in_features=384, out_features=1536, bias=True)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=1536, out_features=384, bias=True)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+    (predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
+  )
+)
+[INFO    ][2026-05-08 20:24:26][root                ][init_video_model         ] MultiSeqWrapper(
+  (backbone): Frozen2DTargetWrapper(
+    (backbone): Eva(
+      (patch_embed): PatchEmbed(
+        (proj): Conv2d(3, 1024, kernel_size=(14, 14), stride=(14, 14))
+        (norm): Identity()
+      )
+      (pos_drop): Dropout(p=0.0, inplace=False)
+      (norm_pre): Identity()
+      (blocks): ModuleList(
+        (0-23): 24 x EvaBlock(
+          (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+          (attn): EvaAttention(
+            (qkv): Linear(in_features=1024, out_features=3072, bias=False)
+            (q_norm): Identity()
+            (k_norm): Identity()
+            (attn_drop): Dropout(p=0.0, inplace=False)
+            (norm): Identity()
+            (proj): Linear(in_features=1024, out_features=1024, bias=True)
+            (proj_drop): Dropout(p=0.0, inplace=False)
+          )
+          (drop_path1): Identity()
+          (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+          (mlp): Mlp(
+            (fc1): Linear(in_features=1024, out_features=4096, bias=True)
+            (act): GELU(approximate='none')
+            (drop1): Dropout(p=0.0, inplace=False)
+            (norm): Identity()
+            (fc2): Linear(in_features=4096, out_features=1024, bias=True)
+            (drop2): Dropout(p=0.0, inplace=False)
+          )
+          (drop_path2): Identity()
+        )
+      )
+      (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+      (fc_norm): Identity()
+      (head_drop): Dropout(p=0.0, inplace=False)
+      (head): Identity()
+      (rope): _CapiPatchRoPE()
+    )
+  )
+)
+[INFO    ][2026-05-08 20:24:26][root                ][init_video_model         ] Encoder number of parameters: 1179648
+[INFO    ][2026-05-08 20:24:26][root                ][init_video_model         ] Predictor number of parameters: 11436928
+[INFO    ][2026-05-08 20:24:26][root                ][init_video_model         ] Target encoder number of parameters: 0
+[INFO    ][2026-05-08 20:24:29][root                ][make_videodataset        ] VideoDataset dataset created
+[INFO    ][2026-05-08 20:24:29][WeightedSampler     ][__init__                 ] Using DistributedWeightedSampler with rank 7 / 16
+[INFO    ][2026-05-08 20:24:29][root                ][make_videodataset        ] VideoDataset unsupervised data loader created
+[INFO    ][2026-05-08 20:24:29][app.vjepa.train     ][main                     ] iterations per epoch/dataset length: 300/399
+[INFO    ][2026-05-08 20:24:29][app.vjepa.train     ][main                     ] Wrapping models in DDP (rank 7)...
+[INFO    ][2026-05-08 20:24:41][app.vjepa.train     ][main                     ] Initializing loader...
+submitit INFO (2026-05-09 08:16:14,357) - Job has timed out. Ran 718 minutes out of requested 720 minutes.
+[INFO    ][2026-05-09 08:16:14][submitit            ][has_timed_out            ] Job has timed out. Ran 718 minutes out of requested 720 minutes.
+submitit WARNING (2026-05-09 08:16:14,423) - Caught signal SIGUSR2 on gcn76.local.snellius.surf.nl: this job is timed-out.
+[WARNING ][2026-05-09 08:16:14][submitit            ][checkpoint_and_try_requeue] Caught signal SIGUSR2 on gcn76.local.snellius.surf.nl: this job is timed-out.
+submitit INFO (2026-05-09 08:16:14,424) - Not checkpointing nor requeuing since I am a slave (procid=7).
+[INFO    ][2026-05-09 08:16:14][submitit            ][checkpoint_and_try_requeue] Not checkpointing nor requeuing since I am a slave (procid=7).
+submitit WARNING (2026-05-09 08:16:14,424) - Bypassing signal SIGTERM
+[WARNING ][2026-05-09 08:16:14][submitit            ][bypass                   ] Bypassing signal SIGTERM
+submitit WARNING (2026-05-09 08:16:14,424) - Bypassing signal SIGCONT
+[WARNING ][2026-05-09 08:16:14][submitit            ][bypass                   ] Bypassing signal SIGCONT
+submitit INFO (2026-05-09 08:19:43,911) - Starting with JobEnvironment(job_id=22595640, hostname=gcn75.local.snellius.surf.nl, local_rank=3(4), node=1(4), global_rank=7(16))
+submitit INFO (2026-05-09 08:19:44,436) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/job_22595640/22595640_submitted.pkl
+INFO:root:loaded pretrain params...
+{   'app': 'vjepa',
+    'cpus_per_task': 16,
+    'data': {   'batch_size': 64,
+                'crop_size': 224,
+                'dataset_fpcs': [16, 16],
+                'dataset_type': 'VideoDataset',
+                'datasets': [   '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
+                                '/scratch-shared/dcanez/data/ssv2/train.csv'],
+                'datasets_weights': [0.65, 0.35],
+                'fps': 4,
+                'num_workers': 10,
+                'patch_size': 14,
+                'persistent_workers': True,
+                'pin_mem': True,
+                'stage': [   {   'dest': 'kinetics_240',
+                                 'format': 'targz_parts',
+                                 'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
+                             {   'dest': 'ssv2',
+                                 'format': 'multipart_tar',
+                                 'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
+                'tubelet_size': 1},
+    'data_aug': {   'auto_augment': False,
+                    'motion_shift': False,
+                    'random_resize_aspect_ratio': [0.75, 1.35],
+                    'random_resize_scale': [0.3, 1.0],
+                    'reprob': 0.0},
+    'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m',
+    'loss': {'loss_exp': 1.0},
+    'mask': [   {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 8,
+                    'spatial_scale': [0.15, 0.15],
+                    'temporal_scale': [1.0, 1.0]},
+                {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 2,
+                    'spatial_scale': [0.7, 0.7],
+                    'temporal_scale': [1.0, 1.0]}],
+    'mem_per_gpu': '180G',
+    'meta': {   'dtype': 'bfloat16',
+                'knn_eval_epoch0': False,
+                'knn_eval_freq': 5,
+                'knn_eval_presets': [   {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
+                                                          'eval_videos_per_class': 25,
+                                                          'num_workers': 8,
+                                                          'train_videos_per_class': 100},
+                                            'preset': 'ucf101'},
+                                        {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
+                                                          'eval_videos_per_class': 100,
+                                                          'linear_probe': True,
+                                                          'num_workers': 8,
+                                                          'pool_type': 'temporal_concat',
+                                                          'train_videos_per_class': 500},
+                                            'preset': 'ssv2_coarse10'}],
+                'load_checkpoint': True,
+                'read_checkpoint': None,
+                'save_every_freq': 5,
+                'seed': 239,
+                'use_sdpa': True,
+                'use_wandb': True,
+                'wandb_project': 'vjepa_ablation'},
+    'metrics': {'sigreg': {}, 'std': {}},
+    'model': {   'class_token': False,
+                 'freeze_backbone': True,
+                 'is_causal': False,
+                 'lora_alpha': 16,
+                 'lora_rank': 8,
+                 'lora_targets': ['attn.qkv', 'attn.proj'],
+                 'model_name': 'vit_large_patch14_capi_lvd1689m',
+                 'num_reg_tokens': 16,
+                 'pred_depth': 6,
+                 'pred_embed_dim': 384,
+                 'pred_num_heads': 12,
+                 'rope_type': 'capi2d',
+                 'stem_type': '2d',
+                 'target_kind': 'frozen_2d',
+                 'target_type': 'vit_large_patch14_capi.lvd1689m',
+                 'uniform_power': True,
+                 'use_activation_checkpointing': True,
+                 'use_lora': True,
+                 'use_mask_tokens': True,
+                 'use_rope': True,
+                 'use_sdpa': True,
+                 'zero_init_mask_tokens': True},
+    'nodes': 4,
+    'optimization': {   'ema': [0.99925, 0.99925],
+                        'epochs': 100,
+                        'final_lr': 0.0001,
+                        'final_weight_decay': 0.04,
+                        'ipe': 300,
+                        'ipe_scale': 1.0,
+                        'lr': 0.0005,
+                        'start_lr': 0.0001,
+                        'warmup': 10,
+                        'weight_decay': 0.04},
+    'tasks_per_node': 4}
+INFO:root:Running pre-training of app: vjepa
+[INFO    ][2026-05-09 08:19:53][app.vjepa.train     ][main                     ] which_dtype='bfloat16'
+[INFO    ][2026-05-09 08:19:53][app.vjepa.train     ][main                     ] Disabling persistent_workers (incompatible with KNN eval)
+[INFO    ][2026-05-09 08:19:53][app.vjepa.train     ][main                     ] NCCL_SOCKET_IFNAME=eno
+[INFO    ][2026-05-09 08:21:30][app.vjepa.train     ][main                     ] Initialized (rank/world-size) 7/16, tasks_per_node=4
+[INFO    ][2026-05-09 08:21:30][root                ][stage_datasets           ] [local_rank 3/4] Staging kinetics_240 (targz_parts)
+[INFO    ][2026-05-09 08:21:30][root                ][_stage_targz_parts       ] [rank 3] Extracting 25/103 tar.gz parts to /scratch-node/dcanez.22595640/kinetics_240
+[INFO    ][2026-05-09 08:21:45][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 2/25 parts
+[INFO    ][2026-05-09 08:21:59][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 4/25 parts
+[INFO    ][2026-05-09 08:22:13][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 6/25 parts
+[INFO    ][2026-05-09 08:22:28][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 8/25 parts
+[INFO    ][2026-05-09 08:22:42][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 10/25 parts
+[INFO    ][2026-05-09 08:22:56][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 12/25 parts
+[INFO    ][2026-05-09 08:23:10][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 14/25 parts
+[INFO    ][2026-05-09 08:23:25][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 16/25 parts
+[INFO    ][2026-05-09 08:23:39][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 18/25 parts
+[INFO    ][2026-05-09 08:23:53][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 20/25 parts
+[INFO    ][2026-05-09 08:24:08][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 22/25 parts
+[INFO    ][2026-05-09 08:24:22][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 24/25 parts
+[INFO    ][2026-05-09 08:24:29][root                ][_stage_targz_parts       ] [local_rank 3] Extracted 25/25 parts
+[INFO    ][2026-05-09 08:24:29][root                ][stage_datasets           ] [local_rank 3/4] Staging ssv2 (multipart_tar)
+[INFO    ][2026-05-09 08:25:33][app.vjepa.train     ][main                     ] Staged datasets: ['/scratch-node/dcanez.22595640/kinetics_240/train.csv', '/scratch-node/dcanez.22595640/ssv2/train.csv']
+[INFO    ][2026-05-09 08:25:57][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
+[INFO    ][2026-05-09 08:26:02][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA applied: 48 modules replaced (rank=8, alpha=16, dropout=0.0, targets=['attn.qkv', 'attn.proj'], scope=blocks)
+[INFO    ][2026-05-09 08:26:02][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA trainable params: 1,179,648
+[INFO    ][2026-05-09 08:26:05][root                ][_build_frozen_2d_target  ] Loaded frozen_2d target from timm: vit_large_patch14_capi.lvd1689m
+[INFO    ][2026-05-09 08:26:05][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-09 08:26:05][root                ][init_video_model         ] VJEPAPredictorMultiSeqWrapper(
+  (backbone): VisionTransformerPredictor(
+    (predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
+    (mask_tokens): ParameterList(
+        (0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+    )
+    (predictor_blocks): ModuleList(
+      (0-5): 6 x Block(
+        (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (attn): RoPEAttention(
+          (qkv): Linear(in_features=384, out_features=1152, bias=True)
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): Linear(in_features=384, out_features=384, bias=True)
+          (proj_drop): Dropout(p=0.0, inplace=False)
+        )
+        (drop_path): Identity()
+        (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (mlp): MLP(
+          (fc1): Linear(in_features=384, out_features=1536, bias=True)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=1536, out_features=384, bias=True)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+    (predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
+  )
+)
+[INFO    ][2026-05-09 08:26:05][root                ][init_video_model         ] MultiSeqWrapper(
+  (backbone): Frozen2DTargetWrapper(
+    (backbone): Eva(
+      (patch_embed): PatchEmbed(
+        (proj): Conv2d(3, 1024, kernel_size=(14, 14), stride=(14, 14))
+        (norm): Identity()
+      )
+      (pos_drop): Dropout(p=0.0, inplace=False)
+      (norm_pre): Identity()
+      (blocks): ModuleList(
+        (0-23): 24 x EvaBlock(
+          (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+          (attn): EvaAttention(
+            (qkv): Linear(in_features=1024, out_features=3072, bias=False)
+            (q_norm): Identity()
+            (k_norm): Identity()
+            (attn_drop): Dropout(p=0.0, inplace=False)
+            (norm): Identity()
+            (proj): Linear(in_features=1024, out_features=1024, bias=True)
+            (proj_drop): Dropout(p=0.0, inplace=False)
+          )
+          (drop_path1): Identity()
+          (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+          (mlp): Mlp(
+            (fc1): Linear(in_features=1024, out_features=4096, bias=True)
+            (act): GELU(approximate='none')
+            (drop1): Dropout(p=0.0, inplace=False)
+            (norm): Identity()
+            (fc2): Linear(in_features=4096, out_features=1024, bias=True)
+            (drop2): Dropout(p=0.0, inplace=False)
+          )
+          (drop_path2): Identity()
+        )
+      )
+      (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+      (fc_norm): Identity()
+      (head_drop): Dropout(p=0.0, inplace=False)
+      (head): Identity()
+      (rope): _CapiPatchRoPE()
+    )
+  )
+)
+[INFO    ][2026-05-09 08:26:05][root                ][init_video_model         ] Encoder number of parameters: 1179648
+[INFO    ][2026-05-09 08:26:05][root                ][init_video_model         ] Predictor number of parameters: 11436928
+[INFO    ][2026-05-09 08:26:05][root                ][init_video_model         ] Target encoder number of parameters: 0
+[INFO    ][2026-05-09 08:26:06][root                ][make_videodataset        ] VideoDataset dataset created
+[INFO    ][2026-05-09 08:26:06][WeightedSampler     ][__init__                 ] Using DistributedWeightedSampler with rank 7 / 16
+[INFO    ][2026-05-09 08:26:06][root                ][make_videodataset        ] VideoDataset unsupervised data loader created
+[INFO    ][2026-05-09 08:26:06][app.vjepa.train     ][main                     ] iterations per epoch/dataset length: 300/399
+[INFO    ][2026-05-09 08:26:06][app.vjepa.train     ][main                     ] Wrapping models in DDP (rank 7)...
+[INFO    ][2026-05-09 08:26:37][root                ][load_checkpoint          ] Loading checkpoint from /scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/latest.pt
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] loaded pretrained encoder from epoch 39 with msg: <All keys matched successfully>
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] loaded pretrained predictor from epoch 39 with msg: <All keys matched successfully>
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] loaded pretrained target encoder from epoch 39 with msg: <All keys matched successfully>
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] loaded optimizers from epoch 39
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] read-path: /scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/latest.pt
+[INFO    ][2026-05-09 08:26:39][app.vjepa.train     ][main                     ] Initializing loader...
+submitit WARNING (2026-05-09 09:56:00,858) - Bypassing signal SIGTERM
+[WARNING ][2026-05-09 09:56:00][submitit            ][bypass                   ] Bypassing signal SIGTERM
+submitit WARNING (2026-05-09 09:56:00,858) - Bypassing signal SIGCONT
+[WARNING ][2026-05-09 09:56:00][submitit            ][bypass                   ] Bypassing signal SIGCONT

vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/job_22595640/22595640_8_log.err

@@ -0,0 +1,324 @@
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
+  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
+[rank8]:[W508 20:23:12.349823475 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
+  scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+submitit WARNING (2026-05-09 08:16:14,380) - Caught signal SIGUSR2 on gcn81.local.snellius.surf.nl: this job is timed-out.
+submitit WARNING (2026-05-09 08:16:14,380) - Bypassing signal SIGTERM
+submitit WARNING (2026-05-09 08:16:14,380) - Bypassing signal SIGCONT
+[rank8]:[W509 08:16:18.184925842 TCPStore.cpp:125] [c10d] recvValue failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58898, remote=[gcn75.local.snellius.surf.nl]:37129): Failed to recv, got 0 bytes. Connection was likely closed. Did the remote server shutdown or crash?
+Exception raised from recvBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:682 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14ddbe209fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a3325d (0x14de0239825d in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x273 (0x14de023961f3 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14ddbf478eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14defe8e2164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14df0f35419a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14df0f3d9100 in /lib64/libc.so.6)
+
+[rank8]:[W509 08:16:18.187182751 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 8] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Failed to recv, got 0 bytes. Connection was likely closed. Did the remote server shutdown or crash?
+[rank8]:[W509 08:16:19.187349367 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58898, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14ddbe209fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14de023976d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14de023961cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14ddbf478eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14defe8e2164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14df0f35419a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14df0f3d9100 in /lib64/libc.so.6)
+
+[rank8]:[W509 08:16:19.190280403 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 8] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank8]:[W509 08:16:20.190426730 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58898, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14ddbe209fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14de023976d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14de023961cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14ddbf478eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14defe8e2164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14df0f35419a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14df0f3d9100 in /lib64/libc.so.6)
+
+[rank8]:[W509 08:16:20.192698499 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 8] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank8]:[W509 08:16:21.192843265 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58898, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14ddbe209fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14de023976d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14de023961cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14ddbf478eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14defe8e2164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14df0f35419a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14df0f3d9100 in /lib64/libc.so.6)
+
+[rank8]:[W509 08:16:21.195117994 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 8] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank8]:[W509 08:16:22.195270810 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58898, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14ddbe209fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14de023976d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14de023961cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14ddbf478eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14defe8e2164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14df0f35419a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14df0f3d9100 in /lib64/libc.so.6)
+
+[rank8]:[W509 08:16:22.197550789 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 8] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank8]:[W509 08:16:23.197682756 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58898, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14ddbe209fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14de023976d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14de023961cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14ddbf478eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14defe8e2164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14df0f35419a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14df0f3d9100 in /lib64/libc.so.6)
+
+[rank8]:[W509 08:16:23.199967455 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 8] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank8]:[W509 08:16:24.200106531 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58898, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14ddbe209fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14de023976d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14de023961cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14ddbf478eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14defe8e2164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14df0f35419a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14df0f3d9100 in /lib64/libc.so.6)
+
+[rank8]:[W509 08:16:24.202258181 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 8] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank8]:[W509 08:16:25.202389757 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58898, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14ddbe209fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14de023976d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14de023961cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14ddbf478eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14defe8e2164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14df0f35419a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14df0f3d9100 in /lib64/libc.so.6)
+
+[rank8]:[W509 08:16:25.204644166 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 8] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank8]:[W509 08:16:26.204772493 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58898, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14ddbe209fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14de023976d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14de023961cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14ddbf478eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14defe8e2164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14df0f35419a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14df0f3d9100 in /lib64/libc.so.6)
+
+[rank8]:[W509 08:16:26.207052372 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 8] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank8]:[W509 08:16:27.207209408 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58898, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14ddbe209fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14de023976d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14de023961cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14ddbf478eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14defe8e2164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14df0f35419a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14df0f3d9100 in /lib64/libc.so.6)
+
+[rank8]:[W509 08:16:27.209472037 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 8] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank8]:[W509 08:16:28.209599554 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58898, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14ddbe209fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14de023976d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14de023961cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14ddbf478eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14defe8e2164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14df0f35419a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14df0f3d9100 in /lib64/libc.so.6)
+
+[rank8]:[W509 08:16:28.211877993 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 8] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank8]:[W509 08:16:29.212024499 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58898, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14ddbe209fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14de023976d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14de023961cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14ddbf478eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14defe8e2164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14df0f35419a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14df0f3d9100 in /lib64/libc.so.6)
+
+[rank8]:[W509 08:16:29.214263378 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 8] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank8]:[W509 08:16:30.214403975 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58898, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14ddbe209fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14de023976d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14de023961cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14ddbf478eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14defe8e2164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14df0f35419a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14df0f3d9100 in /lib64/libc.so.6)
+
+[rank8]:[W509 08:16:30.216674184 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 8] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank8]:[W509 08:16:31.216824470 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58898, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14ddbe209fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14de023976d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14de023961cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14ddbf478eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14defe8e2164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14df0f35419a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14df0f3d9100 in /lib64/libc.so.6)
+
+[rank8]:[W509 08:16:31.219095249 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 8] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank8]:[W509 08:16:32.219245595 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58898, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14ddbe209fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14de023976d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14de023961cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14ddbf478eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14defe8e2164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14df0f35419a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14df0f3d9100 in /lib64/libc.so.6)
+
+[rank8]:[W509 08:16:32.221522094 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 8] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank8]:[W509 08:16:33.221663671 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58898, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14ddbe209fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14de023976d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14de023961cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14ddbf478eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14defe8e2164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14df0f35419a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14df0f3d9100 in /lib64/libc.so.6)
+
+[rank8]:[W509 08:16:33.223918210 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 8] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank8]:[W509 08:16:34.224078956 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58898, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14ddbe209fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14de023976d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14de023961cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14ddbf478eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14defe8e2164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14df0f35419a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14df0f3d9100 in /lib64/libc.so.6)
+
+[rank8]:[W509 08:16:34.226313105 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 8] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank8]:[W509 08:16:35.226460632 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58898, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14ddbe209fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14de023976d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14de023961cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14ddbf478eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14defe8e2164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14df0f35419a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14df0f3d9100 in /lib64/libc.so.6)
+
+[rank8]:[W509 08:16:35.228650581 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 8] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank8]:[W509 08:16:36.228798817 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58898, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14ddbe209fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14de023976d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14de023961cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14ddbf478eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14defe8e2164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14df0f35419a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14df0f3d9100 in /lib64/libc.so.6)
+
+[rank8]:[W509 08:16:36.231063467 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 8] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank8]:[W509 08:16:37.231217393 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58898, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14ddbe209fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14de023976d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14de023961cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14ddbf478eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14defe8e2164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14df0f35419a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14df0f3d9100 in /lib64/libc.so.6)
+
+[rank8]:[W509 08:16:37.233403662 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 8] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank8]:[W509 08:16:38.233541089 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58898, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14ddbe209fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14de023976d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14de023961cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14ddbf478eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14defe8e2164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14df0f35419a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14df0f3d9100 in /lib64/libc.so.6)
+
+[rank8]:[W509 08:16:38.235837608 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 8] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank8]:[W509 08:16:39.235997194 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58898, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14ddbe209fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14de023976d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14de023961cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14ddbf478eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14defe8e2164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14df0f35419a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14df0f3d9100 in /lib64/libc.so.6)
+
+[rank8]:[W509 08:16:39.238255633 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 8] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank8]:[W509 08:16:40.238400139 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58898, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14ddbe209fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14de023976d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14de023961cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14ddbf478eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14defe8e2164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14df0f35419a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14df0f3d9100 in /lib64/libc.so.6)
+
+[rank8]:[W509 08:16:40.240705498 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 8] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank8]:[W509 08:16:41.240846375 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58898, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14ddbe209fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14de023976d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14de023961cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14ddbf478eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14defe8e2164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14df0f35419a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14df0f3d9100 in /lib64/libc.so.6)
+
+[rank8]:[W509 08:16:41.243103394 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 8] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank8]:[W509 08:16:42.243244300 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58898, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14ddbe209fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14de023976d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14de023961cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14ddbf478eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14defe8e2164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14df0f35419a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14df0f3d9100 in /lib64/libc.so.6)
+
+[rank8]:[W509 08:16:42.245511679 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 8] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank8]:[W509 08:16:43.245649026 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58898, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x14ddbe209fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14de023976d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14de023961cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x14ddbf478eec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x14defe8e2164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x14df0f35419a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x14df0f3d9100 in /lib64/libc.so.6)
+
+[rank8]:[W509 08:16:43.247810725 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 8] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[2026-05-09T08:16:46.745] error: namespace_p_join: open failed for /slurm/22595640/.ns: No such file or directory
+[2026-05-09T08:16:46.745] error: namespace_g_join(JobId=22595640 SLUID=s8FXKR4CZ0G200): No such file or directory
+[2026-05-09T08:16:48.391] error: Failed to send MESSAGE_TASK_EXIT: Connection refused
+[2026-05-09T08:16:49.210] error: namespace_p_join: open failed for /slurm/22595640/.ns: No such file or directory
+[2026-05-09T08:16:49.210] error: namespace_g_join(JobId=22595640 SLUID=s8FXKR4CZ0G200): No such file or directory
+[2026-05-09T08:16:50.848] error: Failed to send MESSAGE_TASK_EXIT: Connection refused
+[2026-05-09T08:16:50.851] error: namespace_p_join: open failed for /slurm/22595640/.ns: No such file or directory
+[2026-05-09T08:16:50.851] error: namespace_g_join(JobId=22595640 SLUID=s8FXKR4CZ0G200): No such file or directory
+[2026-05-09T08:16:50.991] error: namespace_p_join: open failed for /slurm/22595640/.ns: No such file or directory
+[2026-05-09T08:16:50.991] error: namespace_g_join(JobId=22595640 SLUID=s8FXKR4CZ0G200): No such file or directory
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
+  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
+[rank8]:[W509 08:24:50.393852755 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
+  scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+submitit WARNING (2026-05-09 09:56:00,842) - Bypassing signal SIGTERM
+submitit WARNING (2026-05-09 09:56:00,843) - Bypassing signal SIGCONT
+[2026-05-09T09:56:33.681] error: namespace_p_join: open failed for /slurm/22595640/.ns: No such file or directory
+[2026-05-09T09:56:33.681] error: namespace_g_join(JobId=22595640 SLUID=s8FXKR5P56S300): No such file or directory
+[2026-05-09T09:56:35.337] error: Failed to send MESSAGE_TASK_EXIT: Connection refused
+[2026-05-09T09:56:36.097] error: namespace_p_join: open failed for /slurm/22595640/.ns: No such file or directory
+[2026-05-09T09:56:36.097] error: namespace_g_join(JobId=22595640 SLUID=s8FXKR5P56S300): No such file or directory
+[2026-05-09T09:56:36.236] error: namespace_p_join: open failed for /slurm/22595640/.ns: No such file or directory
+[2026-05-09T09:56:36.236] error: namespace_g_join(JobId=22595640 SLUID=s8FXKR5P56S300): No such file or directory
+[2026-05-09T09:56:36.274] error: namespace_p_join: open failed for /slurm/22595640/.ns: No such file or directory
+[2026-05-09T09:56:36.275] error: namespace_g_join(JobId=22595640 SLUID=s8FXKR5P56S300): No such file or directory

vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/job_22595640/22595640_8_log.out

@@ -0,0 +1,538 @@
+submitit INFO (2026-05-08 20:18:11,513) - Starting with JobEnvironment(job_id=22595640, hostname=gcn81.local.snellius.surf.nl, local_rank=0(4), node=2(4), global_rank=8(16))
+submitit INFO (2026-05-08 20:18:11,513) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/job_22595640/22595640_submitted.pkl
+INFO:root:loaded pretrain params...
+{   'app': 'vjepa',
+    'cpus_per_task': 16,
+    'data': {   'batch_size': 64,
+                'crop_size': 224,
+                'dataset_fpcs': [16, 16],
+                'dataset_type': 'VideoDataset',
+                'datasets': [   '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
+                                '/scratch-shared/dcanez/data/ssv2/train.csv'],
+                'datasets_weights': [0.65, 0.35],
+                'fps': 4,
+                'num_workers': 10,
+                'patch_size': 14,
+                'persistent_workers': True,
+                'pin_mem': True,
+                'stage': [   {   'dest': 'kinetics_240',
+                                 'format': 'targz_parts',
+                                 'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
+                             {   'dest': 'ssv2',
+                                 'format': 'multipart_tar',
+                                 'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
+                'tubelet_size': 1},
+    'data_aug': {   'auto_augment': False,
+                    'motion_shift': False,
+                    'random_resize_aspect_ratio': [0.75, 1.35],
+                    'random_resize_scale': [0.3, 1.0],
+                    'reprob': 0.0},
+    'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m',
+    'loss': {'loss_exp': 1.0},
+    'mask': [   {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 8,
+                    'spatial_scale': [0.15, 0.15],
+                    'temporal_scale': [1.0, 1.0]},
+                {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 2,
+                    'spatial_scale': [0.7, 0.7],
+                    'temporal_scale': [1.0, 1.0]}],
+    'mem_per_gpu': '180G',
+    'meta': {   'dtype': 'bfloat16',
+                'knn_eval_epoch0': False,
+                'knn_eval_freq': 5,
+                'knn_eval_presets': [   {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
+                                                          'eval_videos_per_class': 25,
+                                                          'num_workers': 8,
+                                                          'train_videos_per_class': 100},
+                                            'preset': 'ucf101'},
+                                        {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
+                                                          'eval_videos_per_class': 100,
+                                                          'linear_probe': True,
+                                                          'num_workers': 8,
+                                                          'pool_type': 'temporal_concat',
+                                                          'train_videos_per_class': 500},
+                                            'preset': 'ssv2_coarse10'}],
+                'load_checkpoint': True,
+                'read_checkpoint': None,
+                'save_every_freq': 5,
+                'seed': 239,
+                'use_sdpa': True,
+                'use_wandb': True,
+                'wandb_project': 'vjepa_ablation'},
+    'metrics': {'sigreg': {}, 'std': {}},
+    'model': {   'class_token': False,
+                 'freeze_backbone': True,
+                 'is_causal': False,
+                 'lora_alpha': 16,
+                 'lora_rank': 8,
+                 'lora_targets': ['attn.qkv', 'attn.proj'],
+                 'model_name': 'vit_large_patch14_capi_lvd1689m',
+                 'num_reg_tokens': 16,
+                 'pred_depth': 6,
+                 'pred_embed_dim': 384,
+                 'pred_num_heads': 12,
+                 'rope_type': 'capi2d',
+                 'stem_type': '2d',
+                 'target_kind': 'frozen_2d',
+                 'target_type': 'vit_large_patch14_capi.lvd1689m',
+                 'uniform_power': True,
+                 'use_activation_checkpointing': True,
+                 'use_lora': True,
+                 'use_mask_tokens': True,
+                 'use_rope': True,
+                 'use_sdpa': True,
+                 'zero_init_mask_tokens': True},
+    'nodes': 4,
+    'optimization': {   'ema': [0.99925, 0.99925],
+                        'epochs': 100,
+                        'final_lr': 0.0001,
+                        'final_weight_decay': 0.04,
+                        'ipe': 300,
+                        'ipe_scale': 1.0,
+                        'lr': 0.0005,
+                        'start_lr': 0.0001,
+                        'warmup': 10,
+                        'weight_decay': 0.04},
+    'tasks_per_node': 4}
+INFO:root:Running pre-training of app: vjepa
+[INFO    ][2026-05-08 20:19:12][app.vjepa.train     ][main                     ] which_dtype='bfloat16'
+[INFO    ][2026-05-08 20:19:12][app.vjepa.train     ][main                     ] Disabling persistent_workers (incompatible with KNN eval)
+[INFO    ][2026-05-08 20:19:12][app.vjepa.train     ][main                     ] NCCL_SOCKET_IFNAME=eno
+[INFO    ][2026-05-08 20:19:14][app.vjepa.train     ][main                     ] Initialized (rank/world-size) 8/16, tasks_per_node=4
+[INFO    ][2026-05-08 20:19:14][root                ][stage_datasets           ] [local_rank 0/4] Staging kinetics_240 (targz_parts)
+[INFO    ][2026-05-08 20:19:14][root                ][_stage_targz_parts       ] [rank 0] Extracting 26/103 tar.gz parts to /scratch-node/dcanez.22595640/kinetics_240
+[INFO    ][2026-05-08 20:19:28][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 2/26 parts
+[INFO    ][2026-05-08 20:19:42][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 4/26 parts
+[INFO    ][2026-05-08 20:19:56][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 6/26 parts
+[INFO    ][2026-05-08 20:20:10][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 8/26 parts
+[INFO    ][2026-05-08 20:20:23][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 10/26 parts
+[INFO    ][2026-05-08 20:20:38][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 12/26 parts
+[INFO    ][2026-05-08 20:20:52][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 14/26 parts
+[INFO    ][2026-05-08 20:21:07][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 16/26 parts
+[INFO    ][2026-05-08 20:21:21][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 18/26 parts
+[INFO    ][2026-05-08 20:21:35][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 20/26 parts
+[INFO    ][2026-05-08 20:21:49][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 22/26 parts
+[INFO    ][2026-05-08 20:22:03][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 24/26 parts
+[INFO    ][2026-05-08 20:22:17][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 26/26 parts
+[INFO    ][2026-05-08 20:22:17][root                ][stage_datasets           ] [local_rank 0/4] Staging ssv2 (multipart_tar)
+[INFO    ][2026-05-08 20:22:17][root                ][_stage_multipart_tar     ] [rank 0] Extracting multipart tar (2 files) to /scratch-node/dcanez.22595640/ssv2
+[INFO    ][2026-05-08 20:23:16][root                ][stage_datasets           ] Data staging completed in 241.7s (4.0min)
+[INFO    ][2026-05-08 20:23:16][root                ][_rewrite_csv             ] Wrote local CSV: /scratch-node/dcanez.22595640/kinetics_240/train.csv (239789 entries)
+[INFO    ][2026-05-08 20:23:17][root                ][_rewrite_csv             ] Wrote local CSV: /scratch-node/dcanez.22595640/ssv2/train.csv (168913 entries)
+[INFO    ][2026-05-08 20:23:17][app.vjepa.train     ][main                     ] Staged datasets: ['/scratch-node/dcanez.22595640/kinetics_240/train.csv', '/scratch-node/dcanez.22595640/ssv2/train.csv']
+[INFO    ][2026-05-08 20:24:23][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
+[INFO    ][2026-05-08 20:24:25][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA applied: 48 modules replaced (rank=8, alpha=16, dropout=0.0, targets=['attn.qkv', 'attn.proj'], scope=blocks)
+[INFO    ][2026-05-08 20:24:25][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA trainable params: 1,179,648
+[INFO    ][2026-05-08 20:24:27][root                ][_build_frozen_2d_target  ] Loaded frozen_2d target from timm: vit_large_patch14_capi.lvd1689m
+[INFO    ][2026-05-08 20:24:28][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-08 20:24:28][root                ][init_video_model         ] VJEPAPredictorMultiSeqWrapper(
+  (backbone): VisionTransformerPredictor(
+    (predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
+    (mask_tokens): ParameterList(
+        (0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+    )
+    (predictor_blocks): ModuleList(
+      (0-5): 6 x Block(
+        (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (attn): RoPEAttention(
+          (qkv): Linear(in_features=384, out_features=1152, bias=True)
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): Linear(in_features=384, out_features=384, bias=True)
+          (proj_drop): Dropout(p=0.0, inplace=False)
+        )
+        (drop_path): Identity()
+        (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (mlp): MLP(
+          (fc1): Linear(in_features=384, out_features=1536, bias=True)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=1536, out_features=384, bias=True)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+    (predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
+  )
+)
+[INFO    ][2026-05-08 20:24:28][root                ][init_video_model         ] MultiSeqWrapper(
+  (backbone): Frozen2DTargetWrapper(
+    (backbone): Eva(
+      (patch_embed): PatchEmbed(
+        (proj): Conv2d(3, 1024, kernel_size=(14, 14), stride=(14, 14))
+        (norm): Identity()
+      )
+      (pos_drop): Dropout(p=0.0, inplace=False)
+      (norm_pre): Identity()
+      (blocks): ModuleList(
+        (0-23): 24 x EvaBlock(
+          (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+          (attn): EvaAttention(
+            (qkv): Linear(in_features=1024, out_features=3072, bias=False)
+            (q_norm): Identity()
+            (k_norm): Identity()
+            (attn_drop): Dropout(p=0.0, inplace=False)
+            (norm): Identity()
+            (proj): Linear(in_features=1024, out_features=1024, bias=True)
+            (proj_drop): Dropout(p=0.0, inplace=False)
+          )
+          (drop_path1): Identity()
+          (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+          (mlp): Mlp(
+            (fc1): Linear(in_features=1024, out_features=4096, bias=True)
+            (act): GELU(approximate='none')
+            (drop1): Dropout(p=0.0, inplace=False)
+            (norm): Identity()
+            (fc2): Linear(in_features=4096, out_features=1024, bias=True)
+            (drop2): Dropout(p=0.0, inplace=False)
+          )
+          (drop_path2): Identity()
+        )
+      )
+      (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+      (fc_norm): Identity()
+      (head_drop): Dropout(p=0.0, inplace=False)
+      (head): Identity()
+      (rope): _CapiPatchRoPE()
+    )
+  )
+)
+[INFO    ][2026-05-08 20:24:28][root                ][init_video_model         ] Encoder number of parameters: 1179648
+[INFO    ][2026-05-08 20:24:28][root                ][init_video_model         ] Predictor number of parameters: 11436928
+[INFO    ][2026-05-08 20:24:28][root                ][init_video_model         ] Target encoder number of parameters: 0
+[INFO    ][2026-05-08 20:24:29][root                ][make_videodataset        ] VideoDataset dataset created
+[INFO    ][2026-05-08 20:24:29][WeightedSampler     ][__init__                 ] Using DistributedWeightedSampler with rank 8 / 16
+[INFO    ][2026-05-08 20:24:29][root                ][make_videodataset        ] VideoDataset unsupervised data loader created
+[INFO    ][2026-05-08 20:24:29][app.vjepa.train     ][main                     ] iterations per epoch/dataset length: 300/399
+[INFO    ][2026-05-08 20:24:29][app.vjepa.train     ][main                     ] Wrapping models in DDP (rank 8)...
+[INFO    ][2026-05-08 20:24:41][app.vjepa.train     ][main                     ] Initializing loader...
+submitit INFO (2026-05-09 08:16:14,367) - Job has timed out. Ran 718 minutes out of requested 720 minutes.
+[INFO    ][2026-05-09 08:16:14][submitit            ][has_timed_out            ] Job has timed out. Ran 718 minutes out of requested 720 minutes.
+submitit WARNING (2026-05-09 08:16:14,380) - Caught signal SIGUSR2 on gcn81.local.snellius.surf.nl: this job is timed-out.
+[WARNING ][2026-05-09 08:16:14][submitit            ][checkpoint_and_try_requeue] Caught signal SIGUSR2 on gcn81.local.snellius.surf.nl: this job is timed-out.
+submitit INFO (2026-05-09 08:16:14,380) - Not checkpointing nor requeuing since I am a slave (procid=8).
+[INFO    ][2026-05-09 08:16:14][submitit            ][checkpoint_and_try_requeue] Not checkpointing nor requeuing since I am a slave (procid=8).
+submitit WARNING (2026-05-09 08:16:14,380) - Bypassing signal SIGTERM
+[WARNING ][2026-05-09 08:16:14][submitit            ][bypass                   ] Bypassing signal SIGTERM
+submitit WARNING (2026-05-09 08:16:14,380) - Bypassing signal SIGCONT
+[WARNING ][2026-05-09 08:16:14][submitit            ][bypass                   ] Bypassing signal SIGCONT
+submitit INFO (2026-05-09 08:19:43,932) - Starting with JobEnvironment(job_id=22595640, hostname=gcn76.local.snellius.surf.nl, local_rank=0(4), node=2(4), global_rank=8(16))
+submitit INFO (2026-05-09 08:19:44,068) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/job_22595640/22595640_submitted.pkl
+INFO:root:loaded pretrain params...
+{   'app': 'vjepa',
+    'cpus_per_task': 16,
+    'data': {   'batch_size': 64,
+                'crop_size': 224,
+                'dataset_fpcs': [16, 16],
+                'dataset_type': 'VideoDataset',
+                'datasets': [   '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
+                                '/scratch-shared/dcanez/data/ssv2/train.csv'],
+                'datasets_weights': [0.65, 0.35],
+                'fps': 4,
+                'num_workers': 10,
+                'patch_size': 14,
+                'persistent_workers': True,
+                'pin_mem': True,
+                'stage': [   {   'dest': 'kinetics_240',
+                                 'format': 'targz_parts',
+                                 'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
+                             {   'dest': 'ssv2',
+                                 'format': 'multipart_tar',
+                                 'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
+                'tubelet_size': 1},
+    'data_aug': {   'auto_augment': False,
+                    'motion_shift': False,
+                    'random_resize_aspect_ratio': [0.75, 1.35],
+                    'random_resize_scale': [0.3, 1.0],
+                    'reprob': 0.0},
+    'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m',
+    'loss': {'loss_exp': 1.0},
+    'mask': [   {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 8,
+                    'spatial_scale': [0.15, 0.15],
+                    'temporal_scale': [1.0, 1.0]},
+                {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 2,
+                    'spatial_scale': [0.7, 0.7],
+                    'temporal_scale': [1.0, 1.0]}],
+    'mem_per_gpu': '180G',
+    'meta': {   'dtype': 'bfloat16',
+                'knn_eval_epoch0': False,
+                'knn_eval_freq': 5,
+                'knn_eval_presets': [   {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
+                                                          'eval_videos_per_class': 25,
+                                                          'num_workers': 8,
+                                                          'train_videos_per_class': 100},
+                                            'preset': 'ucf101'},
+                                        {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
+                                                          'eval_videos_per_class': 100,
+                                                          'linear_probe': True,
+                                                          'num_workers': 8,
+                                                          'pool_type': 'temporal_concat',
+                                                          'train_videos_per_class': 500},
+                                            'preset': 'ssv2_coarse10'}],
+                'load_checkpoint': True,
+                'read_checkpoint': None,
+                'save_every_freq': 5,
+                'seed': 239,
+                'use_sdpa': True,
+                'use_wandb': True,
+                'wandb_project': 'vjepa_ablation'},
+    'metrics': {'sigreg': {}, 'std': {}},
+    'model': {   'class_token': False,
+                 'freeze_backbone': True,
+                 'is_causal': False,
+                 'lora_alpha': 16,
+                 'lora_rank': 8,
+                 'lora_targets': ['attn.qkv', 'attn.proj'],
+                 'model_name': 'vit_large_patch14_capi_lvd1689m',
+                 'num_reg_tokens': 16,
+                 'pred_depth': 6,
+                 'pred_embed_dim': 384,
+                 'pred_num_heads': 12,
+                 'rope_type': 'capi2d',
+                 'stem_type': '2d',
+                 'target_kind': 'frozen_2d',
+                 'target_type': 'vit_large_patch14_capi.lvd1689m',
+                 'uniform_power': True,
+                 'use_activation_checkpointing': True,
+                 'use_lora': True,
+                 'use_mask_tokens': True,
+                 'use_rope': True,
+                 'use_sdpa': True,
+                 'zero_init_mask_tokens': True},
+    'nodes': 4,
+    'optimization': {   'ema': [0.99925, 0.99925],
+                        'epochs': 100,
+                        'final_lr': 0.0001,
+                        'final_weight_decay': 0.04,
+                        'ipe': 300,
+                        'ipe_scale': 1.0,
+                        'lr': 0.0005,
+                        'start_lr': 0.0001,
+                        'warmup': 10,
+                        'weight_decay': 0.04},
+    'tasks_per_node': 4}
+INFO:root:Running pre-training of app: vjepa
+[INFO    ][2026-05-09 08:19:48][app.vjepa.train     ][main                     ] which_dtype='bfloat16'
+[INFO    ][2026-05-09 08:19:48][app.vjepa.train     ][main                     ] Disabling persistent_workers (incompatible with KNN eval)
+[INFO    ][2026-05-09 08:19:48][app.vjepa.train     ][main                     ] NCCL_SOCKET_IFNAME=eno
+[INFO    ][2026-05-09 08:20:51][app.vjepa.train     ][main                     ] Initialized (rank/world-size) 8/16, tasks_per_node=4
+[INFO    ][2026-05-09 08:20:51][root                ][stage_datasets           ] [local_rank 0/4] Staging kinetics_240 (targz_parts)
+[INFO    ][2026-05-09 08:20:51][root                ][_stage_targz_parts       ] [rank 0] Extracting 26/103 tar.gz parts to /scratch-node/dcanez.22595640/kinetics_240
+[INFO    ][2026-05-09 08:21:05][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 2/26 parts
+[INFO    ][2026-05-09 08:21:19][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 4/26 parts
+[INFO    ][2026-05-09 08:21:33][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 6/26 parts
+[INFO    ][2026-05-09 08:21:47][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 8/26 parts
+[INFO    ][2026-05-09 08:22:01][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 10/26 parts
+[INFO    ][2026-05-09 08:22:15][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 12/26 parts
+[INFO    ][2026-05-09 08:22:29][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 14/26 parts
+[INFO    ][2026-05-09 08:22:44][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 16/26 parts
+[INFO    ][2026-05-09 08:22:58][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 18/26 parts
+[INFO    ][2026-05-09 08:23:12][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 20/26 parts
+[INFO    ][2026-05-09 08:23:27][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 22/26 parts
+[INFO    ][2026-05-09 08:23:41][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 24/26 parts
+[INFO    ][2026-05-09 08:23:55][root                ][_stage_targz_parts       ] [local_rank 0] Extracted 26/26 parts
+[INFO    ][2026-05-09 08:23:55][root                ][stage_datasets           ] [local_rank 0/4] Staging ssv2 (multipart_tar)
+[INFO    ][2026-05-09 08:23:55][root                ][_stage_multipart_tar     ] [rank 0] Extracting multipart tar (2 files) to /scratch-node/dcanez.22595640/ssv2
+[INFO    ][2026-05-09 08:25:32][root                ][stage_datasets           ] Data staging completed in 281.1s (4.7min)
+[INFO    ][2026-05-09 08:25:33][root                ][_rewrite_csv             ] Wrote local CSV: /scratch-node/dcanez.22595640/kinetics_240/train.csv (239789 entries)
+[INFO    ][2026-05-09 08:25:33][root                ][_rewrite_csv             ] Wrote local CSV: /scratch-node/dcanez.22595640/ssv2/train.csv (168913 entries)
+[INFO    ][2026-05-09 08:25:33][app.vjepa.train     ][main                     ] Staged datasets: ['/scratch-node/dcanez.22595640/kinetics_240/train.csv', '/scratch-node/dcanez.22595640/ssv2/train.csv']
+[INFO    ][2026-05-09 08:26:26][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
+[INFO    ][2026-05-09 08:26:30][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA applied: 48 modules replaced (rank=8, alpha=16, dropout=0.0, targets=['attn.qkv', 'attn.proj'], scope=blocks)
+[INFO    ][2026-05-09 08:26:30][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA trainable params: 1,179,648
+[INFO    ][2026-05-09 08:26:33][root                ][_build_frozen_2d_target  ] Loaded frozen_2d target from timm: vit_large_patch14_capi.lvd1689m
+[INFO    ][2026-05-09 08:26:33][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-09 08:26:33][root                ][init_video_model         ] VJEPAPredictorMultiSeqWrapper(
+  (backbone): VisionTransformerPredictor(
+    (predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
+    (mask_tokens): ParameterList(
+        (0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+    )
+    (predictor_blocks): ModuleList(
+      (0-5): 6 x Block(
+        (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (attn): RoPEAttention(
+          (qkv): Linear(in_features=384, out_features=1152, bias=True)
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): Linear(in_features=384, out_features=384, bias=True)
+          (proj_drop): Dropout(p=0.0, inplace=False)
+        )
+        (drop_path): Identity()
+        (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (mlp): MLP(
+          (fc1): Linear(in_features=384, out_features=1536, bias=True)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=1536, out_features=384, bias=True)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+    (predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
+  )
+)
+[INFO    ][2026-05-09 08:26:33][root                ][init_video_model         ] MultiSeqWrapper(
+  (backbone): Frozen2DTargetWrapper(
+    (backbone): Eva(
+      (patch_embed): PatchEmbed(
+        (proj): Conv2d(3, 1024, kernel_size=(14, 14), stride=(14, 14))
+        (norm): Identity()
+      )
+      (pos_drop): Dropout(p=0.0, inplace=False)
+      (norm_pre): Identity()
+      (blocks): ModuleList(
+        (0-23): 24 x EvaBlock(
+          (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+          (attn): EvaAttention(
+            (qkv): Linear(in_features=1024, out_features=3072, bias=False)
+            (q_norm): Identity()
+            (k_norm): Identity()
+            (attn_drop): Dropout(p=0.0, inplace=False)
+            (norm): Identity()
+            (proj): Linear(in_features=1024, out_features=1024, bias=True)
+            (proj_drop): Dropout(p=0.0, inplace=False)
+          )
+          (drop_path1): Identity()
+          (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+          (mlp): Mlp(
+            (fc1): Linear(in_features=1024, out_features=4096, bias=True)
+            (act): GELU(approximate='none')
+            (drop1): Dropout(p=0.0, inplace=False)
+            (norm): Identity()
+            (fc2): Linear(in_features=4096, out_features=1024, bias=True)
+            (drop2): Dropout(p=0.0, inplace=False)
+          )
+          (drop_path2): Identity()
+        )
+      )
+      (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+      (fc_norm): Identity()
+      (head_drop): Dropout(p=0.0, inplace=False)
+      (head): Identity()
+      (rope): _CapiPatchRoPE()
+    )
+  )
+)
+[INFO    ][2026-05-09 08:26:33][root                ][init_video_model         ] Encoder number of parameters: 1179648
+[INFO    ][2026-05-09 08:26:33][root                ][init_video_model         ] Predictor number of parameters: 11436928
+[INFO    ][2026-05-09 08:26:33][root                ][init_video_model         ] Target encoder number of parameters: 0
+[INFO    ][2026-05-09 08:26:34][root                ][make_videodataset        ] VideoDataset dataset created
+[INFO    ][2026-05-09 08:26:34][WeightedSampler     ][__init__                 ] Using DistributedWeightedSampler with rank 8 / 16
+[INFO    ][2026-05-09 08:26:34][root                ][make_videodataset        ] VideoDataset unsupervised data loader created
+[INFO    ][2026-05-09 08:26:34][app.vjepa.train     ][main                     ] iterations per epoch/dataset length: 300/399
+[INFO    ][2026-05-09 08:26:34][app.vjepa.train     ][main                     ] Wrapping models in DDP (rank 8)...
+[INFO    ][2026-05-09 08:26:37][root                ][load_checkpoint          ] Loading checkpoint from /scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/latest.pt
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] loaded pretrained encoder from epoch 39 with msg: <All keys matched successfully>
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] loaded pretrained predictor from epoch 39 with msg: <All keys matched successfully>
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] loaded pretrained target encoder from epoch 39 with msg: <All keys matched successfully>
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] loaded optimizers from epoch 39
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] read-path: /scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/latest.pt
+[INFO    ][2026-05-09 08:26:39][app.vjepa.train     ][main                     ] Initializing loader...
+submitit WARNING (2026-05-09 09:56:00,842) - Bypassing signal SIGTERM
+[WARNING ][2026-05-09 09:56:00][submitit            ][bypass                   ] Bypassing signal SIGTERM
+submitit WARNING (2026-05-09 09:56:00,843) - Bypassing signal SIGCONT
+[WARNING ][2026-05-09 09:56:00][submitit            ][bypass                   ] Bypassing signal SIGCONT

vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/job_22595640/22595640_9_log.err

@@ -0,0 +1,305 @@
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
+  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
+[rank9]:[W508 20:22:16.521172962 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
+  scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+submitit WARNING (2026-05-09 08:16:14,422) - Caught signal SIGUSR2 on gcn81.local.snellius.surf.nl: this job is timed-out.
+submitit WARNING (2026-05-09 08:16:14,422) - Bypassing signal SIGTERM
+submitit WARNING (2026-05-09 08:16:14,422) - Bypassing signal SIGCONT
+[rank9]:[W509 08:16:18.184892772 TCPStore.cpp:125] [c10d] recvValue failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58918, remote=[gcn75.local.snellius.surf.nl]:37129): Failed to recv, got 0 bytes. Connection was likely closed. Did the remote server shutdown or crash?
+Exception raised from recvBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:682 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x149258bc0fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a3325d (0x14929cd4f25d in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x273 (0x14929cd4d1f3 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x149259e2feec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x149399299164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1493a9d0b19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1493a9d90100 in /lib64/libc.so.6)
+
+[rank9]:[W509 08:16:18.187128671 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 9] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Failed to recv, got 0 bytes. Connection was likely closed. Did the remote server shutdown or crash?
+[rank9]:[W509 08:16:19.187295348 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58918, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x149258bc0fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14929cd4e6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14929cd4d1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x149259e2feec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x149399299164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1493a9d0b19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1493a9d90100 in /lib64/libc.so.6)
+
+[rank9]:[W509 08:16:19.190513472 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 9] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank9]:[W509 08:16:20.190660378 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58918, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x149258bc0fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14929cd4e6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14929cd4d1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x149259e2feec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x149399299164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1493a9d0b19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1493a9d90100 in /lib64/libc.so.6)
+
+[rank9]:[W509 08:16:20.192910408 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 9] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank9]:[W509 08:16:21.193068154 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58918, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x149258bc0fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14929cd4e6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14929cd4d1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x149259e2feec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x149399299164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1493a9d0b19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1493a9d90100 in /lib64/libc.so.6)
+
+[rank9]:[W509 08:16:21.195326263 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 9] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank9]:[W509 08:16:22.195482179 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58918, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x149258bc0fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14929cd4e6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14929cd4d1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x149259e2feec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x149399299164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1493a9d0b19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1493a9d90100 in /lib64/libc.so.6)
+
+[rank9]:[W509 08:16:22.197666479 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 9] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank9]:[W509 08:16:23.197804635 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58918, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x149258bc0fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14929cd4e6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14929cd4d1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x149259e2feec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x149399299164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1493a9d0b19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1493a9d90100 in /lib64/libc.so.6)
+
+[rank9]:[W509 08:16:23.200092884 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 9] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank9]:[W509 08:16:24.200249010 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58918, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x149258bc0fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14929cd4e6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14929cd4d1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x149259e2feec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x149399299164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1493a9d0b19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1493a9d90100 in /lib64/libc.so.6)
+
+[rank9]:[W509 08:16:24.202525960 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 9] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank9]:[W509 08:16:25.202663076 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58918, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x149258bc0fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14929cd4e6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14929cd4d1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x149259e2feec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x149399299164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1493a9d0b19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1493a9d90100 in /lib64/libc.so.6)
+
+[rank9]:[W509 08:16:25.204843296 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 9] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank9]:[W509 08:16:26.204980902 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58918, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x149258bc0fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14929cd4e6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14929cd4d1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x149259e2feec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x149399299164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1493a9d0b19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1493a9d90100 in /lib64/libc.so.6)
+
+[rank9]:[W509 08:16:26.207238441 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 9] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank9]:[W509 08:16:27.207378847 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58918, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x149258bc0fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14929cd4e6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14929cd4d1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x149259e2feec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x149399299164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1493a9d0b19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1493a9d90100 in /lib64/libc.so.6)
+
+[rank9]:[W509 08:16:27.209599987 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 9] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank9]:[W509 08:16:28.209738063 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58918, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x149258bc0fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14929cd4e6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14929cd4d1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x149259e2feec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x149399299164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1493a9d0b19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1493a9d90100 in /lib64/libc.so.6)
+
+[rank9]:[W509 08:16:28.212025612 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 9] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank9]:[W509 08:16:29.212169858 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58918, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x149258bc0fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14929cd4e6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14929cd4d1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x149259e2feec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x149399299164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1493a9d0b19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1493a9d90100 in /lib64/libc.so.6)
+
+[rank9]:[W509 08:16:29.214551207 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 9] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank9]:[W509 08:16:30.214699853 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58918, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x149258bc0fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14929cd4e6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14929cd4d1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x149259e2feec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x149399299164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1493a9d0b19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1493a9d90100 in /lib64/libc.so.6)
+
+[rank9]:[W509 08:16:30.216944582 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 9] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank9]:[W509 08:16:31.217079759 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58918, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x149258bc0fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14929cd4e6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14929cd4d1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x149259e2feec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x149399299164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1493a9d0b19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1493a9d90100 in /lib64/libc.so.6)
+
+[rank9]:[W509 08:16:31.219233259 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 9] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank9]:[W509 08:16:32.219368605 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58918, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x149258bc0fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14929cd4e6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14929cd4d1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x149259e2feec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x149399299164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1493a9d0b19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1493a9d90100 in /lib64/libc.so.6)
+
+[rank9]:[W509 08:16:32.221668064 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 9] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank9]:[W509 08:16:33.221801620 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58918, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x149258bc0fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14929cd4e6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14929cd4d1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x149259e2feec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x149399299164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1493a9d0b19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1493a9d90100 in /lib64/libc.so.6)
+
+[rank9]:[W509 08:16:33.224038459 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 9] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank9]:[W509 08:16:34.224178316 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58918, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x149258bc0fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14929cd4e6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14929cd4d1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x149259e2feec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x149399299164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1493a9d0b19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1493a9d90100 in /lib64/libc.so.6)
+
+[rank9]:[W509 08:16:34.226423375 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 9] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank9]:[W509 08:16:35.226556761 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58918, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x149258bc0fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14929cd4e6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14929cd4d1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x149259e2feec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x149399299164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1493a9d0b19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1493a9d90100 in /lib64/libc.so.6)
+
+[rank9]:[W509 08:16:35.228825760 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 9] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank9]:[W509 08:16:36.228956937 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58918, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x149258bc0fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14929cd4e6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14929cd4d1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x149259e2feec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x149399299164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1493a9d0b19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1493a9d90100 in /lib64/libc.so.6)
+
+[rank9]:[W509 08:16:36.231185766 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 9] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank9]:[W509 08:16:37.231328632 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58918, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x149258bc0fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14929cd4e6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14929cd4d1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x149259e2feec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x149399299164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1493a9d0b19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1493a9d90100 in /lib64/libc.so.6)
+
+[rank9]:[W509 08:16:37.233516552 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 9] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank9]:[W509 08:16:38.233654878 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58918, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x149258bc0fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14929cd4e6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14929cd4d1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x149259e2feec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x149399299164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1493a9d0b19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1493a9d90100 in /lib64/libc.so.6)
+
+[rank9]:[W509 08:16:38.235955147 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 9] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank9]:[W509 08:16:39.236099333 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58918, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x149258bc0fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14929cd4e6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14929cd4d1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x149259e2feec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x149399299164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1493a9d0b19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1493a9d90100 in /lib64/libc.so.6)
+
+[rank9]:[W509 08:16:39.238355973 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 9] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank9]:[W509 08:16:40.238497829 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58918, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x149258bc0fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14929cd4e6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14929cd4d1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x149259e2feec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x149399299164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1493a9d0b19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1493a9d90100 in /lib64/libc.so.6)
+
+[rank9]:[W509 08:16:40.240697768 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 9] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank9]:[W509 08:16:41.240823385 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58918, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x149258bc0fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14929cd4e6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14929cd4d1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x149259e2feec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x149399299164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1493a9d0b19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1493a9d90100 in /lib64/libc.so.6)
+
+[rank9]:[W509 08:16:41.243032004 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 9] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank9]:[W509 08:16:42.243164351 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58918, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x149258bc0fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14929cd4e6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14929cd4d1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x149259e2feec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x149399299164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1493a9d0b19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1493a9d90100 in /lib64/libc.so.6)
+
+[rank9]:[W509 08:16:42.245438460 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 9] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+[rank9]:[W509 08:16:43.245567996 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=3, addr=[gcn81.local.snellius.surf.nl]:58918, remote=[gcn75.local.snellius.surf.nl]:37129): Broken pipe
+Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):
+frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x149258bc0fdd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libc10.so)
+frame #1: <unknown function> + 0x6a326d1 (0x14929cd4e6d1 in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x24d (0x14929cd4d1cd in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
+frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x44c (0x149259e2feec in /scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/torch/lib/libtorch_cuda.so)
+frame #4: <unknown function> + 0xed164 (0x149399299164 in /sw/arch/RHEL9/EB_production/2025/software/GCCcore/14.2.0/lib64/libstdc++.so.6)
+frame #5: <unknown function> + 0x8a19a (0x1493a9d0b19a in /lib64/libc.so.6)
+frame #6: <unknown function> + 0x10f100 (0x1493a9d90100 in /lib64/libc.so.6)
+
+[rank9]:[W509 08:16:43.247826255 ProcessGroupNCCL.cpp:1802] [PG ID 0 PG GUID 0(default_pg) Rank 9] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe
+/scratch-shared/dcanez/.cache/uv5/virtualenvs/vd/lib/python3.13/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
+  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
+[rank9]:[W509 08:24:24.475564329 ProcessGroupNCCL.cpp:5138] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/utils.py:794: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
+  scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
+/gpfs/scratch1/shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/code/app/vjepa/train.py:955: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
+  with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
+submitit WARNING (2026-05-09 09:56:00,612) - Bypassing signal SIGTERM
+submitit WARNING (2026-05-09 09:56:00,627) - Bypassing signal SIGCONT

vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/job_22595640/22595640_9_log.out

@@ -0,0 +1,530 @@
+submitit INFO (2026-05-08 20:18:11,513) - Starting with JobEnvironment(job_id=22595640, hostname=gcn81.local.snellius.surf.nl, local_rank=1(4), node=2(4), global_rank=9(16))
+submitit INFO (2026-05-08 20:18:11,513) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/job_22595640/22595640_submitted.pkl
+INFO:root:loaded pretrain params...
+{   'app': 'vjepa',
+    'cpus_per_task': 16,
+    'data': {   'batch_size': 64,
+                'crop_size': 224,
+                'dataset_fpcs': [16, 16],
+                'dataset_type': 'VideoDataset',
+                'datasets': [   '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
+                                '/scratch-shared/dcanez/data/ssv2/train.csv'],
+                'datasets_weights': [0.65, 0.35],
+                'fps': 4,
+                'num_workers': 10,
+                'patch_size': 14,
+                'persistent_workers': True,
+                'pin_mem': True,
+                'stage': [   {   'dest': 'kinetics_240',
+                                 'format': 'targz_parts',
+                                 'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
+                             {   'dest': 'ssv2',
+                                 'format': 'multipart_tar',
+                                 'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
+                'tubelet_size': 1},
+    'data_aug': {   'auto_augment': False,
+                    'motion_shift': False,
+                    'random_resize_aspect_ratio': [0.75, 1.35],
+                    'random_resize_scale': [0.3, 1.0],
+                    'reprob': 0.0},
+    'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m',
+    'loss': {'loss_exp': 1.0},
+    'mask': [   {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 8,
+                    'spatial_scale': [0.15, 0.15],
+                    'temporal_scale': [1.0, 1.0]},
+                {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 2,
+                    'spatial_scale': [0.7, 0.7],
+                    'temporal_scale': [1.0, 1.0]}],
+    'mem_per_gpu': '180G',
+    'meta': {   'dtype': 'bfloat16',
+                'knn_eval_epoch0': False,
+                'knn_eval_freq': 5,
+                'knn_eval_presets': [   {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
+                                                          'eval_videos_per_class': 25,
+                                                          'num_workers': 8,
+                                                          'train_videos_per_class': 100},
+                                            'preset': 'ucf101'},
+                                        {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
+                                                          'eval_videos_per_class': 100,
+                                                          'linear_probe': True,
+                                                          'num_workers': 8,
+                                                          'pool_type': 'temporal_concat',
+                                                          'train_videos_per_class': 500},
+                                            'preset': 'ssv2_coarse10'}],
+                'load_checkpoint': True,
+                'read_checkpoint': None,
+                'save_every_freq': 5,
+                'seed': 239,
+                'use_sdpa': True,
+                'use_wandb': True,
+                'wandb_project': 'vjepa_ablation'},
+    'metrics': {'sigreg': {}, 'std': {}},
+    'model': {   'class_token': False,
+                 'freeze_backbone': True,
+                 'is_causal': False,
+                 'lora_alpha': 16,
+                 'lora_rank': 8,
+                 'lora_targets': ['attn.qkv', 'attn.proj'],
+                 'model_name': 'vit_large_patch14_capi_lvd1689m',
+                 'num_reg_tokens': 16,
+                 'pred_depth': 6,
+                 'pred_embed_dim': 384,
+                 'pred_num_heads': 12,
+                 'rope_type': 'capi2d',
+                 'stem_type': '2d',
+                 'target_kind': 'frozen_2d',
+                 'target_type': 'vit_large_patch14_capi.lvd1689m',
+                 'uniform_power': True,
+                 'use_activation_checkpointing': True,
+                 'use_lora': True,
+                 'use_mask_tokens': True,
+                 'use_rope': True,
+                 'use_sdpa': True,
+                 'zero_init_mask_tokens': True},
+    'nodes': 4,
+    'optimization': {   'ema': [0.99925, 0.99925],
+                        'epochs': 100,
+                        'final_lr': 0.0001,
+                        'final_weight_decay': 0.04,
+                        'ipe': 300,
+                        'ipe_scale': 1.0,
+                        'lr': 0.0005,
+                        'start_lr': 0.0001,
+                        'warmup': 10,
+                        'weight_decay': 0.04},
+    'tasks_per_node': 4}
+INFO:root:Running pre-training of app: vjepa
+[INFO    ][2026-05-08 20:19:12][app.vjepa.train     ][main                     ] which_dtype='bfloat16'
+[INFO    ][2026-05-08 20:19:12][app.vjepa.train     ][main                     ] Disabling persistent_workers (incompatible with KNN eval)
+[INFO    ][2026-05-08 20:19:12][app.vjepa.train     ][main                     ] NCCL_SOCKET_IFNAME=eno
+[INFO    ][2026-05-08 20:19:15][app.vjepa.train     ][main                     ] Initialized (rank/world-size) 9/16, tasks_per_node=4
+[INFO    ][2026-05-08 20:19:15][root                ][stage_datasets           ] [local_rank 1/4] Staging kinetics_240 (targz_parts)
+[INFO    ][2026-05-08 20:19:15][root                ][_stage_targz_parts       ] [rank 1] Extracting 26/103 tar.gz parts to /scratch-node/dcanez.22595640/kinetics_240
+[INFO    ][2026-05-08 20:19:28][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 2/26 parts
+[INFO    ][2026-05-08 20:19:41][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 4/26 parts
+[INFO    ][2026-05-08 20:19:55][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 6/26 parts
+[INFO    ][2026-05-08 20:20:09][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 8/26 parts
+[INFO    ][2026-05-08 20:20:23][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 10/26 parts
+[INFO    ][2026-05-08 20:20:38][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 12/26 parts
+[INFO    ][2026-05-08 20:20:50][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 14/26 parts
+[INFO    ][2026-05-08 20:21:05][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 16/26 parts
+[INFO    ][2026-05-08 20:21:19][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 18/26 parts
+[INFO    ][2026-05-08 20:21:33][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 20/26 parts
+[INFO    ][2026-05-08 20:21:47][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 22/26 parts
+[INFO    ][2026-05-08 20:22:02][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 24/26 parts
+[INFO    ][2026-05-08 20:22:16][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 26/26 parts
+[INFO    ][2026-05-08 20:22:16][root                ][stage_datasets           ] [local_rank 1/4] Staging ssv2 (multipart_tar)
+[INFO    ][2026-05-08 20:23:17][app.vjepa.train     ][main                     ] Staged datasets: ['/scratch-node/dcanez.22595640/kinetics_240/train.csv', '/scratch-node/dcanez.22595640/ssv2/train.csv']
+[INFO    ][2026-05-08 20:24:23][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
+[INFO    ][2026-05-08 20:24:25][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA applied: 48 modules replaced (rank=8, alpha=16, dropout=0.0, targets=['attn.qkv', 'attn.proj'], scope=blocks)
+[INFO    ][2026-05-08 20:24:25][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA trainable params: 1,179,648
+[INFO    ][2026-05-08 20:24:27][root                ][_build_frozen_2d_target  ] Loaded frozen_2d target from timm: vit_large_patch14_capi.lvd1689m
+[INFO    ][2026-05-08 20:24:28][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-08 20:24:28][root                ][init_video_model         ] VJEPAPredictorMultiSeqWrapper(
+  (backbone): VisionTransformerPredictor(
+    (predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
+    (mask_tokens): ParameterList(
+        (0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+    )
+    (predictor_blocks): ModuleList(
+      (0-5): 6 x Block(
+        (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (attn): RoPEAttention(
+          (qkv): Linear(in_features=384, out_features=1152, bias=True)
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): Linear(in_features=384, out_features=384, bias=True)
+          (proj_drop): Dropout(p=0.0, inplace=False)
+        )
+        (drop_path): Identity()
+        (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (mlp): MLP(
+          (fc1): Linear(in_features=384, out_features=1536, bias=True)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=1536, out_features=384, bias=True)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+    (predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
+  )
+)
+[INFO    ][2026-05-08 20:24:28][root                ][init_video_model         ] MultiSeqWrapper(
+  (backbone): Frozen2DTargetWrapper(
+    (backbone): Eva(
+      (patch_embed): PatchEmbed(
+        (proj): Conv2d(3, 1024, kernel_size=(14, 14), stride=(14, 14))
+        (norm): Identity()
+      )
+      (pos_drop): Dropout(p=0.0, inplace=False)
+      (norm_pre): Identity()
+      (blocks): ModuleList(
+        (0-23): 24 x EvaBlock(
+          (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+          (attn): EvaAttention(
+            (qkv): Linear(in_features=1024, out_features=3072, bias=False)
+            (q_norm): Identity()
+            (k_norm): Identity()
+            (attn_drop): Dropout(p=0.0, inplace=False)
+            (norm): Identity()
+            (proj): Linear(in_features=1024, out_features=1024, bias=True)
+            (proj_drop): Dropout(p=0.0, inplace=False)
+          )
+          (drop_path1): Identity()
+          (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+          (mlp): Mlp(
+            (fc1): Linear(in_features=1024, out_features=4096, bias=True)
+            (act): GELU(approximate='none')
+            (drop1): Dropout(p=0.0, inplace=False)
+            (norm): Identity()
+            (fc2): Linear(in_features=4096, out_features=1024, bias=True)
+            (drop2): Dropout(p=0.0, inplace=False)
+          )
+          (drop_path2): Identity()
+        )
+      )
+      (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+      (fc_norm): Identity()
+      (head_drop): Dropout(p=0.0, inplace=False)
+      (head): Identity()
+      (rope): _CapiPatchRoPE()
+    )
+  )
+)
+[INFO    ][2026-05-08 20:24:28][root                ][init_video_model         ] Encoder number of parameters: 1179648
+[INFO    ][2026-05-08 20:24:28][root                ][init_video_model         ] Predictor number of parameters: 11436928
+[INFO    ][2026-05-08 20:24:28][root                ][init_video_model         ] Target encoder number of parameters: 0
+[INFO    ][2026-05-08 20:24:29][root                ][make_videodataset        ] VideoDataset dataset created
+[INFO    ][2026-05-08 20:24:29][WeightedSampler     ][__init__                 ] Using DistributedWeightedSampler with rank 9 / 16
+[INFO    ][2026-05-08 20:24:29][root                ][make_videodataset        ] VideoDataset unsupervised data loader created
+[INFO    ][2026-05-08 20:24:29][app.vjepa.train     ][main                     ] iterations per epoch/dataset length: 300/399
+[INFO    ][2026-05-08 20:24:29][app.vjepa.train     ][main                     ] Wrapping models in DDP (rank 9)...
+[INFO    ][2026-05-08 20:24:41][app.vjepa.train     ][main                     ] Initializing loader...
+submitit INFO (2026-05-09 08:16:14,352) - Job has timed out. Ran 718 minutes out of requested 720 minutes.
+[INFO    ][2026-05-09 08:16:14][submitit            ][has_timed_out            ] Job has timed out. Ran 718 minutes out of requested 720 minutes.
+submitit WARNING (2026-05-09 08:16:14,422) - Caught signal SIGUSR2 on gcn81.local.snellius.surf.nl: this job is timed-out.
+[WARNING ][2026-05-09 08:16:14][submitit            ][checkpoint_and_try_requeue] Caught signal SIGUSR2 on gcn81.local.snellius.surf.nl: this job is timed-out.
+submitit INFO (2026-05-09 08:16:14,422) - Not checkpointing nor requeuing since I am a slave (procid=9).
+[INFO    ][2026-05-09 08:16:14][submitit            ][checkpoint_and_try_requeue] Not checkpointing nor requeuing since I am a slave (procid=9).
+submitit WARNING (2026-05-09 08:16:14,422) - Bypassing signal SIGTERM
+[WARNING ][2026-05-09 08:16:14][submitit            ][bypass                   ] Bypassing signal SIGTERM
+submitit WARNING (2026-05-09 08:16:14,422) - Bypassing signal SIGCONT
+[WARNING ][2026-05-09 08:16:14][submitit            ][bypass                   ] Bypassing signal SIGCONT
+submitit INFO (2026-05-09 08:19:43,932) - Starting with JobEnvironment(job_id=22595640, hostname=gcn76.local.snellius.surf.nl, local_rank=1(4), node=2(4), global_rank=9(16))
+submitit INFO (2026-05-09 08:19:44,810) - Loading pickle: /scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/job_22595640/22595640_submitted.pkl
+INFO:root:loaded pretrain params...
+{   'app': 'vjepa',
+    'cpus_per_task': 16,
+    'data': {   'batch_size': 64,
+                'crop_size': 224,
+                'dataset_fpcs': [16, 16],
+                'dataset_type': 'VideoDataset',
+                'datasets': [   '/scratch-shared/dcanez/data/kinetics/k400/train.csv',
+                                '/scratch-shared/dcanez/data/ssv2/train.csv'],
+                'datasets_weights': [0.65, 0.35],
+                'fps': 4,
+                'num_workers': 10,
+                'patch_size': 14,
+                'persistent_workers': True,
+                'pin_mem': True,
+                'stage': [   {   'dest': 'kinetics_240',
+                                 'format': 'targz_parts',
+                                 'src': '/scratch-shared/dcanez/data/kinetics/k400/tars_240/'},
+                             {   'dest': 'ssv2',
+                                 'format': 'multipart_tar',
+                                 'src': '/scratch-nvme/ml-datasets/something-something-v2/'}],
+                'tubelet_size': 1},
+    'data_aug': {   'auto_augment': False,
+                    'motion_shift': False,
+                    'random_resize_aspect_ratio': [0.75, 1.35],
+                    'random_resize_scale': [0.3, 1.0],
+                    'reprob': 0.0},
+    'folder': '/scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m',
+    'loss': {'loss_exp': 1.0},
+    'mask': [   {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 8,
+                    'spatial_scale': [0.15, 0.15],
+                    'temporal_scale': [1.0, 1.0]},
+                {   'aspect_ratio': [0.75, 1.5],
+                    'full_complement': False,
+                    'max_keep': None,
+                    'max_temporal_keep': 1.0,
+                    'num_blocks': 2,
+                    'spatial_scale': [0.7, 0.7],
+                    'temporal_scale': [1.0, 1.0]}],
+    'mem_per_gpu': '180G',
+    'meta': {   'dtype': 'bfloat16',
+                'knn_eval_epoch0': False,
+                'knn_eval_freq': 5,
+                'knn_eval_presets': [   {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/train.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/ucf101/ucfTrainTestlist/val.csv',
+                                                          'eval_videos_per_class': 25,
+                                                          'num_workers': 8,
+                                                          'train_videos_per_class': 100},
+                                            'preset': 'ucf101'},
+                                        {   'config': {   'batch_size': 64,
+                                                          'dataset_train': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/train_coarse10.csv',
+                                                          'dataset_val': '/scratch-shared/mdorkenw/20bn-something-something-v2/something-something-v2-annotations/val_coarse10.csv',
+                                                          'eval_videos_per_class': 100,
+                                                          'linear_probe': True,
+                                                          'num_workers': 8,
+                                                          'pool_type': 'temporal_concat',
+                                                          'train_videos_per_class': 500},
+                                            'preset': 'ssv2_coarse10'}],
+                'load_checkpoint': True,
+                'read_checkpoint': None,
+                'save_every_freq': 5,
+                'seed': 239,
+                'use_sdpa': True,
+                'use_wandb': True,
+                'wandb_project': 'vjepa_ablation'},
+    'metrics': {'sigreg': {}, 'std': {}},
+    'model': {   'class_token': False,
+                 'freeze_backbone': True,
+                 'is_causal': False,
+                 'lora_alpha': 16,
+                 'lora_rank': 8,
+                 'lora_targets': ['attn.qkv', 'attn.proj'],
+                 'model_name': 'vit_large_patch14_capi_lvd1689m',
+                 'num_reg_tokens': 16,
+                 'pred_depth': 6,
+                 'pred_embed_dim': 384,
+                 'pred_num_heads': 12,
+                 'rope_type': 'capi2d',
+                 'stem_type': '2d',
+                 'target_kind': 'frozen_2d',
+                 'target_type': 'vit_large_patch14_capi.lvd1689m',
+                 'uniform_power': True,
+                 'use_activation_checkpointing': True,
+                 'use_lora': True,
+                 'use_mask_tokens': True,
+                 'use_rope': True,
+                 'use_sdpa': True,
+                 'zero_init_mask_tokens': True},
+    'nodes': 4,
+    'optimization': {   'ema': [0.99925, 0.99925],
+                        'epochs': 100,
+                        'final_lr': 0.0001,
+                        'final_weight_decay': 0.04,
+                        'ipe': 300,
+                        'ipe_scale': 1.0,
+                        'lr': 0.0005,
+                        'start_lr': 0.0001,
+                        'warmup': 10,
+                        'weight_decay': 0.04},
+    'tasks_per_node': 4}
+INFO:root:Running pre-training of app: vjepa
+[INFO    ][2026-05-09 08:19:48][app.vjepa.train     ][main                     ] which_dtype='bfloat16'
+[INFO    ][2026-05-09 08:19:48][app.vjepa.train     ][main                     ] Disabling persistent_workers (incompatible with KNN eval)
+[INFO    ][2026-05-09 08:19:48][app.vjepa.train     ][main                     ] NCCL_SOCKET_IFNAME=eno
+[INFO    ][2026-05-09 08:21:22][app.vjepa.train     ][main                     ] Initialized (rank/world-size) 9/16, tasks_per_node=4
+[INFO    ][2026-05-09 08:21:22][root                ][stage_datasets           ] [local_rank 1/4] Staging kinetics_240 (targz_parts)
+[INFO    ][2026-05-09 08:21:22][root                ][_stage_targz_parts       ] [rank 1] Extracting 26/103 tar.gz parts to /scratch-node/dcanez.22595640/kinetics_240
+[INFO    ][2026-05-09 08:21:35][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 2/26 parts
+[INFO    ][2026-05-09 08:21:49][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 4/26 parts
+[INFO    ][2026-05-09 08:22:03][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 6/26 parts
+[INFO    ][2026-05-09 08:22:16][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 8/26 parts
+[INFO    ][2026-05-09 08:22:31][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 10/26 parts
+[INFO    ][2026-05-09 08:22:45][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 12/26 parts
+[INFO    ][2026-05-09 08:22:58][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 14/26 parts
+[INFO    ][2026-05-09 08:23:12][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 16/26 parts
+[INFO    ][2026-05-09 08:23:26][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 18/26 parts
+[INFO    ][2026-05-09 08:23:41][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 20/26 parts
+[INFO    ][2026-05-09 08:23:55][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 22/26 parts
+[INFO    ][2026-05-09 08:24:09][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 24/26 parts
+[INFO    ][2026-05-09 08:24:23][root                ][_stage_targz_parts       ] [local_rank 1] Extracted 26/26 parts
+[INFO    ][2026-05-09 08:24:23][root                ][stage_datasets           ] [local_rank 1/4] Staging ssv2 (multipart_tar)
+[INFO    ][2026-05-09 08:25:33][app.vjepa.train     ][main                     ] Staged datasets: ['/scratch-node/dcanez.22595640/kinetics_240/train.csv', '/scratch-node/dcanez.22595640/ssv2/train.csv']
+[INFO    ][2026-05-09 08:26:26][experiments.stmodels.vision_transformers_v3][vit_large_patch14_capi_lvd1689m] Loading pretrained weights for vit_large_patch14_capi_lvd1689m from https://dl.fbaipublicfiles.com/capi/capi_vitl14_lvd.pth
+[INFO    ][2026-05-09 08:26:30][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA applied: 48 modules replaced (rank=8, alpha=16, dropout=0.0, targets=['attn.qkv', 'attn.proj'], scope=blocks)
+[INFO    ][2026-05-09 08:26:30][experiments.stmodels.lora][apply_lora_to_blocks     ] LoRA trainable params: 1,179,648
+[INFO    ][2026-05-09 08:26:33][root                ][_build_frozen_2d_target  ] Loaded frozen_2d target from timm: vit_large_patch14_capi.lvd1689m
+[INFO    ][2026-05-09 08:26:33][root                ][init_video_model         ] ViTMultiSeqWrapper(
+  (backbone): VisionTransformer(
+    (patch_embed): PatchEmbed3D(
+      (proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
+    )
+    (rope): CAPI2DRoPE()
+    (blocks): ModuleList(
+      (0-23): 24 x Block(
+        (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (rope_impl): CAPI2DRoPE()
+        (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+        (drop_path1): Identity()
+        (drop_path2): Identity()
+        (attn): Attention(
+          (qkv): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=3072, bias=False)
+            (lora_dropout): Identity()
+          )
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): LoRALinear(
+            (lora_A): Linear(in_features=1024, out_features=8, bias=False)
+            (lora_B): Linear(in_features=8, out_features=1024, bias=False)
+            (lora_dropout): Identity()
+          )
+          (proj_drop): Dropout(p=0.0, inplace=False)
+          (rope_impl): CAPI2DRoPE()
+        )
+        (mlp): MLP(
+          (fc1): Linear(in_features=1024, out_features=4096, bias=False)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=4096, out_features=1024, bias=False)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=False)
+  )
+)
+[INFO    ][2026-05-09 08:26:33][root                ][init_video_model         ] VJEPAPredictorMultiSeqWrapper(
+  (backbone): VisionTransformerPredictor(
+    (predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
+    (mask_tokens): ParameterList(
+        (0): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (1): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (2): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+        (3): Parameter containing: [torch.float32 of size 1x1x384 (cuda:0)]
+    )
+    (predictor_blocks): ModuleList(
+      (0-5): 6 x Block(
+        (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (attn): RoPEAttention(
+          (qkv): Linear(in_features=384, out_features=1152, bias=True)
+          (attn_drop): Dropout(p=0.0, inplace=False)
+          (proj): Linear(in_features=384, out_features=384, bias=True)
+          (proj_drop): Dropout(p=0.0, inplace=False)
+        )
+        (drop_path): Identity()
+        (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+        (mlp): MLP(
+          (fc1): Linear(in_features=384, out_features=1536, bias=True)
+          (act): GELU(approximate='none')
+          (fc2): Linear(in_features=1536, out_features=384, bias=True)
+          (drop): Dropout(p=0.0, inplace=False)
+        )
+      )
+    )
+    (predictor_norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
+    (predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
+  )
+)
+[INFO    ][2026-05-09 08:26:33][root                ][init_video_model         ] MultiSeqWrapper(
+  (backbone): Frozen2DTargetWrapper(
+    (backbone): Eva(
+      (patch_embed): PatchEmbed(
+        (proj): Conv2d(3, 1024, kernel_size=(14, 14), stride=(14, 14))
+        (norm): Identity()
+      )
+      (pos_drop): Dropout(p=0.0, inplace=False)
+      (norm_pre): Identity()
+      (blocks): ModuleList(
+        (0-23): 24 x EvaBlock(
+          (norm1): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+          (attn): EvaAttention(
+            (qkv): Linear(in_features=1024, out_features=3072, bias=False)
+            (q_norm): Identity()
+            (k_norm): Identity()
+            (attn_drop): Dropout(p=0.0, inplace=False)
+            (norm): Identity()
+            (proj): Linear(in_features=1024, out_features=1024, bias=True)
+            (proj_drop): Dropout(p=0.0, inplace=False)
+          )
+          (drop_path1): Identity()
+          (norm2): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+          (mlp): Mlp(
+            (fc1): Linear(in_features=1024, out_features=4096, bias=True)
+            (act): GELU(approximate='none')
+            (drop1): Dropout(p=0.0, inplace=False)
+            (norm): Identity()
+            (fc2): Linear(in_features=4096, out_features=1024, bias=True)
+            (drop2): Dropout(p=0.0, inplace=False)
+          )
+          (drop_path2): Identity()
+        )
+      )
+      (norm): RMSNorm((1024,), eps=1e-05, elementwise_affine=True)
+      (fc_norm): Identity()
+      (head_drop): Dropout(p=0.0, inplace=False)
+      (head): Identity()
+      (rope): _CapiPatchRoPE()
+    )
+  )
+)
+[INFO    ][2026-05-09 08:26:33][root                ][init_video_model         ] Encoder number of parameters: 1179648
+[INFO    ][2026-05-09 08:26:33][root                ][init_video_model         ] Predictor number of parameters: 11436928
+[INFO    ][2026-05-09 08:26:33][root                ][init_video_model         ] Target encoder number of parameters: 0
+[INFO    ][2026-05-09 08:26:34][root                ][make_videodataset        ] VideoDataset dataset created
+[INFO    ][2026-05-09 08:26:34][WeightedSampler     ][__init__                 ] Using DistributedWeightedSampler with rank 9 / 16
+[INFO    ][2026-05-09 08:26:34][root                ][make_videodataset        ] VideoDataset unsupervised data loader created
+[INFO    ][2026-05-09 08:26:34][app.vjepa.train     ][main                     ] iterations per epoch/dataset length: 300/399
+[INFO    ][2026-05-09 08:26:35][app.vjepa.train     ][main                     ] Wrapping models in DDP (rank 9)...
+[INFO    ][2026-05-09 08:26:37][root                ][load_checkpoint          ] Loading checkpoint from /scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/latest.pt
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] loaded pretrained encoder from epoch 39 with msg: <All keys matched successfully>
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] loaded pretrained predictor from epoch 39 with msg: <All keys matched successfully>
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] loaded pretrained target encoder from epoch 39 with msg: <All keys matched successfully>
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] loaded optimizers from epoch 39
+[INFO    ][2026-05-09 08:26:39][root                ][load_checkpoint          ] read-path: /scratch-shared/dcanez/runs/vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/latest.pt
+[INFO    ][2026-05-09 08:26:39][app.vjepa.train     ][main                     ] Initializing loader...
+submitit WARNING (2026-05-09 09:56:00,612) - Bypassing signal SIGTERM
+[WARNING ][2026-05-09 09:56:00][submitit            ][bypass                   ] Bypassing signal SIGTERM
+submitit WARNING (2026-05-09 09:56:00,627) - Bypassing signal SIGCONT
+[WARNING ][2026-05-09 09:56:00][submitit            ][bypass                   ] Bypassing signal SIGCONT

vjepa/inflated/d001_lora_frozen2d_bidir_capilvd1689m/latest.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a21ae86b17db56c94d6fdeb4f78bdd7ba05a4de4b58552763f3111549a1e82a2
+size 2573644533