sync run artifacts: inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3
Browse filesThis view is limited to 50 files because it contains too many changes. See raw diff
- inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/best.pt +3 -0
- inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/crash_rank0.log +38 -0
- inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/crash_rank1.log +38 -0
- inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/crash_rank10.log +38 -0
- inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/crash_rank11.log +38 -0
- inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/crash_rank12.log +38 -0
- inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/crash_rank13.log +38 -0
- inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/crash_rank15.log +38 -0
- inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/crash_rank16.log +38 -0
- inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/crash_rank17.log +38 -0
- inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/crash_rank18.log +38 -0
- inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/crash_rank19.log +38 -0
- inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/crash_rank2.log +38 -0
- inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/crash_rank20.log +38 -0
- inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/crash_rank21.log +38 -0
- inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/crash_rank22.log +38 -0
- inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/crash_rank23.log +38 -0
- inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/crash_rank24.log +38 -0
- inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/crash_rank25.log +38 -0
- inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/crash_rank26.log +38 -0
- inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/crash_rank27.log +38 -0
- inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/crash_rank28.log +38 -0
- inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/crash_rank29.log +38 -0
- inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/crash_rank3.log +38 -0
- inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/crash_rank30.log +38 -0
- inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/crash_rank31.log +38 -0
- inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/crash_rank5.log +38 -0
- inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/crash_rank6.log +38 -0
- inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/crash_rank7.log +38 -0
- inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/crash_rank8.log +38 -0
- inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/crash_rank9.log +38 -0
- inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/git-info.txt +2 -0
- inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/job_45423709/45423709_0_log.err +97 -0
- inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/job_45423709/45423709_0_log.out +0 -0
- inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/job_45423709/45423709_10_log.err +13 -0
- inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/job_45423709/45423709_10_log.out +208 -0
- inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/job_45423709/45423709_11_log.err +13 -0
- inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/job_45423709/45423709_11_log.out +208 -0
- inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/job_45423709/45423709_12_log.err +13 -0
- inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/job_45423709/45423709_12_log.out +208 -0
- inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/job_45423709/45423709_13_log.err +15 -0
- inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/job_45423709/45423709_13_log.out +208 -0
- inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/job_45423709/45423709_14_log.err +13 -0
- inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/job_45423709/45423709_14_log.out +208 -0
- inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/job_45423709/45423709_15_log.err +13 -0
- inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/job_45423709/45423709_15_log.out +208 -0
- inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/job_45423709/45423709_16_log.err +13 -0
- inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/job_45423709/45423709_16_log.out +208 -0
- inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/job_45423709/45423709_17_log.err +13 -0
- inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/job_45423709/45423709_17_log.out +208 -0
inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/best.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:0d515fc46d5c6bd80b161589e03f2828b87476151aa2ad282c871a6c83786a81
|
| 3 |
+
size 5114658721
|
inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/crash_rank0.log
ADDED
|
@@ -0,0 +1,38 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Rank 0 crashed at epoch 53, itr 79
|
| 2 |
+
Error: Pin memory thread exited unexpectedly
|
| 3 |
+
|
| 4 |
+
Traceback (most recent call last):
|
| 5 |
+
File "<frozen runpy>", line 198, in _run_module_as_main
|
| 6 |
+
File "<frozen runpy>", line 88, in _run_code
|
| 7 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/_submit.py", line 11, in <module>
|
| 8 |
+
submitit_main()
|
| 9 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 76, in submitit_main
|
| 10 |
+
process_job(args.folder)
|
| 11 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 69, in process_job
|
| 12 |
+
raise error
|
| 13 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 55, in process_job
|
| 14 |
+
result = delayed.result()
|
| 15 |
+
^^^^^^^^^^^^^^^^
|
| 16 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/utils.py", line 137, in result
|
| 17 |
+
self._result = self.function(*self.args, **self.kwargs)
|
| 18 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 19 |
+
File "/leonardo/home/userexternal/dcanezil/vd/app/main_distributed.py", line 120, in __call__
|
| 20 |
+
"Value is YAML-parsed (ints/strings/lists). Repeatable. Applied before submit, so it covers "
|
| 21 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 22 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/scaffold.py", line 17, in main
|
| 23 |
+
return importlib.import_module(f"app.{app}.train").main(args=args, resume_preempt=resume_preempt)
|
| 24 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 25 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py", line 780, in main
|
| 26 |
+
raise e
|
| 27 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py", line 765, in main
|
| 28 |
+
sample = next(loader)
|
| 29 |
+
^^^^^^^^^^^^
|
| 30 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 741, in __next__
|
| 31 |
+
data = self._next_data()
|
| 32 |
+
^^^^^^^^^^^^^^^^^
|
| 33 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 1524, in _next_data
|
| 34 |
+
idx, data = self._get_data()
|
| 35 |
+
^^^^^^^^^^^^^^^^
|
| 36 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 1478, in _get_data
|
| 37 |
+
raise RuntimeError("Pin memory thread exited unexpectedly")
|
| 38 |
+
RuntimeError: Pin memory thread exited unexpectedly
|
inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/crash_rank1.log
ADDED
|
@@ -0,0 +1,38 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Rank 1 crashed at epoch 53, itr 79
|
| 2 |
+
Error: Pin memory thread exited unexpectedly
|
| 3 |
+
|
| 4 |
+
Traceback (most recent call last):
|
| 5 |
+
File "<frozen runpy>", line 198, in _run_module_as_main
|
| 6 |
+
File "<frozen runpy>", line 88, in _run_code
|
| 7 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/_submit.py", line 11, in <module>
|
| 8 |
+
submitit_main()
|
| 9 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 76, in submitit_main
|
| 10 |
+
process_job(args.folder)
|
| 11 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 69, in process_job
|
| 12 |
+
raise error
|
| 13 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 55, in process_job
|
| 14 |
+
result = delayed.result()
|
| 15 |
+
^^^^^^^^^^^^^^^^
|
| 16 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/utils.py", line 137, in result
|
| 17 |
+
self._result = self.function(*self.args, **self.kwargs)
|
| 18 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 19 |
+
File "/leonardo/home/userexternal/dcanezil/vd/app/main_distributed.py", line 120, in __call__
|
| 20 |
+
"Value is YAML-parsed (ints/strings/lists). Repeatable. Applied before submit, so it covers "
|
| 21 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 22 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/scaffold.py", line 17, in main
|
| 23 |
+
return importlib.import_module(f"app.{app}.train").main(args=args, resume_preempt=resume_preempt)
|
| 24 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 25 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py", line 780, in main
|
| 26 |
+
raise e
|
| 27 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py", line 765, in main
|
| 28 |
+
sample = next(loader)
|
| 29 |
+
^^^^^^^^^^^^
|
| 30 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 741, in __next__
|
| 31 |
+
data = self._next_data()
|
| 32 |
+
^^^^^^^^^^^^^^^^^
|
| 33 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 1524, in _next_data
|
| 34 |
+
idx, data = self._get_data()
|
| 35 |
+
^^^^^^^^^^^^^^^^
|
| 36 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 1478, in _get_data
|
| 37 |
+
raise RuntimeError("Pin memory thread exited unexpectedly")
|
| 38 |
+
RuntimeError: Pin memory thread exited unexpectedly
|
inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/crash_rank10.log
ADDED
|
@@ -0,0 +1,38 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Rank 10 crashed at epoch 53, itr 79
|
| 2 |
+
Error: Pin memory thread exited unexpectedly
|
| 3 |
+
|
| 4 |
+
Traceback (most recent call last):
|
| 5 |
+
File "<frozen runpy>", line 198, in _run_module_as_main
|
| 6 |
+
File "<frozen runpy>", line 88, in _run_code
|
| 7 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/_submit.py", line 11, in <module>
|
| 8 |
+
submitit_main()
|
| 9 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 76, in submitit_main
|
| 10 |
+
process_job(args.folder)
|
| 11 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 69, in process_job
|
| 12 |
+
raise error
|
| 13 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 55, in process_job
|
| 14 |
+
result = delayed.result()
|
| 15 |
+
^^^^^^^^^^^^^^^^
|
| 16 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/utils.py", line 137, in result
|
| 17 |
+
self._result = self.function(*self.args, **self.kwargs)
|
| 18 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 19 |
+
File "/leonardo/home/userexternal/dcanezil/vd/app/main_distributed.py", line 120, in __call__
|
| 20 |
+
"Value is YAML-parsed (ints/strings/lists). Repeatable. Applied before submit, so it covers "
|
| 21 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 22 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/scaffold.py", line 17, in main
|
| 23 |
+
return importlib.import_module(f"app.{app}.train").main(args=args, resume_preempt=resume_preempt)
|
| 24 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 25 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py", line 780, in main
|
| 26 |
+
raise e
|
| 27 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py", line 765, in main
|
| 28 |
+
sample = next(loader)
|
| 29 |
+
^^^^^^^^^^^^
|
| 30 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 741, in __next__
|
| 31 |
+
data = self._next_data()
|
| 32 |
+
^^^^^^^^^^^^^^^^^
|
| 33 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 1524, in _next_data
|
| 34 |
+
idx, data = self._get_data()
|
| 35 |
+
^^^^^^^^^^^^^^^^
|
| 36 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 1478, in _get_data
|
| 37 |
+
raise RuntimeError("Pin memory thread exited unexpectedly")
|
| 38 |
+
RuntimeError: Pin memory thread exited unexpectedly
|
inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/crash_rank11.log
ADDED
|
@@ -0,0 +1,38 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Rank 11 crashed at epoch 53, itr 79
|
| 2 |
+
Error: Pin memory thread exited unexpectedly
|
| 3 |
+
|
| 4 |
+
Traceback (most recent call last):
|
| 5 |
+
File "<frozen runpy>", line 198, in _run_module_as_main
|
| 6 |
+
File "<frozen runpy>", line 88, in _run_code
|
| 7 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/_submit.py", line 11, in <module>
|
| 8 |
+
submitit_main()
|
| 9 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 76, in submitit_main
|
| 10 |
+
process_job(args.folder)
|
| 11 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 69, in process_job
|
| 12 |
+
raise error
|
| 13 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 55, in process_job
|
| 14 |
+
result = delayed.result()
|
| 15 |
+
^^^^^^^^^^^^^^^^
|
| 16 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/utils.py", line 137, in result
|
| 17 |
+
self._result = self.function(*self.args, **self.kwargs)
|
| 18 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 19 |
+
File "/leonardo/home/userexternal/dcanezil/vd/app/main_distributed.py", line 120, in __call__
|
| 20 |
+
"Value is YAML-parsed (ints/strings/lists). Repeatable. Applied before submit, so it covers "
|
| 21 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 22 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/scaffold.py", line 17, in main
|
| 23 |
+
return importlib.import_module(f"app.{app}.train").main(args=args, resume_preempt=resume_preempt)
|
| 24 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 25 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py", line 780, in main
|
| 26 |
+
raise e
|
| 27 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py", line 765, in main
|
| 28 |
+
sample = next(loader)
|
| 29 |
+
^^^^^^^^^^^^
|
| 30 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 741, in __next__
|
| 31 |
+
data = self._next_data()
|
| 32 |
+
^^^^^^^^^^^^^^^^^
|
| 33 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 1524, in _next_data
|
| 34 |
+
idx, data = self._get_data()
|
| 35 |
+
^^^^^^^^^^^^^^^^
|
| 36 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 1478, in _get_data
|
| 37 |
+
raise RuntimeError("Pin memory thread exited unexpectedly")
|
| 38 |
+
RuntimeError: Pin memory thread exited unexpectedly
|
inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/crash_rank12.log
ADDED
|
@@ -0,0 +1,38 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Rank 12 crashed at epoch 53, itr 79
|
| 2 |
+
Error: Pin memory thread exited unexpectedly
|
| 3 |
+
|
| 4 |
+
Traceback (most recent call last):
|
| 5 |
+
File "<frozen runpy>", line 198, in _run_module_as_main
|
| 6 |
+
File "<frozen runpy>", line 88, in _run_code
|
| 7 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/_submit.py", line 11, in <module>
|
| 8 |
+
submitit_main()
|
| 9 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 76, in submitit_main
|
| 10 |
+
process_job(args.folder)
|
| 11 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 69, in process_job
|
| 12 |
+
raise error
|
| 13 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 55, in process_job
|
| 14 |
+
result = delayed.result()
|
| 15 |
+
^^^^^^^^^^^^^^^^
|
| 16 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/utils.py", line 137, in result
|
| 17 |
+
self._result = self.function(*self.args, **self.kwargs)
|
| 18 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 19 |
+
File "/leonardo/home/userexternal/dcanezil/vd/app/main_distributed.py", line 120, in __call__
|
| 20 |
+
"Value is YAML-parsed (ints/strings/lists). Repeatable. Applied before submit, so it covers "
|
| 21 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 22 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/scaffold.py", line 17, in main
|
| 23 |
+
return importlib.import_module(f"app.{app}.train").main(args=args, resume_preempt=resume_preempt)
|
| 24 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 25 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py", line 780, in main
|
| 26 |
+
raise e
|
| 27 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py", line 765, in main
|
| 28 |
+
sample = next(loader)
|
| 29 |
+
^^^^^^^^^^^^
|
| 30 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 741, in __next__
|
| 31 |
+
data = self._next_data()
|
| 32 |
+
^^^^^^^^^^^^^^^^^
|
| 33 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 1524, in _next_data
|
| 34 |
+
idx, data = self._get_data()
|
| 35 |
+
^^^^^^^^^^^^^^^^
|
| 36 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 1478, in _get_data
|
| 37 |
+
raise RuntimeError("Pin memory thread exited unexpectedly")
|
| 38 |
+
RuntimeError: Pin memory thread exited unexpectedly
|
inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/crash_rank13.log
ADDED
|
@@ -0,0 +1,38 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Rank 13 crashed at epoch 53, itr 79
|
| 2 |
+
Error: Pin memory thread exited unexpectedly
|
| 3 |
+
|
| 4 |
+
Traceback (most recent call last):
|
| 5 |
+
File "<frozen runpy>", line 198, in _run_module_as_main
|
| 6 |
+
File "<frozen runpy>", line 88, in _run_code
|
| 7 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/_submit.py", line 11, in <module>
|
| 8 |
+
submitit_main()
|
| 9 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 76, in submitit_main
|
| 10 |
+
process_job(args.folder)
|
| 11 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 69, in process_job
|
| 12 |
+
raise error
|
| 13 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 55, in process_job
|
| 14 |
+
result = delayed.result()
|
| 15 |
+
^^^^^^^^^^^^^^^^
|
| 16 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/utils.py", line 137, in result
|
| 17 |
+
self._result = self.function(*self.args, **self.kwargs)
|
| 18 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 19 |
+
File "/leonardo/home/userexternal/dcanezil/vd/app/main_distributed.py", line 120, in __call__
|
| 20 |
+
"Value is YAML-parsed (ints/strings/lists). Repeatable. Applied before submit, so it covers "
|
| 21 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 22 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/scaffold.py", line 17, in main
|
| 23 |
+
return importlib.import_module(f"app.{app}.train").main(args=args, resume_preempt=resume_preempt)
|
| 24 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 25 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py", line 780, in main
|
| 26 |
+
raise e
|
| 27 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py", line 765, in main
|
| 28 |
+
sample = next(loader)
|
| 29 |
+
^^^^^^^^^^^^
|
| 30 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 741, in __next__
|
| 31 |
+
data = self._next_data()
|
| 32 |
+
^^^^^^^^^^^^^^^^^
|
| 33 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 1524, in _next_data
|
| 34 |
+
idx, data = self._get_data()
|
| 35 |
+
^^^^^^^^^^^^^^^^
|
| 36 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 1478, in _get_data
|
| 37 |
+
raise RuntimeError("Pin memory thread exited unexpectedly")
|
| 38 |
+
RuntimeError: Pin memory thread exited unexpectedly
|
inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/crash_rank15.log
ADDED
|
@@ -0,0 +1,38 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Rank 15 crashed at epoch 53, itr 79
|
| 2 |
+
Error: Pin memory thread exited unexpectedly
|
| 3 |
+
|
| 4 |
+
Traceback (most recent call last):
|
| 5 |
+
File "<frozen runpy>", line 198, in _run_module_as_main
|
| 6 |
+
File "<frozen runpy>", line 88, in _run_code
|
| 7 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/_submit.py", line 11, in <module>
|
| 8 |
+
submitit_main()
|
| 9 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 76, in submitit_main
|
| 10 |
+
process_job(args.folder)
|
| 11 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 69, in process_job
|
| 12 |
+
raise error
|
| 13 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 55, in process_job
|
| 14 |
+
result = delayed.result()
|
| 15 |
+
^^^^^^^^^^^^^^^^
|
| 16 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/utils.py", line 137, in result
|
| 17 |
+
self._result = self.function(*self.args, **self.kwargs)
|
| 18 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 19 |
+
File "/leonardo/home/userexternal/dcanezil/vd/app/main_distributed.py", line 120, in __call__
|
| 20 |
+
"Value is YAML-parsed (ints/strings/lists). Repeatable. Applied before submit, so it covers "
|
| 21 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 22 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/scaffold.py", line 17, in main
|
| 23 |
+
return importlib.import_module(f"app.{app}.train").main(args=args, resume_preempt=resume_preempt)
|
| 24 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 25 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py", line 780, in main
|
| 26 |
+
raise e
|
| 27 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py", line 765, in main
|
| 28 |
+
sample = next(loader)
|
| 29 |
+
^^^^^^^^^^^^
|
| 30 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 741, in __next__
|
| 31 |
+
data = self._next_data()
|
| 32 |
+
^^^^^^^^^^^^^^^^^
|
| 33 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 1524, in _next_data
|
| 34 |
+
idx, data = self._get_data()
|
| 35 |
+
^^^^^^^^^^^^^^^^
|
| 36 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 1478, in _get_data
|
| 37 |
+
raise RuntimeError("Pin memory thread exited unexpectedly")
|
| 38 |
+
RuntimeError: Pin memory thread exited unexpectedly
|
inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/crash_rank16.log
ADDED
|
@@ -0,0 +1,38 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Rank 16 crashed at epoch 53, itr 79
|
| 2 |
+
Error: Pin memory thread exited unexpectedly
|
| 3 |
+
|
| 4 |
+
Traceback (most recent call last):
|
| 5 |
+
File "<frozen runpy>", line 198, in _run_module_as_main
|
| 6 |
+
File "<frozen runpy>", line 88, in _run_code
|
| 7 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/_submit.py", line 11, in <module>
|
| 8 |
+
submitit_main()
|
| 9 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 76, in submitit_main
|
| 10 |
+
process_job(args.folder)
|
| 11 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 69, in process_job
|
| 12 |
+
raise error
|
| 13 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 55, in process_job
|
| 14 |
+
result = delayed.result()
|
| 15 |
+
^^^^^^^^^^^^^^^^
|
| 16 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/utils.py", line 137, in result
|
| 17 |
+
self._result = self.function(*self.args, **self.kwargs)
|
| 18 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 19 |
+
File "/leonardo/home/userexternal/dcanezil/vd/app/main_distributed.py", line 120, in __call__
|
| 20 |
+
"Value is YAML-parsed (ints/strings/lists). Repeatable. Applied before submit, so it covers "
|
| 21 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 22 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/scaffold.py", line 17, in main
|
| 23 |
+
return importlib.import_module(f"app.{app}.train").main(args=args, resume_preempt=resume_preempt)
|
| 24 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 25 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py", line 780, in main
|
| 26 |
+
raise e
|
| 27 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py", line 765, in main
|
| 28 |
+
sample = next(loader)
|
| 29 |
+
^^^^^^^^^^^^
|
| 30 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 741, in __next__
|
| 31 |
+
data = self._next_data()
|
| 32 |
+
^^^^^^^^^^^^^^^^^
|
| 33 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 1524, in _next_data
|
| 34 |
+
idx, data = self._get_data()
|
| 35 |
+
^^^^^^^^^^^^^^^^
|
| 36 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 1478, in _get_data
|
| 37 |
+
raise RuntimeError("Pin memory thread exited unexpectedly")
|
| 38 |
+
RuntimeError: Pin memory thread exited unexpectedly
|
inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/crash_rank17.log
ADDED
|
@@ -0,0 +1,38 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Rank 17 crashed at epoch 53, itr 79
|
| 2 |
+
Error: Pin memory thread exited unexpectedly
|
| 3 |
+
|
| 4 |
+
Traceback (most recent call last):
|
| 5 |
+
File "<frozen runpy>", line 198, in _run_module_as_main
|
| 6 |
+
File "<frozen runpy>", line 88, in _run_code
|
| 7 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/_submit.py", line 11, in <module>
|
| 8 |
+
submitit_main()
|
| 9 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 76, in submitit_main
|
| 10 |
+
process_job(args.folder)
|
| 11 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 69, in process_job
|
| 12 |
+
raise error
|
| 13 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 55, in process_job
|
| 14 |
+
result = delayed.result()
|
| 15 |
+
^^^^^^^^^^^^^^^^
|
| 16 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/utils.py", line 137, in result
|
| 17 |
+
self._result = self.function(*self.args, **self.kwargs)
|
| 18 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 19 |
+
File "/leonardo/home/userexternal/dcanezil/vd/app/main_distributed.py", line 120, in __call__
|
| 20 |
+
"Value is YAML-parsed (ints/strings/lists). Repeatable. Applied before submit, so it covers "
|
| 21 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 22 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/scaffold.py", line 17, in main
|
| 23 |
+
return importlib.import_module(f"app.{app}.train").main(args=args, resume_preempt=resume_preempt)
|
| 24 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 25 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py", line 780, in main
|
| 26 |
+
raise e
|
| 27 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py", line 765, in main
|
| 28 |
+
sample = next(loader)
|
| 29 |
+
^^^^^^^^^^^^
|
| 30 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 741, in __next__
|
| 31 |
+
data = self._next_data()
|
| 32 |
+
^^^^^^^^^^^^^^^^^
|
| 33 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 1524, in _next_data
|
| 34 |
+
idx, data = self._get_data()
|
| 35 |
+
^^^^^^^^^^^^^^^^
|
| 36 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 1478, in _get_data
|
| 37 |
+
raise RuntimeError("Pin memory thread exited unexpectedly")
|
| 38 |
+
RuntimeError: Pin memory thread exited unexpectedly
|
inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/crash_rank18.log
ADDED
|
@@ -0,0 +1,38 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Rank 18 crashed at epoch 53, itr 79
|
| 2 |
+
Error: Pin memory thread exited unexpectedly
|
| 3 |
+
|
| 4 |
+
Traceback (most recent call last):
|
| 5 |
+
File "<frozen runpy>", line 198, in _run_module_as_main
|
| 6 |
+
File "<frozen runpy>", line 88, in _run_code
|
| 7 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/_submit.py", line 11, in <module>
|
| 8 |
+
submitit_main()
|
| 9 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 76, in submitit_main
|
| 10 |
+
process_job(args.folder)
|
| 11 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 69, in process_job
|
| 12 |
+
raise error
|
| 13 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 55, in process_job
|
| 14 |
+
result = delayed.result()
|
| 15 |
+
^^^^^^^^^^^^^^^^
|
| 16 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/utils.py", line 137, in result
|
| 17 |
+
self._result = self.function(*self.args, **self.kwargs)
|
| 18 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 19 |
+
File "/leonardo/home/userexternal/dcanezil/vd/app/main_distributed.py", line 120, in __call__
|
| 20 |
+
"Value is YAML-parsed (ints/strings/lists). Repeatable. Applied before submit, so it covers "
|
| 21 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 22 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/scaffold.py", line 17, in main
|
| 23 |
+
return importlib.import_module(f"app.{app}.train").main(args=args, resume_preempt=resume_preempt)
|
| 24 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 25 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py", line 780, in main
|
| 26 |
+
raise e
|
| 27 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py", line 765, in main
|
| 28 |
+
sample = next(loader)
|
| 29 |
+
^^^^^^^^^^^^
|
| 30 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 741, in __next__
|
| 31 |
+
data = self._next_data()
|
| 32 |
+
^^^^^^^^^^^^^^^^^
|
| 33 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 1524, in _next_data
|
| 34 |
+
idx, data = self._get_data()
|
| 35 |
+
^^^^^^^^^^^^^^^^
|
| 36 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 1478, in _get_data
|
| 37 |
+
raise RuntimeError("Pin memory thread exited unexpectedly")
|
| 38 |
+
RuntimeError: Pin memory thread exited unexpectedly
|
inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/crash_rank19.log
ADDED
|
@@ -0,0 +1,38 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Rank 19 crashed at epoch 53, itr 79
|
| 2 |
+
Error: Pin memory thread exited unexpectedly
|
| 3 |
+
|
| 4 |
+
Traceback (most recent call last):
|
| 5 |
+
File "<frozen runpy>", line 198, in _run_module_as_main
|
| 6 |
+
File "<frozen runpy>", line 88, in _run_code
|
| 7 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/_submit.py", line 11, in <module>
|
| 8 |
+
submitit_main()
|
| 9 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 76, in submitit_main
|
| 10 |
+
process_job(args.folder)
|
| 11 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 69, in process_job
|
| 12 |
+
raise error
|
| 13 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 55, in process_job
|
| 14 |
+
result = delayed.result()
|
| 15 |
+
^^^^^^^^^^^^^^^^
|
| 16 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/utils.py", line 137, in result
|
| 17 |
+
self._result = self.function(*self.args, **self.kwargs)
|
| 18 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 19 |
+
File "/leonardo/home/userexternal/dcanezil/vd/app/main_distributed.py", line 120, in __call__
|
| 20 |
+
"Value is YAML-parsed (ints/strings/lists). Repeatable. Applied before submit, so it covers "
|
| 21 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 22 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/scaffold.py", line 17, in main
|
| 23 |
+
return importlib.import_module(f"app.{app}.train").main(args=args, resume_preempt=resume_preempt)
|
| 24 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 25 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py", line 780, in main
|
| 26 |
+
raise e
|
| 27 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py", line 765, in main
|
| 28 |
+
sample = next(loader)
|
| 29 |
+
^^^^^^^^^^^^
|
| 30 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 741, in __next__
|
| 31 |
+
data = self._next_data()
|
| 32 |
+
^^^^^^^^^^^^^^^^^
|
| 33 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 1524, in _next_data
|
| 34 |
+
idx, data = self._get_data()
|
| 35 |
+
^^^^^^^^^^^^^^^^
|
| 36 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 1478, in _get_data
|
| 37 |
+
raise RuntimeError("Pin memory thread exited unexpectedly")
|
| 38 |
+
RuntimeError: Pin memory thread exited unexpectedly
|
inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/crash_rank2.log
ADDED
|
@@ -0,0 +1,38 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Rank 2 crashed at epoch 53, itr 79
|
| 2 |
+
Error: Pin memory thread exited unexpectedly
|
| 3 |
+
|
| 4 |
+
Traceback (most recent call last):
|
| 5 |
+
File "<frozen runpy>", line 198, in _run_module_as_main
|
| 6 |
+
File "<frozen runpy>", line 88, in _run_code
|
| 7 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/_submit.py", line 11, in <module>
|
| 8 |
+
submitit_main()
|
| 9 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 76, in submitit_main
|
| 10 |
+
process_job(args.folder)
|
| 11 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 69, in process_job
|
| 12 |
+
raise error
|
| 13 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 55, in process_job
|
| 14 |
+
result = delayed.result()
|
| 15 |
+
^^^^^^^^^^^^^^^^
|
| 16 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/utils.py", line 137, in result
|
| 17 |
+
self._result = self.function(*self.args, **self.kwargs)
|
| 18 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 19 |
+
File "/leonardo/home/userexternal/dcanezil/vd/app/main_distributed.py", line 120, in __call__
|
| 20 |
+
"Value is YAML-parsed (ints/strings/lists). Repeatable. Applied before submit, so it covers "
|
| 21 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 22 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/scaffold.py", line 17, in main
|
| 23 |
+
return importlib.import_module(f"app.{app}.train").main(args=args, resume_preempt=resume_preempt)
|
| 24 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 25 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py", line 780, in main
|
| 26 |
+
raise e
|
| 27 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py", line 765, in main
|
| 28 |
+
sample = next(loader)
|
| 29 |
+
^^^^^^^^^^^^
|
| 30 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 741, in __next__
|
| 31 |
+
data = self._next_data()
|
| 32 |
+
^^^^^^^^^^^^^^^^^
|
| 33 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 1524, in _next_data
|
| 34 |
+
idx, data = self._get_data()
|
| 35 |
+
^^^^^^^^^^^^^^^^
|
| 36 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 1478, in _get_data
|
| 37 |
+
raise RuntimeError("Pin memory thread exited unexpectedly")
|
| 38 |
+
RuntimeError: Pin memory thread exited unexpectedly
|
inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/crash_rank20.log
ADDED
|
@@ -0,0 +1,38 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Rank 20 crashed at epoch 53, itr 79
|
| 2 |
+
Error: Pin memory thread exited unexpectedly
|
| 3 |
+
|
| 4 |
+
Traceback (most recent call last):
|
| 5 |
+
File "<frozen runpy>", line 198, in _run_module_as_main
|
| 6 |
+
File "<frozen runpy>", line 88, in _run_code
|
| 7 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/_submit.py", line 11, in <module>
|
| 8 |
+
submitit_main()
|
| 9 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 76, in submitit_main
|
| 10 |
+
process_job(args.folder)
|
| 11 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 69, in process_job
|
| 12 |
+
raise error
|
| 13 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 55, in process_job
|
| 14 |
+
result = delayed.result()
|
| 15 |
+
^^^^^^^^^^^^^^^^
|
| 16 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/utils.py", line 137, in result
|
| 17 |
+
self._result = self.function(*self.args, **self.kwargs)
|
| 18 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 19 |
+
File "/leonardo/home/userexternal/dcanezil/vd/app/main_distributed.py", line 120, in __call__
|
| 20 |
+
"Value is YAML-parsed (ints/strings/lists). Repeatable. Applied before submit, so it covers "
|
| 21 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 22 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/scaffold.py", line 17, in main
|
| 23 |
+
return importlib.import_module(f"app.{app}.train").main(args=args, resume_preempt=resume_preempt)
|
| 24 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 25 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py", line 780, in main
|
| 26 |
+
raise e
|
| 27 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py", line 765, in main
|
| 28 |
+
sample = next(loader)
|
| 29 |
+
^^^^^^^^^^^^
|
| 30 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 741, in __next__
|
| 31 |
+
data = self._next_data()
|
| 32 |
+
^^^^^^^^^^^^^^^^^
|
| 33 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 1524, in _next_data
|
| 34 |
+
idx, data = self._get_data()
|
| 35 |
+
^^^^^^^^^^^^^^^^
|
| 36 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 1478, in _get_data
|
| 37 |
+
raise RuntimeError("Pin memory thread exited unexpectedly")
|
| 38 |
+
RuntimeError: Pin memory thread exited unexpectedly
|
inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/crash_rank21.log
ADDED
|
@@ -0,0 +1,38 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Rank 21 crashed at epoch 53, itr 79
|
| 2 |
+
Error: Pin memory thread exited unexpectedly
|
| 3 |
+
|
| 4 |
+
Traceback (most recent call last):
|
| 5 |
+
File "<frozen runpy>", line 198, in _run_module_as_main
|
| 6 |
+
File "<frozen runpy>", line 88, in _run_code
|
| 7 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/_submit.py", line 11, in <module>
|
| 8 |
+
submitit_main()
|
| 9 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 76, in submitit_main
|
| 10 |
+
process_job(args.folder)
|
| 11 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 69, in process_job
|
| 12 |
+
raise error
|
| 13 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 55, in process_job
|
| 14 |
+
result = delayed.result()
|
| 15 |
+
^^^^^^^^^^^^^^^^
|
| 16 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/utils.py", line 137, in result
|
| 17 |
+
self._result = self.function(*self.args, **self.kwargs)
|
| 18 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 19 |
+
File "/leonardo/home/userexternal/dcanezil/vd/app/main_distributed.py", line 120, in __call__
|
| 20 |
+
"Value is YAML-parsed (ints/strings/lists). Repeatable. Applied before submit, so it covers "
|
| 21 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 22 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/scaffold.py", line 17, in main
|
| 23 |
+
return importlib.import_module(f"app.{app}.train").main(args=args, resume_preempt=resume_preempt)
|
| 24 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 25 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py", line 780, in main
|
| 26 |
+
raise e
|
| 27 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py", line 765, in main
|
| 28 |
+
sample = next(loader)
|
| 29 |
+
^^^^^^^^^^^^
|
| 30 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 741, in __next__
|
| 31 |
+
data = self._next_data()
|
| 32 |
+
^^^^^^^^^^^^^^^^^
|
| 33 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 1524, in _next_data
|
| 34 |
+
idx, data = self._get_data()
|
| 35 |
+
^^^^^^^^^^^^^^^^
|
| 36 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 1478, in _get_data
|
| 37 |
+
raise RuntimeError("Pin memory thread exited unexpectedly")
|
| 38 |
+
RuntimeError: Pin memory thread exited unexpectedly
|
inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/crash_rank22.log
ADDED
|
@@ -0,0 +1,38 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Rank 22 crashed at epoch 53, itr 79
|
| 2 |
+
Error: Pin memory thread exited unexpectedly
|
| 3 |
+
|
| 4 |
+
Traceback (most recent call last):
|
| 5 |
+
File "<frozen runpy>", line 198, in _run_module_as_main
|
| 6 |
+
File "<frozen runpy>", line 88, in _run_code
|
| 7 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/_submit.py", line 11, in <module>
|
| 8 |
+
submitit_main()
|
| 9 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 76, in submitit_main
|
| 10 |
+
process_job(args.folder)
|
| 11 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 69, in process_job
|
| 12 |
+
raise error
|
| 13 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 55, in process_job
|
| 14 |
+
result = delayed.result()
|
| 15 |
+
^^^^^^^^^^^^^^^^
|
| 16 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/utils.py", line 137, in result
|
| 17 |
+
self._result = self.function(*self.args, **self.kwargs)
|
| 18 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 19 |
+
File "/leonardo/home/userexternal/dcanezil/vd/app/main_distributed.py", line 120, in __call__
|
| 20 |
+
"Value is YAML-parsed (ints/strings/lists). Repeatable. Applied before submit, so it covers "
|
| 21 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 22 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/scaffold.py", line 17, in main
|
| 23 |
+
return importlib.import_module(f"app.{app}.train").main(args=args, resume_preempt=resume_preempt)
|
| 24 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 25 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py", line 780, in main
|
| 26 |
+
raise e
|
| 27 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py", line 765, in main
|
| 28 |
+
sample = next(loader)
|
| 29 |
+
^^^^^^^^^^^^
|
| 30 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 741, in __next__
|
| 31 |
+
data = self._next_data()
|
| 32 |
+
^^^^^^^^^^^^^^^^^
|
| 33 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 1524, in _next_data
|
| 34 |
+
idx, data = self._get_data()
|
| 35 |
+
^^^^^^^^^^^^^^^^
|
| 36 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 1478, in _get_data
|
| 37 |
+
raise RuntimeError("Pin memory thread exited unexpectedly")
|
| 38 |
+
RuntimeError: Pin memory thread exited unexpectedly
|
inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/crash_rank23.log
ADDED
|
@@ -0,0 +1,38 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Rank 23 crashed at epoch 53, itr 79
|
| 2 |
+
Error: Pin memory thread exited unexpectedly
|
| 3 |
+
|
| 4 |
+
Traceback (most recent call last):
|
| 5 |
+
File "<frozen runpy>", line 198, in _run_module_as_main
|
| 6 |
+
File "<frozen runpy>", line 88, in _run_code
|
| 7 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/_submit.py", line 11, in <module>
|
| 8 |
+
submitit_main()
|
| 9 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 76, in submitit_main
|
| 10 |
+
process_job(args.folder)
|
| 11 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 69, in process_job
|
| 12 |
+
raise error
|
| 13 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 55, in process_job
|
| 14 |
+
result = delayed.result()
|
| 15 |
+
^^^^^^^^^^^^^^^^
|
| 16 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/utils.py", line 137, in result
|
| 17 |
+
self._result = self.function(*self.args, **self.kwargs)
|
| 18 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 19 |
+
File "/leonardo/home/userexternal/dcanezil/vd/app/main_distributed.py", line 120, in __call__
|
| 20 |
+
"Value is YAML-parsed (ints/strings/lists). Repeatable. Applied before submit, so it covers "
|
| 21 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 22 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/scaffold.py", line 17, in main
|
| 23 |
+
return importlib.import_module(f"app.{app}.train").main(args=args, resume_preempt=resume_preempt)
|
| 24 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 25 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py", line 780, in main
|
| 26 |
+
raise e
|
| 27 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py", line 765, in main
|
| 28 |
+
sample = next(loader)
|
| 29 |
+
^^^^^^^^^^^^
|
| 30 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 741, in __next__
|
| 31 |
+
data = self._next_data()
|
| 32 |
+
^^^^^^^^^^^^^^^^^
|
| 33 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 1524, in _next_data
|
| 34 |
+
idx, data = self._get_data()
|
| 35 |
+
^^^^^^^^^^^^^^^^
|
| 36 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 1478, in _get_data
|
| 37 |
+
raise RuntimeError("Pin memory thread exited unexpectedly")
|
| 38 |
+
RuntimeError: Pin memory thread exited unexpectedly
|
inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/crash_rank24.log
ADDED
|
@@ -0,0 +1,38 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Rank 24 crashed at epoch 53, itr 79
|
| 2 |
+
Error: Pin memory thread exited unexpectedly
|
| 3 |
+
|
| 4 |
+
Traceback (most recent call last):
|
| 5 |
+
File "<frozen runpy>", line 198, in _run_module_as_main
|
| 6 |
+
File "<frozen runpy>", line 88, in _run_code
|
| 7 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/_submit.py", line 11, in <module>
|
| 8 |
+
submitit_main()
|
| 9 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 76, in submitit_main
|
| 10 |
+
process_job(args.folder)
|
| 11 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 69, in process_job
|
| 12 |
+
raise error
|
| 13 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 55, in process_job
|
| 14 |
+
result = delayed.result()
|
| 15 |
+
^^^^^^^^^^^^^^^^
|
| 16 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/utils.py", line 137, in result
|
| 17 |
+
self._result = self.function(*self.args, **self.kwargs)
|
| 18 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 19 |
+
File "/leonardo/home/userexternal/dcanezil/vd/app/main_distributed.py", line 120, in __call__
|
| 20 |
+
"Value is YAML-parsed (ints/strings/lists). Repeatable. Applied before submit, so it covers "
|
| 21 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 22 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/scaffold.py", line 17, in main
|
| 23 |
+
return importlib.import_module(f"app.{app}.train").main(args=args, resume_preempt=resume_preempt)
|
| 24 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 25 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py", line 780, in main
|
| 26 |
+
raise e
|
| 27 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py", line 765, in main
|
| 28 |
+
sample = next(loader)
|
| 29 |
+
^^^^^^^^^^^^
|
| 30 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 741, in __next__
|
| 31 |
+
data = self._next_data()
|
| 32 |
+
^^^^^^^^^^^^^^^^^
|
| 33 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 1524, in _next_data
|
| 34 |
+
idx, data = self._get_data()
|
| 35 |
+
^^^^^^^^^^^^^^^^
|
| 36 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 1478, in _get_data
|
| 37 |
+
raise RuntimeError("Pin memory thread exited unexpectedly")
|
| 38 |
+
RuntimeError: Pin memory thread exited unexpectedly
|
inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/crash_rank25.log
ADDED
|
@@ -0,0 +1,38 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Rank 25 crashed at epoch 53, itr 79
|
| 2 |
+
Error: Pin memory thread exited unexpectedly
|
| 3 |
+
|
| 4 |
+
Traceback (most recent call last):
|
| 5 |
+
File "<frozen runpy>", line 198, in _run_module_as_main
|
| 6 |
+
File "<frozen runpy>", line 88, in _run_code
|
| 7 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/_submit.py", line 11, in <module>
|
| 8 |
+
submitit_main()
|
| 9 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 76, in submitit_main
|
| 10 |
+
process_job(args.folder)
|
| 11 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 69, in process_job
|
| 12 |
+
raise error
|
| 13 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 55, in process_job
|
| 14 |
+
result = delayed.result()
|
| 15 |
+
^^^^^^^^^^^^^^^^
|
| 16 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/utils.py", line 137, in result
|
| 17 |
+
self._result = self.function(*self.args, **self.kwargs)
|
| 18 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 19 |
+
File "/leonardo/home/userexternal/dcanezil/vd/app/main_distributed.py", line 120, in __call__
|
| 20 |
+
"Value is YAML-parsed (ints/strings/lists). Repeatable. Applied before submit, so it covers "
|
| 21 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 22 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/scaffold.py", line 17, in main
|
| 23 |
+
return importlib.import_module(f"app.{app}.train").main(args=args, resume_preempt=resume_preempt)
|
| 24 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 25 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py", line 780, in main
|
| 26 |
+
raise e
|
| 27 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py", line 765, in main
|
| 28 |
+
sample = next(loader)
|
| 29 |
+
^^^^^^^^^^^^
|
| 30 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 741, in __next__
|
| 31 |
+
data = self._next_data()
|
| 32 |
+
^^^^^^^^^^^^^^^^^
|
| 33 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 1524, in _next_data
|
| 34 |
+
idx, data = self._get_data()
|
| 35 |
+
^^^^^^^^^^^^^^^^
|
| 36 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 1478, in _get_data
|
| 37 |
+
raise RuntimeError("Pin memory thread exited unexpectedly")
|
| 38 |
+
RuntimeError: Pin memory thread exited unexpectedly
|
inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/crash_rank26.log
ADDED
|
@@ -0,0 +1,38 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Rank 26 crashed at epoch 53, itr 79
|
| 2 |
+
Error: Pin memory thread exited unexpectedly
|
| 3 |
+
|
| 4 |
+
Traceback (most recent call last):
|
| 5 |
+
File "<frozen runpy>", line 198, in _run_module_as_main
|
| 6 |
+
File "<frozen runpy>", line 88, in _run_code
|
| 7 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/_submit.py", line 11, in <module>
|
| 8 |
+
submitit_main()
|
| 9 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 76, in submitit_main
|
| 10 |
+
process_job(args.folder)
|
| 11 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 69, in process_job
|
| 12 |
+
raise error
|
| 13 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 55, in process_job
|
| 14 |
+
result = delayed.result()
|
| 15 |
+
^^^^^^^^^^^^^^^^
|
| 16 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/utils.py", line 137, in result
|
| 17 |
+
self._result = self.function(*self.args, **self.kwargs)
|
| 18 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 19 |
+
File "/leonardo/home/userexternal/dcanezil/vd/app/main_distributed.py", line 120, in __call__
|
| 20 |
+
"Value is YAML-parsed (ints/strings/lists). Repeatable. Applied before submit, so it covers "
|
| 21 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 22 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/scaffold.py", line 17, in main
|
| 23 |
+
return importlib.import_module(f"app.{app}.train").main(args=args, resume_preempt=resume_preempt)
|
| 24 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 25 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py", line 780, in main
|
| 26 |
+
raise e
|
| 27 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py", line 765, in main
|
| 28 |
+
sample = next(loader)
|
| 29 |
+
^^^^^^^^^^^^
|
| 30 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 741, in __next__
|
| 31 |
+
data = self._next_data()
|
| 32 |
+
^^^^^^^^^^^^^^^^^
|
| 33 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 1524, in _next_data
|
| 34 |
+
idx, data = self._get_data()
|
| 35 |
+
^^^^^^^^^^^^^^^^
|
| 36 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 1478, in _get_data
|
| 37 |
+
raise RuntimeError("Pin memory thread exited unexpectedly")
|
| 38 |
+
RuntimeError: Pin memory thread exited unexpectedly
|
inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/crash_rank27.log
ADDED
|
@@ -0,0 +1,38 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Rank 27 crashed at epoch 53, itr 79
|
| 2 |
+
Error: Pin memory thread exited unexpectedly
|
| 3 |
+
|
| 4 |
+
Traceback (most recent call last):
|
| 5 |
+
File "<frozen runpy>", line 198, in _run_module_as_main
|
| 6 |
+
File "<frozen runpy>", line 88, in _run_code
|
| 7 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/_submit.py", line 11, in <module>
|
| 8 |
+
submitit_main()
|
| 9 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 76, in submitit_main
|
| 10 |
+
process_job(args.folder)
|
| 11 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 69, in process_job
|
| 12 |
+
raise error
|
| 13 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 55, in process_job
|
| 14 |
+
result = delayed.result()
|
| 15 |
+
^^^^^^^^^^^^^^^^
|
| 16 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/utils.py", line 137, in result
|
| 17 |
+
self._result = self.function(*self.args, **self.kwargs)
|
| 18 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 19 |
+
File "/leonardo/home/userexternal/dcanezil/vd/app/main_distributed.py", line 120, in __call__
|
| 20 |
+
"Value is YAML-parsed (ints/strings/lists). Repeatable. Applied before submit, so it covers "
|
| 21 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 22 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/scaffold.py", line 17, in main
|
| 23 |
+
return importlib.import_module(f"app.{app}.train").main(args=args, resume_preempt=resume_preempt)
|
| 24 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 25 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py", line 780, in main
|
| 26 |
+
raise e
|
| 27 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py", line 765, in main
|
| 28 |
+
sample = next(loader)
|
| 29 |
+
^^^^^^^^^^^^
|
| 30 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 741, in __next__
|
| 31 |
+
data = self._next_data()
|
| 32 |
+
^^^^^^^^^^^^^^^^^
|
| 33 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 1524, in _next_data
|
| 34 |
+
idx, data = self._get_data()
|
| 35 |
+
^^^^^^^^^^^^^^^^
|
| 36 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 1478, in _get_data
|
| 37 |
+
raise RuntimeError("Pin memory thread exited unexpectedly")
|
| 38 |
+
RuntimeError: Pin memory thread exited unexpectedly
|
inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/crash_rank28.log
ADDED
|
@@ -0,0 +1,38 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Rank 28 crashed at epoch 53, itr 79
|
| 2 |
+
Error: Pin memory thread exited unexpectedly
|
| 3 |
+
|
| 4 |
+
Traceback (most recent call last):
|
| 5 |
+
File "<frozen runpy>", line 198, in _run_module_as_main
|
| 6 |
+
File "<frozen runpy>", line 88, in _run_code
|
| 7 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/_submit.py", line 11, in <module>
|
| 8 |
+
submitit_main()
|
| 9 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 76, in submitit_main
|
| 10 |
+
process_job(args.folder)
|
| 11 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 69, in process_job
|
| 12 |
+
raise error
|
| 13 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 55, in process_job
|
| 14 |
+
result = delayed.result()
|
| 15 |
+
^^^^^^^^^^^^^^^^
|
| 16 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/utils.py", line 137, in result
|
| 17 |
+
self._result = self.function(*self.args, **self.kwargs)
|
| 18 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 19 |
+
File "/leonardo/home/userexternal/dcanezil/vd/app/main_distributed.py", line 120, in __call__
|
| 20 |
+
"Value is YAML-parsed (ints/strings/lists). Repeatable. Applied before submit, so it covers "
|
| 21 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 22 |
+
File "/leonardo/home/userexternal/dcanezil/vd/app/scaffold.py", line 17, in main
|
| 23 |
+
return importlib.import_module(f"app.{app}.train").main(args=args, resume_preempt=resume_preempt)
|
| 24 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 25 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py", line 780, in main
|
| 26 |
+
raise e
|
| 27 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py", line 765, in main
|
| 28 |
+
sample = next(loader)
|
| 29 |
+
^^^^^^^^^^^^
|
| 30 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 741, in __next__
|
| 31 |
+
data = self._next_data()
|
| 32 |
+
^^^^^^^^^^^^^^^^^
|
| 33 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 1524, in _next_data
|
| 34 |
+
idx, data = self._get_data()
|
| 35 |
+
^^^^^^^^^^^^^^^^
|
| 36 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 1478, in _get_data
|
| 37 |
+
raise RuntimeError("Pin memory thread exited unexpectedly")
|
| 38 |
+
RuntimeError: Pin memory thread exited unexpectedly
|
inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/crash_rank29.log
ADDED
|
@@ -0,0 +1,38 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Rank 29 crashed at epoch 53, itr 79
|
| 2 |
+
Error: Pin memory thread exited unexpectedly
|
| 3 |
+
|
| 4 |
+
Traceback (most recent call last):
|
| 5 |
+
File "<frozen runpy>", line 198, in _run_module_as_main
|
| 6 |
+
File "<frozen runpy>", line 88, in _run_code
|
| 7 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/_submit.py", line 11, in <module>
|
| 8 |
+
submitit_main()
|
| 9 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 76, in submitit_main
|
| 10 |
+
process_job(args.folder)
|
| 11 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 69, in process_job
|
| 12 |
+
raise error
|
| 13 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 55, in process_job
|
| 14 |
+
result = delayed.result()
|
| 15 |
+
^^^^^^^^^^^^^^^^
|
| 16 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/utils.py", line 137, in result
|
| 17 |
+
self._result = self.function(*self.args, **self.kwargs)
|
| 18 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 19 |
+
File "/leonardo/home/userexternal/dcanezil/vd/app/main_distributed.py", line 120, in __call__
|
| 20 |
+
"Value is YAML-parsed (ints/strings/lists). Repeatable. Applied before submit, so it covers "
|
| 21 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 22 |
+
File "/leonardo/home/userexternal/dcanezil/vd/app/scaffold.py", line 17, in main
|
| 23 |
+
return importlib.import_module(f"app.{app}.train").main(args=args, resume_preempt=resume_preempt)
|
| 24 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 25 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py", line 780, in main
|
| 26 |
+
raise e
|
| 27 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py", line 765, in main
|
| 28 |
+
sample = next(loader)
|
| 29 |
+
^^^^^^^^^^^^
|
| 30 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 741, in __next__
|
| 31 |
+
data = self._next_data()
|
| 32 |
+
^^^^^^^^^^^^^^^^^
|
| 33 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 1524, in _next_data
|
| 34 |
+
idx, data = self._get_data()
|
| 35 |
+
^^^^^^^^^^^^^^^^
|
| 36 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 1478, in _get_data
|
| 37 |
+
raise RuntimeError("Pin memory thread exited unexpectedly")
|
| 38 |
+
RuntimeError: Pin memory thread exited unexpectedly
|
inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/crash_rank3.log
ADDED
|
@@ -0,0 +1,38 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Rank 3 crashed at epoch 53, itr 79
|
| 2 |
+
Error: Pin memory thread exited unexpectedly
|
| 3 |
+
|
| 4 |
+
Traceback (most recent call last):
|
| 5 |
+
File "<frozen runpy>", line 198, in _run_module_as_main
|
| 6 |
+
File "<frozen runpy>", line 88, in _run_code
|
| 7 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/_submit.py", line 11, in <module>
|
| 8 |
+
submitit_main()
|
| 9 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 76, in submitit_main
|
| 10 |
+
process_job(args.folder)
|
| 11 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 69, in process_job
|
| 12 |
+
raise error
|
| 13 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 55, in process_job
|
| 14 |
+
result = delayed.result()
|
| 15 |
+
^^^^^^^^^^^^^^^^
|
| 16 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/utils.py", line 137, in result
|
| 17 |
+
self._result = self.function(*self.args, **self.kwargs)
|
| 18 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 19 |
+
File "/leonardo/home/userexternal/dcanezil/vd/app/main_distributed.py", line 120, in __call__
|
| 20 |
+
"Value is YAML-parsed (ints/strings/lists). Repeatable. Applied before submit, so it covers "
|
| 21 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 22 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/scaffold.py", line 17, in main
|
| 23 |
+
return importlib.import_module(f"app.{app}.train").main(args=args, resume_preempt=resume_preempt)
|
| 24 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 25 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py", line 780, in main
|
| 26 |
+
raise e
|
| 27 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py", line 765, in main
|
| 28 |
+
sample = next(loader)
|
| 29 |
+
^^^^^^^^^^^^
|
| 30 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 741, in __next__
|
| 31 |
+
data = self._next_data()
|
| 32 |
+
^^^^^^^^^^^^^^^^^
|
| 33 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 1524, in _next_data
|
| 34 |
+
idx, data = self._get_data()
|
| 35 |
+
^^^^^^^^^^^^^^^^
|
| 36 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 1478, in _get_data
|
| 37 |
+
raise RuntimeError("Pin memory thread exited unexpectedly")
|
| 38 |
+
RuntimeError: Pin memory thread exited unexpectedly
|
inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/crash_rank30.log
ADDED
|
@@ -0,0 +1,38 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Rank 30 crashed at epoch 53, itr 79
|
| 2 |
+
Error: Pin memory thread exited unexpectedly
|
| 3 |
+
|
| 4 |
+
Traceback (most recent call last):
|
| 5 |
+
File "<frozen runpy>", line 198, in _run_module_as_main
|
| 6 |
+
File "<frozen runpy>", line 88, in _run_code
|
| 7 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/_submit.py", line 11, in <module>
|
| 8 |
+
submitit_main()
|
| 9 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 76, in submitit_main
|
| 10 |
+
process_job(args.folder)
|
| 11 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 69, in process_job
|
| 12 |
+
raise error
|
| 13 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 55, in process_job
|
| 14 |
+
result = delayed.result()
|
| 15 |
+
^^^^^^^^^^^^^^^^
|
| 16 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/utils.py", line 137, in result
|
| 17 |
+
self._result = self.function(*self.args, **self.kwargs)
|
| 18 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 19 |
+
File "/leonardo/home/userexternal/dcanezil/vd/app/main_distributed.py", line 120, in __call__
|
| 20 |
+
"Value is YAML-parsed (ints/strings/lists). Repeatable. Applied before submit, so it covers "
|
| 21 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 22 |
+
File "/leonardo/home/userexternal/dcanezil/vd/app/scaffold.py", line 17, in main
|
| 23 |
+
return importlib.import_module(f"app.{app}.train").main(args=args, resume_preempt=resume_preempt)
|
| 24 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 25 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py", line 780, in main
|
| 26 |
+
raise e
|
| 27 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py", line 765, in main
|
| 28 |
+
sample = next(loader)
|
| 29 |
+
^^^^^^^^^^^^
|
| 30 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 741, in __next__
|
| 31 |
+
data = self._next_data()
|
| 32 |
+
^^^^^^^^^^^^^^^^^
|
| 33 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 1524, in _next_data
|
| 34 |
+
idx, data = self._get_data()
|
| 35 |
+
^^^^^^^^^^^^^^^^
|
| 36 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 1478, in _get_data
|
| 37 |
+
raise RuntimeError("Pin memory thread exited unexpectedly")
|
| 38 |
+
RuntimeError: Pin memory thread exited unexpectedly
|
inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/crash_rank31.log
ADDED
|
@@ -0,0 +1,38 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Rank 31 crashed at epoch 53, itr 79
|
| 2 |
+
Error: Pin memory thread exited unexpectedly
|
| 3 |
+
|
| 4 |
+
Traceback (most recent call last):
|
| 5 |
+
File "<frozen runpy>", line 198, in _run_module_as_main
|
| 6 |
+
File "<frozen runpy>", line 88, in _run_code
|
| 7 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/_submit.py", line 11, in <module>
|
| 8 |
+
submitit_main()
|
| 9 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 76, in submitit_main
|
| 10 |
+
process_job(args.folder)
|
| 11 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 69, in process_job
|
| 12 |
+
raise error
|
| 13 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 55, in process_job
|
| 14 |
+
result = delayed.result()
|
| 15 |
+
^^^^^^^^^^^^^^^^
|
| 16 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/utils.py", line 137, in result
|
| 17 |
+
self._result = self.function(*self.args, **self.kwargs)
|
| 18 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 19 |
+
File "/leonardo/home/userexternal/dcanezil/vd/app/main_distributed.py", line 120, in __call__
|
| 20 |
+
"Value is YAML-parsed (ints/strings/lists). Repeatable. Applied before submit, so it covers "
|
| 21 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 22 |
+
File "/leonardo/home/userexternal/dcanezil/vd/app/scaffold.py", line 17, in main
|
| 23 |
+
return importlib.import_module(f"app.{app}.train").main(args=args, resume_preempt=resume_preempt)
|
| 24 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 25 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py", line 780, in main
|
| 26 |
+
raise e
|
| 27 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py", line 765, in main
|
| 28 |
+
sample = next(loader)
|
| 29 |
+
^^^^^^^^^^^^
|
| 30 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 741, in __next__
|
| 31 |
+
data = self._next_data()
|
| 32 |
+
^^^^^^^^^^^^^^^^^
|
| 33 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 1524, in _next_data
|
| 34 |
+
idx, data = self._get_data()
|
| 35 |
+
^^^^^^^^^^^^^^^^
|
| 36 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 1478, in _get_data
|
| 37 |
+
raise RuntimeError("Pin memory thread exited unexpectedly")
|
| 38 |
+
RuntimeError: Pin memory thread exited unexpectedly
|
inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/crash_rank5.log
ADDED
|
@@ -0,0 +1,38 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Rank 5 crashed at epoch 53, itr 79
|
| 2 |
+
Error: Pin memory thread exited unexpectedly
|
| 3 |
+
|
| 4 |
+
Traceback (most recent call last):
|
| 5 |
+
File "<frozen runpy>", line 198, in _run_module_as_main
|
| 6 |
+
File "<frozen runpy>", line 88, in _run_code
|
| 7 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/_submit.py", line 11, in <module>
|
| 8 |
+
submitit_main()
|
| 9 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 76, in submitit_main
|
| 10 |
+
process_job(args.folder)
|
| 11 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 69, in process_job
|
| 12 |
+
raise error
|
| 13 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 55, in process_job
|
| 14 |
+
result = delayed.result()
|
| 15 |
+
^^^^^^^^^^^^^^^^
|
| 16 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/utils.py", line 137, in result
|
| 17 |
+
self._result = self.function(*self.args, **self.kwargs)
|
| 18 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 19 |
+
File "/leonardo/home/userexternal/dcanezil/vd/app/main_distributed.py", line 120, in __call__
|
| 20 |
+
"Value is YAML-parsed (ints/strings/lists). Repeatable. Applied before submit, so it covers "
|
| 21 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 22 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/scaffold.py", line 17, in main
|
| 23 |
+
return importlib.import_module(f"app.{app}.train").main(args=args, resume_preempt=resume_preempt)
|
| 24 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 25 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py", line 780, in main
|
| 26 |
+
raise e
|
| 27 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py", line 765, in main
|
| 28 |
+
sample = next(loader)
|
| 29 |
+
^^^^^^^^^^^^
|
| 30 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 741, in __next__
|
| 31 |
+
data = self._next_data()
|
| 32 |
+
^^^^^^^^^^^^^^^^^
|
| 33 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 1524, in _next_data
|
| 34 |
+
idx, data = self._get_data()
|
| 35 |
+
^^^^^^^^^^^^^^^^
|
| 36 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 1478, in _get_data
|
| 37 |
+
raise RuntimeError("Pin memory thread exited unexpectedly")
|
| 38 |
+
RuntimeError: Pin memory thread exited unexpectedly
|
inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/crash_rank6.log
ADDED
|
@@ -0,0 +1,38 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Rank 6 crashed at epoch 53, itr 79
|
| 2 |
+
Error: Pin memory thread exited unexpectedly
|
| 3 |
+
|
| 4 |
+
Traceback (most recent call last):
|
| 5 |
+
File "<frozen runpy>", line 198, in _run_module_as_main
|
| 6 |
+
File "<frozen runpy>", line 88, in _run_code
|
| 7 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/_submit.py", line 11, in <module>
|
| 8 |
+
submitit_main()
|
| 9 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 76, in submitit_main
|
| 10 |
+
process_job(args.folder)
|
| 11 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 69, in process_job
|
| 12 |
+
raise error
|
| 13 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 55, in process_job
|
| 14 |
+
result = delayed.result()
|
| 15 |
+
^^^^^^^^^^^^^^^^
|
| 16 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/utils.py", line 137, in result
|
| 17 |
+
self._result = self.function(*self.args, **self.kwargs)
|
| 18 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 19 |
+
File "/leonardo/home/userexternal/dcanezil/vd/app/main_distributed.py", line 120, in __call__
|
| 20 |
+
"Value is YAML-parsed (ints/strings/lists). Repeatable. Applied before submit, so it covers "
|
| 21 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 22 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/scaffold.py", line 17, in main
|
| 23 |
+
return importlib.import_module(f"app.{app}.train").main(args=args, resume_preempt=resume_preempt)
|
| 24 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 25 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py", line 780, in main
|
| 26 |
+
raise e
|
| 27 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py", line 765, in main
|
| 28 |
+
sample = next(loader)
|
| 29 |
+
^^^^^^^^^^^^
|
| 30 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 741, in __next__
|
| 31 |
+
data = self._next_data()
|
| 32 |
+
^^^^^^^^^^^^^^^^^
|
| 33 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 1524, in _next_data
|
| 34 |
+
idx, data = self._get_data()
|
| 35 |
+
^^^^^^^^^^^^^^^^
|
| 36 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 1478, in _get_data
|
| 37 |
+
raise RuntimeError("Pin memory thread exited unexpectedly")
|
| 38 |
+
RuntimeError: Pin memory thread exited unexpectedly
|
inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/crash_rank7.log
ADDED
|
@@ -0,0 +1,38 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Rank 7 crashed at epoch 53, itr 79
|
| 2 |
+
Error: Pin memory thread exited unexpectedly
|
| 3 |
+
|
| 4 |
+
Traceback (most recent call last):
|
| 5 |
+
File "<frozen runpy>", line 198, in _run_module_as_main
|
| 6 |
+
File "<frozen runpy>", line 88, in _run_code
|
| 7 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/_submit.py", line 11, in <module>
|
| 8 |
+
submitit_main()
|
| 9 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 76, in submitit_main
|
| 10 |
+
process_job(args.folder)
|
| 11 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 69, in process_job
|
| 12 |
+
raise error
|
| 13 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 55, in process_job
|
| 14 |
+
result = delayed.result()
|
| 15 |
+
^^^^^^^^^^^^^^^^
|
| 16 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/utils.py", line 137, in result
|
| 17 |
+
self._result = self.function(*self.args, **self.kwargs)
|
| 18 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 19 |
+
File "/leonardo/home/userexternal/dcanezil/vd/app/main_distributed.py", line 120, in __call__
|
| 20 |
+
"Value is YAML-parsed (ints/strings/lists). Repeatable. Applied before submit, so it covers "
|
| 21 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 22 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/scaffold.py", line 17, in main
|
| 23 |
+
return importlib.import_module(f"app.{app}.train").main(args=args, resume_preempt=resume_preempt)
|
| 24 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 25 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py", line 780, in main
|
| 26 |
+
raise e
|
| 27 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py", line 765, in main
|
| 28 |
+
sample = next(loader)
|
| 29 |
+
^^^^^^^^^^^^
|
| 30 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 741, in __next__
|
| 31 |
+
data = self._next_data()
|
| 32 |
+
^^^^^^^^^^^^^^^^^
|
| 33 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 1524, in _next_data
|
| 34 |
+
idx, data = self._get_data()
|
| 35 |
+
^^^^^^^^^^^^^^^^
|
| 36 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 1478, in _get_data
|
| 37 |
+
raise RuntimeError("Pin memory thread exited unexpectedly")
|
| 38 |
+
RuntimeError: Pin memory thread exited unexpectedly
|
inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/crash_rank8.log
ADDED
|
@@ -0,0 +1,38 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Rank 8 crashed at epoch 53, itr 79
|
| 2 |
+
Error: Pin memory thread exited unexpectedly
|
| 3 |
+
|
| 4 |
+
Traceback (most recent call last):
|
| 5 |
+
File "<frozen runpy>", line 198, in _run_module_as_main
|
| 6 |
+
File "<frozen runpy>", line 88, in _run_code
|
| 7 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/_submit.py", line 11, in <module>
|
| 8 |
+
submitit_main()
|
| 9 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 76, in submitit_main
|
| 10 |
+
process_job(args.folder)
|
| 11 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 69, in process_job
|
| 12 |
+
raise error
|
| 13 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 55, in process_job
|
| 14 |
+
result = delayed.result()
|
| 15 |
+
^^^^^^^^^^^^^^^^
|
| 16 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/utils.py", line 137, in result
|
| 17 |
+
self._result = self.function(*self.args, **self.kwargs)
|
| 18 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 19 |
+
File "/leonardo/home/userexternal/dcanezil/vd/app/main_distributed.py", line 120, in __call__
|
| 20 |
+
"Value is YAML-parsed (ints/strings/lists). Repeatable. Applied before submit, so it covers "
|
| 21 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 22 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/scaffold.py", line 17, in main
|
| 23 |
+
return importlib.import_module(f"app.{app}.train").main(args=args, resume_preempt=resume_preempt)
|
| 24 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 25 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py", line 780, in main
|
| 26 |
+
raise e
|
| 27 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py", line 765, in main
|
| 28 |
+
sample = next(loader)
|
| 29 |
+
^^^^^^^^^^^^
|
| 30 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 741, in __next__
|
| 31 |
+
data = self._next_data()
|
| 32 |
+
^^^^^^^^^^^^^^^^^
|
| 33 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 1524, in _next_data
|
| 34 |
+
idx, data = self._get_data()
|
| 35 |
+
^^^^^^^^^^^^^^^^
|
| 36 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 1478, in _get_data
|
| 37 |
+
raise RuntimeError("Pin memory thread exited unexpectedly")
|
| 38 |
+
RuntimeError: Pin memory thread exited unexpectedly
|
inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/crash_rank9.log
ADDED
|
@@ -0,0 +1,38 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Rank 9 crashed at epoch 53, itr 79
|
| 2 |
+
Error: Pin memory thread exited unexpectedly
|
| 3 |
+
|
| 4 |
+
Traceback (most recent call last):
|
| 5 |
+
File "<frozen runpy>", line 198, in _run_module_as_main
|
| 6 |
+
File "<frozen runpy>", line 88, in _run_code
|
| 7 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/_submit.py", line 11, in <module>
|
| 8 |
+
submitit_main()
|
| 9 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 76, in submitit_main
|
| 10 |
+
process_job(args.folder)
|
| 11 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 69, in process_job
|
| 12 |
+
raise error
|
| 13 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/submission.py", line 55, in process_job
|
| 14 |
+
result = delayed.result()
|
| 15 |
+
^^^^^^^^^^^^^^^^
|
| 16 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/submitit/core/utils.py", line 137, in result
|
| 17 |
+
self._result = self.function(*self.args, **self.kwargs)
|
| 18 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 19 |
+
File "/leonardo/home/userexternal/dcanezil/vd/app/main_distributed.py", line 120, in __call__
|
| 20 |
+
"Value is YAML-parsed (ints/strings/lists). Repeatable. Applied before submit, so it covers "
|
| 21 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 22 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/scaffold.py", line 17, in main
|
| 23 |
+
return importlib.import_module(f"app.{app}.train").main(args=args, resume_preempt=resume_preempt)
|
| 24 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 25 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py", line 780, in main
|
| 26 |
+
raise e
|
| 27 |
+
File "/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py", line 765, in main
|
| 28 |
+
sample = next(loader)
|
| 29 |
+
^^^^^^^^^^^^
|
| 30 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 741, in __next__
|
| 31 |
+
data = self._next_data()
|
| 32 |
+
^^^^^^^^^^^^^^^^^
|
| 33 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 1524, in _next_data
|
| 34 |
+
idx, data = self._get_data()
|
| 35 |
+
^^^^^^^^^^^^^^^^
|
| 36 |
+
File "/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/utils/data/dataloader.py", line 1478, in _get_data
|
| 37 |
+
raise RuntimeError("Pin memory thread exited unexpectedly")
|
| 38 |
+
RuntimeError: Pin memory thread exited unexpectedly
|
inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/git-info.txt
ADDED
|
@@ -0,0 +1,2 @@
|
|
|
|
|
|
|
|
|
|
| 1 |
+
branch: main
|
| 2 |
+
commit: 18cf3c47810c8cfc209d702faf4a9e88e65e19b7
|
inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/job_45423709/45423709_0_log.err
ADDED
|
@@ -0,0 +1,97 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
|
| 2 |
+
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
|
| 3 |
+
wandb: Tracking run with wandb version 0.23.1
|
| 4 |
+
wandb: W&B syncing is set to `offline` in this directory. Run `wandb online` or set WANDB_MODE=online to enable cloud syncing.
|
| 5 |
+
wandb: Run data is saved locally in /leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/wandb/offline-run-20260611_051623-0optsj93
|
| 6 |
+
/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/utils.py:888: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
|
| 7 |
+
scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
|
| 8 |
+
/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py:983: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
|
| 9 |
+
with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
|
| 10 |
+
/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
|
| 11 |
+
return func(*args, **kwargs)
|
| 12 |
+
/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
|
| 13 |
+
return func(*args, **kwargs)
|
| 14 |
+
/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py:983: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
|
| 15 |
+
with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
|
| 16 |
+
/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
|
| 17 |
+
return func(*args, **kwargs)
|
| 18 |
+
/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py:983: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
|
| 19 |
+
with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
|
| 20 |
+
/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
|
| 21 |
+
return func(*args, **kwargs)
|
| 22 |
+
/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py:983: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
|
| 23 |
+
with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
|
| 24 |
+
/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
|
| 25 |
+
return func(*args, **kwargs)
|
| 26 |
+
/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py:983: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
|
| 27 |
+
with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
|
| 28 |
+
/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
|
| 29 |
+
return func(*args, **kwargs)
|
| 30 |
+
/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py:983: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
|
| 31 |
+
with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
|
| 32 |
+
/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
|
| 33 |
+
return func(*args, **kwargs)
|
| 34 |
+
/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py:983: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
|
| 35 |
+
with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
|
| 36 |
+
[rank0]:[E611 13:45:39.787221200 ProcessGroupNCCL.cpp:1825] [PG ID 0 PG GUID 0(default_pg) Rank 0] Observed flight recorder dump signal from another rank via TCPStore.
|
| 37 |
+
[rank0]:[E611 13:45:39.788622908 ProcessGroupNCCL.cpp:1890] [PG ID 0 PG GUID 0(default_pg) Rank 0] Received a dump signal due to a collective timeout from rank 20 and we will try our best to dump the debug info. Last enqueued NCCL work: 410232, last completed NCCL work: 410189.This is most likely caused by incorrect usages of collectives, e.g., wrong sizes used across ranks, the order of collectives is not same for all ranks or the scheduled collective, for some reason, didn't run. Additionally, this can be caused by GIL deadlock or other reasons such as network errors or bugs in the communications library (e.g. NCCL), etc.
|
| 38 |
+
[rank0]:[E611 13:45:39.790346999 ProcessGroupNCCL.cpp:1606] [PG ID 0 PG GUID 0(default_pg) Rank 0] ProcessGroupNCCL preparing to dump debug info. Include stack trace: 1, only active collectives: 0
|
| 39 |
+
[rank0]:[E611 13:45:41.235225001 ProcessGroupNCCL.cpp:688] [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=410190, OpType=ALLREDUCE, NumelIn=395008, NumelOut=395008, Timeout(ms)=900000) ran for 900068 milliseconds before timing out.
|
| 40 |
+
[rank0]:[E611 13:45:41.243231531 ProcessGroupNCCL.cpp:2277] [PG ID 0 PG GUID 0(default_pg) Rank 0] failure detected by watchdog at work sequence id: 410190 PG status: last enqueued work: 410232, last completed work: 410189
|
| 41 |
+
[rank0]:[E611 13:45:41.243253239 ProcessGroupNCCL.cpp:735] Stack trace of the failed collective not found, potentially because FlightRecorder is disabled. You can enable it by setting TORCH_NCCL_TRACE_BUFFER_SIZE to a non-zero value.
|
| 42 |
+
[rank0]:[E611 13:45:54.790244052 ProcessGroupNCCL.cpp:1367] [PG ID 0 PG GUID 0(default_pg) Rank 0] Future for Flight recorder dump in heartbeatMonitor timed out after 15000 ms
|
| 43 |
+
[rank0]:[E611 13:45:54.790310023 ProcessGroupNCCL.cpp:1606] [PG ID 0 PG GUID 0(default_pg) Rank 0] ProcessGroupNCCL preparing to dump debug info. Include stack trace: 0, only active collectives: 0
|
| 44 |
+
[rank0]:[E611 13:46:09.790358485 ProcessGroupNCCL.cpp:1367] [PG ID 0 PG GUID 0(default_pg) Rank 0] Future for Flight recorder dump in heartbeatMonitor timed out after 15000 ms
|
| 45 |
+
srun: error: lrdn1287: task 29: Aborted
|
| 46 |
+
srun: error: lrdn1257: task 19: Aborted
|
| 47 |
+
[rank0]:[E611 13:46:41.243348514 ProcessGroupNCCL.cpp:749] [Rank 0] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
|
| 48 |
+
[rank0]:[E611 13:46:41.243388064 ProcessGroupNCCL.cpp:763] [Rank 0] To avoid data inconsistency, we are taking the entire process down.
|
| 49 |
+
[rank0]:[E611 13:46:41.270463819 ProcessGroupNCCL.cpp:2093] [PG ID 0 PG GUID 0(default_pg) Rank 0] Process group watchdog thread terminated with exception: [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=410190, OpType=ALLREDUCE, NumelIn=395008, NumelOut=395008, Timeout(ms)=900000) ran for 900068 milliseconds before timing out.
|
| 50 |
+
Exception raised from checkTimeout at /pytorch/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:691 (most recent call first):
|
| 51 |
+
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1495355a8fdd in /leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/lib/libc10.so)
|
| 52 |
+
frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x26a (0x14953681ad7a in /leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/lib/libtorch_cuda.so)
|
| 53 |
+
frame #2: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x16a1 (0x14953681f261 in /leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/lib/libtorch_cuda.so)
|
| 54 |
+
frame #3: c10d::ProcessGroupNCCL::Watchdog::run() + 0x105 (0x1495368205a5 in /leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/lib/libtorch_cuda.so)
|
| 55 |
+
frame #4: <unknown function> + 0xe0a53 (0x1496948e2a53 in /leonardo/prod/spack/06/install/0.22/linux-rhel8-icelake/gcc-8.5.0/gcc-12.2.0-lkcazt4letxjj4s7nlhzryoyivevsatz/lib64/libstdc++.so.6)
|
| 56 |
+
frame #5: <unknown function> + 0x81ca (0x1496a960a1ca in /lib64/libpthread.so.0)
|
| 57 |
+
frame #6: clone + 0x43 (0x1496a8aece73 in /lib64/libc.so.6)
|
| 58 |
+
|
| 59 |
+
terminate called after throwing an instance of 'c10::DistBackendError'
|
| 60 |
+
what(): [PG ID 0 PG GUID 0(default_pg) Rank 0] Process group watchdog thread terminated with exception: [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=410190, OpType=ALLREDUCE, NumelIn=395008, NumelOut=395008, Timeout(ms)=900000) ran for 900068 milliseconds before timing out.
|
| 61 |
+
Exception raised from checkTimeout at /pytorch/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:691 (most recent call first):
|
| 62 |
+
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1495355a8fdd in /leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/lib/libc10.so)
|
| 63 |
+
frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x26a (0x14953681ad7a in /leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/lib/libtorch_cuda.so)
|
| 64 |
+
frame #2: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x16a1 (0x14953681f261 in /leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/lib/libtorch_cuda.so)
|
| 65 |
+
frame #3: c10d::ProcessGroupNCCL::Watchdog::run() + 0x105 (0x1495368205a5 in /leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/lib/libtorch_cuda.so)
|
| 66 |
+
frame #4: <unknown function> + 0xe0a53 (0x1496948e2a53 in /leonardo/prod/spack/06/install/0.22/linux-rhel8-icelake/gcc-8.5.0/gcc-12.2.0-lkcazt4letxjj4s7nlhzryoyivevsatz/lib64/libstdc++.so.6)
|
| 67 |
+
frame #5: <unknown function> + 0x81ca (0x1496a960a1ca in /lib64/libpthread.so.0)
|
| 68 |
+
frame #6: clone + 0x43 (0x1496a8aece73 in /lib64/libc.so.6)
|
| 69 |
+
|
| 70 |
+
Exception raised from run at /pytorch/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:2099 (most recent call first):
|
| 71 |
+
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x1495355a8fdd in /leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/lib/libc10.so)
|
| 72 |
+
frame #1: <unknown function> + 0x9bbbc8 (0x149536086bc8 in /leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/torch/lib/libtorch_cuda.so)
|
| 73 |
+
frame #2: <unknown function> + 0xe0a53 (0x1496948e2a53 in /leonardo/prod/spack/06/install/0.22/linux-rhel8-icelake/gcc-8.5.0/gcc-12.2.0-lkcazt4letxjj4s7nlhzryoyivevsatz/lib64/libstdc++.so.6)
|
| 74 |
+
frame #3: <unknown function> + 0x81ca (0x1496a960a1ca in /lib64/libpthread.so.0)
|
| 75 |
+
frame #4: clone + 0x43 (0x1496a8aece73 in /lib64/libc.so.6)
|
| 76 |
+
|
| 77 |
+
srun: error: lrdn1260: task 20: Aborted
|
| 78 |
+
srun: error: lrdn1287: task 31: Aborted
|
| 79 |
+
srun: error: lrdn1183: task 0: Aborted
|
| 80 |
+
/usr/lib64/python3.11/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 34 leaked semaphore objects to clean up at shutdown
|
| 81 |
+
warnings.warn('resource_tracker: There appear to be %d '
|
| 82 |
+
srun: error: lrdn1287: task 30: Aborted
|
| 83 |
+
srun: error: lrdn1270: task 25: Aborted
|
| 84 |
+
srun: error: lrdn1260: tasks 21,23: Aborted
|
| 85 |
+
srun: error: lrdn1270: task 26: Aborted
|
| 86 |
+
srun: error: lrdn1257: task 17: Aborted
|
| 87 |
+
srun: error: lrdn1287: task 28: Aborted
|
| 88 |
+
srun: error: lrdn1187: task 7: Aborted
|
| 89 |
+
srun: error: lrdn1227: task 11: Aborted
|
| 90 |
+
srun: error: lrdn1257: task 18: Aborted
|
| 91 |
+
srun: error: lrdn1260: task 22: Aborted
|
| 92 |
+
srun: error: lrdn1270: tasks 24,27: Aborted
|
| 93 |
+
srun: error: lrdn1257: task 16: Aborted
|
| 94 |
+
srun: error: lrdn1183: tasks 1-3: Aborted
|
| 95 |
+
srun: error: lrdn1187: tasks 4-6: Aborted
|
| 96 |
+
srun: error: lrdn1227: tasks 8-10: Aborted
|
| 97 |
+
srun: error: lrdn1244: tasks 12-15: Aborted
|
inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/job_45423709/45423709_0_log.out
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/job_45423709/45423709_10_log.err
ADDED
|
@@ -0,0 +1,13 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
|
| 2 |
+
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
|
| 3 |
+
/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/utils.py:888: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
|
| 4 |
+
scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
|
| 5 |
+
/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py:983: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
|
| 6 |
+
with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
|
| 7 |
+
[rank10]:[E611 13:45:39.710551653 ProcessGroupNCCL.cpp:1825] [PG ID 0 PG GUID 0(default_pg) Rank 10] Observed flight recorder dump signal from another rank via TCPStore.
|
| 8 |
+
[rank10]:[E611 13:45:39.712340346 ProcessGroupNCCL.cpp:1890] [PG ID 0 PG GUID 0(default_pg) Rank 10] Received a dump signal due to a collective timeout from rank 20 and we will try our best to dump the debug info. Last enqueued NCCL work: 410189, last completed NCCL work: 410188.This is most likely caused by incorrect usages of collectives, e.g., wrong sizes used across ranks, the order of collectives is not same for all ranks or the scheduled collective, for some reason, didn't run. Additionally, this can be caused by GIL deadlock or other reasons such as network errors or bugs in the communications library (e.g. NCCL), etc.
|
| 9 |
+
[rank10]:[E611 13:45:39.714642766 ProcessGroupNCCL.cpp:1606] [PG ID 0 PG GUID 0(default_pg) Rank 10] ProcessGroupNCCL preparing to dump debug info. Include stack trace: 1, only active collectives: 0
|
| 10 |
+
[rank10]:[E611 13:45:54.713534624 ProcessGroupNCCL.cpp:1367] [PG ID 0 PG GUID 0(default_pg) Rank 10] Future for Flight recorder dump in heartbeatMonitor timed out after 15000 ms
|
| 11 |
+
[rank10]:[E611 13:45:54.713611101 ProcessGroupNCCL.cpp:1606] [PG ID 0 PG GUID 0(default_pg) Rank 10] ProcessGroupNCCL preparing to dump debug info. Include stack trace: 0, only active collectives: 0
|
| 12 |
+
[rank10]:[E611 13:46:09.713675849 ProcessGroupNCCL.cpp:1367] [PG ID 0 PG GUID 0(default_pg) Rank 10] Future for Flight recorder dump in heartbeatMonitor timed out after 15000 ms
|
| 13 |
+
[rank10]:[F611 13:54:09.715073282 ProcessGroupNCCL.cpp:1631] [PG ID 0 PG GUID 0(default_pg) Rank 10] [PG ID 0 PG GUID 0(default_pg) Rank 10] Terminating the process after attempting to dump debug info, due to collective timeout or exception.
|
inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/job_45423709/45423709_10_log.out
ADDED
|
@@ -0,0 +1,208 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
submitit INFO (2026-06-11 05:16:06,427) - Starting with JobEnvironment(job_id=45423709, hostname=lrdn1227.leonardo.local, local_rank=2(4), node=2(8), global_rank=10(32))
|
| 2 |
+
submitit INFO (2026-06-11 05:16:06,428) - Loading pickle: /leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/job_45423709/45423709_submitted.pkl
|
| 3 |
+
INFO:root:loaded pretrain params...
|
| 4 |
+
{ 'app': 'vjepa',
|
| 5 |
+
'cpus_per_task': 8,
|
| 6 |
+
'data': { 'batch_size': 32,
|
| 7 |
+
'crop_size': 224,
|
| 8 |
+
'dataset_fpcs': [16, 16],
|
| 9 |
+
'dataset_type': 'VideoDataset',
|
| 10 |
+
'datasets': [ '/leonardo_scratch/fast/EUHPC_D35_082/data/k400/train.csv',
|
| 11 |
+
'/leonardo_scratch/fast/EUHPC_D35_082/data/ssv2/train.csv'],
|
| 12 |
+
'datasets_weights': [0.65, 0.35],
|
| 13 |
+
'fps': 4,
|
| 14 |
+
'num_workers': 8,
|
| 15 |
+
'patch_size': 14,
|
| 16 |
+
'persistent_workers': True,
|
| 17 |
+
'pin_mem': True,
|
| 18 |
+
'tubelet_size': 1},
|
| 19 |
+
'data_aug': { 'auto_augment': False,
|
| 20 |
+
'motion_shift': False,
|
| 21 |
+
'random_resize_aspect_ratio': [0.75, 1.35],
|
| 22 |
+
'random_resize_scale': [0.3, 1.0],
|
| 23 |
+
'reprob': 0.0},
|
| 24 |
+
'folder': '/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3',
|
| 25 |
+
'loss': {'loss_exp': 1.0},
|
| 26 |
+
'mask': [ { 'aspect_ratio': [0.75, 1.5],
|
| 27 |
+
'full_complement': False,
|
| 28 |
+
'max_keep': None,
|
| 29 |
+
'max_temporal_keep': 1.0,
|
| 30 |
+
'num_blocks': 8,
|
| 31 |
+
'spatial_scale': [0.15, 0.15],
|
| 32 |
+
'temporal_scale': [1.0, 1.0]},
|
| 33 |
+
{ 'aspect_ratio': [0.75, 1.5],
|
| 34 |
+
'full_complement': False,
|
| 35 |
+
'max_keep': None,
|
| 36 |
+
'max_temporal_keep': 1.0,
|
| 37 |
+
'num_blocks': 2,
|
| 38 |
+
'spatial_scale': [0.7, 0.7],
|
| 39 |
+
'temporal_scale': [1.0, 1.0]}],
|
| 40 |
+
'mem_per_gpu': '120G',
|
| 41 |
+
'meta': { 'dtype': 'bfloat16',
|
| 42 |
+
'knn_eval_epoch0': False,
|
| 43 |
+
'knn_eval_freq': 5,
|
| 44 |
+
'knn_eval_presets': [ { 'config': { 'batch_size': 8,
|
| 45 |
+
'dataset_train': '/leonardo_scratch/fast/EUHPC_D35_082/data/ssv2/train_coarse10.csv',
|
| 46 |
+
'dataset_val': '/leonardo_scratch/fast/EUHPC_D35_082/data/ssv2/val_coarse10.csv',
|
| 47 |
+
'eval_videos_per_class': 100,
|
| 48 |
+
'linear_probe': True,
|
| 49 |
+
'num_workers': 2,
|
| 50 |
+
'pool_type': 'temporal_concat',
|
| 51 |
+
'train_videos_per_class': 500},
|
| 52 |
+
'preset': 'ssv2_coarse10'}],
|
| 53 |
+
'load_checkpoint': True,
|
| 54 |
+
'read_checkpoint': None,
|
| 55 |
+
'save_every_freq': 5,
|
| 56 |
+
'seed': 239,
|
| 57 |
+
'use_sdpa': True,
|
| 58 |
+
'use_wandb': True,
|
| 59 |
+
'wandb_project': 'vjepa_ablation'},
|
| 60 |
+
'metrics': {'std': {}},
|
| 61 |
+
'model': { 'class_token': True,
|
| 62 |
+
'freeze_backbone': False,
|
| 63 |
+
'is_causal': True,
|
| 64 |
+
'model_name': 'vit_large_patch14_dinov2_lvd142m',
|
| 65 |
+
'pe_type': '2d',
|
| 66 |
+
'pred_depth': 12,
|
| 67 |
+
'pred_embed_dim': 384,
|
| 68 |
+
'pred_num_heads': 6,
|
| 69 |
+
'predictor': 'v2_cross',
|
| 70 |
+
'stem_type': '2d',
|
| 71 |
+
'target_embed_dim': 1024,
|
| 72 |
+
'target_kind': 'ema',
|
| 73 |
+
'uniform_power': True,
|
| 74 |
+
'use_activation_checkpointing': True,
|
| 75 |
+
'use_alibi': False,
|
| 76 |
+
'use_lora': False,
|
| 77 |
+
'use_mask_tokens': True,
|
| 78 |
+
'use_rope': False,
|
| 79 |
+
'use_sdpa': True,
|
| 80 |
+
'zero_init_mask_tokens': True},
|
| 81 |
+
'nodes': 8,
|
| 82 |
+
'optimization': { 'clip_grad': None,
|
| 83 |
+
'ema': [1.0, 0.99925],
|
| 84 |
+
'ema_hold_epochs': 0,
|
| 85 |
+
'ema_ramp_epochs': 30,
|
| 86 |
+
'encoder_layer_decay': 0.9,
|
| 87 |
+
'epochs': 60,
|
| 88 |
+
'final_lr': 0.0005,
|
| 89 |
+
'final_weight_decay': 0.04,
|
| 90 |
+
'ipe': 300,
|
| 91 |
+
'ipe_scale': 1.0,
|
| 92 |
+
'lr': 0.0005,
|
| 93 |
+
'predictor_lr_scale': 3.0,
|
| 94 |
+
'start_lr': 5e-05,
|
| 95 |
+
'warmup': 4,
|
| 96 |
+
'weight_decay': 0.04},
|
| 97 |
+
'tasks_per_node': 4}
|
| 98 |
+
INFO:root:Running pre-training of app: vjepa
|
| 99 |
+
[INFO ][2026-06-11 05:16:17][app.vjepa.train ][main ] which_dtype='bfloat16'
|
| 100 |
+
[INFO ][2026-06-11 05:16:17][app.vjepa.train ][main ] Disabling persistent_workers (incompatible with KNN eval)
|
| 101 |
+
[INFO ][2026-06-11 05:16:17][app.vjepa.train ][main ] NCCL_SOCKET_IFNAME=None
|
| 102 |
+
[INFO ][2026-06-11 05:16:18][app.vjepa.train ][main ] Initialized (rank/world-size) 10/32, tasks_per_node=4
|
| 103 |
+
[WARNING ][2026-06-11 05:16:25][experiments.stmodels.vision_transformers_v3][_build_dinov2_vit ] DINOv2 is an image model — fwd_chunk is unset so all frames will be processed jointly. Set fwd_chunk=1 for per-frame (standard) processing.
|
| 104 |
+
[INFO ][2026-06-11 05:16:40][experiments.stmodels.vision_transformers_v3][_build_dinov2_vit ] Loading pretrained weights for vit_large_patch14_dinov2_lvd142m from timm (vit_large_patch14_dinov2)
|
| 105 |
+
[INFO ][2026-06-11 05:16:53][timm.models._builder][load_pretrained ] Loading pretrained weights from Hugging Face hub (timm/vit_large_patch14_dinov2.lvd142m)
|
| 106 |
+
[INFO ][2026-06-11 05:16:53][timm.models._hub ][load_state_dict_from_hf ] [timm/vit_large_patch14_dinov2.lvd142m] Safe alternative available for 'pytorch_model.bin' (as 'model.safetensors'). Loading weights using safetensors.
|
| 107 |
+
[INFO ][2026-06-11 05:16:53][timm.layers.pos_embed][resample_abs_pos_embed ] Resized position embedding: (37, 37) to (16, 16).
|
| 108 |
+
[INFO ][2026-06-11 05:17:33][root ][init_video_model ] ViTMultiSeqWrapper(
|
| 109 |
+
(backbone): VisionTransformer(
|
| 110 |
+
(patch_embed): PatchEmbed3D(
|
| 111 |
+
(proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
|
| 112 |
+
)
|
| 113 |
+
(blocks): ModuleList(
|
| 114 |
+
(0-23): 24 x Block(
|
| 115 |
+
(norm1): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
|
| 116 |
+
(norm2): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
|
| 117 |
+
(drop_path1): Identity()
|
| 118 |
+
(drop_path2): Identity()
|
| 119 |
+
(attn): Attention(
|
| 120 |
+
(qkv): Linear(in_features=1024, out_features=3072, bias=True)
|
| 121 |
+
(attn_drop): Dropout(p=0.0, inplace=False)
|
| 122 |
+
(proj): Linear(in_features=1024, out_features=1024, bias=True)
|
| 123 |
+
(proj_drop): Dropout(p=0.0, inplace=False)
|
| 124 |
+
)
|
| 125 |
+
(mlp): MLP(
|
| 126 |
+
(fc1): Linear(in_features=1024, out_features=4096, bias=True)
|
| 127 |
+
(act): GELU(approximate='none')
|
| 128 |
+
(fc2): Linear(in_features=4096, out_features=1024, bias=True)
|
| 129 |
+
(drop): Dropout(p=0.0, inplace=False)
|
| 130 |
+
)
|
| 131 |
+
)
|
| 132 |
+
)
|
| 133 |
+
(norm): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
|
| 134 |
+
)
|
| 135 |
+
)
|
| 136 |
+
[INFO ][2026-06-11 05:17:33][root ][init_video_model ] VJEPAPredictorMultiSeqWrapper(
|
| 137 |
+
(backbone): PredictorV2(
|
| 138 |
+
(predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
|
| 139 |
+
(mask_tokens): ParameterList(
|
| 140 |
+
(0): Parameter containing: [torch.float32 of size 1x384 (cuda:0)]
|
| 141 |
+
(1): Parameter containing: [torch.float32 of size 1x384 (cuda:0)]
|
| 142 |
+
(2): Parameter containing: [torch.float32 of size 1x384 (cuda:0)]
|
| 143 |
+
(3): Parameter containing: [torch.float32 of size 1x384 (cuda:0)]
|
| 144 |
+
)
|
| 145 |
+
(predictor_blocks): ModuleList(
|
| 146 |
+
(0-11): 12 x Block(
|
| 147 |
+
(residual1): EfficientResidual(
|
| 148 |
+
(norm): LayerNorm((384,), eps=1e-05, elementwise_affine=True)
|
| 149 |
+
(fn): Attention(
|
| 150 |
+
(q_proj): Linear(in_features=384, out_features=384, bias=False)
|
| 151 |
+
(k_proj): Linear(in_features=384, out_features=384, bias=False)
|
| 152 |
+
(v_proj): Linear(in_features=384, out_features=384, bias=False)
|
| 153 |
+
(proj): Linear(in_features=384, out_features=384, bias=False)
|
| 154 |
+
(rope): Rope()
|
| 155 |
+
)
|
| 156 |
+
)
|
| 157 |
+
(residual2): EfficientResidual(
|
| 158 |
+
(norm): LayerNorm((384,), eps=1e-05, elementwise_affine=True)
|
| 159 |
+
(fn): MLP(
|
| 160 |
+
(fc1): Linear(in_features=384, out_features=1536, bias=False)
|
| 161 |
+
(act): GELU(approximate='none')
|
| 162 |
+
(fc2): Linear(in_features=1536, out_features=384, bias=False)
|
| 163 |
+
(drop): Dropout(p=0.0, inplace=False)
|
| 164 |
+
)
|
| 165 |
+
)
|
| 166 |
+
)
|
| 167 |
+
)
|
| 168 |
+
(predictor_norm): LayerNorm((384,), eps=1e-05, elementwise_affine=True)
|
| 169 |
+
(predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
|
| 170 |
+
)
|
| 171 |
+
)
|
| 172 |
+
[INFO ][2026-06-11 05:17:33][root ][init_video_model ] ViTMultiSeqWrapper(
|
| 173 |
+
(backbone): VisionTransformer(
|
| 174 |
+
(patch_embed): PatchEmbed3D(
|
| 175 |
+
(proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
|
| 176 |
+
)
|
| 177 |
+
(blocks): ModuleList(
|
| 178 |
+
(0-23): 24 x Block(
|
| 179 |
+
(norm1): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
|
| 180 |
+
(norm2): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
|
| 181 |
+
(drop_path1): Identity()
|
| 182 |
+
(drop_path2): Identity()
|
| 183 |
+
(attn): Attention(
|
| 184 |
+
(qkv): Linear(in_features=1024, out_features=3072, bias=True)
|
| 185 |
+
(attn_drop): Dropout(p=0.0, inplace=False)
|
| 186 |
+
(proj): Linear(in_features=1024, out_features=1024, bias=True)
|
| 187 |
+
(proj_drop): Dropout(p=0.0, inplace=False)
|
| 188 |
+
)
|
| 189 |
+
(mlp): MLP(
|
| 190 |
+
(fc1): Linear(in_features=1024, out_features=4096, bias=True)
|
| 191 |
+
(act): GELU(approximate='none')
|
| 192 |
+
(fc2): Linear(in_features=4096, out_features=1024, bias=True)
|
| 193 |
+
(drop): Dropout(p=0.0, inplace=False)
|
| 194 |
+
)
|
| 195 |
+
)
|
| 196 |
+
)
|
| 197 |
+
(norm): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
|
| 198 |
+
)
|
| 199 |
+
)
|
| 200 |
+
[INFO ][2026-06-11 05:17:33][root ][init_video_model ] Encoder number of parameters: 302964736
|
| 201 |
+
[INFO ][2026-06-11 05:17:33][root ][init_video_model ] Predictor number of parameters: 22042240
|
| 202 |
+
[INFO ][2026-06-11 05:17:33][root ][init_video_model ] Target encoder number of parameters: 302964736
|
| 203 |
+
[INFO ][2026-06-11 05:17:34][root ][make_videodataset ] VideoDataset dataset created
|
| 204 |
+
[INFO ][2026-06-11 05:17:34][WeightedSampler ][__init__ ] Using DistributedWeightedSampler with rank 10 / 32
|
| 205 |
+
[INFO ][2026-06-11 05:17:34][root ][make_videodataset ] VideoDataset unsupervised data loader created
|
| 206 |
+
[INFO ][2026-06-11 05:17:34][app.vjepa.train ][main ] iterations per epoch/dataset length: 300/392
|
| 207 |
+
[INFO ][2026-06-11 05:17:34][app.vjepa.train ][main ] Wrapping models in DDP (rank 10)...
|
| 208 |
+
[INFO ][2026-06-11 05:17:35][app.vjepa.train ][main ] Initializing loader...
|
inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/job_45423709/45423709_11_log.err
ADDED
|
@@ -0,0 +1,13 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
|
| 2 |
+
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
|
| 3 |
+
/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/utils.py:888: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
|
| 4 |
+
scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
|
| 5 |
+
/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py:983: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
|
| 6 |
+
with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
|
| 7 |
+
[rank11]:[E611 13:45:39.711227090 ProcessGroupNCCL.cpp:1825] [PG ID 0 PG GUID 0(default_pg) Rank 11] Observed flight recorder dump signal from another rank via TCPStore.
|
| 8 |
+
[rank11]:[E611 13:45:39.712347710 ProcessGroupNCCL.cpp:1890] [PG ID 0 PG GUID 0(default_pg) Rank 11] Received a dump signal due to a collective timeout from rank 20 and we will try our best to dump the debug info. Last enqueued NCCL work: 410188, last completed NCCL work: 410188.This is most likely caused by incorrect usages of collectives, e.g., wrong sizes used across ranks, the order of collectives is not same for all ranks or the scheduled collective, for some reason, didn't run. Additionally, this can be caused by GIL deadlock or other reasons such as network errors or bugs in the communications library (e.g. NCCL), etc.
|
| 9 |
+
[rank11]:[E611 13:45:39.714639647 ProcessGroupNCCL.cpp:1606] [PG ID 0 PG GUID 0(default_pg) Rank 11] ProcessGroupNCCL preparing to dump debug info. Include stack trace: 1, only active collectives: 0
|
| 10 |
+
[rank11]:[E611 13:45:54.713534159 ProcessGroupNCCL.cpp:1367] [PG ID 0 PG GUID 0(default_pg) Rank 11] Future for Flight recorder dump in heartbeatMonitor timed out after 15000 ms
|
| 11 |
+
[rank11]:[E611 13:45:54.713602853 ProcessGroupNCCL.cpp:1606] [PG ID 0 PG GUID 0(default_pg) Rank 11] ProcessGroupNCCL preparing to dump debug info. Include stack trace: 0, only active collectives: 0
|
| 12 |
+
[rank11]:[E611 13:46:09.713669579 ProcessGroupNCCL.cpp:1367] [PG ID 0 PG GUID 0(default_pg) Rank 11] Future for Flight recorder dump in heartbeatMonitor timed out after 15000 ms
|
| 13 |
+
[rank11]:[F611 13:54:09.715074126 ProcessGroupNCCL.cpp:1631] [PG ID 0 PG GUID 0(default_pg) Rank 11] [PG ID 0 PG GUID 0(default_pg) Rank 11] Terminating the process after attempting to dump debug info, due to collective timeout or exception.
|
inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/job_45423709/45423709_11_log.out
ADDED
|
@@ -0,0 +1,208 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
submitit INFO (2026-06-11 05:16:06,427) - Starting with JobEnvironment(job_id=45423709, hostname=lrdn1227.leonardo.local, local_rank=3(4), node=2(8), global_rank=11(32))
|
| 2 |
+
submitit INFO (2026-06-11 05:16:06,428) - Loading pickle: /leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/job_45423709/45423709_submitted.pkl
|
| 3 |
+
INFO:root:loaded pretrain params...
|
| 4 |
+
{ 'app': 'vjepa',
|
| 5 |
+
'cpus_per_task': 8,
|
| 6 |
+
'data': { 'batch_size': 32,
|
| 7 |
+
'crop_size': 224,
|
| 8 |
+
'dataset_fpcs': [16, 16],
|
| 9 |
+
'dataset_type': 'VideoDataset',
|
| 10 |
+
'datasets': [ '/leonardo_scratch/fast/EUHPC_D35_082/data/k400/train.csv',
|
| 11 |
+
'/leonardo_scratch/fast/EUHPC_D35_082/data/ssv2/train.csv'],
|
| 12 |
+
'datasets_weights': [0.65, 0.35],
|
| 13 |
+
'fps': 4,
|
| 14 |
+
'num_workers': 8,
|
| 15 |
+
'patch_size': 14,
|
| 16 |
+
'persistent_workers': True,
|
| 17 |
+
'pin_mem': True,
|
| 18 |
+
'tubelet_size': 1},
|
| 19 |
+
'data_aug': { 'auto_augment': False,
|
| 20 |
+
'motion_shift': False,
|
| 21 |
+
'random_resize_aspect_ratio': [0.75, 1.35],
|
| 22 |
+
'random_resize_scale': [0.3, 1.0],
|
| 23 |
+
'reprob': 0.0},
|
| 24 |
+
'folder': '/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3',
|
| 25 |
+
'loss': {'loss_exp': 1.0},
|
| 26 |
+
'mask': [ { 'aspect_ratio': [0.75, 1.5],
|
| 27 |
+
'full_complement': False,
|
| 28 |
+
'max_keep': None,
|
| 29 |
+
'max_temporal_keep': 1.0,
|
| 30 |
+
'num_blocks': 8,
|
| 31 |
+
'spatial_scale': [0.15, 0.15],
|
| 32 |
+
'temporal_scale': [1.0, 1.0]},
|
| 33 |
+
{ 'aspect_ratio': [0.75, 1.5],
|
| 34 |
+
'full_complement': False,
|
| 35 |
+
'max_keep': None,
|
| 36 |
+
'max_temporal_keep': 1.0,
|
| 37 |
+
'num_blocks': 2,
|
| 38 |
+
'spatial_scale': [0.7, 0.7],
|
| 39 |
+
'temporal_scale': [1.0, 1.0]}],
|
| 40 |
+
'mem_per_gpu': '120G',
|
| 41 |
+
'meta': { 'dtype': 'bfloat16',
|
| 42 |
+
'knn_eval_epoch0': False,
|
| 43 |
+
'knn_eval_freq': 5,
|
| 44 |
+
'knn_eval_presets': [ { 'config': { 'batch_size': 8,
|
| 45 |
+
'dataset_train': '/leonardo_scratch/fast/EUHPC_D35_082/data/ssv2/train_coarse10.csv',
|
| 46 |
+
'dataset_val': '/leonardo_scratch/fast/EUHPC_D35_082/data/ssv2/val_coarse10.csv',
|
| 47 |
+
'eval_videos_per_class': 100,
|
| 48 |
+
'linear_probe': True,
|
| 49 |
+
'num_workers': 2,
|
| 50 |
+
'pool_type': 'temporal_concat',
|
| 51 |
+
'train_videos_per_class': 500},
|
| 52 |
+
'preset': 'ssv2_coarse10'}],
|
| 53 |
+
'load_checkpoint': True,
|
| 54 |
+
'read_checkpoint': None,
|
| 55 |
+
'save_every_freq': 5,
|
| 56 |
+
'seed': 239,
|
| 57 |
+
'use_sdpa': True,
|
| 58 |
+
'use_wandb': True,
|
| 59 |
+
'wandb_project': 'vjepa_ablation'},
|
| 60 |
+
'metrics': {'std': {}},
|
| 61 |
+
'model': { 'class_token': True,
|
| 62 |
+
'freeze_backbone': False,
|
| 63 |
+
'is_causal': True,
|
| 64 |
+
'model_name': 'vit_large_patch14_dinov2_lvd142m',
|
| 65 |
+
'pe_type': '2d',
|
| 66 |
+
'pred_depth': 12,
|
| 67 |
+
'pred_embed_dim': 384,
|
| 68 |
+
'pred_num_heads': 6,
|
| 69 |
+
'predictor': 'v2_cross',
|
| 70 |
+
'stem_type': '2d',
|
| 71 |
+
'target_embed_dim': 1024,
|
| 72 |
+
'target_kind': 'ema',
|
| 73 |
+
'uniform_power': True,
|
| 74 |
+
'use_activation_checkpointing': True,
|
| 75 |
+
'use_alibi': False,
|
| 76 |
+
'use_lora': False,
|
| 77 |
+
'use_mask_tokens': True,
|
| 78 |
+
'use_rope': False,
|
| 79 |
+
'use_sdpa': True,
|
| 80 |
+
'zero_init_mask_tokens': True},
|
| 81 |
+
'nodes': 8,
|
| 82 |
+
'optimization': { 'clip_grad': None,
|
| 83 |
+
'ema': [1.0, 0.99925],
|
| 84 |
+
'ema_hold_epochs': 0,
|
| 85 |
+
'ema_ramp_epochs': 30,
|
| 86 |
+
'encoder_layer_decay': 0.9,
|
| 87 |
+
'epochs': 60,
|
| 88 |
+
'final_lr': 0.0005,
|
| 89 |
+
'final_weight_decay': 0.04,
|
| 90 |
+
'ipe': 300,
|
| 91 |
+
'ipe_scale': 1.0,
|
| 92 |
+
'lr': 0.0005,
|
| 93 |
+
'predictor_lr_scale': 3.0,
|
| 94 |
+
'start_lr': 5e-05,
|
| 95 |
+
'warmup': 4,
|
| 96 |
+
'weight_decay': 0.04},
|
| 97 |
+
'tasks_per_node': 4}
|
| 98 |
+
INFO:root:Running pre-training of app: vjepa
|
| 99 |
+
[INFO ][2026-06-11 05:16:17][app.vjepa.train ][main ] which_dtype='bfloat16'
|
| 100 |
+
[INFO ][2026-06-11 05:16:17][app.vjepa.train ][main ] Disabling persistent_workers (incompatible with KNN eval)
|
| 101 |
+
[INFO ][2026-06-11 05:16:17][app.vjepa.train ][main ] NCCL_SOCKET_IFNAME=None
|
| 102 |
+
[INFO ][2026-06-11 05:16:18][app.vjepa.train ][main ] Initialized (rank/world-size) 11/32, tasks_per_node=4
|
| 103 |
+
[WARNING ][2026-06-11 05:16:25][experiments.stmodels.vision_transformers_v3][_build_dinov2_vit ] DINOv2 is an image model — fwd_chunk is unset so all frames will be processed jointly. Set fwd_chunk=1 for per-frame (standard) processing.
|
| 104 |
+
[INFO ][2026-06-11 05:16:40][experiments.stmodels.vision_transformers_v3][_build_dinov2_vit ] Loading pretrained weights for vit_large_patch14_dinov2_lvd142m from timm (vit_large_patch14_dinov2)
|
| 105 |
+
[INFO ][2026-06-11 05:16:53][timm.models._builder][load_pretrained ] Loading pretrained weights from Hugging Face hub (timm/vit_large_patch14_dinov2.lvd142m)
|
| 106 |
+
[INFO ][2026-06-11 05:16:53][timm.models._hub ][load_state_dict_from_hf ] [timm/vit_large_patch14_dinov2.lvd142m] Safe alternative available for 'pytorch_model.bin' (as 'model.safetensors'). Loading weights using safetensors.
|
| 107 |
+
[INFO ][2026-06-11 05:16:53][timm.layers.pos_embed][resample_abs_pos_embed ] Resized position embedding: (37, 37) to (16, 16).
|
| 108 |
+
[INFO ][2026-06-11 05:17:33][root ][init_video_model ] ViTMultiSeqWrapper(
|
| 109 |
+
(backbone): VisionTransformer(
|
| 110 |
+
(patch_embed): PatchEmbed3D(
|
| 111 |
+
(proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
|
| 112 |
+
)
|
| 113 |
+
(blocks): ModuleList(
|
| 114 |
+
(0-23): 24 x Block(
|
| 115 |
+
(norm1): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
|
| 116 |
+
(norm2): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
|
| 117 |
+
(drop_path1): Identity()
|
| 118 |
+
(drop_path2): Identity()
|
| 119 |
+
(attn): Attention(
|
| 120 |
+
(qkv): Linear(in_features=1024, out_features=3072, bias=True)
|
| 121 |
+
(attn_drop): Dropout(p=0.0, inplace=False)
|
| 122 |
+
(proj): Linear(in_features=1024, out_features=1024, bias=True)
|
| 123 |
+
(proj_drop): Dropout(p=0.0, inplace=False)
|
| 124 |
+
)
|
| 125 |
+
(mlp): MLP(
|
| 126 |
+
(fc1): Linear(in_features=1024, out_features=4096, bias=True)
|
| 127 |
+
(act): GELU(approximate='none')
|
| 128 |
+
(fc2): Linear(in_features=4096, out_features=1024, bias=True)
|
| 129 |
+
(drop): Dropout(p=0.0, inplace=False)
|
| 130 |
+
)
|
| 131 |
+
)
|
| 132 |
+
)
|
| 133 |
+
(norm): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
|
| 134 |
+
)
|
| 135 |
+
)
|
| 136 |
+
[INFO ][2026-06-11 05:17:33][root ][init_video_model ] VJEPAPredictorMultiSeqWrapper(
|
| 137 |
+
(backbone): PredictorV2(
|
| 138 |
+
(predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
|
| 139 |
+
(mask_tokens): ParameterList(
|
| 140 |
+
(0): Parameter containing: [torch.float32 of size 1x384 (cuda:0)]
|
| 141 |
+
(1): Parameter containing: [torch.float32 of size 1x384 (cuda:0)]
|
| 142 |
+
(2): Parameter containing: [torch.float32 of size 1x384 (cuda:0)]
|
| 143 |
+
(3): Parameter containing: [torch.float32 of size 1x384 (cuda:0)]
|
| 144 |
+
)
|
| 145 |
+
(predictor_blocks): ModuleList(
|
| 146 |
+
(0-11): 12 x Block(
|
| 147 |
+
(residual1): EfficientResidual(
|
| 148 |
+
(norm): LayerNorm((384,), eps=1e-05, elementwise_affine=True)
|
| 149 |
+
(fn): Attention(
|
| 150 |
+
(q_proj): Linear(in_features=384, out_features=384, bias=False)
|
| 151 |
+
(k_proj): Linear(in_features=384, out_features=384, bias=False)
|
| 152 |
+
(v_proj): Linear(in_features=384, out_features=384, bias=False)
|
| 153 |
+
(proj): Linear(in_features=384, out_features=384, bias=False)
|
| 154 |
+
(rope): Rope()
|
| 155 |
+
)
|
| 156 |
+
)
|
| 157 |
+
(residual2): EfficientResidual(
|
| 158 |
+
(norm): LayerNorm((384,), eps=1e-05, elementwise_affine=True)
|
| 159 |
+
(fn): MLP(
|
| 160 |
+
(fc1): Linear(in_features=384, out_features=1536, bias=False)
|
| 161 |
+
(act): GELU(approximate='none')
|
| 162 |
+
(fc2): Linear(in_features=1536, out_features=384, bias=False)
|
| 163 |
+
(drop): Dropout(p=0.0, inplace=False)
|
| 164 |
+
)
|
| 165 |
+
)
|
| 166 |
+
)
|
| 167 |
+
)
|
| 168 |
+
(predictor_norm): LayerNorm((384,), eps=1e-05, elementwise_affine=True)
|
| 169 |
+
(predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
|
| 170 |
+
)
|
| 171 |
+
)
|
| 172 |
+
[INFO ][2026-06-11 05:17:33][root ][init_video_model ] ViTMultiSeqWrapper(
|
| 173 |
+
(backbone): VisionTransformer(
|
| 174 |
+
(patch_embed): PatchEmbed3D(
|
| 175 |
+
(proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
|
| 176 |
+
)
|
| 177 |
+
(blocks): ModuleList(
|
| 178 |
+
(0-23): 24 x Block(
|
| 179 |
+
(norm1): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
|
| 180 |
+
(norm2): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
|
| 181 |
+
(drop_path1): Identity()
|
| 182 |
+
(drop_path2): Identity()
|
| 183 |
+
(attn): Attention(
|
| 184 |
+
(qkv): Linear(in_features=1024, out_features=3072, bias=True)
|
| 185 |
+
(attn_drop): Dropout(p=0.0, inplace=False)
|
| 186 |
+
(proj): Linear(in_features=1024, out_features=1024, bias=True)
|
| 187 |
+
(proj_drop): Dropout(p=0.0, inplace=False)
|
| 188 |
+
)
|
| 189 |
+
(mlp): MLP(
|
| 190 |
+
(fc1): Linear(in_features=1024, out_features=4096, bias=True)
|
| 191 |
+
(act): GELU(approximate='none')
|
| 192 |
+
(fc2): Linear(in_features=4096, out_features=1024, bias=True)
|
| 193 |
+
(drop): Dropout(p=0.0, inplace=False)
|
| 194 |
+
)
|
| 195 |
+
)
|
| 196 |
+
)
|
| 197 |
+
(norm): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
|
| 198 |
+
)
|
| 199 |
+
)
|
| 200 |
+
[INFO ][2026-06-11 05:17:33][root ][init_video_model ] Encoder number of parameters: 302964736
|
| 201 |
+
[INFO ][2026-06-11 05:17:33][root ][init_video_model ] Predictor number of parameters: 22042240
|
| 202 |
+
[INFO ][2026-06-11 05:17:33][root ][init_video_model ] Target encoder number of parameters: 302964736
|
| 203 |
+
[INFO ][2026-06-11 05:17:34][root ][make_videodataset ] VideoDataset dataset created
|
| 204 |
+
[INFO ][2026-06-11 05:17:34][WeightedSampler ][__init__ ] Using DistributedWeightedSampler with rank 11 / 32
|
| 205 |
+
[INFO ][2026-06-11 05:17:34][root ][make_videodataset ] VideoDataset unsupervised data loader created
|
| 206 |
+
[INFO ][2026-06-11 05:17:34][app.vjepa.train ][main ] iterations per epoch/dataset length: 300/392
|
| 207 |
+
[INFO ][2026-06-11 05:17:34][app.vjepa.train ][main ] Wrapping models in DDP (rank 11)...
|
| 208 |
+
[INFO ][2026-06-11 05:17:35][app.vjepa.train ][main ] Initializing loader...
|
inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/job_45423709/45423709_12_log.err
ADDED
|
@@ -0,0 +1,13 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
|
| 2 |
+
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
|
| 3 |
+
/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/utils.py:888: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
|
| 4 |
+
scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
|
| 5 |
+
/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py:983: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
|
| 6 |
+
with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
|
| 7 |
+
[rank12]:[E611 13:45:39.661875894 ProcessGroupNCCL.cpp:1825] [PG ID 0 PG GUID 0(default_pg) Rank 12] Observed flight recorder dump signal from another rank via TCPStore.
|
| 8 |
+
[rank12]:[E611 13:45:39.663088686 ProcessGroupNCCL.cpp:1890] [PG ID 0 PG GUID 0(default_pg) Rank 12] Received a dump signal due to a collective timeout from rank 20 and we will try our best to dump the debug info. Last enqueued NCCL work: 410189, last completed NCCL work: 410188.This is most likely caused by incorrect usages of collectives, e.g., wrong sizes used across ranks, the order of collectives is not same for all ranks or the scheduled collective, for some reason, didn't run. Additionally, this can be caused by GIL deadlock or other reasons such as network errors or bugs in the communications library (e.g. NCCL), etc.
|
| 9 |
+
[rank12]:[E611 13:45:39.664590729 ProcessGroupNCCL.cpp:1606] [PG ID 0 PG GUID 0(default_pg) Rank 12] ProcessGroupNCCL preparing to dump debug info. Include stack trace: 1, only active collectives: 0
|
| 10 |
+
[rank12]:[E611 13:45:54.664122358 ProcessGroupNCCL.cpp:1367] [PG ID 0 PG GUID 0(default_pg) Rank 12] Future for Flight recorder dump in heartbeatMonitor timed out after 15000 ms
|
| 11 |
+
[rank12]:[E611 13:45:54.664364906 ProcessGroupNCCL.cpp:1606] [PG ID 0 PG GUID 0(default_pg) Rank 12] ProcessGroupNCCL preparing to dump debug info. Include stack trace: 0, only active collectives: 0
|
| 12 |
+
[rank12]:[E611 13:46:09.664289028 ProcessGroupNCCL.cpp:1367] [PG ID 0 PG GUID 0(default_pg) Rank 12] Future for Flight recorder dump in heartbeatMonitor timed out after 15000 ms
|
| 13 |
+
[rank12]:[F611 13:54:09.665411486 ProcessGroupNCCL.cpp:1631] [PG ID 0 PG GUID 0(default_pg) Rank 12] [PG ID 0 PG GUID 0(default_pg) Rank 12] Terminating the process after attempting to dump debug info, due to collective timeout or exception.
|
inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/job_45423709/45423709_12_log.out
ADDED
|
@@ -0,0 +1,208 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
submitit INFO (2026-06-11 05:16:06,428) - Starting with JobEnvironment(job_id=45423709, hostname=lrdn1244.leonardo.local, local_rank=0(4), node=3(8), global_rank=12(32))
|
| 2 |
+
submitit INFO (2026-06-11 05:16:06,428) - Loading pickle: /leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/job_45423709/45423709_submitted.pkl
|
| 3 |
+
INFO:root:loaded pretrain params...
|
| 4 |
+
{ 'app': 'vjepa',
|
| 5 |
+
'cpus_per_task': 8,
|
| 6 |
+
'data': { 'batch_size': 32,
|
| 7 |
+
'crop_size': 224,
|
| 8 |
+
'dataset_fpcs': [16, 16],
|
| 9 |
+
'dataset_type': 'VideoDataset',
|
| 10 |
+
'datasets': [ '/leonardo_scratch/fast/EUHPC_D35_082/data/k400/train.csv',
|
| 11 |
+
'/leonardo_scratch/fast/EUHPC_D35_082/data/ssv2/train.csv'],
|
| 12 |
+
'datasets_weights': [0.65, 0.35],
|
| 13 |
+
'fps': 4,
|
| 14 |
+
'num_workers': 8,
|
| 15 |
+
'patch_size': 14,
|
| 16 |
+
'persistent_workers': True,
|
| 17 |
+
'pin_mem': True,
|
| 18 |
+
'tubelet_size': 1},
|
| 19 |
+
'data_aug': { 'auto_augment': False,
|
| 20 |
+
'motion_shift': False,
|
| 21 |
+
'random_resize_aspect_ratio': [0.75, 1.35],
|
| 22 |
+
'random_resize_scale': [0.3, 1.0],
|
| 23 |
+
'reprob': 0.0},
|
| 24 |
+
'folder': '/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3',
|
| 25 |
+
'loss': {'loss_exp': 1.0},
|
| 26 |
+
'mask': [ { 'aspect_ratio': [0.75, 1.5],
|
| 27 |
+
'full_complement': False,
|
| 28 |
+
'max_keep': None,
|
| 29 |
+
'max_temporal_keep': 1.0,
|
| 30 |
+
'num_blocks': 8,
|
| 31 |
+
'spatial_scale': [0.15, 0.15],
|
| 32 |
+
'temporal_scale': [1.0, 1.0]},
|
| 33 |
+
{ 'aspect_ratio': [0.75, 1.5],
|
| 34 |
+
'full_complement': False,
|
| 35 |
+
'max_keep': None,
|
| 36 |
+
'max_temporal_keep': 1.0,
|
| 37 |
+
'num_blocks': 2,
|
| 38 |
+
'spatial_scale': [0.7, 0.7],
|
| 39 |
+
'temporal_scale': [1.0, 1.0]}],
|
| 40 |
+
'mem_per_gpu': '120G',
|
| 41 |
+
'meta': { 'dtype': 'bfloat16',
|
| 42 |
+
'knn_eval_epoch0': False,
|
| 43 |
+
'knn_eval_freq': 5,
|
| 44 |
+
'knn_eval_presets': [ { 'config': { 'batch_size': 8,
|
| 45 |
+
'dataset_train': '/leonardo_scratch/fast/EUHPC_D35_082/data/ssv2/train_coarse10.csv',
|
| 46 |
+
'dataset_val': '/leonardo_scratch/fast/EUHPC_D35_082/data/ssv2/val_coarse10.csv',
|
| 47 |
+
'eval_videos_per_class': 100,
|
| 48 |
+
'linear_probe': True,
|
| 49 |
+
'num_workers': 2,
|
| 50 |
+
'pool_type': 'temporal_concat',
|
| 51 |
+
'train_videos_per_class': 500},
|
| 52 |
+
'preset': 'ssv2_coarse10'}],
|
| 53 |
+
'load_checkpoint': True,
|
| 54 |
+
'read_checkpoint': None,
|
| 55 |
+
'save_every_freq': 5,
|
| 56 |
+
'seed': 239,
|
| 57 |
+
'use_sdpa': True,
|
| 58 |
+
'use_wandb': True,
|
| 59 |
+
'wandb_project': 'vjepa_ablation'},
|
| 60 |
+
'metrics': {'std': {}},
|
| 61 |
+
'model': { 'class_token': True,
|
| 62 |
+
'freeze_backbone': False,
|
| 63 |
+
'is_causal': True,
|
| 64 |
+
'model_name': 'vit_large_patch14_dinov2_lvd142m',
|
| 65 |
+
'pe_type': '2d',
|
| 66 |
+
'pred_depth': 12,
|
| 67 |
+
'pred_embed_dim': 384,
|
| 68 |
+
'pred_num_heads': 6,
|
| 69 |
+
'predictor': 'v2_cross',
|
| 70 |
+
'stem_type': '2d',
|
| 71 |
+
'target_embed_dim': 1024,
|
| 72 |
+
'target_kind': 'ema',
|
| 73 |
+
'uniform_power': True,
|
| 74 |
+
'use_activation_checkpointing': True,
|
| 75 |
+
'use_alibi': False,
|
| 76 |
+
'use_lora': False,
|
| 77 |
+
'use_mask_tokens': True,
|
| 78 |
+
'use_rope': False,
|
| 79 |
+
'use_sdpa': True,
|
| 80 |
+
'zero_init_mask_tokens': True},
|
| 81 |
+
'nodes': 8,
|
| 82 |
+
'optimization': { 'clip_grad': None,
|
| 83 |
+
'ema': [1.0, 0.99925],
|
| 84 |
+
'ema_hold_epochs': 0,
|
| 85 |
+
'ema_ramp_epochs': 30,
|
| 86 |
+
'encoder_layer_decay': 0.9,
|
| 87 |
+
'epochs': 60,
|
| 88 |
+
'final_lr': 0.0005,
|
| 89 |
+
'final_weight_decay': 0.04,
|
| 90 |
+
'ipe': 300,
|
| 91 |
+
'ipe_scale': 1.0,
|
| 92 |
+
'lr': 0.0005,
|
| 93 |
+
'predictor_lr_scale': 3.0,
|
| 94 |
+
'start_lr': 5e-05,
|
| 95 |
+
'warmup': 4,
|
| 96 |
+
'weight_decay': 0.04},
|
| 97 |
+
'tasks_per_node': 4}
|
| 98 |
+
INFO:root:Running pre-training of app: vjepa
|
| 99 |
+
[INFO ][2026-06-11 05:16:17][app.vjepa.train ][main ] which_dtype='bfloat16'
|
| 100 |
+
[INFO ][2026-06-11 05:16:17][app.vjepa.train ][main ] Disabling persistent_workers (incompatible with KNN eval)
|
| 101 |
+
[INFO ][2026-06-11 05:16:17][app.vjepa.train ][main ] NCCL_SOCKET_IFNAME=None
|
| 102 |
+
[INFO ][2026-06-11 05:16:18][app.vjepa.train ][main ] Initialized (rank/world-size) 12/32, tasks_per_node=4
|
| 103 |
+
[WARNING ][2026-06-11 05:16:25][experiments.stmodels.vision_transformers_v3][_build_dinov2_vit ] DINOv2 is an image model — fwd_chunk is unset so all frames will be processed jointly. Set fwd_chunk=1 for per-frame (standard) processing.
|
| 104 |
+
[INFO ][2026-06-11 05:16:39][experiments.stmodels.vision_transformers_v3][_build_dinov2_vit ] Loading pretrained weights for vit_large_patch14_dinov2_lvd142m from timm (vit_large_patch14_dinov2)
|
| 105 |
+
[INFO ][2026-06-11 05:16:52][timm.models._builder][load_pretrained ] Loading pretrained weights from Hugging Face hub (timm/vit_large_patch14_dinov2.lvd142m)
|
| 106 |
+
[INFO ][2026-06-11 05:16:52][timm.models._hub ][load_state_dict_from_hf ] [timm/vit_large_patch14_dinov2.lvd142m] Safe alternative available for 'pytorch_model.bin' (as 'model.safetensors'). Loading weights using safetensors.
|
| 107 |
+
[INFO ][2026-06-11 05:16:53][timm.layers.pos_embed][resample_abs_pos_embed ] Resized position embedding: (37, 37) to (16, 16).
|
| 108 |
+
[INFO ][2026-06-11 05:17:34][root ][init_video_model ] ViTMultiSeqWrapper(
|
| 109 |
+
(backbone): VisionTransformer(
|
| 110 |
+
(patch_embed): PatchEmbed3D(
|
| 111 |
+
(proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
|
| 112 |
+
)
|
| 113 |
+
(blocks): ModuleList(
|
| 114 |
+
(0-23): 24 x Block(
|
| 115 |
+
(norm1): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
|
| 116 |
+
(norm2): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
|
| 117 |
+
(drop_path1): Identity()
|
| 118 |
+
(drop_path2): Identity()
|
| 119 |
+
(attn): Attention(
|
| 120 |
+
(qkv): Linear(in_features=1024, out_features=3072, bias=True)
|
| 121 |
+
(attn_drop): Dropout(p=0.0, inplace=False)
|
| 122 |
+
(proj): Linear(in_features=1024, out_features=1024, bias=True)
|
| 123 |
+
(proj_drop): Dropout(p=0.0, inplace=False)
|
| 124 |
+
)
|
| 125 |
+
(mlp): MLP(
|
| 126 |
+
(fc1): Linear(in_features=1024, out_features=4096, bias=True)
|
| 127 |
+
(act): GELU(approximate='none')
|
| 128 |
+
(fc2): Linear(in_features=4096, out_features=1024, bias=True)
|
| 129 |
+
(drop): Dropout(p=0.0, inplace=False)
|
| 130 |
+
)
|
| 131 |
+
)
|
| 132 |
+
)
|
| 133 |
+
(norm): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
|
| 134 |
+
)
|
| 135 |
+
)
|
| 136 |
+
[INFO ][2026-06-11 05:17:34][root ][init_video_model ] VJEPAPredictorMultiSeqWrapper(
|
| 137 |
+
(backbone): PredictorV2(
|
| 138 |
+
(predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
|
| 139 |
+
(mask_tokens): ParameterList(
|
| 140 |
+
(0): Parameter containing: [torch.float32 of size 1x384 (cuda:0)]
|
| 141 |
+
(1): Parameter containing: [torch.float32 of size 1x384 (cuda:0)]
|
| 142 |
+
(2): Parameter containing: [torch.float32 of size 1x384 (cuda:0)]
|
| 143 |
+
(3): Parameter containing: [torch.float32 of size 1x384 (cuda:0)]
|
| 144 |
+
)
|
| 145 |
+
(predictor_blocks): ModuleList(
|
| 146 |
+
(0-11): 12 x Block(
|
| 147 |
+
(residual1): EfficientResidual(
|
| 148 |
+
(norm): LayerNorm((384,), eps=1e-05, elementwise_affine=True)
|
| 149 |
+
(fn): Attention(
|
| 150 |
+
(q_proj): Linear(in_features=384, out_features=384, bias=False)
|
| 151 |
+
(k_proj): Linear(in_features=384, out_features=384, bias=False)
|
| 152 |
+
(v_proj): Linear(in_features=384, out_features=384, bias=False)
|
| 153 |
+
(proj): Linear(in_features=384, out_features=384, bias=False)
|
| 154 |
+
(rope): Rope()
|
| 155 |
+
)
|
| 156 |
+
)
|
| 157 |
+
(residual2): EfficientResidual(
|
| 158 |
+
(norm): LayerNorm((384,), eps=1e-05, elementwise_affine=True)
|
| 159 |
+
(fn): MLP(
|
| 160 |
+
(fc1): Linear(in_features=384, out_features=1536, bias=False)
|
| 161 |
+
(act): GELU(approximate='none')
|
| 162 |
+
(fc2): Linear(in_features=1536, out_features=384, bias=False)
|
| 163 |
+
(drop): Dropout(p=0.0, inplace=False)
|
| 164 |
+
)
|
| 165 |
+
)
|
| 166 |
+
)
|
| 167 |
+
)
|
| 168 |
+
(predictor_norm): LayerNorm((384,), eps=1e-05, elementwise_affine=True)
|
| 169 |
+
(predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
|
| 170 |
+
)
|
| 171 |
+
)
|
| 172 |
+
[INFO ][2026-06-11 05:17:34][root ][init_video_model ] ViTMultiSeqWrapper(
|
| 173 |
+
(backbone): VisionTransformer(
|
| 174 |
+
(patch_embed): PatchEmbed3D(
|
| 175 |
+
(proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
|
| 176 |
+
)
|
| 177 |
+
(blocks): ModuleList(
|
| 178 |
+
(0-23): 24 x Block(
|
| 179 |
+
(norm1): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
|
| 180 |
+
(norm2): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
|
| 181 |
+
(drop_path1): Identity()
|
| 182 |
+
(drop_path2): Identity()
|
| 183 |
+
(attn): Attention(
|
| 184 |
+
(qkv): Linear(in_features=1024, out_features=3072, bias=True)
|
| 185 |
+
(attn_drop): Dropout(p=0.0, inplace=False)
|
| 186 |
+
(proj): Linear(in_features=1024, out_features=1024, bias=True)
|
| 187 |
+
(proj_drop): Dropout(p=0.0, inplace=False)
|
| 188 |
+
)
|
| 189 |
+
(mlp): MLP(
|
| 190 |
+
(fc1): Linear(in_features=1024, out_features=4096, bias=True)
|
| 191 |
+
(act): GELU(approximate='none')
|
| 192 |
+
(fc2): Linear(in_features=4096, out_features=1024, bias=True)
|
| 193 |
+
(drop): Dropout(p=0.0, inplace=False)
|
| 194 |
+
)
|
| 195 |
+
)
|
| 196 |
+
)
|
| 197 |
+
(norm): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
|
| 198 |
+
)
|
| 199 |
+
)
|
| 200 |
+
[INFO ][2026-06-11 05:17:34][root ][init_video_model ] Encoder number of parameters: 302964736
|
| 201 |
+
[INFO ][2026-06-11 05:17:34][root ][init_video_model ] Predictor number of parameters: 22042240
|
| 202 |
+
[INFO ][2026-06-11 05:17:34][root ][init_video_model ] Target encoder number of parameters: 302964736
|
| 203 |
+
[INFO ][2026-06-11 05:17:34][root ][make_videodataset ] VideoDataset dataset created
|
| 204 |
+
[INFO ][2026-06-11 05:17:34][WeightedSampler ][__init__ ] Using DistributedWeightedSampler with rank 12 / 32
|
| 205 |
+
[INFO ][2026-06-11 05:17:34][root ][make_videodataset ] VideoDataset unsupervised data loader created
|
| 206 |
+
[INFO ][2026-06-11 05:17:34][app.vjepa.train ][main ] iterations per epoch/dataset length: 300/392
|
| 207 |
+
[INFO ][2026-06-11 05:17:34][app.vjepa.train ][main ] Wrapping models in DDP (rank 12)...
|
| 208 |
+
[INFO ][2026-06-11 05:17:35][app.vjepa.train ][main ] Initializing loader...
|
inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/job_45423709/45423709_13_log.err
ADDED
|
@@ -0,0 +1,15 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
|
| 2 |
+
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
|
| 3 |
+
/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/utils.py:888: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
|
| 4 |
+
scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
|
| 5 |
+
/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py:983: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
|
| 6 |
+
with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
|
| 7 |
+
[rank13]:[E611 13:45:39.661806418 ProcessGroupNCCL.cpp:1825] [PG ID 0 PG GUID 0(default_pg) Rank 13] Observed flight recorder dump signal from another rank via TCPStore.
|
| 8 |
+
[rank13]:[E611 13:45:39.663091141 ProcessGroupNCCL.cpp:1890] [PG ID 0 PG GUID 0(default_pg) Rank 13] Received a dump signal due to a collective timeout from rank 20 and we will try our best to dump the debug info. Last enqueued NCCL work: 410189, last completed NCCL work: 410188.This is most likely caused by incorrect usages of collectives, e.g., wrong sizes used across ranks, the order of collectives is not same for all ranks or the scheduled collective, for some reason, didn't run. Additionally, this can be caused by GIL deadlock or other reasons such as network errors or bugs in the communications library (e.g. NCCL), etc.
|
| 9 |
+
[rank13]:[E611 13:45:39.664584249 ProcessGroupNCCL.cpp:1606] [PG ID 0 PG GUID 0(default_pg) Rank 13] ProcessGroupNCCL preparing to dump debug info. Include stack trace: 1, only active collectives: 0
|
| 10 |
+
[rank13]:[E611 13:45:54.664112078 ProcessGroupNCCL.cpp:1367] [PG ID 0 PG GUID 0(default_pg) Rank 13] Future for Flight recorder dump in heartbeatMonitor timed out after 15000 ms
|
| 11 |
+
[rank13]:[E611 13:45:54.664177991 ProcessGroupNCCL.cpp:1606] [PG ID 0 PG GUID 0(default_pg) Rank 13] ProcessGroupNCCL preparing to dump debug info. Include stack trace: 0, only active collectives: 0
|
| 12 |
+
[rank13]:[E611 13:46:09.664231889 ProcessGroupNCCL.cpp:1367] [PG ID 0 PG GUID 0(default_pg) Rank 13] Future for Flight recorder dump in heartbeatMonitor timed out after 15000 ms
|
| 13 |
+
[rank13]:[F611 13:54:09.665355619 ProcessGroupNCCL.cpp:1631] [PG ID 0 PG GUID 0(default_pg) Rank 13] [PG ID 0 PG GUID 0(default_pg) Rank 13] Terminating the process after attempting to dump debug info, due to collective timeout or exception.
|
| 14 |
+
/usr/lib64/python3.11/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 34 leaked semaphore objects to clean up at shutdown
|
| 15 |
+
warnings.warn('resource_tracker: There appear to be %d '
|
inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/job_45423709/45423709_13_log.out
ADDED
|
@@ -0,0 +1,208 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
submitit INFO (2026-06-11 05:16:06,428) - Starting with JobEnvironment(job_id=45423709, hostname=lrdn1244.leonardo.local, local_rank=1(4), node=3(8), global_rank=13(32))
|
| 2 |
+
submitit INFO (2026-06-11 05:16:06,428) - Loading pickle: /leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/job_45423709/45423709_submitted.pkl
|
| 3 |
+
INFO:root:loaded pretrain params...
|
| 4 |
+
{ 'app': 'vjepa',
|
| 5 |
+
'cpus_per_task': 8,
|
| 6 |
+
'data': { 'batch_size': 32,
|
| 7 |
+
'crop_size': 224,
|
| 8 |
+
'dataset_fpcs': [16, 16],
|
| 9 |
+
'dataset_type': 'VideoDataset',
|
| 10 |
+
'datasets': [ '/leonardo_scratch/fast/EUHPC_D35_082/data/k400/train.csv',
|
| 11 |
+
'/leonardo_scratch/fast/EUHPC_D35_082/data/ssv2/train.csv'],
|
| 12 |
+
'datasets_weights': [0.65, 0.35],
|
| 13 |
+
'fps': 4,
|
| 14 |
+
'num_workers': 8,
|
| 15 |
+
'patch_size': 14,
|
| 16 |
+
'persistent_workers': True,
|
| 17 |
+
'pin_mem': True,
|
| 18 |
+
'tubelet_size': 1},
|
| 19 |
+
'data_aug': { 'auto_augment': False,
|
| 20 |
+
'motion_shift': False,
|
| 21 |
+
'random_resize_aspect_ratio': [0.75, 1.35],
|
| 22 |
+
'random_resize_scale': [0.3, 1.0],
|
| 23 |
+
'reprob': 0.0},
|
| 24 |
+
'folder': '/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3',
|
| 25 |
+
'loss': {'loss_exp': 1.0},
|
| 26 |
+
'mask': [ { 'aspect_ratio': [0.75, 1.5],
|
| 27 |
+
'full_complement': False,
|
| 28 |
+
'max_keep': None,
|
| 29 |
+
'max_temporal_keep': 1.0,
|
| 30 |
+
'num_blocks': 8,
|
| 31 |
+
'spatial_scale': [0.15, 0.15],
|
| 32 |
+
'temporal_scale': [1.0, 1.0]},
|
| 33 |
+
{ 'aspect_ratio': [0.75, 1.5],
|
| 34 |
+
'full_complement': False,
|
| 35 |
+
'max_keep': None,
|
| 36 |
+
'max_temporal_keep': 1.0,
|
| 37 |
+
'num_blocks': 2,
|
| 38 |
+
'spatial_scale': [0.7, 0.7],
|
| 39 |
+
'temporal_scale': [1.0, 1.0]}],
|
| 40 |
+
'mem_per_gpu': '120G',
|
| 41 |
+
'meta': { 'dtype': 'bfloat16',
|
| 42 |
+
'knn_eval_epoch0': False,
|
| 43 |
+
'knn_eval_freq': 5,
|
| 44 |
+
'knn_eval_presets': [ { 'config': { 'batch_size': 8,
|
| 45 |
+
'dataset_train': '/leonardo_scratch/fast/EUHPC_D35_082/data/ssv2/train_coarse10.csv',
|
| 46 |
+
'dataset_val': '/leonardo_scratch/fast/EUHPC_D35_082/data/ssv2/val_coarse10.csv',
|
| 47 |
+
'eval_videos_per_class': 100,
|
| 48 |
+
'linear_probe': True,
|
| 49 |
+
'num_workers': 2,
|
| 50 |
+
'pool_type': 'temporal_concat',
|
| 51 |
+
'train_videos_per_class': 500},
|
| 52 |
+
'preset': 'ssv2_coarse10'}],
|
| 53 |
+
'load_checkpoint': True,
|
| 54 |
+
'read_checkpoint': None,
|
| 55 |
+
'save_every_freq': 5,
|
| 56 |
+
'seed': 239,
|
| 57 |
+
'use_sdpa': True,
|
| 58 |
+
'use_wandb': True,
|
| 59 |
+
'wandb_project': 'vjepa_ablation'},
|
| 60 |
+
'metrics': {'std': {}},
|
| 61 |
+
'model': { 'class_token': True,
|
| 62 |
+
'freeze_backbone': False,
|
| 63 |
+
'is_causal': True,
|
| 64 |
+
'model_name': 'vit_large_patch14_dinov2_lvd142m',
|
| 65 |
+
'pe_type': '2d',
|
| 66 |
+
'pred_depth': 12,
|
| 67 |
+
'pred_embed_dim': 384,
|
| 68 |
+
'pred_num_heads': 6,
|
| 69 |
+
'predictor': 'v2_cross',
|
| 70 |
+
'stem_type': '2d',
|
| 71 |
+
'target_embed_dim': 1024,
|
| 72 |
+
'target_kind': 'ema',
|
| 73 |
+
'uniform_power': True,
|
| 74 |
+
'use_activation_checkpointing': True,
|
| 75 |
+
'use_alibi': False,
|
| 76 |
+
'use_lora': False,
|
| 77 |
+
'use_mask_tokens': True,
|
| 78 |
+
'use_rope': False,
|
| 79 |
+
'use_sdpa': True,
|
| 80 |
+
'zero_init_mask_tokens': True},
|
| 81 |
+
'nodes': 8,
|
| 82 |
+
'optimization': { 'clip_grad': None,
|
| 83 |
+
'ema': [1.0, 0.99925],
|
| 84 |
+
'ema_hold_epochs': 0,
|
| 85 |
+
'ema_ramp_epochs': 30,
|
| 86 |
+
'encoder_layer_decay': 0.9,
|
| 87 |
+
'epochs': 60,
|
| 88 |
+
'final_lr': 0.0005,
|
| 89 |
+
'final_weight_decay': 0.04,
|
| 90 |
+
'ipe': 300,
|
| 91 |
+
'ipe_scale': 1.0,
|
| 92 |
+
'lr': 0.0005,
|
| 93 |
+
'predictor_lr_scale': 3.0,
|
| 94 |
+
'start_lr': 5e-05,
|
| 95 |
+
'warmup': 4,
|
| 96 |
+
'weight_decay': 0.04},
|
| 97 |
+
'tasks_per_node': 4}
|
| 98 |
+
INFO:root:Running pre-training of app: vjepa
|
| 99 |
+
[INFO ][2026-06-11 05:16:17][app.vjepa.train ][main ] which_dtype='bfloat16'
|
| 100 |
+
[INFO ][2026-06-11 05:16:17][app.vjepa.train ][main ] Disabling persistent_workers (incompatible with KNN eval)
|
| 101 |
+
[INFO ][2026-06-11 05:16:17][app.vjepa.train ][main ] NCCL_SOCKET_IFNAME=None
|
| 102 |
+
[INFO ][2026-06-11 05:16:18][app.vjepa.train ][main ] Initialized (rank/world-size) 13/32, tasks_per_node=4
|
| 103 |
+
[WARNING ][2026-06-11 05:16:25][experiments.stmodels.vision_transformers_v3][_build_dinov2_vit ] DINOv2 is an image model — fwd_chunk is unset so all frames will be processed jointly. Set fwd_chunk=1 for per-frame (standard) processing.
|
| 104 |
+
[INFO ][2026-06-11 05:16:39][experiments.stmodels.vision_transformers_v3][_build_dinov2_vit ] Loading pretrained weights for vit_large_patch14_dinov2_lvd142m from timm (vit_large_patch14_dinov2)
|
| 105 |
+
[INFO ][2026-06-11 05:16:52][timm.models._builder][load_pretrained ] Loading pretrained weights from Hugging Face hub (timm/vit_large_patch14_dinov2.lvd142m)
|
| 106 |
+
[INFO ][2026-06-11 05:16:52][timm.models._hub ][load_state_dict_from_hf ] [timm/vit_large_patch14_dinov2.lvd142m] Safe alternative available for 'pytorch_model.bin' (as 'model.safetensors'). Loading weights using safetensors.
|
| 107 |
+
[INFO ][2026-06-11 05:16:53][timm.layers.pos_embed][resample_abs_pos_embed ] Resized position embedding: (37, 37) to (16, 16).
|
| 108 |
+
[INFO ][2026-06-11 05:17:34][root ][init_video_model ] ViTMultiSeqWrapper(
|
| 109 |
+
(backbone): VisionTransformer(
|
| 110 |
+
(patch_embed): PatchEmbed3D(
|
| 111 |
+
(proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
|
| 112 |
+
)
|
| 113 |
+
(blocks): ModuleList(
|
| 114 |
+
(0-23): 24 x Block(
|
| 115 |
+
(norm1): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
|
| 116 |
+
(norm2): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
|
| 117 |
+
(drop_path1): Identity()
|
| 118 |
+
(drop_path2): Identity()
|
| 119 |
+
(attn): Attention(
|
| 120 |
+
(qkv): Linear(in_features=1024, out_features=3072, bias=True)
|
| 121 |
+
(attn_drop): Dropout(p=0.0, inplace=False)
|
| 122 |
+
(proj): Linear(in_features=1024, out_features=1024, bias=True)
|
| 123 |
+
(proj_drop): Dropout(p=0.0, inplace=False)
|
| 124 |
+
)
|
| 125 |
+
(mlp): MLP(
|
| 126 |
+
(fc1): Linear(in_features=1024, out_features=4096, bias=True)
|
| 127 |
+
(act): GELU(approximate='none')
|
| 128 |
+
(fc2): Linear(in_features=4096, out_features=1024, bias=True)
|
| 129 |
+
(drop): Dropout(p=0.0, inplace=False)
|
| 130 |
+
)
|
| 131 |
+
)
|
| 132 |
+
)
|
| 133 |
+
(norm): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
|
| 134 |
+
)
|
| 135 |
+
)
|
| 136 |
+
[INFO ][2026-06-11 05:17:34][root ][init_video_model ] VJEPAPredictorMultiSeqWrapper(
|
| 137 |
+
(backbone): PredictorV2(
|
| 138 |
+
(predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
|
| 139 |
+
(mask_tokens): ParameterList(
|
| 140 |
+
(0): Parameter containing: [torch.float32 of size 1x384 (cuda:0)]
|
| 141 |
+
(1): Parameter containing: [torch.float32 of size 1x384 (cuda:0)]
|
| 142 |
+
(2): Parameter containing: [torch.float32 of size 1x384 (cuda:0)]
|
| 143 |
+
(3): Parameter containing: [torch.float32 of size 1x384 (cuda:0)]
|
| 144 |
+
)
|
| 145 |
+
(predictor_blocks): ModuleList(
|
| 146 |
+
(0-11): 12 x Block(
|
| 147 |
+
(residual1): EfficientResidual(
|
| 148 |
+
(norm): LayerNorm((384,), eps=1e-05, elementwise_affine=True)
|
| 149 |
+
(fn): Attention(
|
| 150 |
+
(q_proj): Linear(in_features=384, out_features=384, bias=False)
|
| 151 |
+
(k_proj): Linear(in_features=384, out_features=384, bias=False)
|
| 152 |
+
(v_proj): Linear(in_features=384, out_features=384, bias=False)
|
| 153 |
+
(proj): Linear(in_features=384, out_features=384, bias=False)
|
| 154 |
+
(rope): Rope()
|
| 155 |
+
)
|
| 156 |
+
)
|
| 157 |
+
(residual2): EfficientResidual(
|
| 158 |
+
(norm): LayerNorm((384,), eps=1e-05, elementwise_affine=True)
|
| 159 |
+
(fn): MLP(
|
| 160 |
+
(fc1): Linear(in_features=384, out_features=1536, bias=False)
|
| 161 |
+
(act): GELU(approximate='none')
|
| 162 |
+
(fc2): Linear(in_features=1536, out_features=384, bias=False)
|
| 163 |
+
(drop): Dropout(p=0.0, inplace=False)
|
| 164 |
+
)
|
| 165 |
+
)
|
| 166 |
+
)
|
| 167 |
+
)
|
| 168 |
+
(predictor_norm): LayerNorm((384,), eps=1e-05, elementwise_affine=True)
|
| 169 |
+
(predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
|
| 170 |
+
)
|
| 171 |
+
)
|
| 172 |
+
[INFO ][2026-06-11 05:17:34][root ][init_video_model ] ViTMultiSeqWrapper(
|
| 173 |
+
(backbone): VisionTransformer(
|
| 174 |
+
(patch_embed): PatchEmbed3D(
|
| 175 |
+
(proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
|
| 176 |
+
)
|
| 177 |
+
(blocks): ModuleList(
|
| 178 |
+
(0-23): 24 x Block(
|
| 179 |
+
(norm1): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
|
| 180 |
+
(norm2): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
|
| 181 |
+
(drop_path1): Identity()
|
| 182 |
+
(drop_path2): Identity()
|
| 183 |
+
(attn): Attention(
|
| 184 |
+
(qkv): Linear(in_features=1024, out_features=3072, bias=True)
|
| 185 |
+
(attn_drop): Dropout(p=0.0, inplace=False)
|
| 186 |
+
(proj): Linear(in_features=1024, out_features=1024, bias=True)
|
| 187 |
+
(proj_drop): Dropout(p=0.0, inplace=False)
|
| 188 |
+
)
|
| 189 |
+
(mlp): MLP(
|
| 190 |
+
(fc1): Linear(in_features=1024, out_features=4096, bias=True)
|
| 191 |
+
(act): GELU(approximate='none')
|
| 192 |
+
(fc2): Linear(in_features=4096, out_features=1024, bias=True)
|
| 193 |
+
(drop): Dropout(p=0.0, inplace=False)
|
| 194 |
+
)
|
| 195 |
+
)
|
| 196 |
+
)
|
| 197 |
+
(norm): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
|
| 198 |
+
)
|
| 199 |
+
)
|
| 200 |
+
[INFO ][2026-06-11 05:17:34][root ][init_video_model ] Encoder number of parameters: 302964736
|
| 201 |
+
[INFO ][2026-06-11 05:17:34][root ][init_video_model ] Predictor number of parameters: 22042240
|
| 202 |
+
[INFO ][2026-06-11 05:17:34][root ][init_video_model ] Target encoder number of parameters: 302964736
|
| 203 |
+
[INFO ][2026-06-11 05:17:34][root ][make_videodataset ] VideoDataset dataset created
|
| 204 |
+
[INFO ][2026-06-11 05:17:34][WeightedSampler ][__init__ ] Using DistributedWeightedSampler with rank 13 / 32
|
| 205 |
+
[INFO ][2026-06-11 05:17:34][root ][make_videodataset ] VideoDataset unsupervised data loader created
|
| 206 |
+
[INFO ][2026-06-11 05:17:34][app.vjepa.train ][main ] iterations per epoch/dataset length: 300/392
|
| 207 |
+
[INFO ][2026-06-11 05:17:34][app.vjepa.train ][main ] Wrapping models in DDP (rank 13)...
|
| 208 |
+
[INFO ][2026-06-11 05:17:35][app.vjepa.train ][main ] Initializing loader...
|
inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/job_45423709/45423709_14_log.err
ADDED
|
@@ -0,0 +1,13 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
|
| 2 |
+
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
|
| 3 |
+
/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/utils.py:888: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
|
| 4 |
+
scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
|
| 5 |
+
/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py:983: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
|
| 6 |
+
with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
|
| 7 |
+
[rank14]:[E611 13:45:39.661896646 ProcessGroupNCCL.cpp:1825] [PG ID 0 PG GUID 0(default_pg) Rank 14] Observed flight recorder dump signal from another rank via TCPStore.
|
| 8 |
+
[rank14]:[E611 13:45:39.663092437 ProcessGroupNCCL.cpp:1890] [PG ID 0 PG GUID 0(default_pg) Rank 14] Received a dump signal due to a collective timeout from rank 20 and we will try our best to dump the debug info. Last enqueued NCCL work: 410188, last completed NCCL work: 410188.This is most likely caused by incorrect usages of collectives, e.g., wrong sizes used across ranks, the order of collectives is not same for all ranks or the scheduled collective, for some reason, didn't run. Additionally, this can be caused by GIL deadlock or other reasons such as network errors or bugs in the communications library (e.g. NCCL), etc.
|
| 9 |
+
[rank14]:[E611 13:45:39.664592685 ProcessGroupNCCL.cpp:1606] [PG ID 0 PG GUID 0(default_pg) Rank 14] ProcessGroupNCCL preparing to dump debug info. Include stack trace: 1, only active collectives: 0
|
| 10 |
+
[rank14]:[E611 13:45:54.664117847 ProcessGroupNCCL.cpp:1367] [PG ID 0 PG GUID 0(default_pg) Rank 14] Future for Flight recorder dump in heartbeatMonitor timed out after 15000 ms
|
| 11 |
+
[rank14]:[E611 13:45:54.664181789 ProcessGroupNCCL.cpp:1606] [PG ID 0 PG GUID 0(default_pg) Rank 14] ProcessGroupNCCL preparing to dump debug info. Include stack trace: 0, only active collectives: 0
|
| 12 |
+
[rank14]:[E611 13:46:09.664242802 ProcessGroupNCCL.cpp:1367] [PG ID 0 PG GUID 0(default_pg) Rank 14] Future for Flight recorder dump in heartbeatMonitor timed out after 15000 ms
|
| 13 |
+
[rank14]:[F611 13:54:09.665669564 ProcessGroupNCCL.cpp:1631] [PG ID 0 PG GUID 0(default_pg) Rank 14] [PG ID 0 PG GUID 0(default_pg) Rank 14] Terminating the process after attempting to dump debug info, due to collective timeout or exception.
|
inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/job_45423709/45423709_14_log.out
ADDED
|
@@ -0,0 +1,208 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
submitit INFO (2026-06-11 05:16:06,427) - Starting with JobEnvironment(job_id=45423709, hostname=lrdn1244.leonardo.local, local_rank=2(4), node=3(8), global_rank=14(32))
|
| 2 |
+
submitit INFO (2026-06-11 05:16:06,428) - Loading pickle: /leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/job_45423709/45423709_submitted.pkl
|
| 3 |
+
INFO:root:loaded pretrain params...
|
| 4 |
+
{ 'app': 'vjepa',
|
| 5 |
+
'cpus_per_task': 8,
|
| 6 |
+
'data': { 'batch_size': 32,
|
| 7 |
+
'crop_size': 224,
|
| 8 |
+
'dataset_fpcs': [16, 16],
|
| 9 |
+
'dataset_type': 'VideoDataset',
|
| 10 |
+
'datasets': [ '/leonardo_scratch/fast/EUHPC_D35_082/data/k400/train.csv',
|
| 11 |
+
'/leonardo_scratch/fast/EUHPC_D35_082/data/ssv2/train.csv'],
|
| 12 |
+
'datasets_weights': [0.65, 0.35],
|
| 13 |
+
'fps': 4,
|
| 14 |
+
'num_workers': 8,
|
| 15 |
+
'patch_size': 14,
|
| 16 |
+
'persistent_workers': True,
|
| 17 |
+
'pin_mem': True,
|
| 18 |
+
'tubelet_size': 1},
|
| 19 |
+
'data_aug': { 'auto_augment': False,
|
| 20 |
+
'motion_shift': False,
|
| 21 |
+
'random_resize_aspect_ratio': [0.75, 1.35],
|
| 22 |
+
'random_resize_scale': [0.3, 1.0],
|
| 23 |
+
'reprob': 0.0},
|
| 24 |
+
'folder': '/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3',
|
| 25 |
+
'loss': {'loss_exp': 1.0},
|
| 26 |
+
'mask': [ { 'aspect_ratio': [0.75, 1.5],
|
| 27 |
+
'full_complement': False,
|
| 28 |
+
'max_keep': None,
|
| 29 |
+
'max_temporal_keep': 1.0,
|
| 30 |
+
'num_blocks': 8,
|
| 31 |
+
'spatial_scale': [0.15, 0.15],
|
| 32 |
+
'temporal_scale': [1.0, 1.0]},
|
| 33 |
+
{ 'aspect_ratio': [0.75, 1.5],
|
| 34 |
+
'full_complement': False,
|
| 35 |
+
'max_keep': None,
|
| 36 |
+
'max_temporal_keep': 1.0,
|
| 37 |
+
'num_blocks': 2,
|
| 38 |
+
'spatial_scale': [0.7, 0.7],
|
| 39 |
+
'temporal_scale': [1.0, 1.0]}],
|
| 40 |
+
'mem_per_gpu': '120G',
|
| 41 |
+
'meta': { 'dtype': 'bfloat16',
|
| 42 |
+
'knn_eval_epoch0': False,
|
| 43 |
+
'knn_eval_freq': 5,
|
| 44 |
+
'knn_eval_presets': [ { 'config': { 'batch_size': 8,
|
| 45 |
+
'dataset_train': '/leonardo_scratch/fast/EUHPC_D35_082/data/ssv2/train_coarse10.csv',
|
| 46 |
+
'dataset_val': '/leonardo_scratch/fast/EUHPC_D35_082/data/ssv2/val_coarse10.csv',
|
| 47 |
+
'eval_videos_per_class': 100,
|
| 48 |
+
'linear_probe': True,
|
| 49 |
+
'num_workers': 2,
|
| 50 |
+
'pool_type': 'temporal_concat',
|
| 51 |
+
'train_videos_per_class': 500},
|
| 52 |
+
'preset': 'ssv2_coarse10'}],
|
| 53 |
+
'load_checkpoint': True,
|
| 54 |
+
'read_checkpoint': None,
|
| 55 |
+
'save_every_freq': 5,
|
| 56 |
+
'seed': 239,
|
| 57 |
+
'use_sdpa': True,
|
| 58 |
+
'use_wandb': True,
|
| 59 |
+
'wandb_project': 'vjepa_ablation'},
|
| 60 |
+
'metrics': {'std': {}},
|
| 61 |
+
'model': { 'class_token': True,
|
| 62 |
+
'freeze_backbone': False,
|
| 63 |
+
'is_causal': True,
|
| 64 |
+
'model_name': 'vit_large_patch14_dinov2_lvd142m',
|
| 65 |
+
'pe_type': '2d',
|
| 66 |
+
'pred_depth': 12,
|
| 67 |
+
'pred_embed_dim': 384,
|
| 68 |
+
'pred_num_heads': 6,
|
| 69 |
+
'predictor': 'v2_cross',
|
| 70 |
+
'stem_type': '2d',
|
| 71 |
+
'target_embed_dim': 1024,
|
| 72 |
+
'target_kind': 'ema',
|
| 73 |
+
'uniform_power': True,
|
| 74 |
+
'use_activation_checkpointing': True,
|
| 75 |
+
'use_alibi': False,
|
| 76 |
+
'use_lora': False,
|
| 77 |
+
'use_mask_tokens': True,
|
| 78 |
+
'use_rope': False,
|
| 79 |
+
'use_sdpa': True,
|
| 80 |
+
'zero_init_mask_tokens': True},
|
| 81 |
+
'nodes': 8,
|
| 82 |
+
'optimization': { 'clip_grad': None,
|
| 83 |
+
'ema': [1.0, 0.99925],
|
| 84 |
+
'ema_hold_epochs': 0,
|
| 85 |
+
'ema_ramp_epochs': 30,
|
| 86 |
+
'encoder_layer_decay': 0.9,
|
| 87 |
+
'epochs': 60,
|
| 88 |
+
'final_lr': 0.0005,
|
| 89 |
+
'final_weight_decay': 0.04,
|
| 90 |
+
'ipe': 300,
|
| 91 |
+
'ipe_scale': 1.0,
|
| 92 |
+
'lr': 0.0005,
|
| 93 |
+
'predictor_lr_scale': 3.0,
|
| 94 |
+
'start_lr': 5e-05,
|
| 95 |
+
'warmup': 4,
|
| 96 |
+
'weight_decay': 0.04},
|
| 97 |
+
'tasks_per_node': 4}
|
| 98 |
+
INFO:root:Running pre-training of app: vjepa
|
| 99 |
+
[INFO ][2026-06-11 05:16:17][app.vjepa.train ][main ] which_dtype='bfloat16'
|
| 100 |
+
[INFO ][2026-06-11 05:16:17][app.vjepa.train ][main ] Disabling persistent_workers (incompatible with KNN eval)
|
| 101 |
+
[INFO ][2026-06-11 05:16:17][app.vjepa.train ][main ] NCCL_SOCKET_IFNAME=None
|
| 102 |
+
[INFO ][2026-06-11 05:16:18][app.vjepa.train ][main ] Initialized (rank/world-size) 14/32, tasks_per_node=4
|
| 103 |
+
[WARNING ][2026-06-11 05:16:25][experiments.stmodels.vision_transformers_v3][_build_dinov2_vit ] DINOv2 is an image model — fwd_chunk is unset so all frames will be processed jointly. Set fwd_chunk=1 for per-frame (standard) processing.
|
| 104 |
+
[INFO ][2026-06-11 05:16:39][experiments.stmodels.vision_transformers_v3][_build_dinov2_vit ] Loading pretrained weights for vit_large_patch14_dinov2_lvd142m from timm (vit_large_patch14_dinov2)
|
| 105 |
+
[INFO ][2026-06-11 05:16:52][timm.models._builder][load_pretrained ] Loading pretrained weights from Hugging Face hub (timm/vit_large_patch14_dinov2.lvd142m)
|
| 106 |
+
[INFO ][2026-06-11 05:16:52][timm.models._hub ][load_state_dict_from_hf ] [timm/vit_large_patch14_dinov2.lvd142m] Safe alternative available for 'pytorch_model.bin' (as 'model.safetensors'). Loading weights using safetensors.
|
| 107 |
+
[INFO ][2026-06-11 05:16:53][timm.layers.pos_embed][resample_abs_pos_embed ] Resized position embedding: (37, 37) to (16, 16).
|
| 108 |
+
[INFO ][2026-06-11 05:17:34][root ][init_video_model ] ViTMultiSeqWrapper(
|
| 109 |
+
(backbone): VisionTransformer(
|
| 110 |
+
(patch_embed): PatchEmbed3D(
|
| 111 |
+
(proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
|
| 112 |
+
)
|
| 113 |
+
(blocks): ModuleList(
|
| 114 |
+
(0-23): 24 x Block(
|
| 115 |
+
(norm1): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
|
| 116 |
+
(norm2): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
|
| 117 |
+
(drop_path1): Identity()
|
| 118 |
+
(drop_path2): Identity()
|
| 119 |
+
(attn): Attention(
|
| 120 |
+
(qkv): Linear(in_features=1024, out_features=3072, bias=True)
|
| 121 |
+
(attn_drop): Dropout(p=0.0, inplace=False)
|
| 122 |
+
(proj): Linear(in_features=1024, out_features=1024, bias=True)
|
| 123 |
+
(proj_drop): Dropout(p=0.0, inplace=False)
|
| 124 |
+
)
|
| 125 |
+
(mlp): MLP(
|
| 126 |
+
(fc1): Linear(in_features=1024, out_features=4096, bias=True)
|
| 127 |
+
(act): GELU(approximate='none')
|
| 128 |
+
(fc2): Linear(in_features=4096, out_features=1024, bias=True)
|
| 129 |
+
(drop): Dropout(p=0.0, inplace=False)
|
| 130 |
+
)
|
| 131 |
+
)
|
| 132 |
+
)
|
| 133 |
+
(norm): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
|
| 134 |
+
)
|
| 135 |
+
)
|
| 136 |
+
[INFO ][2026-06-11 05:17:34][root ][init_video_model ] VJEPAPredictorMultiSeqWrapper(
|
| 137 |
+
(backbone): PredictorV2(
|
| 138 |
+
(predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
|
| 139 |
+
(mask_tokens): ParameterList(
|
| 140 |
+
(0): Parameter containing: [torch.float32 of size 1x384 (cuda:0)]
|
| 141 |
+
(1): Parameter containing: [torch.float32 of size 1x384 (cuda:0)]
|
| 142 |
+
(2): Parameter containing: [torch.float32 of size 1x384 (cuda:0)]
|
| 143 |
+
(3): Parameter containing: [torch.float32 of size 1x384 (cuda:0)]
|
| 144 |
+
)
|
| 145 |
+
(predictor_blocks): ModuleList(
|
| 146 |
+
(0-11): 12 x Block(
|
| 147 |
+
(residual1): EfficientResidual(
|
| 148 |
+
(norm): LayerNorm((384,), eps=1e-05, elementwise_affine=True)
|
| 149 |
+
(fn): Attention(
|
| 150 |
+
(q_proj): Linear(in_features=384, out_features=384, bias=False)
|
| 151 |
+
(k_proj): Linear(in_features=384, out_features=384, bias=False)
|
| 152 |
+
(v_proj): Linear(in_features=384, out_features=384, bias=False)
|
| 153 |
+
(proj): Linear(in_features=384, out_features=384, bias=False)
|
| 154 |
+
(rope): Rope()
|
| 155 |
+
)
|
| 156 |
+
)
|
| 157 |
+
(residual2): EfficientResidual(
|
| 158 |
+
(norm): LayerNorm((384,), eps=1e-05, elementwise_affine=True)
|
| 159 |
+
(fn): MLP(
|
| 160 |
+
(fc1): Linear(in_features=384, out_features=1536, bias=False)
|
| 161 |
+
(act): GELU(approximate='none')
|
| 162 |
+
(fc2): Linear(in_features=1536, out_features=384, bias=False)
|
| 163 |
+
(drop): Dropout(p=0.0, inplace=False)
|
| 164 |
+
)
|
| 165 |
+
)
|
| 166 |
+
)
|
| 167 |
+
)
|
| 168 |
+
(predictor_norm): LayerNorm((384,), eps=1e-05, elementwise_affine=True)
|
| 169 |
+
(predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
|
| 170 |
+
)
|
| 171 |
+
)
|
| 172 |
+
[INFO ][2026-06-11 05:17:34][root ][init_video_model ] ViTMultiSeqWrapper(
|
| 173 |
+
(backbone): VisionTransformer(
|
| 174 |
+
(patch_embed): PatchEmbed3D(
|
| 175 |
+
(proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
|
| 176 |
+
)
|
| 177 |
+
(blocks): ModuleList(
|
| 178 |
+
(0-23): 24 x Block(
|
| 179 |
+
(norm1): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
|
| 180 |
+
(norm2): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
|
| 181 |
+
(drop_path1): Identity()
|
| 182 |
+
(drop_path2): Identity()
|
| 183 |
+
(attn): Attention(
|
| 184 |
+
(qkv): Linear(in_features=1024, out_features=3072, bias=True)
|
| 185 |
+
(attn_drop): Dropout(p=0.0, inplace=False)
|
| 186 |
+
(proj): Linear(in_features=1024, out_features=1024, bias=True)
|
| 187 |
+
(proj_drop): Dropout(p=0.0, inplace=False)
|
| 188 |
+
)
|
| 189 |
+
(mlp): MLP(
|
| 190 |
+
(fc1): Linear(in_features=1024, out_features=4096, bias=True)
|
| 191 |
+
(act): GELU(approximate='none')
|
| 192 |
+
(fc2): Linear(in_features=4096, out_features=1024, bias=True)
|
| 193 |
+
(drop): Dropout(p=0.0, inplace=False)
|
| 194 |
+
)
|
| 195 |
+
)
|
| 196 |
+
)
|
| 197 |
+
(norm): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
|
| 198 |
+
)
|
| 199 |
+
)
|
| 200 |
+
[INFO ][2026-06-11 05:17:34][root ][init_video_model ] Encoder number of parameters: 302964736
|
| 201 |
+
[INFO ][2026-06-11 05:17:34][root ][init_video_model ] Predictor number of parameters: 22042240
|
| 202 |
+
[INFO ][2026-06-11 05:17:34][root ][init_video_model ] Target encoder number of parameters: 302964736
|
| 203 |
+
[INFO ][2026-06-11 05:17:34][root ][make_videodataset ] VideoDataset dataset created
|
| 204 |
+
[INFO ][2026-06-11 05:17:34][WeightedSampler ][__init__ ] Using DistributedWeightedSampler with rank 14 / 32
|
| 205 |
+
[INFO ][2026-06-11 05:17:34][root ][make_videodataset ] VideoDataset unsupervised data loader created
|
| 206 |
+
[INFO ][2026-06-11 05:17:34][app.vjepa.train ][main ] iterations per epoch/dataset length: 300/392
|
| 207 |
+
[INFO ][2026-06-11 05:17:34][app.vjepa.train ][main ] Wrapping models in DDP (rank 14)...
|
| 208 |
+
[INFO ][2026-06-11 05:17:35][app.vjepa.train ][main ] Initializing loader...
|
inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/job_45423709/45423709_15_log.err
ADDED
|
@@ -0,0 +1,13 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
|
| 2 |
+
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
|
| 3 |
+
/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/utils.py:888: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
|
| 4 |
+
scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
|
| 5 |
+
/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py:983: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
|
| 6 |
+
with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
|
| 7 |
+
[rank15]:[E611 13:45:39.661877651 ProcessGroupNCCL.cpp:1825] [PG ID 0 PG GUID 0(default_pg) Rank 15] Observed flight recorder dump signal from another rank via TCPStore.
|
| 8 |
+
[rank15]:[E611 13:45:39.663084480 ProcessGroupNCCL.cpp:1890] [PG ID 0 PG GUID 0(default_pg) Rank 15] Received a dump signal due to a collective timeout from rank 20 and we will try our best to dump the debug info. Last enqueued NCCL work: 410189, last completed NCCL work: 410188.This is most likely caused by incorrect usages of collectives, e.g., wrong sizes used across ranks, the order of collectives is not same for all ranks or the scheduled collective, for some reason, didn't run. Additionally, this can be caused by GIL deadlock or other reasons such as network errors or bugs in the communications library (e.g. NCCL), etc.
|
| 9 |
+
[rank15]:[E611 13:45:39.664588339 ProcessGroupNCCL.cpp:1606] [PG ID 0 PG GUID 0(default_pg) Rank 15] ProcessGroupNCCL preparing to dump debug info. Include stack trace: 1, only active collectives: 0
|
| 10 |
+
[rank15]:[E611 13:45:54.664119363 ProcessGroupNCCL.cpp:1367] [PG ID 0 PG GUID 0(default_pg) Rank 15] Future for Flight recorder dump in heartbeatMonitor timed out after 15000 ms
|
| 11 |
+
[rank15]:[E611 13:45:54.664196625 ProcessGroupNCCL.cpp:1606] [PG ID 0 PG GUID 0(default_pg) Rank 15] ProcessGroupNCCL preparing to dump debug info. Include stack trace: 0, only active collectives: 0
|
| 12 |
+
[rank15]:[E611 13:46:09.664289511 ProcessGroupNCCL.cpp:1367] [PG ID 0 PG GUID 0(default_pg) Rank 15] Future for Flight recorder dump in heartbeatMonitor timed out after 15000 ms
|
| 13 |
+
[rank15]:[F611 13:54:09.665412463 ProcessGroupNCCL.cpp:1631] [PG ID 0 PG GUID 0(default_pg) Rank 15] [PG ID 0 PG GUID 0(default_pg) Rank 15] Terminating the process after attempting to dump debug info, due to collective timeout or exception.
|
inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/job_45423709/45423709_15_log.out
ADDED
|
@@ -0,0 +1,208 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
submitit INFO (2026-06-11 05:16:06,428) - Starting with JobEnvironment(job_id=45423709, hostname=lrdn1244.leonardo.local, local_rank=3(4), node=3(8), global_rank=15(32))
|
| 2 |
+
submitit INFO (2026-06-11 05:16:06,428) - Loading pickle: /leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/job_45423709/45423709_submitted.pkl
|
| 3 |
+
INFO:root:loaded pretrain params...
|
| 4 |
+
{ 'app': 'vjepa',
|
| 5 |
+
'cpus_per_task': 8,
|
| 6 |
+
'data': { 'batch_size': 32,
|
| 7 |
+
'crop_size': 224,
|
| 8 |
+
'dataset_fpcs': [16, 16],
|
| 9 |
+
'dataset_type': 'VideoDataset',
|
| 10 |
+
'datasets': [ '/leonardo_scratch/fast/EUHPC_D35_082/data/k400/train.csv',
|
| 11 |
+
'/leonardo_scratch/fast/EUHPC_D35_082/data/ssv2/train.csv'],
|
| 12 |
+
'datasets_weights': [0.65, 0.35],
|
| 13 |
+
'fps': 4,
|
| 14 |
+
'num_workers': 8,
|
| 15 |
+
'patch_size': 14,
|
| 16 |
+
'persistent_workers': True,
|
| 17 |
+
'pin_mem': True,
|
| 18 |
+
'tubelet_size': 1},
|
| 19 |
+
'data_aug': { 'auto_augment': False,
|
| 20 |
+
'motion_shift': False,
|
| 21 |
+
'random_resize_aspect_ratio': [0.75, 1.35],
|
| 22 |
+
'random_resize_scale': [0.3, 1.0],
|
| 23 |
+
'reprob': 0.0},
|
| 24 |
+
'folder': '/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3',
|
| 25 |
+
'loss': {'loss_exp': 1.0},
|
| 26 |
+
'mask': [ { 'aspect_ratio': [0.75, 1.5],
|
| 27 |
+
'full_complement': False,
|
| 28 |
+
'max_keep': None,
|
| 29 |
+
'max_temporal_keep': 1.0,
|
| 30 |
+
'num_blocks': 8,
|
| 31 |
+
'spatial_scale': [0.15, 0.15],
|
| 32 |
+
'temporal_scale': [1.0, 1.0]},
|
| 33 |
+
{ 'aspect_ratio': [0.75, 1.5],
|
| 34 |
+
'full_complement': False,
|
| 35 |
+
'max_keep': None,
|
| 36 |
+
'max_temporal_keep': 1.0,
|
| 37 |
+
'num_blocks': 2,
|
| 38 |
+
'spatial_scale': [0.7, 0.7],
|
| 39 |
+
'temporal_scale': [1.0, 1.0]}],
|
| 40 |
+
'mem_per_gpu': '120G',
|
| 41 |
+
'meta': { 'dtype': 'bfloat16',
|
| 42 |
+
'knn_eval_epoch0': False,
|
| 43 |
+
'knn_eval_freq': 5,
|
| 44 |
+
'knn_eval_presets': [ { 'config': { 'batch_size': 8,
|
| 45 |
+
'dataset_train': '/leonardo_scratch/fast/EUHPC_D35_082/data/ssv2/train_coarse10.csv',
|
| 46 |
+
'dataset_val': '/leonardo_scratch/fast/EUHPC_D35_082/data/ssv2/val_coarse10.csv',
|
| 47 |
+
'eval_videos_per_class': 100,
|
| 48 |
+
'linear_probe': True,
|
| 49 |
+
'num_workers': 2,
|
| 50 |
+
'pool_type': 'temporal_concat',
|
| 51 |
+
'train_videos_per_class': 500},
|
| 52 |
+
'preset': 'ssv2_coarse10'}],
|
| 53 |
+
'load_checkpoint': True,
|
| 54 |
+
'read_checkpoint': None,
|
| 55 |
+
'save_every_freq': 5,
|
| 56 |
+
'seed': 239,
|
| 57 |
+
'use_sdpa': True,
|
| 58 |
+
'use_wandb': True,
|
| 59 |
+
'wandb_project': 'vjepa_ablation'},
|
| 60 |
+
'metrics': {'std': {}},
|
| 61 |
+
'model': { 'class_token': True,
|
| 62 |
+
'freeze_backbone': False,
|
| 63 |
+
'is_causal': True,
|
| 64 |
+
'model_name': 'vit_large_patch14_dinov2_lvd142m',
|
| 65 |
+
'pe_type': '2d',
|
| 66 |
+
'pred_depth': 12,
|
| 67 |
+
'pred_embed_dim': 384,
|
| 68 |
+
'pred_num_heads': 6,
|
| 69 |
+
'predictor': 'v2_cross',
|
| 70 |
+
'stem_type': '2d',
|
| 71 |
+
'target_embed_dim': 1024,
|
| 72 |
+
'target_kind': 'ema',
|
| 73 |
+
'uniform_power': True,
|
| 74 |
+
'use_activation_checkpointing': True,
|
| 75 |
+
'use_alibi': False,
|
| 76 |
+
'use_lora': False,
|
| 77 |
+
'use_mask_tokens': True,
|
| 78 |
+
'use_rope': False,
|
| 79 |
+
'use_sdpa': True,
|
| 80 |
+
'zero_init_mask_tokens': True},
|
| 81 |
+
'nodes': 8,
|
| 82 |
+
'optimization': { 'clip_grad': None,
|
| 83 |
+
'ema': [1.0, 0.99925],
|
| 84 |
+
'ema_hold_epochs': 0,
|
| 85 |
+
'ema_ramp_epochs': 30,
|
| 86 |
+
'encoder_layer_decay': 0.9,
|
| 87 |
+
'epochs': 60,
|
| 88 |
+
'final_lr': 0.0005,
|
| 89 |
+
'final_weight_decay': 0.04,
|
| 90 |
+
'ipe': 300,
|
| 91 |
+
'ipe_scale': 1.0,
|
| 92 |
+
'lr': 0.0005,
|
| 93 |
+
'predictor_lr_scale': 3.0,
|
| 94 |
+
'start_lr': 5e-05,
|
| 95 |
+
'warmup': 4,
|
| 96 |
+
'weight_decay': 0.04},
|
| 97 |
+
'tasks_per_node': 4}
|
| 98 |
+
INFO:root:Running pre-training of app: vjepa
|
| 99 |
+
[INFO ][2026-06-11 05:16:17][app.vjepa.train ][main ] which_dtype='bfloat16'
|
| 100 |
+
[INFO ][2026-06-11 05:16:17][app.vjepa.train ][main ] Disabling persistent_workers (incompatible with KNN eval)
|
| 101 |
+
[INFO ][2026-06-11 05:16:17][app.vjepa.train ][main ] NCCL_SOCKET_IFNAME=None
|
| 102 |
+
[INFO ][2026-06-11 05:16:18][app.vjepa.train ][main ] Initialized (rank/world-size) 15/32, tasks_per_node=4
|
| 103 |
+
[WARNING ][2026-06-11 05:16:25][experiments.stmodels.vision_transformers_v3][_build_dinov2_vit ] DINOv2 is an image model — fwd_chunk is unset so all frames will be processed jointly. Set fwd_chunk=1 for per-frame (standard) processing.
|
| 104 |
+
[INFO ][2026-06-11 05:16:39][experiments.stmodels.vision_transformers_v3][_build_dinov2_vit ] Loading pretrained weights for vit_large_patch14_dinov2_lvd142m from timm (vit_large_patch14_dinov2)
|
| 105 |
+
[INFO ][2026-06-11 05:16:52][timm.models._builder][load_pretrained ] Loading pretrained weights from Hugging Face hub (timm/vit_large_patch14_dinov2.lvd142m)
|
| 106 |
+
[INFO ][2026-06-11 05:16:52][timm.models._hub ][load_state_dict_from_hf ] [timm/vit_large_patch14_dinov2.lvd142m] Safe alternative available for 'pytorch_model.bin' (as 'model.safetensors'). Loading weights using safetensors.
|
| 107 |
+
[INFO ][2026-06-11 05:16:53][timm.layers.pos_embed][resample_abs_pos_embed ] Resized position embedding: (37, 37) to (16, 16).
|
| 108 |
+
[INFO ][2026-06-11 05:17:33][root ][init_video_model ] ViTMultiSeqWrapper(
|
| 109 |
+
(backbone): VisionTransformer(
|
| 110 |
+
(patch_embed): PatchEmbed3D(
|
| 111 |
+
(proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
|
| 112 |
+
)
|
| 113 |
+
(blocks): ModuleList(
|
| 114 |
+
(0-23): 24 x Block(
|
| 115 |
+
(norm1): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
|
| 116 |
+
(norm2): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
|
| 117 |
+
(drop_path1): Identity()
|
| 118 |
+
(drop_path2): Identity()
|
| 119 |
+
(attn): Attention(
|
| 120 |
+
(qkv): Linear(in_features=1024, out_features=3072, bias=True)
|
| 121 |
+
(attn_drop): Dropout(p=0.0, inplace=False)
|
| 122 |
+
(proj): Linear(in_features=1024, out_features=1024, bias=True)
|
| 123 |
+
(proj_drop): Dropout(p=0.0, inplace=False)
|
| 124 |
+
)
|
| 125 |
+
(mlp): MLP(
|
| 126 |
+
(fc1): Linear(in_features=1024, out_features=4096, bias=True)
|
| 127 |
+
(act): GELU(approximate='none')
|
| 128 |
+
(fc2): Linear(in_features=4096, out_features=1024, bias=True)
|
| 129 |
+
(drop): Dropout(p=0.0, inplace=False)
|
| 130 |
+
)
|
| 131 |
+
)
|
| 132 |
+
)
|
| 133 |
+
(norm): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
|
| 134 |
+
)
|
| 135 |
+
)
|
| 136 |
+
[INFO ][2026-06-11 05:17:33][root ][init_video_model ] VJEPAPredictorMultiSeqWrapper(
|
| 137 |
+
(backbone): PredictorV2(
|
| 138 |
+
(predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
|
| 139 |
+
(mask_tokens): ParameterList(
|
| 140 |
+
(0): Parameter containing: [torch.float32 of size 1x384 (cuda:0)]
|
| 141 |
+
(1): Parameter containing: [torch.float32 of size 1x384 (cuda:0)]
|
| 142 |
+
(2): Parameter containing: [torch.float32 of size 1x384 (cuda:0)]
|
| 143 |
+
(3): Parameter containing: [torch.float32 of size 1x384 (cuda:0)]
|
| 144 |
+
)
|
| 145 |
+
(predictor_blocks): ModuleList(
|
| 146 |
+
(0-11): 12 x Block(
|
| 147 |
+
(residual1): EfficientResidual(
|
| 148 |
+
(norm): LayerNorm((384,), eps=1e-05, elementwise_affine=True)
|
| 149 |
+
(fn): Attention(
|
| 150 |
+
(q_proj): Linear(in_features=384, out_features=384, bias=False)
|
| 151 |
+
(k_proj): Linear(in_features=384, out_features=384, bias=False)
|
| 152 |
+
(v_proj): Linear(in_features=384, out_features=384, bias=False)
|
| 153 |
+
(proj): Linear(in_features=384, out_features=384, bias=False)
|
| 154 |
+
(rope): Rope()
|
| 155 |
+
)
|
| 156 |
+
)
|
| 157 |
+
(residual2): EfficientResidual(
|
| 158 |
+
(norm): LayerNorm((384,), eps=1e-05, elementwise_affine=True)
|
| 159 |
+
(fn): MLP(
|
| 160 |
+
(fc1): Linear(in_features=384, out_features=1536, bias=False)
|
| 161 |
+
(act): GELU(approximate='none')
|
| 162 |
+
(fc2): Linear(in_features=1536, out_features=384, bias=False)
|
| 163 |
+
(drop): Dropout(p=0.0, inplace=False)
|
| 164 |
+
)
|
| 165 |
+
)
|
| 166 |
+
)
|
| 167 |
+
)
|
| 168 |
+
(predictor_norm): LayerNorm((384,), eps=1e-05, elementwise_affine=True)
|
| 169 |
+
(predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
|
| 170 |
+
)
|
| 171 |
+
)
|
| 172 |
+
[INFO ][2026-06-11 05:17:33][root ][init_video_model ] ViTMultiSeqWrapper(
|
| 173 |
+
(backbone): VisionTransformer(
|
| 174 |
+
(patch_embed): PatchEmbed3D(
|
| 175 |
+
(proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
|
| 176 |
+
)
|
| 177 |
+
(blocks): ModuleList(
|
| 178 |
+
(0-23): 24 x Block(
|
| 179 |
+
(norm1): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
|
| 180 |
+
(norm2): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
|
| 181 |
+
(drop_path1): Identity()
|
| 182 |
+
(drop_path2): Identity()
|
| 183 |
+
(attn): Attention(
|
| 184 |
+
(qkv): Linear(in_features=1024, out_features=3072, bias=True)
|
| 185 |
+
(attn_drop): Dropout(p=0.0, inplace=False)
|
| 186 |
+
(proj): Linear(in_features=1024, out_features=1024, bias=True)
|
| 187 |
+
(proj_drop): Dropout(p=0.0, inplace=False)
|
| 188 |
+
)
|
| 189 |
+
(mlp): MLP(
|
| 190 |
+
(fc1): Linear(in_features=1024, out_features=4096, bias=True)
|
| 191 |
+
(act): GELU(approximate='none')
|
| 192 |
+
(fc2): Linear(in_features=4096, out_features=1024, bias=True)
|
| 193 |
+
(drop): Dropout(p=0.0, inplace=False)
|
| 194 |
+
)
|
| 195 |
+
)
|
| 196 |
+
)
|
| 197 |
+
(norm): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
|
| 198 |
+
)
|
| 199 |
+
)
|
| 200 |
+
[INFO ][2026-06-11 05:17:33][root ][init_video_model ] Encoder number of parameters: 302964736
|
| 201 |
+
[INFO ][2026-06-11 05:17:33][root ][init_video_model ] Predictor number of parameters: 22042240
|
| 202 |
+
[INFO ][2026-06-11 05:17:33][root ][init_video_model ] Target encoder number of parameters: 302964736
|
| 203 |
+
[INFO ][2026-06-11 05:17:34][root ][make_videodataset ] VideoDataset dataset created
|
| 204 |
+
[INFO ][2026-06-11 05:17:34][WeightedSampler ][__init__ ] Using DistributedWeightedSampler with rank 15 / 32
|
| 205 |
+
[INFO ][2026-06-11 05:17:34][root ][make_videodataset ] VideoDataset unsupervised data loader created
|
| 206 |
+
[INFO ][2026-06-11 05:17:34][app.vjepa.train ][main ] iterations per epoch/dataset length: 300/392
|
| 207 |
+
[INFO ][2026-06-11 05:17:34][app.vjepa.train ][main ] Wrapping models in DDP (rank 15)...
|
| 208 |
+
[INFO ][2026-06-11 05:17:35][app.vjepa.train ][main ] Initializing loader...
|
inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/job_45423709/45423709_16_log.err
ADDED
|
@@ -0,0 +1,13 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
|
| 2 |
+
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
|
| 3 |
+
/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/utils.py:888: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
|
| 4 |
+
scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
|
| 5 |
+
/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py:983: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
|
| 6 |
+
with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
|
| 7 |
+
[rank16]:[E611 13:45:39.770256158 ProcessGroupNCCL.cpp:1825] [PG ID 0 PG GUID 0(default_pg) Rank 16] Observed flight recorder dump signal from another rank via TCPStore.
|
| 8 |
+
[rank16]:[E611 13:45:39.771489089 ProcessGroupNCCL.cpp:1890] [PG ID 0 PG GUID 0(default_pg) Rank 16] Received a dump signal due to a collective timeout from rank 20 and we will try our best to dump the debug info. Last enqueued NCCL work: 410188, last completed NCCL work: 410188.This is most likely caused by incorrect usages of collectives, e.g., wrong sizes used across ranks, the order of collectives is not same for all ranks or the scheduled collective, for some reason, didn't run. Additionally, this can be caused by GIL deadlock or other reasons such as network errors or bugs in the communications library (e.g. NCCL), etc.
|
| 9 |
+
[rank16]:[E611 13:45:39.771610499 ProcessGroupNCCL.cpp:1606] [PG ID 0 PG GUID 0(default_pg) Rank 16] ProcessGroupNCCL preparing to dump debug info. Include stack trace: 1, only active collectives: 0
|
| 10 |
+
[rank16]:[E611 13:45:54.771658391 ProcessGroupNCCL.cpp:1367] [PG ID 0 PG GUID 0(default_pg) Rank 16] Future for Flight recorder dump in heartbeatMonitor timed out after 15000 ms
|
| 11 |
+
[rank16]:[E611 13:45:54.771746849 ProcessGroupNCCL.cpp:1606] [PG ID 0 PG GUID 0(default_pg) Rank 16] ProcessGroupNCCL preparing to dump debug info. Include stack trace: 0, only active collectives: 0
|
| 12 |
+
[rank16]:[E611 13:46:09.771797259 ProcessGroupNCCL.cpp:1367] [PG ID 0 PG GUID 0(default_pg) Rank 16] Future for Flight recorder dump in heartbeatMonitor timed out after 15000 ms
|
| 13 |
+
[rank16]:[F611 13:54:09.772946524 ProcessGroupNCCL.cpp:1631] [PG ID 0 PG GUID 0(default_pg) Rank 16] [PG ID 0 PG GUID 0(default_pg) Rank 16] Terminating the process after attempting to dump debug info, due to collective timeout or exception.
|
inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/job_45423709/45423709_16_log.out
ADDED
|
@@ -0,0 +1,208 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
submitit INFO (2026-06-11 05:16:06,427) - Starting with JobEnvironment(job_id=45423709, hostname=lrdn1257.leonardo.local, local_rank=0(4), node=4(8), global_rank=16(32))
|
| 2 |
+
submitit INFO (2026-06-11 05:16:06,427) - Loading pickle: /leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/job_45423709/45423709_submitted.pkl
|
| 3 |
+
INFO:root:loaded pretrain params...
|
| 4 |
+
{ 'app': 'vjepa',
|
| 5 |
+
'cpus_per_task': 8,
|
| 6 |
+
'data': { 'batch_size': 32,
|
| 7 |
+
'crop_size': 224,
|
| 8 |
+
'dataset_fpcs': [16, 16],
|
| 9 |
+
'dataset_type': 'VideoDataset',
|
| 10 |
+
'datasets': [ '/leonardo_scratch/fast/EUHPC_D35_082/data/k400/train.csv',
|
| 11 |
+
'/leonardo_scratch/fast/EUHPC_D35_082/data/ssv2/train.csv'],
|
| 12 |
+
'datasets_weights': [0.65, 0.35],
|
| 13 |
+
'fps': 4,
|
| 14 |
+
'num_workers': 8,
|
| 15 |
+
'patch_size': 14,
|
| 16 |
+
'persistent_workers': True,
|
| 17 |
+
'pin_mem': True,
|
| 18 |
+
'tubelet_size': 1},
|
| 19 |
+
'data_aug': { 'auto_augment': False,
|
| 20 |
+
'motion_shift': False,
|
| 21 |
+
'random_resize_aspect_ratio': [0.75, 1.35],
|
| 22 |
+
'random_resize_scale': [0.3, 1.0],
|
| 23 |
+
'reprob': 0.0},
|
| 24 |
+
'folder': '/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3',
|
| 25 |
+
'loss': {'loss_exp': 1.0},
|
| 26 |
+
'mask': [ { 'aspect_ratio': [0.75, 1.5],
|
| 27 |
+
'full_complement': False,
|
| 28 |
+
'max_keep': None,
|
| 29 |
+
'max_temporal_keep': 1.0,
|
| 30 |
+
'num_blocks': 8,
|
| 31 |
+
'spatial_scale': [0.15, 0.15],
|
| 32 |
+
'temporal_scale': [1.0, 1.0]},
|
| 33 |
+
{ 'aspect_ratio': [0.75, 1.5],
|
| 34 |
+
'full_complement': False,
|
| 35 |
+
'max_keep': None,
|
| 36 |
+
'max_temporal_keep': 1.0,
|
| 37 |
+
'num_blocks': 2,
|
| 38 |
+
'spatial_scale': [0.7, 0.7],
|
| 39 |
+
'temporal_scale': [1.0, 1.0]}],
|
| 40 |
+
'mem_per_gpu': '120G',
|
| 41 |
+
'meta': { 'dtype': 'bfloat16',
|
| 42 |
+
'knn_eval_epoch0': False,
|
| 43 |
+
'knn_eval_freq': 5,
|
| 44 |
+
'knn_eval_presets': [ { 'config': { 'batch_size': 8,
|
| 45 |
+
'dataset_train': '/leonardo_scratch/fast/EUHPC_D35_082/data/ssv2/train_coarse10.csv',
|
| 46 |
+
'dataset_val': '/leonardo_scratch/fast/EUHPC_D35_082/data/ssv2/val_coarse10.csv',
|
| 47 |
+
'eval_videos_per_class': 100,
|
| 48 |
+
'linear_probe': True,
|
| 49 |
+
'num_workers': 2,
|
| 50 |
+
'pool_type': 'temporal_concat',
|
| 51 |
+
'train_videos_per_class': 500},
|
| 52 |
+
'preset': 'ssv2_coarse10'}],
|
| 53 |
+
'load_checkpoint': True,
|
| 54 |
+
'read_checkpoint': None,
|
| 55 |
+
'save_every_freq': 5,
|
| 56 |
+
'seed': 239,
|
| 57 |
+
'use_sdpa': True,
|
| 58 |
+
'use_wandb': True,
|
| 59 |
+
'wandb_project': 'vjepa_ablation'},
|
| 60 |
+
'metrics': {'std': {}},
|
| 61 |
+
'model': { 'class_token': True,
|
| 62 |
+
'freeze_backbone': False,
|
| 63 |
+
'is_causal': True,
|
| 64 |
+
'model_name': 'vit_large_patch14_dinov2_lvd142m',
|
| 65 |
+
'pe_type': '2d',
|
| 66 |
+
'pred_depth': 12,
|
| 67 |
+
'pred_embed_dim': 384,
|
| 68 |
+
'pred_num_heads': 6,
|
| 69 |
+
'predictor': 'v2_cross',
|
| 70 |
+
'stem_type': '2d',
|
| 71 |
+
'target_embed_dim': 1024,
|
| 72 |
+
'target_kind': 'ema',
|
| 73 |
+
'uniform_power': True,
|
| 74 |
+
'use_activation_checkpointing': True,
|
| 75 |
+
'use_alibi': False,
|
| 76 |
+
'use_lora': False,
|
| 77 |
+
'use_mask_tokens': True,
|
| 78 |
+
'use_rope': False,
|
| 79 |
+
'use_sdpa': True,
|
| 80 |
+
'zero_init_mask_tokens': True},
|
| 81 |
+
'nodes': 8,
|
| 82 |
+
'optimization': { 'clip_grad': None,
|
| 83 |
+
'ema': [1.0, 0.99925],
|
| 84 |
+
'ema_hold_epochs': 0,
|
| 85 |
+
'ema_ramp_epochs': 30,
|
| 86 |
+
'encoder_layer_decay': 0.9,
|
| 87 |
+
'epochs': 60,
|
| 88 |
+
'final_lr': 0.0005,
|
| 89 |
+
'final_weight_decay': 0.04,
|
| 90 |
+
'ipe': 300,
|
| 91 |
+
'ipe_scale': 1.0,
|
| 92 |
+
'lr': 0.0005,
|
| 93 |
+
'predictor_lr_scale': 3.0,
|
| 94 |
+
'start_lr': 5e-05,
|
| 95 |
+
'warmup': 4,
|
| 96 |
+
'weight_decay': 0.04},
|
| 97 |
+
'tasks_per_node': 4}
|
| 98 |
+
INFO:root:Running pre-training of app: vjepa
|
| 99 |
+
[INFO ][2026-06-11 05:16:17][app.vjepa.train ][main ] which_dtype='bfloat16'
|
| 100 |
+
[INFO ][2026-06-11 05:16:17][app.vjepa.train ][main ] Disabling persistent_workers (incompatible with KNN eval)
|
| 101 |
+
[INFO ][2026-06-11 05:16:17][app.vjepa.train ][main ] NCCL_SOCKET_IFNAME=None
|
| 102 |
+
[INFO ][2026-06-11 05:16:18][app.vjepa.train ][main ] Initialized (rank/world-size) 16/32, tasks_per_node=4
|
| 103 |
+
[WARNING ][2026-06-11 05:16:25][experiments.stmodels.vision_transformers_v3][_build_dinov2_vit ] DINOv2 is an image model — fwd_chunk is unset so all frames will be processed jointly. Set fwd_chunk=1 for per-frame (standard) processing.
|
| 104 |
+
[INFO ][2026-06-11 05:16:40][experiments.stmodels.vision_transformers_v3][_build_dinov2_vit ] Loading pretrained weights for vit_large_patch14_dinov2_lvd142m from timm (vit_large_patch14_dinov2)
|
| 105 |
+
[INFO ][2026-06-11 05:16:53][timm.models._builder][load_pretrained ] Loading pretrained weights from Hugging Face hub (timm/vit_large_patch14_dinov2.lvd142m)
|
| 106 |
+
[INFO ][2026-06-11 05:16:53][timm.models._hub ][load_state_dict_from_hf ] [timm/vit_large_patch14_dinov2.lvd142m] Safe alternative available for 'pytorch_model.bin' (as 'model.safetensors'). Loading weights using safetensors.
|
| 107 |
+
[INFO ][2026-06-11 05:16:53][timm.layers.pos_embed][resample_abs_pos_embed ] Resized position embedding: (37, 37) to (16, 16).
|
| 108 |
+
[INFO ][2026-06-11 05:17:34][root ][init_video_model ] ViTMultiSeqWrapper(
|
| 109 |
+
(backbone): VisionTransformer(
|
| 110 |
+
(patch_embed): PatchEmbed3D(
|
| 111 |
+
(proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
|
| 112 |
+
)
|
| 113 |
+
(blocks): ModuleList(
|
| 114 |
+
(0-23): 24 x Block(
|
| 115 |
+
(norm1): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
|
| 116 |
+
(norm2): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
|
| 117 |
+
(drop_path1): Identity()
|
| 118 |
+
(drop_path2): Identity()
|
| 119 |
+
(attn): Attention(
|
| 120 |
+
(qkv): Linear(in_features=1024, out_features=3072, bias=True)
|
| 121 |
+
(attn_drop): Dropout(p=0.0, inplace=False)
|
| 122 |
+
(proj): Linear(in_features=1024, out_features=1024, bias=True)
|
| 123 |
+
(proj_drop): Dropout(p=0.0, inplace=False)
|
| 124 |
+
)
|
| 125 |
+
(mlp): MLP(
|
| 126 |
+
(fc1): Linear(in_features=1024, out_features=4096, bias=True)
|
| 127 |
+
(act): GELU(approximate='none')
|
| 128 |
+
(fc2): Linear(in_features=4096, out_features=1024, bias=True)
|
| 129 |
+
(drop): Dropout(p=0.0, inplace=False)
|
| 130 |
+
)
|
| 131 |
+
)
|
| 132 |
+
)
|
| 133 |
+
(norm): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
|
| 134 |
+
)
|
| 135 |
+
)
|
| 136 |
+
[INFO ][2026-06-11 05:17:34][root ][init_video_model ] VJEPAPredictorMultiSeqWrapper(
|
| 137 |
+
(backbone): PredictorV2(
|
| 138 |
+
(predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
|
| 139 |
+
(mask_tokens): ParameterList(
|
| 140 |
+
(0): Parameter containing: [torch.float32 of size 1x384 (cuda:0)]
|
| 141 |
+
(1): Parameter containing: [torch.float32 of size 1x384 (cuda:0)]
|
| 142 |
+
(2): Parameter containing: [torch.float32 of size 1x384 (cuda:0)]
|
| 143 |
+
(3): Parameter containing: [torch.float32 of size 1x384 (cuda:0)]
|
| 144 |
+
)
|
| 145 |
+
(predictor_blocks): ModuleList(
|
| 146 |
+
(0-11): 12 x Block(
|
| 147 |
+
(residual1): EfficientResidual(
|
| 148 |
+
(norm): LayerNorm((384,), eps=1e-05, elementwise_affine=True)
|
| 149 |
+
(fn): Attention(
|
| 150 |
+
(q_proj): Linear(in_features=384, out_features=384, bias=False)
|
| 151 |
+
(k_proj): Linear(in_features=384, out_features=384, bias=False)
|
| 152 |
+
(v_proj): Linear(in_features=384, out_features=384, bias=False)
|
| 153 |
+
(proj): Linear(in_features=384, out_features=384, bias=False)
|
| 154 |
+
(rope): Rope()
|
| 155 |
+
)
|
| 156 |
+
)
|
| 157 |
+
(residual2): EfficientResidual(
|
| 158 |
+
(norm): LayerNorm((384,), eps=1e-05, elementwise_affine=True)
|
| 159 |
+
(fn): MLP(
|
| 160 |
+
(fc1): Linear(in_features=384, out_features=1536, bias=False)
|
| 161 |
+
(act): GELU(approximate='none')
|
| 162 |
+
(fc2): Linear(in_features=1536, out_features=384, bias=False)
|
| 163 |
+
(drop): Dropout(p=0.0, inplace=False)
|
| 164 |
+
)
|
| 165 |
+
)
|
| 166 |
+
)
|
| 167 |
+
)
|
| 168 |
+
(predictor_norm): LayerNorm((384,), eps=1e-05, elementwise_affine=True)
|
| 169 |
+
(predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
|
| 170 |
+
)
|
| 171 |
+
)
|
| 172 |
+
[INFO ][2026-06-11 05:17:34][root ][init_video_model ] ViTMultiSeqWrapper(
|
| 173 |
+
(backbone): VisionTransformer(
|
| 174 |
+
(patch_embed): PatchEmbed3D(
|
| 175 |
+
(proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
|
| 176 |
+
)
|
| 177 |
+
(blocks): ModuleList(
|
| 178 |
+
(0-23): 24 x Block(
|
| 179 |
+
(norm1): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
|
| 180 |
+
(norm2): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
|
| 181 |
+
(drop_path1): Identity()
|
| 182 |
+
(drop_path2): Identity()
|
| 183 |
+
(attn): Attention(
|
| 184 |
+
(qkv): Linear(in_features=1024, out_features=3072, bias=True)
|
| 185 |
+
(attn_drop): Dropout(p=0.0, inplace=False)
|
| 186 |
+
(proj): Linear(in_features=1024, out_features=1024, bias=True)
|
| 187 |
+
(proj_drop): Dropout(p=0.0, inplace=False)
|
| 188 |
+
)
|
| 189 |
+
(mlp): MLP(
|
| 190 |
+
(fc1): Linear(in_features=1024, out_features=4096, bias=True)
|
| 191 |
+
(act): GELU(approximate='none')
|
| 192 |
+
(fc2): Linear(in_features=4096, out_features=1024, bias=True)
|
| 193 |
+
(drop): Dropout(p=0.0, inplace=False)
|
| 194 |
+
)
|
| 195 |
+
)
|
| 196 |
+
)
|
| 197 |
+
(norm): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
|
| 198 |
+
)
|
| 199 |
+
)
|
| 200 |
+
[INFO ][2026-06-11 05:17:34][root ][init_video_model ] Encoder number of parameters: 302964736
|
| 201 |
+
[INFO ][2026-06-11 05:17:34][root ][init_video_model ] Predictor number of parameters: 22042240
|
| 202 |
+
[INFO ][2026-06-11 05:17:34][root ][init_video_model ] Target encoder number of parameters: 302964736
|
| 203 |
+
[INFO ][2026-06-11 05:17:34][root ][make_videodataset ] VideoDataset dataset created
|
| 204 |
+
[INFO ][2026-06-11 05:17:34][WeightedSampler ][__init__ ] Using DistributedWeightedSampler with rank 16 / 32
|
| 205 |
+
[INFO ][2026-06-11 05:17:34][root ][make_videodataset ] VideoDataset unsupervised data loader created
|
| 206 |
+
[INFO ][2026-06-11 05:17:34][app.vjepa.train ][main ] iterations per epoch/dataset length: 300/392
|
| 207 |
+
[INFO ][2026-06-11 05:17:34][app.vjepa.train ][main ] Wrapping models in DDP (rank 16)...
|
| 208 |
+
[INFO ][2026-06-11 05:17:35][app.vjepa.train ][main ] Initializing loader...
|
inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/job_45423709/45423709_17_log.err
ADDED
|
@@ -0,0 +1,13 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
/leonardo/home/userexternal/dcanezil/vd/.venv/lib64/python3.11/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
|
| 2 |
+
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
|
| 3 |
+
/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/utils.py:888: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
|
| 4 |
+
scaler = torch.cuda.amp.GradScaler() if mixed_precision else None
|
| 5 |
+
/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/code/app/vjepa/train.py:983: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
|
| 6 |
+
with torch.cuda.amp.autocast(dtype=dtype, enabled=mixed_precision):
|
| 7 |
+
[rank17]:[E611 13:45:39.770240374 ProcessGroupNCCL.cpp:1825] [PG ID 0 PG GUID 0(default_pg) Rank 17] Observed flight recorder dump signal from another rank via TCPStore.
|
| 8 |
+
[rank17]:[E611 13:45:39.771483136 ProcessGroupNCCL.cpp:1890] [PG ID 0 PG GUID 0(default_pg) Rank 17] Received a dump signal due to a collective timeout from rank 20 and we will try our best to dump the debug info. Last enqueued NCCL work: 410188, last completed NCCL work: 410188.This is most likely caused by incorrect usages of collectives, e.g., wrong sizes used across ranks, the order of collectives is not same for all ranks or the scheduled collective, for some reason, didn't run. Additionally, this can be caused by GIL deadlock or other reasons such as network errors or bugs in the communications library (e.g. NCCL), etc.
|
| 9 |
+
[rank17]:[E611 13:45:39.771596628 ProcessGroupNCCL.cpp:1606] [PG ID 0 PG GUID 0(default_pg) Rank 17] ProcessGroupNCCL preparing to dump debug info. Include stack trace: 1, only active collectives: 0
|
| 10 |
+
[rank17]:[E611 13:45:54.771687361 ProcessGroupNCCL.cpp:1367] [PG ID 0 PG GUID 0(default_pg) Rank 17] Future for Flight recorder dump in heartbeatMonitor timed out after 15000 ms
|
| 11 |
+
[rank17]:[E611 13:45:54.771766657 ProcessGroupNCCL.cpp:1606] [PG ID 0 PG GUID 0(default_pg) Rank 17] ProcessGroupNCCL preparing to dump debug info. Include stack trace: 0, only active collectives: 0
|
| 12 |
+
[rank17]:[E611 13:46:09.771853903 ProcessGroupNCCL.cpp:1367] [PG ID 0 PG GUID 0(default_pg) Rank 17] Future for Flight recorder dump in heartbeatMonitor timed out after 15000 ms
|
| 13 |
+
[rank17]:[F611 13:54:09.772944303 ProcessGroupNCCL.cpp:1631] [PG ID 0 PG GUID 0(default_pg) Rank 17] [PG ID 0 PG GUID 0(default_pg) Rank 17] Terminating the process after attempting to dump debug info, due to collective timeout or exception.
|
inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/job_45423709/45423709_17_log.out
ADDED
|
@@ -0,0 +1,208 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
submitit INFO (2026-06-11 05:16:06,427) - Starting with JobEnvironment(job_id=45423709, hostname=lrdn1257.leonardo.local, local_rank=1(4), node=4(8), global_rank=17(32))
|
| 2 |
+
submitit INFO (2026-06-11 05:16:06,428) - Loading pickle: /leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3/job_45423709/45423709_submitted.pkl
|
| 3 |
+
INFO:root:loaded pretrain params...
|
| 4 |
+
{ 'app': 'vjepa',
|
| 5 |
+
'cpus_per_task': 8,
|
| 6 |
+
'data': { 'batch_size': 32,
|
| 7 |
+
'crop_size': 224,
|
| 8 |
+
'dataset_fpcs': [16, 16],
|
| 9 |
+
'dataset_type': 'VideoDataset',
|
| 10 |
+
'datasets': [ '/leonardo_scratch/fast/EUHPC_D35_082/data/k400/train.csv',
|
| 11 |
+
'/leonardo_scratch/fast/EUHPC_D35_082/data/ssv2/train.csv'],
|
| 12 |
+
'datasets_weights': [0.65, 0.35],
|
| 13 |
+
'fps': 4,
|
| 14 |
+
'num_workers': 8,
|
| 15 |
+
'patch_size': 14,
|
| 16 |
+
'persistent_workers': True,
|
| 17 |
+
'pin_mem': True,
|
| 18 |
+
'tubelet_size': 1},
|
| 19 |
+
'data_aug': { 'auto_augment': False,
|
| 20 |
+
'motion_shift': False,
|
| 21 |
+
'random_resize_aspect_ratio': [0.75, 1.35],
|
| 22 |
+
'random_resize_scale': [0.3, 1.0],
|
| 23 |
+
'reprob': 0.0},
|
| 24 |
+
'folder': '/leonardo_work/EUHPC_D35_082/runs/dcanezil/vjepa/inflated/d068_full_ema_dinov2l_ssv2k400_bs1024_16f_llrd0p9_emainv_ramp30_predlr3',
|
| 25 |
+
'loss': {'loss_exp': 1.0},
|
| 26 |
+
'mask': [ { 'aspect_ratio': [0.75, 1.5],
|
| 27 |
+
'full_complement': False,
|
| 28 |
+
'max_keep': None,
|
| 29 |
+
'max_temporal_keep': 1.0,
|
| 30 |
+
'num_blocks': 8,
|
| 31 |
+
'spatial_scale': [0.15, 0.15],
|
| 32 |
+
'temporal_scale': [1.0, 1.0]},
|
| 33 |
+
{ 'aspect_ratio': [0.75, 1.5],
|
| 34 |
+
'full_complement': False,
|
| 35 |
+
'max_keep': None,
|
| 36 |
+
'max_temporal_keep': 1.0,
|
| 37 |
+
'num_blocks': 2,
|
| 38 |
+
'spatial_scale': [0.7, 0.7],
|
| 39 |
+
'temporal_scale': [1.0, 1.0]}],
|
| 40 |
+
'mem_per_gpu': '120G',
|
| 41 |
+
'meta': { 'dtype': 'bfloat16',
|
| 42 |
+
'knn_eval_epoch0': False,
|
| 43 |
+
'knn_eval_freq': 5,
|
| 44 |
+
'knn_eval_presets': [ { 'config': { 'batch_size': 8,
|
| 45 |
+
'dataset_train': '/leonardo_scratch/fast/EUHPC_D35_082/data/ssv2/train_coarse10.csv',
|
| 46 |
+
'dataset_val': '/leonardo_scratch/fast/EUHPC_D35_082/data/ssv2/val_coarse10.csv',
|
| 47 |
+
'eval_videos_per_class': 100,
|
| 48 |
+
'linear_probe': True,
|
| 49 |
+
'num_workers': 2,
|
| 50 |
+
'pool_type': 'temporal_concat',
|
| 51 |
+
'train_videos_per_class': 500},
|
| 52 |
+
'preset': 'ssv2_coarse10'}],
|
| 53 |
+
'load_checkpoint': True,
|
| 54 |
+
'read_checkpoint': None,
|
| 55 |
+
'save_every_freq': 5,
|
| 56 |
+
'seed': 239,
|
| 57 |
+
'use_sdpa': True,
|
| 58 |
+
'use_wandb': True,
|
| 59 |
+
'wandb_project': 'vjepa_ablation'},
|
| 60 |
+
'metrics': {'std': {}},
|
| 61 |
+
'model': { 'class_token': True,
|
| 62 |
+
'freeze_backbone': False,
|
| 63 |
+
'is_causal': True,
|
| 64 |
+
'model_name': 'vit_large_patch14_dinov2_lvd142m',
|
| 65 |
+
'pe_type': '2d',
|
| 66 |
+
'pred_depth': 12,
|
| 67 |
+
'pred_embed_dim': 384,
|
| 68 |
+
'pred_num_heads': 6,
|
| 69 |
+
'predictor': 'v2_cross',
|
| 70 |
+
'stem_type': '2d',
|
| 71 |
+
'target_embed_dim': 1024,
|
| 72 |
+
'target_kind': 'ema',
|
| 73 |
+
'uniform_power': True,
|
| 74 |
+
'use_activation_checkpointing': True,
|
| 75 |
+
'use_alibi': False,
|
| 76 |
+
'use_lora': False,
|
| 77 |
+
'use_mask_tokens': True,
|
| 78 |
+
'use_rope': False,
|
| 79 |
+
'use_sdpa': True,
|
| 80 |
+
'zero_init_mask_tokens': True},
|
| 81 |
+
'nodes': 8,
|
| 82 |
+
'optimization': { 'clip_grad': None,
|
| 83 |
+
'ema': [1.0, 0.99925],
|
| 84 |
+
'ema_hold_epochs': 0,
|
| 85 |
+
'ema_ramp_epochs': 30,
|
| 86 |
+
'encoder_layer_decay': 0.9,
|
| 87 |
+
'epochs': 60,
|
| 88 |
+
'final_lr': 0.0005,
|
| 89 |
+
'final_weight_decay': 0.04,
|
| 90 |
+
'ipe': 300,
|
| 91 |
+
'ipe_scale': 1.0,
|
| 92 |
+
'lr': 0.0005,
|
| 93 |
+
'predictor_lr_scale': 3.0,
|
| 94 |
+
'start_lr': 5e-05,
|
| 95 |
+
'warmup': 4,
|
| 96 |
+
'weight_decay': 0.04},
|
| 97 |
+
'tasks_per_node': 4}
|
| 98 |
+
INFO:root:Running pre-training of app: vjepa
|
| 99 |
+
[INFO ][2026-06-11 05:16:17][app.vjepa.train ][main ] which_dtype='bfloat16'
|
| 100 |
+
[INFO ][2026-06-11 05:16:17][app.vjepa.train ][main ] Disabling persistent_workers (incompatible with KNN eval)
|
| 101 |
+
[INFO ][2026-06-11 05:16:17][app.vjepa.train ][main ] NCCL_SOCKET_IFNAME=None
|
| 102 |
+
[INFO ][2026-06-11 05:16:18][app.vjepa.train ][main ] Initialized (rank/world-size) 17/32, tasks_per_node=4
|
| 103 |
+
[WARNING ][2026-06-11 05:16:25][experiments.stmodels.vision_transformers_v3][_build_dinov2_vit ] DINOv2 is an image model — fwd_chunk is unset so all frames will be processed jointly. Set fwd_chunk=1 for per-frame (standard) processing.
|
| 104 |
+
[INFO ][2026-06-11 05:16:40][experiments.stmodels.vision_transformers_v3][_build_dinov2_vit ] Loading pretrained weights for vit_large_patch14_dinov2_lvd142m from timm (vit_large_patch14_dinov2)
|
| 105 |
+
[INFO ][2026-06-11 05:16:53][timm.models._builder][load_pretrained ] Loading pretrained weights from Hugging Face hub (timm/vit_large_patch14_dinov2.lvd142m)
|
| 106 |
+
[INFO ][2026-06-11 05:16:53][timm.models._hub ][load_state_dict_from_hf ] [timm/vit_large_patch14_dinov2.lvd142m] Safe alternative available for 'pytorch_model.bin' (as 'model.safetensors'). Loading weights using safetensors.
|
| 107 |
+
[INFO ][2026-06-11 05:16:53][timm.layers.pos_embed][resample_abs_pos_embed ] Resized position embedding: (37, 37) to (16, 16).
|
| 108 |
+
[INFO ][2026-06-11 05:17:34][root ][init_video_model ] ViTMultiSeqWrapper(
|
| 109 |
+
(backbone): VisionTransformer(
|
| 110 |
+
(patch_embed): PatchEmbed3D(
|
| 111 |
+
(proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
|
| 112 |
+
)
|
| 113 |
+
(blocks): ModuleList(
|
| 114 |
+
(0-23): 24 x Block(
|
| 115 |
+
(norm1): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
|
| 116 |
+
(norm2): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
|
| 117 |
+
(drop_path1): Identity()
|
| 118 |
+
(drop_path2): Identity()
|
| 119 |
+
(attn): Attention(
|
| 120 |
+
(qkv): Linear(in_features=1024, out_features=3072, bias=True)
|
| 121 |
+
(attn_drop): Dropout(p=0.0, inplace=False)
|
| 122 |
+
(proj): Linear(in_features=1024, out_features=1024, bias=True)
|
| 123 |
+
(proj_drop): Dropout(p=0.0, inplace=False)
|
| 124 |
+
)
|
| 125 |
+
(mlp): MLP(
|
| 126 |
+
(fc1): Linear(in_features=1024, out_features=4096, bias=True)
|
| 127 |
+
(act): GELU(approximate='none')
|
| 128 |
+
(fc2): Linear(in_features=4096, out_features=1024, bias=True)
|
| 129 |
+
(drop): Dropout(p=0.0, inplace=False)
|
| 130 |
+
)
|
| 131 |
+
)
|
| 132 |
+
)
|
| 133 |
+
(norm): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
|
| 134 |
+
)
|
| 135 |
+
)
|
| 136 |
+
[INFO ][2026-06-11 05:17:34][root ][init_video_model ] VJEPAPredictorMultiSeqWrapper(
|
| 137 |
+
(backbone): PredictorV2(
|
| 138 |
+
(predictor_embed): Linear(in_features=1024, out_features=384, bias=True)
|
| 139 |
+
(mask_tokens): ParameterList(
|
| 140 |
+
(0): Parameter containing: [torch.float32 of size 1x384 (cuda:0)]
|
| 141 |
+
(1): Parameter containing: [torch.float32 of size 1x384 (cuda:0)]
|
| 142 |
+
(2): Parameter containing: [torch.float32 of size 1x384 (cuda:0)]
|
| 143 |
+
(3): Parameter containing: [torch.float32 of size 1x384 (cuda:0)]
|
| 144 |
+
)
|
| 145 |
+
(predictor_blocks): ModuleList(
|
| 146 |
+
(0-11): 12 x Block(
|
| 147 |
+
(residual1): EfficientResidual(
|
| 148 |
+
(norm): LayerNorm((384,), eps=1e-05, elementwise_affine=True)
|
| 149 |
+
(fn): Attention(
|
| 150 |
+
(q_proj): Linear(in_features=384, out_features=384, bias=False)
|
| 151 |
+
(k_proj): Linear(in_features=384, out_features=384, bias=False)
|
| 152 |
+
(v_proj): Linear(in_features=384, out_features=384, bias=False)
|
| 153 |
+
(proj): Linear(in_features=384, out_features=384, bias=False)
|
| 154 |
+
(rope): Rope()
|
| 155 |
+
)
|
| 156 |
+
)
|
| 157 |
+
(residual2): EfficientResidual(
|
| 158 |
+
(norm): LayerNorm((384,), eps=1e-05, elementwise_affine=True)
|
| 159 |
+
(fn): MLP(
|
| 160 |
+
(fc1): Linear(in_features=384, out_features=1536, bias=False)
|
| 161 |
+
(act): GELU(approximate='none')
|
| 162 |
+
(fc2): Linear(in_features=1536, out_features=384, bias=False)
|
| 163 |
+
(drop): Dropout(p=0.0, inplace=False)
|
| 164 |
+
)
|
| 165 |
+
)
|
| 166 |
+
)
|
| 167 |
+
)
|
| 168 |
+
(predictor_norm): LayerNorm((384,), eps=1e-05, elementwise_affine=True)
|
| 169 |
+
(predictor_proj): Linear(in_features=384, out_features=1024, bias=True)
|
| 170 |
+
)
|
| 171 |
+
)
|
| 172 |
+
[INFO ][2026-06-11 05:17:34][root ][init_video_model ] ViTMultiSeqWrapper(
|
| 173 |
+
(backbone): VisionTransformer(
|
| 174 |
+
(patch_embed): PatchEmbed3D(
|
| 175 |
+
(proj): Conv3d(3, 1024, kernel_size=(1, 14, 14), stride=(1, 14, 14))
|
| 176 |
+
)
|
| 177 |
+
(blocks): ModuleList(
|
| 178 |
+
(0-23): 24 x Block(
|
| 179 |
+
(norm1): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
|
| 180 |
+
(norm2): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
|
| 181 |
+
(drop_path1): Identity()
|
| 182 |
+
(drop_path2): Identity()
|
| 183 |
+
(attn): Attention(
|
| 184 |
+
(qkv): Linear(in_features=1024, out_features=3072, bias=True)
|
| 185 |
+
(attn_drop): Dropout(p=0.0, inplace=False)
|
| 186 |
+
(proj): Linear(in_features=1024, out_features=1024, bias=True)
|
| 187 |
+
(proj_drop): Dropout(p=0.0, inplace=False)
|
| 188 |
+
)
|
| 189 |
+
(mlp): MLP(
|
| 190 |
+
(fc1): Linear(in_features=1024, out_features=4096, bias=True)
|
| 191 |
+
(act): GELU(approximate='none')
|
| 192 |
+
(fc2): Linear(in_features=4096, out_features=1024, bias=True)
|
| 193 |
+
(drop): Dropout(p=0.0, inplace=False)
|
| 194 |
+
)
|
| 195 |
+
)
|
| 196 |
+
)
|
| 197 |
+
(norm): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
|
| 198 |
+
)
|
| 199 |
+
)
|
| 200 |
+
[INFO ][2026-06-11 05:17:34][root ][init_video_model ] Encoder number of parameters: 302964736
|
| 201 |
+
[INFO ][2026-06-11 05:17:34][root ][init_video_model ] Predictor number of parameters: 22042240
|
| 202 |
+
[INFO ][2026-06-11 05:17:34][root ][init_video_model ] Target encoder number of parameters: 302964736
|
| 203 |
+
[INFO ][2026-06-11 05:17:34][root ][make_videodataset ] VideoDataset dataset created
|
| 204 |
+
[INFO ][2026-06-11 05:17:34][WeightedSampler ][__init__ ] Using DistributedWeightedSampler with rank 17 / 32
|
| 205 |
+
[INFO ][2026-06-11 05:17:34][root ][make_videodataset ] VideoDataset unsupervised data loader created
|
| 206 |
+
[INFO ][2026-06-11 05:17:34][app.vjepa.train ][main ] iterations per epoch/dataset length: 300/392
|
| 207 |
+
[INFO ][2026-06-11 05:17:34][app.vjepa.train ][main ] Wrapping models in DDP (rank 17)...
|
| 208 |
+
[INFO ][2026-06-11 05:17:35][app.vjepa.train ][main ] Initializing loader...
|