Hugging Face's logo

Amshaker
/
WorldMem

Model card Files
xet
 Community
1
Amshaker commited on
Commit
42929a5
·
verified ·
1 Parent(s): 19d1879

Upload folder using huggingface_hub (part 2)

Browse files
Files changed (3) hide show
  1. slurm_logs/worldmem-native-p0-p3-175k-85729_0.out +0 -0
  2. slurm_logs/worldmem-native-p0-p3-175k-85729_1.out +0 -0
  3. slurm_logs/worldmem-native-p0-p3-175k-85729_2.out +269 -0
slurm_logs/worldmem-native-p0-p3-175k-85729_0.out ADDED
The diff for this file is too large to render. See raw diff
 
slurm_logs/worldmem-native-p0-p3-175k-85729_1.out ADDED
The diff for this file is too large to render. See raw diff
 
slurm_logs/worldmem-native-p0-p3-175k-85729_2.out ADDED
@@ -0,0 +1,269 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ***************************************************
2
+ You have loaded a buildenv module
3
+ ***************************************************
4
+ The buildenv-gcccuda/12.9.1-gcc11 module makes available:
5
+ - GCC: 11
6
+ - CUDA: 12.9.1
7
+ - OpenMPI: 5.0.8
8
+ - FFTW: 3.3.10
9
+ - OpenBLAS: 0.3.30
10
+ - ScaLAPACK: 2.2.2
11
+ - MAGMA: 2.9.0
12
+ - Eigen: 5.0.0
13
+ - OpenCV: 4.12.0
14
+ - LLVM: 21.1.3
15
+
16
+ These libraries are accessible via the standard environment
17
+ variables CPATH (for headers) and LIBRARY_PATH (for libraries),
18
+ which are picked up automatically by the compiler toolchain.
19
+
20
+ {
21
+ "profile": "P2",
22
+ "native_commit": "c1cc91bdfb2dda52fe309280e5ed046b1faba448",
23
+ "selector_commit": "c4f048c7e564b57d70755a0d954d3af04aabd2fd",
24
+ "changed_native_files": [
25
+ "datasets/video/minecraft_video_dataset.py",
26
+ "algorithms/worldmem/df_video.py"
27
+ ],
28
+ "retained_revisit_slots": 4,
29
+ "native_selection": "Run native eight-slot selection unchanged; retain its leading revisit slots verbatim",
30
+ "shortage": "Leave unfilled reserved slots at their original native selections",
31
+ "dropout": "Native repeat-one-frame training augmentation is unchanged"
32
+ }
33
+ /proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/lightning/fabric/__init__.py:40: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
34
+ Native imports ready: 1.26.4 2.4.1+cu121
35
+ P2: native code c1cc91bdfb2dda52fe309280e5ed046b1faba448, selector overlay c4f048c7e564b57d70755a0d954d3af04aabd2fd, evaluation 85805
36
+ P2 stage 1: resume=0 target=30000 checkpoint=none
37
+ /proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/_internal/defaults_list.py:251: UserWarning: In 'training': Defaults list is missing `_self_`. See https://hydra.cc/docs/1.2/upgrades/1.0_to_1.1/default_composition_order for more information
38
+ warnings.warn(msg, UserWarning)
39
+ /proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/_internal/defaults_list.py:251: UserWarning: In 'training': Defaults list is missing `_self_`. See https://hydra.cc/docs/1.2/upgrades/1.0_to_1.1/default_composition_order for more information
40
+ warnings.warn(msg, UserWarning)
41
+ /proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/_internal/defaults_list.py:251: UserWarning: In 'training': Defaults list is missing `_self_`. See https://hydra.cc/docs/1.2/upgrades/1.0_to_1.1/default_composition_order for more information
42
+ warnings.warn(msg, UserWarning)
43
+ /proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/_internal/defaults_list.py:251: UserWarning: In 'training': Defaults list is missing `_self_`. See https://hydra.cc/docs/1.2/upgrades/1.0_to_1.1/default_composition_order for more information
44
+ warnings.warn(msg, UserWarning)
45
+ /proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/_internal/defaults_list.py:251: UserWarning: In 'training': Defaults list is missing `_self_`. See https://hydra.cc/docs/1.2/upgrades/1.0_to_1.1/default_composition_order for more information
46
+ warnings.warn(msg, UserWarning)
47
+ /proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/_internal/defaults_list.py:251: UserWarning: In 'training': Defaults list is missing `_self_`. See https://hydra.cc/docs/1.2/upgrades/1.0_to_1.1/default_composition_order for more information
48
+ warnings.warn(msg, UserWarning)
49
+ /proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/_internal/defaults_list.py:251: UserWarning: In 'training': Defaults list is missing `_self_`. See https://hydra.cc/docs/1.2/upgrades/1.0_to_1.1/default_composition_order for more information
50
+ warnings.warn(msg, UserWarning)
51
+ /proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/hydra/_internal/defaults_list.py:251: UserWarning: In 'training': Defaults list is missing `_self_`. See https://hydra.cc/docs/1.2/upgrades/1.0_to_1.1/default_composition_order for more information
52
+ warnings.warn(msg, UserWarning)
53
+ /proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/lightning/fabric/__init__.py:40: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
54
+ /proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/lightning/fabric/__init__.py:40: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
55
+ /proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/lightning/fabric/__init__.py:40: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
56
+ /proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/lightning/fabric/__init__.py:40: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
57
+ /proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/lightning/fabric/__init__.py:40: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
58
+ /proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/lightning/fabric/__init__.py:40: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
59
+ /proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/lightning/fabric/__init__.py:40: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
60
+ /proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/lightning/fabric/__init__.py:40: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
61
+ [2026-09-15 13:56:59,653][lightning.fabric.utilities.seed][INFO] - [rank: 7] Seed set to 42
62
+ INFO: [rank: 7] Seed set to 42
63
+ [2026-09-15 13:56:59,738][lightning.fabric.utilities.seed][INFO] - [rank: 1] Seed set to 42
64
+ INFO: [rank: 1] Seed set to 42
65
+ [2026-09-15 13:56:59,827][lightning.fabric.utilities.seed][INFO] - [rank: 0] Seed set to 42
66
+ INFO: [rank: 0] Seed set to 42
67
+ [2026-09-15 13:56:59,943][lightning.fabric.utilities.seed][INFO] - [rank: 2] Seed set to 42
68
+ INFO: [rank: 2] Seed set to 42
69
+ [2026-09-15 13:57:00,012][lightning.fabric.utilities.seed][INFO] - [rank: 4] Seed set to 42
70
+ INFO: [rank: 4] Seed set to 42
71
+ [2026-09-15 13:57:00,057][lightning.fabric.utilities.seed][INFO] - [rank: 6] Seed set to 42
72
+ INFO: [rank: 6] Seed set to 42
73
+ [2026-09-15 13:57:00,254][lightning.fabric.utilities.seed][INFO] - [rank: 5] Seed set to 42
74
+ INFO: [rank: 5] Seed set to 42
75
+ [2026-09-15 13:57:00,787][lightning.fabric.utilities.seed][INFO] - [rank: 3] Seed set to 42
76
+ INFO: [rank: 3] Seed set to 42
77
+ /proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
78
+ warnings.warn(
79
+ /proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=AlexNet_Weights.IMAGENET1K_V1`. You can also use `weights=AlexNet_Weights.DEFAULT` to get the most up-to-date weights.
80
+ warnings.warn(msg)
81
+ /proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
82
+ warnings.warn(
83
+ /proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=AlexNet_Weights.IMAGENET1K_V1`. You can also use `weights=AlexNet_Weights.DEFAULT` to get the most up-to-date weights.
84
+ warnings.warn(msg)
85
+ /proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
86
+ warnings.warn(
87
+ /proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
88
+ warnings.warn(
89
+ /proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=AlexNet_Weights.IMAGENET1K_V1`. You can also use `weights=AlexNet_Weights.DEFAULT` to get the most up-to-date weights.
90
+ warnings.warn(msg)
91
+ /proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=AlexNet_Weights.IMAGENET1K_V1`. You can also use `weights=AlexNet_Weights.DEFAULT` to get the most up-to-date weights.
92
+ warnings.warn(msg)
93
+ /proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
94
+ warnings.warn(
95
+ /proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=AlexNet_Weights.IMAGENET1K_V1`. You can also use `weights=AlexNet_Weights.DEFAULT` to get the most up-to-date weights.
96
+ warnings.warn(msg)
97
+ /proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
98
+ warnings.warn(
99
+ /proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=AlexNet_Weights.IMAGENET1K_V1`. You can also use `weights=AlexNet_Weights.DEFAULT` to get the most up-to-date weights.
100
+ warnings.warn(msg)
101
+ /proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/lpips/lpips.py:107: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
102
+ self.load_state_dict(torch.load(model_path, map_location='cpu'), strict=False)
103
+ /proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/lpips/lpips.py:107: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
104
+ self.load_state_dict(torch.load(model_path, map_location='cpu'), strict=False)
105
+ /proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/lpips/lpips.py:107: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
106
+ self.load_state_dict(torch.load(model_path, map_location='cpu'), strict=False)
107
+ /proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/lpips/lpips.py:107: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
108
+ self.load_state_dict(torch.load(model_path, map_location='cpu'), strict=False)
109
+ /proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
110
+ warnings.warn(
111
+ /proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=AlexNet_Weights.IMAGENET1K_V1`. You can also use `weights=AlexNet_Weights.DEFAULT` to get the most up-to-date weights.
112
+ warnings.warn(msg)
113
+ /proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/lpips/lpips.py:107: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
114
+ self.load_state_dict(torch.load(model_path, map_location='cpu'), strict=False)
115
+ /proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/lpips/lpips.py:107: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
116
+ self.load_state_dict(torch.load(model_path, map_location='cpu'), strict=False)
117
+ /proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
118
+ warnings.warn(
119
+ /proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=AlexNet_Weights.IMAGENET1K_V1`. You can also use `weights=AlexNet_Weights.DEFAULT` to get the most up-to-date weights.
120
+ warnings.warn(msg)
121
+ /proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/lpips/lpips.py:107: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
122
+ self.load_state_dict(torch.load(model_path, map_location='cpu'), strict=False)
123
+ /proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/lpips/lpips.py:107: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
124
+ self.load_state_dict(torch.load(model_path, map_location='cpu'), strict=False)
125
+ /proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/lightning/pytorch/plugins/precision/amp.py:54: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
126
+ /proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/lightning/pytorch/plugins/precision/amp.py:54: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
127
+ /proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/lightning/pytorch/plugins/precision/amp.py:54: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
128
+ /proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/lightning/pytorch/plugins/precision/amp.py:54: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
129
+ /proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/lightning/pytorch/plugins/precision/amp.py:54: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
130
+ [2026-09-15 13:57:09,154][pytorch_lightning.utilities.rank_zero][INFO] - Using 16bit Automatic Mixed Precision (AMP)
131
+ /proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/lightning/pytorch/plugins/precision/amp.py:54: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
132
+ [2026-09-15 13:57:09,323][pytorch_lightning.utilities.rank_zero][INFO] - GPU available: True (cuda), used: True
133
+ [2026-09-15 13:57:09,323][pytorch_lightning.utilities.rank_zero][INFO] - TPU available: False, using: 0 TPU cores
134
+ [2026-09-15 13:57:09,323][pytorch_lightning.utilities.rank_zero][INFO] - IPU available: False, using: 0 IPUs
135
+ [2026-09-15 13:57:09,323][pytorch_lightning.utilities.rank_zero][INFO] - HPU available: False, using: 0 HPUs
136
+ [2026-09-15 13:57:09,324][pytorch_lightning.utilities.rank_zero][INFO] - `Trainer(limit_val_batches=1)` was configured so 1 batch will be used.
137
+ /proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/lightning/pytorch/plugins/precision/amp.py:54: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
138
+ /proj/cvl/users/x_fahkh2/envs/worldmem/lib/python3.10/site-packages/lightning/pytorch/plugins/precision/amp.py:54: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
139
+ [2026-09-15 13:57:11,215][pytorch_lightning.utilities.rank_zero][INFO] - Model weights loaded.
140
+ [2026-09-15 13:57:12,566][pytorch_lightning.utilities.rank_zero][INFO] - Model weights loaded.
141
+ [2026-09-15 13:57:15,622][lightning.fabric.utilities.seed][INFO] - [rank: 1] Seed set to 42
142
+ INFO: [rank: 1] Seed set to 42
143
+ [2026-09-15 13:57:15,622][lightning.fabric.utilities.seed][INFO] - [rank: 2] Seed set to 42
144
+ INFO: [rank: 2] Seed set to 42
145
+ INFO: [rank: 4] Seed set to 42
146
+ [2026-09-15 13:57:15,623][lightning.fabric.utilities.seed][INFO] - [rank: 4] Seed set to 42
147
+ INFO: [rank: 7] Seed set to 42
148
+ [2026-09-15 13:57:15,625][lightning.fabric.utilities.seed][INFO] - [rank: 7] Seed set to 42
149
+ INFO: [rank: 6] Seed set to 42
150
+ [2026-09-15 13:57:15,635][lightning.fabric.utilities.seed][INFO] - [rank: 6] Seed set to 42
151
+ INFO: Initializing distributed: GLOBAL_RANK: 4, MEMBER: 5/8
152
+ [2026-09-15 13:57:15,638][lightning.fabric.utilities.distributed][INFO] - Initializing distributed: GLOBAL_RANK: 4, MEMBER: 5/8
153
+ [2026-09-15 13:57:15,640][lightning.fabric.utilities.distributed][INFO] - Initializing distributed: GLOBAL_RANK: 1, MEMBER: 2/8
154
+ INFO: Initializing distributed: GLOBAL_RANK: 1, MEMBER: 2/8
155
+ INFO: [rank: 5] Seed set to 42
156
+ [2026-09-15 13:57:15,641][lightning.fabric.utilities.seed][INFO] - [rank: 5] Seed set to 42
157
+ INFO: Initializing distributed: GLOBAL_RANK: 6, MEMBER: 7/8
158
+ [2026-09-15 13:57:15,644][lightning.fabric.utilities.distributed][INFO] - Initializing distributed: GLOBAL_RANK: 6, MEMBER: 7/8
159
+ [2026-09-15 13:57:15,651][lightning.fabric.utilities.distributed][INFO] - Initializing distributed: GLOBAL_RANK: 5, MEMBER: 6/8
160
+ INFO: Initializing distributed: GLOBAL_RANK: 5, MEMBER: 6/8
161
+ [2026-09-15 13:57:15,651][lightning.fabric.utilities.seed][INFO] - [rank: 0] Seed set to 42
162
+ INFO: [rank: 0] Seed set to 42
163
+ [2026-09-15 13:57:15,662][lightning.fabric.utilities.distributed][INFO] - Initializing distributed: GLOBAL_RANK: 0, MEMBER: 1/8
164
+ INFO: Initializing distributed: GLOBAL_RANK: 0, MEMBER: 1/8
165
+ [2026-09-15 13:57:15,697][lightning.fabric.utilities.seed][INFO] - [rank: 3] Seed set to 42
166
+ INFO: [rank: 3] Seed set to 42
167
+ [2026-09-15 13:57:15,709][lightning.fabric.utilities.distributed][INFO] - Initializing distributed: GLOBAL_RANK: 3, MEMBER: 4/8
168
+ INFO: Initializing distributed: GLOBAL_RANK: 3, MEMBER: 4/8
169
+ [2026-09-15 13:57:16,238][lightning.fabric.utilities.distributed][INFO] - Initializing distributed: GLOBAL_RANK: 2, MEMBER: 3/8
170
+ INFO: Initializing distributed: GLOBAL_RANK: 2, MEMBER: 3/8
171
+ [2026-09-15 13:57:16,751][lightning.fabric.utilities.distributed][INFO] - Initializing distributed: GLOBAL_RANK: 7, MEMBER: 8/8
172
+ INFO: Initializing distributed: GLOBAL_RANK: 7, MEMBER: 8/8
173
+ [2026-09-15 13:57:21,747][pytorch_lightning.utilities.rank_zero][INFO] - ----------------------------------------------------------------------------------------------------
174
+ distributed_backend=nccl
175
+ All distributed processes registered. Starting with 8 processes
176
+ ----------------------------------------------------------------------------------------------------
177
+
178
+ wandb: WARNING The anonymous setting has no effect and will be removed in a future version.
179
+ wandb: WARNING `resume` will be ignored since W&B syncing is set to `offline`. Starting a new run with run id panx53h2.
180
+ wandb: Tracking run with wandb version 0.28.1
181
+ wandb: W&B syncing is set to `offline` in this directory. Run `wandb online` or set WANDB_MODE=online to enable cloud syncing.
182
+ wandb: Run data is saved locally in /proj/cvl/users/x_fahkh2/DeMemWM/results/worldmem_native_p2_8h200_bs8_global64_seed42_175k/stages/stage_1/wandb/offline-run-20260915_135726-panx53h2
183
+ wandb: View this run in the terminal with `wandb leet`
184
+ INFO: LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1,2,3,4,5,6,7]
185
+ INFO: LOCAL_RANK: 1 - CUDA_VISIBLE_DEVICES: [0,1,2,3,4,5,6,7]
186
+ INFO: LOCAL_RANK: 6 - CUDA_VISIBLE_DEVICES: [0,1,2,3,4,5,6,7]
187
+ [2026-09-15 13:57:29,646][lightning.pytorch.accelerators.cuda][INFO] - LOCAL_RANK: 1 - CUDA_VISIBLE_DEVICES: [0,1,2,3,4,5,6,7]
188
+ INFO: LOCAL_RANK: 3 - CUDA_VISIBLE_DEVICES: [0,1,2,3,4,5,6,7]
189
+ [2026-09-15 13:57:29,646][lightning.pytorch.accelerators.cuda][INFO] - LOCAL_RANK: 6 - CUDA_VISIBLE_DEVICES: [0,1,2,3,4,5,6,7]
190
+ INFO: LOCAL_RANK: 7 - CUDA_VISIBLE_DEVICES: [0,1,2,3,4,5,6,7]
191
+ INFO: LOCAL_RANK: 4 - CUDA_VISIBLE_DEVICES: [0,1,2,3,4,5,6,7]
192
+ [2026-09-15 13:57:29,646][lightning.pytorch.accelerators.cuda][INFO] - LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1,2,3,4,5,6,7]
193
+ [2026-09-15 13:57:29,646][lightning.pytorch.accelerators.cuda][INFO] - LOCAL_RANK: 3 - CUDA_VISIBLE_DEVICES: [0,1,2,3,4,5,6,7]
194
+ INFO: LOCAL_RANK: 5 - CUDA_VISIBLE_DEVICES: [0,1,2,3,4,5,6,7]
195
+ [2026-09-15 13:57:29,646][lightning.pytorch.accelerators.cuda][INFO] - LOCAL_RANK: 7 - CUDA_VISIBLE_DEVICES: [0,1,2,3,4,5,6,7]
196
+ [2026-09-15 13:57:29,646][lightning.pytorch.accelerators.cuda][INFO] - LOCAL_RANK: 4 - CUDA_VISIBLE_DEVICES: [0,1,2,3,4,5,6,7]
197
+ [2026-09-15 13:57:29,646][lightning.pytorch.accelerators.cuda][INFO] - LOCAL_RANK: 2 - CUDA_VISIBLE_DEVICES: [0,1,2,3,4,5,6,7]
198
+ INFO: LOCAL_RANK: 2 - CUDA_VISIBLE_DEVICES: [0,1,2,3,4,5,6,7]
199
+ [2026-09-15 13:57:29,646][lightning.pytorch.accelerators.cuda][INFO] - LOCAL_RANK: 5 - CUDA_VISIBLE_DEVICES: [0,1,2,3,4,5,6,7]
200
+ INFO:
201
+ | Name | Type | Params
202
+ ---------------------------------------------------------------------------------
203
+ 0 | diffusion_model | Diffusion | 945 M
204
+ 1 | validation_lpips_model | LearnedPerceptualImagePatchSimilarity | 2.5 M
205
+ 2 | vae | AutoencoderKL | 229 M
206
+ ---------------------------------------------------------------------------------
207
+ 1.2 B Trainable params
208
+ 2.5 M Non-trainable params
209
+ 1.2 B Total params
210
+ 4,707.521 Total estimated model params size (MB)
211
+ [2026-09-15 13:57:30,514][lightning.pytorch.callbacks.model_summary][INFO] -
212
+ | Name | Type | Params
213
+ ---------------------------------------------------------------------------------
214
+ 0 | diffusion_model | Diffusion | 945 M
215
+ 1 | validation_lpips_model | LearnedPerceptualImagePatchSimilarity | 2.5 M
216
+ 2 | vae | AutoencoderKL | 229 M
217
+ ---------------------------------------------------------------------------------
218
+ 1.2 B Trainable params
219
+ 2.5 M Non-trainable params
220
+ 1.2 B Total params
221
+ 4,707.521 Total estimated model params size (MB)
222
+ INFO: SLURM auto-requeueing enabled. Setting signal handlers.
223
+ [2026-09-15 13:57:33,165][lightning.pytorch.trainer.connectors.signal_connector][INFO] - SLURM auto-requeueing enabled. Setting signal handlers.
224
+ INFO: SLURM auto-requeueing enabled. Setting signal handlers.
225
+ INFO: SLURM auto-requeueing enabled. Setting signal handlers.
226
+ INFO: SLURM auto-requeueing enabled. Setting signal handlers.
227
+ INFO: SLURM auto-requeueing enabled. Setting signal handlers.
228
+ INFO: SLURM auto-requeueing enabled. Setting signal handlers.
229
+ INFO: SLURM auto-requeueing enabled. Setting signal handlers.
230
+ [2026-09-15 13:57:33,165][lightning.pytorch.trainer.connectors.signal_connector][INFO] - SLURM auto-requeueing enabled. Setting signal handlers.
231
+ [2026-09-15 13:57:33,165][lightning.pytorch.trainer.connectors.signal_connector][INFO] - SLURM auto-requeueing enabled. Setting signal handlers.
232
+ [2026-09-15 13:57:33,166][lightning.pytorch.trainer.connectors.signal_connector][INFO] - SLURM auto-requeueing enabled. Setting signal handlers.
233
+ [2026-09-15 13:57:33,166][lightning.pytorch.trainer.connectors.signal_connector][INFO] - SLURM auto-requeueing enabled. Setting signal handlers.
234
+ [2026-09-15 13:57:33,166][lightning.pytorch.trainer.connectors.signal_connector][INFO] - SLURM auto-requeueing enabled. Setting signal handlers.
235
+ INFO: SLURM auto-requeueing enabled. Setting signal handlers.
236
+ [2026-09-15 13:57:33,165][lightning.pytorch.trainer.connectors.signal_connector][INFO] - SLURM auto-requeueing enabled. Setting signal handlers.
237
+ [2026-09-15 13:57:33,166][lightning.pytorch.trainer.connectors.signal_connector][INFO] - SLURM auto-requeueing enabled. Setting signal handlers.
238
+
239
+ srun: got SIGCONT
240
+ INFO: [rank: 1] Received SIGTERM: 15
241
+ [2026-09-15 13:58:02,048][lightning.pytorch.trainer.connectors.signal_connector][INFO] - [rank: 1] Received SIGTERM: 15
242
+ srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
243
+ [2026-09-15 13:58:02,049][lightning.pytorch.trainer.connectors.signal_connector][INFO] - [rank: 7] Received SIGTERM: 15
244
+ INFO: [rank: 7] Received SIGTERM: 15
245
+ INFO: [rank: 0] Received SIGTERM: 15
246
+ [2026-09-15 13:58:02,050][lightning.pytorch.trainer.connectors.signal_connector][INFO] - Bypassing SIGTERM: 15
247
+ INFO: Bypassing SIGTERM: 15
248
+ [2026-09-15 13:58:02,050][lightning.pytorch.trainer.connectors.signal_connector][INFO] - [rank: 3] Received SIGTERM: 15
249
+ [2026-09-15 13:58:02,050][lightning.pytorch.trainer.connectors.signal_connector][INFO] - Bypassing SIGTERM: 15
250
+ INFO: [rank: 3] Received SIGTERM: 15
251
+ INFO: Bypassing SIGTERM: 15
252
+ [2026-09-15 13:58:02,050][lightning.pytorch.trainer.connectors.signal_connector][INFO] - [rank: 4] Received SIGTERM: 15
253
+ [2026-09-15 13:58:02,050][lightning.pytorch.trainer.connectors.signal_connector][INFO] - Bypassing SIGTERM: 15
254
+ INFO: [rank: 4] Received SIGTERM: 15
255
+ INFO: Bypassing SIGTERM: 15
256
+ [2026-09-15 13:58:02,050][lightning.pytorch.trainer.connectors.signal_connector][INFO] - Bypassing SIGTERM: 15
257
+ INFO: Bypassing SIGTERM: 15
258
+ INFO: [rank: 5] Received SIGTERM: 15
259
+ [2026-09-15 13:58:02,050][lightning.pytorch.trainer.connectors.signal_connector][INFO] - [rank: 5] Received SIGTERM: 15
260
+ 0.62it/s, v_num=53h2]
261
+ [2026-09-15 13:58:02,051][lightning.pytorch.trainer.connectors.signal_connector][INFO] - Bypassing SIGTERM: 15
262
+ INFO: Bypassing SIGTERM: 15
263
+ INFO: Bypassing SIGTERM: 15
264
+ [2026-09-15 13:58:02,051][lightning.pytorch.trainer.connectors.signal_connector][INFO] - Bypassing SIGTERM: 15
265
+ INFO: [rank: 6] Received SIGTERM: 15
266
+ [2026-09-15 13:58:02,051][lightning.pytorch.trainer.connectors.signal_connector][INFO] - [rank: 6] Received SIGTERM: 15
267
+ [2026-09-15 13:58:02,052][lightning.pytorch.trainer.connectors.signal_connector][INFO] - Bypassing SIGTERM: 15
268
+ INFO: Bypassing SIGTERM: 15
269
+ [2026-09-15T13:58:02.052] error: *** STEP 85802.0 ON node105 CANCELLED AT 2026-09-15T13:58:02 DUE to SIGNAL Killed ***