FlowWM/causal_droidstd/step_5000/rollout.log

WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1785085726.472938   86049 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
I0000 00:00:1785085726.502470   86049 cpu_feature_guard.cc:227] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: SSE4.1 SSE4.2 AVX AVX2 AVX512F AVX512_VNNI AVX512_BF16 AVX512_FP16 AVX_VNNI AMX_TILE AMX_INT8 AMX_BF16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/mnt/data/sftp/data/hunght23/quangpt3_data/masked-visual-actions/Ctrl-World/scripts/rollout_replay_traj.py:298: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
  _peek = torch.load(args.val_model_path, map_location='cpu')
[rollout] seeded with 1234 -- initial noise is now reproducible across runs
[rollout] checkpoint unet.conv_in=24ch -> causal_mode=True flow_stream=False

Loading pipeline components...:   0%|          | 0/5 [00:00<?, ?it/s]
Loading pipeline components...:  40%|████      | 2/5 [00:01<00:02,  1.29it/s]
Loading pipeline components...: 100%|██████████| 5/5 [00:02<00:00,  2.35it/s]
Loading pipeline components...: 100%|██████████| 5/5 [00:02<00:00,  2.14it/s]
Expected types for unet: (<class 'models.unet_spatio_temporal_condition.UNetSpatioTemporalConditionModel'>,), got <class 'diffusers.models.unets.unet_spatio_temporal_condition.UNetSpatioTemporalConditionModel'>.
/mnt/data/sftp/data/hunght23/quangpt3_data/masked-visual-actions/Ctrl-World/scripts/rollout_replay_traj.py:318: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
  state_dict = torch.load(args.val_model_path, map_location='cpu')
/mnt/data/sftp/data/hunght23/quangpt3_data/masked-visual-actions/Ctrl-World/scripts/rollout_replay_traj.py:351: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
  zero_flow = torch.load(zero_flow_path, map_location='cpu')
replace the unet to support action condition and frame_level pose!
load world model success
[rollout] unet conv_in=24ch  flow_stream=True  causal_mode=True  guidance_mode=off  w_m=0.0 w_c=0.0
rollout with replay
[rollout] history_idx = [0, 0, -8, -6, -4, -2]   one slot = pred_step-1 = 4 GT frames
[rollout]   gathers GT frames [prefill(frame 0), prefill(frame 0), -28, -20, -12, -4] relative to the current frame. Training (dataset_droid_exp33.py:171-176) uses six UNIFORM steps of one chunk span, i.e. [-24, -20, -16, -12, -8, -4].
Ground truth frames ids [ 8  9 10 11 12 13 14 15]
Traceback (most recent call last):
  File "/mnt/data/sftp/data/hunght23/quangpt3_data/masked-visual-actions/Ctrl-World/scripts/rollout_replay_traj.py", line 765, in <module>
    eef_probe, _, _, _, _ = Agent.get_traj_info(val_id_i, start_idx=start_idx_i, steps=8)
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/data/sftp/data/hunght23/quangpt3_data/masked-visual-actions/Ctrl-World/scripts/rollout_replay_traj.py", line 444, in get_traj_info
    joint_pos = np.array(anno['joints'])
                         ~~~~^^^^^^^^^^
KeyError: 'joints'