Buckets:
| spec: /srv/data/foundation-physics-graph-model/outputs/cosmos_input/spec_depth_vis.json | |
| output: /srv/data/foundation-physics-graph-model/outputs/cosmos_runs/depth_vis | |
| devices: 1 | |
| hf_token: hf_AvnPC... (the gated downloads must use this one) | |
| [07-31 15:54:41|INFO|packages/cosmos-oss/cosmos_oss/init.py:96:_init_log_files] Log saved to /srv/data/foundation-physics-graph-model/outputs/cosmos_runs/depth_vis/console.log | |
| /srv/data/foundation-physics-graph-model/third_party/cosmos-transfer2.5/.venv/lib/python3.10/site-packages/torch/cuda/__init__.py:789: UserWarning: Can't initialize NVML | |
| warnings.warn("Can't initialize NVML") | |
| [07-31 15:55:01|INFO|cosmos_transfer2/_src/imaginaire/utils/checkpoint_db.py:297:download] Downloading checkpoint Wan2.1/vae(685afcaa-4de2-42fe-b7b9-69f7a2dee4d8) | |
| [07-31 15:55:01|INFO|cosmos_transfer2/_src/imaginaire/utils/checkpoint_db.py:165:_hf_download] uvx 'hf>=1.3.5' download nvidia/Cosmos-Predict2.5-2B --repo-type model --revision f176dc95b4a70f53ce01c4b302851595e7322b00 tokenizer.pth | |
| path=/home/quang/.cache/huggingface/hub/models--nvidia--Cosmos-Predict2.5-2B/snapshots/f176dc95b4a70f53ce01c4b302851595e7322b00/tokenizer.pth | |
| [07-31 15:55:21|INFO|cosmos_transfer2/_src/imaginaire/utils/checkpoint_db.py:297:download] Downloading checkpoint Qwen/Qwen2.5-VL-7B-Instruct(7219c6c7-f878-4137-bbdb-76842ea85e70) | |
| [07-31 15:55:21|INFO|cosmos_transfer2/_src/imaginaire/utils/checkpoint_db.py:165:_hf_download] uvx 'hf>=1.3.5' download nvidia/Cosmos-Reason1-7B --repo-type model --revision 3210bec0495fdc7a8d3dbb8d58da5711eab4b423 --include '*' | |
| path=/home/quang/.cache/huggingface/hub/models--nvidia--Cosmos-Reason1-7B/snapshots/3210bec0495fdc7a8d3dbb8d58da5711eab4b423 | |
| [07-31 16:02:08|INFO|cosmos_transfer2/_src/imaginaire/utils/checkpoint_db.py:297:download] Downloading checkpoint nvidia/Cosmos-Reason1.1-7B(cb3e3ffa-7b08-4c34-822d-61c7aa31a14f) | |
| [07-31 16:02:08|INFO|cosmos_transfer2/_src/imaginaire/utils/checkpoint_db.py:165:_hf_download] uvx 'hf>=1.3.5' download nvidia/Cosmos-Reason1-7B --repo-type model --revision 3210bec0495fdc7a8d3dbb8d58da5711eab4b423 --include '*' | |
| path=/home/quang/.cache/huggingface/hub/models--nvidia--Cosmos-Reason1-7B/snapshots/3210bec0495fdc7a8d3dbb8d58da5711eab4b423 | |
| [07-31 16:02:30|INFO|cosmos_transfer2/_src/imaginaire/utils/checkpoint_db.py:297:download] Downloading checkpoint nvidia/Cosmos-Transfer2.5-2B/general/edge(61f5694b-0ad5-4ecd-8ad7-c8545627d125) | |
| [07-31 16:02:30|INFO|cosmos_transfer2/_src/imaginaire/utils/checkpoint_db.py:165:_hf_download] uvx 'hf>=1.3.5' download nvidia/Cosmos-Transfer2.5-2B --repo-type model --revision b67b64abda3801a9aceddbff2bdb86126c06db74 general/edge/61f5694b-0ad5-4ecd-8ad7-c8545627d125_ema_bf16.pt | |
| path=/home/quang/.cache/huggingface/hub/models--nvidia--Cosmos-Transfer2.5-2B/snapshots/b67b64abda3801a9aceddbff2bdb86126c06db74/general/edge/61f5694b-0ad5-4ecd-8ad7-c8545627d125_ema_bf16.pt | |
| [07-31 16:04:53|INFO|cosmos_transfer2/_src/imaginaire/utils/checkpoint_db.py:297:download] Downloading checkpoint nvidia/Cosmos-Transfer2.5-2B/general/edge(61f5694b-0ad5-4ecd-8ad7-c8545627d125) | |
| [07-31 16:04:56|INFO|cosmos_transfer2/_src/imaginaire/utils/checkpoint_db.py:297:download] Downloading checkpoint nvidia/Cosmos-Transfer2.5-2B/general/blur(ba2f44f2-c726-4fe7-949f-597069d9b91c) | |
| [07-31 16:04:56|INFO|cosmos_transfer2/_src/imaginaire/utils/checkpoint_db.py:165:_hf_download] uvx 'hf>=1.3.5' download nvidia/Cosmos-Transfer2.5-2B --repo-type model --revision eb5325b77d358944da58a690157dd2b8071bbf85 general/blur/ba2f44f2-c726-4fe7-949f-597069d9b91c_ema_bf16.pt | |
| path=/home/quang/.cache/huggingface/hub/models--nvidia--Cosmos-Transfer2.5-2B/snapshots/eb5325b77d358944da58a690157dd2b8071bbf85/general/blur/ba2f44f2-c726-4fe7-949f-597069d9b91c_ema_bf16.pt | |
| [07-31 16:07:16|INFO|cosmos_transfer2/_src/imaginaire/utils/checkpoint_db.py:297:download] Downloading checkpoint nvidia/Cosmos-Transfer2.5-2B/general/depth(626e6618-bfcd-4d9a-a077-1409e2ce353f) | |
| [07-31 16:07:16|INFO|cosmos_transfer2/_src/imaginaire/utils/checkpoint_db.py:165:_hf_download] uvx 'hf>=1.3.5' download nvidia/Cosmos-Transfer2.5-2B --repo-type model --revision dea7737ca29dd8d9086413c6dc5724b8250a0bb4 general/depth/626e6618-bfcd-4d9a-a077-1409e2ce353f_ema_bf16.pt | |
| path=/home/quang/.cache/huggingface/hub/models--nvidia--Cosmos-Transfer2.5-2B/snapshots/dea7737ca29dd8d9086413c6dc5724b8250a0bb4/general/depth/626e6618-bfcd-4d9a-a077-1409e2ce353f_ema_bf16.pt | |
| [07-31 16:09:35|INFO|cosmos_transfer2/_src/imaginaire/utils/checkpoint_db.py:297:download] Downloading checkpoint nvidia/Cosmos-Transfer2.5-2B/general/seg(5136ef49-6d8d-42e8-8abf-7dac722a304a) | |
| [07-31 16:09:35|INFO|cosmos_transfer2/_src/imaginaire/utils/checkpoint_db.py:165:_hf_download] uvx 'hf>=1.3.5' download nvidia/Cosmos-Transfer2.5-2B --repo-type model --revision 23057a4167b89de89a4a397fdbf3887994d115eb general/seg/5136ef49-6d8d-42e8-8abf-7dac722a304a_ema_bf16.pt | |
| path=/home/quang/.cache/huggingface/hub/models--nvidia--Cosmos-Transfer2.5-2B/snapshots/23057a4167b89de89a4a397fdbf3887994d115eb/general/seg/5136ef49-6d8d-42e8-8abf-7dac722a304a_ema_bf16.pt | |
| [07-31 16:11:56|INFO|cosmos_transfer2/inference.py:146:__init__] Found 2 hint keys across all samples | |
| [07-31 16:11:56|WARNING|cosmos_transfer2/inference.py:148:__init__] Loading the multicontrol model. Multicontrol inference is not strictly equal to single control | |
| [07-31 16:11:56|INFO|cosmos_transfer2/inference.py:156:__init__] Saved config to /srv/data/foundation-physics-graph-model/outputs/cosmos_runs/depth_vis/config.yaml | |
| [07-31 16:11:56|INFO|cosmos_transfer2/inference.py:163:generate] Generating 1 samples: ['pick_place_books_depth_vis'] | |
| [07-31 16:11:56|INFO|cosmos_transfer2/inference.py:167:generate] [1/1] Processing sample pick_place_books_depth_vis | |
| [07-31 16:11:56|INFO|cosmos_transfer2/inference.py:208:_generate_sample] Saved arguments to /srv/data/foundation-physics-graph-model/outputs/cosmos_runs/depth_vis/pick_place_books_depth_vis.json | |
| [07-31 16:11:56|WARNING|cosmos_transfer2/inference.py:239:_generate_sample] Guardrail checks on prompt are disabled | |
| [07-31 16:11:56|INFO|cosmos_transfer2/inference.py:242:_generate_sample] Processing the following paths: {'vis': None, 'vis_mask': None, 'vis_mask_prompt': None, 'depth': '/srv/data/foundation-physics-graph-model/outputs/cosmos_input/depth.mp4', 'depth_mask': None, 'depth_mask_prompt': None} | |
| Generating samples: 0%| | 0/35 [00:00<?, ?it/s] Generating samples: 3%|▎ | 1/35 [00:11<06:42, 11.84s/it] Generating samples: 6%|▌ | 2/35 [00:22<06:13, 11.31s/it] Generating samples: 9%|▊ | 3/35 [00:33<05:58, 11.20s/it] Generating samples: 11%|█▏ | 4/35 [00:44<05:45, 11.15s/it] Generating samples: 14%|█▍ | 5/35 [00:55<05:30, 11.02s/it] Generating samples: 17%|█▋ | 6/35 [01:06<05:17, 10.95s/it] Generating samples: 20%|██ | 7/35 [01:17<05:07, 10.97s/it] Generating samples: 23%|██▎ | 8/35 [01:28<04:56, 10.99s/it] Generating samples: 26%|██▌ | 9/35 [01:39<04:46, 11.01s/it] Generating samples: 29%|██▊ | 10/35 [01:50<04:35, 11.03s/it] Generating samples: 31%|███▏ | 11/35 [02:01<04:23, 11.00s/it] Generating samples: 34%|███▍ | 12/35 [02:12<04:12, 10.96s/it] Generating samples: 37%|███▋ | 13/35 [02:23<04:00, 10.94s/it] Generating samples: 40%|████ | 14/35 [02:34<03:49, 10.93s/it] Generating samples: 43%|████▎ | 15/35 [02:45<03:38, 10.91s/it] Generating samples: 46%|████▌ | 16/35 [02:55<03:25, 10.83s/it] Generating samples: 49%|████▊ | 17/35 [03:06<03:13, 10.78s/it] Generating samples: 51%|█████▏ | 18/35 [03:17<03:02, 10.76s/it] Generating samples: 54%|█████▍ | 19/35 [03:27<02:51, 10.74s/it] Generating samples: 57%|█████▋ | 20/35 [03:38<02:40, 10.72s/it] Generating samples: 60%|██████ | 21/35 [03:49<02:30, 10.72s/it] Generating samples: 63%|██████▎ | 22/35 [04:00<02:19, 10.76s/it] Generating samples: 66%|██████▌ | 23/35 [04:10<02:09, 10.76s/it] Generating samples: 69%|██████▊ | 24/35 [04:21<01:58, 10.74s/it] Generating samples: 71%|███████▏ | 25/35 [04:32<01:47, 10.74s/it] Generating samples: 74%|███████▍ | 26/35 [04:43<01:37, 10.85s/it] Generating samples: 77%|███████▋ | 27/35 [04:54<01:27, 10.89s/it] Generating samples: 80%|████████ | 28/35 [05:05<01:16, 10.94s/it] Generating samples: 83%|████████▎ | 29/35 [05:16<01:05, 10.96s/it] Generating samples: 86%|████████▌ | 30/35 [05:27<00:54, 10.99s/it] Generating samples: 89%|████████▊ | 31/35 [05:38<00:43, 10.94s/it] Generating samples: 91%|█████████▏| 32/35 [05:49<00:32, 10.91s/it] Generating samples: 94%|█████████▍| 33/35 [06:00<00:21, 10.93s/it] Generating samples: 97%|█████████▋| 34/35 [06:11<00:10, 10.93s/it] Generating samples: 100%|██████████| 35/35 [06:22<00:00, 10.93s/it] Generating samples: 100%|██████████| 35/35 [06:22<00:00, 10.92s/it] | |
| Traceback (most recent call last): | |
| File "/srv/data/foundation-physics-graph-model/third_party/cosmos-transfer2.5/examples/inference.py", line 94, in <module> | |
| main(args) | |
| File "/srv/data/foundation-physics-graph-model/third_party/cosmos-transfer2.5/examples/inference.py", line 78, in main | |
| inference.generate(inference_samples, output_dir=args.setup.output_dir) | |
| File "/srv/data/foundation-physics-graph-model/third_party/cosmos-transfer2.5/cosmos_transfer2/inference.py", line 168, in generate | |
| output_path = self._generate_sample(sample, output_dir, sample_id=i_sample) | |
| File "/srv/data/foundation-physics-graph-model/third_party/cosmos-transfer2.5/cosmos_transfer2/inference.py", line 260, in _generate_sample | |
| output_video, control_video_dict, mask_video_dict, fps, _ = self.inference_pipeline.generate_img2world( | |
| File "/srv/data/foundation-physics-graph-model/third_party/cosmos-transfer2.5/.venv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context | |
| return func(*args, **kwargs) | |
| File "/srv/data/foundation-physics-graph-model/third_party/cosmos-transfer2.5/cosmos_transfer2/_src/transfer2/inference/inference_pipeline.py", line 642, in generate_img2world | |
| sample = self.model.generate_samples_from_batch( | |
| File "/srv/data/foundation-physics-graph-model/third_party/cosmos-transfer2.5/.venv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context | |
| return func(*args, **kwargs) | |
| File "/srv/data/foundation-physics-graph-model/third_party/cosmos-transfer2.5/cosmos_transfer2/_src/transfer2/models/vid2vid_model_control_vace_rectified_flow.py", line 389, in generate_samples_from_batch | |
| velocity_fn = self.get_velocity_fn_from_batch(data_batch, guidance, is_negative_prompt=is_negative_prompt) | |
| File "/srv/data/foundation-physics-graph-model/third_party/cosmos-transfer2.5/cosmos_transfer2/_src/transfer2/models/vid2vid_model_control_vace_rectified_flow.py", line 253, in get_velocity_fn_from_batch | |
| _, x0, control_condition = self.get_data_and_condition(data_batch) | |
| File "/srv/data/foundation-physics-graph-model/third_party/cosmos-transfer2.5/cosmos_transfer2/_src/transfer2/models/vid2vid_model_control_vace_rectified_flow.py", line 97, in get_data_and_condition | |
| latent_control_input += self.get_control_latent(latent_state, control_input, control_input_mask) | |
| File "/srv/data/foundation-physics-graph-model/third_party/cosmos-transfer2.5/cosmos_transfer2/_src/transfer2/models/vid2vid_model_control_vace_rectified_flow.py", line 204, in get_control_latent | |
| latent_control_input.append(self.encode(control_input).contiguous().to(**self.tensor_kwargs)) | |
| File "/srv/data/foundation-physics-graph-model/third_party/cosmos-transfer2.5/.venv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context | |
| return func(*args, **kwargs) | |
| File "/srv/data/foundation-physics-graph-model/third_party/cosmos-transfer2.5/cosmos_transfer2/_src/predict2/models/text2world_model_rectified_flow.py", line 986, in encode | |
| return self.tokenizer.encode(state) | |
| File "/srv/data/foundation-physics-graph-model/third_party/cosmos-transfer2.5/cosmos_transfer2/_src/predict2/tokenizers/wan2pt1.py", line 999, in encode | |
| latents = self.model.encode(state, clear_encoder_cache=not self.keep_encoder_cache) | |
| File "/srv/data/foundation-physics-graph-model/third_party/cosmos-transfer2.5/.venv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context | |
| return func(*args, **kwargs) | |
| File "/srv/data/foundation-physics-graph-model/third_party/cosmos-transfer2.5/cosmos_transfer2/_src/predict2/tokenizers/wan2pt1.py", line 826, in encode | |
| latent = self.model.encode(videos, self.scale, clear_encoder_cache) | |
| File "/srv/data/foundation-physics-graph-model/third_party/cosmos-transfer2.5/cosmos_transfer2/_src/predict2/tokenizers/wan2pt1.py", line 516, in encode | |
| out_ = self.encoder( | |
| File "/srv/data/foundation-physics-graph-model/third_party/cosmos-transfer2.5/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl | |
| return self._call_impl(*args, **kwargs) | |
| File "/srv/data/foundation-physics-graph-model/third_party/cosmos-transfer2.5/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl | |
| return forward_call(*args, **kwargs) | |
| File "/srv/data/foundation-physics-graph-model/third_party/cosmos-transfer2.5/cosmos_transfer2/_src/predict2/tokenizers/wan2pt1.py", line 333, in forward | |
| x = layer(x, feat_cache, feat_idx) | |
| File "/srv/data/foundation-physics-graph-model/third_party/cosmos-transfer2.5/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl | |
| return self._call_impl(*args, **kwargs) | |
| File "/srv/data/foundation-physics-graph-model/third_party/cosmos-transfer2.5/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl | |
| return forward_call(*args, **kwargs) | |
| File "/srv/data/foundation-physics-graph-model/third_party/cosmos-transfer2.5/cosmos_transfer2/_src/predict2/tokenizers/wan2pt1.py", line 217, in forward | |
| x = layer(x, feat_cache[idx]) | |
| File "/srv/data/foundation-physics-graph-model/third_party/cosmos-transfer2.5/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl | |
| return self._call_impl(*args, **kwargs) | |
| File "/srv/data/foundation-physics-graph-model/third_party/cosmos-transfer2.5/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl | |
| return forward_call(*args, **kwargs) | |
| File "/srv/data/foundation-physics-graph-model/third_party/cosmos-transfer2.5/cosmos_transfer2/_src/predict2/tokenizers/wan2pt1.py", line 62, in forward | |
| return super().forward(x) | |
| File "/srv/data/foundation-physics-graph-model/third_party/cosmos-transfer2.5/.venv/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 725, in forward | |
| return self._conv_forward(input, self.weight, self.bias) | |
| File "/srv/data/foundation-physics-graph-model/third_party/cosmos-transfer2.5/.venv/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 720, in _conv_forward | |
| return F.conv3d( | |
| RuntimeError: NVML_SUCCESS == DriverAPI::get()->nvmlInit_v2_() INTERNAL ASSERT FAILED at "/pytorch/c10/cuda/CUDACachingAllocator.cpp":983, please report a bug to PyTorch. | |
Xet Storage Details
- Size:
- 15.7 kB
- Xet hash:
- 98fdc260c270d359a274116084dc637dcfdb8176ea27abfe21d8f3e626e1b06a
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.