logs/cosmos_wait_run.log

waiting for a GPU with >= 90 GB free (polling every 300s)
16:22:40  best: GPU0 with 27 GB free (need 90)
16:27:40  best: GPU0 with 138 GB free (need 90)
launching on GPU0
spec:    /srv/data/foundation-physics-graph-model/outputs/cosmos_input/spec_depth_vis.json
output:  /srv/data/foundation-physics-graph-model/outputs/cosmos_runs/depth_vis
devices: 0
hf_token: hf_AvnPC... (the gated downloads must use this one)
[07-31 16:27:42|INFO|packages/cosmos-oss/cosmos_oss/init.py:96:_init_log_files] Log saved to /srv/data/foundation-physics-graph-model/outputs/cosmos_runs/depth_vis/console.log
[07-31 16:28:04|INFO|cosmos_transfer2/_src/imaginaire/utils/checkpoint_db.py:297:download] Downloading checkpoint Wan2.1/vae(685afcaa-4de2-42fe-b7b9-69f7a2dee4d8)
[07-31 16:28:04|INFO|cosmos_transfer2/_src/imaginaire/utils/checkpoint_db.py:165:_hf_download] uvx 'hf>=1.3.5' download nvidia/Cosmos-Predict2.5-2B --repo-type model --revision f176dc95b4a70f53ce01c4b302851595e7322b00 tokenizer.pth
path=/home/quang/.cache/huggingface/hub/models--nvidia--Cosmos-Predict2.5-2B/snapshots/f176dc95b4a70f53ce01c4b302851595e7322b00/tokenizer.pth
[07-31 16:28:07|INFO|cosmos_transfer2/_src/imaginaire/utils/checkpoint_db.py:297:download] Downloading checkpoint Qwen/Qwen2.5-VL-7B-Instruct(7219c6c7-f878-4137-bbdb-76842ea85e70)
[07-31 16:28:07|INFO|cosmos_transfer2/_src/imaginaire/utils/checkpoint_db.py:165:_hf_download] uvx 'hf>=1.3.5' download nvidia/Cosmos-Reason1-7B --repo-type model --revision 3210bec0495fdc7a8d3dbb8d58da5711eab4b423 --include '*'
path=/home/quang/.cache/huggingface/hub/models--nvidia--Cosmos-Reason1-7B/snapshots/3210bec0495fdc7a8d3dbb8d58da5711eab4b423
[07-31 16:28:09|INFO|cosmos_transfer2/_src/imaginaire/utils/checkpoint_db.py:297:download] Downloading checkpoint nvidia/Cosmos-Reason1.1-7B(cb3e3ffa-7b08-4c34-822d-61c7aa31a14f)
[07-31 16:28:09|INFO|cosmos_transfer2/_src/imaginaire/utils/checkpoint_db.py:165:_hf_download] uvx 'hf>=1.3.5' download nvidia/Cosmos-Reason1-7B --repo-type model --revision 3210bec0495fdc7a8d3dbb8d58da5711eab4b423 --include '*'
path=/home/quang/.cache/huggingface/hub/models--nvidia--Cosmos-Reason1-7B/snapshots/3210bec0495fdc7a8d3dbb8d58da5711eab4b423
[07-31 16:28:31|INFO|cosmos_transfer2/_src/imaginaire/utils/checkpoint_db.py:297:download] Downloading checkpoint nvidia/Cosmos-Transfer2.5-2B/general/edge(61f5694b-0ad5-4ecd-8ad7-c8545627d125)
[07-31 16:28:31|INFO|cosmos_transfer2/_src/imaginaire/utils/checkpoint_db.py:165:_hf_download] uvx 'hf>=1.3.5' download nvidia/Cosmos-Transfer2.5-2B --repo-type model --revision b67b64abda3801a9aceddbff2bdb86126c06db74 general/edge/61f5694b-0ad5-4ecd-8ad7-c8545627d125_ema_bf16.pt
path=/home/quang/.cache/huggingface/hub/models--nvidia--Cosmos-Transfer2.5-2B/snapshots/b67b64abda3801a9aceddbff2bdb86126c06db74/general/edge/61f5694b-0ad5-4ecd-8ad7-c8545627d125_ema_bf16.pt
[07-31 16:28:39|INFO|cosmos_transfer2/_src/imaginaire/utils/checkpoint_db.py:297:download] Downloading checkpoint nvidia/Cosmos-Transfer2.5-2B/general/edge(61f5694b-0ad5-4ecd-8ad7-c8545627d125)
[07-31 16:28:42|INFO|cosmos_transfer2/_src/imaginaire/utils/checkpoint_db.py:297:download] Downloading checkpoint nvidia/Cosmos-Transfer2.5-2B/general/blur(ba2f44f2-c726-4fe7-949f-597069d9b91c)
[07-31 16:28:42|INFO|cosmos_transfer2/_src/imaginaire/utils/checkpoint_db.py:165:_hf_download] uvx 'hf>=1.3.5' download nvidia/Cosmos-Transfer2.5-2B --repo-type model --revision eb5325b77d358944da58a690157dd2b8071bbf85 general/blur/ba2f44f2-c726-4fe7-949f-597069d9b91c_ema_bf16.pt
path=/home/quang/.cache/huggingface/hub/models--nvidia--Cosmos-Transfer2.5-2B/snapshots/eb5325b77d358944da58a690157dd2b8071bbf85/general/blur/ba2f44f2-c726-4fe7-949f-597069d9b91c_ema_bf16.pt
[07-31 16:28:48|INFO|cosmos_transfer2/_src/imaginaire/utils/checkpoint_db.py:297:download] Downloading checkpoint nvidia/Cosmos-Transfer2.5-2B/general/depth(626e6618-bfcd-4d9a-a077-1409e2ce353f)
[07-31 16:28:48|INFO|cosmos_transfer2/_src/imaginaire/utils/checkpoint_db.py:165:_hf_download] uvx 'hf>=1.3.5' download nvidia/Cosmos-Transfer2.5-2B --repo-type model --revision dea7737ca29dd8d9086413c6dc5724b8250a0bb4 general/depth/626e6618-bfcd-4d9a-a077-1409e2ce353f_ema_bf16.pt
path=/home/quang/.cache/huggingface/hub/models--nvidia--Cosmos-Transfer2.5-2B/snapshots/dea7737ca29dd8d9086413c6dc5724b8250a0bb4/general/depth/626e6618-bfcd-4d9a-a077-1409e2ce353f_ema_bf16.pt
[07-31 16:28:52|INFO|cosmos_transfer2/_src/imaginaire/utils/checkpoint_db.py:297:download] Downloading checkpoint nvidia/Cosmos-Transfer2.5-2B/general/seg(5136ef49-6d8d-42e8-8abf-7dac722a304a)
[07-31 16:28:52|INFO|cosmos_transfer2/_src/imaginaire/utils/checkpoint_db.py:165:_hf_download] uvx 'hf>=1.3.5' download nvidia/Cosmos-Transfer2.5-2B --repo-type model --revision 23057a4167b89de89a4a397fdbf3887994d115eb general/seg/5136ef49-6d8d-42e8-8abf-7dac722a304a_ema_bf16.pt
path=/home/quang/.cache/huggingface/hub/models--nvidia--Cosmos-Transfer2.5-2B/snapshots/23057a4167b89de89a4a397fdbf3887994d115eb/general/seg/5136ef49-6d8d-42e8-8abf-7dac722a304a_ema_bf16.pt
[07-31 16:28:57|INFO|cosmos_transfer2/inference.py:146:__init__] Found 2 hint keys across all samples
[07-31 16:28:57|WARNING|cosmos_transfer2/inference.py:148:__init__] Loading the multicontrol model. Multicontrol inference is not strictly equal to single control
[07-31 16:28:58|INFO|cosmos_transfer2/inference.py:156:__init__] Saved config to /srv/data/foundation-physics-graph-model/outputs/cosmos_runs/depth_vis/config.yaml
[07-31 16:28:58|INFO|cosmos_transfer2/inference.py:163:generate] Generating 1 samples: ['pick_place_books_depth_vis']
[07-31 16:28:58|INFO|cosmos_transfer2/inference.py:167:generate] [1/1] Processing sample pick_place_books_depth_vis
[07-31 16:28:58|INFO|cosmos_transfer2/inference.py:208:_generate_sample] Saved arguments to /srv/data/foundation-physics-graph-model/outputs/cosmos_runs/depth_vis/pick_place_books_depth_vis.json
[07-31 16:28:58|WARNING|cosmos_transfer2/inference.py:239:_generate_sample] Guardrail checks on prompt are disabled
[07-31 16:28:58|INFO|cosmos_transfer2/inference.py:242:_generate_sample] Processing the following paths: {'vis': None, 'vis_mask': None, 'vis_mask_prompt': None, 'depth': '/srv/data/foundation-physics-graph-model/outputs/cosmos_input/depth.mp4', 'depth_mask': None, 'depth_mask_prompt': None}

Generating samples:   0%|          | 0/35 [00:00<?, ?it/s]
Generating samples:   3%|▎         | 1/35 [00:11<06:26, 11.37s/it]
Generating samples:   6%|▌         | 2/35 [00:21<05:58, 10.87s/it]
Generating samples:   9%|▊         | 3/35 [00:32<05:43, 10.75s/it]
Generating samples:  11%|█▏        | 4/35 [00:43<05:31, 10.69s/it]
Generating samples:  14%|█▍        | 5/35 [00:53<05:19, 10.64s/it]
Generating samples:  17%|█▋        | 6/35 [01:04<05:06, 10.58s/it]
Generating samples:  20%|██        | 7/35 [01:14<04:56, 10.59s/it]
Generating samples:  23%|██▎       | 8/35 [01:25<04:45, 10.59s/it]
Generating samples:  26%|██▌       | 9/35 [01:35<04:34, 10.56s/it]
Generating samples:  29%|██▊       | 10/35 [01:46<04:24, 10.57s/it]
Generating samples:  31%|███▏      | 11/35 [01:56<04:13, 10.55s/it]
Generating samples:  34%|███▍      | 12/35 [02:07<04:02, 10.56s/it]
Generating samples:  37%|███▋      | 13/35 [02:18<03:52, 10.57s/it]
Generating samples:  40%|████      | 14/35 [02:28<03:41, 10.57s/it]
Generating samples:  43%|████▎     | 15/35 [02:39<03:31, 10.56s/it]
Generating samples:  46%|████▌     | 16/35 [02:49<03:20, 10.56s/it]
Generating samples:  49%|████▊     | 17/35 [03:00<03:10, 10.56s/it]
Generating samples:  51%|█████▏    | 18/35 [03:10<02:59, 10.55s/it]
Generating samples:  54%|█████▍    | 19/35 [03:21<02:48, 10.54s/it]
Generating samples:  57%|█████▋    | 20/35 [03:31<02:37, 10.53s/it]
Generating samples:  60%|██████    | 21/35 [03:42<02:27, 10.54s/it]
Generating samples:  63%|██████▎   | 22/35 [03:53<02:17, 10.56s/it]
Generating samples:  66%|██████▌   | 23/35 [04:03<02:06, 10.55s/it]
Generating samples:  69%|██████▊   | 24/35 [04:14<01:56, 10.56s/it]
Generating samples:  71%|███████▏  | 25/35 [04:24<01:45, 10.56s/it]
Generating samples:  74%|███████▍  | 26/35 [04:35<01:34, 10.54s/it]
Generating samples:  77%|███████▋  | 27/35 [04:45<01:24, 10.54s/it]
Generating samples:  80%|████████  | 28/35 [04:56<01:13, 10.51s/it]
Generating samples:  83%|████████▎ | 29/35 [05:06<01:03, 10.53s/it]
Generating samples:  86%|████████▌ | 30/35 [05:17<00:52, 10.52s/it]
Generating samples:  89%|████████▊ | 31/35 [05:27<00:41, 10.50s/it]
Generating samples:  91%|█████████▏| 32/35 [05:38<00:31, 10.52s/it]
Generating samples:  94%|█████████▍| 33/35 [05:48<00:20, 10.50s/it]
Generating samples:  97%|█████████▋| 34/35 [05:59<00:10, 10.47s/it]
Generating samples: 100%|██████████| 35/35 [06:09<00:00, 10.45s/it]
Generating samples: 100%|██████████| 35/35 [06:09<00:00, 10.56s/it]

Generating samples:   0%|          | 0/35 [00:00<?, ?it/s]
Generating samples:   3%|▎         | 1/35 [00:10<05:55, 10.47s/it]
Generating samples:   6%|▌         | 2/35 [00:20<05:44, 10.45s/it]
Generating samples:   9%|▊         | 3/35 [00:31<05:36, 10.50s/it]
Generating samples:  11%|█▏        | 4/35 [00:41<05:25, 10.51s/it]
Generating samples:  14%|█▍        | 5/35 [00:52<05:15, 10.52s/it]
Generating samples:  17%|█▋        | 6/35 [01:03<05:04, 10.51s/it]
Generating samples:  20%|██        | 7/35 [01:13<04:53, 10.48s/it]
Generating samples:  23%|██▎       | 8/35 [01:23<04:42, 10.47s/it]
Generating samples:  26%|██▌       | 9/35 [01:34<04:32, 10.46s/it]
Generating samples:  29%|██▊       | 10/35 [01:44<04:21, 10.46s/it]
Generating samples:  31%|███▏      | 11/35 [01:55<04:11, 10.46s/it]
Generating samples:  34%|███▍      | 12/35 [02:05<04:01, 10.49s/it]
Generating samples:  37%|███▋      | 13/35 [02:16<03:51, 10.51s/it]
Generating samples:  40%|████      | 14/35 [02:26<03:41, 10.52s/it]
Generating samples:  43%|████▎     | 15/35 [02:37<03:30, 10.51s/it]
Generating samples:  46%|████▌     | 16/35 [02:47<03:19, 10.51s/it]
Generating samples:  49%|████▊     | 17/35 [02:58<03:09, 10.52s/it]
Generating samples:  51%|█████▏    | 18/35 [03:08<02:58, 10.51s/it]
Generating samples:  54%|█████▍    | 19/35 [03:19<02:48, 10.51s/it]
Generating samples:  57%|█████▋    | 20/35 [03:29<02:37, 10.48s/it]
Generating samples:  60%|██████    | 21/35 [03:40<02:26, 10.47s/it]
Generating samples:  63%|██████▎   | 22/35 [03:50<02:16, 10.48s/it]
Generating samples:  66%|██████▌   | 23/35 [04:01<02:05, 10.49s/it]
Generating samples:  69%|██████▊   | 24/35 [04:11<01:55, 10.49s/it]
Generating samples:  71%|███████▏  | 25/35 [04:22<01:44, 10.47s/it]
Generating samples:  74%|███████▍  | 26/35 [04:32<01:34, 10.45s/it]
Generating samples:  77%|███████▋  | 27/35 [04:42<01:23, 10.41s/it]
Generating samples:  80%|████████  | 28/35 [04:53<01:12, 10.41s/it]
Generating samples:  83%|████████▎ | 29/35 [05:03<01:02, 10.43s/it]
Generating samples:  86%|████████▌ | 30/35 [05:14<00:52, 10.44s/it]
Generating samples:  89%|████████▊ | 31/35 [05:24<00:41, 10.46s/it]
Generating samples:  91%|█████████▏| 32/35 [05:35<00:31, 10.45s/it]
Generating samples:  94%|█████████▍| 33/35 [05:45<00:20, 10.46s/it]
Generating samples:  97%|█████████▋| 34/35 [05:56<00:10, 10.47s/it]
Generating samples: 100%|██████████| 35/35 [06:06<00:00, 10.47s/it]
Generating samples: 100%|██████████| 35/35 [06:06<00:00, 10.48s/it]

Generating samples:   0%|          | 0/35 [00:00<?, ?it/s]
Generating samples:   3%|▎         | 1/35 [00:10<05:55, 10.46s/it]
Generating samples:   6%|▌         | 2/35 [00:20<05:44, 10.44s/it]
Generating samples:   9%|▊         | 3/35 [00:31<05:35, 10.47s/it]
Generating samples:  11%|█▏        | 4/35 [00:41<05:24, 10.46s/it]
Generating samples:  14%|█▍        | 5/35 [00:52<05:14, 10.50s/it]
Generating samples:  17%|█▋        | 6/35 [01:02<05:04, 10.51s/it]
Generating samples:  20%|██        | 7/35 [01:13<04:54, 10.51s/it]
Generating samples:  23%|██▎       | 8/35 [01:23<04:43, 10.52s/it]
Generating samples:  26%|██▌       | 9/35 [01:34<04:33, 10.51s/it]
Generating samples:  29%|██▊       | 10/35 [01:44<04:22, 10.50s/it]
Generating samples:  31%|███▏      | 11/35 [01:55<04:12, 10.52s/it]
Generating samples:  34%|███▍      | 12/35 [02:05<04:01, 10.48s/it]
Generating samples:  37%|███▋      | 13/35 [02:16<03:51, 10.51s/it]
Generating samples:  40%|████      | 14/35 [02:26<03:40, 10.49s/it]
Generating samples:  43%|████▎     | 15/35 [02:37<03:30, 10.51s/it]
Generating samples:  46%|████▌     | 16/35 [02:47<03:19, 10.50s/it]
Generating samples:  49%|████▊     | 17/35 [02:58<03:09, 10.50s/it]
Generating samples:  51%|█████▏    | 18/35 [03:08<02:58, 10.50s/it]
Generating samples:  54%|█████▍    | 19/35 [03:19<02:47, 10.48s/it]
Generating samples:  57%|█████▋    | 20/35 [03:29<02:37, 10.48s/it]
Generating samples:  60%|██████    | 21/35 [03:40<02:26, 10.47s/it]
Generating samples:  63%|██████▎   | 22/35 [03:50<02:16, 10.49s/it]
Generating samples:  66%|██████▌   | 23/35 [04:01<02:05, 10.47s/it]
Generating samples:  69%|██████▊   | 24/35 [04:11<01:55, 10.46s/it]
Generating samples:  71%|███████▏  | 25/35 [04:22<01:44, 10.47s/it]
Generating samples:  74%|███████▍  | 26/35 [04:32<01:34, 10.46s/it]
Generating samples:  77%|███████▋  | 27/35 [04:43<01:23, 10.47s/it]
Generating samples:  80%|████████  | 28/35 [04:53<01:13, 10.47s/it]
Generating samples:  83%|████████▎ | 29/35 [05:04<01:02, 10.47s/it]
Generating samples:  86%|████████▌ | 30/35 [05:14<00:52, 10.46s/it]
Generating samples:  89%|████████▊ | 31/35 [05:24<00:41, 10.44s/it]
Generating samples:  91%|█████████▏| 32/35 [05:35<00:31, 10.46s/it]
Generating samples:  94%|█████████▍| 33/35 [05:45<00:20, 10.41s/it]
Generating samples:  97%|█████████▋| 34/35 [05:56<00:10, 10.43s/it]
Generating samples: 100%|██████████| 35/35 [06:06<00:00, 10.41s/it]
Generating samples: 100%|██████████| 35/35 [06:06<00:00, 10.47s/it]
[libx264 @ 0x6f6b840] VBV maxrate unspecified, assuming CBR
[07-31 16:50:37|INFO|cosmos_transfer2/inference.py:311:_generate_sample] vis control video saved to /srv/data/foundation-physics-graph-model/outputs/cosmos_runs/depth_vis/pick_place_books_depth_vis_control_vis.mp4
[libx264 @ 0x13fc5840] VBV maxrate unspecified, assuming CBR
[07-31 16:50:41|INFO|cosmos_transfer2/inference.py:311:_generate_sample] depth control video saved to /srv/data/foundation-physics-graph-model/outputs/cosmos_runs/depth_vis/pick_place_books_depth_vis_control_depth.mp4
[07-31 16:50:41|WARNING|cosmos_transfer2/inference.py:335:_generate_sample] Guardrail checks on video are disabled
[libx264 @ 0x23f68840] VBV maxrate unspecified, assuming CBR
[07-31 16:50:45|SUCCESS|cosmos_transfer2/inference.py:343:_generate_sample] Generated video saved to /srv/data/foundation-physics-graph-model/outputs/cosmos_runs/depth_vis/pick_place_books_depth_vis.mp4