|
|
| ***************************************** |
| Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. |
| ***************************************** |
| /home/ubuntu/Isaac-GR00T/.venv/lib/python3.10/site-packages/albumentations/__init__.py:13: UserWarning: A new version of Albumentations is available: 2.0.8 (you have 1.4.18). Upgrade using: pip install -U albumentations. To disable automatic update checks, set the environment variable NO_ALBUMENTATIONS_UPDATE to 1. |
| check_for_updates() |
| /home/ubuntu/Isaac-GR00T/.venv/lib/python3.10/site-packages/albumentations/__init__.py:13: UserWarning: A new version of Albumentations is available: 2.0.8 (you have 1.4.18). Upgrade using: pip install -U albumentations. To disable automatic update checks, set the environment variable NO_ALBUMENTATIONS_UPDATE to 1. |
| check_for_updates() |
| /home/ubuntu/Isaac-GR00T/.venv/lib/python3.10/site-packages/albumentations/__init__.py:13: UserWarning: A new version of Albumentations is available: 2.0.8 (you have 1.4.18). Upgrade using: pip install -U albumentations. To disable automatic update checks, set the environment variable NO_ALBUMENTATIONS_UPDATE to 1. |
| check_for_updates() |
| /home/ubuntu/Isaac-GR00T/gr00t/experiment/experiment.py:98: UserWarning: image_crop_size and image_target_size will be deprecated in the future. Please use shortest_image_edge and crop_fraction instead. |
| warnings.warn( |
| /home/ubuntu/Isaac-GR00T/gr00t/experiment/experiment.py:98: UserWarning: image_crop_size and image_target_size will be deprecated in the future. Please use shortest_image_edge and crop_fraction instead. |
| warnings.warn( |
| /home/ubuntu/Isaac-GR00T/gr00t/experiment/experiment.py:98: UserWarning: image_crop_size and image_target_size will be deprecated in the future. Please use shortest_image_edge and crop_fraction instead. |
| warnings.warn( |
| /home/ubuntu/Isaac-GR00T/.venv/lib/python3.10/site-packages/albumentations/__init__.py:13: UserWarning: A new version of Albumentations is available: 2.0.8 (you have 1.4.18). Upgrade using: pip install -U albumentations. To disable automatic update checks, set the environment variable NO_ALBUMENTATIONS_UPDATE to 1. |
| check_for_updates() |
| /home/ubuntu/Isaac-GR00T/gr00t/experiment/experiment.py:98: UserWarning: image_crop_size and image_target_size will be deprecated in the future. Please use shortest_image_edge and crop_fraction instead. |
| warnings.warn( |
| 05/28/2026 08:31:17 - INFO - Saved config to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-083109/experiment_cfg |
| wandb: Currently logged in as: lucafrat (lucafrat-microsoft) to https: |
| Flash Attention 2 only supports torch.float16 and torch.bfloat16 dtypes, but the current dype in Qwen3VLForConditionalGeneration is torch.float32. You should run training or inference using Automatic Mixed-Precision via the `with torch.autocast(device_type='torch_device'):` decorator, or load the model with the `dtype` argument. Example: `model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="flash_attention_2", dtype=torch.float16)` |
| Flash Attention 2 only supports torch.float16 and torch.bfloat16 dtypes, but the current dype in Qwen3VLModel is torch.float32. You should run training or inference using Automatic Mixed-Precision via the `with torch.autocast(device_type='torch_device'):` decorator, or load the model with the `dtype` argument. Example: `model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="flash_attention_2", dtype=torch.float16)` |
| Flash Attention 2 only supports torch.float16 and torch.bfloat16 dtypes, but the current dype in Qwen3VLVisionModel is torch.float32. You should run training or inference using Automatic Mixed-Precision via the `with torch.autocast(device_type='torch_device'):` decorator, or load the model with the `dtype` argument. Example: `model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="flash_attention_2", dtype=torch.float16)` |
| Flash Attention 2 only supports torch.float16 and torch.bfloat16 dtypes, but the current dype in Qwen3VLTextModel is torch.float32. You should run training or inference using Automatic Mixed-Precision via the `with torch.autocast(device_type='torch_device'):` decorator, or load the model with the `dtype` argument. Example: `model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="flash_attention_2", dtype=torch.float16)` |
| Flash Attention 2 only supports torch.float16 and torch.bfloat16 dtypes, but the current dype in Qwen3VLForConditionalGeneration is torch.float32. You should run training or inference using Automatic Mixed-Precision via the `with torch.autocast(device_type='torch_device'):` decorator, or load the model with the `dtype` argument. Example: `model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="flash_attention_2", dtype=torch.float16)` |
| Flash Attention 2 only supports torch.float16 and torch.bfloat16 dtypes, but the current dype in Qwen3VLModel is torch.float32. You should run training or inference using Automatic Mixed-Precision via the `with torch.autocast(device_type='torch_device'):` decorator, or load the model with the `dtype` argument. Example: `model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="flash_attention_2", dtype=torch.float16)` |
| Flash Attention 2 only supports torch.float16 and torch.bfloat16 dtypes, but the current dype in Qwen3VLVisionModel is torch.float32. You should run training or inference using Automatic Mixed-Precision via the `with torch.autocast(device_type='torch_device'):` decorator, or load the model with the `dtype` argument. Example: `model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="flash_attention_2", dtype=torch.float16)` |
| Flash Attention 2 only supports torch.float16 and torch.bfloat16 dtypes, but the current dype in Qwen3VLForConditionalGeneration is torch.float32. You should run training or inference using Automatic Mixed-Precision via the `with torch.autocast(device_type='torch_device'):` decorator, or load the model with the `dtype` argument. Example: `model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="flash_attention_2", dtype=torch.float16)` |
| Flash Attention 2 only supports torch.float16 and torch.bfloat16 dtypes, but the current dype in Qwen3VLTextModel is torch.float32. You should run training or inference using Automatic Mixed-Precision via the `with torch.autocast(device_type='torch_device'):` decorator, or load the model with the `dtype` argument. Example: `model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="flash_attention_2", dtype=torch.float16)` |
| Flash Attention 2 only supports torch.float16 and torch.bfloat16 dtypes, but the current dype in Qwen3VLModel is torch.float32. You should run training or inference using Automatic Mixed-Precision via the `with torch.autocast(device_type='torch_device'):` decorator, or load the model with the `dtype` argument. Example: `model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="flash_attention_2", dtype=torch.float16)` |
| Flash Attention 2 only supports torch.float16 and torch.bfloat16 dtypes, but the current dype in Qwen3VLVisionModel is torch.float32. You should run training or inference using Automatic Mixed-Precision via the `with torch.autocast(device_type='torch_device'):` decorator, or load the model with the `dtype` argument. Example: `model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="flash_attention_2", dtype=torch.float16)` |
| Flash Attention 2 only supports torch.float16 and torch.bfloat16 dtypes, but the current dype in Qwen3VLTextModel is torch.float32. You should run training or inference using Automatic Mixed-Precision via the `with torch.autocast(device_type='torch_device'):` decorator, or load the model with the `dtype` argument. Example: `model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="flash_attention_2", dtype=torch.float16)` |
| wandb: Tracking run with wandb version 0.23.0 |
| wandb: Run data is saved locally in /home/ubuntu/Isaac-GR00T/wandb/run-20260528_083117-4nqwtde8 |
| wandb: Run `wandb offline` to turn off syncing. |
| wandb: Syncing run run-2026-05-28-083109 |
| wandb: ⭐️ View project at https: |
| wandb: 🚀 View run at https: |
| Flash Attention 2 only supports torch.float16 and torch.bfloat16 dtypes, but the current dype in Qwen3VLForConditionalGeneration is torch.float32. You should run training or inference using Automatic Mixed-Precision via the `with torch.autocast(device_type='torch_device'):` decorator, or load the model with the `dtype` argument. Example: `model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="flash_attention_2", dtype=torch.float16)` |
| Flash Attention 2 only supports torch.float16 and torch.bfloat16 dtypes, but the current dype in Qwen3VLModel is torch.float32. You should run training or inference using Automatic Mixed-Precision via the `with torch.autocast(device_type='torch_device'):` decorator, or load the model with the `dtype` argument. Example: `model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="flash_attention_2", dtype=torch.float16)` |
| Flash Attention 2 only supports torch.float16 and torch.bfloat16 dtypes, but the current dype in Qwen3VLVisionModel is torch.float32. You should run training or inference using Automatic Mixed-Precision via the `with torch.autocast(device_type='torch_device'):` decorator, or load the model with the `dtype` argument. Example: `model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="flash_attention_2", dtype=torch.float16)` |
| Flash Attention 2 only supports torch.float16 and torch.bfloat16 dtypes, but the current dype in Qwen3VLTextModel is torch.float32. You should run training or inference using Automatic Mixed-Precision via the `with torch.autocast(device_type='torch_device'):` decorator, or load the model with the `dtype` argument. Example: `model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="flash_attention_2", dtype=torch.float16)` |
| /home/ubuntu/Isaac-GR00T/gr00t/model/modules/dit.py:255: FutureWarning: Accessing config attribute `compute_dtype` directly via 'AlternateVLDiT' object attribute is deprecated. Please access 'compute_dtype' over 'AlternateVLDiT's config object instead, e.g. 'unet.config.compute_dtype'. |
| embedding_dim=self.inner_dim, compute_dtype=self.compute_dtype |
| /home/ubuntu/Isaac-GR00T/gr00t/model/modules/dit.py:286: FutureWarning: Accessing config attribute `output_dim` directly via 'AlternateVLDiT' object attribute is deprecated. Please access 'output_dim' over 'AlternateVLDiT's config object instead, e.g. 'unet.config.output_dim'. |
| self.proj_out_2 = nn.Linear(self.inner_dim, self.output_dim) |
| Total number of DiT parameters: 1091722240 |
| /home/ubuntu/Isaac-GR00T/gr00t/model/modules/dit.py:255: FutureWarning: Accessing config attribute `compute_dtype` directly via 'AlternateVLDiT' object attribute is deprecated. Please access 'compute_dtype' over 'AlternateVLDiT's config object instead, e.g. 'unet.config.compute_dtype'. |
| embedding_dim=self.inner_dim, compute_dtype=self.compute_dtype |
| /home/ubuntu/Isaac-GR00T/gr00t/model/modules/dit.py:286: FutureWarning: Accessing config attribute `output_dim` directly via 'AlternateVLDiT' object attribute is deprecated. Please access 'output_dim' over 'AlternateVLDiT's config object instead, e.g. 'unet.config.output_dim'. |
| self.proj_out_2 = nn.Linear(self.inner_dim, self.output_dim) |
| Total number of DiT parameters: 1091722240 |
| /home/ubuntu/Isaac-GR00T/gr00t/model/modules/dit.py:255: FutureWarning: Accessing config attribute `compute_dtype` directly via 'AlternateVLDiT' object attribute is deprecated. Please access 'compute_dtype' over 'AlternateVLDiT's config object instead, e.g. 'unet.config.compute_dtype'. |
| embedding_dim=self.inner_dim, compute_dtype=self.compute_dtype |
| /home/ubuntu/Isaac-GR00T/gr00t/model/modules/dit.py:286: FutureWarning: Accessing config attribute `output_dim` directly via 'AlternateVLDiT' object attribute is deprecated. Please access 'output_dim' over 'AlternateVLDiT's config object instead, e.g. 'unet.config.output_dim'. |
| self.proj_out_2 = nn.Linear(self.inner_dim, self.output_dim) |
| Total number of DiT parameters: 1091722240 |
| /home/ubuntu/Isaac-GR00T/gr00t/model/modules/dit.py:255: FutureWarning: Accessing config attribute `compute_dtype` directly via 'AlternateVLDiT' object attribute is deprecated. Please access 'compute_dtype' over 'AlternateVLDiT's config object instead, e.g. 'unet.config.compute_dtype'. |
| embedding_dim=self.inner_dim, compute_dtype=self.compute_dtype |
| /home/ubuntu/Isaac-GR00T/gr00t/model/modules/dit.py:286: FutureWarning: Accessing config attribute `output_dim` directly via 'AlternateVLDiT' object attribute is deprecated. Please access 'output_dim' over 'AlternateVLDiT's config object instead, e.g. 'unet.config.output_dim'. |
| self.proj_out_2 = nn.Linear(self.inner_dim, self.output_dim) |
| Total number of DiT parameters: 1091722240 |
| 05/28/2026 08:31:21 - INFO - Using AlternateVLDiT for diffusion model |
| Total number of SelfAttentionTransformer parameters: 201433088 |
| Total number of SelfAttentionTransformer parameters: 201433088 |
| Total number of SelfAttentionTransformer parameters: 201433088 |
| Total number of SelfAttentionTransformer parameters: 201433088 |
|
Loading checkpoint shards: 0%| | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards: 50%|█████ | 1/2 [00:04<00:04, 4.05s/it]
Loading checkpoint shards: 50%|█████ | 1/2 [00:04<00:04, 4.02s/it]
Loading checkpoint shards: 50%|█████ | 1/2 [00:04<00:04, 4.11s/it]
Loading checkpoint shards: 50%|█████ | 1/2 [00:03<00:03, 3.98s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:05<00:00, 2.58s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:05<00:00, 2.80s/it] |
|
Loading checkpoint shards: 100%|██████████| 2/2 [00:05<00:00, 2.57s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:05<00:00, 2.79s/it] |
|
Loading checkpoint shards: 100%|██████████| 2/2 [00:05<00:00, 2.64s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:05<00:00, 2.86s/it] |
|
Loading checkpoint shards: 100%|██████████| 2/2 [00:05<00:00, 2.54s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:05<00:00, 2.75s/it] |
| 05/28/2026 08:31:30 - INFO - Total parameters: 3,144,016,000 |
| 05/28/2026 08:31:30 - INFO - Trainable parameters: 1,620,515,968 (51.54%) |
|
Initializing datasets: 0%| | 0/1 [00:00<?, ?it/s]/home/ubuntu/Isaac-GR00T/.venv/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:4631: UserWarning: No device id is provided via `init_process_group` or `barrier `. Using the current device set by the user. |
| warnings.warn( # warn only once |
| [rank1]:[W528 08:31:32.390787316 ProcessGroupNCCL.cpp:4718] [PG ID 0 PG GUID 0 Rank 1] using GPU 1 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can pecify device_id in init_process_group() to force use of a particular device. |
|
Initializing datasets: 0%| | 0/1 [00:00<?, ?it/s]/home/ubuntu/Isaac-GR00T/.venv/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:4631: UserWarning: No device id is provided via `init_process_group` or `barrier `. Using the current device set by the user. |
| warnings.warn( # warn only once |
| [rank3]:[W528 08:31:33.638147504 ProcessGroupNCCL.cpp:4718] [PG ID 0 PG GUID 0 Rank 3] using GPU 3 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can pecify device_id in init_process_group() to force use of a particular device. |
|
Initializing datasets: 0%| | 0/1 [00:00<?, ?it/s]/home/ubuntu/Isaac-GR00T/.venv/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:4631: UserWarning: No device id is provided via `init_process_group` or `barrier `. Using the current device set by the user. |
| warnings.warn( # warn only once |
| [rank2]:[W528 08:31:33.890201587 ProcessGroupNCCL.cpp:4718] [PG ID 0 PG GUID 0 Rank 2] using GPU 2 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can pecify device_id in init_process_group() to force use of a particular device. |
|
Initializing datasets: 0%| | 0/1 [00:00<?, ?it/s]Generating stats for /home/ubuntu/groot-files/dataset_wbc_train |
| /home/ubuntu/Isaac-GR00T/.venv/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:4631: UserWarning: No device id is provided via `init_process_group` or `barrier `. Using the current device set by the user. |
| warnings.warn( # warn only once |
| [rank0]:[W528 08:31:34.468049813 ProcessGroupNCCL.cpp:4718] [PG ID 0 PG GUID 0 Rank 0] using GPU 0 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can pecify device_id in init_process_group() to force use of a particular device. |
| Generated 79 shards for dataset /home/ubuntu/groot-files/dataset_wbc_trainGenerated 79 shards for dataset /home/ubuntu/groot-files/dataset_wbc_train |
| Total steps: 80551, average shard length: 1019.632911392405, shard length std: 55.72199920427534 |
|
Initializing datasets: 100%|██████████| 1/1 [00:01<00:00, 1.41s/it]
Initializing datasets: 100%|██████████| 1/1 [00:01<00:00, 1.41s/it] |
|
|
| Total steps: 80551, average shard length: 1019.632911392405, shard length std: 55.72199920427534 |
|
Initializing datasets: 100%|██████████| 1/1 [00:00<00:00, 1.03it/s]
Initializing datasets: 100%|██████████| 1/1 [00:00<00:00, 1.03it/s] |
| Generated 79 shards for dataset /home/ubuntu/groot-files/dataset_wbc_trainGenerated 79 shards for dataset /home/ubuntu/groot-files/dataset_wbc_train |
|
|
| Total steps: 80551, average shard length: 1019.632911392405, shard length std: 55.72199920427534Total steps: 80551, average shard length: 1019.632911392405, shard length std: 55.72199920427534 |
|
|
|
Initializing datasets: 100%|██████████| 1/1 [00:01<00:00, 1.91s/it]
Initializing datasets: 100%|██████████| 1/1 [00:01<00:00, 1.67s/it]
Initializing datasets: 100%|██████████| 1/1 [00:01<00:00, 1.67s/it]
Initializing datasets: 100%|██████████| 1/1 [00:01<00:00, 1.91s/it] |
|
|
| 05/28/2026 08:31:34 - INFO - Overriding statistics for embodiment 'unitree_g1_sonic' |
| 05/28/2026 08:31:34 - INFO - Saved dataset statistics for inference |
| Generated 8 shards for dataset /home/ubuntu/groot-files/dataset_wbc_eval |
| Total steps: 8147, average shard length: 1018.375, shard length std: 45.367768018715665 |
| Generated 8 shards for dataset /home/ubuntu/groot-files/dataset_wbc_eval |
| Total steps: 8147, average shard length: 1018.375, shard length std: 45.367768018715665 |
| 05/28/2026 08:31:36 - INFO - Overriding statistics for embodiment 'unitree_g1_sonic' |
| Generated 8 shards for dataset /home/ubuntu/groot-files/dataset_wbc_eval |
| Total steps: 8147, average shard length: 1018.375, shard length std: 45.367768018715665 |
| Generated 8 shards for dataset /home/ubuntu/groot-files/dataset_wbc_eval |
| Total steps: 8147, average shard length: 1018.375, shard length std: 45.367768018715665 |
| 05/28/2026 08:31:36 - WARNING - No valid checkpoint found in output directory (/home/ubuntu/groot-files/checkpoints/run-2026-05-28-083109) |
| 05/28/2026 08:31:36 - INFO - Held-out eval enabled: /home/ubuntu/groot-files/dataset_wbc_eval every 1000 steps |
| 05/28/2026 08:31:36 - WARNING - No valid checkpoint found in output directory (/home/ubuntu/groot-files/checkpoints/run-2026-05-28-083109) |
| 05/28/2026 08:31:36 - WARNING - No valid checkpoint found in output directory (/home/ubuntu/groot-files/checkpoints/run-2026-05-28-083109) |
| 05/28/2026 08:31:36 - INFO - 🚀 Starting training... |
| 05/28/2026 08:31:36 - WARNING - No valid checkpoint found in output directory (/home/ubuntu/groot-files/checkpoints/run-2026-05-28-083109) |
| Current global step: 0 |
| Creating custom train dataloader |
| Current global step: 0 |
| Creating custom train dataloader |
| Current global step: 0 |
| Creating custom train dataloader |
| Current global step: 0 |
| Creating custom train dataloader |
| Rank 2, Worker 2: Caching shard...Rank 2, Worker 1: Caching shard... |
|
|
| Rank 2, Worker 0: Caching shard... |
| Rank 2, Worker 3: Caching shard... |
| Rank 2, Worker 4: Caching shard... |
| Rank 2, Worker 5: Caching shard... |
| Rank 3, Worker 0: Caching shard... |
| Rank 3, Worker 3: Caching shard...Rank 3, Worker 2: Caching shard...Rank 3, Worker 1: Caching shard... |
|
|
|
|
| Rank 3, Worker 4: Caching shard... |
| Rank 3, Worker 5: Caching shard... |
| Rank 1, Worker 1: Caching shard... |
| Rank 1, Worker 2: Caching shard...Rank 1, Worker 3: Caching shard... |
|
|
| Rank 1, Worker 4: Caching shard... |
| Rank 1, Worker 5: Caching shard... |
| Rank 1, Worker 0: Caching shard... |
|
0%| | 0/300 [00:00<?, ?it/s]Rank 0, Worker 2: Caching shard... |
| Rank 0, Worker 0: Caching shard...Rank 0, Worker 1: Caching shard... |
| |
| Rank 0, Worker 3: Caching shard... |
| Rank 0, Worker 4: Caching shard... |
| Rank 0, Worker 5: Caching shard... |
| Rank 3, Worker 2: Wait for shard 73 in dataset 0 in 20.18 seconds |
| Rank 3, Worker 2: Caching shard... |
| Rank 1, Worker 1: Wait for shard 65 in dataset 0 in 20.08 seconds |
| Rank 1, Worker 1: Caching shard... |
| Rank 3, Worker 1: Wait for shard 3 in dataset 0 in 20.43 seconds |
| Rank 3, Worker 1: Caching shard... |
| Rank 2, Worker 4: Wait for shard 30 in dataset 0 in 20.91 seconds |
| Rank 2, Worker 4: Caching shard... |
| Rank 3, Worker 5: Wait for shard 62 in dataset 0 in 21.06 seconds |
| Rank 3, Worker 5: Caching shard... |
| Rank 2, Worker 0: Wait for shard 39 in dataset 0 in 21.16 seconds |
| Rank 2, Worker 0: Caching shard... |
| Rank 3, Worker 4: Wait for shard 23 in dataset 0 in 21.14 seconds |
| Rank 3, Worker 4: Caching shard... |
| Rank 3, Worker 3: Wait for shard 6 in dataset 0 in 21.17 seconds |
| Rank 3, Worker 3: Caching shard... |
| Rank 2, Worker 2: Wait for shard 9 in dataset 0 in 21.63 seconds |
| Rank 2, Worker 2: Caching shard... |
| Rank 0, Worker 4: Wait for shard 22 in dataset 0 in 20.46 seconds |
| Rank 0, Worker 4: Caching shard... |
| Rank 1, Worker 4: Wait for shard 25 in dataset 0 in 21.50 seconds |
| Rank 1, Worker 4: Caching shard... |
| Rank 1, Worker 0: Wait for shard 11 in dataset 0 in 21.50 seconds |
| Rank 1, Worker 0: Caching shard... |
| Rank 2, Worker 1: Wait for shard 28 in dataset 0 in 21.85 seconds |
| Rank 2, Worker 1: Caching shard... |
| Rank 2, Worker 3: Wait for shard 29 in dataset 0 in 21.94 seconds |
| Rank 2, Worker 3: Caching shard... |
| Rank 0, Worker 5: Wait for shard 50 in dataset 0 in 20.90 seconds |
| Rank 0, Worker 5: Caching shard... |
| Rank 0, Worker 3: Wait for shard 2 in dataset 0 in 21.31 seconds |
| Rank 0, Worker 3: Caching shard... |
| Rank 2, Worker 5: Wait for shard 63 in dataset 0 in 22.87 seconds |
| Rank 2, Worker 5: Caching shard... |
| Rank 1, Worker 3: Wait for shard 5 in dataset 0 in 22.72 seconds |
| Rank 1, Worker 3: Caching shard... |
| Rank 1, Worker 2: Wait for shard 34 in dataset 0 in 23.64 seconds |
| Rank 1, Worker 2: Caching shard... |
| Rank 1, Worker 5: Wait for shard 45 in dataset 0 in 23.74 seconds |
| Rank 1, Worker 5: Caching shard... |
| Rank 0, Worker 2: Wait for shard 49 in dataset 0 in 22.83 seconds |
| Rank 0, Worker 2: Caching shard... |
| Rank 3, Worker 0: Wait for shard 40 in dataset 0 in 24.02 seconds |
| Rank 3, Worker 0: Caching shard... |
| Rank 0, Worker 0: Wait for shard 53 in dataset 0 in 23.24 seconds |
| Rank 0, Worker 0: Caching shard... |
| Rank 0, Worker 1: Wait for shard 47 in dataset 0 in 24.14 seconds |
| Rank 0, Worker 1: Caching shard... |
| Could not estimate the number of tokens of the input, floating-point operations will not be computed |
| Could not estimate the number of tokens of the input, floating-point operations will not be computed |
| Could not estimate the number of tokens of the input, floating-point operations will not be computed |
| Could not estimate the number of tokens of the input, floating-point operations will not be computed |
|
0%| | 1/300 [00:26<2:10:12, 26.13s/it]
1%| | 2/300 [00:26<54:23, 10.95s/it]
1%| | 3/300 [00:26<30:10, 6.10s/it]
1%|▏ | 4/300 [00:27<18:53, 3.83s/it]
2%|▏ | 5/300 [00:27<12:40, 2.58s/it]
2%|▏ | 6/300 [00:27<08:50, 1.81s/it]
2%|▏ | 7/300 [00:28<06:26, 1.32s/it]
3%|▎ | 8/300 [00:28<04:50, 1.00it/s]
3%|▎ | 9/300 [00:28<03:41, 1.31it/s]
3%|▎ | 10/300 [00:28<02:56, 1.64it/s]
{'loss': 1.1818, 'grad_norm': 0.24104289710521698, 'learning_rate': 6e-05} |
|
3%|▎ | 10/300 [00:28<02:56, 1.64it/s]
4%|▎ | 11/300 [00:29<02:27, 1.95it/s]
4%|▍ | 12/300 [00:29<02:13, 2.16it/s]
4%|▍ | 13/300 [00:29<01:59, 2.41it/s]
5%|▍ | 14/300 [00:30<01:49, 2.60it/s]
5%|▌ | 15/300 [00:30<01:45, 2.70it/s]
5%|▌ | 16/300 [00:30<01:41, 2.79it/s]
6%|▌ | 17/300 [00:31<01:34, 3.00it/s]
6%|▌ | 18/300 [00:31<01:32, 3.04it/s]
6%|▋ | 19/300 [00:31<01:28, 3.17it/s]
7%|▋ | 20/300 [00:32<01:29, 3.14it/s]
{'loss': 1.1533, 'grad_norm': 0.17898637056350708, 'learning_rate': 9.99514040972383e-05} |
|
7%|▋ | 20/300 [00:32<01:29, 3.14it/s]
7%|▋ | 21/300 [00:32<01:28, 3.14it/s]
7%|▋ | 22/300 [00:32<01:27, 3.19it/s]
8%|▊ | 23/300 [00:32<01:22, 3.38it/s]
8%|▊ | 24/300 [00:33<01:18, 3.53it/s]
8%|▊ | 25/300 [00:33<01:17, 3.53it/s]
9%|▊ | 26/300 [00:33<01:17, 3.54it/s]
9%|▉ | 27/300 [00:34<01:17, 3.50it/s]
9%|▉ | 28/300 [00:34<01:21, 3.32it/s]
10%|▉ | 29/300 [00:34<01:21, 3.31it/s]
10%|█ | 30/300 [00:34<01:21, 3.30it/s]
{'loss': 1.1299, 'grad_norm': 0.17362689971923828, 'learning_rate': 9.940578445376258e-05} |
|
10%|█ | 30/300 [00:34<01:21, 3.30it/s]
10%|█ | 31/300 [00:35<01:21, 3.29it/s]
11%|█ | 32/300 [00:35<01:18, 3.41it/s]
11%|█ | 33/300 [00:35<01:17, 3.46it/s]
11%|█▏ | 34/300 [00:36<01:14, 3.57it/s]
12%|█▏ | 35/300 [00:36<01:13, 3.60it/s]
12%|█▏ | 36/300 [00:36<01:17, 3.40it/s]
12%|█▏ | 37/300 [00:36<01:14, 3.51it/s]
13%|█▎ | 38/300 [00:37<01:15, 3.46it/s]
13%|█▎ | 39/300 [00:37<01:14, 3.52it/s]
13%|█▎ | 40/300 [00:37<01:16, 3.38it/s]
{'loss': 1.1172, 'grad_norm': 0.1847437471151352, 'learning_rate': 9.826044551386744e-05} |
|
13%|█▎ | 40/300 [00:37<01:16, 3.38it/s]
14%|█▎ | 41/300 [00:38<01:16, 3.39it/s]
14%|█▍ | 42/300 [00:38<01:17, 3.35it/s]
14%|█▍ | 43/300 [00:38<01:14, 3.43it/s]
15%|█▍ | 44/300 [00:39<01:15, 3.38it/s]
15%|█▌ | 45/300 [00:39<01:18, 3.24it/s]
15%|█▌ | 46/300 [00:39<01:15, 3.36it/s]
16%|█▌ | 47/300 [00:39<01:18, 3.23it/s]
16%|█▌ | 48/300 [00:40<01:18, 3.20it/s]
16%|█▋ | 49/300 [00:40<01:17, 3.23it/s]
17%|█▋ | 50/300 [00:40<01:15, 3.32it/s]
{'loss': 1.1125, 'grad_norm': 0.13554123044013977, 'learning_rate': 9.652929014076593e-05} |
|
17%|█▋ | 50/300 [00:40<01:15, 3.32it/s]/home/ubuntu/Isaac-GR00T/.venv/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:4631: UserWarning: No device id is provided via `init_process_group` or `barrier `. Using the current device set by the user. |
| warnings.warn( # warn only once |
| /home/ubuntu/Isaac-GR00T/.venv/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:4631: UserWarning: No device id is provided via `init_process_group` or `barrier `. Using the current device set by the user. |
| warnings.warn( # warn only once |
| /home/ubuntu/Isaac-GR00T/.venv/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:4631: UserWarning: No device id is provided via `init_process_group` or `barrier `. Using the current device set by the user. |
| warnings.warn( # warn only once |
| /home/ubuntu/Isaac-GR00T/.venv/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:4631: UserWarning: No device id is provided via `init_process_group` or `barrier `. Using the current device set by the user. |
| warnings.warn( # warn only once |
| Copying experiment config directory /home/ubuntu/groot-files/checkpoints/run-2026-05-28-083109/experiment_cfg to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-083109/checkpoint-50/experiment_cfg |
| Copying processor directory /home/ubuntu/groot-files/checkpoints/run-2026-05-28-083109/processor to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-083109/checkpoint-50 |
| Copying wandb_config.json from /home/ubuntu/groot-files/checkpoints/run-2026-05-28-083109/wandb_config.json to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-083109/checkpoint-50/wandb_config.json |
|
17%|█▋ | 51/300 [01:08<35:28, 8.55s/it]
17%|█▋ | 52/300 [01:08<24:56, 6.03s/it]
18%|█▊ | 53/300 [01:09<17:35, 4.28s/it]
18%|█▊ | 54/300 [01:09<12:28, 3.04s/it]
18%|█▊ | 55/300 [01:09<08:54, 2.18s/it]
19%|█▊ | 56/300 [01:09<06:24, 1.58s/it]
19%|█▉ | 57/300 [01:09<04:40, 1.15s/it]
19%|█▉ | 58/300 [01:09<03:27, 1.17it/s]
20%|█▉ | 59/300 [01:10<02:36, 1.54it/s]
20%|██ | 60/300 [01:10<02:01, 1.98it/s]
{'loss': 1.1205, 'grad_norm': 0.16369155049324036, 'learning_rate': 9.42333322156023e-05} |
|
20%|██ | 60/300 [01:10<02:01, 1.98it/s]
20%|██ | 61/300 [01:10<01:36, 2.47it/s]
21%|██ | 62/300 [01:10<01:19, 3.00it/s]
21%|██ | 63/300 [01:10<01:07, 3.53it/s]
21%|██▏ | 64/300 [01:10<00:58, 4.02it/s]
22%|██▏ | 65/300 [01:11<00:52, 4.46it/s]
22%|██▏ | 66/300 [01:11<00:48, 4.84it/s]
22%|██▏ | 67/300 [01:11<00:45, 5.11it/s]
23%|██▎ | 68/300 [01:11<00:43, 5.34it/s]
23%|██▎ | 69/300 [01:11<00:41, 5.53it/s]
23%|██▎ | 70/300 [01:11<00:40, 5.64it/s]
{'loss': 1.1092, 'grad_norm': 0.1633358895778656, 'learning_rate': 9.140044155740101e-05} |
|
23%|██▎ | 70/300 [01:11<00:40, 5.64it/s]
24%|██▎ | 71/300 [01:12<00:39, 5.73it/s]
24%|██▍ | 72/300 [01:12<00:39, 5.81it/s]
24%|██▍ | 73/300 [01:12<00:38, 5.82it/s]
25%|██▍ | 74/300 [01:12<00:38, 5.87it/s]
25%|██▌ | 75/300 [01:12<00:38, 5.91it/s]
25%|██▌ | 76/300 [01:12<00:37, 5.94it/s]
26%|██▌ | 77/300 [01:13<00:37, 5.96it/s]
26%|██▌ | 78/300 [01:13<00:37, 5.97it/s]
26%|██▋ | 79/300 [01:13<00:37, 5.93it/s]
27%|██▋ | 80/300 [01:13<00:37, 5.94it/s]
{'loss': 1.1094, 'grad_norm': 0.13908536732196808, 'learning_rate': 8.806500562121723e-05} |
|
27%|██▋ | 80/300 [01:13<00:37, 5.94it/s]
27%|██▋ | 81/300 [01:13<00:36, 5.94it/s]
27%|██▋ | 82/300 [01:13<00:36, 5.96it/s]
28%|██▊ | 83/300 [01:14<00:36, 5.98it/s]
28%|██▊ | 84/300 [01:14<00:36, 6.00it/s]
28%|██▊ | 85/300 [01:14<00:36, 5.96it/s]
29%|██▊ | 86/300 [01:14<00:35, 5.97it/s]
29%|██▉ | 87/300 [01:14<00:35, 5.98it/s]
29%|██▉ | 88/300 [01:14<00:35, 5.98it/s]
30%|██▉ | 89/300 [01:15<00:35, 6.00it/s]
30%|███ | 90/300 [01:15<00:35, 5.93it/s]
{'loss': 1.1111, 'grad_norm': 0.15955759584903717, 'learning_rate': 8.4267512081015e-05} |
|
30%|███ | 90/300 [01:15<00:35, 5.93it/s]
30%|███ | 91/300 [01:15<00:35, 5.88it/s]
31%|███ | 92/300 [01:15<00:35, 5.91it/s]
31%|███ | 93/300 [01:15<00:34, 5.94it/s]
31%|███▏ | 94/300 [01:15<00:34, 5.96it/s]
32%|███▏ | 95/300 [01:16<00:34, 5.97it/s]
32%|███▏ | 96/300 [01:16<00:34, 5.93it/s]
32%|███▏ | 97/300 [01:16<00:34, 5.91it/s]
33%|███▎ | 98/300 [01:16<00:34, 5.93it/s]
33%|███▎ | 99/300 [01:16<00:33, 5.96it/s]
33%|███▎ | 100/300 [01:16<00:33, 5.99it/s]
{'loss': 1.1105, 'grad_norm': 0.20263013243675232, 'learning_rate': 8.005405736415126e-05} |
|
33%|███▎ | 100/300 [01:16<00:33, 5.99it/s]/home/ubuntu/Isaac-GR00T/.venv/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:4631: UserWarning: No device id is provided via `init_process_group` or `barrier `. Using the current device set by the user. |
| warnings.warn( # warn only once |
| Copying experiment config directory /home/ubuntu/groot-files/checkpoints/run-2026-05-28-083109/experiment_cfg to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-083109/checkpoint-100/experiment_cfg |
| Copying processor directory /home/ubuntu/groot-files/checkpoints/run-2026-05-28-083109/processor to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-083109/checkpoint-100 |
| Copying wandb_config.json from /home/ubuntu/groot-files/checkpoints/run-2026-05-28-083109/wandb_config.json to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-083109/checkpoint-100/wandb_config.json |
|
34%|███▎ | 101/300 [01:43<27:09, 8.19s/it]
34%|███▍ | 102/300 [01:43<19:04, 5.78s/it]
34%|███▍ | 103/300 [01:44<13:27, 4.10s/it]
35%|███▍ | 104/300 [01:44<09:32, 2.92s/it]
35%|███▌ | 105/300 [01:44<06:48, 2.09s/it]
35%|███▌ | 106/300 [01:44<04:53, 1.52s/it]
36%|███▌ | 107/300 [01:44<03:34, 1.11s/it]
36%|███▌ | 108/300 [01:44<02:39, 1.21it/s]
36%|███▋ | 109/300 [01:45<02:00, 1.58it/s]
37%|███▋ | 110/300 [01:45<01:33, 2.02it/s]
{'loss': 1.1064, 'grad_norm': 0.17603614926338196, 'learning_rate': 7.547578710319174e-05} |
|
37%|███▋ | 110/300 [01:45<01:33, 2.02it/s]
37%|███▋ | 111/300 [01:45<01:14, 2.52it/s]
37%|███▋ | 112/300 [01:45<01:01, 3.05it/s]
38%|███▊ | 113/300 [01:45<00:52, 3.58it/s]
38%|███▊ | 114/300 [01:46<00:45, 4.05it/s]
38%|███▊ | 115/300 [01:46<00:41, 4.46it/s]
39%|███▊ | 116/300 [01:46<00:38, 4.79it/s]
39%|███▉ | 117/300 [01:46<00:35, 5.11it/s]
39%|███▉ | 118/300 [01:46<00:34, 5.35it/s]
40%|███▉ | 119/300 [01:46<00:32, 5.52it/s]
40%|████ | 120/300 [01:47<00:32, 5.61it/s]
{'loss': 1.0955, 'grad_norm': 0.2784798741340637, 'learning_rate': 7.058827529721525e-05} |
|
40%|████ | 120/300 [01:47<00:32, 5.61it/s]
40%|████ | 121/300 [01:47<00:31, 5.67it/s]
41%|████ | 122/300 [01:47<00:31, 5.72it/s]
41%|████ | 123/300 [01:47<00:30, 5.81it/s]
41%|████▏ | 124/300 [01:47<00:29, 5.87it/s]
42%|████▏ | 125/300 [01:47<00:29, 5.90it/s]
42%|████▏ | 126/300 [01:48<00:29, 5.85it/s]
42%|████▏ | 127/300 [01:48<00:29, 5.85it/s]
43%|████▎ | 128/300 [01:48<00:29, 5.85it/s]
43%|████▎ | 129/300 [01:48<00:29, 5.89it/s]
43%|████▎ | 130/300 [01:48<00:28, 5.92it/s]
{'loss': 1.0814, 'grad_norm': 0.2869728207588196, 'learning_rate': 6.545084971874738e-05} |
|
43%|████▎ | 130/300 [01:48<00:28, 5.92it/s]
44%|████▎ | 131/300 [01:48<00:28, 5.92it/s]
44%|████▍ | 132/300 [01:49<00:28, 5.89it/s]
44%|████▍ | 133/300 [01:49<00:28, 5.87it/s]
45%|████▍ | 134/300 [01:49<00:28, 5.86it/s]
45%|████▌ | 135/300 [01:49<00:27, 5.91it/s]
45%|████▌ | 136/300 [01:49<00:27, 5.94it/s]
46%|████▌ | 137/300 [01:49<00:27, 5.96it/s]
46%|████▌ | 138/300 [01:50<00:27, 5.97it/s]
46%|████▋ | 139/300 [01:50<00:27, 5.94it/s]
47%|████▋ | 140/300 [01:50<00:26, 5.96it/s]
{'loss': 1.0734, 'grad_norm': 0.22003582119941711, 'learning_rate': 6.012587175496961e-05} |
|
47%|████▋ | 140/300 [01:50<00:26, 5.96it/s]
47%|████▋ | 141/300 [01:50<00:26, 5.93it/s]
47%|████▋ | 142/300 [01:50<00:26, 5.95it/s]
48%|████▊ | 143/300 [01:50<00:26, 5.97it/s]
48%|████▊ | 144/300 [01:51<00:26, 5.96it/s]
48%|████▊ | 145/300 [01:51<00:26, 5.92it/s]
49%|████▊ | 146/300 [01:51<00:25, 5.95it/s]
49%|████▉ | 147/300 [01:51<00:25, 5.93it/s]
49%|████▉ | 148/300 [01:51<00:25, 5.95it/s]
50%|████▉ | 149/300 [01:51<00:25, 5.92it/s]
50%|█████ | 150/300 [01:52<00:25, 5.94it/s]
{'loss': 1.0592, 'grad_norm': 0.30818697810173035, 'learning_rate': 5.467797942495589e-05} |
|
50%|█████ | 150/300 [01:52<00:25, 5.94it/s]/home/ubuntu/Isaac-GR00T/.venv/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:4631: UserWarning: No device id is provided via `init_process_group` or `barrier `. Using the current device set by the user. |
| warnings.warn( # warn only once |
| Copying experiment config directory /home/ubuntu/groot-files/checkpoints/run-2026-05-28-083109/experiment_cfg to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-083109/checkpoint-150/experiment_cfg |
| Copying processor directory /home/ubuntu/groot-files/checkpoints/run-2026-05-28-083109/processor to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-083109/checkpoint-150 |
| Copying wandb_config.json from /home/ubuntu/groot-files/checkpoints/run-2026-05-28-083109/wandb_config.json to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-083109/checkpoint-150/wandb_config.json |
|
50%|█████ | 151/300 [02:21<22:00, 8.86s/it]
51%|█████ | 152/300 [02:21<15:25, 6.25s/it]
51%|█████ | 153/300 [02:21<10:50, 4.43s/it]
51%|█████▏ | 154/300 [02:21<07:39, 3.15s/it]
52%|█████▏ | 155/300 [02:21<05:26, 2.25s/it]
52%|█████▏ | 156/300 [02:22<03:54, 1.63s/it]
52%|█████▏ | 157/300 [02:22<02:50, 1.19s/it]
53%|█████▎ | 158/300 [02:22<02:05, 1.13it/s]
53%|█████▎ | 159/300 [02:22<01:34, 1.49it/s]
53%|█████▎ | 160/300 [02:22<01:12, 1.92it/s]
{'loss': 1.042, 'grad_norm': 0.315724641084671, 'learning_rate': 4.917330276168208e-05} |
|
53%|█████▎ | 160/300 [02:22<01:12, 1.92it/s]
54%|█████▎ | 161/300 [02:22<00:57, 2.41it/s]
54%|█████▍ | 162/300 [02:23<00:46, 2.94it/s]
54%|█████▍ | 163/300 [02:23<00:39, 3.46it/s]
55%|█████▍ | 164/300 [02:23<00:34, 3.95it/s]
55%|█████▌ | 165/300 [02:23<00:30, 4.39it/s]
55%|█████▌ | 166/300 [02:23<00:28, 4.74it/s]
56%|█████▌ | 167/300 [02:23<00:26, 5.05it/s]
56%|█████▌ | 168/300 [02:24<00:24, 5.30it/s]
56%|█████▋ | 169/300 [02:24<00:23, 5.47it/s]
57%|█████▋ | 170/300 [02:24<00:23, 5.60it/s]
{'loss': 1.0178, 'grad_norm': 0.36880528926849365, 'learning_rate': 4.367866108300769e-05} |
|
57%|█████▋ | 170/300 [02:24<00:23, 5.60it/s]
57%|█████▋ | 171/300 [02:24<00:22, 5.67it/s]
57%|█████▋ | 172/300 [02:24<00:22, 5.77it/s]
58%|█████▊ | 173/300 [02:24<00:21, 5.84it/s]
58%|█████▊ | 174/300 [02:25<00:21, 5.90it/s]
58%|█████▊ | 175/300 [02:25<00:21, 5.90it/s]
59%|█████▊ | 176/300 [02:25<00:20, 5.91it/s]
59%|█████▉ | 177/300 [02:25<00:20, 5.91it/s]
59%|█████▉ | 178/300 [02:25<00:20, 5.94it/s]
60%|█████▉ | 179/300 [02:25<00:20, 5.92it/s]
60%|██████ | 180/300 [02:26<00:20, 5.94it/s]
{'loss': 0.997, 'grad_norm': 0.39281517267227173, 'learning_rate': 3.826075189567296e-05} |
|
60%|██████ | 180/300 [02:26<00:20, 5.94it/s]
60%|██████ | 181/300 [02:26<00:20, 5.91it/s]
61%|██████ | 182/300 [02:26<00:19, 5.91it/s]
61%|██████ | 183/300 [02:26<00:19, 5.91it/s]
61%|██████▏ | 184/300 [02:26<00:19, 5.94it/s]
62%|██████▏ | 185/300 [02:26<00:19, 5.96it/s]
62%|██████▏ | 186/300 [02:27<00:19, 6.00it/s]
62%|██████▏ | 187/300 [02:27<00:18, 5.96it/s]
63%|██████▎ | 188/300 [02:27<00:18, 5.93it/s]
63%|██████▎ | 189/300 [02:27<00:18, 5.93it/s]
63%|██████▎ | 190/300 [02:27<00:18, 5.95it/s]
{'loss': 0.9755, 'grad_norm': 0.3882957398891449, 'learning_rate': 3.298534127791785e-05} |
|
63%|██████▎ | 190/300 [02:27<00:18, 5.95it/s]
64%|██████▎ | 191/300 [02:27<00:18, 5.95it/s]
64%|██████▍ | 192/300 [02:28<00:18, 5.97it/s]
64%|██████▍ | 193/300 [02:28<00:17, 5.95it/s]
65%|██████▍ | 194/300 [02:28<00:17, 5.93it/s]
65%|██████▌ | 195/300 [02:28<00:17, 5.92it/s]
65%|██████▌ | 196/300 [02:28<00:17, 5.95it/s]
66%|██████▌ | 197/300 [02:28<00:17, 5.97it/s]
66%|██████▌ | 198/300 [02:29<00:17, 5.98it/s]
66%|██████▋ | 199/300 [02:29<00:16, 5.95it/s]
67%|██████▋ | 200/300 [02:29<00:16, 5.94it/s]
{'loss': 0.9598, 'grad_norm': 0.5218529105186462, 'learning_rate': 2.79164655683813e-05} |
|
67%|██████▋ | 200/300 [02:29<00:16, 5.94it/s]/home/ubuntu/Isaac-GR00T/.venv/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:4631: UserWarning: No device id is provided via `init_process_group` or `barrier `. Using the current device set by the user. |
| warnings.warn( # warn only once |
| Copying experiment config directory /home/ubuntu/groot-files/checkpoints/run-2026-05-28-083109/experiment_cfg to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-083109/checkpoint-200/experiment_cfg |
| Copying processor directory /home/ubuntu/groot-files/checkpoints/run-2026-05-28-083109/processor to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-083109/checkpoint-200 |
| Copying wandb_config.json from /home/ubuntu/groot-files/checkpoints/run-2026-05-28-083109/wandb_config.json to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-083109/checkpoint-200/wandb_config.json |
|
67%|██████▋ | 201/300 [02:57<13:51, 8.40s/it]
67%|██████▋ | 202/300 [02:57<09:40, 5.93s/it]
68%|██████▊ | 203/300 [02:57<06:47, 4.20s/it]
68%|██████▊ | 204/300 [02:57<04:46, 2.99s/it]
68%|██████▊ | 205/300 [02:57<03:23, 2.14s/it]
69%|██████▊ | 206/300 [02:57<02:25, 1.55s/it]
69%|██████▉ | 207/300 [02:58<01:45, 1.14s/it]
69%|██████▉ | 208/300 [02:58<01:17, 1.18it/s]
70%|██████▉ | 209/300 [02:58<00:58, 1.56it/s]
70%|███████ | 210/300 [02:58<00:44, 2.00it/s]
{'loss': 0.9238, 'grad_norm': 0.42040392756462097, 'learning_rate': 2.3115654051696095e-05} |
|
70%|███████ | 210/300 [02:58<00:44, 2.00it/s]
70%|███████ | 211/300 [02:58<00:35, 2.50it/s]
71%|███████ | 212/300 [02:58<00:29, 3.02it/s]
71%|███████ | 213/300 [02:59<00:24, 3.54it/s]
71%|███████▏ | 214/300 [02:59<00:21, 4.03it/s]
72%|███████▏ | 215/300 [02:59<00:18, 4.48it/s]
72%|███████▏ | 216/300 [02:59<00:17, 4.85it/s]
72%|███████▏ | 217/300 [02:59<00:16, 5.15it/s]
73%|███████▎ | 218/300 [02:59<00:15, 5.36it/s]
73%|███████▎ | 219/300 [03:00<00:14, 5.51it/s]
73%|███████▎ | 220/300 [03:00<00:14, 5.63it/s]
{'loss': 0.9209, 'grad_norm': 0.5135362148284912, 'learning_rate': 1.8641182076323148e-05} |
|
73%|███████▎ | 220/300 [03:00<00:14, 5.63it/s]
74%|███████▎ | 221/300 [03:00<00:13, 5.72it/s]
74%|███████▍ | 222/300 [03:00<00:13, 5.81it/s]
74%|███████▍ | 223/300 [03:00<00:13, 5.88it/s]
75%|███████▍ | 224/300 [03:00<00:12, 5.88it/s]
75%|███████▌ | 225/300 [03:01<00:12, 5.88it/s]
75%|███████▌ | 226/300 [03:01<00:12, 5.93it/s]
76%|███████▌ | 227/300 [03:01<00:12, 5.96it/s]
76%|███████▌ | 228/300 [03:01<00:12, 5.98it/s]
76%|███████▋ | 229/300 [03:01<00:11, 5.99it/s]
77%|███████▋ | 230/300 [03:01<00:11, 5.95it/s]
{'loss': 0.914, 'grad_norm': 0.4933675527572632, 'learning_rate': 1.4547363670763137e-05} |
|
77%|███████▋ | 230/300 [03:01<00:11, 5.95it/s]
77%|███████▋ | 231/300 [03:02<00:11, 5.91it/s]
77%|███████▋ | 232/300 [03:02<00:11, 5.95it/s]
78%|███████▊ | 233/300 [03:02<00:11, 5.95it/s]
78%|███████▊ | 234/300 [03:02<00:11, 6.00it/s]
78%|███████▊ | 235/300 [03:02<00:10, 6.00it/s]
79%|███████▊ | 236/300 [03:02<00:10, 5.96it/s]
79%|███████▉ | 237/300 [03:03<00:10, 5.94it/s]
79%|███████▉ | 238/300 [03:03<00:10, 5.96it/s]
80%|███████▉ | 239/300 [03:03<00:10, 5.94it/s]
80%|████████ | 240/300 [03:03<00:10, 5.96it/s]
{'loss': 0.9023, 'grad_norm': 0.43049290776252747, 'learning_rate': 1.0883892244826172e-05} |
|
80%|████████ | 240/300 [03:03<00:10, 5.96it/s]
80%|████████ | 241/300 [03:03<00:09, 5.93it/s]
81%|████████ | 242/300 [03:03<00:09, 5.91it/s]
81%|████████ | 243/300 [03:04<00:09, 5.88it/s]
81%|████████▏ | 244/300 [03:04<00:09, 5.91it/s]
82%|████████▏ | 245/300 [03:04<00:09, 5.94it/s]
82%|████████▏ | 246/300 [03:04<00:09, 5.96it/s]
82%|████████▏ | 247/300 [03:04<00:08, 5.96it/s]
83%|████████▎ | 248/300 [03:04<00:08, 5.94it/s]
83%|████████▎ | 249/300 [03:05<00:08, 5.94it/s]
83%|████████▎ | 250/300 [03:05<00:08, 5.95it/s]
{'loss': 0.881, 'grad_norm': 0.6489596366882324, 'learning_rate': 7.695237378953223e-06} |
|
83%|████████▎ | 250/300 [03:05<00:08, 5.95it/s]/home/ubuntu/Isaac-GR00T/.venv/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:4631: UserWarning: No device id is provided via `init_process_group` or `barrier `. Using the current device set by the user. |
| warnings.warn( # warn only once |
| Copying experiment config directory /home/ubuntu/groot-files/checkpoints/run-2026-05-28-083109/experiment_cfg to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-083109/checkpoint-250/experiment_cfg |
| Copying processor directory /home/ubuntu/groot-files/checkpoints/run-2026-05-28-083109/processor to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-083109/checkpoint-250 |
| Copying wandb_config.json from /home/ubuntu/groot-files/checkpoints/run-2026-05-28-083109/wandb_config.json to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-083109/checkpoint-250/wandb_config.json |
|
84%|████████▎ | 251/300 [03:34<07:11, 8.81s/it]
84%|████████▍ | 252/300 [03:34<04:59, 6.23s/it]
84%|████████▍ | 253/300 [03:34<03:27, 4.42s/it]
85%|████████▍ | 254/300 [03:34<02:24, 3.15s/it]
85%|████████▌ | 255/300 [03:35<01:41, 2.26s/it]
85%|████████▌ | 256/300 [03:35<01:11, 1.63s/it]
86%|████████▌ | 257/300 [03:35<00:51, 1.20s/it]
86%|████████▌ | 258/300 [03:35<00:37, 1.12it/s]
86%|████████▋ | 259/300 [03:35<00:27, 1.48it/s]
87%|████████▋ | 260/300 [03:35<00:21, 1.90it/s]
{'loss': 0.8806, 'grad_norm': 0.46545666456222534, 'learning_rate': 5.020105023749644e-06} |
|
87%|████████▋ | 260/300 [03:35<00:21, 1.90it/s]
87%|████████▋ | 261/300 [03:36<00:16, 2.37it/s]
87%|████████▋ | 262/300 [03:36<00:13, 2.87it/s]
88%|████████▊ | 263/300 [03:36<00:10, 3.37it/s]
88%|████████▊ | 264/300 [03:36<00:09, 3.85it/s]
88%|████████▊ | 265/300 [03:36<00:08, 4.29it/s]
89%|████████▊ | 266/300 [03:36<00:07, 4.65it/s]
89%|████████▉ | 267/300 [03:37<00:06, 4.90it/s]
89%|████████▉ | 268/300 [03:37<00:06, 5.11it/s]
90%|████████▉ | 269/300 [03:37<00:05, 5.23it/s]
90%|█████████ | 270/300 [03:37<00:05, 5.25it/s]
{'loss': 0.8742, 'grad_norm': 0.4423169195652008, 'learning_rate': 2.890967662177285e-06} |
|
90%|█████████ | 270/300 [03:37<00:05, 5.25it/s]
90%|█████████ | 271/300 [03:37<00:05, 5.37it/s]
91%|█████████ | 272/300 [03:38<00:05, 5.42it/s]
91%|█████████ | 273/300 [03:38<00:05, 5.38it/s]
91%|█████████▏| 274/300 [03:38<00:04, 5.39it/s]
92%|█████████▏| 275/300 [03:38<00:04, 5.40it/s]
92%|█████████▏| 276/300 [03:38<00:04, 5.50it/s]
92%|█████████▏| 277/300 [03:38<00:04, 5.56it/s]
93%|█████████▎| 278/300 [03:39<00:03, 5.56it/s]
93%|█████████▎| 279/300 [03:39<00:03, 5.57it/s]
93%|█████████▎| 280/300 [03:39<00:03, 5.62it/s]
{'loss': 0.8695, 'grad_norm': 0.5047191381454468, 'learning_rate': 1.333670137599713e-06} |
|
93%|█████████▎| 280/300 [03:39<00:03, 5.62it/s]
94%|█████████▎| 281/300 [03:39<00:03, 5.45it/s]
94%|█████████▍| 282/300 [03:39<00:03, 5.44it/s]
94%|█████████▍| 283/300 [03:40<00:03, 5.54it/s]
95%|█████████▍| 284/300 [03:40<00:02, 5.39it/s]
95%|█████████▌| 285/300 [03:40<00:02, 5.21it/s]
95%|█████████▌| 286/300 [03:40<00:02, 5.15it/s]
96%|█████████▌| 287/300 [03:40<00:02, 5.12it/s]
96%|█████████▌| 288/300 [03:41<00:02, 5.25it/s]
96%|█████████▋| 289/300 [03:41<00:02, 5.30it/s]
97%|█████████▋| 290/300 [03:41<00:01, 5.41it/s]
{'loss': 0.8673, 'grad_norm': 0.46933674812316895, 'learning_rate': 3.6711593239417973e-07} |
|
97%|█████████▋| 290/300 [03:41<00:01, 5.41it/s]
97%|█████████▋| 291/300 [03:41<00:01, 5.49it/s]
97%|█████████▋| 292/300 [03:41<00:01, 5.55it/s]
98%|█████████▊| 293/300 [03:41<00:01, 5.55it/s]
98%|█████████▊| 294/300 [03:42<00:01, 5.59it/s]
98%|█████████▊| 295/300 [03:42<00:00, 5.64it/s]
99%|█████████▊| 296/300 [03:42<00:00, 5.68it/s]
99%|█████████▉| 297/300 [03:42<00:00, 5.64it/s]
99%|█████████▉| 298/300 [03:42<00:00, 5.69it/s]
100%|█████████▉| 299/300 [03:42<00:00, 5.69it/s]
100%|██████████| 300/300 [03:43<00:00, 5.67it/s]
{'loss': 0.872, 'grad_norm': 0.4605172276496887, 'learning_rate': 3.0377052828489683e-09} |
|
100%|██████████| 300/300 [03:43<00:00, 5.67it/s]/home/ubuntu/Isaac-GR00T/.venv/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:4631: UserWarning: No device id is provided via `init_process_group` or `barrier `. Using the current device set by the user. |
| warnings.warn( # warn only once |
| Copying experiment config directory /home/ubuntu/groot-files/checkpoints/run-2026-05-28-083109/experiment_cfg to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-083109/checkpoint-300/experiment_cfg |
| Copying processor directory /home/ubuntu/groot-files/checkpoints/run-2026-05-28-083109/processor to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-083109/checkpoint-300 |
| Copying wandb_config.json from /home/ubuntu/groot-files/checkpoints/run-2026-05-28-083109/wandb_config.json to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-083109/checkpoint-300/wandb_config.json |
|
{'train_runtime': 252.1375, 'train_samples_per_second': 38.074, 'train_steps_per_second': 1.19, 'train_loss': 1.0189680989583334} |
|
100%|██████████| 300/300 [04:12<00:00, 5.67it/s]
100%|██████████| 300/300 [04:12<00:00, 1.19it/s] |
| 05/28/2026 08:36:07 - INFO - Model saved to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-083109 |
| 05/28/2026 08:36:07 - INFO - Training completed! |
| [1;34mwandb[0m: |
| [1;34mwandb[0m: 🚀 View run [33mrun-2026-05-28-083109[0m at: [34m[0m |
| [1;34mwandb[0m: Find logs at: [1;35mwandb/run-20260528_083117-4nqwtde8/logs[0m |
|
|
