Upload GR00T N1.5 fine-tuned policy at step 110000

README.md

@@ -0,0 +1,130 @@
+---
+library_name: lerobot
+pipeline_tag: robotics
+base_model: nvidia/GR00T-N1.5-3B
+base_model_relation: finetune
+datasets:
+- CoRL2026-CSI/Isaaclab-so101_11task_baseCaP_3300epi
+license: other
+license_name: nvidia-license
+license_link: https://developer.download.nvidia.com/licenses/NVIDIA-OneWay-Noncommercial-License-22Mar2022.pdf
+tags:
+- robotics
+- lerobot
+- groot
+- gr00t-n1.5
+- so101
+- imitation-learning
+- flow-matching
+- safetensors
+---
+
+# CoRL2026-CSI/IsaacLab-so101-multi-gr00t
+
+This is a LeRobot GR00T N1.5 policy fine-tuned from
+[`nvidia/GR00T-N1.5-3B`](https://huggingface.co/nvidia/GR00T-N1.5-3B) on
+[`CoRL2026-CSI/Isaaclab-so101_11task_baseCaP_3300epi`](https://huggingface.co/datasets/CoRL2026-CSI/Isaaclab-so101_11task_baseCaP_3300epi).
+
+The model is intended for SO-101 style manipulation experiments using RGB observations,
+robot proprioception, language instructions, and continuous action chunks. It is uploaded
+as a LeRobot policy checkpoint and should be loaded through the matching LeRobot GR00T
+implementation used for training.
+
+## Model Details
+
+- **Policy type:** GR00T N1.5
+- **Base model:** `nvidia/GR00T-N1.5-3B`
+- **Tokenizer assets:** `lerobot/eagle2hg-processor-groot-n1p5`
+- **Embodiment tag:** `new_embodiment`
+- **Observation steps:** `1`
+- **Action chunk size:** `16`
+- **Action steps:** `16`
+- **Max state/action dims:** `64` / `32`
+
+## Fine-Tuning Setup
+
+- **Training steps:** `110000`
+- **Approx. epochs:** `7.99`
+- **Final training samples:** `28160000`
+- **Final training loss:** `0.010196`
+- **Runtime:** `76.72 hours`
+- **Micro batch size:** `64`
+- **Gradient accumulation steps:** `2`
+- **Effective batch size:** `256`
+- **Optimizer:** `adamw`, lr `0.0001`, weight decay `1e-05`
+- **Scheduler:** `cosine_decay_with_warmup`
+- **Mixed precision:** bf16 policy weights/config enabled: `true`
+- **VLM/LLM backbone fine-tuned:** `false`
+- **Vision tower fine-tuned:** `false`
+- **Action projector fine-tuned:** `true`
+- **Action DiT fine-tuned:** `true`
+
+The GR00T vision-language backbone was frozen for this run. The action head projector
+and flow-matching diffusion transformer were fine-tuned.
+
+## Inputs
+
+- `observation.ee_pos.robot_xyzrpy`: `STATE`, shape `[6]`
+- `observation.gripper_binary`: `STATE`, shape `[1]`
+- `observation.images.left_wrist`: `VISUAL`, shape `[3, 480, 640]`
+- `observation.images.top`: `VISUAL`, shape `[3, 480, 640]`
+- `observation.state`: `STATE`, shape `[6]`
+- `observation.state.radian_urdf0`: `STATE`, shape `[6]`
+
+## Outputs
+
+- `action`: `ACTION`, shape `[6]`
+- `action.radian_urdf0`: `ACTION`, shape `[6]`
+
+## Usage
+
+Install and use the same LeRobot checkout/environment that contains the GR00T policy
+implementation, then point `policy.path` to this Hub repo.
+
+```bash
+lerobot-record \
+  --robot.type=<your_robot> \
+  --dataset.repo_id=<your_eval_dataset_repo> \
+  --policy.path=CoRL2026-CSI/IsaacLab-so101-multi-gr00t \
+  --episodes=10
+```
+
+For local Python usage, load the policy with LeRobot's policy factory or GR00T policy
+loader from the training checkout.
+
+## Evaluation
+
+This upload records the offline training run metrics only. No rollout success rate is
+claimed here unless a separate real/sim evaluation is added later.
+
+Final logged training metrics:
+
+- loss: `0.010196`
+- grad norm: `0.193945`
+- update time: `1.2521 s/step`
+- dataloading time: `0.0046 s/step`
+
+## Limitations and Safety
+
+This model is a robot control policy and can produce unsafe actions if deployed on
+hardware without appropriate validation, workspace limits, emergency stop handling, and
+task-specific safety checks. Test in simulation or a constrained setup before any
+physical deployment.
+
+The model is specialized to the training dataset and embodiment configuration. It may
+not transfer reliably to different cameras, calibration, action spaces, robot hardware,
+or tasks without further validation or fine-tuning.
+
+## License and Terms
+
+This model is a fine-tune of `nvidia/GR00T-N1.5-3B`; users are responsible for complying
+with the NVIDIA model license and any dataset/license constraints. See the base model
+card and NVIDIA license terms linked in the metadata.
+
+## Files
+
+- `model.safetensors`: fine-tuned policy weights
+- `config.json`: LeRobot GR00T policy config
+- `train_config.json`: training configuration
+- `policy_preprocessor.json` and `policy_postprocessor.json`: LeRobot processor pipelines
+- `policy_*_step_*.safetensors`: normalization/statistics state used by processors

config.json

@@ -0,0 +1,113 @@
+{
+    "type": "groot",
+    "n_obs_steps": 1,
+    "input_features": {
+        "observation.state": {
+            "type": "STATE",
+            "shape": [
+                6
+            ]
+        },
+        "observation.images.top": {
+            "type": "VISUAL",
+            "shape": [
+                3,
+                480,
+                640
+            ]
+        },
+        "observation.images.left_wrist": {
+            "type": "VISUAL",
+            "shape": [
+                3,
+                480,
+                640
+            ]
+        },
+        "observation.ee_pos.robot_xyzrpy": {
+            "type": "STATE",
+            "shape": [
+                6
+            ]
+        },
+        "observation.gripper_binary": {
+            "type": "STATE",
+            "shape": [
+                1
+            ]
+        },
+        "observation.state.radian_urdf0": {
+            "type": "STATE",
+            "shape": [
+                6
+            ]
+        }
+    },
+    "output_features": {
+        "action": {
+            "type": "ACTION",
+            "shape": [
+                6
+            ]
+        },
+        "action.radian_urdf0": {
+            "type": "ACTION",
+            "shape": [
+                6
+            ]
+        }
+    },
+    "device": "cuda",
+    "use_amp": false,
+    "use_peft": false,
+    "push_to_hub": true,
+    "repo_id": "CoRL2026-CSI/IsaacLab-so101-multi-gr00t",
+    "private": null,
+    "tags": null,
+    "license": null,
+    "pretrained_path": null,
+    "chunk_size": 16,
+    "n_action_steps": 16,
+    "max_state_dim": 64,
+    "max_action_dim": 32,
+    "normalization_mapping": {
+        "VISUAL": "IDENTITY",
+        "STATE": "MEAN_STD",
+        "ACTION": "MEAN_STD"
+    },
+    "image_size": [
+        224,
+        224
+    ],
+    "base_model_path": "nvidia/GR00T-N1.5-3B",
+    "tokenizer_assets_repo": "lerobot/eagle2hg-processor-groot-n1p5",
+    "embodiment_tag": "new_embodiment",
+    "tune_llm": false,
+    "tune_visual": false,
+    "tune_projector": true,
+    "tune_diffusion_model": true,
+    "lora_rank": 0,
+    "lora_alpha": 16,
+    "lora_dropout": 0.1,
+    "lora_full_model": false,
+    "optimizer_lr": 0.0001,
+    "optimizer_betas": [
+        0.95,
+        0.999
+    ],
+    "optimizer_eps": 1e-08,
+    "optimizer_weight_decay": 1e-05,
+    "warmup_ratio": 0.05,
+    "use_bf16": true,
+    "video_backend": "decord",
+    "balance_dataset_weights": true,
+    "balance_trajectory_weights": true,
+    "dataset_paths": null,
+    "output_dir": "./tmp/gr00t",
+    "save_steps": 1000,
+    "max_steps": 10000,
+    "batch_size": 32,
+    "dataloader_num_workers": 8,
+    "report_to": "wandb",
+    "resume": false
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f4fa283b77eed3cf2e41e0303fc2ea99b7ac226c049db5a35b5890055cc6ef17
+size 5448472032

policy_postprocessor.json

@@ -0,0 +1,20 @@
+{
+  "name": "policy_postprocessor",
+  "steps": [
+    {
+      "registry_name": "groot_action_unpack_unnormalize_v1",
+      "config": {
+        "env_action_dim": 6,
+        "normalize_min_max": true
+      },
+      "state_file": "policy_postprocessor_step_0_groot_action_unpack_unnormalize_v1.safetensors"
+    },
+    {
+      "registry_name": "device_processor",
+      "config": {
+        "device": "cpu",
+        "float_dtype": null
+      }
+    }
+  ]
+}

policy_postprocessor_step_0_groot_action_unpack_unnormalize_v1.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8401150b9e477f4110fc4ba7e65d7e522b31fd33b6d0ce5a01bcda6e26ca0a95
+size 19016

policy_preprocessor.json

@@ -0,0 +1,52 @@
+{
+  "name": "policy_preprocessor",
+  "steps": [
+    {
+      "registry_name": "rename_observations_processor",
+      "config": {
+        "rename_map": {}
+      }
+    },
+    {
+      "registry_name": "to_batch_processor",
+      "config": {}
+    },
+    {
+      "registry_name": "groot_pack_inputs_v3",
+      "config": {
+        "state_horizon": 1,
+        "action_horizon": 16,
+        "max_state_dim": 64,
+        "max_action_dim": 32,
+        "language_key": "task",
+        "formalize_language": false,
+        "embodiment_tag": "new_embodiment",
+        "embodiment_mapping": {
+          "new_embodiment": 31,
+          "oxe_droid": 17,
+          "agibot_genie1": 26,
+          "gr1": 24,
+          "so100": 2,
+          "unitree_g1": 3
+        },
+        "normalize_min_max": true
+      },
+      "state_file": "policy_preprocessor_step_2_groot_pack_inputs_v3.safetensors"
+    },
+    {
+      "registry_name": "groot_eagle_encode_v3",
+      "config": {}
+    },
+    {
+      "registry_name": "groot_eagle_collate_v3",
+      "config": {}
+    },
+    {
+      "registry_name": "device_processor",
+      "config": {
+        "device": "cuda",
+        "float_dtype": null
+      }
+    }
+  ]
+}

policy_preprocessor_step_2_groot_pack_inputs_v3.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8401150b9e477f4110fc4ba7e65d7e522b31fd33b6d0ce5a01bcda6e26ca0a95
+size 19016

train_config.json

@@ -0,0 +1,257 @@
+{
+    "dataset": {
+        "repo_id": "CoRL2026-CSI/Isaaclab-so101_11task_baseCaP_3300epi",
+        "root": null,
+        "episodes": null,
+        "image_transforms": {
+            "enable": false,
+            "max_num_transforms": 3,
+            "random_order": false,
+            "tfs": {
+                "brightness": {
+                    "weight": 1.0,
+                    "type": "ColorJitter",
+                    "kwargs": {
+                        "brightness": [
+                            0.8,
+                            1.2
+                        ]
+                    }
+                },
+                "contrast": {
+                    "weight": 1.0,
+                    "type": "ColorJitter",
+                    "kwargs": {
+                        "contrast": [
+                            0.8,
+                            1.2
+                        ]
+                    }
+                },
+                "saturation": {
+                    "weight": 1.0,
+                    "type": "ColorJitter",
+                    "kwargs": {
+                        "saturation": [
+                            0.5,
+                            1.5
+                        ]
+                    }
+                },
+                "hue": {
+                    "weight": 1.0,
+                    "type": "ColorJitter",
+                    "kwargs": {
+                        "hue": [
+                            -0.05,
+                            0.05
+                        ]
+                    }
+                },
+                "sharpness": {
+                    "weight": 1.0,
+                    "type": "SharpnessJitter",
+                    "kwargs": {
+                        "sharpness": [
+                            0.5,
+                            1.5
+                        ]
+                    }
+                },
+                "affine": {
+                    "weight": 1.0,
+                    "type": "RandomAffine",
+                    "kwargs": {
+                        "degrees": [
+                            -5.0,
+                            5.0
+                        ],
+                        "translate": [
+                            0.05,
+                            0.05
+                        ]
+                    }
+                }
+            }
+        },
+        "revision": null,
+        "use_imagenet_stats": true,
+        "video_backend": "torchcodec",
+        "streaming": false
+    },
+    "env": null,
+    "policy": {
+        "type": "groot",
+        "n_obs_steps": 1,
+        "input_features": {
+            "observation.state": {
+                "type": "STATE",
+                "shape": [
+                    6
+                ]
+            },
+            "observation.images.top": {
+                "type": "VISUAL",
+                "shape": [
+                    3,
+                    480,
+                    640
+                ]
+            },
+            "observation.images.left_wrist": {
+                "type": "VISUAL",
+                "shape": [
+                    3,
+                    480,
+                    640
+                ]
+            },
+            "observation.ee_pos.robot_xyzrpy": {
+                "type": "STATE",
+                "shape": [
+                    6
+                ]
+            },
+            "observation.gripper_binary": {
+                "type": "STATE",
+                "shape": [
+                    1
+                ]
+            },
+            "observation.state.radian_urdf0": {
+                "type": "STATE",
+                "shape": [
+                    6
+                ]
+            }
+        },
+        "output_features": {
+            "action": {
+                "type": "ACTION",
+                "shape": [
+                    6
+                ]
+            },
+            "action.radian_urdf0": {
+                "type": "ACTION",
+                "shape": [
+                    6
+                ]
+            }
+        },
+        "device": "cuda",
+        "use_amp": false,
+        "use_peft": false,
+        "push_to_hub": true,
+        "repo_id": "CoRL2026-CSI/IsaacLab-so101-multi-gr00t",
+        "private": null,
+        "tags": null,
+        "license": null,
+        "pretrained_path": null,
+        "chunk_size": 16,
+        "n_action_steps": 16,
+        "max_state_dim": 64,
+        "max_action_dim": 32,
+        "normalization_mapping": {
+            "VISUAL": "IDENTITY",
+            "STATE": "MEAN_STD",
+            "ACTION": "MEAN_STD"
+        },
+        "image_size": [
+            224,
+            224
+        ],
+        "base_model_path": "nvidia/GR00T-N1.5-3B",
+        "tokenizer_assets_repo": "lerobot/eagle2hg-processor-groot-n1p5",
+        "embodiment_tag": "new_embodiment",
+        "tune_llm": false,
+        "tune_visual": false,
+        "tune_projector": true,
+        "tune_diffusion_model": true,
+        "lora_rank": 0,
+        "lora_alpha": 16,
+        "lora_dropout": 0.1,
+        "lora_full_model": false,
+        "optimizer_lr": 0.0001,
+        "optimizer_betas": [
+            0.95,
+            0.999
+        ],
+        "optimizer_eps": 1e-08,
+        "optimizer_weight_decay": 1e-05,
+        "warmup_ratio": 0.05,
+        "use_bf16": true,
+        "video_backend": "decord",
+        "balance_dataset_weights": true,
+        "balance_trajectory_weights": true,
+        "dataset_paths": null,
+        "output_dir": "./tmp/gr00t",
+        "save_steps": 1000,
+        "max_steps": 10000,
+        "batch_size": 32,
+        "dataloader_num_workers": 8,
+        "report_to": "wandb",
+        "resume": false
+    },
+    "output_dir": "/home/work/hscho/corl_2026/AutoDataCollector/lerobot/outputs/train/gr00tn15_20260508_160623",
+    "job_name": "gr00tn15_20260508_160623",
+    "resume": false,
+    "seed": 1000,
+    "cudnn_deterministic": false,
+    "num_workers": 4,
+    "dataloader_prefetch_factor": 1,
+    "dataloader_persistent_workers": true,
+    "dataloader_pin_memory": true,
+    "batch_size": 64,
+    "gradient_accumulation_steps": 2,
+    "ddp_find_unused_parameters": false,
+    "profile_timing": false,
+    "groot_preprocess_in_workers": true,
+    "steps": 110000,
+    "eval_freq": 0,
+    "log_freq": 10,
+    "tolerance_s": 0.0001,
+    "save_checkpoint": true,
+    "save_freq": 20000,
+    "use_policy_training_preset": true,
+    "optimizer": {
+        "type": "adamw",
+        "lr": 0.0001,
+        "weight_decay": 1e-05,
+        "grad_clip_norm": 10.0,
+        "betas": [
+            0.95,
+            0.999
+        ],
+        "eps": 1e-08
+    },
+    "scheduler": {
+        "type": "cosine_decay_with_warmup",
+        "num_warmup_steps": 500,
+        "num_decay_steps": 10000,
+        "peak_lr": 0.0001,
+        "decay_lr": 1e-05
+    },
+    "eval": {
+        "n_episodes": 50,
+        "batch_size": 50,
+        "use_async_envs": false
+    },
+    "wandb": {
+        "enable": true,
+        "disable_artifact": true,
+        "project": "lerobot-groot",
+        "entity": null,
+        "notes": null,
+        "run_id": "sz16gh68",
+        "mode": null
+    },
+    "peft": null,
+    "use_rabc": false,
+    "rabc_progress_path": null,
+    "rabc_kappa": 0.01,
+    "rabc_epsilon": 1e-06,
+    "rabc_head_mode": "sparse",
+    "rename_map": {},
+    "checkpoint_path": null
+}