IDEAS-Lab-Northwestern/gr00t-n16-sim2real-clutter-task0048-sim

GR00T-N1.6 - Sim2Real Clutter (task_0048) (joint, 2-cam)

NVIDIA Isaac GR00T-N1.6-3B fine-tuned on the ManiGuard clutter-task0048 base task (sim Franka Panda). Part of the ManiGuard VLA benchmark - GR00T vs pi0.5 on the same task families with identical data, cameras, and controller.

Model

• Base: nvidia/GR00T-N1.6-3B - Eagle (nvidia/Eagle-Block2A-2B-v2) VLM + flow-matching DiT action head
• Embodiment: NEW_EMBODIMENT - Franka Panda, 8-D joint state/action (7 arm joints + 1 gripper)
• Cameras (2): image_left (overview) + wrist (256x256)
• Action: arm = state-relative chunks (reconstructed to absolute at inference), gripper = absolute; 16-step horizon at 30 fps (0.53 s); NON_EEF (joint space)
• Tuning: GR00T-N1.6 default - VLM (LLM + visual) frozen, train projector + diffusion action head (no LoRA)

Training

• single-card config, bf16, global batch 4, 50000 steps (~8 epochs over 23,904 frames), cosine LR (peak 2.5e-5, sqrt-scaled), warmup 0.05
• Data: IDEAS-Lab-Northwestern/sim2real-clutter-task0048-sim-joint-5cam; videos decoded as H.264 for GR00T's torchcodec loader

Usage

Load with Gr00tPolicy from Isaac-GR00T (n1d6), --embodiment-tag NEW_EMBODIMENT. The included processor/ carries the normalization stats + modality config.

WARNING - Convention (must match at eval): joint-space JointController (absolute joint targets, NON_EEF) + 2 cameras (image_left overview + wrist). A mismatched controller or camera set silently feeds an out-of-distribution input.