JeffrinSam/GR00T-N1.7-G1-BrainCo-Pick

GR00T N1.7 — Unitree G1 BrainCo Pick (Ego-Centric)

Fine-tuned checkpoint of nvidia/GR00T-N1.7-3B on 1,598 real-robot episodes of tabletop pick-and-place tasks performed by a Unitree G1 humanoid equipped with BrainCo dexterous hands.

This model uses a single ego-centric head camera (no wrist cameras) and predicts 26-DOF arm + hand joint commands for left-arm manipulation.

---

Model Details

Property	Value
Base model	nvidia/GR00T-N1.7-3B
Architecture	VLM backbone (Cosmos-Reason2-2B / Qwen3-VL) + Diffusion action head (AlternateVLDiT)
Total parameters	~3B
Robot	Unitree G1 humanoid + BrainCo dexterous hands
Camera	1× head stereo-left (ego-centric only)
State dim	26D (14D arm joints + 12D hand state)
Action dim	26D (14D relative arm targets + 12D absolute hand commands)
Action horizon	16 steps at 30 fps
Inference timesteps	4 (flow matching)
Final training loss	0.020
Training steps	20,000
Training hardware	1× NVIDIA RTX 5090 (32 GB)

---

Training Data

8 picking tasks, 1,598 total episodes from the Unitree G1 BrainCo dataset:

Task	Episodes	Object	Notes
GraspOreo	201	Oreo cookie box	Rigid box
GraspRubiksCube	197	Rubik's cube	Rigid, 3D geometry
PickApple	200	Apple	Soft, deformable
PickCharger	200	USB charger	Small, fragile
PickDoll	200	Doll	Articulated, soft
PickDrink	201	Bottle / can	Cylindrical, slippery
PickTissues	206	Tissue box	Light, crushable
PickToothpaste	193	Toothpaste tube	Small cylinder

Data format: GR00T-flavored LeRobot v2.1 (per-episode HDF5-backed parquet + mp4 videos).

Data pipeline: Raw episodes were captured as human-teleoperated demonstrations, IK-retargeted to the G1 robot via training_data_g1brainco_newik.npz, and converted from LeRobot v3.0 to v2.1 format with the preprocessing script at examples/G1_Brainco/preprocess_brainco_pick.py in the Isaac GR00T repo.

---

Training Configuration

base_model:              nvidia/GR00T-N1.7-3B
embodiment_tag:          NEW_EMBODIMENT
max_steps:               20000
learning_rate:           1e-4
warmup_ratio:            0.05
weight_decay:            1e-5
optimizer:               paged_adamw_8bit
global_batch_size:       1
gradient_accumulation:   64          # effective batch = 64
gradient_checkpointing:  true
num_shards_per_epoch:    2000
shard_size:              512
episode_sampling_rate:   0.1
color_jitter:            brightness=0.3, contrast=0.4, saturation=0.5, hue=0.08
tune_llm:                false
tune_visual:             false
tune_projector:          true
tune_diffusion_model:    true

Loss Curve

Step	Loss	LR
10	1.230	9e-7
1000	0.0942	9.99e-5
2000	0.0676	9.93e-5
4000	0.0559	9.40e-5
6000	0.0431	8.39e-5
8000	0.0328	7.01e-5
10000	0.0406	5.41e-5
14000	0.0280	2.27e-5
18000	0.0302	2.71e-6
20000	0.0200	~0

---

Modality Configuration

# g1_brainco_config_pick_ego.py
config = {
    "video": ModalityConfig(
        delta_indices=[0],
        modality_keys=["observation.images.head_stereo_left"],  # ego-centric only
    ),
    "state": ModalityConfig(
        delta_indices=[0],
        modality_keys=[
            "observation.state.arm_q",       # 14D arm joint positions (rad)
            "observation.state.hand_state",  # 12D BrainCo finger state
        ],
    ),
    "action": ModalityConfig(
        delta_indices=list(range(16)),       # 16-step prediction horizon
        modality_keys=[
            "action.arm_q",    # 14D relative arm joint targets
            "action.hand_cmd", # 12D absolute hand commands
        ],
        action_configs=[
            ActionConfig(rep=ActionRepresentation.RELATIVE, type=ActionType.NON_EEF,
                         state_key="observation.state.arm_q"),
            ActionConfig(rep=ActionRepresentation.ABSOLUTE, type=ActionType.NON_EEF),
        ],
    ),
    "language": ModalityConfig(
        delta_indices=[0],
        modality_keys=["annotation.human.task_description"],
    ),
}

---

Quick Start

Install

git clone https://github.com/NVIDIA/Isaac-GR00T
cd Isaac-GR00T
uv sync --all-extras
# or: pip install -e ".[dev]"

Download the checkpoint

from huggingface_hub import snapshot_download

model_path = snapshot_download("JeffrinSam/GR00T-N1.7-G1-BrainCo-Pick")

Run inference server

python gr00t/eval/run_gr00t_server.py \
    --model-path JeffrinSam/GR00T-N1.7-G1-BrainCo-Pick \
    --embodiment-tag NEW_EMBODIMENT \
    --modality-config-path modality_config.py \
    --denoising-steps 4

Open-loop evaluation

python gr00t/eval/open_loop_eval.py \
    --dataset-path /path/to/brainco_pick/G1_Brainco_PickApple_Dataset \
    --embodiment-tag NEW_EMBODIMENT \
    --model-path JeffrinSam/GR00T-N1.7-G1-BrainCo-Pick \
    --modality-config-path modality_config.py

Python inference

import numpy as np
from gr00t.model.gr00t_n1d7.setup import Gr00tN1d7Pipeline

pipeline = Gr00tN1d7Pipeline.from_pretrained(
    "JeffrinSam/GR00T-N1.7-G1-BrainCo-Pick",
    modality_config_path="modality_config.py",
)

# Single inference call
output = pipeline.get_action(
    observation={
        # Head camera frame: (H, W, 3) uint8
        "observation.images.head_stereo_left": head_frame,
        # 14D arm joint positions (radians)
        "observation.state.arm_q": arm_q,
        # 12D BrainCo finger state
        "observation.state.hand_state": hand_state,
    },
    language_instruction="Pick up the apple and place it in the bowl",
)

# output["action.arm_q"]   → (16, 14) relative arm delta targets
# output["action.hand_cmd"] → (16, 12) absolute hand commands

---

Deploying on Real Unitree G1 + BrainCo Hands

Hardware requirements

• Unitree G1 humanoid robot
• BrainCo dexterous hands (6-DOF per hand)
• Head stereo-left camera (640×480 RGB, 30 fps)
• Inference host: ≥16 GB VRAM GPU (RTX 4090 / H100 / Jetson Thor)

Action space mapping

output["action.arm_q"]    (16, 14) — relative delta from current arm_q
    [0:7]   left arm  : shoulder_pitch, shoulder_roll, shoulder_yaw,
                         elbow_pitch, elbow_roll, wrist_pitch, wrist_yaw
    [7:14]  right arm : (same order; right arm stationary in training data)

output["action.hand_cmd"] (16, 12) — absolute finger positions [0, 1]
    [0:6]   left hand : thumb, index, middle, ring, pinky + spread
    [6:12]  right hand: (same order)

Control loop (30 Hz skeleton)

import numpy as np
from gr00t.model.gr00t_n1d7.setup import Gr00tN1d7Pipeline

pipeline = Gr00tN1d7Pipeline.from_pretrained(
    "JeffrinSam/GR00T-N1.7-G1-BrainCo-Pick",
    modality_config_path="modality_config.py",
)

action_buffer = None
action_step = 0
HORIZON = 16

while True:
    # 1. Read sensors
    head_img  = robot.get_camera("head_stereo_left")          # (480, 640, 3) uint8
    arm_q     = robot.get_arm_joint_positions()               # (14,) radians
    hand_state = robot.get_hand_state()                       # (12,) [0,1]

    # 2. Infer (every HORIZON steps to amortise diffusion cost)
    if action_step == 0 or action_buffer is None:
        output = pipeline.get_action(
            observation={
                "observation.images.head_stereo_left": head_img,
                "observation.state.arm_q": arm_q,
                "observation.state.hand_state": hand_state,
            },
            language_instruction="Pick up the apple",
        )
        action_buffer = output
        action_step = 0

    # 3. Execute current step
    delta_arm  = action_buffer["action.arm_q"][action_step]    # (14,)
    hand_cmd   = action_buffer["action.hand_cmd"][action_step] # (12,)

    target_arm = arm_q + delta_arm
    target_arm = np.clip(target_arm, -np.pi, np.pi)

    robot.set_arm_joints(target_arm, kp=15, kd=0.5)
    robot.set_hand_joints(hand_cmd)

    action_step = (action_step + 1) % HORIZON
    time.sleep(1 / 30.0)

---

Files in this Repository

File	Description
config.json	Model architecture config (GR00T N1.7)
model-0000*-of-00003.safetensors	Fine-tuned weights (sharded, ~12 GB total)
model.safetensors.index.json	Shard index
statistics.json	Per-embodiment normalization stats (mean/std for state, action)
embodiment_id.json	Embodiment ID table used during training
modality_config.py	Modality config (ego-only camera, 26D state/action) — required for inference
trainer_state.json	Full training log (loss per step)

---

Reproducing Training

1. Clone the repo

   git clone https://github.com/NVIDIA/Isaac-GR00T
   cd Isaac-GR00T
   uv sync --all-extras
   

2. Download and preprocess the datasets

   python examples/G1_Brainco/preprocess_brainco_pick.py
   # Downloads 8 datasets from unitreerobotics HF org → dataset_g1/brainco_pick/
   

3. Launch fine-tuning

   bash examples/G1_Brainco/finetune_g1_brainco_pick.sh
   

   Hardware notes for 32 GB VRAM (RTX 5090):

   PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
   --gradient_checkpointing
   --optim paged_adamw_8bit
   --global_batch_size 1 --gradient_accumulation_steps 64
   # Peak ~26 GB, ~5.8 s/step, 20 000 steps ≈ 32 h
   

---

Limitations

• Trained on left-arm only demonstrations; right-arm commands are zero throughout.
• Single ego-centric head camera — no wrist-camera feedback. Fine grasping of very small objects may be less precise than a 3-camera setup.
• Data collected in a specific lab tabletop environment; real-world lighting and clutter variation may affect performance.
• Not trained for whole-body or locomotion control.

---

Citation

If you use this checkpoint, please cite the base GR00T N1.7 model:

@misc{gr00tn17_2025,
  title  = {Isaac GR00T N1.7: Open Foundation Model for Generalized Humanoid Control},
  author = {NVIDIA},
  year   = {2025},
  url    = {https://huggingface.co/nvidia/GR00T-N1.7-3B}
}

And the Unitree G1 BrainCo datasets:

@misc{unitree_brainco_2025,
  title  = {Unitree G1 BrainCo Manipulation Dataset},
  author = {Unitree Robotics},
  year   = {2025},
  url    = {https://huggingface.co/unitreerobotics}
}

---

License

Weights are released under the same Apache 2.0 license as the base GR00T N1.7 model.