GR00T N1.7, TAPTIC multitask (Panda + XHand), batch 8, 200k steps planned

Training still in progress: this is checkpoint step 100000 (branch step-100000; main = this step).

  • Base: nvidia/GR00T-N1.7-3B, embodiment new_embodiment, LeRobot (mfincato fork) train_ben.py.
  • Data (local, not published): 13 tasks / 912 episodes / 336,658 frames, 10 fps, cams wrist+left 480x640, state 40 (7 Panda joints deg, 12 XHand joints, 15 finger forces, ee pose xyz+rotvec), action 18 (ee pose + 12 XHand joints). = TAPTIC 11-task union + screw_light_bulb (54 eps) + graspglasstattileno (76) + graspglasshaptictattile (70). The graspglass datasets stored the absolute flange rotation; converted to the common gripper_down frame, w' = rotvec(Rx(pi)^T R) (verified against Panda FK). Episodes < 20 frames excluded.
  • Training: batch 8, 200k steps planned, cosine LR with 10k warmup, image transforms on, seed 42, bf16; tails fix (GROOT_KEEP_TAILS=1: last frames of each episode kept, action chunk padded).
  • Held-out (5% per task, shuffled seed 42; merged-dataset episode indices): [3, 14, 35, 81, 94, 113, 117, 128, 131, 195, 211, 269, 275, 286, 294, 301, 302, 327, 361, 377, 379, 414, 427, 453, 475, 522, 534, 540, 542, 566, 578, 586, 609, 612, 629, 650, 672, 706, 710, 713, 724, 732, 736, 792, 793, 795, 824, 855, 857, 860, 866]

Open-loop validation on held-out episodes (this step)

Every 10th frame, first 16 predicted actions vs recorded: ee position error, geodesic rotation error, mean XHand joint error.

overall pos 43.51 mm, rot 8.49 deg, hand 6.31 deg (1732 chunks)

task pos mm rot deg hand deg
Take the elliptical cylinder and put it 26.1 7.09 4.73
Take the credit card and insert it in th 43.2 8.99 3.84
Take the fruits and put them in the bask 39.9 8.25 7.69
Open the bottle, unscrew the cap. 29.5 8.88 8.32
Pick up one cup from the stack of cups. 35.9 7.61 3.96
Gently grasp the plastic glass and move 43.9 5.83 3.67
Clear the table by putting the objects i 77.0 12.46 12.04
Slide the car across the surface, to rea 43.4 4.73 5.83
Clean the table by wiping the surface. 62.3 9.42 8.28
Pick up the pipette and carefully transf 37.8 5.0 5.68
Pick the pipette and move it without lef 41.4 14.32 11.8
Screw the light bulb into the socket, th 43.2 11.16 8.39
Pick the glass and place it in the plate 45.1 6.09 3.96
Downloads last month
-
Safetensors
Model size
3B params
Tensor type
F32
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support