nvidia

/

Cosmos3-Nano

like 302

Follow

NVIDIA 63.2k

Hi Cosmos / AgiBotWorld team,

I am using nvidia/Cosmos3-Nano through vLLM-Omni with:

• domain_name="agibotworld"
• view_point="concat_view"
• action_chunk_size=16
• raw_action_dim=29
• 17-frame, 10 FPS concat-view windows

The inverse-dynamics endpoint returns valid action chunks with shape (16, 29). I can also feed each
returned chunk inline through:

extra_params["action"] = action_chunk
extra_params["action_mode"] = "forward_dynamics"

and Cosmos successfully generates a forward-dynamics video rollout.

I am trying to map these actions to an AgiBot G1 Isaac Sim model for IK -> FK validation.

From the published AgiBotWorld action example and returned chunks, the 29-D structure appears to be
grouped approximately as:

0:7 pose-like block: [x, y, z, qw, qx, qy, qz]
7:9 two scalar body/head-like values
9:16 left end-effector pose-like block
16:19 left hand/gripper-like values
19:26 right end-effector pose-like block
26:29 right hand/gripper-like values

Could you clarify the official specification?

1. What is the exact meaning and field order of all 29 dimensions?
2. Are the pose blocks expressed in the robot base frame, waist frame, torso frame, world frame, or
   another frame?
3. Is there a reference G1 URDF/USD, joint ordering, and an example converting the 29-D action into G1
   joint targets or EE IK targets?

For context, I tested a simple rigid transform from the inferred Cosmos EE pose blocks to FK poses from
our G1 simulator. The fit was poor, which suggests either the 29-D values need a domain normalizer, the
assumed fields/frame are wrong, or inverse-dynamics predictions should not be used as paired coordinate-
calibration labels.

Any official schema, normalizer, or reference conversion code would be very helpful. Thanks.