| --- |
| license: apache-2.0 |
| library_name: lerobot |
| tags: [robotics, lerobot, mtdit-spatial, multi-task-dit, relative-actions] |
| --- |
| |
| # MTDiT-Spatial — relative actions, end-effector space (13-dim pose, 6D rotation) |
|
|
| `mtdit_spatial` with `vision_feature_mode=cls`, which is byte-identical to stock |
| `multi_task_dit`. Trained **from scratch** on the 949-episode `base4` multi-task mixture |
| (900 train / 49 held out, 8 tasks, 3 cameras, 50 Hz). |
|
|
| Checkpoint **step 070000** of a 100K-step run — the held-out-loss minimum, not the final step |
| (earliest of a four-way tie at 0.0074 (70K/75K/85K/90K); fewest steps at the floor). |
|
|
| ## What this is for |
|
|
| One cell of a 2x2 matrix crossing action representation (relative joints vs relative |
| end-effector) with policy (MTDiT vs pi0.5), testing whether a relative action target makes the |
| policy attend to its cameras. **It does not.** These weights are published as a negative result. |
|
|
| | | value | |
| |---|---| |
| | held-out loss | 0.0074 | |
| | camera-sensitivity ratio | 0.119 | |
| | prediction error | 0.104 | |
| | null test | 0.00000 (pass) | |
|
|
| Measured on `multitask-eefabs6d-v30` episodes 93-97 -- the same croissant episodes, in end-effector space -- scored on position dims 0:3 (metres). |
|
|
| The ratio is **not comparable to the joint-space sibling**: this one divides a change in predicted end-effector position (metres) by the natural spread of that quantity, the other divides degrees by degrees. Only within-representation comparisons are meaningful, which is also why no GROUNDED/WEAK/BLIND word is given here -- those boundaries were calibrated on joint-space targets. |
|
|
| Camera sensitivity holds `observation.state` fixed, swaps in another episode's cameras from a |
| state-matched frame, and divides the resulting change in predicted arm joints by the natural |
| cross-episode spread of the ground truth. **1.0 means the policy tracks the scene; 0.0 means it |
| ignores it.** Measured on croissant held-out episodes 93-97 restricted to the "pick up the |
| croissant" phase, four inference seeds under common random numbers, padded steps excluded, with |
| the null test (each variant fed the anchor's own cameras) required to return 0. |
|
|
| Read the ratio together with the prediction error: a high ratio with a high error is an unstable |
| policy moving because any input moved, not a grounded one. The error here is well inside the 0.8 |
| reliability gate. |
|
|
| ## Action representation |
|
|
| Targets are **relative**: `action[t+k] - state[anchor]`, one anchor per chunk, added back after |
| inference. The gripper stays absolute (`relative_exclude_joints=['gripper']`) because it is a |
| near-binary command with a 70-unit single-step jump, not a pose. |
|
|
| Normalisation statistics must match the representation, so this trains against `l5vel-peng/multitask-eefrel-h32`, a |
| metadata-only sibling of the mixture whose `meta/stats.json` holds offset quantiles computed at |
| the policy horizon. Training against absolute statistics would squash every target. |
|
|
| ## Configuration |
|
|
| ``` |
| horizon 32 n_action_steps 24 n_obs_steps 2 |
| hidden_dim 512 num_layers 4 num_heads 8 dropout 0.1 |
| diffusion / DDPM, num_train_timesteps 100 |
| optimizer_lr 3e-4 weight_decay 0 cosine to 100K, warmup 0 |
| CLIP openai/clip-vit-base-patch16, vision_encoder_lr_multiplier 0.1 |
| images resize [240,320] -> random crop [224,224] |
| norm VISUAL MEAN_STD | STATE MIN_MAX | ACTION QUANTILES |
| batch 64, bf16, seed 1000 |
| ``` |
|
|
| ## Loading |
|
|
| ```python |
| from lerobot.policies.factory import make_policy # needs lerobot_policy_mtdit_spatial installed |
| ``` |
|
|
| Requires the [`lerobot_policy_mtdit_spatial`](https://github.com/) plugin, which registers the |
| `mtdit_spatial` policy type. Loading with `--policy.path` inherits this checkpoint's config; |
| `--policy.pretrained_path` loads weights only and resets stored settings to defaults. |
|
|
| Apache-2.0, inherited from LeRobot. |
|
|
