---
license: apache-2.0
library_name: lerobot
tags: [robotics, lerobot, mtdit-spatial, multi-task-dit, relative-actions]
---

# MTDiT-Spatial — relative actions, end-effector space (13-dim pose, 6D rotation)

`mtdit_spatial` with `vision_feature_mode=cls`, which is byte-identical to stock
`multi_task_dit`. Trained **from scratch** on the 949-episode `base4` multi-task mixture
(900 train / 49 held out, 8 tasks, 3 cameras, 50 Hz).

Checkpoint **step 070000** of a 100K-step run — the held-out-loss minimum, not the final step
(earliest of a four-way tie at 0.0074 (70K/75K/85K/90K); fewest steps at the floor).

## What this is for

One cell of a 2x2 matrix crossing action representation (relative joints vs relative
end-effector) with policy (MTDiT vs pi0.5), testing whether a relative action target makes the
policy attend to its cameras. **It does not.** These weights are published as a negative result.

| | value |
|---|---|
| held-out loss | 0.0074 |
| camera-sensitivity ratio | 0.119 |
| prediction error | 0.104 |
| null test | 0.00000 (pass) |

Measured on `multitask-eefabs6d-v30` episodes 93-97 -- the same croissant episodes, in end-effector space -- scored on position dims 0:3 (metres).

The ratio is **not comparable to the joint-space sibling**: this one divides a change in predicted end-effector position (metres) by the natural spread of that quantity, the other divides degrees by degrees. Only within-representation comparisons are meaningful, which is also why no GROUNDED/WEAK/BLIND word is given here -- those boundaries were calibrated on joint-space targets.

Camera sensitivity holds `observation.state` fixed, swaps in another episode's cameras from a
state-matched frame, and divides the resulting change in predicted arm joints by the natural
cross-episode spread of the ground truth. **1.0 means the policy tracks the scene; 0.0 means it
ignores it.** Measured on croissant held-out episodes 93-97 restricted to the "pick up the
croissant" phase, four inference seeds under common random numbers, padded steps excluded, with
the null test (each variant fed the anchor's own cameras) required to return 0.

Read the ratio together with the prediction error: a high ratio with a high error is an unstable
policy moving because any input moved, not a grounded one. The error here is well inside the 0.8
reliability gate.

## Action representation

Targets are **relative**: `action[t+k] - state[anchor]`, one anchor per chunk, added back after
inference. The gripper stays absolute (`relative_exclude_joints=['gripper']`) because it is a
near-binary command with a 70-unit single-step jump, not a pose.

Normalisation statistics must match the representation, so this trains against `l5vel-peng/multitask-eefrel-h32`, a
metadata-only sibling of the mixture whose `meta/stats.json` holds offset quantiles computed at
the policy horizon. Training against absolute statistics would squash every target.

## Configuration

```
horizon 32   n_action_steps 24   n_obs_steps 2
hidden_dim 512   num_layers 4   num_heads 8   dropout 0.1
diffusion / DDPM, num_train_timesteps 100
optimizer_lr 3e-4   weight_decay 0   cosine to 100K, warmup 0
CLIP openai/clip-vit-base-patch16, vision_encoder_lr_multiplier 0.1
images resize [240,320] -> random crop [224,224]
norm  VISUAL MEAN_STD | STATE MIN_MAX | ACTION QUANTILES
batch 64, bf16, seed 1000
```

## Loading

```python
from lerobot.policies.factory import make_policy   # needs lerobot_policy_mtdit_spatial installed
```

Requires the [`lerobot_policy_mtdit_spatial`](https://github.com/) plugin, which registers the
`mtdit_spatial` policy type. Loading with `--policy.path` inherits this checkpoint's config;
`--policy.pretrained_path` loads weights only and resets stored settings to defaults.

Apache-2.0, inherited from LeRobot.