fm-dit / README.md
blyu413's picture
Add paused B5 B8 A20 step-6000 FM-DiT training continuation packages
c19c80d verified
|
Raw History Blame Contribute Delete
15.8 kB
---
tags:
- robotics
- humanoid
- imitation-learning
- flow-matching
---
# FM-DiT — Humanoid Benchmark
Flow-matching Diffusion Transformer checkpoints trained on Humanoid Benchmark tasks.
The A20 policy uses a ResNet18 image encoder and an AdaLN DiT action backbone
(hidden dimension 384, 4 layers, 8 attention heads), with 10 midpoint sampling steps.
## B7 black clothes to hamper (trimmed data), batch 512, steps 500-4000, with resume state (2026-11-17)
Checkpoints: [`b7-black-clothes-hamper/b512/step-000500/`](b7-black-clothes-hamper/b512/step-000500) to
[`step-004000/`](b7-black-clothes-hamper/b512/step-004000), every 500 steps; `step-004000` is the run's last checkpoint
(the one evaluated below).
Trained with `evaluation/baselines/train.py --policy fm_dit --contract g1_sonic_dfx6` (Humanoid Benchmark commit
`028df1bb`; `code/hb_baselines_v1/` is byte-identical to its `evaluation/baselines` and `evaluation/augmentation`) on all
107 episodes (83,660 frames, about 24.5 epochs) of `humanoid-benchmark/b7-black-clothes-hamper` at revision
`949da46e09a180010cacd439ddf7dd5dbde00de9` (the trimmed B7 black-garment data). Same architecture as A20 (ResNet18 +
AdaLN DiT, hidden 384, 4 layers, 8 heads, 10 midpoint sampling steps), `hb_baselines/1` format. Training: one RTX 4090D,
batch size 512, 4,000 optimizer steps, seed 0, no held-out episodes, Humanoid Benchmark augmentation v2
(`hb-aug-v2-20261109`), AdamW (lr 1e-4, betas 0.95/0.999, weight decay 1e-6), the ImageNet-initialized ResNet18
fine-tuned at 0.1 x the learning rate with frozen BatchNorm, 500-step linear warm-up then cosine decay to 0 at step
4,000, bf16 autocast, EMA weights (0.998 at step 4,000). The preflight on 64 real samples found state 44-D without
velocity, only the 29 body joints perturbed (at most 2.68 deg), action labels and the non-body state bit-identical.
Final flow-matching loss 0.108; offline check (normalized action MSE on 256 training frames) 0.0712 at step 500,
0.0267 at 2,000 and 0.0240 at 4,000 (per checkpoint in `train_log.jsonl` and each `config.json`).
Inputs: Unitree G1 state44 and separate `ego_view` and `chest` RGB images, resized to 168 x 224. Output: 16 actions of
dimension 76 (64 SONIC motion-token values and 6 measured values for each INSPIRE hand). Training instruction: "Walk to
the clothes stand, take the black garment off the hanger with the right hand and drop it into the laundry hamper."
Closed-loop result (Humanoid Benchmark `debug/newton/b7_policy_eval.py`, protocol `b7_black_v1` without the expert's grip
pin; 50 unseen robot starts, seeds 700000-700049, 45 policy seconds): **0/50**. In every run the robot walks about
0.9 m towards the clothes stand and stops short of it; no run touched the garment. Same seeds and protocol: ACT
(batch 256, 8,000 steps, same data and augmentation) 2/50.
Files under `b7-black-clothes-hamper/b512/`: `step-000500/` ... `step-004000/` (`model.pt` EMA weights, `config.json`
with the full configuration, normalization statistics and the offline metrics, `protocol.json` with the weight size and
SHA-256), `train_state.pt` (resume state at step 4,000: raw policy weights, AdamW, LR scheduler, EMA, step,
configuration; its EMA equals `step-004000/model.pt`), `run.json` (command line and configuration), `train_log.jsonl`,
and `SHA256SUMS` (these files and the source snapshot; `sha256sum -c b7-black-clothes-hamper/b512/SHA256SUMS` from the
repository root). Load and resume as the A18 example above with `'b7-black-clothes-hamper/b512/step-004000'`,
`robot='g1'`, `--contract g1_sonic_dfx6` and the dataset `humanoid-benchmark/b7-black-clothes-hamper` at the revision
above (the resume itself was not test-run from this layout).
## A20 pre-reach trim v2, batch 512, step 4000 (2026-11-17)
Checkpoint: [`a20-basket-transfer/b512-trimv2/step-004000/`](a20-basket-transfer/b512-trimv2/step-004000).
Trained on 100 demonstrations (54,433 frames) from
`humanoid-benchmark/a20-basket-transfer-pre-reach-trim-expert-v2`, with batch size 512,
4,000 optimizer steps, seed 0, EMA weights (decay 0.9999), and Humanoid Benchmark
augmentation v2. These weights use the `hb_baselines/1` format.
Inputs: Unitree G1 state44 and separate `ego_view` and `chest` RGB images, resized
to 168 x 224. Output: 16 actions of dimension 76 (64 SONIC motion-token values and
6 measured values for each INSPIRE hand). Normalization is stored in the weights.
The ResNet18 visual backbone was initialized with ImageNet weights and fine-tuned.
The checkpoint contains `model.pt`, `config.json` and `protocol.json` (weight size
and SHA-256). `SHA256SUMS` covers the checkpoint and the accompanying inference
source snapshot. The snapshot is under `code/hb_baselines_v1/`; its requirements
record the training environment. Optimizer/resume state is excluded.
After downloading this repository and installing its inference dependencies,
load the checkpoint from the repository root:
```python
import sys
sys.path.insert(0, 'code/hb_baselines_v1')
from baselines.runtime import BaselinePolicy
policy = BaselinePolicy(
'a20-basket-transfer/b512-trimv2/step-004000',
device='cpu', policy='fm_dit', robot='g1',
)
print(policy.info())
```
The saved weights passed SHA-256 verification and strict CPU loading before upload.
The offline metrics in `config.json` use training frames; they are not held-out
or closed-loop task success rates.
## A18 pour water and A17 unscrew bottle (H2 + Sharpa), batch 512, step 4000, with resume state (2026-11-17)
| Task | Checkpoint | Training data (all 100 demonstrations) | Frames | Epochs |
| --- | --- | --- | --- | --- |
| A18 pour water | [`a18-pour-water/b512/step-004000/`](a18-pour-water/b512/step-004000) | `humanoid-benchmark/a18-pour-water-expert-v1` @ `8e956a10` | 72,752 | 28.2 |
| A17 unscrew bottle | [`a17-unscrew-bottle/b512/step-004000/`](a17-unscrew-bottle/b512/step-004000) | `humanoid-benchmark/a17-unscrew-bottle-expert-v1` @ `407e6e20` | 115,333 | 17.8 |
Same architecture as A20 (ResNet18 + AdaLN DiT, hidden 384, 4 layers, 8 heads, 10 midpoint sampling steps),
`hb_baselines/1` format. Training: batch size 512, 4,000 optimizer steps, seed 0, no held-out episodes, Humanoid
Benchmark augmentation v2 (`hb-aug-v2-20261109`), AdamW (lr 1e-4, betas 0.95/0.999, weight decay 1e-6), the
ImageNet-initialized ResNet18 fine-tuned at 0.1 x the learning rate with frozen BatchNorm, 500-step linear warm-up
then cosine decay to 0 at step 4,000, bf16 autocast. The saved weights are the EMA (Diffusion Policy warm-up
schedule, maximum decay 0.9999; 0.998 at step 4,000). Source: Humanoid Benchmark commit `028df1bb`
(`evaluation/baselines`, `evaluation/augmentation`).
Inputs: Unitree H2 with Sharpa hands, state78 (`observation.state` 75 + `observation.projected_gravity` 3) and
separate `head` and `chest` RGB images, resized to 168 x 224. Output: 16 actions of dimension 75
(`action.body_target` 31 + `action.hands` 44). Normalization is stored in the weights.
Closed-loop result (simulation, 50 random layouts per task, 2026-11-17): **0/50 on both tasks**. The arm reaches the
pitcher handle / the bottle, but the hand does not close on it. The flow-matching loss was still falling when the
learning rate reached 0, so these runs are undertrained; the resume state below is provided to continue them.
### Files
Under `<task>/b512/`:
- `step-004000/`: `model.pt` (EMA weights), `config.json` (full training configuration, normalization statistics,
offline metrics on 256 training frames) and `protocol.json` (weight size and SHA-256).
- `train_state.pt`: the resume state at step 4,000: raw (non-EMA) policy weights, AdamW state, LR-scheduler state,
EMA weights, step counter and training configuration. Its EMA weights are identical to `step-004000/model.pt`.
- `run.json`: the original command line, configuration and augmentation preflight. `train_log.jsonl`: loss,
learning rate and EMA decay every 100 steps, and the offline metrics of the save.
- `SHA256SUMS`: covers these files and the source snapshot; check it from the repository root, e.g.
`sha256sum -c a18-pour-water/b512/SHA256SUMS`.
`code/hb_baselines_v1/` now also holds the training sources (`baselines/train.py`, `baselines/data.py`,
`baselines/adapters.py` and `augmentation/`) from the same commit as the inference files, which are unchanged.
### Load for inference
```python
import sys
sys.path.insert(0, 'code/hb_baselines_v1')
from baselines.runtime import BaselinePolicy
policy = BaselinePolicy(
'a18-pour-water/b512/step-004000',
device='cpu', policy='fm_dit', robot='h2_sharpa',
)
print(policy.info())
```
### Resume training
```bash
hf download humanoid-benchmark/fm-dit --include 'a18-pour-water/b512/*' --include 'code/*' --local-dir fm-dit
hf download humanoid-benchmark/a18-pour-water-expert-v1 --repo-type dataset \
--revision 8e956a102c84f991d2745a8cc66155c9991b742d --local-dir a18-pour-water-expert-v1
mkdir -p runs/a18_fm_dit && cp fm-dit/a18-pour-water/b512/train_state.pt runs/a18_fm_dit/
python fm-dit/code/hb_baselines_v1/baselines/train.py --policy fm_dit --contract h2_sharpa \
--dataset a18-pour-water-expert-v1 --out runs/a18_fm_dit --resume \
--steps 60000 --batch-size 512 --save-every 5000 --device cuda:0
```
For A17 use `a17-unscrew-bottle/b512/` and `humanoid-benchmark/a17-unscrew-bottle-expert-v1` at revision
`407e6e20db2debc9b77fc2813e3445f549d51a33`.
- `--resume` reads `<out>/train_state.pt` and continues from step 4,000; check that the log prints
`{"event": "resumed", "step": 4000}`, because without that file the run starts from step 0. It refuses another policy configuration
(keep `--policy fm_dit`, `--contract h2_sharpa` and the defaults for horizon, cameras, image size, backbone and
`--model`) and another augmentation (keep the default `--augmentation hb-v2` with `--shift 0`). Batch size,
workers, save interval and device may change.
- `--steps` is the new total. The schedule is recomputed for it, without a second warm-up: the learning rate
restarts at 0.97 x 1e-4 for `--steps 30000` or 0.99 x 1e-4 for `--steps 60000` and decays by cosine to 0 at the
new total. The EMA continues on the global step.
- The run writes `checkpoints/step-NNNNNN/` and `checkpoints/final` into `--out`, replaces `train_state.pt` at every
save, appends to `train_log.jsonl` and rewrites `run.json` with the new command (the original stays here).
Outside a git checkout `code_commit` is recorded as null; the source is commit `028df1bb`.
- The first run decodes the frames of both cameras once into `--cache-dir` (default `~/hb_models/baselines/cache`;
about 15 GB for A18 and 26 GB for A17). The augmentation runs on the CPU (about 17 ms of one core per item), so
`--workers` sets the speed; the original runs used 14 workers each.
- Environment: `code/hb_baselines_v1/baselines/requirements.txt` (Python 3.12, torch 2.9.0 with CUDA 12.8).
Verified before upload: both checkpoints load strictly on CPU from this layout, and a resume from this layout
continued each run from step 4,000 to 4,002 on CPU and saved a new checkpoint.
## B6 pedal bin, batch 512, step 4000, with resume state (2026-11-17)
Checkpoint: [`b6-pedal-bin/b512/step-004000/`](b6-pedal-bin/b512/step-004000) (the run's last checkpoint).
Trained with `evaluation/baselines/train.py --policy fm_dit --contract g1_sonic_dfx6` (Humanoid Benchmark commit
`4385573e`; `code/hb_baselines_v1/` is byte-identical to its `evaluation/baselines` and `evaluation/augmentation`) on
the first 100 episodes (65,127 frames, about 31.4 epochs) of the B6 v5t7 ready-start expert data, the data of the
GR00T `b6-pedal-bin/b256` and Psi0 `b6-pedal-bin/b512` checkpoints (`humanoid-benchmark/b6-pedal-bin` `v5/`, episodes
0-99). Same architecture as A20 (ResNet18 + AdaLN DiT, hidden 384, 4 layers, 8 heads, 10 midpoint sampling steps),
`hb_baselines/1` format. Training: batch size 512 (fits the 32 GB RTX 5090), 4,000 optimizer steps, seed 0, no held-out
episodes, Humanoid Benchmark augmentation v2 (`hb-aug-v2-20261109`; joint-angle bias +-2 deg plus 0.25 deg noise on
the 29 body joints only), AdamW (lr 1e-4, betas 0.95/0.999, weight decay 1e-6), the ImageNet-initialized ResNet18
fine-tuned at 0.1 x the learning rate with frozen BatchNorm, **200-step** linear warm-up then cosine decay to 0 at step
4,000, bf16 autocast, EMA weights (0.998 at step 4,000). Before training a real batch of 512 was checked: state 44-D
without velocity, only the body joints perturbed, action labels bit-identical to the parquet files
(`batch_check_b6_v5.json`). Final flow-matching loss 0.109; offline check (normalized action MSE on 256 training
frames) 0.026.
Inputs: Unitree G1 state44 and separate `ego_view` and `chest` RGB images, resized to 168 x 224. Output: 16 actions of
dimension 76 (64 SONIC motion-token values and 6 measured values for each INSPIRE hand). Training instruction: "Pick up
the chip barrel from the cart, walk to the pedal bin, step on its pedal and drop it in."
Closed-loop result (Humanoid Benchmark `debug/newton/b6_policy_eval.py --protocol v5_ready_start` with the v5 task
configuration; 50 unseen layouts, seeds 180000-180049, 45 policy seconds): **0/50**; the robot never touched the
barrel. Same seeds and protocol: GR00T `b6-pedal-bin/b256/step-004000` 39/50, Psi0 `b6-pedal-bin/b512/step-004000`
30/50, ACT (`humanoid-benchmark/act` `b6-pedal-bin/b512/step-004000`, this recipe) 8/50, DP 0/50. With the dataset's
own frames the checkpoint predicts the recorded actions (held-out episodes as well as trained ones); in closed loop its
first head-camera frames differ from the dataset's starts, and it does not set off towards the cart.
Files under `b6-pedal-bin/b512/`: `step-004000/` (`model.pt` EMA weights, `config.json` with the full configuration,
normalization statistics and the offline metrics, `protocol.json` with the weight size and SHA-256), `train_state.pt`
(resume state at step 4,000: raw policy weights, AdamW, LR scheduler, EMA, step, configuration; its EMA equals
`step-004000/model.pt`), `run.json` (command line, configuration, augmentation preflight), `train_log.jsonl`,
`train.log`, `batch_check_b6_v5.json`, and `SHA256SUMS` (these files and the source snapshot; `sha256sum -c
b6-pedal-bin/b512/SHA256SUMS` from the repository root). Load and resume as the A18 example above with
`'b6-pedal-bin/b512/step-004000'`, `robot='g1'`, `--contract g1_sonic_dfx6` and the dataset
`humanoid-benchmark/b6-pedal-bin` `v5/` with `--episodes 0:100` (the resume itself was not test-run from this layout).
## B5, B8 and A20 paused continuation packages (2026-11-17)
The following independent runs resumed the local step-4,000 baselines and were paused by the owner.
Each package contains a **step-6,000** inference checkpoint and full optimizer/EMA resume state,
configuration, logs, dataset revision, hashes and portable resume instructions. The intended total
was 20,000; later logged updates were not checkpointed. These are separate from the existing
A20 `b512-trimv2-cont` step-10,000/20,000/30,000 lineage.
| Task | Continuation package | Saved step |
| --- | --- | --- |
| B5 | [b5-chair-push-standing-start/b512-cont-20261117/](b5-chair-push-standing-start/b512-cont-20261117/) | 6,000 |
| B8 | [b8-table-wipe/b512-cont-20261117/](b8-table-wipe/b512-cont-20261117/) | 6,000 |
| A20 | [a20-basket-transfer/b512-trimv2-cont-20261117/](a20-basket-transfer/b512-trimv2-cont-20261117/) | 6,000 |
Shared frozen training code and the opt-in resume launcher: [`code/fmdit_continue_20261117/`](code/fmdit_continue_20261117/).
The three source datasets are referenced at pinned revisions and are not redistributed here.
Offline errors are training-frame checks, not held-out or closed-loop performance.