fm-dit / README.md
blyu413's picture
Add paused B5 B8 A20 step-6000 FM-DiT training continuation packages
c19c80d verified
|
Raw History Blame Contribute Delete
15.8 kB
metadata
tags:
  - robotics
  - humanoid
  - imitation-learning
  - flow-matching

FM-DiT — Humanoid Benchmark

Flow-matching Diffusion Transformer checkpoints trained on Humanoid Benchmark tasks. The A20 policy uses a ResNet18 image encoder and an AdaLN DiT action backbone (hidden dimension 384, 4 layers, 8 attention heads), with 10 midpoint sampling steps.

B7 black clothes to hamper (trimmed data), batch 512, steps 500-4000, with resume state (2026-11-17)

Checkpoints: b7-black-clothes-hamper/b512/step-000500/ to step-004000/, every 500 steps; step-004000 is the run's last checkpoint (the one evaluated below).

Trained with evaluation/baselines/train.py --policy fm_dit --contract g1_sonic_dfx6 (Humanoid Benchmark commit 028df1bb; code/hb_baselines_v1/ is byte-identical to its evaluation/baselines and evaluation/augmentation) on all 107 episodes (83,660 frames, about 24.5 epochs) of humanoid-benchmark/b7-black-clothes-hamper at revision 949da46e09a180010cacd439ddf7dd5dbde00de9 (the trimmed B7 black-garment data). Same architecture as A20 (ResNet18 + AdaLN DiT, hidden 384, 4 layers, 8 heads, 10 midpoint sampling steps), hb_baselines/1 format. Training: one RTX 4090D, batch size 512, 4,000 optimizer steps, seed 0, no held-out episodes, Humanoid Benchmark augmentation v2 (hb-aug-v2-20261109), AdamW (lr 1e-4, betas 0.95/0.999, weight decay 1e-6), the ImageNet-initialized ResNet18 fine-tuned at 0.1 x the learning rate with frozen BatchNorm, 500-step linear warm-up then cosine decay to 0 at step 4,000, bf16 autocast, EMA weights (0.998 at step 4,000). The preflight on 64 real samples found state 44-D without velocity, only the 29 body joints perturbed (at most 2.68 deg), action labels and the non-body state bit-identical. Final flow-matching loss 0.108; offline check (normalized action MSE on 256 training frames) 0.0712 at step 500, 0.0267 at 2,000 and 0.0240 at 4,000 (per checkpoint in train_log.jsonl and each config.json).

Inputs: Unitree G1 state44 and separate ego_view and chest RGB images, resized to 168 x 224. Output: 16 actions of dimension 76 (64 SONIC motion-token values and 6 measured values for each INSPIRE hand). Training instruction: "Walk to the clothes stand, take the black garment off the hanger with the right hand and drop it into the laundry hamper."

Closed-loop result (Humanoid Benchmark debug/newton/b7_policy_eval.py, protocol b7_black_v1 without the expert's grip pin; 50 unseen robot starts, seeds 700000-700049, 45 policy seconds): 0/50. In every run the robot walks about 0.9 m towards the clothes stand and stops short of it; no run touched the garment. Same seeds and protocol: ACT (batch 256, 8,000 steps, same data and augmentation) 2/50.

Files under b7-black-clothes-hamper/b512/: step-000500/ ... step-004000/ (model.pt EMA weights, config.json with the full configuration, normalization statistics and the offline metrics, protocol.json with the weight size and SHA-256), train_state.pt (resume state at step 4,000: raw policy weights, AdamW, LR scheduler, EMA, step, configuration; its EMA equals step-004000/model.pt), run.json (command line and configuration), train_log.jsonl, and SHA256SUMS (these files and the source snapshot; sha256sum -c b7-black-clothes-hamper/b512/SHA256SUMS from the repository root). Load and resume as the A18 example above with 'b7-black-clothes-hamper/b512/step-004000', robot='g1', --contract g1_sonic_dfx6 and the dataset humanoid-benchmark/b7-black-clothes-hamper at the revision above (the resume itself was not test-run from this layout).

A20 pre-reach trim v2, batch 512, step 4000 (2026-11-17)

Checkpoint: a20-basket-transfer/b512-trimv2/step-004000/.

Trained on 100 demonstrations (54,433 frames) from humanoid-benchmark/a20-basket-transfer-pre-reach-trim-expert-v2, with batch size 512, 4,000 optimizer steps, seed 0, EMA weights (decay 0.9999), and Humanoid Benchmark augmentation v2. These weights use the hb_baselines/1 format.

Inputs: Unitree G1 state44 and separate ego_view and chest RGB images, resized to 168 x 224. Output: 16 actions of dimension 76 (64 SONIC motion-token values and 6 measured values for each INSPIRE hand). Normalization is stored in the weights. The ResNet18 visual backbone was initialized with ImageNet weights and fine-tuned.

The checkpoint contains model.pt, config.json and protocol.json (weight size and SHA-256). SHA256SUMS covers the checkpoint and the accompanying inference source snapshot. The snapshot is under code/hb_baselines_v1/; its requirements record the training environment. Optimizer/resume state is excluded.

After downloading this repository and installing its inference dependencies, load the checkpoint from the repository root:

import sys
sys.path.insert(0, 'code/hb_baselines_v1')
from baselines.runtime import BaselinePolicy

policy = BaselinePolicy(
    'a20-basket-transfer/b512-trimv2/step-004000',
    device='cpu', policy='fm_dit', robot='g1',
)
print(policy.info())

The saved weights passed SHA-256 verification and strict CPU loading before upload. The offline metrics in config.json use training frames; they are not held-out or closed-loop task success rates.

A18 pour water and A17 unscrew bottle (H2 + Sharpa), batch 512, step 4000, with resume state (2026-11-17)

Task Checkpoint Training data (all 100 demonstrations) Frames Epochs
A18 pour water a18-pour-water/b512/step-004000/ humanoid-benchmark/a18-pour-water-expert-v1 @ 8e956a10 72,752 28.2
A17 unscrew bottle a17-unscrew-bottle/b512/step-004000/ humanoid-benchmark/a17-unscrew-bottle-expert-v1 @ 407e6e20 115,333 17.8

Same architecture as A20 (ResNet18 + AdaLN DiT, hidden 384, 4 layers, 8 heads, 10 midpoint sampling steps), hb_baselines/1 format. Training: batch size 512, 4,000 optimizer steps, seed 0, no held-out episodes, Humanoid Benchmark augmentation v2 (hb-aug-v2-20261109), AdamW (lr 1e-4, betas 0.95/0.999, weight decay 1e-6), the ImageNet-initialized ResNet18 fine-tuned at 0.1 x the learning rate with frozen BatchNorm, 500-step linear warm-up then cosine decay to 0 at step 4,000, bf16 autocast. The saved weights are the EMA (Diffusion Policy warm-up schedule, maximum decay 0.9999; 0.998 at step 4,000). Source: Humanoid Benchmark commit 028df1bb (evaluation/baselines, evaluation/augmentation).

Inputs: Unitree H2 with Sharpa hands, state78 (observation.state 75 + observation.projected_gravity 3) and separate head and chest RGB images, resized to 168 x 224. Output: 16 actions of dimension 75 (action.body_target 31 + action.hands 44). Normalization is stored in the weights.

Closed-loop result (simulation, 50 random layouts per task, 2026-11-17): 0/50 on both tasks. The arm reaches the pitcher handle / the bottle, but the hand does not close on it. The flow-matching loss was still falling when the learning rate reached 0, so these runs are undertrained; the resume state below is provided to continue them.

Files

Under <task>/b512/:

  • step-004000/: model.pt (EMA weights), config.json (full training configuration, normalization statistics, offline metrics on 256 training frames) and protocol.json (weight size and SHA-256).
  • train_state.pt: the resume state at step 4,000: raw (non-EMA) policy weights, AdamW state, LR-scheduler state, EMA weights, step counter and training configuration. Its EMA weights are identical to step-004000/model.pt.
  • run.json: the original command line, configuration and augmentation preflight. train_log.jsonl: loss, learning rate and EMA decay every 100 steps, and the offline metrics of the save.
  • SHA256SUMS: covers these files and the source snapshot; check it from the repository root, e.g. sha256sum -c a18-pour-water/b512/SHA256SUMS.

code/hb_baselines_v1/ now also holds the training sources (baselines/train.py, baselines/data.py, baselines/adapters.py and augmentation/) from the same commit as the inference files, which are unchanged.

Load for inference

import sys
sys.path.insert(0, 'code/hb_baselines_v1')
from baselines.runtime import BaselinePolicy

policy = BaselinePolicy(
    'a18-pour-water/b512/step-004000',
    device='cpu', policy='fm_dit', robot='h2_sharpa',
)
print(policy.info())

Resume training

hf download humanoid-benchmark/fm-dit --include 'a18-pour-water/b512/*' --include 'code/*' --local-dir fm-dit
hf download humanoid-benchmark/a18-pour-water-expert-v1 --repo-type dataset \
    --revision 8e956a102c84f991d2745a8cc66155c9991b742d --local-dir a18-pour-water-expert-v1
mkdir -p runs/a18_fm_dit && cp fm-dit/a18-pour-water/b512/train_state.pt runs/a18_fm_dit/
python fm-dit/code/hb_baselines_v1/baselines/train.py --policy fm_dit --contract h2_sharpa \
    --dataset a18-pour-water-expert-v1 --out runs/a18_fm_dit --resume \
    --steps 60000 --batch-size 512 --save-every 5000 --device cuda:0

For A17 use a17-unscrew-bottle/b512/ and humanoid-benchmark/a17-unscrew-bottle-expert-v1 at revision 407e6e20db2debc9b77fc2813e3445f549d51a33.

  • --resume reads <out>/train_state.pt and continues from step 4,000; check that the log prints {"event": "resumed", "step": 4000}, because without that file the run starts from step 0. It refuses another policy configuration (keep --policy fm_dit, --contract h2_sharpa and the defaults for horizon, cameras, image size, backbone and --model) and another augmentation (keep the default --augmentation hb-v2 with --shift 0). Batch size, workers, save interval and device may change.
  • --steps is the new total. The schedule is recomputed for it, without a second warm-up: the learning rate restarts at 0.97 x 1e-4 for --steps 30000 or 0.99 x 1e-4 for --steps 60000 and decays by cosine to 0 at the new total. The EMA continues on the global step.
  • The run writes checkpoints/step-NNNNNN/ and checkpoints/final into --out, replaces train_state.pt at every save, appends to train_log.jsonl and rewrites run.json with the new command (the original stays here). Outside a git checkout code_commit is recorded as null; the source is commit 028df1bb.
  • The first run decodes the frames of both cameras once into --cache-dir (default ~/hb_models/baselines/cache; about 15 GB for A18 and 26 GB for A17). The augmentation runs on the CPU (about 17 ms of one core per item), so --workers sets the speed; the original runs used 14 workers each.
  • Environment: code/hb_baselines_v1/baselines/requirements.txt (Python 3.12, torch 2.9.0 with CUDA 12.8).

Verified before upload: both checkpoints load strictly on CPU from this layout, and a resume from this layout continued each run from step 4,000 to 4,002 on CPU and saved a new checkpoint.

B6 pedal bin, batch 512, step 4000, with resume state (2026-11-17)

Checkpoint: b6-pedal-bin/b512/step-004000/ (the run's last checkpoint).

Trained with evaluation/baselines/train.py --policy fm_dit --contract g1_sonic_dfx6 (Humanoid Benchmark commit 4385573e; code/hb_baselines_v1/ is byte-identical to its evaluation/baselines and evaluation/augmentation) on the first 100 episodes (65,127 frames, about 31.4 epochs) of the B6 v5t7 ready-start expert data, the data of the GR00T b6-pedal-bin/b256 and Psi0 b6-pedal-bin/b512 checkpoints (humanoid-benchmark/b6-pedal-bin v5/, episodes 0-99). Same architecture as A20 (ResNet18 + AdaLN DiT, hidden 384, 4 layers, 8 heads, 10 midpoint sampling steps), hb_baselines/1 format. Training: batch size 512 (fits the 32 GB RTX 5090), 4,000 optimizer steps, seed 0, no held-out episodes, Humanoid Benchmark augmentation v2 (hb-aug-v2-20261109; joint-angle bias +-2 deg plus 0.25 deg noise on the 29 body joints only), AdamW (lr 1e-4, betas 0.95/0.999, weight decay 1e-6), the ImageNet-initialized ResNet18 fine-tuned at 0.1 x the learning rate with frozen BatchNorm, 200-step linear warm-up then cosine decay to 0 at step 4,000, bf16 autocast, EMA weights (0.998 at step 4,000). Before training a real batch of 512 was checked: state 44-D without velocity, only the body joints perturbed, action labels bit-identical to the parquet files (batch_check_b6_v5.json). Final flow-matching loss 0.109; offline check (normalized action MSE on 256 training frames) 0.026.

Inputs: Unitree G1 state44 and separate ego_view and chest RGB images, resized to 168 x 224. Output: 16 actions of dimension 76 (64 SONIC motion-token values and 6 measured values for each INSPIRE hand). Training instruction: "Pick up the chip barrel from the cart, walk to the pedal bin, step on its pedal and drop it in."

Closed-loop result (Humanoid Benchmark debug/newton/b6_policy_eval.py --protocol v5_ready_start with the v5 task configuration; 50 unseen layouts, seeds 180000-180049, 45 policy seconds): 0/50; the robot never touched the barrel. Same seeds and protocol: GR00T b6-pedal-bin/b256/step-004000 39/50, Psi0 b6-pedal-bin/b512/step-004000 30/50, ACT (humanoid-benchmark/act b6-pedal-bin/b512/step-004000, this recipe) 8/50, DP 0/50. With the dataset's own frames the checkpoint predicts the recorded actions (held-out episodes as well as trained ones); in closed loop its first head-camera frames differ from the dataset's starts, and it does not set off towards the cart.

Files under b6-pedal-bin/b512/: step-004000/ (model.pt EMA weights, config.json with the full configuration, normalization statistics and the offline metrics, protocol.json with the weight size and SHA-256), train_state.pt (resume state at step 4,000: raw policy weights, AdamW, LR scheduler, EMA, step, configuration; its EMA equals step-004000/model.pt), run.json (command line, configuration, augmentation preflight), train_log.jsonl, train.log, batch_check_b6_v5.json, and SHA256SUMS (these files and the source snapshot; sha256sum -c b6-pedal-bin/b512/SHA256SUMS from the repository root). Load and resume as the A18 example above with 'b6-pedal-bin/b512/step-004000', robot='g1', --contract g1_sonic_dfx6 and the dataset humanoid-benchmark/b6-pedal-bin v5/ with --episodes 0:100 (the resume itself was not test-run from this layout).

B5, B8 and A20 paused continuation packages (2026-11-17)

The following independent runs resumed the local step-4,000 baselines and were paused by the owner. Each package contains a step-6,000 inference checkpoint and full optimizer/EMA resume state, configuration, logs, dataset revision, hashes and portable resume instructions. The intended total was 20,000; later logged updates were not checkpointed. These are separate from the existing A20 b512-trimv2-cont step-10,000/20,000/30,000 lineage.

Shared frozen training code and the opt-in resume launcher: code/fmdit_continue_20261117/. The three source datasets are referenced at pinned revisions and are not redistributed here. Offline errors are training-frame checks, not held-out or closed-loop performance.