Download README.md from humanoid-benchmark/fm-dit: direct link, hf CLI and curl.
- Browser
- Download file 15.8 kB
-
https://huggingface.co/humanoid-benchmark/fm-dit/resolve/main/README.md
- Command line
-
hf download hf://humanoid-benchmark/fm-dit/README.md
-
curl -L -o README.md https://huggingface.co/humanoid-benchmark/fm-dit/resolve/main/README.md
tags:
- robotics
- humanoid
- imitation-learning
- flow-matching
FM-DiT — Humanoid Benchmark
Flow-matching Diffusion Transformer checkpoints trained on Humanoid Benchmark tasks. The A20 policy uses a ResNet18 image encoder and an AdaLN DiT action backbone (hidden dimension 384, 4 layers, 8 attention heads), with 10 midpoint sampling steps.
B7 black clothes to hamper (trimmed data), batch 512, steps 500-4000, with resume state (2026-11-17)
Checkpoints: b7-black-clothes-hamper/b512/step-000500/ to
step-004000/, every 500 steps; step-004000 is the run's last checkpoint
(the one evaluated below).
Trained with evaluation/baselines/train.py --policy fm_dit --contract g1_sonic_dfx6 (Humanoid Benchmark commit
028df1bb; code/hb_baselines_v1/ is byte-identical to its evaluation/baselines and evaluation/augmentation) on all
107 episodes (83,660 frames, about 24.5 epochs) of humanoid-benchmark/b7-black-clothes-hamper at revision
949da46e09a180010cacd439ddf7dd5dbde00de9 (the trimmed B7 black-garment data). Same architecture as A20 (ResNet18 +
AdaLN DiT, hidden 384, 4 layers, 8 heads, 10 midpoint sampling steps), hb_baselines/1 format. Training: one RTX 4090D,
batch size 512, 4,000 optimizer steps, seed 0, no held-out episodes, Humanoid Benchmark augmentation v2
(hb-aug-v2-20261109), AdamW (lr 1e-4, betas 0.95/0.999, weight decay 1e-6), the ImageNet-initialized ResNet18
fine-tuned at 0.1 x the learning rate with frozen BatchNorm, 500-step linear warm-up then cosine decay to 0 at step
4,000, bf16 autocast, EMA weights (0.998 at step 4,000). The preflight on 64 real samples found state 44-D without
velocity, only the 29 body joints perturbed (at most 2.68 deg), action labels and the non-body state bit-identical.
Final flow-matching loss 0.108; offline check (normalized action MSE on 256 training frames) 0.0712 at step 500,
0.0267 at 2,000 and 0.0240 at 4,000 (per checkpoint in train_log.jsonl and each config.json).
Inputs: Unitree G1 state44 and separate ego_view and chest RGB images, resized to 168 x 224. Output: 16 actions of
dimension 76 (64 SONIC motion-token values and 6 measured values for each INSPIRE hand). Training instruction: "Walk to
the clothes stand, take the black garment off the hanger with the right hand and drop it into the laundry hamper."
Closed-loop result (Humanoid Benchmark debug/newton/b7_policy_eval.py, protocol b7_black_v1 without the expert's grip
pin; 50 unseen robot starts, seeds 700000-700049, 45 policy seconds): 0/50. In every run the robot walks about
0.9 m towards the clothes stand and stops short of it; no run touched the garment. Same seeds and protocol: ACT
(batch 256, 8,000 steps, same data and augmentation) 2/50.
Files under b7-black-clothes-hamper/b512/: step-000500/ ... step-004000/ (model.pt EMA weights, config.json
with the full configuration, normalization statistics and the offline metrics, protocol.json with the weight size and
SHA-256), train_state.pt (resume state at step 4,000: raw policy weights, AdamW, LR scheduler, EMA, step,
configuration; its EMA equals step-004000/model.pt), run.json (command line and configuration), train_log.jsonl,
and SHA256SUMS (these files and the source snapshot; sha256sum -c b7-black-clothes-hamper/b512/SHA256SUMS from the
repository root). Load and resume as the A18 example above with 'b7-black-clothes-hamper/b512/step-004000',
robot='g1', --contract g1_sonic_dfx6 and the dataset humanoid-benchmark/b7-black-clothes-hamper at the revision
above (the resume itself was not test-run from this layout).
A20 pre-reach trim v2, batch 512, step 4000 (2026-11-17)
Checkpoint: a20-basket-transfer/b512-trimv2/step-004000/.
Trained on 100 demonstrations (54,433 frames) from
humanoid-benchmark/a20-basket-transfer-pre-reach-trim-expert-v2, with batch size 512,
4,000 optimizer steps, seed 0, EMA weights (decay 0.9999), and Humanoid Benchmark
augmentation v2. These weights use the hb_baselines/1 format.
Inputs: Unitree G1 state44 and separate ego_view and chest RGB images, resized
to 168 x 224. Output: 16 actions of dimension 76 (64 SONIC motion-token values and
6 measured values for each INSPIRE hand). Normalization is stored in the weights.
The ResNet18 visual backbone was initialized with ImageNet weights and fine-tuned.
The checkpoint contains model.pt, config.json and protocol.json (weight size
and SHA-256). SHA256SUMS covers the checkpoint and the accompanying inference
source snapshot. The snapshot is under code/hb_baselines_v1/; its requirements
record the training environment. Optimizer/resume state is excluded.
After downloading this repository and installing its inference dependencies, load the checkpoint from the repository root:
import sys
sys.path.insert(0, 'code/hb_baselines_v1')
from baselines.runtime import BaselinePolicy
policy = BaselinePolicy(
'a20-basket-transfer/b512-trimv2/step-004000',
device='cpu', policy='fm_dit', robot='g1',
)
print(policy.info())
The saved weights passed SHA-256 verification and strict CPU loading before upload.
The offline metrics in config.json use training frames; they are not held-out
or closed-loop task success rates.
A18 pour water and A17 unscrew bottle (H2 + Sharpa), batch 512, step 4000, with resume state (2026-11-17)
| Task | Checkpoint | Training data (all 100 demonstrations) | Frames | Epochs |
|---|---|---|---|---|
| A18 pour water | a18-pour-water/b512/step-004000/ |
humanoid-benchmark/a18-pour-water-expert-v1 @ 8e956a10 |
72,752 | 28.2 |
| A17 unscrew bottle | a17-unscrew-bottle/b512/step-004000/ |
humanoid-benchmark/a17-unscrew-bottle-expert-v1 @ 407e6e20 |
115,333 | 17.8 |
Same architecture as A20 (ResNet18 + AdaLN DiT, hidden 384, 4 layers, 8 heads, 10 midpoint sampling steps),
hb_baselines/1 format. Training: batch size 512, 4,000 optimizer steps, seed 0, no held-out episodes, Humanoid
Benchmark augmentation v2 (hb-aug-v2-20261109), AdamW (lr 1e-4, betas 0.95/0.999, weight decay 1e-6), the
ImageNet-initialized ResNet18 fine-tuned at 0.1 x the learning rate with frozen BatchNorm, 500-step linear warm-up
then cosine decay to 0 at step 4,000, bf16 autocast. The saved weights are the EMA (Diffusion Policy warm-up
schedule, maximum decay 0.9999; 0.998 at step 4,000). Source: Humanoid Benchmark commit 028df1bb
(evaluation/baselines, evaluation/augmentation).
Inputs: Unitree H2 with Sharpa hands, state78 (observation.state 75 + observation.projected_gravity 3) and
separate head and chest RGB images, resized to 168 x 224. Output: 16 actions of dimension 75
(action.body_target 31 + action.hands 44). Normalization is stored in the weights.
Closed-loop result (simulation, 50 random layouts per task, 2026-11-17): 0/50 on both tasks. The arm reaches the pitcher handle / the bottle, but the hand does not close on it. The flow-matching loss was still falling when the learning rate reached 0, so these runs are undertrained; the resume state below is provided to continue them.
Files
Under <task>/b512/:
step-004000/:model.pt(EMA weights),config.json(full training configuration, normalization statistics, offline metrics on 256 training frames) andprotocol.json(weight size and SHA-256).train_state.pt: the resume state at step 4,000: raw (non-EMA) policy weights, AdamW state, LR-scheduler state, EMA weights, step counter and training configuration. Its EMA weights are identical tostep-004000/model.pt.run.json: the original command line, configuration and augmentation preflight.train_log.jsonl: loss, learning rate and EMA decay every 100 steps, and the offline metrics of the save.SHA256SUMS: covers these files and the source snapshot; check it from the repository root, e.g.sha256sum -c a18-pour-water/b512/SHA256SUMS.
code/hb_baselines_v1/ now also holds the training sources (baselines/train.py, baselines/data.py,
baselines/adapters.py and augmentation/) from the same commit as the inference files, which are unchanged.
Load for inference
import sys
sys.path.insert(0, 'code/hb_baselines_v1')
from baselines.runtime import BaselinePolicy
policy = BaselinePolicy(
'a18-pour-water/b512/step-004000',
device='cpu', policy='fm_dit', robot='h2_sharpa',
)
print(policy.info())
Resume training
hf download humanoid-benchmark/fm-dit --include 'a18-pour-water/b512/*' --include 'code/*' --local-dir fm-dit
hf download humanoid-benchmark/a18-pour-water-expert-v1 --repo-type dataset \
--revision 8e956a102c84f991d2745a8cc66155c9991b742d --local-dir a18-pour-water-expert-v1
mkdir -p runs/a18_fm_dit && cp fm-dit/a18-pour-water/b512/train_state.pt runs/a18_fm_dit/
python fm-dit/code/hb_baselines_v1/baselines/train.py --policy fm_dit --contract h2_sharpa \
--dataset a18-pour-water-expert-v1 --out runs/a18_fm_dit --resume \
--steps 60000 --batch-size 512 --save-every 5000 --device cuda:0
For A17 use a17-unscrew-bottle/b512/ and humanoid-benchmark/a17-unscrew-bottle-expert-v1 at revision
407e6e20db2debc9b77fc2813e3445f549d51a33.
--resumereads<out>/train_state.ptand continues from step 4,000; check that the log prints{"event": "resumed", "step": 4000}, because without that file the run starts from step 0. It refuses another policy configuration (keep--policy fm_dit,--contract h2_sharpaand the defaults for horizon, cameras, image size, backbone and--model) and another augmentation (keep the default--augmentation hb-v2with--shift 0). Batch size, workers, save interval and device may change.--stepsis the new total. The schedule is recomputed for it, without a second warm-up: the learning rate restarts at 0.97 x 1e-4 for--steps 30000or 0.99 x 1e-4 for--steps 60000and decays by cosine to 0 at the new total. The EMA continues on the global step.- The run writes
checkpoints/step-NNNNNN/andcheckpoints/finalinto--out, replacestrain_state.ptat every save, appends totrain_log.jsonland rewritesrun.jsonwith the new command (the original stays here). Outside a git checkoutcode_commitis recorded as null; the source is commit028df1bb. - The first run decodes the frames of both cameras once into
--cache-dir(default~/hb_models/baselines/cache; about 15 GB for A18 and 26 GB for A17). The augmentation runs on the CPU (about 17 ms of one core per item), so--workerssets the speed; the original runs used 14 workers each. - Environment:
code/hb_baselines_v1/baselines/requirements.txt(Python 3.12, torch 2.9.0 with CUDA 12.8).
Verified before upload: both checkpoints load strictly on CPU from this layout, and a resume from this layout continued each run from step 4,000 to 4,002 on CPU and saved a new checkpoint.
B6 pedal bin, batch 512, step 4000, with resume state (2026-11-17)
Checkpoint: b6-pedal-bin/b512/step-004000/ (the run's last checkpoint).
Trained with evaluation/baselines/train.py --policy fm_dit --contract g1_sonic_dfx6 (Humanoid Benchmark commit
4385573e; code/hb_baselines_v1/ is byte-identical to its evaluation/baselines and evaluation/augmentation) on
the first 100 episodes (65,127 frames, about 31.4 epochs) of the B6 v5t7 ready-start expert data, the data of the
GR00T b6-pedal-bin/b256 and Psi0 b6-pedal-bin/b512 checkpoints (humanoid-benchmark/b6-pedal-bin v5/, episodes
0-99). Same architecture as A20 (ResNet18 + AdaLN DiT, hidden 384, 4 layers, 8 heads, 10 midpoint sampling steps),
hb_baselines/1 format. Training: batch size 512 (fits the 32 GB RTX 5090), 4,000 optimizer steps, seed 0, no held-out
episodes, Humanoid Benchmark augmentation v2 (hb-aug-v2-20261109; joint-angle bias +-2 deg plus 0.25 deg noise on
the 29 body joints only), AdamW (lr 1e-4, betas 0.95/0.999, weight decay 1e-6), the ImageNet-initialized ResNet18
fine-tuned at 0.1 x the learning rate with frozen BatchNorm, 200-step linear warm-up then cosine decay to 0 at step
4,000, bf16 autocast, EMA weights (0.998 at step 4,000). Before training a real batch of 512 was checked: state 44-D
without velocity, only the body joints perturbed, action labels bit-identical to the parquet files
(batch_check_b6_v5.json). Final flow-matching loss 0.109; offline check (normalized action MSE on 256 training
frames) 0.026.
Inputs: Unitree G1 state44 and separate ego_view and chest RGB images, resized to 168 x 224. Output: 16 actions of
dimension 76 (64 SONIC motion-token values and 6 measured values for each INSPIRE hand). Training instruction: "Pick up
the chip barrel from the cart, walk to the pedal bin, step on its pedal and drop it in."
Closed-loop result (Humanoid Benchmark debug/newton/b6_policy_eval.py --protocol v5_ready_start with the v5 task
configuration; 50 unseen layouts, seeds 180000-180049, 45 policy seconds): 0/50; the robot never touched the
barrel. Same seeds and protocol: GR00T b6-pedal-bin/b256/step-004000 39/50, Psi0 b6-pedal-bin/b512/step-004000
30/50, ACT (humanoid-benchmark/act b6-pedal-bin/b512/step-004000, this recipe) 8/50, DP 0/50. With the dataset's
own frames the checkpoint predicts the recorded actions (held-out episodes as well as trained ones); in closed loop its
first head-camera frames differ from the dataset's starts, and it does not set off towards the cart.
Files under b6-pedal-bin/b512/: step-004000/ (model.pt EMA weights, config.json with the full configuration,
normalization statistics and the offline metrics, protocol.json with the weight size and SHA-256), train_state.pt
(resume state at step 4,000: raw policy weights, AdamW, LR scheduler, EMA, step, configuration; its EMA equals
step-004000/model.pt), run.json (command line, configuration, augmentation preflight), train_log.jsonl,
train.log, batch_check_b6_v5.json, and SHA256SUMS (these files and the source snapshot; sha256sum -c b6-pedal-bin/b512/SHA256SUMS from the repository root). Load and resume as the A18 example above with
'b6-pedal-bin/b512/step-004000', robot='g1', --contract g1_sonic_dfx6 and the dataset
humanoid-benchmark/b6-pedal-bin v5/ with --episodes 0:100 (the resume itself was not test-run from this layout).
B5, B8 and A20 paused continuation packages (2026-11-17)
The following independent runs resumed the local step-4,000 baselines and were paused by the owner.
Each package contains a step-6,000 inference checkpoint and full optimizer/EMA resume state,
configuration, logs, dataset revision, hashes and portable resume instructions. The intended total
was 20,000; later logged updates were not checkpointed. These are separate from the existing
A20 b512-trimv2-cont step-10,000/20,000/30,000 lineage.
| Task | Continuation package | Saved step |
|---|---|---|
| B5 | b5-chair-push-standing-start/b512-cont-20261117/ | 6,000 |
| B8 | b8-table-wipe/b512-cont-20261117/ | 6,000 |
| A20 | a20-basket-transfer/b512-trimv2-cont-20261117/ | 6,000 |
Shared frozen training code and the opt-in resume launcher: code/fmdit_continue_20261117/.
The three source datasets are referenced at pinned revisions and are not redistributed here.
Offline errors are training-frame checks, not held-out or closed-loop performance.