|
Download README.md from humanoid-benchmark/act: direct link, hf CLI and curl.
- Browser
- Download file 8 kB
-
https://huggingface.co/humanoid-benchmark/act/resolve/main/README.md
- Command line
-
hf download hf://humanoid-benchmark/act/README.md
-
curl -L -o README.md https://huggingface.co/humanoid-benchmark/act/resolve/main/README.md
8 kB
| tags: | |
| - robotics | |
| - humanoid | |
| - imitation-learning | |
| # ACT β Humanoid Benchmark | |
| ACT (Action Chunking Transformer, ResNet18 backbone, CVAE; Psi0 `baselines/act`) checkpoints trained on Humanoid Benchmark tasks. | |
| ## Checkpoints | |
| 100 demonstrations per task. Every row is one training run; the epoch folders are its published checkpoints. | |
| | Folder | Task | Robot | Cameras (model input) | Global batch | Proprio augmentation | Checkpoints | | |
| |---|---|---|---|---|---|---| | |
| | `b6-pedal-bin/` | B6 β pick up a chip barrel, step on the pedal bin, drop it in | Unitree G1 (Inspire hands) | ego_view, chest | 696 | earlier recipe (with delay/drop) | `epoch-020` | | |
| | `a3-fridge-open/` | A3 β open the fridge door | Unitree G1 (Inspire hands) | ego_view, chest | 696 | earlier recipe (with delay/drop) | `epoch-010`, `epoch-020` | | |
| | `a7-cabinet-close/` | A7 β close the cabinet door | Unitree G1 (Inspire hands) | ego_view, chest | 696 | earlier recipe (with delay/drop) | `epoch-010`, `epoch-020` | | |
| | `a10-trolley-push/` | A10 β push a trolley | Unitree G1 (Inspire hands) | ego_view, chest | 696 | earlier recipe (with delay/drop) | `epoch-010`, `epoch-020` | | |
| | `a20-basket-transfer/` | A20 β transfer a basket | Unitree G1 (Inspire hands) | ego_view, chest | 696 | earlier recipe (with delay/drop) | `epoch-010`, `epoch-020` | | |
| | `b7-black-clothes-hamper/` | B7 β take the black garment off the hanger and drop it into the hamper | Unitree G1 (Inspire hands) | ego_view, chest | 696 | earlier recipe (with delay/drop) | `epoch-010`, `epoch-020` | | |
| | `a18-pour-water/` | A18 β pour water from a pitcher into a glass | H2 (Sharpa hands) | head, chest | 696 | earlier recipe (with delay/drop) | `epoch-010`, `epoch-020` | | |
| | `a17-unscrew-bottle/` | A17 β unscrew the cap of a bottle | H2 (Sharpa hands) | head, chest | 696 | earlier recipe (with delay/drop) | `epoch-010`, `epoch-020` | | |
| ## What an epoch folder contains | |
| Every `epoch-XXX/` folder is self-contained for evaluation and deployment: weights, model configuration, normalization | |
| statistics and `protocol.json` (task, data, input/output dimensions, cameras, batch, steps, augmentation, file list). | |
| Simple training records are included for reproducibility (training plan, training log, configs, metrics); optimizer / | |
| scheduler / RNG / DeepSpeed resume state is not. `code/` holds the adapter code used for training and inference | |
| (`code/VERSIONS.json`: upstream commits and package versions). | |
| ## Inputs and augmentation | |
| Model inputs are identical across all benchmark models: G1 state 44-D (29 body joint angles + 6 + 6 measured hand values | |
| + 3 projected gravity), H2 state 78-D (75 joint angles + 3 projected gravity), no joint velocities. Cameras: G1 ego view + | |
| chest; H2 head + chest (A14: head + chest + wrist). The third-person camera recorded in the datasets is for data review | |
| only and is never a model input. Each camera is a separate image input, except for Delta Flow and DiT4DiT, which | |
| place the views side by side in one frame. | |
| Training uses the Humanoid Benchmark online data augmentation spec v2 (15 photometric / geometric / sensor / lighting | |
| effects, one of them boosted to 1.5x per sample) with the model's own image augmentation turned off, plus joint-angle | |
| bias and noise on the proprioception (column "Proprio augmentation"; the earlier recipe also delayed the | |
| proprioception and dropped action-history rows). | |
| ## A20 pre-reach trim v2, batch 512, step 4000 (2026-11-17) | |
| Checkpoint: [`a20-basket-transfer/b512-trimv2/step-004000/`](a20-basket-transfer/b512-trimv2/step-004000). | |
| Trained on 100 demonstrations (54,433 frames) from | |
| `humanoid-benchmark/a20-basket-transfer-pre-reach-trim-expert-v2`, with batch size 512, | |
| 4,000 optimizer steps, seed 0, EMA weights (decay 0.9999), and Humanoid Benchmark | |
| augmentation v2. These weights use the `hb_baselines/1` format. | |
| Inputs: Unitree G1 state44 and separate `ego_view` and `chest` RGB images, resized | |
| to 168 x 224. Output: 16 actions of dimension 76 (64 SONIC motion-token values and | |
| 6 measured values for each INSPIRE hand). Normalization is stored in the weights. | |
| The ResNet18 visual backbone was initialized with ImageNet weights and fine-tuned. | |
| The checkpoint contains `model.pt`, `config.json` and `protocol.json` (weight size | |
| and SHA-256). `SHA256SUMS` covers the checkpoint and the accompanying inference | |
| source snapshot. The snapshot is under `code/hb_baselines_v1/`; its requirements | |
| record the training environment. Optimizer/resume state is excluded. | |
| After downloading this repository and installing its inference dependencies, | |
| load the checkpoint from the repository root: | |
| ```python | |
| import sys | |
| sys.path.insert(0, 'code/hb_baselines_v1') | |
| from baselines.runtime import BaselinePolicy | |
| policy = BaselinePolicy( | |
| 'a20-basket-transfer/b512-trimv2/step-004000', | |
| device='cpu', policy='act', robot='g1', | |
| ) | |
| print(policy.info()) | |
| ``` | |
| The saved weights passed SHA-256 verification and strict CPU loading before upload. | |
| The offline metrics in `config.json` use training frames; they are not held-out | |
| or closed-loop task success rates. | |
| This release uses the accompanying benchmark baseline loader. The earlier | |
| `epoch-XXX` releases retain their original Psi0 ACT loader and file format. | |
| ## B6 batch 512, step 4000 (2026-11-17) | |
| Checkpoint: [`b6-pedal-bin/b512/step-004000/`](b6-pedal-bin/b512/step-004000). A new training run on B6; the earlier | |
| B6 release `b6-pedal-bin/epoch-020/` (Psi0 ACT loader, batch 696, earlier augmentation recipe) is kept unchanged. | |
| Trained with `evaluation/baselines/train.py --policy act --contract g1_sonic_dfx6` (Humanoid_benchmark commit | |
| 4385573e) on the first 100 episodes (65,127 frames) of the B6 v5t7 ready-start expert data, the data of the GR00T | |
| `b6-pedal-bin/b256` and Psi0 `b6-pedal-bin/b512` checkpoints: batch size 512, 4,000 optimizer steps (about 31.4 | |
| epochs), AdamW lr 1e-4, 200-step linear warm-up then cosine decay to step 4000, seed 0, EMA weights (decay 0.9999), | |
| bf16 autocast, ImageNet-initialized ResNet18 fine-tuned at 0.1x the learning rate (frozen BatchNorm), and Humanoid | |
| Benchmark augmentation v2 (one call per sample on both views; joint-angle bias +-2 deg plus 0.25 deg noise on the 29 | |
| body joints only; no proprio delay or history drop; native random shift off). Before training, a real batch of 512 | |
| was checked for every model: state 44-D without velocity, only the body joints perturbed, action labels bit-identical | |
| to the parquet files (`training_records/batch_check_b6_v5.json`). Final loss: L1 0.050, KL 0.006; offline check | |
| (normalized action MSE on 256 training frames) 0.0054. These weights use the `hb_baselines/1` format and the same | |
| `code/hb_baselines_v1/` loader as the A20 release above (byte-identical to the training code). | |
| Inputs: Unitree G1 state44 and separate `ego_view` and `chest` RGB images, resized to 168 x 224. Output: 16 actions of | |
| dimension 76 (64 SONIC motion-token values and 6 measured values for each INSPIRE hand). Normalization is stored in the | |
| weights. Training instruction: "Pick up the chip barrel from the cart, walk to the pedal bin, step on its pedal and | |
| drop it in." | |
| Closed-loop evaluation (Humanoid_benchmark `debug/newton/b6_policy_eval.py --protocol v5_ready_start` with the v5 task | |
| configuration; 50 unseen layouts, seeds 180000-180049, 45 policy seconds; success = the barrel inside the bin and | |
| released): **8/50** (Wilson 95 % 8-29 %); the barrel touched 45, lifted 28, pedal pressed 10, in the bin 10. Same | |
| seeds and protocol: GR00T `b6-pedal-bin/b256/step-004000` 39/50, Psi0 `b6-pedal-bin/b512/step-004000` 30/50, | |
| FM-DiT 0/50, DP 0/50 (trained with this recipe). | |
| Files: `model.pt`, `config.json`, `protocol.json` (weight size and SHA-256), `SHA256SUMS` (the checkpoint, the | |
| loader snapshot and the training records) and `training_records/` (run configuration `run.json`, `train_log.jsonl`, | |
| `train.log`, the pre-training batch check). Optimizer/resume state is excluded. Load it as the A20 example above with | |
| `'b6-pedal-bin/b512/step-004000'`. | |