HINT-WAM: Hindsight-Informed Training for World Action Models

HINT-WAM post-trains the world action model Fast-WAM. During training, the model acts as its own teacher: given hindsight (the future video of a demonstration), it supervises its own action expert on the denoising states that its sampler actually visits, using an energy-score objective. At deployment, only the student is used, so the policy has exactly the architecture and inference cost of Fast-WAM and never sees the future.

  • Paper: link coming soon
  • Code: link coming soon

Checkpoints

File Benchmark Trained from Notes
libero90/hintwam_libero90.pt LIBERO-90 Fast-WAM LIBERO checkpoint, after SFT (step 3k) main result
config.yaml Stage-B training config of the LIBERO checkpoint

Results

Success rates (%) with 50 trials per task (LIBERO-90) or 50 episodes per task and setting (RoboTwin 2.0, hard subset of 25 tasks).

Method LIBERO-90 RoboTwin clean RoboTwin randomized
Fast-WAM (released) 15.56 86.48 83.44
SFT 78.88 โ€“ โ€“
Full SFT (video + action expert) 81.24 84.12 83.24
BC control (same extra steps, no teacher) 71.56 85.36 82.08
HINT-WAM 85.18 87.04 83.68

Usage

Download the config together with a checkpoint:

from huggingface_hub import hf_hub_download

repo = "<hf-username>/HINT-WAM"
config = hf_hub_download(repo, "config.yaml")
ckpt = hf_hub_download(repo, "libero90/hintwam_libero90.pt")   # or robotwin/hintwam_robotwin.pt

or download everything with hf download <hf-username>/HINT-WAM --local-dir checkpoints/hintwam.

The checkpoints load exactly like the released Fast-WAM checkpoints. Closed-loop evaluation uses the LIBERO and RoboTwin evaluators of Fast-WAM with the stock settings (10 Euler steps; on LIBERO, replanning every 10 control steps) and the normalization statistics of the released Fast-WAM checkpoints.

Training

Stage 0 (SFT) Stage A (teacher) Stage B (HINT-WAM)
Init. released Fast-WAM SFT step 3k, frozen SFT step 3k
Trainable action expert hindsight encoder + adapters (13M) action expert
Learning rate 1e-4 3e-4 1e-4
Steps 5,000 2,000 2,000

All stages: AdamW, cosine schedule, effective batch size 192, 8 GPUs (24 GB), bfloat16, DeepSpeed ZeRO-2. The video expert is frozen throughout. Stage B uses the energy score with M = 4 student samples and a behavior-cloning anchor (both weights 1). No hyperparameter was tuned on RoboTwin.

License and acknowledgements

Released under the MIT license, as is Fast-WAM. HINT-WAM builds on Fast-WAM, LIBERO and RoboTwin 2.0.

Citation

Coming soon.

Downloads last month
-
Video Preview
loading

Paper for tuandattttt/HINT-WAM