FACT: A Multimodal Force–Action–Track Foundation Model for Human-to-Robot Dexterity

Training code for FACT. The policy is a block-causal transformer trained from scratch on T-Rex-format LeRobot v2.1 data; it predicts 16-step, 20 Hz delta-base EEF62 action chunks together with the 16 following EEF62 states (coarse flow, τ 1→0.4), followed by a force-only tactile refinement stage (τ 0.4→0, refreshed at 5 Hz from 10 F6 tactile sensors). Frozen perception (Wan2.2 VAE image tokenizer, umT5-XXL text encoder) is used offline only.

Layout

fact/                 model + data package
  model.py            FactConfig, FactCoarseTransformer, FactPolicy (forward / sample / refine_action_suffix)
  force.py            force-only refinement stage;  tactile_vq.py  tactile VQ
  data.py             FactEpisode / FactAnchorDataset / FactMultiEpisodeDataset / collate
  data_utils.py       20 Hz timestamp grid, 6-D rotation, delta-base EEF62 math
  runtime_stats.py    q01/q99 normalisation statistics, delta-base -> absolute inversion
  perception/         Wan2.2 VAE, umT5 encoder, tokenizer (cache building only)
configs/              fact_base.json (209 M), fact_base_text.json (+ text conditioning, 213 M), fact_small.json (109 M)
scripts/
  train_full.sh       full training launcher: offline caches -> torch.distributed DDP (single / multi node)
  train_ddp.py        DDP trainer (bf16, cosine LR, checkpoint / resume, W&B, monitoring video)
  build_cache_dataset.py   VAE latent cache (shardable over GPUs) and umT5 text cache for a dataset
  check_dataset_ready.py   dataset structure / statistics validation
  eval_multi.py       open-loop evaluation of a checkpoint over many episodes
  train.py / eval.py / build_cache.py / run_single_episode.sh / collect.py   single-episode protocol
  param_count.py      parameter budget of a config
data_tools/           build_merged_dataset.py (merge several datasets), north_ces_fk_params.json (Sharpa FK)
docs/                 METHOD.md (architecture), DATA_FORMAT.md (dataset layout), FACT_MODEL_zh.md
results/              single-episode benchmark tables and raw baseline JSON

Installation

pip install -e ".[train]"        # torch>=2.8, pyarrow, einops, transformers, wandb, decord2, ...

Frozen perception weights (from the Wan2.2-TI2V-5B release) under checkpoints/Wan2.2-TI2V-5B/ (or WAN22_CKPT_DIR=...): Wan2.2_VAE.pth, models_t5_umt5-xxl-enc-bf16.pth, google/umt5-xxl/tokenizer.json.

Data

LeRobot v2.1 datasets in the T-Rex force layout (see docs/DATA_FORMAT.md): per-episode parquet with timestamp, observation.state_eef62 [62], action.eef62_absolute [62], observation.tactile_force [60], three RGB views (head_left, left_wrist, right_wrist) and meta/{info.json, episodes.jsonl, tasks.jsonl, stats.json, relative_stats_dreamzero.json, modality.json}.

python scripts/check_dataset_ready.py --dataset-root /data/trex_full_force
python data_tools/build_merged_dataset.py --source trex=/data/trex_full_force:/cache/trex_full_force \
    --source sharpadex=/data/SharpaDex_trex --out /data/ALL_trex --cache /cache/ALL_trex   # optional merge

Training

DATA_ROOT=/data/trex_full_force OUTPUT_DIR=checkpoints/fact_trex NUM_GPUS=16 PER_DEVICE_BATCH=8 \
bash scripts/train_full.sh                         # extra args are forwarded to scripts/train_ddp.py

The launcher builds the caches on first use (<CACHE_DIR>/latents/episode_XXXXXX.npy float16 [N,48,10,20]; <CACHE_DIR>/text_cache.npz), validates the dataset and runs python -m torch.distributed.run scripts/train_ddp.py --config configs/fact_base_text.json --blocks 4 .... Defaults: 100 k steps, global batch 16×8, AdamW lr 3e-4 cosine (warm-up 2000), weight decay 0.01, grad clip 1.0, bf16 (force stage fp32, lr×0.3), checkpoint every 10 k steps, --resume auto. Multi-node: run the same command on every node with NNODES=<n> NODE_RANK=<i> MASTER_ADDR=<node0>. Env overrides: DATA_ROOT OUTPUT_DIR CACHE_DIR FACT_CONFIG WAN22_CKPT_DIR NUM_GPUS PER_DEVICE_BATCH LR MAX_STEPS SAVE_STEPS VIDEO_STEPS WANDB_PROJECT WANDB_RUN_NAME.

Single-episode protocol (results/single_episode_benchmark.md):

bash scripts/run_single_episode.sh fact_base 0 --config-json "$(cat configs/fact_base.json)" \
    --steps 3000 --batch 8 --lr 2e-4 --bf16

Evaluation

python scripts/eval_multi.py --checkpoint checkpoints/fact_trex/checkpoint-100000 \
    --dataset-root /data/trex_full_force --cache-dir cache/trex_full_force \
    --episodes 0-99 --anchors-per-episode 8 --out eval.json --bf16

Reports normalised 62-D delta-base action MSE (cascade / coarse-only / coarse-to-zero, arm / hand) and future-state MSE. FactPolicy.sample() returns a chunk; FactPolicy.refine_action_suffix() re-runs the force stage at offsets 0/4/8/12 on new tactile readings; fact.runtime_stats.delta_base_to_absolute() converts the output to absolute EEF poses.

License

MIT (LICENSE). fact/perception/ is derived from the Wan2.2 release and keeps its license.