---
license: mit
tags:
- robotics
- tactile
- vla
---
# FACT: A Multimodal Force–Action–Track Foundation Model for Human-to-Robot Dexterity

Training code for FACT. The policy is a block-causal transformer trained from scratch on
T-Rex-format LeRobot v2.1 data; it predicts 16-step, 20 Hz delta-base EEF62 action chunks
together with the 16 following EEF62 states (coarse flow, τ 1→0.4), followed by a force-only
tactile refinement stage (τ 0.4→0, refreshed at 5 Hz from 10 F6 tactile sensors).
Frozen perception (Wan2.2 VAE image tokenizer, umT5-XXL text encoder) is used offline only.

## Layout

```
fact/                 model + data package
  model.py            FactConfig, FactCoarseTransformer, FactPolicy (forward / sample / refine_action_suffix)
  force.py            force-only refinement stage;  tactile_vq.py  tactile VQ
  data.py             FactEpisode / FactAnchorDataset / FactMultiEpisodeDataset / collate
  data_utils.py       20 Hz timestamp grid, 6-D rotation, delta-base EEF62 math
  runtime_stats.py    q01/q99 normalisation statistics, delta-base -> absolute inversion
  perception/         Wan2.2 VAE, umT5 encoder, tokenizer (cache building only)
configs/              fact_base.json (209 M), fact_base_text.json (+ text conditioning, 213 M), fact_small.json (109 M)
scripts/
  train_full.sh       full training launcher: offline caches -> torch.distributed DDP (single / multi node)
  train_ddp.py        DDP trainer (bf16, cosine LR, checkpoint / resume, W&B, monitoring video)
  build_cache_dataset.py   VAE latent cache (shardable over GPUs) and umT5 text cache for a dataset
  check_dataset_ready.py   dataset structure / statistics validation
  eval_multi.py       open-loop evaluation of a checkpoint over many episodes
  train.py / eval.py / build_cache.py / run_single_episode.sh / collect.py   single-episode protocol
  param_count.py      parameter budget of a config
data_tools/           build_merged_dataset.py (merge several datasets), north_ces_fk_params.json (Sharpa FK)
docs/                 METHOD.md (architecture), DATA_FORMAT.md (dataset layout), FACT_MODEL_zh.md
results/              single-episode benchmark tables and raw baseline JSON
```

## Installation

```bash
pip install -e ".[train]"        # torch>=2.8, pyarrow, einops, transformers, wandb, decord2, ...
```

Frozen perception weights (from the Wan2.2-TI2V-5B release) under `checkpoints/Wan2.2-TI2V-5B/`
(or `WAN22_CKPT_DIR=...`): `Wan2.2_VAE.pth`, `models_t5_umt5-xxl-enc-bf16.pth`,
`google/umt5-xxl/tokenizer.json`.

## Data

LeRobot v2.1 datasets in the T-Rex force layout (see `docs/DATA_FORMAT.md`): per-episode parquet
with `timestamp`, `observation.state_eef62` [62], `action.eef62_absolute` [62],
`observation.tactile_force` [60], three RGB views (`head_left`, `left_wrist`, `right_wrist`) and
`meta/{info.json, episodes.jsonl, tasks.jsonl, stats.json, relative_stats_dreamzero.json, modality.json}`.

```bash
python scripts/check_dataset_ready.py --dataset-root /data/trex_full_force
python data_tools/build_merged_dataset.py --source trex=/data/trex_full_force:/cache/trex_full_force \
    --source sharpadex=/data/SharpaDex_trex --out /data/ALL_trex --cache /cache/ALL_trex   # optional merge
```

## Training

```bash
DATA_ROOT=/data/trex_full_force OUTPUT_DIR=checkpoints/fact_trex NUM_GPUS=16 PER_DEVICE_BATCH=8 \
bash scripts/train_full.sh                         # extra args are forwarded to scripts/train_ddp.py
```

The launcher builds the caches on first use (`<CACHE_DIR>/latents/episode_XXXXXX.npy` float16
[N,48,10,20]; `<CACHE_DIR>/text_cache.npz`), validates the dataset and runs
`python -m torch.distributed.run scripts/train_ddp.py --config configs/fact_base_text.json --blocks 4 ...`.
Defaults: 100 k steps, global batch 16×8, AdamW lr 3e-4 cosine (warm-up 2000), weight decay 0.01,
grad clip 1.0, bf16 (force stage fp32, lr×0.3), checkpoint every 10 k steps, `--resume auto`.
Multi-node: run the same command on every node with `NNODES=<n> NODE_RANK=<i> MASTER_ADDR=<node0>`.
Env overrides: `DATA_ROOT OUTPUT_DIR CACHE_DIR FACT_CONFIG WAN22_CKPT_DIR NUM_GPUS PER_DEVICE_BATCH LR
MAX_STEPS SAVE_STEPS VIDEO_STEPS WANDB_PROJECT WANDB_RUN_NAME`.

Single-episode protocol (`results/single_episode_benchmark.md`):

```bash
bash scripts/run_single_episode.sh fact_base 0 --config-json "$(cat configs/fact_base.json)" \
    --steps 3000 --batch 8 --lr 2e-4 --bf16
```

## Evaluation

```bash
python scripts/eval_multi.py --checkpoint checkpoints/fact_trex/checkpoint-100000 \
    --dataset-root /data/trex_full_force --cache-dir cache/trex_full_force \
    --episodes 0-99 --anchors-per-episode 8 --out eval.json --bf16
```

Reports normalised 62-D delta-base action MSE (cascade / coarse-only / coarse-to-zero, arm / hand)
and future-state MSE. `FactPolicy.sample()` returns a chunk; `FactPolicy.refine_action_suffix()`
re-runs the force stage at offsets 0/4/8/12 on new tactile readings;
`fact.runtime_stats.delta_base_to_absolute()` converts the output to absolute EEF poses.

## License

MIT (`LICENSE`). `fact/perception/` is derived from the Wan2.2 release and keeps its license.