| --- |
| license: mit |
| tags: |
| - robotics |
| - tactile |
| - vla |
| --- |
| # FACT: A Multimodal Force–Action–Track Foundation Model for Human-to-Robot Dexterity |
|
|
| Training code and released checkpoints for FACT. The policy is a block-causal transformer that |
| predicts 16-step, 20 Hz delta-base EEF62 action chunks together with the 16 following EEF62 |
| states (coarse flow, τ 1→0.4), followed by a force-only tactile refinement stage (τ 0.4→0, |
| refreshed at 5 Hz from 10 F6 tactile sensors). Frozen perception (Wan2.2 VAE image tokenizer, |
| umT5-XXL text encoder) is used offline only. |
|
|
| ## Checkpoints |
|
|
| | name | trunk | params | files | |
| |---|---|---|---| |
| | `checkpoints/fact_small` | 12-layer block-causal transformer, d=1024 | 213 M | `fact_config.json`, `fact_model.pt` (fp32) | |
| | `checkpoints/fact_large` | 30-layer Wan-style block-causal DiT, d=3072 | 5.09 B | `fact_large_config.json`, `fact_large_model.pt` (bf16 trunk, fp32 force stage) | |
|
|
| ```python |
| from fact.model import FactPolicy # fact_small |
| from fact.large import FactLargePolicy # fact_large |
| |
| policy = FactLargePolicy.load("checkpoints/fact_large").to("cuda").eval() |
| out = policy.sample(batch, seed=0, mode="cascade") # out["action_pred"]: [B, 16, 64], out["future_state_pred"]: [B, 16, 64] |
| ``` |
|
|
| Both policies share the batch contract (`state` [B,K,64], `obs_latents` [B,K,48,10,20], |
| `current_force` [B,K,10,6], `force_history` [B,K,16,10,6], `force_history_valid` [B,K,16], |
| `text_tokens` [B,64,4096], `text_mask` [B,64]) and the output convention (normalised 62-D |
| delta-base actions, padded to 64; `fact.runtime_stats.delta_base_to_absolute()` inverts them). |
|
|
| ## Layout |
|
|
| ``` |
| fact/ model + data package |
| model.py FactConfig, FactCoarseTransformer, FactPolicy (forward / sample / refine_action_suffix) |
| large/ FactLargeConfig, FactLargePolicy, FactLargeWanModel (Wan-style DiT trunk, fact/large/wan/) |
| force.py force-only refinement stage; tactile_vq.py tactile VQ |
| data.py FactEpisode / FactAnchorDataset / FactMultiEpisodeDataset / collate |
| data_utils.py 20 Hz timestamp grid, 6-D rotation, delta-base EEF62 math |
| runtime_stats.py normalisation statistics, delta-base -> absolute inversion |
| perception/ Wan2.2 VAE, umT5 encoder, tokenizer (cache building only) |
| configs/ fact_small.json, fact_base.json, fact_base_text.json, fact_large.json |
| scripts/ |
| train_full.sh fact_small / fact_base training launcher (offline caches -> torch.distributed DDP) |
| train_ddp.py DDP trainer for the small/base configs |
| train_large.sh fact_large launcher (memory-mapped arrays + per-source statistics -> DDP / ZeRO-1) |
| train_large_ddp.py DDP trainer for fact_large (async checkpointing, held-out open-loop eval) |
| prepare_arrays.py memory-mapped episode arrays + per-source statistics for train_large |
| build_cache_dataset.py VAE latent cache (shardable over GPUs) and umT5 text cache for a dataset |
| check_dataset_ready.py dataset structure / statistics validation |
| eval_multi.py open-loop evaluation of a checkpoint over many episodes |
| param_count.py, param_count_large.py parameter budget of a config |
| train.py / eval.py / build_cache.py / run_single_episode.sh / collect.py single-episode protocol |
| data_tools/ build_merged_dataset.py (merge several datasets), north_ces_fk_params.json (FK parameters) |
| docs/ METHOD.md (fact_small / fact_base), FACT_LARGE.md (fact_large), DATA_FORMAT.md (dataset layout) |
| ``` |
|
|
| ## Installation |
|
|
| ```bash |
| pip install -e ".[train]" # torch>=2.8, diffusers, pyarrow, einops, transformers, decord2, ... |
| ``` |
|
|
| Frozen perception weights (from the Wan2.2-TI2V-5B release) under `checkpoints/Wan2.2-TI2V-5B/` |
| (or `WAN22_CKPT_DIR=...`): `Wan2.2_VAE.pth`, `models_t5_umt5-xxl-enc-bf16.pth`, |
| `google/umt5-xxl/tokenizer.json`. `fact/large/wan/` can use FlashAttention / cuDNN fused attention |
| when installed and falls back to PyTorch SDPA otherwise. |
|
|
| ## Data |
|
|
| LeRobot v2.1 datasets in the FACT force layout (see `docs/DATA_FORMAT.md`): per-episode parquet |
| with `timestamp`, `observation.state_eef62` [62], `action.eef62_absolute` [62], |
| `observation.tactile_force` [60], three RGB views (`head_left`, `left_wrist`, `right_wrist`) and |
| `meta/{info.json, episodes.jsonl, tasks.jsonl, stats.json, relative_stats_dreamzero.json, modality.json}`. |
|
|
| ```bash |
| python scripts/check_dataset_ready.py --dataset-root /data/my_dataset |
| python data_tools/build_merged_dataset.py --source a=/data/dataset_a:/cache/dataset_a \ |
| --source b=/data/dataset_b --out /data/merged --cache /cache/merged # optional merge |
| ``` |
|
|
| ## Training |
|
|
| ```bash |
| # fact_small / fact_base |
| DATA_ROOT=/data/my_dataset OUTPUT_DIR=checkpoints/run NUM_GPUS=16 PER_DEVICE_BATCH=8 bash scripts/train_full.sh |
| |
| # fact_large (5.09 B, from scratch) |
| python scripts/prepare_arrays.py --dataset-root /data/my_dataset --cache-dir /cache/my_dataset |
| DATA_ROOT=/data/my_dataset CACHE_DIR=/cache/my_dataset OUTPUT_DIR=checkpoints/run_large NUM_GPUS=16 \ |
| bash scripts/train_large.sh |
| ``` |
|
|
| Extra arguments are forwarded to the respective `train_*_ddp.py`. The launchers build the caches |
| on first use (`<CACHE_DIR>/latents/episode_XXXXXX.npy` float16 [N,48,10,20]; |
| `<CACHE_DIR>/text_cache.npz`), validate the dataset and run `python -m torch.distributed.run`. |
| Multi-node: run the same command on every node with `NNODES=<n> NODE_RANK=<i> MASTER_ADDR=<node0>`. |
| Env overrides: `DATA_ROOT OUTPUT_DIR CACHE_DIR FACT_CONFIG WAN22_CKPT_DIR NUM_GPUS PER_DEVICE_BATCH LR |
| MAX_STEPS SAVE_STEPS` (see the launcher headers for the full list). |
| |
| ## Evaluation |
| |
| ```bash |
| python scripts/eval_multi.py --checkpoint checkpoints/fact_small \ |
| --dataset-root /data/my_dataset --cache-dir /cache/my_dataset \ |
| --episodes 0-99 --anchors-per-episode 8 --out eval.json --bf16 |
| ``` |
| |
| Reports normalised 62-D delta-base action MSE (cascade / coarse-only / coarse-to-zero, arm / hand) |
| and future-state MSE. `sample()` returns a chunk; `FactPolicy.refine_action_suffix()` re-runs the |
| force stage at offsets 0/4/8/12 on new tactile readings. |
| |
| ## License |
| |
| MIT (`LICENSE`). `fact/perception/` and `fact/large/wan/` are derived from the Wan2.2 release and keep its license. |
| |
