README.md

---
license: mit
library_name: pytorch
pipeline_tag: robotics
tags:
  - vision-language-action
  - robotics
  - libero
  - progress-estimation
  - failure-detection
  - flow-matching
base_model: Qwen/Qwen3-VL-4B-Instruct
---

# PACE · LIBERO

> ✍️ **Fei Ni**, Zhuo Chen, Xianze Yao, Zerui Chen, Changrui Chen, Yifu Yuan, Shan Luo, Jianye Hao, Jiankang Deng<br>
> 💻 Code: [github.com/Fei-Ni/PACE](https://github.com/Fei-Ni/PACE)

PACE is a vision-language-action policy that internalizes task progress: it predicts a *foresight anchor* (an embedding of the finished task), reads its current progress and a forward commitment against it, and conditions its action decoder on them. This repository holds one checkpoint trained jointly on the four LIBERO suites.

## Result

LIBERO success rate (%), 10 tasks × 50 trials per suite, seed 7, evaluated with `examples/LIBERO/eval_libero.sh` from the code repository.

| Spatial | Object | Goal | Long | **Average** |
|---|---|---|---|---|
| 99.0 | 99.8 | 99.4 | 96.2 | **98.6** |

The paper (Table 1) reports 99.0 / 99.8 / 97.8 / 98.0, average 98.7. This checkpoint was retrained with the released code; the recipe is described in the code repository's `docs/TRAINING.md`.

## Failure alarm

Every response of the policy server also carries a failure alarm (`audit_flag`, with `audit_score` and `audit_stall`): it fires when the progress the policy commits to keeps going undelivered, or when its progress reading stops rising. With the thresholds in `config.yaml` (a shared 0.4 for unmet commitments; the stall limit calibrated on other checkpoints, never on this one), it flags **92.9%** of the 28 failed episodes of the evaluation above and 7.2% of the 1,972 successful ones, with a median of half the episode still left. It only reports and never changes the actions. Details and a calibration script: `docs/PROGRESS.md` in the code repository.

## Architecture

| Component | Choice |
|---|---|
| Vision-language backbone | Qwen3-VL-4B-Instruct |
| Action head | flow-matching DiT-B (16 layers), chunk of 8 actions, 4 denoising steps |
| Progress encoder | DINOv2 ViT-B/14, frozen (CLS, 768-d) |
| Progress interface | 4 goal-query tokens → foresight anchor; p<sub>cur</sub>, Δp<sub>cur</sub> readouts; Δp<sub>prev</sub> input token; anchor + p<sub>cur</sub> + Δp<sub>cur</sub> prepended to the DiT |
| Inputs | third-person + wrist images (224 × 224) at t and at the latest query at or before t − 7 (t − 8 with the LIBERO client), instruction; no proprioception |
| Actions | 7-D end-effector delta (xyz, rpy, gripper), normalized with `dataset_statistics.json` (key `franka`) |

## Training data

The four `*_no_noops` LIBERO datasets (Spatial, Object, Goal, 10) in LeRobot format, 30K steps at batch 64.

## Files

```
PACE-LIBERO/
├── README.md
├── config.yaml               # model config (base_vlm = Qwen/Qwen3-VL-4B-Instruct)
├── dataset_statistics.json   # action normalization statistics
└── pytorch_model.pt          # full model state dict
```

## How to load

```bash
huggingface-cli download spikefly/PACE-LIBERO --local-dir ./PACE-LIBERO
```

```python
from pace.model.framework.base_framework import baseframework

policy = baseframework.from_pretrained("./PACE-LIBERO").cuda().eval()   # or "spikefly/PACE-LIBERO"
```

Qwen3-VL-4B-Instruct (processor and architecture) and the DINOv2 ViT-B/14 weights are fetched on first use; set `BASE_VLM` / `DINOV2_WEIGHTS` to local copies to skip the downloads. For evaluation, see `examples/LIBERO/` in the code repository.

## Sibling checkpoint

| Checkpoint | Benchmark |
|---|---|
| [spikefly/PACE-SimplerEnv](https://huggingface.co/spikefly/PACE-SimplerEnv) | SimplerEnv WidowX |

## Citation

```bibtex
@misc{ni2026pace,
  title  = {{PACE}: Progress Actively-internalized Conditioning Execution for Long-horizon Manipulation},
  author = {Ni, Fei and Chen, Zhuo and Yao, Xianze and Chen, Zerui and Chen, Changrui and Yuan, Yifu and Luo, Shan and Hao, Jianye and Deng, Jiankang},
  year   = {2026}
}
```

## License

MIT for the PACE weights and code. The backbone weights derive from Qwen3-VL-4B-Instruct (Apache-2.0) and the progress encoder is DINOv2 (Apache-2.0); LIBERO data follows its own license.