README.md

---
license: mit
library_name: pytorch
pipeline_tag: robotics
tags:
  - vision-language-action
  - robotics
  - simplerenv
  - widowx
  - bridge
  - progress-estimation
  - failure-detection
  - flow-matching
base_model: Qwen/Qwen3-VL-4B-Instruct
---

# PACE · SimplerEnv (WidowX)

> ✍️ **Fei Ni**, Zhuo Chen, Xianze Yao, Zerui Chen, Changrui Chen, Yifu Yuan, Shan Luo, Jianye Hao, Jiankang Deng<br>
> 💻 Code: [github.com/Fei-Ni/PACE](https://github.com/Fei-Ni/PACE)

PACE is a vision-language-action policy that internalizes task progress: it predicts a *foresight anchor* (an embedding of the finished task), reads its current progress and a forward commitment against it, and conditions its action decoder on them. This repository holds the WidowX checkpoint evaluated on SimplerEnv.

## Result

SimplerEnv WidowX success rate (%), ManiSkill3 environments, 24 episodes per task (seeds 0–23), StarVLA's default adaptive action ensembling, evaluated with `examples/SimplerEnv/eval_simpler.sh` from the code repository.

| Put Spoon on Towel | Put Carrot on Plate | Stack Green Block on Yellow Block | Put Eggplant in Basket | **Average** |
|---|---|---|---|---|
| 87.5 | 70.8 | 62.5 | 100.0 | **80.2** |

The paper (Table 1) reports an average of 75.0. This checkpoint was retrained with the released code.

## Failure alarm

Every response of the policy server also carries a failure alarm (`audit_flag`, with `audit_score` and `audit_stall`): it fires when the progress the policy commits to keeps going undelivered, or when its progress reading stops rising. With the thresholds in `config.yaml` (a shared 0.4 for unmet commitments; the stall limit calibrated on other checkpoints, never on this one), it flags 6 of the 19 failed episodes of the evaluation above and 5 of the 77 successful ones; most of the failures it misses are near misses (the block lands on the other block and slides off) whose progress reads like a success. It only reports and never changes the actions. Details and a calibration script: `docs/PROGRESS.md` in the code repository.

## Architecture

| Component | Choice |
|---|---|
| Vision-language backbone | Qwen3-VL-4B-Instruct |
| Action head | flow-matching DiT-B (16 layers), chunk of 16 actions, 4 denoising steps |
| Progress encoder | DINOv2 ViT-B/14, frozen (CLS, 768-d) |
| Progress interface | 4 goal-query tokens → foresight anchor; p<sub>cur</sub>, Δp<sub>cur</sub> readouts; Δp<sub>prev</sub> input token; anchor + p<sub>cur</sub> + Δp<sub>cur</sub> prepended to the DiT |
| Inputs | third-person image (224 × 224) at t and t − 15, instruction; no proprioception |
| Actions | 7-D end-effector delta (xyz, rpy, gripper), normalized with `dataset_statistics.json` (key `oxe_bridge`) |

## Training

Training scripts and recipes are in the code repository (`docs/TRAINING.md`).

## Files

```
PACE-SimplerEnv/
├── README.md
├── config.yaml               # model config (base_vlm = Qwen/Qwen3-VL-4B-Instruct)
├── dataset_statistics.json   # action normalization statistics
└── pytorch_model.pt          # full model state dict
```

## How to load

```bash
huggingface-cli download spikefly/PACE-SimplerEnv --local-dir ./PACE-SimplerEnv
```

```python
from pace.model.framework.base_framework import baseframework

policy = baseframework.from_pretrained("./PACE-SimplerEnv").cuda().eval()   # or "spikefly/PACE-SimplerEnv"
```

Qwen3-VL-4B-Instruct (processor and architecture) and the DINOv2 ViT-B/14 weights are fetched on first use; set `BASE_VLM` / `DINOV2_WEIGHTS` to local copies to skip the downloads. For evaluation, see `examples/SimplerEnv/` in the code repository.

## Sibling checkpoint

| Checkpoint | Benchmark |
|---|---|
| [spikefly/PACE-LIBERO](https://huggingface.co/spikefly/PACE-LIBERO) | LIBERO (4 suites) |

## Citation

```bibtex
@misc{ni2026pace,
  title  = {{PACE}: Progress Actively-internalized Conditioning Execution for Long-horizon Manipulation},
  author = {Ni, Fei and Chen, Zhuo and Yao, Xianze and Chen, Zerui and Chen, Changrui and Yuan, Yifu and Luo, Shan and Hao, Jianye and Deng, Jiankang},
  year   = {2026}
}
```

## License

MIT for the PACE weights and code. The weights derive from StarVLA's Qwen3VL-GR00T-Bridge-RT-1 checkpoint (see its model card) and from Qwen3-VL-4B-Instruct (Apache-2.0); the progress encoder is DINOv2 (Apache-2.0); BridgeData V2 and SimplerEnv follow their own licenses.