SeeYooLeo/CIG-RoboCasa-Kitchen

[NeurIPS 2026] Counterfactual Instruction Grounding
for Vision-Language-Action Models

Shiyu Liu · Yu Zhou · Meng Liu · Xuanming Guo · Liqiang Nie

📖 Overview

Counterfactual Instruction Grounding (CIG) helps vision-language-action policies remain sensitive to language instructions during execution. Given the same observation and demonstrated action chunk, CIG trains the policy to distinguish the commanded instruction from counterfactual alternatives.

CIG augments imitation learning during continuation training, without changing the policy architecture or collecting additional demonstrations. This repository provides four robocasa-kitchen checkpoints covering two policy families:

• OpenVLA-OFT: continuous-regression action prediction.
• π0.5: flow-matching action generation.

🧠 Method

CIG builds a candidate instruction set, scores the same action chunk under each candidate, and optimizes the commanded instruction's contrastive posterior. OFT uses negative mean absolute action error; π0.5 uses flow-matching residual energy. CIG gradients update the VLM representation, with a saturation gate, while imitation learning trains the action head or expert.

📊 robocasa-kitchen results

Success rate (%) on 24 single-stage tasks, evaluated with 50 rollouts per task. These are the results reported in the manuscript; differences are absolute percentage points.

Policy	Pick & Place	Open & Close	Knob & Lever	Button	Insertion	Average
OpenVLA-OFT	14.8	63.7	50.4	54.0	31.0	40.7
OpenVLA-OFT + CIG	20.3	73.3	48.4	85.3	22.0	47.7
π0.5	56.5	83.0	62.4	80.7	38.0	65.8
π0.5 + CIG	64.8	86.7	65.6	84.0	47.0	71.3

CIG improves average success by 7.0 points for OFT and 5.5 points for π0.5. π0.5 improves across all five categories; OFT's category-level gains vary.

Manuscript example on PnPMicrowaveToCounter, comparing continued imitation learning and CIG. Triangles and stars indicate start and end waypoints.

🥽 VR data collection

We collect demonstrations with the Quest 3S / WebXR interface for Stretch 3. See the VR guide for setup, data collection and replay.

🤗 Choose a checkpoint

Checkpoint	Policy	Contents	Size
oft-baseline	OpenVLA-OFT	Merged model, action head, proprioceptive projector, tokenizer and statistics	15.45 GB
oft-cig	OpenVLA-OFT + CIG	Same layout as OFT baseline	15.45 GB
pi05-baseline	π0.5	PyTorch SafeTensors and normalization statistics	7.23 GB
pi05-cig	π0.5 + CIG	PyTorch SafeTensors and normalization statistics	7.47 GB

Sizes are rounded decimal GB. Download the complete checkpoint folder, including its configuration and normalization files. OFT LoRA updates are already merged; the π0.5 baseline is already converted from JAX to PyTorch.

📥 Download

From the companion robocasa-kitchen code release root:

pip install -U huggingface_hub
hf download SeeYooLeo/CIG-RoboCasa-Kitchen \
  --revision 4bdc391cb1dd6df8faa978e55aebf31e40c39da4 \
  --include "oft-baseline/*" --include "oft-cig/*" \
  --include "pi05-baseline/*" --include "pi05-cig/*" \
  --local-dir weights
python verify_weights.py

The pinned revision contains the verified weights. All 38 checkpoint files match the size and SHA-256 values in weights_manifest.json. The usage guide includes single-checkpoint download, standalone checksum verification, policy-server commands and evaluation.

🗂️ Data and training

We train on the official robocasa-kitchen demonstrations. The π0.5 baseline uses the official openpi training scripts, and CIG continues training from the corresponding imitation-learning baselines. Installation, data preparation and training instructions are provided in the companion code release.

🧪 Inference and evaluation

Use oft/serve.py for OFT or pi05/scripts/serve.py for π0.5. See the usage guide for inference and evaluation commands.

🙏 Acknowledgments and licenses

Built on OpenVLA-OFT, Prismatic VLMs, openpi, and robocasa-kitchen.

The repository's Apache-2.0 metadata does not replace upstream model terms. OFT derives from OpenVLA and its Llama-2 backbone; π0.5 includes Gemma-derived components. See third-party notices and model terms for the applicable upstream notices and links.

📄 Citation

@misc{liu2026cig,
  title  = {Counterfactual Instruction Grounding for Vision-Language-Action Models},
  author = {Liu, Shiyu and Zhou, Yu and Liu, Meng and Guo, Xuanming and Nie, Liqiang},
  year   = {2026}
}