ALPS / README.md

ALPS weights, LIBERO policies and model card

3714471 1 day ago

8.42 kB

---
license: apache-2.0
pipeline_tag: image-feature-extraction
tags:
- vision
- vision-transformer
- world-models
- robotics
- representation-learning
- image-text
- libero
- pytorch
datasets:
- mlfoundations/datacomp_1b
- UCSC-VLAA/Recap-DataComp-1B
---
# 🏔️ ALPS: Visual State Encoders for World Models

**ALPS** (A Language-aligned, Pixel-renderable State encoder) is a pretrained ViT-B/16 that brings physical-state
information, native language alignment, and pixel reconstruction into one set of patch tokens. Its frozen tokens serve as
the visual state of world models and robot policies, its native text tower relates instructions and goals stated in words
to the scene, and an evaluation decoder renders its tokens back into images.

[Project page](https://ucsc-vlaa.github.io/ALPS/) · [Code](https://github.com/UCSC-VLAA/ALPS) · [Paper](https://arxiv.org/abs/2610.30120)

## Files

| File | Content | Size |
|---|---|---|
| `alps_vitb16.safetensors` | **ALPS** (Stage 1 + Stage 2): the image encoder and the language interface | 786 MB |
| `alps_stage1_vitb16.safetensors` | **ALPS (Stage 1)**, before cross-view refinement: the image encoder and the language interface | 786 MB |
| `libero_policies/` | the eight ALPS policies of the LIBERO experiments (Table 2 of the paper), see below | 8 x 35 MB |
| `SHA256SUMS` | checksums of every file above | |

## Use

```bash
git clone https://github.com/UCSC-VLAA/ALPS.git && cd ALPS
pip install -e .                 # torch, numpy, pillow
```

```python
import torch, alps
from PIL import Image
model = alps.load("alps-vitb16")                          # downloads alps_vitb16.safetensors from this repository
x = alps.center_crop_transform(Image.open("image.jpg"))   # [1, 3, 224, 224], normalized
with torch.no_grad():
    out = model.encode_image(x)
    z = out["tokens"]            # [1, 196, 768]  raw patch tokens: the visual state
    img = model.image_embedding(tokens=z)                      # L2-normalized
    txt = model.text_embedding(["a robot arm opens the top drawer", "a photo of a dog"])
    probs = (model.contrastive_scale() * img @ txt.T).softmax(-1)
```

`alps.load("alps-stage1-vitb16")` loads ALPS (Stage 1). Simulator frames: resize the whole frame to 224
(`alps.resize_transform`). Use float32 or bfloat16 (float16 overflows in the deeper blocks). More in the
[repository README](https://github.com/UCSC-VLAA/ALPS).

## Model

* Architecture: ViT-B/16 (width 768, 12 blocks, 12 heads, MLP 3,072), class token, fixed 2-D sine-cosine position
  embedding, no patch-embedding bias, LayerNorm epsilon 1e-6. 85.6M parameters.
* Output: 196 raw patch tokens of width 768 at 224 x 224 pixels (the visual state), without a final LayerNorm.
* Language interface (from Stage 1, unchanged by Stage 2): a 12-layer text encoder of width 768 (109.7M parameters,
  WordPiece vocabulary, 32 tokens), linear image projection of the mean token, linear text projection, learned
  contrastive scale.

## Training

* Stage 1, from scratch: DataComp-1B images with their web captions and the synthetic captions of the LLaVA-LLaMA-3
  captioner of Recap-DataComp-1B. Objectives: image-text contrastive alignment, caption prediction, pixel reconstruction,
  and cross-view dense correspondence, all on the same patch tokens. 12.8 billion samples at 128 pixels, then 1.0 billion
  samples at 224 pixels.
* Stage 2, cross-view self-distillation on single images: from an augmented crop, the student predicts the standardized
  tokens of the frozen Stage 1 encoder on a clean crop of the same image; only patches of the clean crop whose centers lie
  inside the augmented crop contribute to the loss. 5 million draws from a curated collection of public image-editing
  datasets, used as still images only (no edit pairs or instructions).
* Still images and text only: no video, no action supervision, no teacher model other than the frozen Stage 1 copy of
  ALPS itself.

## Results (from the paper)

In a comparison with fifteen public encoders, all frozen and evaluated with the same protocols, ALPS has the lowest
aggregate LIBERO state-probe error among the ViT-B encoders (.829, relative to ALPS (Stage 1)), matching the 3.5x larger
V-JEPA 2-L; its Flickr30k image-to-text recall at 1 (93.2) is within 2 points of the best image-text encoders; and its
reconstruction l1 error (.120) is lower than that of every compared encoder not trained to reconstruct pixels or
latents. Relative to ALPS (Stage 1), cross-view refinement reduces state-probe error by 17% while changing image-text
retrieval by at most 0.5 points.

## LIBERO policies

`libero_policies/` holds the eight ALPS policies of Table 2: the LIBERO transformer policy trained by behavior cloning
on frozen ALPS tokens (50 demonstrations per task), with instructions embedded by LIBERO's BERT or by ALPS's own text
tower. Each folder, named by its suite and instruction embedding, has the run's `config.json` (only `experiment_dir` and
`experiment_name` follow the folder name) and `multitask_model_ep<N>.safetensors`, its 8.8M trainable parameters;
the frozen encoder is `alps_vitb16.safetensors` of this repository.

| Suite | Instructions | Folder | Success in Table 2 (%) |
|---|---|---|---|
| LIBERO-Long | BERT | `libero_policies/LIBERO_10/bert` | 69.7 |
| LIBERO-Goal | BERT | `libero_policies/LIBERO_GOAL/bert` | 85.1 |
| LIBERO-Spatial | BERT | `libero_policies/LIBERO_SPATIAL/bert` | 86.5 |
| LIBERO-Object | BERT | `libero_policies/LIBERO_OBJECT/bert` | 92.2 |
| LIBERO-Long | ALPS text tower | `libero_policies/LIBERO_10/alps_text` | 78.5 |
| LIBERO-Goal | ALPS text tower | `libero_policies/LIBERO_GOAL/alps_text` | 87.4 |
| LIBERO-Spatial | ALPS text tower | `libero_policies/LIBERO_SPATIAL/alps_text` | 82.6 |
| LIBERO-Object | ALPS text tower | `libero_policies/LIBERO_OBJECT/alps_text` | 93.7 |

Success is the mean of three evaluations of each policy with 50 rollouts per task. The rollouts sample their actions and
the simulator is not seeded, so a new evaluation differs from each of the paper's three about as much as they differ from
one another. To evaluate a policy, set up the patched LIBERO as described in
[`eval/libero_policy/README.md`](https://github.com/UCSC-VLAA/ALPS/tree/main/eval/libero_policy) of the code repository, then:

```bash
huggingface-cli download UCSC-VLAA/ALPS alps_vitb16.safetensors --local-dir alps_hf
huggingface-cli download UCSC-VLAA/ALPS --include "libero_policies/*" --local-dir alps_hf
export LIBERO_ROOT=<patched LIBERO checkout> ALPS_WEIGHTS=$PWD/alps_hf/alps_vitb16.safetensors \
       TASK_EMBEDDINGS_DIR=<ALPS repository>/eval/libero_policy/paper_records/raw/task_embeddings
bash <ALPS repository>/eval/libero_policy/scripts/evaluate_policy.sh \
     alps_hf/libero_policies/LIBERO_OBJECT/bert 0
```

The files were written from the training checkpoints of the runs with `scripts/export_policy_weights.py` of the code
repository (the frozen network dropped, every other tensor unchanged). For each policy, the frozen network stored in its
checkpoint equals, bit for bit, the encoder of `alps_vitb16.safetensors` with the settings that `evaluate_policy.py`
applies to these configurations.

## Intended use and limitations

Intended use: a frozen visual state for world models, robot policies, and probes; image-text retrieval and zero-shot
classification through the native text encoder; pixel reconstruction through an evaluation decoder trained on the
frozen tokens.

* Evaluated with the ViT-B/16 backbone on simulation benchmarks (LIBERO, DINO-WM tasks, Reacher); real-robot use is
  untested.
* Pretrained on still images only; fine-grained temporal actions are not its strength.
* The language interface was trained on web and synthetic captions and inherits their biases.

## License

Apache-2.0, for the weights in this repository and the code in [UCSC-VLAA/ALPS](https://github.com/UCSC-VLAA/ALPS).

## Citation

```bibtex
@article{liu2026alps,
  title   = {ALPS: Visual State Encoders for World Models},
  author  = {Liu, Yanqing and Zhang, Letian and Liu, Chengzhi and Wang, Zijun and Mei, Jieru and Fei, Hongliang and Zhou, Yuyin and Xie, Cihang},
  journal = {arXiv preprint arXiv:2610.30120},
  year    = {2026}
}
```

## Acknowledgements

We would like to thank the TPU Research Cloud (TRC) program, the Google Cloud Research Credits program, and the Google TPU Builder Award for supporting our computing needs.

	---
	license: apache-2.0
	pipeline_tag: image-feature-extraction
	tags:
	- vision
	- vision-transformer
	- world-models
	- robotics
	- representation-learning
	- image-text
	- libero
	- pytorch
	datasets:
	- mlfoundations/datacomp_1b
	- UCSC-VLAA/Recap-DataComp-1B
	---

	# 🏔️ ALPS: Visual State Encoders for World Models

	ALPS (A Language-aligned, Pixel-renderable State encoder) is a pretrained ViT-B/16 that brings physical-state
	information, native language alignment, and pixel reconstruction into one set of patch tokens. Its frozen tokens serve as
	the visual state of world models and robot policies, its native text tower relates instructions and goals stated in words
	to the scene, and an evaluation decoder renders its tokens back into images.

	[Project page](https://ucsc-vlaa.github.io/ALPS/) · [Code](https://github.com/UCSC-VLAA/ALPS) · [Paper](https://arxiv.org/abs/2610.30120)

	## Files

	\| File \| Content \| Size \|
	\|---\|---\|---\|
	\| `alps_vitb16.safetensors` \| ALPS (Stage 1 + Stage 2): the image encoder and the language interface \| 786 MB \|
	\| `alps_stage1_vitb16.safetensors` \| ALPS (Stage 1), before cross-view refinement: the image encoder and the language interface \| 786 MB \|
	\| `libero_policies/` \| the eight ALPS policies of the LIBERO experiments (Table 2 of the paper), see below \| 8 x 35 MB \|
	\| `SHA256SUMS` \| checksums of every file above \| \|

	## Use

	```bash
	git clone https://github.com/UCSC-VLAA/ALPS.git && cd ALPS
	pip install -e . # torch, numpy, pillow
	```

	```python
	import torch, alps
	from PIL import Image

	model = alps.load("alps-vitb16") # downloads alps_vitb16.safetensors from this repository
	x = alps.center_crop_transform(Image.open("image.jpg")) # [1, 3, 224, 224], normalized

	with torch.no_grad():
	out = model.encode_image(x)
	z = out["tokens"] # [1, 196, 768] raw patch tokens: the visual state
	img = model.image_embedding(tokens=z) # L2-normalized
	txt = model.text_embedding(["a robot arm opens the top drawer", "a photo of a dog"])
	probs = (model.contrastive_scale() * img @ txt.T).softmax(-1)
	```

	`alps.load("alps-stage1-vitb16")` loads ALPS (Stage 1). Simulator frames: resize the whole frame to 224
	(`alps.resize_transform`). Use float32 or bfloat16 (float16 overflows in the deeper blocks). More in the
	[repository README](https://github.com/UCSC-VLAA/ALPS).

	## Model

	* Architecture: ViT-B/16 (width 768, 12 blocks, 12 heads, MLP 3,072), class token, fixed 2-D sine-cosine position
	embedding, no patch-embedding bias, LayerNorm epsilon 1e-6. 85.6M parameters.
	* Output: 196 raw patch tokens of width 768 at 224 x 224 pixels (the visual state), without a final LayerNorm.
	* Language interface (from Stage 1, unchanged by Stage 2): a 12-layer text encoder of width 768 (109.7M parameters,
	WordPiece vocabulary, 32 tokens), linear image projection of the mean token, linear text projection, learned
	contrastive scale.

	## Training

	* Stage 1, from scratch: DataComp-1B images with their web captions and the synthetic captions of the LLaVA-LLaMA-3
	captioner of Recap-DataComp-1B. Objectives: image-text contrastive alignment, caption prediction, pixel reconstruction,
	and cross-view dense correspondence, all on the same patch tokens. 12.8 billion samples at 128 pixels, then 1.0 billion
	samples at 224 pixels.
	* Stage 2, cross-view self-distillation on single images: from an augmented crop, the student predicts the standardized
	tokens of the frozen Stage 1 encoder on a clean crop of the same image; only patches of the clean crop whose centers lie
	inside the augmented crop contribute to the loss. 5 million draws from a curated collection of public image-editing
	datasets, used as still images only (no edit pairs or instructions).
	* Still images and text only: no video, no action supervision, no teacher model other than the frozen Stage 1 copy of
	ALPS itself.

	## Results (from the paper)

	In a comparison with fifteen public encoders, all frozen and evaluated with the same protocols, ALPS has the lowest
	aggregate LIBERO state-probe error among the ViT-B encoders (.829, relative to ALPS (Stage 1)), matching the 3.5x larger
	V-JEPA 2-L; its Flickr30k image-to-text recall at 1 (93.2) is within 2 points of the best image-text encoders; and its
	reconstruction l1 error (.120) is lower than that of every compared encoder not trained to reconstruct pixels or
	latents. Relative to ALPS (Stage 1), cross-view refinement reduces state-probe error by 17% while changing image-text
	retrieval by at most 0.5 points.

	## LIBERO policies

	`libero_policies/` holds the eight ALPS policies of Table 2: the LIBERO transformer policy trained by behavior cloning
	on frozen ALPS tokens (50 demonstrations per task), with instructions embedded by LIBERO's BERT or by ALPS's own text
	tower. Each folder, named by its suite and instruction embedding, has the run's `config.json` (only `experiment_dir` and
	`experiment_name` follow the folder name) and `multitask_model_ep<N>.safetensors`, its 8.8M trainable parameters;
	the frozen encoder is `alps_vitb16.safetensors` of this repository.

	\| Suite \| Instructions \| Folder \| Success in Table 2 (%) \|
	\|---\|---\|---\|---\|
	\| LIBERO-Long \| BERT \| `libero_policies/LIBERO_10/bert` \| 69.7 \|
	\| LIBERO-Goal \| BERT \| `libero_policies/LIBERO_GOAL/bert` \| 85.1 \|
	\| LIBERO-Spatial \| BERT \| `libero_policies/LIBERO_SPATIAL/bert` \| 86.5 \|
	\| LIBERO-Object \| BERT \| `libero_policies/LIBERO_OBJECT/bert` \| 92.2 \|
	\| LIBERO-Long \| ALPS text tower \| `libero_policies/LIBERO_10/alps_text` \| 78.5 \|
	\| LIBERO-Goal \| ALPS text tower \| `libero_policies/LIBERO_GOAL/alps_text` \| 87.4 \|
	\| LIBERO-Spatial \| ALPS text tower \| `libero_policies/LIBERO_SPATIAL/alps_text` \| 82.6 \|
	\| LIBERO-Object \| ALPS text tower \| `libero_policies/LIBERO_OBJECT/alps_text` \| 93.7 \|

	Success is the mean of three evaluations of each policy with 50 rollouts per task. The rollouts sample their actions and
	the simulator is not seeded, so a new evaluation differs from each of the paper's three about as much as they differ from
	one another. To evaluate a policy, set up the patched LIBERO as described in
	[`eval/libero_policy/README.md`](https://github.com/UCSC-VLAA/ALPS/tree/main/eval/libero_policy) of the code repository, then:

	```bash
	huggingface-cli download UCSC-VLAA/ALPS alps_vitb16.safetensors --local-dir alps_hf
	huggingface-cli download UCSC-VLAA/ALPS --include "libero_policies/*" --local-dir alps_hf
	export LIBERO_ROOT=<patched LIBERO checkout> ALPS_WEIGHTS=$PWD/alps_hf/alps_vitb16.safetensors \
	TASK_EMBEDDINGS_DIR=<ALPS repository>/eval/libero_policy/paper_records/raw/task_embeddings
	bash <ALPS repository>/eval/libero_policy/scripts/evaluate_policy.sh \
	alps_hf/libero_policies/LIBERO_OBJECT/bert 0
	```

	The files were written from the training checkpoints of the runs with `scripts/export_policy_weights.py` of the code
	repository (the frozen network dropped, every other tensor unchanged). For each policy, the frozen network stored in its
	checkpoint equals, bit for bit, the encoder of `alps_vitb16.safetensors` with the settings that `evaluate_policy.py`
	applies to these configurations.

	## Intended use and limitations

	Intended use: a frozen visual state for world models, robot policies, and probes; image-text retrieval and zero-shot
	classification through the native text encoder; pixel reconstruction through an evaluation decoder trained on the
	frozen tokens.

	* Evaluated with the ViT-B/16 backbone on simulation benchmarks (LIBERO, DINO-WM tasks, Reacher); real-robot use is
	untested.
	* Pretrained on still images only; fine-grained temporal actions are not its strength.
	* The language interface was trained on web and synthetic captions and inherits their biases.

	## License

	Apache-2.0, for the weights in this repository and the code in [UCSC-VLAA/ALPS](https://github.com/UCSC-VLAA/ALPS).

	## Citation

	```bibtex
	@article{liu2026alps,
	title = {ALPS: Visual State Encoders for World Models},
	author = {Liu, Yanqing and Zhang, Letian and Liu, Chengzhi and Wang, Zijun and Mei, Jieru and Fei, Hongliang and Zhou, Yuyin and Xie, Cihang},
	journal = {arXiv preprint arXiv:2610.30120},
	year = {2026}
	}
	```

	## Acknowledgements

	We would like to thank the TPU Research Cloud (TRC) program, the Google Cloud Research Credits program, and the Google TPU Builder Award for supporting our computing needs.