Add model card
#1
by nielsr HF Staff - opened
README.md
ADDED
|
@@ -0,0 +1,77 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: cc-by-nc-sa-4.0
|
| 3 |
+
pipeline_tag: robotics
|
| 4 |
+
---
|
| 5 |
+
|
| 6 |
+
# LWM — Latent World Model for Goal-Conditioned Navigation
|
| 7 |
+
|
| 8 |
+
This repository contains the pretrained weights for the latent world model (LWM) and the AR+ policy described in [Predicting Consequences and Reinforcing Navigation Policies with Latent World Models](https://huggingface.co/papers/2608.26190) (ECCV 2026 Spotlight).
|
| 9 |
+
|
| 10 |
+
**Project page:** https://wzm206.github.io/latent-world-model-nav
|
| 11 |
+
**Code:** https://github.com/wzm206/LWM
|
| 12 |
+
|
| 13 |
+
The system consists of two components:
|
| 14 |
+
- **World model** (`LatentWorldModel`): fuses current and goal observations, scores candidate action trajectories by predicted distance to the goal.
|
| 15 |
+
- **Policy** (`ARPlusPolicy`): auto-regressively samples candidate trajectories, which the world model ranks.
|
| 16 |
+
|
| 17 |
+
## Weights
|
| 18 |
+
|
| 19 |
+
Download the checkpoints:
|
| 20 |
+
|
| 21 |
+
```bash
|
| 22 |
+
hf download wzm206/LWM world_model.pt ar_plus.pt --local-dir weights/
|
| 23 |
+
```
|
| 24 |
+
|
| 25 |
+
## Usage
|
| 26 |
+
|
| 27 |
+
### Command line
|
| 28 |
+
|
| 29 |
+
```bash
|
| 30 |
+
python inference.py --now-img path/to/now.jpg --goal-img path/to/goal.jpg
|
| 31 |
+
```
|
| 32 |
+
|
| 33 |
+
### Python API
|
| 34 |
+
|
| 35 |
+
```python
|
| 36 |
+
import torch
|
| 37 |
+
from PIL import Image
|
| 38 |
+
|
| 39 |
+
from lwm import ActionTokenizer, ARPlusPolicy, LatentWorldModel
|
| 40 |
+
from lwm.preprocess import get_image_transform, load_kmeans_trajectories
|
| 41 |
+
|
| 42 |
+
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
|
| 43 |
+
transform = get_image_transform()
|
| 44 |
+
|
| 45 |
+
# Images must be 224x224 RGB, ImageNet-normalized.
|
| 46 |
+
img_now = transform(Image.open("now.jpg").convert("RGB")).unsqueeze(0).to(device)
|
| 47 |
+
img_obj = transform(Image.open("goal.jpg").convert("RGB")).unsqueeze(0).to(device)
|
| 48 |
+
|
| 49 |
+
# World model
|
| 50 |
+
wm = LatentWorldModel()
|
| 51 |
+
wm.load_state_dict(torch.load("weights/world_model.pt", map_location="cpu"))
|
| 52 |
+
wm.to(device).eval()
|
| 53 |
+
|
| 54 |
+
codebook = load_kmeans_trajectories("tokenizer/zed2_traj.json").to(device)
|
| 55 |
+
kmeans_idx, point_idx = wm.eval_wm(img_now, img_obj, codebook)
|
| 56 |
+
|
| 57 |
+
# Policy
|
| 58 |
+
policy = ARPlusPolicy()
|
| 59 |
+
policy.load_state_dict(torch.load("weights/ar_plus.pt", map_location="cpu"))
|
| 60 |
+
policy.to(device).eval()
|
| 61 |
+
|
| 62 |
+
tokenizer = ActionTokenizer("tokenizer/zed2_64.json")
|
| 63 |
+
token_seqs = policy.roll_out(img_now, img_obj, num_sample=32, temperature=1.0) # (1, 32, 65)
|
| 64 |
+
actions, lengths = tokenizer.batch_decode(token_seqs.view(32, 65)) # (32, 63, 3)
|
| 65 |
+
reward = wm.get_reward(img_now, img_obj, 0.1 * actions.view(1, 32, 63, 3), lengths)
|
| 66 |
+
best = reward[0].argmax().item()
|
| 67 |
+
```
|
| 68 |
+
|
| 69 |
+
## Input / Output Conventions
|
| 70 |
+
|
| 71 |
+
- **Images**: RGB, 224×224, ImageNet-normalized.
|
| 72 |
+
- **Actions**: `(x, y, yaw)` in normalized units (multiply by 10 to get meters/radians).
|
| 73 |
+
- Trajectories have up to 63 steps; the world model outputs predicted distance-to-goal (lower is better).
|
| 74 |
+
|
| 75 |
+
## License
|
| 76 |
+
|
| 77 |
+
The CroCo backbone (`lwm/croco/`) is derived from [CroCo](https://github.com/naver/croco) and is licensed under **CC BY-NC-SA 4.0** (non-commercial). The released checkpoints are for research / non-commercial use.
|