Add model card

#1
by nielsr HF Staff - opened
Files changed (1) hide show
  1. README.md +77 -0
README.md ADDED
@@ -0,0 +1,77 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: cc-by-nc-sa-4.0
3
+ pipeline_tag: robotics
4
+ ---
5
+
6
+ # LWM — Latent World Model for Goal-Conditioned Navigation
7
+
8
+ This repository contains the pretrained weights for the latent world model (LWM) and the AR+ policy described in [Predicting Consequences and Reinforcing Navigation Policies with Latent World Models](https://huggingface.co/papers/2608.26190) (ECCV 2026 Spotlight).
9
+
10
+ **Project page:** https://wzm206.github.io/latent-world-model-nav
11
+ **Code:** https://github.com/wzm206/LWM
12
+
13
+ The system consists of two components:
14
+ - **World model** (`LatentWorldModel`): fuses current and goal observations, scores candidate action trajectories by predicted distance to the goal.
15
+ - **Policy** (`ARPlusPolicy`): auto-regressively samples candidate trajectories, which the world model ranks.
16
+
17
+ ## Weights
18
+
19
+ Download the checkpoints:
20
+
21
+ ```bash
22
+ hf download wzm206/LWM world_model.pt ar_plus.pt --local-dir weights/
23
+ ```
24
+
25
+ ## Usage
26
+
27
+ ### Command line
28
+
29
+ ```bash
30
+ python inference.py --now-img path/to/now.jpg --goal-img path/to/goal.jpg
31
+ ```
32
+
33
+ ### Python API
34
+
35
+ ```python
36
+ import torch
37
+ from PIL import Image
38
+
39
+ from lwm import ActionTokenizer, ARPlusPolicy, LatentWorldModel
40
+ from lwm.preprocess import get_image_transform, load_kmeans_trajectories
41
+
42
+ device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
43
+ transform = get_image_transform()
44
+
45
+ # Images must be 224x224 RGB, ImageNet-normalized.
46
+ img_now = transform(Image.open("now.jpg").convert("RGB")).unsqueeze(0).to(device)
47
+ img_obj = transform(Image.open("goal.jpg").convert("RGB")).unsqueeze(0).to(device)
48
+
49
+ # World model
50
+ wm = LatentWorldModel()
51
+ wm.load_state_dict(torch.load("weights/world_model.pt", map_location="cpu"))
52
+ wm.to(device).eval()
53
+
54
+ codebook = load_kmeans_trajectories("tokenizer/zed2_traj.json").to(device)
55
+ kmeans_idx, point_idx = wm.eval_wm(img_now, img_obj, codebook)
56
+
57
+ # Policy
58
+ policy = ARPlusPolicy()
59
+ policy.load_state_dict(torch.load("weights/ar_plus.pt", map_location="cpu"))
60
+ policy.to(device).eval()
61
+
62
+ tokenizer = ActionTokenizer("tokenizer/zed2_64.json")
63
+ token_seqs = policy.roll_out(img_now, img_obj, num_sample=32, temperature=1.0) # (1, 32, 65)
64
+ actions, lengths = tokenizer.batch_decode(token_seqs.view(32, 65)) # (32, 63, 3)
65
+ reward = wm.get_reward(img_now, img_obj, 0.1 * actions.view(1, 32, 63, 3), lengths)
66
+ best = reward[0].argmax().item()
67
+ ```
68
+
69
+ ## Input / Output Conventions
70
+
71
+ - **Images**: RGB, 224×224, ImageNet-normalized.
72
+ - **Actions**: `(x, y, yaw)` in normalized units (multiply by 10 to get meters/radians).
73
+ - Trajectories have up to 63 steps; the world model outputs predicted distance-to-goal (lower is better).
74
+
75
+ ## License
76
+
77
+ The CroCo backbone (`lwm/croco/`) is derived from [CroCo](https://github.com/naver/croco) and is licensed under **CC BY-NC-SA 4.0** (non-commercial). The released checkpoints are for research / non-commercial use.