Add WNM-3D model card
Browse files
README.md
CHANGED
|
@@ -1,3 +1,107 @@
|
|
| 1 |
---
|
| 2 |
license: fair-noncommercial-research-license
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
license: fair-noncommercial-research-license
|
| 3 |
+
tags:
|
| 4 |
+
- robotics
|
| 5 |
+
- vision-language-navigation
|
| 6 |
+
- embodied-ai
|
| 7 |
+
- world-model
|
| 8 |
+
- video-generation
|
| 9 |
+
- pytorch
|
| 10 |
---
|
| 11 |
+
|
| 12 |
+
# WNM-3D
|
| 13 |
+
|
| 14 |
+
WNM-3D is a generative world navigation model for continuous
|
| 15 |
+
vision-language navigation. It converts monocular egocentric RGB history into
|
| 16 |
+
persistent geometry-aware scene tokens and jointly generates future views and
|
| 17 |
+
navigation actions for closed-loop control.
|
| 18 |
+
|
| 19 |
+
- [Project page](https://wnm-3d.github.io/)
|
| 20 |
+
- [Paper](https://arxiv.org/abs/2608.07267)
|
| 21 |
+
- [Source code and documentation](https://github.com/TeleHuman/WNM-3D)
|
| 22 |
+
- [GN0 / GN-Bench](https://github.com/TeleHuman/GN0)
|
| 23 |
+
- [GN-Matrix dataset](https://huggingface.co/datasets/TeleEmbodied/GN-Matrix)
|
| 24 |
+
|
| 25 |
+
## Released Checkpoints
|
| 26 |
+
|
| 27 |
+
| Directory | Training stage | Recommended use |
|
| 28 |
+
| --- | --- | --- |
|
| 29 |
+
| `wnm_3d_stage1_release` | Offline A\* SFT | Stage-I analysis and initialization |
|
| 30 |
+
| `wnm_3d_stage2_release` | Closed-loop DAgger-SFT | Stage-II analysis and initialization |
|
| 31 |
+
| `wnm_3d_stage3_release` | Counterfactual DanceGRPO | Evaluation and inference |
|
| 32 |
+
|
| 33 |
+
The Stage-III checkpoint is the primary released policy. Each directory is a
|
| 34 |
+
self-contained inference checkpoint; standalone Wan, UMT5-XXL, and VGGT-Ω
|
| 35 |
+
initialization weights are not required for evaluation.
|
| 36 |
+
|
| 37 |
+
## Download
|
| 38 |
+
|
| 39 |
+
```bash
|
| 40 |
+
hf download TeleEmbodied/WNM-3D \
|
| 41 |
+
--include "wnm_3d_stage3_release/**" \
|
| 42 |
+
--local-dir checkpoints
|
| 43 |
+
```
|
| 44 |
+
|
| 45 |
+
## Inference
|
| 46 |
+
|
| 47 |
+
Install WNM-3D by following the
|
| 48 |
+
[installation guide](https://github.com/TeleHuman/WNM-3D/blob/main/docs/INSTALLATION.md),
|
| 49 |
+
then launch a single policy replica:
|
| 50 |
+
|
| 51 |
+
```bash
|
| 52 |
+
bash scripts/inference/wnm_3d_server.sh \
|
| 53 |
+
--model-path checkpoints/wnm_3d_stage3_release \
|
| 54 |
+
--cuda-devices 0 \
|
| 55 |
+
--num-replicas 1 \
|
| 56 |
+
--base-port 8000
|
| 57 |
+
```
|
| 58 |
+
|
| 59 |
+
Run the GN-Bench client from a sibling GN0 checkout:
|
| 60 |
+
|
| 61 |
+
```bash
|
| 62 |
+
cd ../GN0
|
| 63 |
+
bash scripts/evaluation/eval_remote.sh \
|
| 64 |
+
--exp-config configs/gn_bench/interiorgs/test_unseen.yaml \
|
| 65 |
+
--enable-stall-recovery \
|
| 66 |
+
--num-gpus 1 \
|
| 67 |
+
--result-dir tmp/eval/wnm_3d_test_unseen
|
| 68 |
+
```
|
| 69 |
+
|
| 70 |
+
The reference single-replica deployment was validated on an NVIDIA H100 80 GB
|
| 71 |
+
GPU and used approximately 27 GiB of GPU memory after loading. A released
|
| 72 |
+
checkpoint occupies approximately 26 GiB. These figures are observations from
|
| 73 |
+
the reference configuration, not strict minimum requirements.
|
| 74 |
+
|
| 75 |
+
## Intended Use and Limitations
|
| 76 |
+
|
| 77 |
+
WNM-3D is intended for research on embodied navigation, world models, and
|
| 78 |
+
closed-loop vision-language navigation. The released policy was developed and
|
| 79 |
+
evaluated with the InteriorGS scenes and GN-Matrix task annotations used by
|
| 80 |
+
GN-Bench. Performance may not transfer to new simulators, sensors, scene
|
| 81 |
+
distributions, languages, or physical robots without additional validation.
|
| 82 |
+
|
| 83 |
+
Generated actions can fail or behave unexpectedly. Do not use the model as a
|
| 84 |
+
safety-critical controller or deploy it around people, property, or physical
|
| 85 |
+
systems without appropriate safeguards and human oversight.
|
| 86 |
+
|
| 87 |
+
## License
|
| 88 |
+
|
| 89 |
+
The model repository is published under the FAIR Noncommercial Research
|
| 90 |
+
License shown in its Hugging Face metadata. The WNM-3D source repository
|
| 91 |
+
contains both Apache-2.0 portions and vendored components governed by separate
|
| 92 |
+
terms, including the FAIR Noncommercial Research License and DINOv3 License.
|
| 93 |
+
Review the complete
|
| 94 |
+
[license boundaries and notices](https://github.com/TeleHuman/WNM-3D/blob/main/docs/THIRD_PARTY_NOTICES.md)
|
| 95 |
+
before use or redistribution. Datasets and upstream initialization weights are
|
| 96 |
+
subject to their providers' terms.
|
| 97 |
+
|
| 98 |
+
## Citation
|
| 99 |
+
|
| 100 |
+
```bibtex
|
| 101 |
+
@article{huang2026wnm_3d,
|
| 102 |
+
title={WNM-3D: A World Navigation Model with 3D Scene Conditioning for Closed-Loop VLN},
|
| 103 |
+
author={Huang, Yuehao and Wu, Yunzi and Zhang, Xiaotao and Li, Xinhai and Dong, Jiankun and Lv, Jiajun and Zhang, Chi and Bai, Chenjia and Liu, Yong and Li, Xuelong},
|
| 104 |
+
journal={arXiv preprint arXiv:2608.07267},
|
| 105 |
+
year={2026}
|
| 106 |
+
}
|
| 107 |
+
```
|