Add robotics metadata, library name and paper links

#2
by nielsr HF Staff - opened
Files changed (1) hide show
  1. README.md +25 -108
README.md CHANGED
@@ -1,30 +1,34 @@
1
- # WALL-OSS
 
 
 
 
 
 
2
 
3
  <div align="left">
4
 
5
  <p align="center">
6
- <img src="assets/logo.png" width="600"/>
7
  <p>
8
 
9
  <div align="center">
10
 
11
- [![Paper](https://img.shields.io/badge/📄%20Paper-PDF-EA1B22?style=for-the-badge&logo=adobeacrobatreader&logoColor=fff)](https://x2robot.cn-wlcb.ufileos.com/wall_oss.pdf)
12
  &nbsp;&nbsp;
13
  [![Hugging Face](https://img.shields.io/badge/Hugging%20Face-x--square--robot-FFB000?style=for-the-badge&logo=huggingface&logoColor=000)](https://huggingface.co/x-square-robot)
14
  &nbsp;&nbsp;
15
  [![GitHub](https://img.shields.io/badge/GitHub-181717?style=for-the-badge&logo=github&logoColor=fff)](https://github.com/X-Square-Robot/wall-x)
16
  &nbsp;&nbsp;
17
- [![Project Page](https://img.shields.io/badge/Project-1E90FF?style=for-the-badge&logo=google-chrome&logoColor=fff)](https://x2robot.com/en/research/68bc2cde8497d7f238dde690)
18
 
19
  </div>
20
 
21
  </div>
22
 
23
- ## <a href="https://x2robot.cn-wlcb.ufileos.com/wall_oss.pdf" target="_blank"><strong>WALL-OSS: Igniting VLMs toward the Embodied Space</strong></a>
24
 
25
- We introduce **WALL-OSS**, an end-to-end embodied foundation model that leverages large-scale multimodal pretraining to achieve (1) embodiment-aware vision--language understanding, (2) strong language--action association, and (3) robust manipulation capability.
26
- Our approach employs a tightly coupled architecture and multi-strategies training curriculum that enables Unified Cross-Level CoT—seamlessly unifying instruction reasoning, subgoal decomposition, and fine-grained action synthesis within a single differentiable framework.
27
- Our results show that WALL-OSS attains high success on complex long-horizon manipulations, demonstrates strong instruction-following capabilities, complex understanding and reasoning, and outperforms strong baselines, thereby providing a reliable and scalable path from VLMs to embodied foundation models.
28
 
29
  ## 🎬 Video Demos
30
 
@@ -36,8 +40,6 @@ Our results show that WALL-OSS attains high success on complex long-horizon mani
36
  <p><strong>WALL-OSS in Action: Demonstrating advanced manipulation capabilities and embodied AI performance</strong></p>
37
  </div>
38
 
39
-
40
-
41
  ## 🚀 Quick Start
42
 
43
  ### Installation
@@ -72,119 +74,34 @@ model.eval()
72
  device = "cuda" if torch.cuda.is_available() else "cpu"
73
  model = model.to(device).bfloat16()
74
 
75
- # Your inference code here...
76
- ```
77
-
78
- ## 🎯 Supervised Fine-Tuning (SFT)
79
-
80
- For training Wall-X on your robotics datasets, please refer to our comprehensive training guide:
81
-
82
- **📖 [Training Documentation](https://github.com/X-Square-Robot/wall-x/blob/main/workspace/README.md)**
83
-
84
- The training process includes:
85
- - **Dataset Preparation**: How to prepare your robotics datasets in LeRobot format
86
- - **Configuration Setup**: Detailed configuration for GPU setup, model paths, and robot DOF settings
87
- - **Training Scripts**: Ready-to-use training scripts with proper hyperparameters
88
-
89
- ### Quick Training Start
90
-
91
- ```bash
92
- # Run training (see workspace/README.md for detailed configuration)
93
- bash ./workspace/lerobot_example/run.sh
94
- ```
95
-
96
- ## 🔮 Inference
97
-
98
- For detailed inference examples and model evaluation:
99
-
100
- **📖 [Inference Documentation](https://github.com/X-Square-Robot/wall-x/blob/main/scripts/)**
101
-
102
- ### Basic Inference Example
103
-
104
- ```python
105
- import torch
106
- from wall_x.model.qwen2_5_based.modeling_qwen2_5_vl_act import Qwen2_5_VLMoEForAction
107
-
108
- # Load model
109
- model_path = "X-Square-Robot/wall-x"
110
- model = Qwen2_5_VLMoEForAction.from_pretrained(model_path)
111
- model.eval()
112
-
113
- # Setup
114
  batch_size = 1
115
  seq_length = 50
116
- device = "cuda" if torch.cuda.is_available() else "cpu"
117
- model = model.to(device).bfloat16()
118
 
119
  # Prepare inputs (example with synthetic data)
120
- torch.manual_seed(0)
121
- input_ids = torch.randint(0, len(model.processor.tokenizer), (batch_size, seq_length), dtype=torch.long)
122
- attention_mask = torch.ones((batch_size, seq_length), dtype=torch.long)
123
- moe_token_types = torch.zeros((batch_size, seq_length), dtype=torch.long)
124
- position_ids = torch.arange(seq_length, dtype=torch.long).unsqueeze(0).expand(batch_size, -1)
125
-
126
- # Robotics-specific inputs
127
- proprioception = torch.randn((batch_size, 1, 20), dtype=torch.float32) # Joint states
128
- agent_pos_mask = torch.ones((batch_size, 1, 20), dtype=torch.float32)
129
- dof_mask = torch.ones((batch_size, 32, 20), dtype=torch.float32) # DOF mask
130
- dataset_names = ["x2_normal"]
131
-
132
- # Move to device
133
- inputs = {
134
- "input_ids": input_ids.to(device),
135
- "attention_mask": attention_mask.to(device),
136
- "moe_token_types": moe_token_types.to(device),
137
- "position_ids": position_ids.to(device),
138
- "proprioception": proprioception.to(device).bfloat16(),
139
- "agent_pos_mask": agent_pos_mask.to(device).bfloat16(),
140
- "dof_mask": dof_mask.to(device).bfloat16(),
141
- "dataset_names": dataset_names,
142
- "mode": "validate"
143
- }
144
 
145
- # Run inference
146
  with torch.no_grad():
147
- outputs = model(**inputs)
148
  print(f"Output logits shape: {outputs.logits.shape}")
149
  ```
150
 
151
- ### Advanced Inference Scripts
152
-
153
- For production-ready inference and evaluation scripts:
154
-
155
- ```bash
156
- # Basic inference test
157
- python ./scripts/fake_inference.py
158
-
159
- # Generate open-loop comparison plots
160
- python ./scripts/draw_openloop_plot.py
161
- ```
162
-
163
- **📁 [View all inference scripts](https://github.com/X-Square-Robot/wall-x/tree/main/scripts)**
164
-
165
- ## 📚 Complete Documentation
166
-
167
- For comprehensive setup, training, and inference instructions:
168
-
169
- ### 🚀 **[Visit our GitHub Repository](https://github.com/X-Square-Robot/wall-x)**
170
 
171
- The repository contains:
172
- - **Detailed Installation Guide**: Complete environment setup with all dependencies
173
- - **Training Tutorials**: Step-by-step SFT process with LeRobot datasets
174
- - **Inference Examples**: Multiple inference scripts and evaluation tools
175
- - **Configuration Templates**: Ready-to-use configs for different robot setups
176
- - **Troubleshooting Guide**: Common issues and solutions
177
 
178
- ## 📄 Cite Us
179
 
180
  If you find WALL-OSS models useful, please cite:
181
 
182
  ```bibtex
183
  @misc{walloss_paper_2025,
184
- title = {WALL-OSS: Igniting VLMs toward the Embodied Space},
185
  author = {X Square Robot},
186
- year = {2025},
187
- howpublished = {\url{https://x2robot.cn-wlcb.ufileos.com/wall_oss.pdf}},
188
- note = {White paper}
189
  }
190
- ```
 
1
+ ---
2
+ license: apache-2.0
3
+ library_name: lerobot
4
+ pipeline_tag: robotics
5
+ ---
6
+
7
+ # WALL-OSS: Wall-OSS-0.5
8
 
9
  <div align="left">
10
 
11
  <p align="center">
12
+ <img src="https://huggingface.co/X-Square-Robot/wall-oss-flow/resolve/main/assets/logo.png" width="600"/>
13
  <p>
14
 
15
  <div align="center">
16
 
17
+ [![Paper](https://img.shields.io/badge/📄%20Paper-PDF-EA1B22?style=for-the-badge&logo=adobeacrobatreader&logoColor=fff)](https://huggingface.co/papers/2605.30877)
18
  &nbsp;&nbsp;
19
  [![Hugging Face](https://img.shields.io/badge/Hugging%20Face-x--square--robot-FFB000?style=for-the-badge&logo=huggingface&logoColor=000)](https://huggingface.co/x-square-robot)
20
  &nbsp;&nbsp;
21
  [![GitHub](https://img.shields.io/badge/GitHub-181717?style=for-the-badge&logo=github&logoColor=fff)](https://github.com/X-Square-Robot/wall-x)
22
  &nbsp;&nbsp;
23
+ [![Project Page](https://img.shields.io/badge/Project-1E90FF?style=for-the-badge&logo=google-chrome&logoColor=fff)](https://x2robot.com/en/oss)
24
 
25
  </div>
26
 
27
  </div>
28
 
29
+ Wall-OSS-0.5 is an open-source 4B Vision-Language-Action (VLA) model built upon a 3B VLM backbone augmented with action-generation components. It is designed to provide executable robot behavior directly from pretraining. The model was introduced in the [Wall-OSS-0.5 Technical Report](https://huggingface.co/papers/2605.30877).
30
 
31
+ Before task-specific fine-tuning, Wall-OSS-0.5 achieves non-trivial zero-shot real-robot behavior. After fine-tuning, it serves as a strong adaptation prior, significantly outperforming previous baselines in manipulation tasks.
 
 
32
 
33
  ## 🎬 Video Demos
34
 
 
40
  <p><strong>WALL-OSS in Action: Demonstrating advanced manipulation capabilities and embodied AI performance</strong></p>
41
  </div>
42
 
 
 
43
  ## 🚀 Quick Start
44
 
45
  ### Installation
 
74
  device = "cuda" if torch.cuda.is_available() else "cpu"
75
  model = model.to(device).bfloat16()
76
 
77
+ # Basic inference example
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
78
  batch_size = 1
79
  seq_length = 50
 
 
80
 
81
  # Prepare inputs (example with synthetic data)
82
+ input_ids = torch.randint(0, len(model.processor.tokenizer), (batch_size, seq_length), dtype=torch.long).to(device)
83
+ attention_mask = torch.ones((batch_size, seq_length), dtype=torch.long).to(device)
84
+ proprioception = torch.randn((batch_size, 1, 20), dtype=torch.float32).to(device).bfloat16()
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
85
 
 
86
  with torch.no_grad():
87
+ outputs = model(input_ids=input_ids, attention_mask=attention_mask, proprioception=proprioception, mode="validate")
88
  print(f"Output logits shape: {outputs.logits.shape}")
89
  ```
90
 
91
+ ## 🎯 Supervised Fine-Tuning (SFT)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
92
 
93
+ For training Wall-X on your robotics datasets using the [LeRobot](https://github.com/huggingface/lerobot) format, please refer to the [Training Documentation](https://github.com/X-Square-Robot/wall-x/blob/main/workspace/README.md).
 
 
 
 
 
94
 
95
+ ## 📄 Citation
96
 
97
  If you find WALL-OSS models useful, please cite:
98
 
99
  ```bibtex
100
  @misc{walloss_paper_2025,
101
+ title = {Wall-OSS-0.5 Technical Report},
102
  author = {X Square Robot},
103
+ year = {2026},
104
+ howpublished = {\url{https://huggingface.co/papers/2605.30877}},
105
+ note = {Technical Report}
106
  }
107
+ ```