README.md

---
license: apache-2.0

tags:
  - humanoid-robotics
  - loco-manipulation
  - world-action-model
  - embodied-ai
  - robot-learning
  - vision-language-action
  - diffusion
  - latent-world-model
  - unitree-g1
---

# ω-0: A Latent Predictive World Action Model for Concurrent Humanoid Loco-Manipulation

**ω-0 (OMEGA-0)** is a latent predictive world action model for real-world humanoid **concurrent loco-manipulation**.

Given a language instruction, visual observations, and robot proprioceptive state, ω-0 predicts **controller-compatible whole-body action latents** for humanoid robot execution. Instead of reconstructing future RGB videos in pixel space, ω-0 predicts compact future visual representations and couples them with diffusion-based whole-body action generation.

The released checkpoint is intended to be used with the official OMEGA-0 codebase for inference and real-robot deployment.

> **Paper:** *ω-0: A Latent Predictive World Action Model for Concurrent Humanoid Loco-Manipulation*  
> **Code:** https://github.com/gentlefress/OMEGA-0  
> **Project Page:** https://gentlefress.github.io/OMEGA-0_page/  
> **Dataset:** https://huggingface.co/datasets/keycharon/omega-HOME

---

## 🌟 Highlights

- **Whole-body loco-manipulation:** jointly coordinates locomotion, posture, balance, and manipulation.
- **Latent predictive modeling:** predicts future visual representations rather than reconstructing future frames in pixel space.
- **Action generation:** generates whole-body action latents compatible with the SONIC controller.
- **Multimodal conditioning:** supports language, visual observations, and robot state.
- **Real-world humanoid deployment:** demonstrated on the Unitree G1.
- **Multiple visual viewpoints:** supports egocentric RGB, exocentric RGB, and exocentric depth.
- **Single unified policy:** one model is used across a diverse set of household loco-manipulation tasks.
- **Open-source workflow:** the official repository provides training, inference, teleoperation, recording, and deployment code.

---

## 🧠 Model Overview

ω-0 follows a simple principle:

> **Predict how the scene will evolve, while directly generating how the whole body should act.**

The model combines three complementary components:

```text
Language Instruction
        │
        ├──────────────┐
        │              │
Visual Observation   Robot State
        │              │
        └──────┬───────┘
               ↓
        Multimodal Encoder
               │
               ↓
     ┌─────────────────────┐
     │   ω-0 WAM Predictor │
     └─────────────────────┘
          │           │
          │           └────────→ Future Visual Latent
          │
          └────────────────────→ Whole-body Action Latent
                                      │
                                      ↓
                                   SONIC
                                      │
                                      ↓
                                Unitree G1
```

The predictive branch provides a compact representation of future scene evolution, while the action branch generates controller-compatible whole-body actions.

---

## 📥 Inputs

Depending on the configured model and deployment setup, ω-0 can use:

### Language

A natural-language task instruction, for example:

```text
"Pick up the clothes from the washing machine."
```

### Visual Observation

Supported observation modes include:

- Egocentric RGB
- Exocentric RGB
- Exocentric depth

### Robot Proprioception

Robot state information can be used as specified by the corresponding configuration.

---

## 📤 Outputs

The model predicts **whole-body action latents**, which are subsequently decoded/interpreted by the OMEGA-0 action pipeline and executed through the **SONIC** whole-body controller.

The checkpoint therefore should **not** be interpreted as a standalone joint-position predictor.

The complete execution pipeline is:

```text
Instruction + Observation + Robot State
                    ↓
                  ω-0
                    ↓
          Whole-body Action Latent
                    ↓
              Action Decoder
                    ↓
                  SONIC
                    ↓
               Unitree G1
```

---

## 🤖 Real-World Platform

The released system targets the following experimental setup:

- **Robot:** Unitree G1
- **Hands:** Inspire dexterous hands
- **Controller:** SONIC
- **Egocentric camera:** ZED Mini
- **Teleoperation:** PICO body tracking through XRoboToolkit

Hardware and software configurations may need to be adapted to your own setup.

---

## 🚀 Quick Start

### 1. Clone the official repository

```bash
git clone https://github.com/gentlefress/OMEGA-0.git
cd OMEGA-0
```

### 2. Install the environment

The official repository uses `uv`:

```bash
uv sync --locked --extra train --extra deploy --python 3.10 --inexact
source .venv/bin/activate

uv pip install flash-attn --no-build-isolation
```

Verify the installation:

```bash
python -c "import omega, torch; print('PyTorch:', torch.__version__); print('CUDA:', torch.cuda.is_available())"
```

### 3. Download the checkpoint

Download the checkpoint from this Hugging Face repository and place it in your local model directory.

For example:

```text
/path/to/omega0_checkpoint/
```

### 4. Configure the inference server

The official OMEGA-0 repository provides an inference configuration under:

```text
src/configs/serve.yaml
```

For an exported checkpoint, configure the model artifact:

```yaml
model:
  artifact: /path/to/omega0_checkpoint
  device: cuda
```

Then validate the configuration:

```bash
python -m omega.inference.run --config src/configs/serve.yaml --check
```

Launch the inference server:

```bash
OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=4 \
HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 \
python -m omega.inference.run --config src/configs/serve.yaml
```

The default inference server listens on:

```text
127.0.0.1:8014
```

For multi-machine deployment, configure the server and client addresses according to your network setup.

---

## 🦾 Robot Deployment

The official repository provides a robot client for Unitree G1 deployment.

Set the inference endpoint and task instruction:

```bash
export INFERENCE_ENDPOINT=http://127.0.0.1:8014
export INSTRUCTION="Pick up the clothes from the washing machine."
```

Then run:

```bash
python -m omega_real.run --config real/configs/deploy.yaml --check
python -m omega_real.run --config real/configs/deploy.yaml --arm
```

Please follow the official deployment instructions and safety procedures before enabling robot commands.

---

## 🏋️ Training

The released codebase contains two main training stages:

### Stage 1 — Action-Token Pretraining

The model first learns to associate language and visual observations with whole-body motion tokens.

```text
Language + Visual Observation
              ↓
           Qwen3-VL
              ↓
       Whole-body motion tokens
```

### Stage 2 — WAM Fine-Tuning

The model is further optimized to jointly predict future visual representations and whole-body action latents.

```text
Language + Observation + State
              ↓
           ω-0 WAM
        ↙             ↘
Future Visual      Action Latent
Representation
                      ↓
                    SONIC
```

Training and dataset preparation instructions are available in the official repository.

---

## 📊 ω-HOME Dataset

ω-0 is accompanied by **ω-HOME**, a 40+ hour real-world household humanoid dataset containing synchronized multimodal observations, whole-body motion, robot states, and action latents.

The dataset is available on Hugging Face:

https://huggingface.co/datasets/keycharon/omega-HOME

---

## 🎯 Intended Use

This checkpoint is intended for:

- Research on humanoid robot loco-manipulation
- World action models
- Whole-body robot learning
- Vision-language-action modeling
- Latent predictive modeling
- Real-world humanoid manipulation
- Reproduction and extension of the ω-0 experiments

The checkpoint is primarily a **research model** and is not intended for unsupervised deployment in safety-critical environments.

---

## ⚠️ Limitations and Safety

ω-0 is a research system and should be deployed with appropriate hardware-level and software-level safety mechanisms.

In particular:

- Real-robot behavior depends on the underlying robot, controller, camera, calibration, and environment.
- The released model is developed and evaluated primarily on the Unitree G1 platform.
- Performance may degrade under substantial distribution shifts in scenes, objects, embodiments, or tasks.
- Generated actions should be checked by the downstream controller and deployment stack.
- Do not operate the robot near people without appropriate safety precautions.
- The model should not be treated as a guaranteed safe or autonomous control policy.

---

## 📚 Citation

If you use ω-0 or this checkpoint in your research, please cite:

```bibtex
@article{li2026omega0,
  title   = {{$\omega$-0: A Latent Predictive World Action Model for Concurrent Humanoid Loco-Manipulation}},
  author  = {Zhe Li and Zhenzhe Zhang and Yangyang Wei and Wenjie Zhang and
             Xichen Yuan and Peiyuan Zhi and Gen Li and Xinying Guo and
             Fengjie Gao and Jianfei Yang and Shanghang Zhang},
  journal = {arXiv preprint arXiv:2608.06375},
  year    = {2026},
  doi     = {10.48550/arXiv.2608.06375},
  url     = {https://arxiv.org/abs/2608.06375}
}
```

---

## 🙏 Acknowledgements

We acknowledge the open-source projects that contributed to the development of OMEGA-0, including:

- Ψ₀
- GR00T-WholeBodyControl
- Qwen3-VL
- V-JEPA
- Wan

Please refer to the official repository and the corresponding third-party licenses for the exact dependencies and attribution requirements.

---

## 📄 License

OMEGA-0 code is released under the **Apache 2.0 License**. Third-party code, models, and assets remain subject to their respective licenses.

Please check the license terms of all external components before redistribution or commercial use.