--- license: apache-2.0 tags: - humanoid-robotics - loco-manipulation - world-action-model - embodied-ai - robot-learning - vision-language-action - diffusion - latent-world-model - unitree-g1 --- # ω-0: A Latent Predictive World Action Model for Concurrent Humanoid Loco-Manipulation **ω-0 (OMEGA-0)** is a latent predictive world action model for real-world humanoid **concurrent loco-manipulation**. Given a language instruction, visual observations, and robot proprioceptive state, ω-0 predicts **controller-compatible whole-body action latents** for humanoid robot execution. Instead of reconstructing future RGB videos in pixel space, ω-0 predicts compact future visual representations and couples them with diffusion-based whole-body action generation. The released checkpoint is intended to be used with the official OMEGA-0 codebase for inference and real-robot deployment. > **Paper:** *ω-0: A Latent Predictive World Action Model for Concurrent Humanoid Loco-Manipulation* > **Code:** https://github.com/gentlefress/OMEGA-0 > **Project Page:** https://gentlefress.github.io/OMEGA-0_page/ > **Dataset:** https://huggingface.co/datasets/keycharon/omega-HOME --- ## 🌟 Highlights - **Whole-body loco-manipulation:** jointly coordinates locomotion, posture, balance, and manipulation. - **Latent predictive modeling:** predicts future visual representations rather than reconstructing future frames in pixel space. - **Action generation:** generates whole-body action latents compatible with the SONIC controller. - **Multimodal conditioning:** supports language, visual observations, and robot state. - **Real-world humanoid deployment:** demonstrated on the Unitree G1. - **Multiple visual viewpoints:** supports egocentric RGB, exocentric RGB, and exocentric depth. - **Single unified policy:** one model is used across a diverse set of household loco-manipulation tasks. - **Open-source workflow:** the official repository provides training, inference, teleoperation, recording, and deployment code. --- ## 🧠 Model Overview ω-0 follows a simple principle: > **Predict how the scene will evolve, while directly generating how the whole body should act.** The model combines three complementary components: ```text Language Instruction │ ├──────────────┐ │ │ Visual Observation Robot State │ │ └──────┬───────┘ ↓ Multimodal Encoder │ ↓ ┌─────────────────────┐ │ ω-0 WAM Predictor │ └─────────────────────┘ │ │ │ └────────→ Future Visual Latent │ └────────────────────→ Whole-body Action Latent │ ↓ SONIC │ ↓ Unitree G1 ``` The predictive branch provides a compact representation of future scene evolution, while the action branch generates controller-compatible whole-body actions. --- ## 📥 Inputs Depending on the configured model and deployment setup, ω-0 can use: ### Language A natural-language task instruction, for example: ```text "Pick up the clothes from the washing machine." ``` ### Visual Observation Supported observation modes include: - Egocentric RGB - Exocentric RGB - Exocentric depth ### Robot Proprioception Robot state information can be used as specified by the corresponding configuration. --- ## 📤 Outputs The model predicts **whole-body action latents**, which are subsequently decoded/interpreted by the OMEGA-0 action pipeline and executed through the **SONIC** whole-body controller. The checkpoint therefore should **not** be interpreted as a standalone joint-position predictor. The complete execution pipeline is: ```text Instruction + Observation + Robot State ↓ ω-0 ↓ Whole-body Action Latent ↓ Action Decoder ↓ SONIC ↓ Unitree G1 ``` --- ## 🤖 Real-World Platform The released system targets the following experimental setup: - **Robot:** Unitree G1 - **Hands:** Inspire dexterous hands - **Controller:** SONIC - **Egocentric camera:** ZED Mini - **Teleoperation:** PICO body tracking through XRoboToolkit Hardware and software configurations may need to be adapted to your own setup. --- ## 🚀 Quick Start ### 1. Clone the official repository ```bash git clone https://github.com/gentlefress/OMEGA-0.git cd OMEGA-0 ``` ### 2. Install the environment The official repository uses `uv`: ```bash uv sync --locked --extra train --extra deploy --python 3.10 --inexact source .venv/bin/activate uv pip install flash-attn --no-build-isolation ``` Verify the installation: ```bash python -c "import omega, torch; print('PyTorch:', torch.__version__); print('CUDA:', torch.cuda.is_available())" ``` ### 3. Download the checkpoint Download the checkpoint from this Hugging Face repository and place it in your local model directory. For example: ```text /path/to/omega0_checkpoint/ ``` ### 4. Configure the inference server The official OMEGA-0 repository provides an inference configuration under: ```text src/configs/serve.yaml ``` For an exported checkpoint, configure the model artifact: ```yaml model: artifact: /path/to/omega0_checkpoint device: cuda ``` Then validate the configuration: ```bash python -m omega.inference.run --config src/configs/serve.yaml --check ``` Launch the inference server: ```bash OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=4 \ HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 \ python -m omega.inference.run --config src/configs/serve.yaml ``` The default inference server listens on: ```text 127.0.0.1:8014 ``` For multi-machine deployment, configure the server and client addresses according to your network setup. --- ## 🦾 Robot Deployment The official repository provides a robot client for Unitree G1 deployment. Set the inference endpoint and task instruction: ```bash export INFERENCE_ENDPOINT=http://127.0.0.1:8014 export INSTRUCTION="Pick up the clothes from the washing machine." ``` Then run: ```bash python -m omega_real.run --config real/configs/deploy.yaml --check python -m omega_real.run --config real/configs/deploy.yaml --arm ``` Please follow the official deployment instructions and safety procedures before enabling robot commands. --- ## 🏋️ Training The released codebase contains two main training stages: ### Stage 1 — Action-Token Pretraining The model first learns to associate language and visual observations with whole-body motion tokens. ```text Language + Visual Observation ↓ Qwen3-VL ↓ Whole-body motion tokens ``` ### Stage 2 — WAM Fine-Tuning The model is further optimized to jointly predict future visual representations and whole-body action latents. ```text Language + Observation + State ↓ ω-0 WAM ↙ ↘ Future Visual Action Latent Representation ↓ SONIC ``` Training and dataset preparation instructions are available in the official repository. --- ## 📊 ω-HOME Dataset ω-0 is accompanied by **ω-HOME**, a 40+ hour real-world household humanoid dataset containing synchronized multimodal observations, whole-body motion, robot states, and action latents. The dataset is available on Hugging Face: https://huggingface.co/datasets/keycharon/omega-HOME --- ## 🎯 Intended Use This checkpoint is intended for: - Research on humanoid robot loco-manipulation - World action models - Whole-body robot learning - Vision-language-action modeling - Latent predictive modeling - Real-world humanoid manipulation - Reproduction and extension of the ω-0 experiments The checkpoint is primarily a **research model** and is not intended for unsupervised deployment in safety-critical environments. --- ## ⚠️ Limitations and Safety ω-0 is a research system and should be deployed with appropriate hardware-level and software-level safety mechanisms. In particular: - Real-robot behavior depends on the underlying robot, controller, camera, calibration, and environment. - The released model is developed and evaluated primarily on the Unitree G1 platform. - Performance may degrade under substantial distribution shifts in scenes, objects, embodiments, or tasks. - Generated actions should be checked by the downstream controller and deployment stack. - Do not operate the robot near people without appropriate safety precautions. - The model should not be treated as a guaranteed safe or autonomous control policy. --- ## 📚 Citation If you use ω-0 or this checkpoint in your research, please cite: ```bibtex @article{li2026omega0, title = {{$\omega$-0: A Latent Predictive World Action Model for Concurrent Humanoid Loco-Manipulation}}, author = {Zhe Li and Zhenzhe Zhang and Yangyang Wei and Wenjie Zhang and Xichen Yuan and Peiyuan Zhi and Gen Li and Xinying Guo and Fengjie Gao and Jianfei Yang and Shanghang Zhang}, journal = {arXiv preprint arXiv:2608.06375}, year = {2026}, doi = {10.48550/arXiv.2608.06375}, url = {https://arxiv.org/abs/2608.06375} } ``` --- ## 🙏 Acknowledgements We acknowledge the open-source projects that contributed to the development of OMEGA-0, including: - Ψ₀ - GR00T-WholeBodyControl - Qwen3-VL - V-JEPA - Wan Please refer to the official repository and the corresponding third-party licenses for the exact dependencies and attribution requirements. --- ## 📄 License OMEGA-0 code is released under the **Apache 2.0 License**. Third-party code, models, and assets remain subject to their respective licenses. Please check the license terms of all external components before redistribution or commercial use.