README.md

---
license: apache-2.0
pipeline_tag: any-to-any
tags:
- any-to-any
- hunyuan
- hunyuan_vl_mot
- unified_mot
- vision-language
- Embodied
- image-text-to-text
- image-to-text
- MoT
- flow-matching
---

<div align="center">
<h1>RxBrain</h1>
<p><b>Embodied Cognition Foundation Model with Joint Language–Visual Reasoning and Imagination</b></p>
<p><i>Tencent Robotics X × Futian Laboratory × Tencent Hy Team</i></p>

<a href="https://arxiv.org/abs/2607.14187"><img src="https://img.shields.io/badge/arXiv-2607.14187-b31b1b.svg" alt="arXiv"></a>
<a href="https://huggingface.co/tencent/Hy-Embodied-RxBrain-1.0"><img src="https://img.shields.io/badge/🤗%20HuggingFace-Model-FFD21E" alt="Model"></a>
<a href="https://tairos.tencent.com/openSourceModels/hy-embodied-rxbrain-1.0"><img src="https://img.shields.io/badge/🌐%20Project-Page-2ea44f" alt="Project Page"></a>
<a href="https://huggingface.co/spaces/hugging-apps/rxbrain-embodied-cognition"><img src="https://img.shields.io/badge/🤗%20HuggingFace-Demo-FFD21E" alt="Demo"></a>
<a href="https://github.com/Tencent-Hunyuan/Hy-Embodied-RxBrain-1.0"><img src="https://img.shields.io/badge/GitHub-Code-181717?logo=github&logoColor=white" alt="GitHub"></a>
<a href="./LICENSE"><img src="https://img.shields.io/badge/License-Apache_2.0-blue.svg" alt="License"></a>

</div>

<div align="center">
<img src="https://github.com/Tencent-Hunyuan/Hy-Embodied-RxBrain-1.0/blob/main/assets/teaser.png?raw=true" alt="RxBrain — capability overview" width="90%">
</div>

<div align="center">
<video src="https://huggingface.co/tencent/Hy-Embodied-RxBrain-1.0/resolve/main/assets/RxBrain_demo.mp4" controls width="90%"></video>
</div>

## 🔥 Updates

  * **`[2026-07]`** 🎉 We release **Hy-Embodied-RxBrain-1.0** — the technical report, official inference code, and model weights.

## 📖 Introduction

**RxBrain** (`Hy-Embodied-RxBrain-1.0`) is a **unified multimodal foundation model for embodied cognition** — a single model that couples language reasoning with visual imagination to deliver three core capabilities:

  * 🤖 **Embodied Understanding & Reasoning** — question answering and chain-of-thought over images and multi-frame video.
  * 🔮 **World State Prediction** — imagine the near-future frames an action produces in the physical world.
  * 🧩 **Joint Subgoal Planning** — decompose a task into steps, emitting for each step *both* the next action (language) *and* the goal image it should reach (vision).

These capabilities are unified through **interleaved generation**: within a single autoregressive sequence RxBrain alternates reasoning text and flow-matched imagined frames — a learned `<Image>` token decides when to imagine — so an embodied plan couples *what to do* with *what the world should look like*, step by step.

## ⭐️ Key Features

  * 🧠 **Unified Mixture-of-Transformers (MoT):** A ~6.2B-parameter backbone with modality-specific pathways (text / vision / generation), so understanding and image synthesis share one autoregressive model instead of separate towers.
  * 🎨 **Flow-Matching Image Head:** Imagined frames are produced by a flow-matching head decoding into a frozen **FLUX** VAE latent space, enabling text-to-image, multi-frame world-model rollout, and goal-image planning.
  * 🔗 **Interleaved Reasoning + Imagination:** Text reasoning and generated frames are emitted in one sequence, coupling symbolic plans with visual goals.

## 📋 TODO

* [ ] Open-source **RxBrain-Bench**, a comprehensive benchmark for embodied cognition.
* [ ] Open-source the fine-tuning code for **Visual Question Answering (VQA)**, **Multi Frames Generation**, and **Interleave Generation**.
* [x] Release the **technical report**, **inference code**, and **model weights**.

## 🛠️ Dependencies and Installation

### Prerequisites

- 🖥️ **Operating System**: Linux (recommended)
- 🐍 **Python**: 3.10+
- ⚡ **CUDA**: 12.x, an NVIDIA GPU (required for `flash-attn`)
- 🔥 **PyTorch**: 2.10

### Installation

1. **Install the specific Transformers version required for this model** (it provides the `hunyuan_vl_mot` backbone that `unified_mot` builds on):
```bash
pip install git+https://github.com/huggingface/transformers@9293856c419762ebf98fbe2bd9440f9ce7069f1a
```
> **Note:** A stock `transformers` release does **not** yet include `hunyuan_vl_mot`; this pinned commit is required. We will merge the improvements into the Transformers main branch later.

2. **Clone the inference code and install the remaining dependencies:**
```bash
git clone https://github.com/Tencent-Hunyuan/Hy-Embodied-RxBrain-1.0.git
cd Hy-Embodied-RxBrain-1.0
pip install -r requirements.txt
```

### Model Download

| Component | Params | Source |
|---|:---:|---|
| **Hy-Embodied-RxBrain-1.0** | ~6.2 B | [🤗 tencent/Hy-Embodied-RxBrain-1.0](https://huggingface.co/tencent/Hy-Embodied-RxBrain-1.0) |
| FLUX VAE (`ae.safetensors`) | 83.8 M | Obtain from the [FLUX](https://github.com/black-forest-labs/flux) distribution |

Download the weights to a **local directory** — the loader reads the checkpoint files directly, so `--ckpt` must be a local path, **not** the Hub repo id:

```bash
pip install -U "huggingface_hub[cli]"
hf download tencent/Hy-Embodied-RxBrain-1.0 --local-dir ./Hy-Embodied-RxBrain-1.0
```

The VQA (understanding) path needs **only** the main weights. Image generation (T2I / world-model rollout / interleaved planning) additionally requires the external **FLUX VAE** `ae.safetensors`.


## 🚀 Quick Start with Transformers

Load the Transformers processor together with the `UnifiedMoT` classes shipped in this repo, then run understanding (VQA). Run this from the repo root so the `model` package is importable, and point `MODEL_PATH` at your **local** download (see [Model Download](#model-download)).

```python
import torch
from transformers.models.hunyuan_vl_mot import HunYuanVLMoTProcessor
from model import UnifiedMoTForConditionalGeneration, maybe_init_generation_path
from vqa_inference import answer

MODEL_PATH = "./Hy-Embodied-RxBrain-1.0"  # local checkpoint directory, not the Hub id
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
dtype = torch.bfloat16

# Load processor & model
processor = HunYuanVLMoTProcessor.from_pretrained(MODEL_PATH, trust_remote_code=True)
model = UnifiedMoTForConditionalGeneration.from_pretrained(MODEL_PATH, dtype=dtype)
maybe_init_generation_path(model, model_load_path=MODEL_PATH)  # wires up the generation path
model.to(device).eval()

# Ask a question about an image
text = answer(
    model, processor,
    image_paths=["demo_cases/bridgev2_move_toy/input/obs_1.jpg"],
    question="What objects are on the stovetop, and where is the green toy?",
    device=device, dtype=dtype, max_new_tokens=256,
)
print(text)
```

> **Note:** RxBrain uses a custom interleaved text/image decoding loop rather than the standard `model.generate` API. The `answer(...)` helper (in `vqa_inference.py`) wraps that loop for the understanding case; image generation and planning have their own entry points below.

The same tasks are also available as ready-to-run scripts:

<details open>
<summary><b>① Visual Question Answering (VQA)</b> — image(s) + question → answer text</summary>

Pure autoregressive text understanding — **no VAE / flow-matching needed**.

```bash
python vqa_inference.py \
    --ckpt ./Hy-Embodied-RxBrain-1.0 \
    --images demo_cases/bridgev2_move_toy/input/obs_1.jpg \
    --question "What objects are on the stovetop, and where is the green toy?" \
    --max_new_tokens 256
```
</details>


<details open>
<summary><b>② Text-to-Image (T2I)</b></summary>

```bash
python text2image_inference.py \
    --ckpt ./Hy-Embodied-RxBrain-1.0 --vae /path/to/ae.safetensors \
    --prompt "a watercolor painting of a cat" \
    --height 256 --width 256 --num_steps 25 --out out.png

# with classifier-free guidance
python text2image_inference.py \
    --ckpt ./Hy-Embodied-RxBrain-1.0 --vae /path/to/ae.safetensors \
    --prompt "a watercolor painting of a cat" \
    --cfg_scale 5.0 --num_steps 50 --out out.png
```
</details>

<details open>
<summary><b>③ Multi-Frame World-Model Rollout</b> — imagine future frames from an observation</summary>

```bash
python multiframe_inference.py \
    --ckpt ./Hy-Embodied-RxBrain-1.0 --vae /path/to/ae.safetensors \
    --frames /path/to/obs.jpg --task "imagine the next frames" \
    --num_frames 4 --num_steps 50 --out_dir multiframe_out
```
</details>

<details>
<summary><b>④ Interleaved Embodied Planning</b> — text plan + goal images, step by step</summary>

Runs interleaved planning on a bundled scene. See [`demo_cases/README.md`](https://github.com/Tencent-Hunyuan/Hy-Embodied-RxBrain-1.0/blob/main/demo_cases/README.md) for details.

```bash
CASE=umi_fold_sock
python interleave_inference.py \
    --ckpt ./Hy-Embodied-RxBrain-1.0 --vae /path/to/ae.safetensors \
    --frames  demo_cases/$CASE/input/*.jpg \
    --task    "$(cat demo_cases/$CASE/prompt.txt)" \
    --max_frames 5 --num_steps 50 --out_dir out_$CASE
```
</details>

## 📊 Evaluation

RxBrain is evaluated on embodied understanding, spatial reasoning, and imagination/generation benchmarks. For detailed metrics and methodology, please refer to our [technical report](https://arxiv.org/abs/2607.14187).