| --- |
| license: other |
| language: |
| - multilingual |
| pipeline_tag: image-text-to-text |
| tags: |
| - hunyuan |
| - hunyuan_vl_mot |
| - unified_mot |
| - vision-language |
| - Embodied |
| - image-to-text |
| - any-to-any |
| - MoT |
| - flow-matching |
| --- |
| |
| <div align="center"> |
| <h1>RxBrain</h1> |
| <p><b>Embodied Cognition Foundation Model with Joint Language–Visual Reasoning and Imagination</b></p> |
| <p><i>Tencent Robotics X × Futian Laboratory × Tencent Hunyuan</i></p> |
|
|
| <a href="https://github.com/Tencent-Hunyuan/Hy-Embodied-RxBrain-1.0/blob/main/assets/RxBrain_v0.pdf"><img src="https://img.shields.io/badge/Paper-Report-red?logo=adobeacrobatreader" alt="Tech Report"></a> |
| <a href="https://huggingface.co/tencent/Hy-Embodied-RxBrain-1.0"><img src="https://img.shields.io/badge/Models-HuggingFace-yellow?logo=huggingface" alt="Models"></a> |
| <a href="https://github.com/Tencent-Hunyuan/Hy-Embodied-RxBrain-1.0"><img src="https://img.shields.io/badge/GitHub-Repo-181717?logo=github&logoColor=white" alt="GitHub"></a> |
|
|
| </div> |
|
|
| <div align="center"> |
| <img src="https://github.com/Tencent-Hunyuan/Hy-Embodied-RxBrain-1.0/blob/main/assets/teaser.png?raw=true" alt="RxBrain — capability overview" width="90%"> |
| </div> |
|
|
| <div align="center"> |
| <video src="https://huggingface.co/tencent/Hy-Embodied-RxBrain-1.0/resolve/main/assets/RxBrain_demo.mp4" controls width="90%"></video> |
| </div> |
|
|
| ## 🔥 Updates |
|
|
| * **`[2026-07]`** 🎉 We release **Hy-Embodied-RxBrain-1.0** — the technical report, official inference code, and model weights. |
|
|
| ## 📖 Introduction |
|
|
| **RxBrain** (`Hy-Embodied-RxBrain-1.0`) is a **unified multimodal foundation model for embodied cognition** — a single model that couples language reasoning with visual imagination to deliver three core capabilities: |
|
|
| * 🤖 **Embodied Understanding & Reasoning** — question answering and chain-of-thought over images and multi-frame video. |
| * 🔮 **World State Prediction** — imagine the near-future frames an action produces in the physical world. |
| * 🧩 **Joint Subgoal Planning** — decompose a task into steps, emitting for each step *both* the next action (language) *and* the goal image it should reach (vision). |
|
|
| These capabilities are unified through **interleaved generation**: within a single autoregressive sequence RxBrain alternates reasoning text and flow-matched imagined frames — a learned `<Image>` token decides when to imagine — so an embodied plan couples *what to do* with *what the world should look like*, step by step. |
|
|
| ## ⭐️ Key Features |
|
|
| * 🧠 **Unified Mixture-of-Transformers (MoT):** A ~6.2B-parameter backbone with modality-specific pathways (text / vision / generation), so understanding and image synthesis share one autoregressive model instead of separate towers. |
| * 🎨 **Flow-Matching Image Head:** Imagined frames are produced by a flow-matching head decoding into a frozen **FLUX** VAE latent space, enabling text-to-image, multi-frame world-model rollout, and goal-image planning. |
| * 🔗 **Interleaved Reasoning + Imagination:** Text reasoning and generated frames are emitted in one sequence, coupling symbolic plans with visual goals. |
|
|
| ## 📅 Roadmap |
|
|
| - [x] Transformers Inference (understanding + generation) |
| - [ ] vLLM Inference |
| - [ ] Fine-tuning Code |
| - [ ] Online Gradio Demo |
|
|
| ## 🛠️ Dependencies and Installation |
|
|
| ### Prerequisites |
|
|
| - 🖥️ **Operating System**: Linux (recommended) |
| - 🐍 **Python**: 3.10+ |
| - ⚡ **CUDA**: 12.x, an NVIDIA GPU (required for `flash-attn`) |
| - 🔥 **PyTorch**: 2.10 |
|
|
| ### Installation |
|
|
| 1. **Install the specific Transformers version required for this model** (it provides the `hunyuan_vl_mot` backbone that `unified_mot` builds on): |
| ```bash |
| pip install git+https://github.com/huggingface/transformers@9293856c419762ebf98fbe2bd9440f9ce7069f1a |
| ``` |
| > **Note:** A stock `transformers` release does **not** yet include `hunyuan_vl_mot`; this pinned commit is required. We will merge the improvements into the Transformers main branch later. |
|
|
| 2. **Clone the inference code and install the remaining dependencies:** |
| ```bash |
| git clone https://github.com/Tencent-Hunyuan/Hy-Embodied-RxBrain-1.0.git |
| cd Hy-Embodied-RxBrain-1.0 |
| pip install -r requirements.txt |
| ``` |
|
|
| ### Model Download |
|
|
| | Component | Params | Source | |
| |---|:---:|---| |
| | **Hy-Embodied-RxBrain-1.0** | ~6.2 B | [🤗 tencent/Hy-Embodied-RxBrain-1.0](https://huggingface.co/tencent/Hy-Embodied-RxBrain-1.0) | |
| | FLUX VAE (`ae.safetensors`) | 83.8 M | Obtain from the [FLUX](https://github.com/black-forest-labs/flux) distribution | |
|
|
| Download the weights to a **local directory** — the loader reads the checkpoint files directly, so `--ckpt` must be a local path, **not** the Hub repo id: |
|
|
| ```bash |
| pip install -U "huggingface_hub[cli]" |
| hf download tencent/Hy-Embodied-RxBrain-1.0 --local-dir ./Hy-Embodied-RxBrain-1.0 |
| ``` |
|
|
| The VQA (understanding) path needs **only** the main weights. Image generation (T2I / world-model rollout / interleaved planning) additionally requires the external **FLUX VAE** `ae.safetensors`. |
|
|
|
|
| ## 🚀 Quick Start with Transformers |
|
|
| Load the Transformers processor together with the `UnifiedMoT` classes shipped in this repo, then run understanding (VQA). Run this from the repo root so the `model` package is importable, and point `MODEL_PATH` at your **local** download (see [Model Download](#model-download)). |
|
|
| ```python |
| import torch |
| from transformers.models.hunyuan_vl_mot import HunYuanVLMoTProcessor |
| from model import UnifiedMoTForConditionalGeneration, maybe_init_generation_path |
| from vqa_inference import answer |
| |
| MODEL_PATH = "./Hy-Embodied-RxBrain-1.0" # local checkpoint directory, not the Hub id |
| device = torch.device("cuda" if torch.cuda.is_available() else "cpu") |
| dtype = torch.bfloat16 |
| |
| # Load processor & model |
| processor = HunYuanVLMoTProcessor.from_pretrained(MODEL_PATH, trust_remote_code=True) |
| model = UnifiedMoTForConditionalGeneration.from_pretrained(MODEL_PATH, dtype=dtype) |
| maybe_init_generation_path(model, model_load_path=MODEL_PATH) # wires up the generation path |
| model.to(device).eval() |
| |
| # Ask a question about an image |
| text = answer( |
| model, processor, |
| image_paths=["demo_cases/bridgev2_move_toy/input/obs_1.jpg"], |
| question="What objects are on the stovetop, and where is the green toy?", |
| device=device, dtype=dtype, max_new_tokens=256, |
| ) |
| print(text) |
| ``` |
|
|
| > **Note:** RxBrain uses a custom interleaved text/image decoding loop rather than the standard `model.generate` API. The `answer(...)` helper (in `vqa_inference.py`) wraps that loop for the understanding case; image generation and planning have their own entry points below. |
| |
| The same tasks are also available as ready-to-run scripts: |
| |
| <details open> |
| <summary><b>① Visual Question Answering (VQA)</b> — image(s) + question → answer text</summary> |
| |
| Pure autoregressive text understanding — **no VAE / flow-matching needed**. |
| |
| ```bash |
| python vqa_inference.py \ |
| --ckpt ./Hy-Embodied-RxBrain-1.0 \ |
| --images demo_cases/bridgev2_move_toy/input/obs_1.jpg \ |
| --question "What objects are on the stovetop, and where is the green toy?" \ |
| --max_new_tokens 256 |
| ``` |
| </details> |
| |
|
|
| <details open> |
| <summary><b>② Text-to-Image (T2I)</b></summary> |
|
|
| ```bash |
| python text2image_inference.py \ |
| --ckpt ./Hy-Embodied-RxBrain-1.0 --vae /path/to/ae.safetensors \ |
| --prompt "a watercolor painting of a cat" \ |
| --height 256 --width 256 --num_steps 25 --out out.png |
| |
| # with classifier-free guidance |
| python text2image_inference.py \ |
| --ckpt ./Hy-Embodied-RxBrain-1.0 --vae /path/to/ae.safetensors \ |
| --prompt "a watercolor painting of a cat" \ |
| --cfg_scale 5.0 --num_steps 50 --out out.png |
| ``` |
| </details> |
|
|
| <details open> |
| <summary><b>③ Multi-Frame World-Model Rollout</b> — imagine future frames from an observation</summary> |
|
|
| ```bash |
| python multiframe_inference.py \ |
| --ckpt ./Hy-Embodied-RxBrain-1.0 --vae /path/to/ae.safetensors \ |
| --frames /path/to/obs.jpg --task "imagine the next frames" \ |
| --num_frames 4 --num_steps 50 --out_dir multiframe_out |
| ``` |
| </details> |
|
|
| <details> |
| <summary><b>④ Interleaved Embodied Planning</b> — text plan + goal images, step by step</summary> |
|
|
| Runs interleaved planning on a bundled scene. See [`demo_cases/README.md`](https://github.com/Tencent-Hunyuan/Hy-Embodied-RxBrain-1.0/blob/main/demo_cases/README.md) for details. |
|
|
| ```bash |
| CASE=umi_fold_sock |
| python interleave_inference.py \ |
| --ckpt ./Hy-Embodied-RxBrain-1.0 --vae /path/to/ae.safetensors \ |
| --frames demo_cases/$CASE/input/*.jpg \ |
| --task "$(cat demo_cases/$CASE/prompt.txt)" \ |
| --max_frames 5 --num_steps 50 --out_dir out_$CASE |
| ``` |
| </details> |
|
|
| ## 📊 Evaluation |
|
|
| RxBrain is evaluated on embodied understanding, spatial reasoning, and imagination/generation benchmarks. For detailed metrics and methodology, please refer to our [technical report](https://github.com/Tencent-Hunyuan/Hy-Embodied-RxBrain-1.0/blob/main/assets/RxBrain_v0.pdf). |
|
|
|
|
