README.md

<div align="center">
  <picture>
      <img src="https://raw.githubusercontent.com/FluxVLA/FluxVLA/main/assets/fluxvla.png" width="50%" alt="FluxVLA Engine">
  </picture>
</div>
<hr>

<div align="center" style="line-height:1">
  <a href="https://github.com/limxdynamics/FluxVLA" target="_blank"><img alt="GitHub" src="https://img.shields.io/badge/GitHub-FluxVLA-181717?logo=github&logoColor=white"/></a>
  <a href="https://fluxvla.limxdynamics.com" target="_blank"><img alt="Docs" src="https://img.shields.io/badge/Docs-English-2ea44f"/></a>
  <a href="https://fluxvla.limxdynamics.com/zh/" target="_blank"><img alt="中文文档" src="https://img.shields.io/badge/Docs-中文文档-2ea44f"/></a>
</div>

<div align="center" style="line-height: 1;">
  <a href="https://huggingface.co/limxdynamics/FluxVLAEngine" target="_blank"><img alt="Hugging Face" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-FluxVLAEngine-ffc107?color=ffc107&logoColor=white"/></a>
  <a href="https://github.com/limxdynamics/FluxVLA" target="_blank"><img alt="License" src="https://img.shields.io/badge/License-Apache--2.0-blue.svg"/></a>
</div>

<div align="center" style="line-height: 1;">
<a href="https://github.com/limxdynamics/FluxVLA/issues/1"><img src="https://img.shields.io/badge/WeChat-green?logo=wechat"></a>
<a href="https://github.com/limxdynamics/FluxVLA/issues/1"><img src="https://img.shields.io/badge/Feishu-3370FF?logo=lark&logoColor=white"></a>
</div>

<p align="center">
📖&nbsp;&nbsp;<a href="https://fluxvla.limxdynamics.com" target="_blank">Documentation</a>
&nbsp;|&nbsp;
🤗&nbsp;&nbsp;<a href="https://huggingface.co/limxdynamics/FluxVLAEngine">Model Hub</a>
</p>


## 1. Model Introduction

FluxVLA Engine is an integrated engineering platform designed for embodied intelligence applications. It follows the core design principles of unified configuration, standardized interfaces, module decoupling, and deployability, forming a complete engineering loop from data collection to real-world deployment. With a focus on building a "standardized industrial-academic-research foundation," FluxVLA significantly lowers the engineering threshold for VLA (Visual Language Agent) research and development.

### Key Features

🧩 **Rich VLA Model Zoo**: Built-in support for mainstream VLA families including OpenVLA, LLaVA-VLA, GR00T, Pi0, Pi0.5, and DreamZero, enabling fast comparison, fine-tuning, and deployment across different model architectures.

⚡ **Extreme Inference Speed**: CUDA-accelerated operator fusion. With the GR00T-N1.5, the system achieves a high-frequency of 42.8Hz on the RTX 5090.

🛠 **All-in-One**: A unified configuration file manages critical parameters for data, models, training, evaluation, inference, and deployment, ensuring easy reproducibility and deployment.

🖥 **Flexible Configuration**: Supports mainstream visual backbones, LLM (Large Language Models), VLM (Vision-Language Models), and VLA (Vision-Language Agents). You can also create and customize your own VLA.

🤖 **Real-world Deployment Ready**: In addition to mainstream simulators, FluxVLA comes with ready-to-use deployment scripts for Aloha, Tron2, and UR3, eliminating the need for custom integration.

💨 **Smooth Motion**: Optimized for real-world training/inference with RTC trajectory guidance, ensuring accurate and continuous motion even at 40Hz+ frequencies.

📊 **Strong Benchmark Performance**: Provides competitive LIBERO benchmark results across spatial, object, goal, and long-horizon tasks, making it easier to evaluate VLA models under standardized settings.

🛰 **Remote Inference Ready**: Includes a ZMQ-based server/client inference framework, enabling GPU-offloaded remote inference for resource-constrained robot-side devices.

🧪 **Scalable Training Pipeline**: Supports distributed training with DDP/FSDP, LoRA fine-tuning, eval-after-train, and checkpoint-based resume, covering both local debugging and cluster-scale training workflows.

## 2. Model Zoo

FluxVLA Engine supports multiple VLA model families, each optimized for different use cases:

<div align="center">

| Family | Parameters | Key Features | Typical Use Cases |
|:---:|:---:|:---|:---|
| **GR00T** | 3B | High-frequency action prediction, CUDA-optimized | Real-time robot control, deployment |
| **Pi0 / Pi0.5** | 3B | Flow-matching VLA, smooth trajectories | Continuous control, manipulation tasks |
| **DreamZero** | 23B | Diffusion/flow-based generation | Complex motion planning, LIBERO tasks |
| **LLaVA-VLA** | 3B-7B | VLM-based architecture | Vision-language grounding, instruction following |
| **OpenVLA** | 7B | Open-source baseline | Benchmark comparison, fine-tuning |

</div>

### Available Pretrained Models

| Model | Size | Download Link |
|:---:|:---:|:---|
| GR00T N1.5 | 3B | [Hugging Face](https://huggingface.co/nvidia/GR00T-N1.5-3B) |
| PI0 Base | 3B | [Hugging Face](https://huggingface.co/limxdynamics/FluxVLAEngine/tree/main/pi0_base) |
| PI0.5 Base | 3B | [Hugging Face](https://huggingface.co/limxdynamics/FluxVLAEngine/tree/main/pi05_base) |
| PI0.5 LIBERO | 3B | [Hugging Face](https://huggingface.co/limxdynamics/FluxVLAEngine/tree/main/pi05_libero) |
| OpenVLA | 7B | [Hugging Face](https://huggingface.co/openvla/openvla-7b-finetuned-libero-10) |

### Trained FluxVLA Checkpoints

| Model | Training Data | Performance | Download |
|:---:|:---:|:---:|:---|
| PI0.5 PaliGemma | LIBERO-10 | 96.0% avg | [Hugging Face](https://huggingface.co/limxdynamics/FluxVLAEngine/tree/main/pi05_paligemma_libero_10_full_finetune_bs64) |
| Cosmos3-Edge | LIBERO benchmark suites | 94.65% avg | [Spatial](https://huggingface.co/limxdynamics/FluxVLAEngine/tree/main/cosmos3edge_libero_spatial_full_finetune_bs16) · [Object](https://huggingface.co/limxdynamics/FluxVLAEngine/tree/main/cosmos3edge_libero_object_full_finetune_bs16) · [Goal](https://huggingface.co/limxdynamics/FluxVLAEngine/tree/main/cosmos3edge_libero_goal_full_finetune_bs16) · [Long](https://huggingface.co/limxdynamics/FluxVLAEngine/tree/main/cosmos3edge_libero_10_full_finetune_bs16) |
| GR00T Eagle 3B | LIBERO-10 | 89.4% avg | [Hugging Face](https://huggingface.co/limxdynamics/FluxVLAEngine/tree/main/gr00t_eagle_3b_libero_10_full_finetune_bs64) |

## 3. Evaluation Results

FluxVLA demonstrates state-of-the-art performance across multiple LIBERO benchmark suites:

<div align="center">

### LIBERO Benchmark Performance

| Model | Libero-Spatial | Libero-Object | Libero-Goal | Libero-Long | Average |
|:---:|:---:|:---:|:---:|:---:|:---:|
| **FluxVLA (Pi0.5)** | 98.6 | 99.0 | 97.8 | 96.0±1.0 | 97.85 |
| **FluxVLA (Qwen3VL+GR00T)** | 98.6 | 99.6 | 95.6 | 92.2±1.8 | 96.50 |
| **FluxVLA (Cosmos3-Edge)** | 97.4 | 94.0 | 95.6 | 91.6 | 94.65 |
| **FluxVLA (DreamZero)** | 96.8 | 97.4 | 90.8±1.5 | 93.6 | 94.65 |
| **FluxVLA (GR00T)** | 96.2 | 96.8 | 93.4 | 89.4±1.5 | 93.95 |

</div>

### Inference Speed Comparison

<div align="center">

| Model | Hardware | Inference Speed | Acceleration |
|:---:|:---:|:---:|:---:|
| GR00T N1.5 (Optimized) | RTX 5090 | 42.8 Hz | 3.2x |
| GR00T N1.5 (Baseline) | RTX 5090 | 13.4 Hz | 1.0x |
| Pi0.5 (Optimized) | RTX 5090 | 38.5 Hz | 2.8x |
| Pi0.5 (Baseline) | RTX 5090 | 13.7 Hz | 1.0x |

</div>

*Note: Optimized versions use CUDA-accelerated operator fusion and Triton kernels.*

## 4. Architecture Overview

FluxVLA Engine supports flexible architecture combinations:

<div align="center">

<p align="center">
  <img src="https://raw.githubusercontent.com/FluxVLA/FluxVLA/main/assets/framework.png" alt="Framework Architecture" width="800">
</p>

### Supported Components

| Component Type | Options |
|:---:|:---|
| **Vision Encoders** | DINOv2 ViT-Large, SigLIP ViT-SO400M, SigLIP2, PaliGemma |
| **Language Models** | Qwen2.5 (3B/7B), Llama 2 (7B) |
| **Vision-Language Models** | Qwen2.5-VL (3B), PaliGemma (3B) |
| **Action Heads** | Diffusion, Flow Matching, Direct Regression |
| **Precision** | FP32, FP16, BF16|

</div>

## 5. Installation & Deployment

### Quick Installation

```bash
# Clone repository
git clone https://github.com/limxdynamics/FluxVLA.git
cd FluxVLA

# Create environment
conda create -n fluxvla python=3.10 -y
conda activate fluxvla

# Install PyTorch (CUDA 12.4)
pip install torch==2.6.0 torchvision==0.21.0 torchaudio==2.6.0 \
  --index-url https://download.pytorch.org/whl/cu124

# Install flash-attention
pip install psutil ninja packaging
MAX_JOBS=8 pip install flash-attn==2.5.5 --no-build-isolation

# Install FluxVLA
pip install -r requirements.txt
pip install --no-build-isolation -e .
```

### Deployment Options

FluxVLA supports multiple deployment scenarios:

1. **Local Training & Evaluation**: Single-node or multi-node distributed training with DDP/FSDP
2. **Real-Robot Deployment**: Direct on-robot inference with optimized runtime
3. **Remote Inference**: GPU server + robot client architecture via ZMQ
4. **Simulation**: LIBERO, Isaac Sim integration

### Hardware Requirements

| Deployment Type | Minimum GPU | Recommended GPU | Memory |
|:---:|:---:|:---:|:---:|
| Training (3B model) | RTX 3090 (24GB) | A100 (40GB) | 32GB+ RAM |
| Inference (optimized) | RTX 4090 (24GB) | RTX 5090 (32GB) | 16GB+ RAM |
| Remote Inference Server | RTX 4090 (24GB) | RTX 5090 (32GB) | 16GB+ RAM |
| Robot Client | CPU only | - | 8GB+ RAM |

## 6. Model Usage

### Training

Train a VLA model on LIBERO-10 dataset:

```bash
# Single-node training
torchrun \
  --standalone \
  --nnodes 1 \
  --nproc-per-node 2 \
  scripts/train.py \
  --config configs/pi05/pi05_paligemma_libero_10_full_finetune.py \
  --work-dir ./checkpoints/pi05_paligemma_libero_10_full_finetune \
  --cfg-options train_dataloader.per_device_batch_size=2

# Or use the launcher script
bash scripts/train.sh \
  configs/gr00t/gr00t_eagle_3b_libero_10_full_finetune.py \
  work_dirs/gr00t_eagle_3b_libero_10_full_finetune
```

### Evaluation

Evaluate a trained model on LIBERO benchmarks:

```bash
# Evaluate on LIBERO-10
torchrun \
  --standalone \
  --nnodes 1 \
  --nproc-per-node 2 \
  scripts/eval.py \
  --config configs/pi05/pi05_paligemma_libero_10_full_finetune.py \
  --ckpt-path checkpoints/pi05_paligemma_libero_10_full_finetune_bs64/checkpoints/latest-checkpoint.safetensors

# Or use the launcher script
bash scripts/eval.sh \
  configs/pi05/pi05_paligemma_libero_10_full_finetune.py \
  checkpoints/pi05_paligemma_libero_10_full_finetune_bs64/checkpoints/latest-checkpoint.safetensors
```

### Inference

#### Simulation Inference

```python
import torch
from fluxvla.models import build_model
from fluxvla.datasets import build_dataset
from fluxvla.transforms import build_transforms

# Load configuration
config = load_config('configs/pi05/pi05_paligemma_libero_10_full_finetune.py')

# Build model
model = build_model(config.model)
model.load_checkpoint('path/to/checkpoint.safetensors')
model.eval()
model.cuda()

# Build transforms
transforms = build_transforms(config.transforms)

# Inference loop
with torch.no_grad():
    # Prepare input
    image = transforms['image'](raw_image)
    instruction = "Pick up the red cube"
    
    # Get action prediction
    action = model.predict(
        image=image,
        instruction=instruction,
        proprioception=robot_state
    )
    
    # Execute action on robot
    robot.execute(action)
```

#### Real-Robot Inference

```bash
# Direct on-robot inference
python scripts/inference_real_robot.py \
  --config configs/pi05/pi05_paligemma_aloha_full_finetune.py \
  --ckpt-path checkpoints/pi05_paligemma_libero_10_full_finetune_bs64/checkpoints/latest-checkpoint.safetensors
```

#### Remote Inference

For deployment scenarios where the robot cannot host the full model:

```bash
# On GPU server
python scripts/remote_inference_server.py \
  --config configs/pi05/pi05_paligemma_aloha_remote_inference.py \
  --ckpt-path checkpoints/pi05_paligemma_libero_10_full_finetune_bs64/checkpoints/latest-checkpoint.safetensors \
  --host 0.0.0.0 \
  --port 5555

# On robot client
bash scripts/remote_inference_client.sh
```


## 7. Advanced Features

### RTC (Real-Time Control) Trajectory Guidance

FluxVLA includes RTC trajectory guidance for smooth, continuous motion control:

<p align="center">
  <img src="https://raw.githubusercontent.com/FluxVLA/FluxVLA/main/assets/rtc_comparison_prefix_len_5.png" alt="RTC Comparison" width="600">
</p>

RTC improves action continuity by conditioning on recent action history, reducing jitter and improving success rates in real-world deployment.

```python
# Enable RTC in config
model = dict(
    type='Pi05VLA',
    rtc_enabled=True,
    rtc_prefix_length=5,
    rtc_temperature=0.1,
)
```

### Inference Acceleration

FluxVLA provides multiple acceleration techniques:

1. **CUDA Operator Fusion**: Custom CUDA kernels for attention and MLP layers
2. **Triton Kernels**: Optimized Triton implementations for key operations
3. **CUDA Graph**: Graph capture for reduced kernel launch overhead
4. **Mixed Precision**: FP16/BF16 inference with minimal accuracy loss

```bash
# Enable acceleration in config
python scripts/inference.py \
  --config configs/pi05/pi05_paligemma_libero_10_full_finetune.py \
  --ckpt-path checkpoints/latest.safetensors \
  --use-cuda-graph \
  --use-triton-kernels
```

### Distributed Training

FluxVLA supports both DDP and FSDP for multi-GPU training:

```bash
# Multi-node training with FSDP
torchrun \
  --nnodes 4 \
  --nproc-per-node 8 \
  --rdzv-backend c10d \
  --rdzv-endpoint $MASTER_ADDR:$MASTER_PORT \
  scripts/train.py \
  --config configs/pi05/pi05_paligemma_libero_10_full_finetune.py \
  --work-dir ./checkpoints/distributed_training
```

### LoRA Fine-tuning

For efficient fine-tuning with limited compute:

```python
# Enable LoRA in config
lora = dict(
    enabled=True,
    r=16,
    lora_alpha=32,
    target_modules=['q_proj', 'v_proj', 'k_proj', 'o_proj'],
    lora_dropout=0.05,
)
```

## 8. Supported Datasets

FluxVLA supports multiple robot learning datasets:

| Dataset | Tasks | Episodes | Download |
|:---:|:---:|:---:|:---|
| LIBERO-Spatial | 10 | ~5K | [Hugging Face](https://huggingface.co/datasets/limxdynamics/FluxVLAData/tree/main/libero_spatial_no_noops_lerobotv2.1) |
| LIBERO-Object | 10 | ~5K | [Hugging Face](https://huggingface.co/datasets/limxdynamics/FluxVLAData/tree/main/libero_object_no_noops_lerobotv2.1) |
| LIBERO-Goal | 10 | ~5K | [Hugging Face](https://huggingface.co/datasets/limxdynamics/FluxVLAData/tree/main/libero_goal_no_noops_lerobotv2.1) |
| LIBERO-10 | 10 | ~5K | [Hugging Face](https://huggingface.co/datasets/limxdynamics/FluxVLAData/tree/main/libero_10_no_noops_lerobotv2.1) |
| Aloha Real Robot | Custom | Variable | [Hugging Face](https://huggingface.co/datasets/limxdynamics/FluxVLAData/tree/main/RealRobot_AgileX_aloha_lerobot_v2) |
| UR3 Real Robot | Custom | Variable | [Hugging Face](https://huggingface.co/datasets/limxdynamics/FluxVLAData/tree/main/RealRobot_UR3_Chem_lerobot_v2) |

## 9. Citation

If you use FluxVLA in your research, please cite:

```bibtex
@software{fluxvla2026,
  title={FluxVLA Engine: An All-in-One VLA Engineering Platform for Embodied AI},
  author={LimX Dynamics},
  year={2026},
  url={https://github.com/limxdynamics/FluxVLA},
  note={GitHub repository}
}
```

## 10. License

The FluxVLA code is released under the **Apache-2.0 License**. See [LICENSE](LICENSE) for details.

Model checkpoints derived from third-party base models remain subject to their respective licenses. The Cosmos3-Edge checkpoints are derived from NVIDIA Cosmos3-Edge and are subject to the [Open Model Development and Weight License 1.1 (OpenMDW 1.1)](https://openmdw.ai/license/1-1/).

## 11. Acknowledgements

FluxVLA builds on and benefits from the following open-source projects:

- [LeRobot](https://github.com/huggingface/lerobot) - Robot learning datasets and tools
- [NVIDIA Isaac GR00T](https://developer.nvidia.com/isaac/groot) - Foundation model for humanoid robots
- [OpenVLA](https://github.com/openvla/openvla) - Open-source VLA baseline
- [OpenPI](https://github.com/OpenDriveLab/OpenPI) - Flow-matching VLA framework
- [LLaVA](https://github.com/haotian-liu/LLaVA) - Vision-language model architecture
- [Qwen](https://github.com/QwenLM/Qwen) - Large language models
- [DeepSpeed](https://github.com/microsoft/DeepSpeed) - Distributed training optimization
- [Triton](https://github.com/openai/triton) - GPU programming framework
- [RTC](https://github.com/rail-berkeley/rtc) - Real-time control trajectory guidance

We thank the authors and contributors of these projects for their excellent work.

## 12. Contact Us

For questions, feedback, collaboration, or technical support, please contact us:

- **Mason**: [mason@limxdynamics.com](mailto:mason@limxdynamics.com)
- **Wayne Mao**: [waynemao@limxdynamics.com](mailto:waynemao@limxdynamics.com)

You can also:
- Open an issue on [GitHub](https://github.com/limxdynamics/FluxVLA/issues)
- Visit our documentation: [English](https://fluxvla.limxdynamics.com) | [中文](https://fluxvla.limxdynamics.com/zh/)
- Check out our Hugging Face page: [limxdynamics/FluxVLAEngine](https://huggingface.co/limxdynamics/FluxVLAEngine)

---

**Project Links:**
- 🌐 GitHub: <https://github.com/limxdynamics/FluxVLA>
- 🤗 Hugging Face: <https://huggingface.co/limxdynamics/FluxVLAEngine>
- 📖 English Documentation: <https://fluxvla.limxdynamics.com>
- 📖 中文文档: <https://fluxvla.limxdynamics.com/zh/>