📖 Documentation
|
🤗 Model Hub
## 1. Model Introduction
FluxVLA Engine is an integrated engineering platform designed for embodied intelligence applications. It follows the core design principles of unified configuration, standardized interfaces, module decoupling, and deployability, forming a complete engineering loop from data collection to real-world deployment. With a focus on building a "standardized industrial-academic-research foundation," FluxVLA significantly lowers the engineering threshold for VLA (Visual Language Agent) research and development.
### Key Features
🧩 **Rich VLA Model Zoo**: Built-in support for mainstream VLA families including OpenVLA, LLaVA-VLA, GR00T, Pi0, Pi0.5, and DreamZero, enabling fast comparison, fine-tuning, and deployment across different model architectures.
⚡ **Extreme Inference Speed**: CUDA-accelerated operator fusion. With the GR00T-N1.5, the system achieves a high-frequency of 42.8Hz on the RTX 5090.
🛠 **All-in-One**: A unified configuration file manages critical parameters for data, models, training, evaluation, inference, and deployment, ensuring easy reproducibility and deployment.
🖥 **Flexible Configuration**: Supports mainstream visual backbones, LLM (Large Language Models), VLM (Vision-Language Models), and VLA (Vision-Language Agents). You can also create and customize your own VLA.
🤖 **Real-world Deployment Ready**: In addition to mainstream simulators, FluxVLA comes with ready-to-use deployment scripts for Aloha, Tron2, and UR3, eliminating the need for custom integration.
💨 **Smooth Motion**: Optimized for real-world training/inference with RTC trajectory guidance, ensuring accurate and continuous motion even at 40Hz+ frequencies.
📊 **Strong Benchmark Performance**: Provides competitive LIBERO benchmark results across spatial, object, goal, and long-horizon tasks, making it easier to evaluate VLA models under standardized settings.
🛰 **Remote Inference Ready**: Includes a ZMQ-based server/client inference framework, enabling GPU-offloaded remote inference for resource-constrained robot-side devices.
🧪 **Scalable Training Pipeline**: Supports distributed training with DDP/FSDP, LoRA fine-tuning, eval-after-train, and checkpoint-based resume, covering both local debugging and cluster-scale training workflows.
## 2. Model Zoo
FluxVLA Engine supports multiple VLA model families, each optimized for different use cases:
| Family | Parameters | Key Features | Typical Use Cases |
|:---:|:---:|:---|:---|
| **GR00T** | 3B | High-frequency action prediction, CUDA-optimized | Real-time robot control, deployment |
| **Pi0 / Pi0.5** | 3B | Flow-matching VLA, smooth trajectories | Continuous control, manipulation tasks |
| **DreamZero** | 23B | Diffusion/flow-based generation | Complex motion planning, LIBERO tasks |
| **LLaVA-VLA** | 3B-7B | VLM-based architecture | Vision-language grounding, instruction following |
| **OpenVLA** | 7B | Open-source baseline | Benchmark comparison, fine-tuning |
### Available Pretrained Models
| Model | Size | Download Link |
|:---:|:---:|:---|
| GR00T N1.5 | 3B | [Hugging Face](https://huggingface.co/nvidia/GR00T-N1.5-3B) |
| PI0 Base | 3B | [Hugging Face](https://huggingface.co/limxdynamics/FluxVLAEngine/tree/main/pi0_base) |
| PI0.5 Base | 3B | [Hugging Face](https://huggingface.co/limxdynamics/FluxVLAEngine/tree/main/pi05_base) |
| PI0.5 LIBERO | 3B | [Hugging Face](https://huggingface.co/limxdynamics/FluxVLAEngine/tree/main/pi05_libero) |
| OpenVLA | 7B | [Hugging Face](https://huggingface.co/openvla/openvla-7b-finetuned-libero-10) |
### Trained FluxVLA Checkpoints
| Model | Training Data | Performance | Download |
|:---:|:---:|:---:|:---|
| PI0.5 PaliGemma | LIBERO-10 | 96.0% avg | [Hugging Face](https://huggingface.co/limxdynamics/FluxVLAEngine/tree/main/pi05_paligemma_libero_10_full_finetune_bs64) |
| Cosmos3-Edge | LIBERO benchmark suites | 94.65% avg | [Spatial](https://huggingface.co/limxdynamics/FluxVLAEngine/tree/main/cosmos3edge_libero_spatial_full_finetune_bs16) · [Object](https://huggingface.co/limxdynamics/FluxVLAEngine/tree/main/cosmos3edge_libero_object_full_finetune_bs16) · [Goal](https://huggingface.co/limxdynamics/FluxVLAEngine/tree/main/cosmos3edge_libero_goal_full_finetune_bs16) · [Long](https://huggingface.co/limxdynamics/FluxVLAEngine/tree/main/cosmos3edge_libero_10_full_finetune_bs16) |
| GR00T Eagle 3B | LIBERO-10 | 89.4% avg | [Hugging Face](https://huggingface.co/limxdynamics/FluxVLAEngine/tree/main/gr00t_eagle_3b_libero_10_full_finetune_bs64) |
## 3. Evaluation Results
FluxVLA demonstrates state-of-the-art performance across multiple LIBERO benchmark suites:
### LIBERO Benchmark Performance
| Model | Libero-Spatial | Libero-Object | Libero-Goal | Libero-Long | Average |
|:---:|:---:|:---:|:---:|:---:|:---:|
| **FluxVLA (Pi0.5)** | 98.6 | 99.0 | 97.8 | 96.0±1.0 | 97.85 |
| **FluxVLA (Qwen3VL+GR00T)** | 98.6 | 99.6 | 95.6 | 92.2±1.8 | 96.50 |
| **FluxVLA (Cosmos3-Edge)** | 97.4 | 94.0 | 95.6 | 91.6 | 94.65 |
| **FluxVLA (DreamZero)** | 96.8 | 97.4 | 90.8±1.5 | 93.6 | 94.65 |
| **FluxVLA (GR00T)** | 96.2 | 96.8 | 93.4 | 89.4±1.5 | 93.95 |
### Inference Speed Comparison
| Model | Hardware | Inference Speed | Acceleration |
|:---:|:---:|:---:|:---:|
| GR00T N1.5 (Optimized) | RTX 5090 | 42.8 Hz | 3.2x |
| GR00T N1.5 (Baseline) | RTX 5090 | 13.4 Hz | 1.0x |
| Pi0.5 (Optimized) | RTX 5090 | 38.5 Hz | 2.8x |
| Pi0.5 (Baseline) | RTX 5090 | 13.7 Hz | 1.0x |
*Note: Optimized versions use CUDA-accelerated operator fusion and Triton kernels.*
## 4. Architecture Overview
FluxVLA Engine supports flexible architecture combinations:
### Supported Components
| Component Type | Options |
|:---:|:---|
| **Vision Encoders** | DINOv2 ViT-Large, SigLIP ViT-SO400M, SigLIP2, PaliGemma |
| **Language Models** | Qwen2.5 (3B/7B), Llama 2 (7B) |
| **Vision-Language Models** | Qwen2.5-VL (3B), PaliGemma (3B) |
| **Action Heads** | Diffusion, Flow Matching, Direct Regression |
| **Precision** | FP32, FP16, BF16|
## 5. Installation & Deployment
### Quick Installation
```bash
# Clone repository
git clone https://github.com/limxdynamics/FluxVLA.git
cd FluxVLA
# Create environment
conda create -n fluxvla python=3.10 -y
conda activate fluxvla
# Install PyTorch (CUDA 12.4)
pip install torch==2.6.0 torchvision==0.21.0 torchaudio==2.6.0 \
--index-url https://download.pytorch.org/whl/cu124
# Install flash-attention
pip install psutil ninja packaging
MAX_JOBS=8 pip install flash-attn==2.5.5 --no-build-isolation
# Install FluxVLA
pip install -r requirements.txt
pip install --no-build-isolation -e .
```
### Deployment Options
FluxVLA supports multiple deployment scenarios:
1. **Local Training & Evaluation**: Single-node or multi-node distributed training with DDP/FSDP
2. **Real-Robot Deployment**: Direct on-robot inference with optimized runtime
3. **Remote Inference**: GPU server + robot client architecture via ZMQ
4. **Simulation**: LIBERO, Isaac Sim integration
### Hardware Requirements
| Deployment Type | Minimum GPU | Recommended GPU | Memory |
|:---:|:---:|:---:|:---:|
| Training (3B model) | RTX 3090 (24GB) | A100 (40GB) | 32GB+ RAM |
| Inference (optimized) | RTX 4090 (24GB) | RTX 5090 (32GB) | 16GB+ RAM |
| Remote Inference Server | RTX 4090 (24GB) | RTX 5090 (32GB) | 16GB+ RAM |
| Robot Client | CPU only | - | 8GB+ RAM |
## 6. Model Usage
### Training
Train a VLA model on LIBERO-10 dataset:
```bash
# Single-node training
torchrun \
--standalone \
--nnodes 1 \
--nproc-per-node 2 \
scripts/train.py \
--config configs/pi05/pi05_paligemma_libero_10_full_finetune.py \
--work-dir ./checkpoints/pi05_paligemma_libero_10_full_finetune \
--cfg-options train_dataloader.per_device_batch_size=2
# Or use the launcher script
bash scripts/train.sh \
configs/gr00t/gr00t_eagle_3b_libero_10_full_finetune.py \
work_dirs/gr00t_eagle_3b_libero_10_full_finetune
```
### Evaluation
Evaluate a trained model on LIBERO benchmarks:
```bash
# Evaluate on LIBERO-10
torchrun \
--standalone \
--nnodes 1 \
--nproc-per-node 2 \
scripts/eval.py \
--config configs/pi05/pi05_paligemma_libero_10_full_finetune.py \
--ckpt-path checkpoints/pi05_paligemma_libero_10_full_finetune_bs64/checkpoints/latest-checkpoint.safetensors
# Or use the launcher script
bash scripts/eval.sh \
configs/pi05/pi05_paligemma_libero_10_full_finetune.py \
checkpoints/pi05_paligemma_libero_10_full_finetune_bs64/checkpoints/latest-checkpoint.safetensors
```
### Inference
#### Simulation Inference
```python
import torch
from fluxvla.models import build_model
from fluxvla.datasets import build_dataset
from fluxvla.transforms import build_transforms
# Load configuration
config = load_config('configs/pi05/pi05_paligemma_libero_10_full_finetune.py')
# Build model
model = build_model(config.model)
model.load_checkpoint('path/to/checkpoint.safetensors')
model.eval()
model.cuda()
# Build transforms
transforms = build_transforms(config.transforms)
# Inference loop
with torch.no_grad():
# Prepare input
image = transforms['image'](raw_image)
instruction = "Pick up the red cube"
# Get action prediction
action = model.predict(
image=image,
instruction=instruction,
proprioception=robot_state
)
# Execute action on robot
robot.execute(action)
```
#### Real-Robot Inference
```bash
# Direct on-robot inference
python scripts/inference_real_robot.py \
--config configs/pi05/pi05_paligemma_aloha_full_finetune.py \
--ckpt-path checkpoints/pi05_paligemma_libero_10_full_finetune_bs64/checkpoints/latest-checkpoint.safetensors
```
#### Remote Inference
For deployment scenarios where the robot cannot host the full model:
```bash
# On GPU server
python scripts/remote_inference_server.py \
--config configs/pi05/pi05_paligemma_aloha_remote_inference.py \
--ckpt-path checkpoints/pi05_paligemma_libero_10_full_finetune_bs64/checkpoints/latest-checkpoint.safetensors \
--host 0.0.0.0 \
--port 5555
# On robot client
bash scripts/remote_inference_client.sh
```
## 7. Advanced Features
### RTC (Real-Time Control) Trajectory Guidance
FluxVLA includes RTC trajectory guidance for smooth, continuous motion control:
RTC improves action continuity by conditioning on recent action history, reducing jitter and improving success rates in real-world deployment.
```python
# Enable RTC in config
model = dict(
type='Pi05VLA',
rtc_enabled=True,
rtc_prefix_length=5,
rtc_temperature=0.1,
)
```
### Inference Acceleration
FluxVLA provides multiple acceleration techniques:
1. **CUDA Operator Fusion**: Custom CUDA kernels for attention and MLP layers
2. **Triton Kernels**: Optimized Triton implementations for key operations
3. **CUDA Graph**: Graph capture for reduced kernel launch overhead
4. **Mixed Precision**: FP16/BF16 inference with minimal accuracy loss
```bash
# Enable acceleration in config
python scripts/inference.py \
--config configs/pi05/pi05_paligemma_libero_10_full_finetune.py \
--ckpt-path checkpoints/latest.safetensors \
--use-cuda-graph \
--use-triton-kernels
```
### Distributed Training
FluxVLA supports both DDP and FSDP for multi-GPU training:
```bash
# Multi-node training with FSDP
torchrun \
--nnodes 4 \
--nproc-per-node 8 \
--rdzv-backend c10d \
--rdzv-endpoint $MASTER_ADDR:$MASTER_PORT \
scripts/train.py \
--config configs/pi05/pi05_paligemma_libero_10_full_finetune.py \
--work-dir ./checkpoints/distributed_training
```
### LoRA Fine-tuning
For efficient fine-tuning with limited compute:
```python
# Enable LoRA in config
lora = dict(
enabled=True,
r=16,
lora_alpha=32,
target_modules=['q_proj', 'v_proj', 'k_proj', 'o_proj'],
lora_dropout=0.05,
)
```
## 8. Supported Datasets
FluxVLA supports multiple robot learning datasets:
| Dataset | Tasks | Episodes | Download |
|:---:|:---:|:---:|:---|
| LIBERO-Spatial | 10 | ~5K | [Hugging Face](https://huggingface.co/datasets/limxdynamics/FluxVLAData/tree/main/libero_spatial_no_noops_lerobotv2.1) |
| LIBERO-Object | 10 | ~5K | [Hugging Face](https://huggingface.co/datasets/limxdynamics/FluxVLAData/tree/main/libero_object_no_noops_lerobotv2.1) |
| LIBERO-Goal | 10 | ~5K | [Hugging Face](https://huggingface.co/datasets/limxdynamics/FluxVLAData/tree/main/libero_goal_no_noops_lerobotv2.1) |
| LIBERO-10 | 10 | ~5K | [Hugging Face](https://huggingface.co/datasets/limxdynamics/FluxVLAData/tree/main/libero_10_no_noops_lerobotv2.1) |
| Aloha Real Robot | Custom | Variable | [Hugging Face](https://huggingface.co/datasets/limxdynamics/FluxVLAData/tree/main/RealRobot_AgileX_aloha_lerobot_v2) |
| UR3 Real Robot | Custom | Variable | [Hugging Face](https://huggingface.co/datasets/limxdynamics/FluxVLAData/tree/main/RealRobot_UR3_Chem_lerobot_v2) |
## 9. Citation
If you use FluxVLA in your research, please cite:
```bibtex
@software{fluxvla2026,
title={FluxVLA Engine: An All-in-One VLA Engineering Platform for Embodied AI},
author={LimX Dynamics},
year={2026},
url={https://github.com/limxdynamics/FluxVLA},
note={GitHub repository}
}
```
## 10. License
The FluxVLA code is released under the **Apache-2.0 License**. See [LICENSE](LICENSE) for details.
Model checkpoints derived from third-party base models remain subject to their respective licenses. The Cosmos3-Edge checkpoints are derived from NVIDIA Cosmos3-Edge and are subject to the [Open Model Development and Weight License 1.1 (OpenMDW 1.1)](https://openmdw.ai/license/1-1/).
## 11. Acknowledgements
FluxVLA builds on and benefits from the following open-source projects:
- [LeRobot](https://github.com/huggingface/lerobot) - Robot learning datasets and tools
- [NVIDIA Isaac GR00T](https://developer.nvidia.com/isaac/groot) - Foundation model for humanoid robots
- [OpenVLA](https://github.com/openvla/openvla) - Open-source VLA baseline
- [OpenPI](https://github.com/OpenDriveLab/OpenPI) - Flow-matching VLA framework
- [LLaVA](https://github.com/haotian-liu/LLaVA) - Vision-language model architecture
- [Qwen](https://github.com/QwenLM/Qwen) - Large language models
- [DeepSpeed](https://github.com/microsoft/DeepSpeed) - Distributed training optimization
- [Triton](https://github.com/openai/triton) - GPU programming framework
- [RTC](https://github.com/rail-berkeley/rtc) - Real-time control trajectory guidance
We thank the authors and contributors of these projects for their excellent work.
## 12. Contact Us
For questions, feedback, collaboration, or technical support, please contact us:
- **Mason**: [mason@limxdynamics.com](mailto:mason@limxdynamics.com)
- **Wayne Mao**: [waynemao@limxdynamics.com](mailto:waynemao@limxdynamics.com)
You can also:
- Open an issue on [GitHub](https://github.com/limxdynamics/FluxVLA/issues)
- Visit our documentation: [English](https://fluxvla.limxdynamics.com) | [中文](https://fluxvla.limxdynamics.com/zh/)
- Check out our Hugging Face page: [limxdynamics/FluxVLAEngine](https://huggingface.co/limxdynamics/FluxVLAEngine)
---
**Project Links:**
- 🌐 GitHub:
- 🤗 Hugging Face:
- 📖 English Documentation:
- 📖 中文文档: 
