GEAR-VLA-9B

⭐️ Project Page   |   📖 Paper   |   🗂️ Dataset

🔥 Introduction

GEAR-VLA is a vision-language-action framework designed to connect embodied perception, geometric reasoning, and robot control. This repository releases GEAR-VLA-9B, the vision-language model (VLM) checkpoint obtained after the Embodied Pretraining stage.

GEAR-VLA introduces three complementary designs:

🎯 Coarse-to-Fine Action Learning. The framework combines latent action representations from manipulation videos with discretized robot actions. This design first captures high-level interaction dynamics and subsequently learns precise embodiment-specific control.

🧭 Semantic-Aligned 3D Geometry Integration. A trainable VGGT branch extracts multi-view 3D geometry while preserving the semantic knowledge of the frozen 2D visual pathway. The two feature streams are aligned and fused before entering the language model.

🤖 Embodiment Canonicalization. Embodiment-aware state inputs and a unified relative end-effector action space confine robot-specific variation to the low-level interface. This design supports transfer across robot platforms without introducing robot-specific semantic prompts into the shared representation.

The released checkpoint provides the VLM foundation used by the complete GEAR-VLA system. It does not include the downstream DiT action expert, robot-state projector, or a complete continuous-control policy.

📚 Embodied Pretraining

Embodied Pretraining uses approximately 4.189 million general vision-language and embodied-perception samples, together with 23,027.6 hours of robot and egocentric manipulation data. All tasks are formulated as autoregressive token prediction within a unified training objective.

For robot trajectories with action annotations, the model predicts both FAST-style action tokens and latent action identifiers. For videos without action annotations, it learns from latent action identifiers. Pretraining was conducted for 350,000 iterations on 240 NVIDIA H200 GPUs, with a batch size of eight per GPU.

🏗️ Model Architecture

GEAR-VLA-9B uses a dual-branch visual architecture. The original 2D visual encoder remains frozen to preserve pretrained semantic representations, while VGGT serves as a trainable 3D spatial encoder for multi-view geometric reasoning. The current implementation uses the VGGT Aggregator and does not use its camera, depth, point, or tracking heads.

The 2D features H_2D and 3D features H_3D are concatenated along the feature dimension and passed through an expanded MLP projector. The projector weights associated with the 2D pathway are inherited from the pretrained VLM, whereas the newly introduced 3D pathway is zero-initialized. This initialization preserves the original visual-language behavior at the start of training while allowing geometry-aware features to be learned progressively.

📊 VLM Capability Evaluation

We evaluated the current GEAR-VLA-9B Embodied Pretraining checkpoint under the same evaluation protocols and obtained consistent results.

The evaluation covers EgoPlan2, Where2Place, RefSpatial-bench, ShareRobot-affordance, ShareRobot-trajectory, BLINK, EmbSpatial, ERQA, SAT, and CVBench. These benchmarks primarily assess embodied reasoning, spatial understanding, grounding, affordance prediction, and trajectory reasoning.

These scores characterize the embodied capabilities of the VLM checkpoint. They should not be interpreted as robot-task success rates for the complete continuous-control policy.

🏆 Complete GEAR-VLA System Results

The complete GEAR-VLA system, which includes downstream action-policy components that are not part of this release, achieves the following results reported in the paper:

Evaluation setting	Result
LIBERO	98.7%
LIBERO-Plus	88.7%
RoboTwin, clean	91.1%
RoboTwin, randomized	89.9%
AgileX	85.9%
LDT-01	81.0%

Across 6,360 trials involving 212 previously unseen objects, the complete system achieves a grasping success rate of 90.1%.

📦 Released Checkpoint

This repository contains the VLM parameters produced by the Embodied Pretraining stage. The checkpoint is intended for embodied visual-language understanding, geometric reasoning, and initialization of downstream GEAR-VLA training.

The release does not contain the DiT action expert, robot-state projector, embodiment-specific action heads, or downstream continuous-control policy. Reproducing the complete-system robot results therefore requires the remaining policy components and downstream training procedure described in the paper.

⚠️ Limitations

The VGGT branch and multi-stage training procedure increase computational and memory requirements. The current geometry integration also relies on multi-view inputs. Single-camera settings, mobile manipulation, and scenes with severe occlusion may require additional adaptation.

🚀 Quick Start

Using 🤗 Transformers to Chat

The checkpoint uses custom Transformers code and the VGGT backbone. Install the required packages before loading the model.

pip install -U "transformers>=4.48.3" huggingface_hub accelerate
pip install git+https://github.com/facebookresearch/vggt.git

Python code

import torch
from huggingface_hub import snapshot_download
from PIL import Image
from transformers import AutoModel, AutoProcessor


REPO_ID = "yuanzhang/GEAR-VLA"

# Download the complete checkpoint first because the custom processor loads
# its configuration and tokenizer from a local snapshot.
model_dir = snapshot_download(REPO_ID)

processor = AutoProcessor.from_pretrained(
    model_dir,
    trust_remote_code=True,
)
model = AutoModel.from_pretrained(
    model_dir,
    torch_dtype=torch.bfloat16,
    low_cpu_mem_usage=True,
    trust_remote_code=True,
    device_map="auto",
).eval()

# Supply one or more synchronized views of the same scene.
views = [
    Image.open("./view_1.jpg").convert("RGB"),
    Image.open("./view_2.jpg").convert("RGB"),
]

# The processor creates aligned inputs for both visual branches:
# pixel_values: [V, 3, 448, 448]
# vggt_images:  [1, V, 3, 518, 518]
image_inputs = processor.preprocess_images(views)
pixel_values = image_inputs["pixel_values"].to(
    device="cuda",
    dtype=torch.bfloat16,
)
vggt_images = image_inputs["vggt_images"].to(
    device="cuda",
    dtype=torch.bfloat16,
)

question = (
    "\n".join(["<image>"] * len(views))
    + "\nDescribe the scene and reason about the spatial relationships "
      "between the visible objects."
)
generation_config = {
    "do_sample": False,
    "max_new_tokens": 512,
}

response = model.chat(
    processor.tokenizer,
    pixel_values,
    question,
    generation_config,
    num_patches_list=[1] * len(views),
    vggt_images=vggt_images,
)
print(response)

For the architecture, training procedure, and evaluation protocol, see the paper.