README.md

# MagicBot-VGA

This repository documents how to evaluate our RoboTwin model
[`zaleni/MagicBot-VGA-Robotwin`](https://huggingface.co/zaleni/MagicBot-VGA-Robotwin)
with the MagicBot-VGA codebase.

[![Repository](https://img.shields.io/badge/Repository-GitHub-181717?logo=github)](https://github.com/zaleni/MagicBot-VGA)
[![Model](https://img.shields.io/badge/Model-HuggingFace-FFD21E?logo=huggingface&logoColor=000000)](https://huggingface.co/zaleni/MagicBot-VGA-Robotwin)

This README focuses on RoboTwin 2.0 environment preparation and evaluation.

It covers:

- MagicBot environment installation
- RoboTwin evaluation setup
- required external model assets
- single-task evaluation
- 50-task randomized evaluation
- CVPR 2026 RoboTwin Track 11-task evaluation
- submission package generation for the leaderboard workflow

## 1. Requirements

The codebase is built and tested with:

- Python 3.10
- CUDA 12.8
- PyTorch 2.7.1

We recommend using a Linux machine with NVIDIA GPUs.

## 2. Install the MagicBot Base Environment

Clone the repository:

```bash
git clone https://github.com/zaleni/MagicBot-VGA.git
cd MagicBot-VGA
```

Create a conda environment:

```bash
conda create -y -n magicbot python=3.10
conda activate magicbot
pip install --upgrade pip
```

Install the basic system dependencies used by the codebase:

```bash
conda install -c conda-forge ffmpeg=7.1.1 svt-av1 -y
```

Install PyTorch for CUDA 12.8:

```bash
pip install torch==2.7.1 torchvision==0.22.1 torchaudio==2.7.1 \
  --index-url https://download.pytorch.org/whl/cu128
```

Install Python dependencies:

```bash
pip install torchcodec numpy scipy transformers==4.57.1 mediapy loguru pytest omegaconf
pip install -e .
```

## 3. Qwen3-VL Dependency

For `CubeV2`, the recommended dependency is the official Hugging Face `Qwen3-VL`
implementation provided by `transformers>=4.57.0`.

In this repository, `CubeV2` imports Qwen3-VL directly from:

```python
from transformers.models.qwen3_vl import modeling_qwen3_vl
from transformers.models.qwen3_vl import Qwen3VLForConditionalGeneration, Qwen3VLTextModel
```

So for standard evaluation, you do not need to patch `transformers` if your environment
already uses a recent enough official version such as `transformers==4.57.1`.

This repo also contains a vendored replacement file under:

```text
src/lerobot/policies/cubev2/transformers_replace/models/qwen3_vl/modeling_qwen3_vl.py
```

That file is best understood as a repo-side override copy. Most users evaluating
`zaleni/MagicBot-VGA-Robotwin` should not need it unless they intentionally want to
reproduce a specific local patched behavior.

## 4. Prepare RoboTwin for Evaluation

This section is specifically for RoboTwin evaluation. If you only want to load the
model or run other parts of the codebase, the extra RoboTwin setup below is not required.

### Option A: initialize the bundled RoboTwin submodule

```bash
git submodule update --init third_party/RoboTwin
```

### Option B: copy an existing RoboTwin checkout

You do not have to download RoboTwin from scratch if you already have a prepared copy.
You can copy it into this repository instead.

The evaluation code assumes RoboTwin is located exactly at:

```text
<repo_root>/third_party/RoboTwin
```

So a valid layout looks like:

```text
MagicBot-VGA/
  evaluation/
  launch/
  src/
  third_party/
    RoboTwin/
```

If your RoboTwin directory already exists elsewhere, either:

- copy it to `third_party/RoboTwin`, or
- create a symlink at `third_party/RoboTwin` pointing to your existing RoboTwin directory

This path requirement comes from the evaluation code, which imports RoboTwin modules
and task configs from `third_party/RoboTwin` directly.

### Install RoboTwin-specific system dependency

RoboTwin rendering requires Vulkan:

```bash
sudo apt install -y libvulkan1 mesa-vulkan-drivers vulkan-tools
```

### Install RoboTwin Python dependencies and assets

```bash
cp evaluation/RoboTwin/requirements.txt third_party/RoboTwin/script/requirements.txt
cd third_party/RoboTwin
bash script/_install.sh
bash script/_download_assets.sh
cd ../../
```

For more RoboTwin installation details, you can also refer to the official documentation:
https://robotwin-platform.github.io/doc/usage/robotwin-install.html

## 5. Prepare External Model Assets

The released checkpoint `zaleni/MagicBot-VGA-Robotwin` is intended to be lightweight.
For RoboTwin action evaluation, you should provide the external backbone/tokenizer assets explicitly.

Recommended values:

- Qwen3-VL backbone and processor: `Qwen/Qwen3-VL-2B-Instruct`
- Cosmos tokenizer: `nvidia/Cosmos-Tokenizer-CI8x8`

You can use either:

- public Hugging Face repo ids
- local directories downloaded in advance

Example for offline/local usage:

```bash
QWEN3_VL_PATH=/path/to/Qwen3-VL-2B-Instruct
COSMOS_TOKENIZER_PATH=/path/to/Cosmos-Tokenizer-CI8x8
```

For standard RoboTwin action evaluation, we recommend disabling DA3 teacher instantiation:

```bash
DISABLE_DA3_TEACHER_FOR_EVAL=true
```

This avoids loading the frozen DA3 teacher during evaluation while keeping the policy architecture compatible.

## 6. Single-Task Evaluation

The most direct way is to call `evaluation/RoboTwin/inference.py` on a single RoboTwin task.

Example: evaluate task `0` (`adjust_bottle`) on `demo_clean`:

```bash
cd third_party/RoboTwin

python ../../evaluation/RoboTwin/inference.py \
  --args.ckpt_path zaleni/MagicBot-VGA-Robotwin \
  --args.video_dir ../../evaluation/RoboTwin/output_magicbot/demo_clean/task_00 \
  --args.task_config demo_clean \
  --args.task_idx 0 \
  --args.action_mode delta \
  --args.stats_key aloha \
  --args.dtype bfloat16 \
  --args.qwen3_vl_pretrained_path Qwen/Qwen3-VL-2B-Instruct \
  --args.qwen3_vl_processor_path Qwen/Qwen3-VL-2B-Instruct \
  --args.cosmos_tokenizer_path_or_name nvidia/Cosmos-Tokenizer-CI8x8 \
  --args.disable_3d_teacher_for_eval
```

If you use local asset directories, replace the public repo ids with your local paths.

Important arguments:

- `--args.ckpt_path`: model repo id or local `pretrained_model` directory
- `--args.task_config`: `demo_clean` or `demo_randomized`
- `--args.task_idx`: task index in `evaluation/RoboTwin/inference.py`
- `--args.action_mode`: usually `delta` for this model
- `--args.stats_key`: usually `aloha` for RoboTwin
- `--args.dtype`: `bfloat16` is recommended on modern GPUs

Outputs are written to `--args.video_dir`, including:

- replay videos
- `summary.json`
- `summary.txt`

## 7. 50-Task Randomized Evaluation

For batch evaluation on RoboTwin randomized tasks, use:

```bash
PRETRAINED_CKPT=zaleni/MagicBot-VGA-Robotwin \
QWEN3_VL_PRETRAINED_PATH=Qwen/Qwen3-VL-2B-Instruct \
QWEN3_VL_PROCESSOR_PATH=Qwen/Qwen3-VL-2B-Instruct \
COSMOS_TOKENIZER_PATH_OR_NAME=nvidia/Cosmos-Tokenizer-CI8x8 \
DISABLE_DA3_TEACHER_FOR_EVAL=true \
GPU_IDS=0,1 \
MAX_JOBS_PER_GPU=2 \
bash evaluation/RoboTwin/eval_randomized_50.sh
```

Useful environment variables:

- `PRETRAINED_CKPT`: model repo id or local checkpoint directory
- `GPU_IDS`: comma-separated GPU ids, for example `0,1,2,3`
- `MAX_JOBS_PER_GPU`: parallel RoboTwin jobs per GPU
- `TASK_CONFIG`: defaults to `demo_randomized`
- `TEST_NUM`: number of episodes per task
- `DTYPE`: `bfloat16` or `float32`
- `BASE_OUTPUT_PATH`: output root directory

This script writes:

- per-task logs and videos under `tasks/`
- aggregated `summary.json`
- aggregated `summary.txt`

## 8. Evaluate a Continuous Task Range

`eval_randomized_50.sh` supports continuous ranges through:

- `START_TASK_IDX`
- `TASK_COUNT`

Example: evaluate tasks `10` to `19`:

```bash
PRETRAINED_CKPT=zaleni/MagicBot-VGA-Robotwin \
QWEN3_VL_PRETRAINED_PATH=Qwen/Qwen3-VL-2B-Instruct \
QWEN3_VL_PROCESSOR_PATH=Qwen/Qwen3-VL-2B-Instruct \
COSMOS_TOKENIZER_PATH_OR_NAME=nvidia/Cosmos-Tokenizer-CI8x8 \
DISABLE_DA3_TEACHER_FOR_EVAL=true \
START_TASK_IDX=10 \
TASK_COUNT=10 \
bash evaluation/RoboTwin/eval_randomized_50.sh
```

## 9. Evaluate the CVPR 2026 RoboTwin Track 11-Task Subset

For the Hugging Face leaderboard
[`open-gigaai/CVPR-2026-RoboTwin-Track-LeaderBoard`](https://huggingface.co/spaces/open-gigaai/CVPR-2026-RoboTwin-Track-LeaderBoard),
we use the following 11-task subset:

```text
[2, 3, 9, 10, 12, 15, 17, 25, 28, 30, 44]
```

The exact task names in `evaluation/RoboTwin/inference.py` are:

- `blocks_ranking_rgb`
- `blocks_ranking_size`
- `handover_mic`
- `hanging_mug`
- `move_can_pot`
- `move_stapler_pad`
- `open_microwave`
- `place_can_basket`
- `place_dual_shoes`
- `place_fan`
- `stack_blocks_three`

The current batch script does not take a sparse task list directly, so the recommended approach is to run a shell loop:

```bash
cd third_party/RoboTwin

TASKS=(2 3 9 10 12 15 17 25 28 30 44)
for t in "${TASKS[@]}"; do
  python ../../evaluation/RoboTwin/inference.py \
    --args.ckpt_path zaleni/MagicBot-VGA-Robotwin \
    --args.video_dir ../../evaluation/RoboTwin/output_magicbot/custom_subset/task_${t} \
    --args.task_config demo_randomized \
    --args.task_idx "${t}" \
    --args.action_mode delta \
    --args.stats_key aloha \
    --args.dtype bfloat16 \
    --args.qwen3_vl_pretrained_path Qwen/Qwen3-VL-2B-Instruct \
    --args.qwen3_vl_processor_path Qwen/Qwen3-VL-2B-Instruct \
    --args.cosmos_tokenizer_path_or_name nvidia/Cosmos-Tokenizer-CI8x8 \
    --args.disable_3d_teacher_for_eval
done
```

This produces one output directory per task, each containing replay videos plus `summary.json` and `summary.txt`.

## 10. Package the 11-Task Submission and Export Success Rates

After you finish the randomized evaluation run, you can convert those 11 tasks into a submission-style folder with:

```bash
python util_scripts/package_robotwin_submission.py \
  --run /path/to/output_randomized_50/<run_name>/summary.txt \
  --dst /path/to/output_randomized_50/<run_name>/submission_package \
  --overwrite
```

If you also want to bundle a policy folder, add:

```bash
  --policy-dir /path/to/policy/Your_Policy
```

The packaging script will:

- create `submission_package/<task_name>/episode0.mp4`, `episode1.mp4`, ...
- preserve the 11-task ordering by task index
- write `package_manifest.txt`
- write `selected_task_summary.json`
- write `selected_task_summary.txt`

The selected-task summary files include:

- per-task `success_rate`
- per-task `success_count` and `test_num`
- `avg_task_success_rate` across the 11 tasks
- `overall_episode_success_rate` across all episodes in the 11-task subset

This is useful when you want a leaderboard-facing summary for the competition subset rather than the full randomized-50 report.

## 11. Task Index Reference

Task indices are defined in [`evaluation/RoboTwin/inference.py`](evaluation/RoboTwin/inference.py).

For example:

- `0`: `adjust_bottle`
- `2`: `blocks_ranking_rgb`
- `3`: `blocks_ranking_size`
- `9`: `handover_mic`
- `10`: `hanging_mug`
- `12`: `move_can_pot`
- `15`: `move_stapler_pad`
- `17`: `open_microwave`
- `25`: `place_can_basket`
- `28`: `place_dual_shoes`
- `30`: `place_fan`
- `44`: `stack_blocks_three`

## 12. Common Notes

- `inference.py` can load checkpoints from either a local directory or a Hugging Face repo id.
- If your server cannot access Hugging Face online, download the external assets in advance and pass local paths.
- If you use the lightweight checkpoint release for action evaluation, keeping `--args.disable_3d_teacher_for_eval` enabled is recommended.
- If you want to inspect reconstructed future images during inference, enable `--args.decode_image_flag`, though this is not required for standard RoboTwin scoring.

## 13. Model Link

Released RoboTwin checkpoint:

- https://huggingface.co/zaleni/MagicBot-VGA-Robotwin

## 14. Acknowledgments

MagicBot-VGA is developed on top of the excellent InternVLA framework. Our codebase
started from that foundation and has since been substantially modified and extended
for our own model architecture, training pipeline, and evaluation workflow.

We sincerely thank the [InternVLA](https://github.com/InternRobotics/InternVLA-A1)
authors and contributors for open-sourcing their framework and making follow-up
research and development much easier.

We also thank the following open-source projects:

- [InternVLA](https://github.com/InternRobotics/InternVLA-A1)
- [LeRobot](https://github.com/huggingface/lerobot)
- [RoboTwin](https://github.com/RoboTwin-Platform/RoboTwin)
- [Qwen3-VL](https://github.com/QwenLM/Qwen3-VL)
- [NVIDIA Cosmos](https://github.com/nvidia-cosmos)