---
license: mit
tags:
  - robotics
  - vision-language-action
  - manipulation
  - maniskill
library_name: pytorch
---

# MiniRT

A small autoregressive vision-language-action policy for the SO-101 arm, and the hierarchy it needs
to become selective.

Code, evaluation harness and the full report: **https://github.com/charliechin424/MiniRT**

**Act one.** One 96 MB weight drives eight ManiSkill3 manipulation tasks from free-form English at
90.3% mean success over 300 episodes per task.

**Act two.** The same architecture scores 0% when the task requires *choosing* — "put the blue can in
the bin" with four objects on the table. It reaches, grasps and lifts competently and takes the
wrong object; colour grounding is a cross-modal binding it cannot learn at the motor level. Moving
that job into a 0.1 M-parameter grounder that shares the policy's own vision backbone recovers 26%
end-to-end, with no privileged knowledge of which object is which.

## Files

| file | what it is |
|---|---|
| `v7_mt_generalist_fp16.pt` | act one — one weight, eight tasks. 47.9 M params, 96 MB fp16. |
| `v8_mt_colorsort_fp16.pt` | act one plus the ColorSort skill; the warm-start ancestor of the act-two policy. |
| `colorsort/v15cs_frozen_fp16.pt` | act two — the delivered position-conditioned VLA. Frozen vision backbone so the grounder and the policy genuinely share one encoder: 48.1 M total. |
| `colorsort/v15cs_frozen_int8.pt` | the same policy as W8A8 integer compute, 57.7 MB. Costs success — see below. Ships with `colorsort/v15cs_frozen_int8_params.json`, the per-layer quantisation parameter table. |
| `colorsort/grounder_r4_mix20.pt` | act two — the MiniRT-Grounder heads (0.104 M params on a DINOv2-S backbone). Trained at `img_size=448`. |

## Numbers

Act one, `success_at_end` over 300 episodes (3 × 100, seed 3000), `physx_cpu`:

| Task | success |
|---|---|
| ReachCube | 99.3 |
| ReachCan | 99.0 |
| LiftCube | 95.7 |
| StackCube | 93.0 |
| PlaceCube | 91.3 |
| LiftCan | 89.3 |
| PlaceCan | 84.7 |
| StackCan | 70.3 |
| **mean** | **90.3** |

Unseen instruction phrasings: seven of eight tasks show no degradation.

Act two, 100 episodes, seed 3000:

| | privileged ceiling | grounded closed loop | n=2 | n=3 |
|---|---|---|---|---|
| **frozen 48.1 M (delivered)** | 29% | **26%** | 41% | 10% |

Perception costs three points; insisting one encoder serve both the grounder and the policy costs
about half the success rate. That is the trade-off, stated rather than hidden.

## Running these

The evaluation is **not** reproducible from the weights alone. It needs the ColorSort environment,
which is a modification to an upstream repository and ships in the code repo as `squint_patch/`, and
it needs several flags whose defaults are wrong for these checkpoints — `--min-area 6` in
particular silently costs eleven points if left at its default. The README has the exact commands:

**https://github.com/charliechin424/MiniRT**

## Notes

- Every number is measured on `physx_cpu`. `physx_gpu` diverges physically, per task and in either
  direction; do not compare across backends.
- The act-two checkpoint has catastrophically forgotten the eight base tasks. Its ancestor
  `v8_mt_colorsort` has not. This is a design consequence — six generations of fine-tuning with zero
  base-task replay — not a defect, and it is recorded here so that comparing the two checkpoints
  does not lead to the wrong conclusion.
- **The int8 build costs success and is labelled so.** fp16 26/100, int8 22/100; at the object level
  58% against 47% over 249 objects. Both acceptance gates still pass, so it is usable — choose it for
  the 40% size reduction, not because it is free. The four-point headline gap sits on the noise band
  of a 100-episode run and read alone would suggest no effect; the paired per-seed comparison
  (identical scenes and wording) shows int8 worse on 37 seeds and better on 17, p ~ 0.009. On the
  single-task predecessor quantisation was genuinely free, which is why this is worth stating.

## Credit

Built on [squint](https://github.com/aalmuzairee/squint) (SO-101 ManiSkill3 environments),
[ManiSkill3](https://github.com/haosulab/ManiSkill), [LeRobot](https://github.com/huggingface/lerobot)
and [DINOv2](https://github.com/facebookresearch/dinov2).