| --- |
| license: mit |
| tags: |
| - robotics |
| - vision-language-action |
| - manipulation |
| - maniskill |
| library_name: pytorch |
| --- |
| |
| # MiniRT |
|
|
| A small autoregressive vision-language-action policy for the SO-101 arm, and the hierarchy it needs |
| to become selective. |
|
|
| Code, evaluation harness and the full report: **https://github.com/charliechin424/MiniRT** |
|
|
| **Act one.** One 96 MB weight drives eight ManiSkill3 manipulation tasks from free-form English at |
| 90.3% mean success over 300 episodes per task. |
|
|
| **Act two.** The same architecture scores 0% when the task requires *choosing* — "put the blue can in |
| the bin" with four objects on the table. It reaches, grasps and lifts competently and takes the |
| wrong object; colour grounding is a cross-modal binding it cannot learn at the motor level. Moving |
| that job into a 0.1 M-parameter grounder that shares the policy's own vision backbone recovers 26% |
| end-to-end, with no privileged knowledge of which object is which. |
|
|
| ## Files |
|
|
| | file | what it is | |
| |---|---| |
| | `v7_mt_generalist_fp16.pt` | act one — one weight, eight tasks. 47.9 M params, 96 MB fp16. | |
| | `v8_mt_colorsort_fp16.pt` | act one plus the ColorSort skill; the warm-start ancestor of the act-two policy. | |
| | `colorsort/v15cs_frozen_fp16.pt` | act two — the delivered position-conditioned VLA. Frozen vision backbone so the grounder and the policy genuinely share one encoder: 48.1 M total. | |
| | `colorsort/v15cs_frozen_int8.pt` | the same policy as W8A8 integer compute, 57.7 MB. Costs success — see below. Ships with `colorsort/v15cs_frozen_int8_params.json`, the per-layer quantisation parameter table. | |
| | `colorsort/grounder_r4_mix20.pt` | act two — the MiniRT-Grounder heads (0.104 M params on a DINOv2-S backbone). Trained at `img_size=448`. | |
|
|
| ## Numbers |
|
|
| Act one, `success_at_end` over 300 episodes (3 × 100, seed 3000), `physx_cpu`: |
|
|
| | Task | success | |
| |---|---| |
| | ReachCube | 99.3 | |
| | ReachCan | 99.0 | |
| | LiftCube | 95.7 | |
| | StackCube | 93.0 | |
| | PlaceCube | 91.3 | |
| | LiftCan | 89.3 | |
| | PlaceCan | 84.7 | |
| | StackCan | 70.3 | |
| | **mean** | **90.3** | |
|
|
| Unseen instruction phrasings: seven of eight tasks show no degradation. |
|
|
| Act two, 100 episodes, seed 3000: |
|
|
| | | privileged ceiling | grounded closed loop | n=2 | n=3 | |
| |---|---|---|---|---| |
| | **frozen 48.1 M (delivered)** | 29% | **26%** | 41% | 10% | |
|
|
| Perception costs three points; insisting one encoder serve both the grounder and the policy costs |
| about half the success rate. That is the trade-off, stated rather than hidden. |
|
|
| ## Running these |
|
|
| The evaluation is **not** reproducible from the weights alone. It needs the ColorSort environment, |
| which is a modification to an upstream repository and ships in the code repo as `squint_patch/`, and |
| it needs several flags whose defaults are wrong for these checkpoints — `--min-area 6` in |
| particular silently costs eleven points if left at its default. The README has the exact commands: |
|
|
| **https://github.com/charliechin424/MiniRT** |
|
|
| ## Notes |
|
|
| - Every number is measured on `physx_cpu`. `physx_gpu` diverges physically, per task and in either |
| direction; do not compare across backends. |
| - The act-two checkpoint has catastrophically forgotten the eight base tasks. Its ancestor |
| `v8_mt_colorsort` has not. This is a design consequence — six generations of fine-tuning with zero |
| base-task replay — not a defect, and it is recorded here so that comparing the two checkpoints |
| does not lead to the wrong conclusion. |
| - **The int8 build costs success and is labelled so.** fp16 26/100, int8 22/100; at the object level |
| 58% against 47% over 249 objects. Both acceptance gates still pass, so it is usable — choose it for |
| the 40% size reduction, not because it is free. The four-point headline gap sits on the noise band |
| of a 100-episode run and read alone would suggest no effect; the paired per-seed comparison |
| (identical scenes and wording) shows int8 worse on 37 seeds and better on 17, p ~ 0.009. On the |
| single-task predecessor quantisation was genuinely free, which is why this is worth stating. |
|
|
| ## Credit |
|
|
| Built on [squint](https://github.com/aalmuzairee/squint) (SO-101 ManiSkill3 environments), |
| [ManiSkill3](https://github.com/haosulab/ManiSkill), [LeRobot](https://github.com/huggingface/lerobot) |
| and [DINOv2](https://github.com/facebookresearch/dinov2). |
|
|
