model card: rewrite around the two acts; drop the delisted generations

README.md

@@ -4,61 +4,85 @@ tags:
   - robotics
   - vision-language-action
   - manipulation
-  - so-101
-  - quantization
-  - autoregressive-policy
+  - maniskill
 library_name: pytorch
 ---
 
-# MiniRT1 — <200MB Strictly-Autoregressive VLA for SO-101 Pick-and-Place
+# MiniRT
 
-Model checkpoints for the **MiniRT1** project. A visuomotor policy that generates
-6-DoF joint-delta actions **token-by-token (strictly autoregressive)**, trained by
-privileged teacher-student distillation + DAgger in ManiSkill3's SO-101
-`SO101PlaceCube-v1` environment.
+A small autoregressive vision-language-action policy for the SO-101 arm, and the hierarchy it needs
+to become selective.
 
-- **Code + full technical report:** https://github.com/charliechin424/MiniRT1
-- **Eval protocol:** `evaluation/eval_maniskill.py`, 300 episodes, seed 3000, `success_at_end`
+Code, evaluation harness and the full report: **https://github.com/charliechin424/MiniRT**
 
-## Checkpoints
+**Act one.** One 96 MB weight drives eight ManiSkill3 manipulation tasks from free-form English at
+90.3% mean success over 300 episodes per task.
 
-| File | success (300 ep) | size | description |
-|---|---|---|---|
-| `v6_dino_fp16.pt` | **92%** | 96 MB | project best — DINOv2-S backbone VLA (fp16) |
-| `v6_dino_int8_static_pg64.pt` | **92%** | 57.6 MB | W8A8 true integer-compute (chip bit-path, per-group-64 fixed-point); zero loss |
-| `v6_dino_w4a8.pt` | 91% | 57.6 MB | 4-bit weights / 8-bit activations, near-lossless |
-| `v4_fp16.pt` | 84% | 74 MB | ResNet18 backbone, no language |
-| `v5_lang_fp16.pt` | 83% | 74 MB | +frozen-MiniLM language token (instruction-conditioned VLA) |
-| `rl_expert_teacher.pt` | 86.6% | 1.2 MB | state-based PPO teacher (privileged 49-D state; distillation source) |
-| `task_chain_squint.npz` | — | 0.15 MB | PoE guided-decoding action-token Markov prior (Phase 8/9) |
+**Act two.** The same architecture scores 0% when the task requires *choosing* — "put the blue can in
+the bin" with four objects on the table. It reaches, grasps and lifts competently and takes the
+wrong object; colour grounding is a cross-modal binding it cannot learn at the motor level. Moving
+that job into a 0.1 M-parameter grounder that shares the policy's own vision backbone recovers 26%
+end-to-end, with no privileged knowledge of which object is which.
 
-int8/W4A8 checkpoints carry `quant`/`group_size`/`w_bits`/`a_bits` fields; the
-adapter (`ar_policy/policy_adapter.py`) rebuilds the integer-compute module
-structure automatically. See the GitHub repo for the exact loading and eval code.
+## Files
 
-## `ablation/` — intermediate checkpoints (reproduce the paper's tables)
+| file | what it is |
+|---|---|
+| `v7_mt_generalist_fp16.pt` | act one — one weight, eight tasks. 47.9 M params, 96 MB fp16. |
+| `v8_mt_colorsort_fp16.pt` | act one plus the ColorSort skill; the warm-start ancestor of the act-two policy. |
+| `colorsort/v15cs_frozen_fp16.pt` | act two — the delivered position-conditioned VLA. Frozen vision backbone so the grounder and the policy genuinely share one encoder: 48.1 M total. |
+| `colorsort/grounder_r4_mix20.pt` | act two — the MiniRT-Grounder heads (0.104 M params on a DINOv2-S backbone). Trained at `img_size=448`. |
 
-The root files above are the final deliverables. `ablation/` holds every
-intermediate model behind the report's ablation tables:
+## Numbers
 
-- `v1_fp16.pt` (1%, chunk=10 exposure-bias failure), `v2_fp16.pt` (45%), `v3_fp16.pt` (52%) — the training-recipe evolution
-- v4/v5 int8 variants: `{v4,v5}_int8_static_{pg64,pc}.pt`, `v5_int8_weightonly.pt`
-- v6 quantization sweep: `v6_int8_static_pc.pt`, `v6_w4a8_clip.pt`, `v6_w4a4_{rtn,clip,dyn}.pt` (Phase 9A low-bit study)
-- 9C data-scaling weak students: `v6_sub{200,800,3200}_fp16.pt` (trained on 200/800/3200 episodes; the "guidance vs data size" curve)
+Act one, `success_at_end` over 300 episodes (3 × 100, seed 3000), `physx_cpu`:
 
-## Usage
+| Task | success |
+|---|---|
+| ReachCube | 99.3 |
+| ReachCan | 99.0 |
+| LiftCube | 95.7 |
+| StackCube | 93.0 |
+| PlaceCube | 91.3 |
+| LiftCan | 89.3 |
+| PlaceCan | 84.7 |
+| StackCan | 70.3 |
+| **mean** | **90.3** |
 
-```bash
-# grab a checkpoint
-hf download charliechin424/MiniRT1 v6_dino_fp16.pt --local-dir ./ckpts
+Unseen instruction phrasings: seven of eight tasks show no degradation.
 
-# then, in the MiniRT1 repo (needs the `sim` conda env + squint env):
-MINIRT1_CKPT=./ckpts/v6_dino_fp16.pt \
-  python evaluation/eval_maniskill.py --policy ar_policy/policy_adapter.py:make_policy --episodes 300
-```
+Act two, 100 episodes, seed 3000:
 
-## Citation
+| | privileged ceiling | grounded closed loop | n=2 | n=3 |
+|---|---|---|---|---|
+| **frozen 48.1 M (delivered)** | 29% | **26%** | 41% | 10% |
 
-Built on [squint](https://github.com/aalmuzairee/squint) (SO-101 ManiSkill3 env),
-[LeRobot](https://github.com/huggingface/lerobot), [DINOv2](https://github.com/facebookresearch/dinov2).
-Methodology: Learning by Cheating (CoRL 2019), DAgger (AISTATS 2011), RT-1/OpenVLA action tokenization.
+Perception costs three points; insisting one encoder serve both the grounder and the policy costs
+about half the success rate. That is the trade-off, stated rather than hidden.
+
+## Running these
+
+The evaluation is **not** reproducible from the weights alone. It needs the ColorSort environment,
+which is a modification to an upstream repository and ships in the code repo as `squint_patch/`, and
+it needs several flags whose defaults are wrong for these checkpoints — `--min-area 6` in
+particular silently costs eleven points if left at its default. The README has the exact commands:
+
+**https://github.com/charliechin424/MiniRT**
+
+## Notes
+
+- Every number is measured on `physx_cpu`. `physx_gpu` diverges physically, per task and in either
+  direction; do not compare across backends.
+- The act-two checkpoint has catastrophically forgotten the eight base tasks. Its ancestor
+  `v8_mt_colorsort` has not. This is a design consequence — six generations of fine-tuning with zero
+  base-task replay — not a defect, and it is recorded here so that comparing the two checkpoints
+  does not lead to the wrong conclusion.
+- No integer-compute (int8) checkpoint of the act-two model is published. The conversion exists and
+  its calibration is correct, but its closed-loop behaviour has not been measured, and publishing it
+  on the strength of the single-task predecessor's result would be an unearned claim.
+
+## Credit
+
+Built on [squint](https://github.com/aalmuzairee/squint) (SO-101 ManiSkill3 environments),
+[ManiSkill3](https://github.com/haosulab/ManiSkill), [LeRobot](https://github.com/huggingface/lerobot)
+and [DINOv2](https://github.com/facebookresearch/dinov2).