OpenRAL
/

rskill-molmoact2-multi-so101-nf4

@@ -1,192 +1,228 @@
 ---
-library_name: transformers
 tags:
-  - molmoact2
-  - robotics
-  - image-text-to-text
-  - so100
-  - so101
 ---
-<img src="assets/MolmoAct2.svg" alt="MolmoAct Logo" height="50">
-# **MolmoAct2-SO100_101**
-MolmoAct2 is an open vision-language-action model for robot control. It builds on Molmo2-ER and attaches a flow-matching continuous action expert that conditions on the VLM key-value cache through a per-layer connection.
-This checkpoint is fine-tuned on the SO-100/101 mixture with absolute joint-pose control and annotated language instructions. It is intended for both further fine-tuning and SO-100/101 policy inference.
-## Quick Links
-- 📂 Models: [Models](https://huggingface.co/collections/allenai/molmoact2-models), [Finetuned Models](https://huggingface.co/collections/allenai/molmoact2-finetuned-models)
-- 📂 Datasets: [MolmoAct2-BimanualYAM Dataset](https://huggingface.co/collections/allenai/molmoact2-datasets), [MolmoAct2 Datasets](https://huggingface.co/collections/allenai/molmoact2-datasets), [Molmo2-ER Datasets](https://huggingface.co/collections/allenai/molmo2-er-datasets)
-- 📄 Paper: [arXiv:2605.02881](https://arxiv.org/abs/2605.02881)
-- 💻 Code: [allenai/molmoact2](https://github.com/allenai/molmoact2)
-- 🎥 Blog Post: [MolmoAct2](https://allenai.org/blog/molmoact2)
-## Intended Use
-Use this checkpoint for SO-100/101 inference or for further fine-tuning. Dataset normalization metadata is stored in `norm_stats.json`. pass `norm_tag="so100_so101_molmoact2"` at inference time.
-Continuous action prediction is the intended and recommended inference mode. Discrete action prediction is exposed for parity and debugging, but we use continuous actions by default.
-## Install
-```bash
-pip install torch transformers pillow numpy huggingface_hub
-```
-## Sample Input
-This sample comes from `Beegbrain/pick_lemon_and_drop_in_bowl`, episode 0, frame 0. Camera order for this checkpoint does not matter. random camera order is acceptable.
-| Realsense Top RGB | Realsense Side RGB |
 | --- | --- |
-| ![Sample realsense top RGB](assets/sample_realsense_top_rgb.png) | ![Sample realsense side RGB](assets/sample_realsense_side_rgb.png) |
 ```python
-from huggingface_hub import hf_hub_download
-from PIL import Image
-import numpy as np
-repo_id = "allenai/MolmoAct2-SO100_101"
-top_rgb = Image.open(
-    hf_hub_download(repo_id, "assets/sample_realsense_top_rgb.png")
-).convert("RGB")
-side_rgb = Image.open(
-    hf_hub_download(repo_id, "assets/sample_realsense_side_rgb.png")
-).convert("RGB")
-task = "Move the arm towards the lemon, grasp it, lift it up, and drop it into the red bowl."
-robot_state = np.array(
-    [
-        -0.52734375,
-        189.140625,
-        181.40625,
-        60.64453125,
-        -3.603515625,
-        1.0971786975860596,
-    ],
-    dtype=np.float32,
-)
-```
-## Continuous Actions
-```python
-import numpy as np
-import torch
-from huggingface_hub import hf_hub_download
-from PIL import Image
-from transformers import AutoModelForImageTextToText, AutoProcessor
-repo_id = "allenai/MolmoAct2-SO100_101"
-top_rgb = Image.open(
-    hf_hub_download(repo_id, "assets/sample_realsense_top_rgb.png")
-).convert("RGB")
-side_rgb = Image.open(
-    hf_hub_download(repo_id, "assets/sample_realsense_side_rgb.png")
-).convert("RGB")
-task = "Move the arm towards the lemon, grasp it, lift it up, and drop it into the red bowl."
-robot_state = np.array(
-    [
-        -0.52734375,
-        189.140625,
-        181.40625,
-        60.64453125,
-        -3.603515625,
-        1.0971786975860596,
-    ],
-    dtype=np.float32,
-)
-processor = AutoProcessor.from_pretrained(repo_id, trust_remote_code=True)
-model = AutoModelForImageTextToText.from_pretrained(
-    repo_id,
-    trust_remote_code=True,
-    dtype=torch.float32,
-).to("cuda").eval()
-out = model.predict_action(
-    processor=processor,
-    images=[top_rgb, side_rgb],
-    task=task,
-    state=robot_state,
-    norm_tag="so100_so101_molmoact2",
-    inference_action_mode="continuous",
-    enable_depth_reasoning=False,
-    num_steps=10,
-    normalize_language=True,
-    enable_cuda_graph=True,
-)
-actions = out.actions
 ```
-MolmoAct2 was trained with mixed precision. For our reported experiments, we ran inference in `float32`. This path uses the most GPU memory: roughly 26GB with CUDA graph enabled, or around 24GB without CUDA graph.
-If you have a GPU with less memory, you can run inference with `bfloat16` instead:
-```python
-model = AutoModelForImageTextToText.from_pretrained(
-    repo_id,
-    trust_remote_code=True,
-    dtype=torch.bfloat16,
-).to("cuda").eval()
-with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16):
-    out = model.predict_action(...)
 ```
-Using `bfloat16` is much more memory efficient and can run under 16GB of GPU memory in our tests. It usually does not hurt performance much.
-Images may be PIL images or RGB arrays. Camera order does not need to be fixed for this checkpoint. random camera order is acceptable. `state` is the raw robot state, and actions are returned in robot scale.
-`normalize_language=True` is the default. It lowercases the task string and removes trailing sentence punctuation to match training preprocessing. Set it to `False` if you need to preserve the task text exactly.
-`enable_cuda_graph=True` is the default. The first few calls can be slow because the model warms up and captures CUDA graphs. run several random warm-up calls before measuring deployment latency. `num_steps` controls the continuous flow solver and defaults to the checkpoint config value, 10.
-Depth reasoning is disabled for this checkpoint. Calling `enable_depth_reasoning=True` will raise an error.
-## Discrete Actions
-Discrete action inference requires a caller-provided action tokenizer. It is not saved in this repository. Discrete mode decodes action tokens directly. the continuous action expert is not used.
-```python
-action_tokenizer = AutoProcessor.from_pretrained(
-    "allenai/MolmoAct2-FAST-Tokenizer",
-    trust_remote_code=True,
-)
-out = model.predict_action(
-    processor=processor,
-    images=[top_rgb, side_rgb],
-    task=task,
-    state=robot_state,
-    norm_tag="so100_so101_molmoact2",
-    inference_action_mode="discrete",
-    action_tokenizer=action_tokenizer,
-    enable_depth_reasoning=False,
-)
-```
-## Model and Hardware Safety
-MolmoAct2 generate robot actions from visual observations and language instructions, but their behavior may vary across embodiments, environments, and hardware configurations. Users should carefully validate model outputs before deployment, especially when operating physical robots or other actuated systems. Where possible, actions should be monitored through interpretable intermediate outputs (adaptive depth map), simulation rollouts, action limits, or other safety checks before execution on hardware. The model’s action space should be bounded by the training data, robot controller limits, and task-specific safety constraints, including limits on speed, workspace, torque, and contact force. Users should follow the hardware manufacturer’s safety guidelines, use appropriate emergency-stop mechanisms, and operate the system only in a safely configured environment with human supervision.
-## Citation
-```bibtex
-@misc{fang2026molmoact2actionreasoningmodels,
-      title={MolmoAct2: Action Reasoning Models for Real-world Deployment},
-      author={Haoquan Fang and Jiafei Duan and Donovan Clay and Sam Wang and Shuo Liu and Weikai Huang and Xiang Fan and Wei-Chuan Tsai and Shirui Chen and Yi Ru Wang and Shanli Xing and Jaemin Cho and Jae Sung Park and Ainaz Eftekhar and Peter Sushko and Karen Farley and Angad Wadhwa and Cole Harrison and Winson Han and Ying-Chun Lee and Eli VanderBilt and Rose Hendrix and Suveen Ellawela and Lucas Ngoo and Joyce Chai and Zhongzheng Ren and Ali Farhadi and Dieter Fox and Ranjay Krishna},
-      year={2026},
-      eprint={2605.02881},
-      archivePrefix={arXiv},
-      primaryClass={cs.RO},
-      url={https://arxiv.org/abs/2605.02881},
-}
-```

 ---
+language:
+- en
+license: apache-2.0
+pipeline_tag: robotics
 tags:
+- OpenRAL
+- rskill
+- molmoact2
+- vision-language-action
+- nf4
+- 4-bit
+- so100_follower
+- so101_follower
+- transformers
+- vla
+- so101
+- so100
+- manipulation
+base_model:
+- allenai/MolmoAct2-SO100_101
+base_model_relation: quantized
+inference: false
 ---
+# rskill-molmoact2-so101-nf4
+> **OpenRAL rSkill** — MolmoAct2 (Ai2's open action reasoning model: a
+> Molmo2-ER embodied-reasoning VLM backbone with a flow-matching
+> continuous-action expert) finetuned on the
+> [SO-100/SO-101](https://huggingface.co/allenai/MolmoAct2-SO100_101) teleop
+> mixture and NF4-quantized so the ~5.5 B-param model fits an 8 GB GPU.
+> Robots: SO-100 and SO-101 follower arms. **Apache-2.0 weights** — commercial
+> use permitted.
+This package wraps `hf://OpenRAL/rskill-molmoact2-so101-nf4` (an
+NF4-quantized mirror of `allenai/MolmoAct2-SO100_101`) with a `rskill.yaml`
+manifest that adds capability checking, license surfacing, latency budgets,
+and local registry integration. It does **not** copy model weights — they
+live on the Hub.
+> **Required sim config knob:** this checkpoint uses normalization statistics
+> tagged `"so100_so101_molmoact2"`. Any `SimEnvironment` config that drives
+> this rSkill must set `vla.extra.norm_tag: "so100_so101_molmoact2"` —
+> omitting it silently applies the adapter's default `"libero"` norm stats and
+> produces garbage actions.
+## What this skill does
+Performs tabletop manipulation — picking, placing, grasping, and transporting
+objects — on the SO-100 and SO-101 follower arms. The MolmoAct2 backbone
+reasons about the scene in 3D and the flow-matching action expert emits a
+continuous absolute joint-position action chunk that the adapter replays one
+step at a time.
+| Field | Value |
+| --- | --- |
+| Actions | pick, place, pick_and_place, grasp |
+| Objects | diverse tabletop objects |
+| Scenes  | tabletop |
+| Embodiments | `so100_follower`, `so101_follower` |
+## How it works
+MolmoAct2 grafts a modern DiT-style flow-matching continuous-action expert
+onto the Molmo2-ER discrete-token VLM via per-layer KV-cache conditioning
+(arXiv:2605.02881). It ships as a transformers **custom-code** model
+(`trust_remote_code`, `auto_map` → `MolmoAct2ForConditionalGeneration`), not a
+lerobot policy. The OpenRAL `molmoact2` adapter
+(`python/sim/src/openral_sim/policies/molmoact2.py`) loads it via
+`AutoModelForImageTextToText.from_pretrained` + `AutoProcessor` from the
+manifest's `source_repo`, NF4-quantizes every Linear with ≥4M weight elements
+via bitsandbytes, overlays the prequantized pack from `weights_uri`, then drives
+it through the checkpoint's own `predict_action(...)` continuous-action API. Two
+RGB camera streams plus a 6-D proprio state go in; a `(chunk_size, 6)` absolute
+joint-position chunk comes out, replayed one step at a time and re-inferred
+when the queue empties.
+The adapter reads `norm_tag` from `vla.extra.norm_tag`; this rSkill requires
+`"so100_so101_molmoact2"` — set it explicitly in every `SimEnvironment` config.
+### Observation → action contract
+| Direction | Key | Shape | Notes |
+| --- | --- | --- | --- |
+| in | `observation.images.camera1` | `(1, 3, H, W) float32 [0,1]` | overhead view (→ model `top`) |
+| in | `observation.images.camera2` | `(1, 3, H, W) float32 [0,1]` | wrist/side view (→ model `side`) |
+| in | `observation.state`           | `(1, 6)` float32                | SO-101 6-D joint positions (rad) |
+| out | action chunk                  | `(10, 6)` float32               | absolute joint-position targets |
+**Camera aliases (for `so101_box` scene):** `oak_top → top`, `wrist → side`.
+Override per-scene via `vla.extra` if your scene uses different camera names.
+## Upstream model / training
+The wrapped weights come from Ai2's `allenai/MolmoAct2-SO100_101` checkpoint —
+the base `allenai/MolmoAct2` foundation model finetuned on the SO-100/SO-101
+teleop dataset mixture with absolute joint-pose control and annotated language
+instructions. This rSkill repackages an NF4-quantized mirror of those weights;
+it does **not** retrain or copy the full-precision weights.
+| Field | Value |
+| --- | --- |
+| Source repo | [`allenai/MolmoAct2-SO100_101`](https://huggingface.co/allenai/MolmoAct2-SO100_101) |
+| Base model  | [`allenai/MolmoAct2`](https://huggingface.co/allenai/MolmoAct2) |
+| Paper       | [arxiv:2605.02881](https://arxiv.org/abs/2605.02881) — *MolmoAct2: Action Reasoning Models for Real-world Deployment* |
+| License     | apache-2.0 (code + weights) |
+| Parameters  | ~5.5 B |
+| Training data | SO-100/SO-101 teleop mixture (absolute joint-pose, annotated language) |
+| norm_tag    | `"so100_so101_molmoact2"` — **required** in `vla.extra.norm_tag` |
+## Supported robots
+| Robot | Embodiment tag | Status | Notes |
+| --- | --- | --- | --- |
+| SO-101 follower | `so101_follower` | ⚡ experimental | Native training embodiment; numbers not yet locally reproduced. |
+| SO-100 follower | `so100_follower` | ⚡ experimental | Shares identical 6-DoF kinematics; covered by training mixture. |
+## Sensors required
+| Key | Modality | Min resolution | Format |
+| --- | --- | --- | --- |
+| `observation.images.camera1` | RGB | 224 × 224 | `float32` |
+| `observation.images.camera2` | RGB | 224 × 224 | `float32` |
+| `observation.state`          | proprioception | (6,) | `float32` |
+## Manifest summary
+| Field | Value |
 | --- | --- |
+| `name` | `OpenRAL/rskill-molmoact2-so101-nf4` |
+| `version` | `0.1.0` |
+| `license` | `apache-2.0` |
+| `role` | `s1` |
+| `embodiment_tags` | `["so100_follower", "so101_follower"]` |
+| `runtime` / `quantization.dtype` | `pytorch` / `int4` (NF4) |
+| `weights_uri` | `hf://OpenRAL/rskill-molmoact2-so101-nf4` |
+| `chunk_size` / `n_action_steps` | 10 / 10 (full chunk replay) |
+| `latency_budget.per_chunk_ms` | 1000 ms |
+| `commercial_use_allowed` | `true` (Apache-2.0) |
+| `image_preprocessing.image_max_crops` | `4` (secondary vision lever; processor default is 8 — see Memory note) |
+| **`norm_tag` (vla.extra)** | **`"so100_so101_molmoact2"` — required** |
+Full schema: [`openral_core.schemas.RSkillManifest`](../../python/core/src/openral_core/schemas.py).
+## Quick start
 ```python
+from openral_rskill.loader import rSkill
+pkg = rSkill.from_yaml("rskills/molmoact2-so101-nf4/rskill.yaml")
+print(pkg.manifest.name, pkg.manifest.version)
 ```
+```bash
+# CLI:
+uv run openral rskill install OpenRAL/rskill-molmoact2-so101-nf4
+uv run openral rskill check                # does this host meet the requirements?
 ```
+### Sim config snippet
+```yaml
+vla:
+  id: molmoact2
+  weights_uri: rskills/molmoact2-so101-nf4
+  extra:
+    norm_tag: "so100_so101_molmoact2"   # REQUIRED — default "libero" is wrong for this checkpoint
+    # image_max_crops: 6                # optional secondary lever; manifest pins 4 (see note)
+```
+> **Memory note (measured on an 8 GiB RTX 4070, transformers 5.x).** NF4 makes
+> the model ~6.0 GiB resident (the bf16 vocab embeddings + vision tower
+> dominate; the nf4 Linears are ~3.5 GiB) and it peaks **~7.63 GiB** during a
+> chunk — right at the edge of an 8 GiB card (which exposes only ~7.6 GiB
+> usable). The decisive enabler is the **CUDA expandable-segments allocator**:
+> without it the first forward's ~1.5 GiB embedding `cat` cannot be placed
+> contiguously and OOMs. The molmoact2 adapter turns this on automatically
+> (`PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True`, via
+> `_enable_expandable_segments`) before its first CUDA allocation; export it
+> yourself if other GPU work in the process allocates before the policy loads.
+> `image_max_crops` (pinned to 4 here) is a *secondary* lever — it bounds the
+> vision tile count but does **not** by itself decide the 8 GiB fit on these
+> checkpoints, and transformers 5.x's fast image processor largely ignores it.
+> Leave ~0.4 GiB of headroom: don't run other GPU processes alongside it.
+## Reproduction
+```bash
+just bootstrap && uv sync --all-packages
+# Closed-loop rollout against the SO-101 box scene (NF4 weights fit an 8 GB GPU):
+openral sim run --config scenes/sim/so101_tube_insertion.yaml \
+                --rskill rskills/molmoact2-so101-nf4 \
+                --vla.extra.norm_tag so100_so101_molmoact2
+```
+Producing / refreshing the NF4 weights on the Hub (one-shot, needs a CUDA
+host):
+```bash
+HF_TOKEN=<write-token> uv run python tools/quantize_rskill.py \
+    --source allenai/MolmoAct2-SO100_101 \
+    --target OpenRAL/rskill-molmoact2-so101-nf4 \
+    --loader transformers --trust-remote-code
+```
+## Evaluation
+`eval/so101.json::status` is **pending** — no locally-reproduced benchmark
+numbers are available yet. Run the reproduction command in
+`eval/so101.json::source.reproduction_cli` to populate.
+## License
+This rSkill package (`rskill.yaml`, `README.md`, `eval/so101.json`) is
+**Apache-2.0**. The wrapped weights at
+`hf://OpenRAL/rskill-molmoact2-so101-nf4` (NF4 mirror of
+`allenai/MolmoAct2-SO100_101`) are also released under **Apache-2.0** by Ai2 —
+commercial use is permitted; review the upstream LICENSE before deployment.
+## See also
+- [`robots/so101_follower/README.md`](../../robots/so101_follower/README.md) — RobotDescription manifest.
+- [`robots/so100_follower/README.md`](../../robots/so100_follower/README.md) — SO-100 variant.
+- [`scenes/sim/so101_tube_insertion.yaml`](../../scenes/sim/so101_tube_insertion.yaml) — SO-101 sim scene config.
+- [`rskills/molmoact2-libero-nf4/README.md`](../molmoact2-libero-nf4/README.md) — MolmoAct2 LIBERO variant (Franka Panda).
+- [CLAUDE.md §6.4](../../CLAUDE.md) — rSkill packaging contract.