docs: HF model card — best-of-both front-matter derived from manifest

946a7db verified 13 days ago

9.62 kB

language:
  - en
license: apache-2.0
pipeline_tag: robotics
tags:
  - OpenRAL
  - rskill
  - molmoact2
  - vision-language-action
  - nf4
  - 4-bit
  - so100_follower
  - so101_follower
  - transformers
  - vla
  - so101
  - so100
  - manipulation
base_model:
  - allenai/MolmoAct2-SO100_101
base_model_relation: quantized
inference: false

rskill-molmoact2-so101-nf4

OpenRAL rSkill — MolmoAct2 (Ai2's open action reasoning model: a Molmo2-ER embodied-reasoning VLM backbone with a flow-matching continuous-action expert) finetuned on the SO-100/SO-101 teleop mixture and NF4-quantized so the ~5.5 B-param model fits an 8 GB GPU. Robots: SO-100 and SO-101 follower arms. Apache-2.0 weights — commercial use permitted.

This package wraps hf://OpenRAL/rskill-molmoact2-so101-nf4 (an NF4-quantized mirror of allenai/MolmoAct2-SO100_101) with a rskill.yaml manifest that adds capability checking, license surfacing, latency budgets, and local registry integration. It does not copy model weights — they live on the Hub.

Required sim config knob: this checkpoint uses normalization statistics tagged "so100_so101_molmoact2". Any SimEnvironment config that drives this rSkill must set vla.extra.norm_tag: "so100_so101_molmoact2" — omitting it silently applies the adapter's default "libero" norm stats and produces garbage actions.

What this skill does

Performs tabletop manipulation — picking, placing, grasping, and transporting objects — on the SO-100 and SO-101 follower arms. The MolmoAct2 backbone reasons about the scene in 3D and the flow-matching action expert emits a continuous absolute joint-position action chunk that the adapter replays one step at a time.

Field	Value
Actions	pick, place, pick_and_place, grasp
Objects	diverse tabletop objects
Scenes	tabletop
Embodiments	`so100_follower`, `so101_follower`

How it works

MolmoAct2 grafts a modern DiT-style flow-matching continuous-action expert onto the Molmo2-ER discrete-token VLM via per-layer KV-cache conditioning (arXiv:2605.02881). It ships as a transformers custom-code model (trust_remote_code, auto_map → MolmoAct2ForConditionalGeneration), not a lerobot policy. The OpenRAL molmoact2 adapter (python/sim/src/openral_sim/policies/molmoact2.py) loads it via AutoModelForImageTextToText.from_pretrained + AutoProcessor from the manifest's source_repo, NF4-quantizes every Linear with ≥4M weight elements via bitsandbytes, overlays the prequantized pack from weights_uri, then drives it through the checkpoint's own predict_action(...) continuous-action API. Two RGB camera streams plus a 6-D proprio state go in; a (chunk_size, 6) absolute joint-position chunk comes out, replayed one step at a time and re-inferred when the queue empties.

The adapter reads norm_tag from vla.extra.norm_tag; this rSkill requires "so100_so101_molmoact2" — set it explicitly in every SimEnvironment config.

Observation → action contract

Direction	Key	Shape	Notes
in	`observation.images.camera1`	`(1, 3, H, W) float32 [0,1]`	overhead view (→ model `top`)
in	`observation.images.camera2`	`(1, 3, H, W) float32 [0,1]`	wrist/side view (→ model `side`)
in	`observation.state`	`(1, 6)` float32	SO-101 6-D joint positions (rad)
out	action chunk	`(10, 6)` float32	absolute joint-position targets

Camera aliases (for so101_box scene): oak_top → top, wrist → side. Override per-scene via vla.extra if your scene uses different camera names.

Upstream model / training

The wrapped weights come from Ai2's allenai/MolmoAct2-SO100_101 checkpoint — the base allenai/MolmoAct2 foundation model finetuned on the SO-100/SO-101 teleop dataset mixture with absolute joint-pose control and annotated language instructions. This rSkill repackages an NF4-quantized mirror of those weights; it does not retrain or copy the full-precision weights.

Field	Value
Source repo	`allenai/MolmoAct2-SO100_101`
Base model	`allenai/MolmoAct2`
Paper	arxiv:2605.02881 — MolmoAct2: Action Reasoning Models for Real-world Deployment
License	apache-2.0 (code + weights)
Parameters	~5.5 B
Training data	SO-100/SO-101 teleop mixture (absolute joint-pose, annotated language)
norm_tag	`"so100_so101_molmoact2"` — required in `vla.extra.norm_tag`

Supported robots

Robot	Embodiment tag	Status	Notes
SO-101 follower	`so101_follower`	⚡ experimental	Native training embodiment; numbers not yet locally reproduced.
SO-100 follower	`so100_follower`	⚡ experimental	Shares identical 6-DoF kinematics; covered by training mixture.

Sensors required

Key	Modality	Min resolution	Format
`observation.images.camera1`	RGB	224 × 224	`float32`
`observation.images.camera2`	RGB	224 × 224	`float32`
`observation.state`	proprioception	(6,)	`float32`

Manifest summary

Field	Value
`name`	`OpenRAL/rskill-molmoact2-so101-nf4`
`version`	`0.1.0`
`license`	`apache-2.0`
`role`	`s1`
`embodiment_tags`	`["so100_follower", "so101_follower"]`
`runtime` / `quantization.dtype`	`pytorch` / `int4` (NF4)
`weights_uri`	`hf://OpenRAL/rskill-molmoact2-so101-nf4`
`chunk_size` / `n_action_steps`	10 / 10 (full chunk replay)
`latency_budget.per_chunk_ms`	1000 ms
`commercial_use_allowed`	`true` (Apache-2.0)
`image_preprocessing.image_max_crops`	`4` (secondary vision lever; processor default is 8 — see Memory note)
`norm_tag` (vla.extra)	`"so100_so101_molmoact2"` — required

Full schema: openral_core.schemas.RSkillManifest.

Quick start

from openral_rskill.loader import rSkill

pkg = rSkill.from_yaml("rskills/molmoact2-so101-nf4/rskill.yaml")
print(pkg.manifest.name, pkg.manifest.version)

# CLI:
uv run openral rskill install OpenRAL/rskill-molmoact2-so101-nf4
uv run openral rskill check                # does this host meet the requirements?

Sim config snippet

vla:
  id: molmoact2
  weights_uri: rskills/molmoact2-so101-nf4
  extra:
    norm_tag: "so100_so101_molmoact2"   # REQUIRED — default "libero" is wrong for this checkpoint
    # image_max_crops: 6                # optional secondary lever; manifest pins 4 (see note)

Memory note (measured on an 8 GiB RTX 4070, transformers 5.x). NF4 makes the model 6.0 GiB resident (the bf16 vocab embeddings + vision tower dominate; the nf4 Linears are ~3.5 GiB) and it peaks **7.63 GiB** during a chunk — right at the edge of an 8 GiB card (which exposes only ~7.6 GiB usable). The decisive enabler is the CUDA expandable-segments allocator: without it the first forward's ~1.5 GiB embedding cat cannot be placed contiguously and OOMs. The molmoact2 adapter turns this on automatically (PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True, via _enable_expandable_segments) before its first CUDA allocation; export it yourself if other GPU work in the process allocates before the policy loads. image_max_crops (pinned to 4 here) is a secondary lever — it bounds the vision tile count but does not by itself decide the 8 GiB fit on these checkpoints, and transformers 5.x's fast image processor largely ignores it. Leave ~0.4 GiB of headroom: don't run other GPU processes alongside it.

Reproduction

just bootstrap && uv sync --all-packages

# Closed-loop rollout against the SO-101 box scene (NF4 weights fit an 8 GB GPU):
openral sim run --config scenes/sim/so101_tube_insertion.yaml \
                --rskill rskills/molmoact2-so101-nf4 \
                --vla.extra.norm_tag so100_so101_molmoact2

Producing / refreshing the NF4 weights on the Hub (one-shot, needs a CUDA host):

HF_TOKEN=<write-token> uv run python tools/quantize_rskill.py \
    --source allenai/MolmoAct2-SO100_101 \
    --target OpenRAL/rskill-molmoact2-so101-nf4 \
    --loader transformers --trust-remote-code

Evaluation

eval/so101.json::status is pending — no locally-reproduced benchmark numbers are available yet. Run the reproduction command in eval/so101.json::source.reproduction_cli to populate.

License

This rSkill package (rskill.yaml, README.md, eval/so101.json) is Apache-2.0. The wrapped weights at hf://OpenRAL/rskill-molmoact2-so101-nf4 (NF4 mirror of allenai/MolmoAct2-SO100_101) are also released under Apache-2.0 by Ai2 — commercial use is permitted; review the upstream LICENSE before deployment.

OpenRAL
/

rskill-molmoact2-multi-so101-nf4