Buckets:

rerun
/

molmoact2

Files

xet

rerun/molmoact2 / README.md

bhoram

12 days ago

preview code

download

raw

5.81 kB

# MolmoAct2-BimanualYAM — Rerun RRD conversion

[Rerun](https://rerun.io) recordings of the [MolmoAct2-BimanualYAM](https://huggingface.co/datasets/allenai/MolmoAct2-BimanualYAM-Dataset) dataset: 32,246 teleoperated episodes (about 704 hours) of bimanual tabletop manipulation on two I2RT YAM arms, with three cameras.
Converted from the full release.

Each episode becomes several `.rrd` files, one per _layer_, sharing a single `recording_id`.
Open them in the Rerun Viewer, or register them as a catalog dataset.
Layers live in their own directories, so one episode is spread across the bucket like this:

```
base/episode_<NNNNN>.rrd       # the episode as recorded ('base' layer), with its metadata as properties
urdf/…                         # the two posed arms: forward kinematics from the joint state
cameras/…                      # the three cameras placed in the scene
stats/…                        # the source's statistics of the episode, and how far each arm moves
assets/urdf-model-left.rrd     # the left arm's meshes, one for the whole dataset
assets/urdf-model-right.rrd    # the right arm's meshes, one for the whole dataset
blueprints/default.rbl         # default viewer layout, one for the whole dataset
```

`<NNNNN>` is the source `episode_index`, zero-padded to five digits (00000…32245).

## Provenance and attribution

This dataset is a converted derivative of [allenai/MolmoAct2-BimanualYAM-Dataset](https://huggingface.co/datasets/allenai/MolmoAct2-BimanualYAM-Dataset), released by the Allen Institute for AI.
The original authors released it under the Apache License 2.0, at source revision [`e9f21ae1`](https://huggingface.co/datasets/allenai/MolmoAct2-BimanualYAM-Dataset/tree/e9f21ae15074330839f2ac25ed4b49d76dfa1f9c), dated 2026-06-11.

All credit for the underlying data collection belongs to the original authors.
The original authors do not endorse this conversion.

## Changes from the source

- The source packs many consecutive episodes into each parquet and mp4 file.
  Every episode becomes its own Rerun recording.
- The base layer keeps the state, action, and task of every frame, and copies each camera's AV1 samples without re-encoding, except that a video window starting on a delta frame has its frames up to the next keyframe re-encoded.
- Three layers are added beyond the base: `urdf` poses the two arms, `cameras` places the three cameras, and `stats` holds the per-episode statistics from `meta/episodes` (not the dataset-wide `meta/stats.json`) and how far each arm moves.
  The source has no calibration, so the arm spacing, the top camera pose, and the camera intrinsics are estimates.

## Size and scale

The full RRD dataset is 2.42 TB in 128,987 files, converted from 2.35 TB of source.
A single episode's base layer ranges roughly from 0.5 to 200 MB, with a median of about 50 MB.
The base layer is about the size of the source.
The urdf layer adds about 2% on top, the cameras layer about 27 KB per episode, and the stats layer about 40 KB.
The two arm models are 3.3 MB each.

## How to use

Fetch one episode from the bucket and view it locally:

```bash
pip install "huggingface_hub[cli]" rerun-sdk
# Every layer of one episode, plus the two arm models and the default blueprint
hf buckets sync hf://buckets/rerun/molmoact2 . \
  --include "*/episode_<NNNNN>.rrd" \
  --include "assets/*.rrd" \
  --include "blueprints/default.rbl"
rerun */episode_<NNNNN>.rrd blueprints/default.rbl
```

Keep the `*` so every layer loads.

> **Note:** The viewer opens local files through its Viewer catalog by default, and it can then ignore the blueprint file.
> Turn off **Settings** → **Viewer catalog** → **Load files via Viewer catalog** before running the command above.

Opened as above, the 3D view shows the three cameras but not the arms.

The recommended way of viewing the dataset with a 3D view is to register the episodes on a catalog with the two `assets/urdf-model-*.rrd` files as the dataset's assets.
The [Local Catalog](https://github.com/rerun-io/rrd-datasets/tree/main/examples/molmoact2#4-local-catalog) section in the code repo provides instructions.

These recordings were written with Rerun 0.38.1.
A newer viewer reads them as they are in most cases; [`rerun rrd migrate`](https://rerun.io/docs/reference/cli#rerun-rrd-migrate) updates them if it cannot.

To copy the whole bucket (about 2.4 TB), mirror it bucket-to-bucket rather than to a local disk.
The bucket is reachable over an [S3-compatible API](https://huggingface.co/docs/hub/main/en/storage-buckets-s3#import-data-using-rclone), so `rclone` can stream it straight into your own S3-compatible storage.

## Scope

Conversion code is published at [rerun-io/rrd-datasets](https://github.com/rerun-io/rrd-datasets).
The code and these recordings demonstrate the conversion; they are not a versioned dataset release.
Files may be regenerated or replaced, and converting the same episode twice does not give identical bytes.

Interesting observations of the source data, including known issues, are listed in [observations.md](https://github.com/rerun-io/rrd-datasets/blob/main/examples/molmoact2/observations.md).

## License

Apache License 2.0, the same license as the source dataset — the full text ships as the `LICENSE` file at the root of this bucket.
If you redistribute or build on this data, keep the attribution above and comply with the license terms.

The two arm models, `assets/urdf-model-left.rrd` and `assets/urdf-model-right.rrd`, are the only files not derived from the source dataset.
They carry geometry from the YAM arm model in [i2rt-robotics/i2rt](https://github.com/i2rt-robotics/i2rt), under the MIT License, Copyright (c) I2RT Robotics.
Its full text ships as `LICENSE-YAM` at the root of this bucket, and applies to those two files.

Xet Storage Details

Size:: 5.81 kB
Xet hash:: 102c3be6f0481c150ba315331752ccfae2ee54c408a0034f9065fc9da751f979

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.

	# MolmoAct2-BimanualYAM — Rerun RRD conversion

	[Rerun](https://rerun.io) recordings of the [MolmoAct2-BimanualYAM](https://huggingface.co/datasets/allenai/MolmoAct2-BimanualYAM-Dataset) dataset: 32,246 teleoperated episodes (about 704 hours) of bimanual tabletop manipulation on two I2RT YAM arms, with three cameras.
	Converted from the full release.

	Each episode becomes several `.rrd` files, one per _layer_, sharing a single `recording_id`.
	Open them in the Rerun Viewer, or register them as a catalog dataset.
	Layers live in their own directories, so one episode is spread across the bucket like this:

	```
	base/episode_<NNNNN>.rrd # the episode as recorded ('base' layer), with its metadata as properties
	urdf/… # the two posed arms: forward kinematics from the joint state
	cameras/… # the three cameras placed in the scene
	stats/… # the source's statistics of the episode, and how far each arm moves
	assets/urdf-model-left.rrd # the left arm's meshes, one for the whole dataset
	assets/urdf-model-right.rrd # the right arm's meshes, one for the whole dataset
	blueprints/default.rbl # default viewer layout, one for the whole dataset
	```

	`<NNNNN>` is the source `episode_index`, zero-padded to five digits (00000…32245).

	## Provenance and attribution

	This dataset is a converted derivative of [allenai/MolmoAct2-BimanualYAM-Dataset](https://huggingface.co/datasets/allenai/MolmoAct2-BimanualYAM-Dataset), released by the Allen Institute for AI.
	The original authors released it under the Apache License 2.0, at source revision [`e9f21ae1`](https://huggingface.co/datasets/allenai/MolmoAct2-BimanualYAM-Dataset/tree/e9f21ae15074330839f2ac25ed4b49d76dfa1f9c), dated 2026-06-11.

	All credit for the underlying data collection belongs to the original authors.
	The original authors do not endorse this conversion.

	## Changes from the source

	- The source packs many consecutive episodes into each parquet and mp4 file.
	Every episode becomes its own Rerun recording.
	- The base layer keeps the state, action, and task of every frame, and copies each camera's AV1 samples without re-encoding, except that a video window starting on a delta frame has its frames up to the next keyframe re-encoded.
	- Three layers are added beyond the base: `urdf` poses the two arms, `cameras` places the three cameras, and `stats` holds the per-episode statistics from `meta/episodes` (not the dataset-wide `meta/stats.json`) and how far each arm moves.
	The source has no calibration, so the arm spacing, the top camera pose, and the camera intrinsics are estimates.

	## Size and scale

	The full RRD dataset is 2.42 TB in 128,987 files, converted from 2.35 TB of source.
	A single episode's base layer ranges roughly from 0.5 to 200 MB, with a median of about 50 MB.
	The base layer is about the size of the source.
	The urdf layer adds about 2% on top, the cameras layer about 27 KB per episode, and the stats layer about 40 KB.
	The two arm models are 3.3 MB each.

	## How to use

	Fetch one episode from the bucket and view it locally:

	```bash
	pip install "huggingface_hub[cli]" rerun-sdk

	# Every layer of one episode, plus the two arm models and the default blueprint
	hf buckets sync hf://buckets/rerun/molmoact2 . \
	--include "*/episode_<NNNNN>.rrd" \
	--include "assets/*.rrd" \
	--include "blueprints/default.rbl"

	rerun */episode_<NNNNN>.rrd blueprints/default.rbl
	```

	Keep the `*` so every layer loads.

	> Note: The viewer opens local files through its Viewer catalog by default, and it can then ignore the blueprint file.
	> Turn off Settings → Viewer catalog → Load files via Viewer catalog before running the command above.

	Opened as above, the 3D view shows the three cameras but not the arms.

	The recommended way of viewing the dataset with a 3D view is to register the episodes on a catalog with the two `assets/urdf-model-*.rrd` files as the dataset's assets.
	The [Local Catalog](https://github.com/rerun-io/rrd-datasets/tree/main/examples/molmoact2#4-local-catalog) section in the code repo provides instructions.

	These recordings were written with Rerun 0.38.1.
	A newer viewer reads them as they are in most cases; [`rerun rrd migrate`](https://rerun.io/docs/reference/cli#rerun-rrd-migrate) updates them if it cannot.

	To copy the whole bucket (about 2.4 TB), mirror it bucket-to-bucket rather than to a local disk.
	The bucket is reachable over an [S3-compatible API](https://huggingface.co/docs/hub/main/en/storage-buckets-s3#import-data-using-rclone), so `rclone` can stream it straight into your own S3-compatible storage.

	## Scope

	Conversion code is published at [rerun-io/rrd-datasets](https://github.com/rerun-io/rrd-datasets).
	The code and these recordings demonstrate the conversion; they are not a versioned dataset release.
	Files may be regenerated or replaced, and converting the same episode twice does not give identical bytes.

	Interesting observations of the source data, including known issues, are listed in [observations.md](https://github.com/rerun-io/rrd-datasets/blob/main/examples/molmoact2/observations.md).

	## License

	Apache License 2.0, the same license as the source dataset — the full text ships as the `LICENSE` file at the root of this bucket.
	If you redistribute or build on this data, keep the attribution above and comply with the license terms.

	The two arm models, `assets/urdf-model-left.rrd` and `assets/urdf-model-right.rrd`, are the only files not derived from the source dataset.
	They carry geometry from the YAM arm model in [i2rt-robotics/i2rt](https://github.com/i2rt-robotics/i2rt), under the MIT License, Copyright (c) I2RT Robotics.
	Its full text ships as `LICENSE-YAM` at the root of this bucket, and applies to those two files.