RACER-augmented RLBench Dataset Guide

This guide explains how to load and convert the RACER-augmented RLBench dataset with the Robometer pipeline.

Sources:

Dataset card: https://huggingface.co/datasets/sled-umich/RACER-augmented_rlbench
Example JSON: https://huggingface.co/datasets/sled-umich/RACER-augmented_rlbench/blob/main/samples/close_jar/0/language_description.json

Overview

Train/validation split under train/ and val/ directories (sometimes train/samples/)
Each task (e.g., close_jar) contains multiple numbered episodes with:
- language_description.json (contains task_goal and per-frame subgoal entries)
- Camera folders: front_rgb/, left_shoulder_rgb/, right_shoulder_rgb (or right_shoudler_rgb), wrist_rgb/

We use task_goal as the language instruction for all trajectories.

Success trajectories: full expert episode for each camera view
Failure trajectories: for any expert frame with heuristic failures in augmentation, construct a failure episode consisting of expert frames up to that frame (inclusive), for each camera view

Configuration

# configs/data_gen_configs/racer_train.yaml

dataset:
  dataset_path: ./datasets/racer
  dataset_name: racer_train

output:
  output_dir: ./robometer_dataset/racer_train_rfm
  max_trajectories: -1
  max_frames: 64
  use_video: true
  fps: 10
  shortest_edge_size: 240
  center_crop: false

hub:
  push_to_hub: true
  hub_repo_id: racer_train_rfm

For validation, use racer_val.yaml analogously.

Loader

File: dataset_upload/dataset_loaders/racer_loader.py
Function: load_racer_dataset(dataset_path, dataset_name)
Notes:
- Handles both train/ and validation/ (and the samples/ subfolder if present)
- Uses task_goal from language_description.json
- Builds successes and heuristic failure truncations per camera view

Usage

uv run python -m dataset_upload.generate_hf_dataset --config_path=dataset_upload/configs/data_gen_configs/racer_train.yaml

This will:

Load expert and heuristic failure episodes
Generate web-optimized videos per camera view
Create a HuggingFace dataset for the train split (use the val YAML for validation)