robometer/dataset_upload/dataset_guides/NewMITFranka.md

Robometer MIT Franka Dataset Guide

This guide explains how to integrate and use the Robometer MIT Franka dataset with the Robometer pipeline. Source: https://drive.google.com/drive/folders/1dd62YeKY9-bTeK5fkjljzoFkNX2uZ1YD?usp=sharing

Overview

• Robometer MIT Franka is a robotics dataset with quality-labeled trajectories for manipulation tasks
• The dataset contains pre-recorded MP4 videos organized by task and quality level
• Quality labels include: successful, suboptimal, and failure
• Each trajectory has two camera views: external (ext) and wrist

Dataset Structure

The dataset is organized as follows:

ROBOMETER_MIT_Franka/
  foldtowel/
    success/
      Tue_Jan_20_23_49_14_2026_ext.mp4
      Tue_Jan_20_23_49_14_2026_wrist.mp4
      ...
    suboptimal/
      Tue_Jan_20_23_52_01_2026_ext.mp4
      Tue_Jan_20_23_52_01_2026_wrist.mp4
      ...
    failure/
      Tue_Jan_20_23_54_53_2026_ext.mp4
      Tue_Jan_20_23_54_53_2026_wrist.mp4
      ...
  movebanana/
    success/
    suboptimal/
    failure/
  movemouse/
  pourpebble/
  pulltissue/
  putspoon/
  stirpot/

Tasks Included

1. foldtowel: Fold the towel in half.
2. movebanana: Pick up the banana from the blue plate and place it on the green plate.
3. movemouse: Pick up the mouse and place it right next to the laptop, while avoiding spilling coffee.
4. pourpebble: Pour the pebbles from the cup onto the plate.
5. pulltissue: Pull a tissue from the tissue box.
6. putspoon: Pick up the spoon and place it inside the cup.
7. stirpot: Pick up the spatula and stir the beans in the pot.

Camera Views

Each trajectory has two synchronized camera views:

• ext (external): Third-person view of the robot and workspace
• wrist: First-person view from the robot's wrist camera

By default, the loader processes both views as separate trajectories, each with the same task instruction but different visual observations. You can optionally exclude wrist camera views using the exclude_wrist_cam configuration option.

Quality Labels

• success: Trajectory successfully completes the task
• suboptimal: Trajectory completes the task but with inefficiencies or errors
• failure: Trajectory fails to complete the task

Configuration

Configuration file: dataset_upload/configs/data_gen_configs/new_mit_franka.yaml

dataset:
  dataset_path: ~/robometer/datasets/ROBOMETER_MIT_Franka
  dataset_name: rfm_new_mit_franka_rfm
  exclude_wrist_cam: false  # Set to true to only process external camera views

output:
  output_dir: ./robometer_dataset/rfm_new_mit_franka_rfm
  max_trajectories: -1  # -1 for all trajectories
  max_frames: 32
  use_video: true
  fps: 10
  shortest_edge_size: 240
  center_crop: false
  num_workers: 4

hub:
  push_to_hub: true
  hub_repo_id: rfm_new_mit_franka_rfm

Camera View Options

The dataset includes both external and wrist camera views. You can configure which views to process:

• exclude_wrist_cam: false (default): Process both external and wrist camera views as separate trajectories
• exclude_wrist_cam: true: Only process external camera views, skip wrist camera entirely

This is useful if you only want third-person views or need to reduce dataset size.

Usage

Convert Dataset to HuggingFace Format

uv run python -m dataset_upload.generate_hf_dataset --config_path=dataset_upload/configs/data_gen_configs/new_mit_franka.yaml

To only process external camera views (exclude wrist camera):

uv run python -m dataset_upload.generate_hf_dataset --config_path=dataset_upload/configs/data_gen_configs/new_mit_franka.yaml --dataset.exclude_wrist_cam=true

This will:

• Scan all task folders and their quality subfolders
• Load the corresponding MP4 videos for both camera views (or only external if exclude_wrist_cam: true)
• Process and resample videos to the specified frame count and FPS
• Generate language embeddings for task instructions
• Create a HuggingFace dataset with proper quality labels
• Optionally push to HuggingFace Hub

Quality Label Mapping

The loader automatically normalizes quality labels:

• "success" → "successful"
• "failure" → "failure" (unchanged)
• "suboptimal" → "suboptimal" (unchanged)

Output Format

The generated dataset will have the following schema:

• id: Unique trajectory identifier
• task: Task instruction text
• lang_vector: Language embedding of the task
• data_source: Dataset name
• frames: Path to video file (or sequence of images)
• is_robot: Boolean (True for this dataset)
• quality_label: One of "successful", "suboptimal", "failure"
• preference_group_id: None for this dataset
• preference_rank: None for this dataset

Task Instructions Mapping

The loader uses the following mapping from folder names to natural language instructions:

Folder Name	Task Instruction
foldtowel	Fold the towel in half
movebanana	Pick up the banana from the blue plate and place it on the green plate
movemouse	Pick up the mouse and place it right next to the laptop, while avoiding spilling coffee
pourpebble	Pour the pebbles from the cup onto the plate
pulltissue	Pull a tissue from the tissue box
putspoon	Pick up the spoon and place it inside the cup
stirpot	Pick up the spatula and stir the beans in the pot

These mappings are defined in the loader file (new_mit_franka_loader.py) and can be updated if needed.

Notes

• Videos are already in MP4 format, so the loader reads them directly using OpenCV
• The loader supports parallel processing with configurable worker count
• Language embeddings are cached to avoid redundant computations
• Output videos maintain the same content but are resampled to the specified frame count and FPS
• Both camera views (ext and wrist) are processed as separate trajectories by default
• Use exclude_wrist_cam: true to only process external camera views and reduce dataset size by ~50%

Troubleshooting

Video File Not Found

• Ensure the dataset path points to the ROBOMETER_MIT_Franka directory
• Check that video files exist in the expected task/quality subfolders

Missing Task Folders

• The loader expects folders named: foldtowel, movebanana, movemouse, pourpebble, pulltissue, putspoon, stirpot
• Each task folder should contain subfolders: success, suboptimal, failure

OpenCV Issues

• If you encounter video codec issues, ensure OpenCV is properly installed with video support
• Try: pip install opencv-python-headless or pip install opencv-python

Unknown Task Warning

• If a task folder is not in the TASK_INSTRUCTIONS mapping, it will be skipped
• Add new tasks to the mapping in new_mit_franka_loader.py

Filtering Camera Views

• Set exclude_wrist_cam: true in the YAML config to skip wrist camera videos
• Or pass it as a command-line argument: dataset.exclude_wrist_cam=true
• This will process only external camera views, roughly halving the dataset size

Example Output

After processing, you'll have a directory structure like:

robometer_dataset/rfm_new_mit_franka_rfm/
  rfm_new_mit_franka_rfm/
    shard_0000/
      episode_000000/
        foldtowel_ext.mp4
      episode_000001/
        foldtowel_wrist.mp4
      ...

With exclude_wrist_cam: true, only external camera videos are included:

robometer_dataset/rfm_new_mit_franka_rfm/
  rfm_new_mit_franka_rfm/
    shard_0000/
      episode_000000/
        foldtowel_ext.mp4
      episode_000001/
        movebanana_ext.mp4
      ...

And a HuggingFace Dataset with all trajectory metadata.

Dataset Statistics

The dataset contains approximately 304 video files (152 trajectory pairs with ext and wrist views) across 7 different manipulation tasks, with varying distributions of successful, suboptimal, and failed demonstrations.

When exclude_wrist_cam: true is set, approximately 152 external camera videos are processed (one per trajectory).