Enhance model card for Large Video Planner Enables Generalizable Robot Control

README.md

@@ -1,4 +1,21 @@
-This folder contains the trained large video planner checkpoints (14B parameter) and  all the metadata for eight dataset sources: Agibot-world, droid, bridge, language-tables, Pandas(filtered), SomethingSomethingV2, ego4d, epic_kitchens.  We release a `merged_metadata.csv` and a `cleaned_metadata.csv` for each. 
+---
+pipeline_tag: robotics
+license: apache-2.0
+---
+
+# Large Video Planner Enables Generalizable Robot Control
+
+This repository contains the trained large video planner checkpoints (14B parameter) for the model presented in the paper [Large Video Planner Enables Generalizable Robot Control](https://huggingface.co/papers/2512.15840).
+
+The Large Video Planner explores an alternative paradigm of using large-scale video pretraining as a primary modality for building robot foundation models. It produces zero-shot video plans for novel scenes and tasks, which are then post-processed to extract executable robot actions.
+
+-   **Project Page:** [https://www.boyuan.space/large-video-planner/](https://www.boyuan.space/large-video-planner/)
+-   **GitHub Repository:** [https://github.com/buoyancy99/large-video-planner](https://github.com/buoyancy99/large-video-planner)
+-   **Hugging Face Demo:** [https://huggingface.co/spaces/KempnerInstituteAI/LVP](https://huggingface.co/spaces/KempnerInstituteAI/LVP)
+
+---
+
+This folder contains the trained large video planner checkpoints (14B parameter) and all the metadata for eight dataset sources: Agibot-world, droid, bridge, language-tables, Pandas(filtered), SomethingSomethingV2, ego4d, epic_kitchens. We release a `merged_metadata.csv` and a `cleaned_metadata.csv` for each.
 We also released our test set in `data/ours_test/` with the images and text instructions gathered from third-parties.
 
 ## Trained Checkpoints for our Large Video Planner
@@ -7,27 +24,44 @@ We also released our test set in `data/ours_test/` with the images and text inst
 ## Dataset format
 We train on a mixture of datasets, so we define a unified dataset format for consistency and ease of management.
 
-Each dataset includes a global metadata file, typically named `metadata_merged.csv`, which contains key information for each video clip. 
+Each dataset includes a global metadata file, typically named `metadata_merged.csv`, which contains key information for each video clip.
 
 The file is named as metdata_**merged**.csv because each video clip may have multiple recaptions. Instead of saving the captions for each video into a list within a single csv row, we just create another row on the `metadata_merged.csv`. So `metadata_merged.csv` may contain multiple rows referring to the same video with different captions. For some dataset, we also provide a `cleaned_metadata.csv`, which contains a deduplicated version of the metadata (one entry per video) but excludes the additional recaptions.
 
 Important fields of the global metadata includes:
-1. `video_path`: Relative path (from the metadata file) to the video clip.
-2. `trim_start`, and `trim_end` (optional): Specifies the trimmed segment of the clip. If absent, the full video is used.
-3. `gemini_caption`: Action-focused caption generated by Gemini Flash 2.0.
-4. `original_caption`: Original caption from the source dataset; used when no Gemini caption is available.
-5. `prompt_embed_path`: Path to precomputed T5 prompt embeddings (not released due to large size).
-6. `stable_brightess` (optional): 1.0 if brightness is stable, 0.0 otherwise. We recommend removing videos with `stable_brightess == 0.0`
-7. `stable_background` (optional): Either 1.0 or 0.0.  Recommend to remove videos with `stable_background == 0.0`, this indicates the video has large average optical flow magnitudes, which very likely contains large background motions. 
-8. `detected_hand_in_first_frame` (optional): 1.0 if a human hand is detected in the first frame, 0.0 otherwise. Videos with 0.0 often cause embodiment ambiguity and should be filtered out.
-9. There are some other fields which can help you understand more about this clips. `n_frames`, `n_fps`, `height`, `width`, ... etc.
-
-
-
+1.  `video_path`: Relative path (from the metadata file) to the video clip.
+2.  `trim_start`, and `trim_end` (optional): Specifies the trimmed segment of the clip. If absent, the full video is used.
+3.  `gemini_caption`: Action-focused caption generated by Gemini Flash 2.0.
+4.  `original_caption`: Original caption from the source dataset; used when no Gemini caption is available.
+5.  `prompt_embed_path`: Path to precomputed T5 prompt embeddings (not released due to large size).
+6.  `stable_brightess` (optional): 1.0 if brightness is stable, 0.0 otherwise. We recommend removing videos with `stable_brightess == 0.0`
+7.  `stable_background` (optional): Either 1.0 or 0.0. Recommend to remove videos with `stable_background == 0.0`, this indicates the video has large average optical flow magnitudes, which very likely contains large background motions.
+8.  `detected_hand_in_first_frame` (optional): 1.0 if a human hand is detected in the first frame, 0.0 otherwise. Videos with 0.0 often cause embodiment ambiguity and should be filtered out.
+9.  There are some other fields which can help you understand more about this clips. `n_frames`, `n_fps`, `height`, `width`, ... etc.
 
 ## Downloading the dataset
 We provide dataset-specific download scripts for AgiBot World, DROID, Ego4D, EpicKitchens, and Something-Something in their respective dataset.py files within the `datasets/` folder of the relased code.
 
-For downloading the filtered Pandas subset, we provide the unique `youtube_key_segment` for each video_clip, and the `trim_start`, and `trim_end` for each clip.  To download these subset, please download the official metadata from [Pandas-70M](https://snap-research.github.io/Panda-70M/), then using the `youtube_key_segment` to find the URL of the video clips and then download with your own online video downloader.
+For downloading the filtered Pandas subset, we provide the unique `youtube_key_segment` for each video_clip, and the `trim_start`, and `trim_end` for each clip. To download these subset, please download the official metadata from [Pandas-70M](https://snap-research.github.io/Panda-70M/), then using the `youtube_key_segment` to find the URL of the video clips and then download with your own online video downloader.
+
+## Sample Usage (Inference)
+
+To run inference with the Large Video Planner, first ensure your environment is set up and checkpoints are downloaded as described in the [GitHub repository's instructions](https://github.com/buoyancy99/large-video-planner#instructions-for-running-the-code).
 
+Then, use the following command for basic inference:
 
+```bash
+mkdir -p <your-output-folder>
+python -m main \
+  +name=<your_exp_name> \
+  experiment=exp_video \
+  algorithm=wan_i2v \
+  dataset=ours_test \
+  experiment.tasks=[validation] \
+  algorithm.logging.video_type=single \
+  experiment.num_nodes=1 \
+  experiment.validation.limit_batch=null \
+  algorithm.hist_guidance=1.5 \
+  algorithm.lang_guidance=2.5
+```
+Replace `<your-output-folder>` and `<your_exp_name>` with your desired values. Refer to the [GitHub repository](https://github.com/buoyancy99/large-video-planner) for detailed explanations of arguments and further instructions, including how to download the checkpoints.