|
|
--- |
|
|
pipeline_tag: robotics |
|
|
license: apache-2.0 |
|
|
--- |
|
|
|
|
|
# Large Video Planner Enables Generalizable Robot Control |
|
|
|
|
|
This repository contains the trained large video planner checkpoints (14B parameter) for the model presented in the paper [Large Video Planner Enables Generalizable Robot Control](https://huggingface.co/papers/2512.15840). |
|
|
|
|
|
The Large Video Planner explores an alternative paradigm of using large-scale video pretraining as a primary modality for building robot foundation models. It produces zero-shot video plans for novel scenes and tasks, which are then post-processed to extract executable robot actions. |
|
|
|
|
|
- **Project Page:** [https://www.boyuan.space/large-video-planner/](https://www.boyuan.space/large-video-planner/) |
|
|
- **GitHub Repository:** [https://github.com/buoyancy99/large-video-planner](https://github.com/buoyancy99/large-video-planner) |
|
|
- **Hugging Face Demo:** [https://huggingface.co/spaces/KempnerInstituteAI/LVP](https://huggingface.co/spaces/KempnerInstituteAI/LVP) |
|
|
|
|
|
--- |
|
|
|
|
|
This folder contains the trained large video planner checkpoints (14B parameter) and all the metadata for eight dataset sources: Agibot-world, droid, bridge, language-tables, Pandas(filtered), SomethingSomethingV2, ego4d, epic_kitchens. We release a `merged_metadata.csv` and a `cleaned_metadata.csv` for each. |
|
|
We also released our test set in `data/ours_test/` with the images and text instructions gathered from third-parties. |
|
|
|
|
|
## Trained Checkpoints for our Large Video Planner |
|
|
`checkpoints/lvp_14B.ckpt` is the trained weights for the transformer backbone. |
|
|
|
|
|
## Dataset format |
|
|
We train on a mixture of datasets, so we define a unified dataset format for consistency and ease of management. |
|
|
|
|
|
Each dataset includes a global metadata file, typically named `metadata_merged.csv`, which contains key information for each video clip. |
|
|
|
|
|
The file is named as metdata_**merged**.csv because each video clip may have multiple recaptions. Instead of saving the captions for each video into a list within a single csv row, we just create another row on the `metadata_merged.csv`. So `metadata_merged.csv` may contain multiple rows referring to the same video with different captions. For some dataset, we also provide a `cleaned_metadata.csv`, which contains a deduplicated version of the metadata (one entry per video) but excludes the additional recaptions. |
|
|
|
|
|
Important fields of the global metadata includes: |
|
|
1. `video_path`: Relative path (from the metadata file) to the video clip. |
|
|
2. `trim_start`, and `trim_end` (optional): Specifies the trimmed segment of the clip. If absent, the full video is used. |
|
|
3. `gemini_caption`: Action-focused caption generated by Gemini Flash 2.0. |
|
|
4. `original_caption`: Original caption from the source dataset; used when no Gemini caption is available. |
|
|
5. `prompt_embed_path`: Path to precomputed T5 prompt embeddings (not released due to large size). |
|
|
6. `stable_brightess` (optional): 1.0 if brightness is stable, 0.0 otherwise. We recommend removing videos with `stable_brightess == 0.0` |
|
|
7. `stable_background` (optional): Either 1.0 or 0.0. Recommend to remove videos with `stable_background == 0.0`, this indicates the video has large average optical flow magnitudes, which very likely contains large background motions. |
|
|
8. `detected_hand_in_first_frame` (optional): 1.0 if a human hand is detected in the first frame, 0.0 otherwise. Videos with 0.0 often cause embodiment ambiguity and should be filtered out. |
|
|
9. There are some other fields which can help you understand more about this clips. `n_frames`, `n_fps`, `height`, `width`, ... etc. |
|
|
|
|
|
## Downloading the dataset |
|
|
We provide dataset-specific download scripts for AgiBot World, DROID, Ego4D, EpicKitchens, and Something-Something in their respective dataset.py files within the `datasets/` folder of the relased code. |
|
|
|
|
|
For downloading the filtered Pandas subset, we provide the unique `youtube_key_segment` for each video_clip, and the `trim_start`, and `trim_end` for each clip. To download these subset, please download the official metadata from [Pandas-70M](https://snap-research.github.io/Panda-70M/), then using the `youtube_key_segment` to find the URL of the video clips and then download with your own online video downloader. |
|
|
|
|
|
## Sample Usage (Inference) |
|
|
|
|
|
To run inference with the Large Video Planner, first ensure your environment is set up and checkpoints are downloaded as described in the [GitHub repository's instructions](https://github.com/buoyancy99/large-video-planner#instructions-for-running-the-code). |
|
|
|
|
|
Then, use the following command for basic inference: |
|
|
|
|
|
```bash |
|
|
mkdir -p <your-output-folder> |
|
|
python -m main \ |
|
|
+name=<your_exp_name> \ |
|
|
experiment=exp_video \ |
|
|
algorithm=wan_i2v \ |
|
|
dataset=ours_test \ |
|
|
experiment.tasks=[validation] \ |
|
|
algorithm.logging.video_type=single \ |
|
|
experiment.num_nodes=1 \ |
|
|
experiment.validation.limit_batch=null \ |
|
|
algorithm.hist_guidance=1.5 \ |
|
|
algorithm.lang_guidance=2.5 |
|
|
``` |
|
|
Replace `<your-output-folder>` and `<your_exp_name>` with your desired values. Refer to the [GitHub repository](https://github.com/buoyancy99/large-video-planner) for detailed explanations of arguments and further instructions, including how to download the checkpoints. |
