Instructions to use MBZUAI/WorldGuide-Ckpt with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Diffusers
How to use MBZUAI/WorldGuide-Ckpt with Diffusers:
pip install -U diffusers transformers accelerate
import torch from diffusers import DiffusionPipeline from diffusers.utils import load_image, export_to_video # switch to "mps" for apple devices pipe = DiffusionPipeline.from_pretrained("MBZUAI/WorldGuide-Ckpt", dtype=torch.bfloat16, device_map="cuda") pipe.to("cuda") prompt = "A man with short gray hair plays a red electric guitar." image = load_image( "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/guitar-man.png" ) output = pipe(image=image, prompt=prompt).frames[0] export_to_video(output, "output.mp4") - Notebooks
- Google Colab
- Kaggle
Add model card
Browse files
README.md
ADDED
|
@@ -0,0 +1,59 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
pipeline_tag: image-to-video
|
| 3 |
+
library_name: diffusers
|
| 4 |
+
language:
|
| 5 |
+
- en
|
| 6 |
+
tags:
|
| 7 |
+
- world-model
|
| 8 |
+
- video-generation
|
| 9 |
+
- image-to-video
|
| 10 |
+
---
|
| 11 |
+
|
| 12 |
+
<div align="center">
|
| 13 |
+
|
| 14 |
+
<img src="https://mbzuai-oryx.github.io/WorldGuide/assets/worldguide-logo.png" alt="WorldGuide" width="120"/>
|
| 15 |
+
|
| 16 |
+
# WorldGuide: Goal-Directed Video World Model for Procedural Task Execution
|
| 17 |
+
|
| 18 |
+
[Ankan Deria](https://ankan8145.github.io), [Komal Kumar](https://komalkumar.org), [Hisham Cholakkal](https://hishamcholakkal.com), [Fahad Shahbaz Khan](https://sites.google.com/view/fahadkhans), [Salman Khan](https://salman-h-khan.github.io)
|
| 19 |
+
|
| 20 |
+
**Mohamed bin Zayed University of Artificial Intelligence**
|
| 21 |
+
|
| 22 |
+
[](https://mbzuai-oryx.github.io/WorldGuide/)
|
| 23 |
+
[](https://github.com/mbzuai-oryx/WorldGuide)
|
| 24 |
+
|
| 25 |
+
</div>
|
| 26 |
+
|
| 27 |
+
Official checkpoints for **WorldGuide**. Method details, results and training code are on the [project page](https://mbzuai-oryx.github.io/WorldGuide/) and on [GitHub](https://github.com/mbzuai-oryx/WorldGuide).
|
| 28 |
+
|
| 29 |
+
## Checkpoints
|
| 30 |
+
|
| 31 |
+
```text
|
| 32 |
+
WorldGuide-Ckpt/
|
| 33 |
+
├── transformer/ # WorldGuide Video DiT → --action_ckpt
|
| 34 |
+
└── text_encoder/llm/ # WorldGuide ContextPlanner → --planner_model_path
|
| 35 |
+
```
|
| 36 |
+
|
| 37 |
+
## Usage
|
| 38 |
+
|
| 39 |
+
```bash
|
| 40 |
+
git clone https://github.com/mbzuai-oryx/WorldGuide.git
|
| 41 |
+
cd WorldGuide
|
| 42 |
+
pip install -r requirements.txt
|
| 43 |
+
|
| 44 |
+
# Download this repository and all other required components into ./ckpts
|
| 45 |
+
python download_models.py --hf_token <your_hf_token>
|
| 46 |
+
|
| 47 |
+
# Run closed-loop inference
|
| 48 |
+
bash scripts/inference/run_v5_qwen_closed_loop_memory.sh
|
| 49 |
+
```
|
| 50 |
+
|
| 51 |
+
## Citation
|
| 52 |
+
|
| 53 |
+
```bibtex
|
| 54 |
+
@misc{worldguide,
|
| 55 |
+
title = {WorldGuide: Goal-Directed Video World Model for Procedural Task Execution},
|
| 56 |
+
author = {Deria, Ankan and Kumar, Komal and Cholakkal, Hisham and Khan, Fahad Shahbaz and Khan, Salman},
|
| 57 |
+
year = {2026}
|
| 58 |
+
}
|
| 59 |
+
```
|