Spaces:
Sleeping
Sleeping
| title: "PPMem: Persistent Planar Memory for Video World Models" | |
| emoji: 🛩️ | |
| colorFrom: red | |
| colorTo: yellow | |
| sdk: gradio | |
| sdk_version: 6.30.0 | |
| app_file: app.py | |
| short_description: Camera-posed video world model with planar memory | |
| python_version: "3.12" | |
| startup_duration_timeout: 1h | |
| pinned: false | |
| license: apache-2.0 | |
| # PPMem: Persistent Planar Memory for Video World Models | |
| Demo of **PPMem** ([paper](https://huggingface.co/papers/2610.23795) · [code](https://github.com/robbyant-research/PPMem) · [weights `Robbyant-Research/PPMem`](https://huggingface.co/Robbyant-Research/PPMem)): an interactive camera-posed video world model built on a Wan 2.2 TI2V-5B DiT that maintains a **persistent memory of 3D planes**. Give it a starting image and a camera trajectory; it generates a 25-frame (97-frame latent-space) RGB video from the first pose, plus the planar geometry (per-patch 3D planes), depth and normal maps, and a world-space planar mesh it predicts for that trajectory. | |
| ## How it works | |
| This Space runs the repo's single-image rollout path (`step0` → `PPMemImageToVideoPipeline`) 1:1: | |
| 1. **Depth bootstrap** — metric depth + intrinsics for the first frame come from the bundled `geometry.npz` (authors' file) or, when none is supplied, from MoGe-2 single-image estimation. | |
| 2. **step0** — the frame-0 image and depth are reprojected along the camera trajectory to build the projected RGB / mask / depth conditioning latents, prefix (very-first + context) latents, and the per-frame intrinsics/extrinsics metadata. | |
| 3. **Denoise** — the PPMem DiT denoises one 25-frame chunk with flow-match UniPC sampling (shift 5.0, guidance 5.0), conditioned on those latents and the UMT5 text prompt embeds, with its plane-memory module keeping planar state persistent across frames. | |
| 4. **Geometry pass** — one final forward with the plane head enabled predicts per-patch 3D planes; these are exported as a world-space planar mesh (PLY), turbo-colormapped depth maps and normal maps. | |
| Outputs: **RGB video** (1024×576 @ 15 fps), **depth video** (turbo colormap, rendered from the predicted planes), **normal maps**, and the **planar mesh** (`gr.Model3D`). | |
| ## Inputs | |
| - **Start image** — the first frame. | |
| - **Camera trajectory** (`.npy`) — optional `(N, 4, 4)` camera-to-world poses. Bundled examples use the authors' trajectories; without one a slow forward dolly is synthesized. | |
| - **Geometry** (`.npz`) — optional depth/focal bootstrap with `depth, fx, fy, cx, cy` keys; when omitted MoGe-2 estimates it. | |
| - **Prompt** — describes the scene (used as in the paper: full scene description). | |
| Advanced: seed, steps (default 50), guidance (default 5.0). | |
| ## Notes | |
| - Runs on ZeroGPU (RTX PRO 6000 Blackwell); one request takes ~5 minutes at 50 steps plus queue. Lowering the steps slider shortens both the runtime and the GPU-time reservation. | |
| - `diff_rect_rasterization` (the repo's rectangle rasterizer) is installed from a wheel built for this hardware (sm_120 / cu130 / torch 2.11), so the DiT's `render_primitives` path is active. The published depth/normal export additionally uses the repo's pure-`numpy` `save_depth_maps` / `save_normal_maps` path. | |
| - `flash_attn` is the FA2 varlen kernel exposed under the FA3 module name the repo imports (`flash_attn_interface.py` shim); FA3 itself cannot run on sm_120. | |
| - Model and code by the PPMem authors (Apache-2.0). Example assets are the authors' `examples/` files (Apache-2.0). |