README.md

metadata

title: 'PPMem: Persistent Planar Memory for Video World Models'
emoji: 🛩️
colorFrom: red
colorTo: yellow
sdk: gradio
sdk_version: 6.30.0
app_file: app.py
short_description: Camera-posed video world model with planar memory
python_version: '3.12'
startup_duration_timeout: 1h
pinned: false
license: apache-2.0

PPMem: Persistent Planar Memory for Video World Models

Demo of PPMem (paper · code · weights Robbyant-Research/PPMem): an interactive camera-posed video world model built on a Wan 2.2 TI2V-5B DiT that maintains a persistent memory of 3D planes. Give it a starting image and a camera trajectory; it generates a 25-frame (97-frame latent-space) RGB video from the first pose, plus the planar geometry (per-patch 3D planes), depth and normal maps, and a world-space planar mesh it predicts for that trajectory.

How it works

This Space runs the repo's single-image rollout path (step0 → PPMemImageToVideoPipeline) 1:1:

1. Depth bootstrap — metric depth + intrinsics for the first frame come from the bundled geometry.npz (authors' file) or, when none is supplied, from MoGe-2 single-image estimation.
2. step0 — the frame-0 image and depth are reprojected along the camera trajectory to build the projected RGB / mask / depth conditioning latents, prefix (very-first + context) latents, and the per-frame intrinsics/extrinsics metadata.
3. Denoise — the PPMem DiT denoises one 25-frame chunk with flow-match UniPC sampling (shift 5.0, guidance 5.0), conditioned on those latents and the UMT5 text prompt embeds, with its plane-memory module keeping planar state persistent across frames.
4. Geometry pass — one final forward with the plane head enabled predicts per-patch 3D planes; these are exported as a world-space planar mesh (PLY), turbo-colormapped depth maps and normal maps.

Outputs: RGB video (1024×576 @ 15 fps), depth video (turbo colormap, rendered from the predicted planes), normal maps, and the planar mesh (gr.Model3D).

Inputs

• Start image — the first frame.
• Camera trajectory (.npy) — optional (N, 4, 4) camera-to-world poses. Bundled examples use the authors' trajectories; without one a slow forward dolly is synthesized.
• Geometry (.npz) — optional depth/focal bootstrap with depth, fx, fy, cx, cy keys; when omitted MoGe-2 estimates it.
• Prompt — describes the scene (used as in the paper: full scene description).

Advanced: seed, steps (default 50), guidance (default 5.0).

Notes

• Runs on ZeroGPU (RTX PRO 6000 Blackwell); one request takes ~5 minutes at 50 steps plus queue. Lowering the steps slider shortens both the runtime and the GPU-time reservation.
• diff_rect_rasterization (the repo's rectangle rasterizer) is installed from a wheel built for this hardware (sm_120 / cu130 / torch 2.11), so the DiT's render_primitives path is active. The published depth/normal export additionally uses the repo's pure-numpy save_depth_maps / save_normal_maps path.
• flash_attn is the FA2 varlen kernel exposed under the FA3 module name the repo imports (flash_attn_interface.py shim); FA3 itself cannot run on sm_120.
• Model and code by the PPMem authors (Apache-2.0). Example assets are the authors' examples/ files (Apache-2.0).