README.md

metadata

title: SCoPE Camera Controlled Video
emoji: 🎥
colorFrom: indigo
colorTo: green
sdk: gradio
sdk_version: 6.15.1
app_file: app.py
python_version: 3.10.13
startup_duration_timeout: 1h
pinned: false
license: apache-2.0
short_description: Steer a 3D camera path through any still image
models:
  - TencentARC/SCoPE
  - lightx2v/Wan2.2-Lightning
tags:
  - image-to-video
  - camera-control
  - wan2.2

SCoPE — camera-trajectory controlled image-to-video

TencentARC/SCoPE retrofits Wan2.2-I2V-A14B with Sightline-Coordinate Positional Encoding: Plücker camera rays are normalised, gated and injected directly into the DiT's self-attention queries and keys, so an explicit 3D camera path steers the generated shot. Give it a first frame and a camera move, and it returns an 81-frame, 832×480, 16 fps clip. A caption is used for the scene content; if you leave it blank, a small vision-language model (Qwen/Qwen2-VL-2B-Instruct) writes one from your first frame automatically.

• Paper / project page: https://visual-ai.github.io/scope/
• Reference code: https://github.com/TencentARC/SCoPE

What this Space runs

The app uses SCoPE's own code (scope/) and its vendored DiffSynth-Studio pipeline (diffsynth/, restored to the v1.1.8 modules the release was cut against), driven exactly as scope/inference.py does: pipe(prompt=…, input_image=…, camera_control_panshot={pose, x_fov, xi}, height=480, width=832, num_frames=81, sigma_shift=5.0, switch_DiT_boundary=0.9, camera_cfg_scale=1.0, lock_first_frame=False, tiled=False).

Camera paths are OpenCV camera-to-world matrices of shape [81, 3, 4], expressed relative to the first frame, which is the format SCoPE was trained on. The 16 presets in trajectories/ are taken from the release's own examples/ trajectory set (re-expressed relative to frame 0), and the "camera motion scale" slider rescales only their translation component. Every preset carries a real translation; pure in-place rotations were dropped because their effect is hard to see.

Deviations from the reference configuration

The full 67 GB checkpoint at the paper's sampling settings (40 steps, cfg_scale=3.5) needs several minutes of GPU per clip, far beyond a ZeroGPU slot. To fit, this Space:

1. Serves both 14B experts in fp8 (torchao, Float8DynamicActivationFloat8WeightConfig) and the UMT5-XXL text encoder in int8. SCoPE's Plücker/gate MLPs are deliberately left in bf16.
2. Fuses the Wan2.2-Lightning 4-step distillation LoRA (Wan2.2-I2V-A14B-4steps-lora-rank64-Seko-V1, strength 1.0) into both experts and samples 4 steps at cfg_scale = 1.0, per the LoRA's reference workflow (shift 5, split 2+2 — which is what switch_DiT_boundary = 0.9 produces).
3. Streams the checkpoint shard by shard, deleting each file once its tensors are quantised, so the 71 GB download never has to exist on disk at once.

Expect slightly softer detail and less prompt adherence than the official 40-step samples; camera control itself is architectural and unaffected by the shorter schedule.

Credits

Model, code and example images © the SCoPE authors (Apache-2.0). The showcase images in examples/ are drawn from the SCoPE release: three AI-generated scenes, one iStock aerial clip (rice-field motorbike), and two OmniWorld character scenes (kept in their own "with characters" group). Automatic prompting uses Qwen/Qwen2-VL-2B-Instruct (Apache-2.0).