README copy.md

DragStream

Streaming Drag-Oriented Interactive Video Manipulation: Drag Anything, Anytime!

Junbao Zhou¹ · Yuan Zhou¹ · Kesen Zhao¹ · Qingshan Xu¹ · Beier Zhu¹ · Richang Hong² · Hanwang Zhang^1
1Nanyang Technological University ²Hefei University of Technology

---

Achieving streaming, fine-grained control over the outputs of autoregressive video diffusion models remains challenging, making it difficult to ensure that they consistently align with user expectations. To bridge this gap, we propose stReaming drag-oriEnted interactiVe vidEo manipuLation (REVEL), a new task that enables users to modify generated videos anytime on anything via fine-grained, interactive drag. Beyond DragVideo and SG-I2V, REVEL unifies drag-style video manipulation as editing and animating video frames with both supporting user-specified translation, deformation, and rotation effects, making drag operations versatile. In resolving REVEL, we observe: i) drag-induced perturbations accumulate in latent space, causing severe latent distribution drift that halts the drag process; ii) streaming drag is easily disturbed by context frames, thereby yielding visually unnatural outcomes. We thus propose a training-free approach, DragStream, comprising: i) an adaptive distribution self-rectification strategy that leverages neighboring frames' statistics to effectively constrain the drift of latent embeddings; ii) a spatial-frequency selective optimization mechanism, allowing the model to fully exploit contextual information while mitigating its interference via selectively propagating visual cues along generation. Our method can be seamlessly integrated into existing autoregressive video diffusion models, and extensive experiments firmly demonstrate the effectiveness of our DragStream

---

Requirements

We tested this repo on the following setup:

• Nvidia GPU with at least 40 GB memory.
• Linux operating system.
• 64 GB RAM.

Other hardware setup could also work but hasn't been tested.

Installation

1. Follow Self-Forcing to Install Dependencies

Create a conda environment and install dependencies:

conda create -n drag-stream python=3.10 -y
conda activate drag-stream
pip install -r requirements.txt
pip install flash-attn --no-build-isolation
python setup.py develop

2. Follow Self-Forcing to Download Checkpoints

huggingface-cli download Wan-AI/Wan2.1-T2V-1.3B --local-dir wan_models/Wan2.1-T2V-1.3B
huggingface-cli download gdhe17/Self-Forcing checkpoints --local-dir ./checkpoints

3. Follow Segment-Anything to Install SAM

git clone git@github.com:facebookresearch/segment-anything.git
cd segment-anything; pip install -e .

4. Follow Segment-Anything to Download SAM Checkpoint

• ViT-H SAM model.

Drag/Animate Video with GUI

python app.py

CLI Inference with Saved Trajectories

python offline_run.py

Reproducibility

To ensure every Drag/Animation is performed on the same generated video given the same input conditions, we set the random seed before the initialization of random noise and before the generation process.

Please refer to the set_seed(seed) in inference.py, stream_inference.py, click_gui_video.py, and offline_run.py for details.

Acknowledgements

This codebase is built on top of the open-source implementation of Self-Forcing.