README.md

<div align="center">

# From Draft to Draft-Free: One-Step Video Object Removal via Privileged Distillation and Fast Planting

**Zizhao Chen, Ping Wei<sup>†</sup>, Guang Dai, Jingdong Wang, Mengmeng Wang<sup>†</sup>**

Xi'an Jiaotong University &nbsp;·&nbsp; SGIT AI Lab &nbsp;·&nbsp;
Zhejiang University of Technology &nbsp;·&nbsp; Baidu

<sup>†</sup> Corresponding authors

<a href="https://bigd233.github.io/D2DF/"><img src="docs/assets/project-page-badge.svg" alt="Project Page"></a>
<a href="https://arxiv.org/abs/2607.14976"><img src="https://img.shields.io/badge/arXiv-Paper-b31b1b?logo=arxiv&logoColor=white" alt="arXiv Paper"></a>
<a href="https://huggingface.co/BigD233333/D2DF"><img src="https://img.shields.io/badge/%F0%9F%A4%97-Hugging_Face-FFD21E" alt="Hugging Face"></a>
<a href="https://github.com/bigD233/D2DF"><img src="https://img.shields.io/badge/GitHub-Code-181717?logo=github&logoColor=white" alt="GitHub Code"></a>

</div>

<p align="center">
  <img src="docs/assets/teaser.png" alt="Overview of the D2DF framework" width="100%">
</p>

**D2DF** is a three-stage framework for one-step video object removal. It first
distills a multi-step draft-guided diffusion teacher into **D2DF-DG**, a one-step
draft refiner, through **Prior-Privileged Consistency Distillation (PPCD)**. It then
uses the lightweight **Self-Guided Fast Planting (SGFP)** module to construct
pseudo-drafts in latent space, producing **D2DF-DF**, a fully draft-free one-step
generator.

This repository provides standalone inference code for both D2DF-DF and D2DF-DG
using the official Hugging Face `diffusers` and `transformers` packages.

## Installation

```bash
git clone https://github.com/bigD233/D2DF.git
cd D2DF
pip install -r requirements.txt
```

We recommend Python 3.11 and an NVIDIA GPU with CUDA support. The released
inference code was tested with PyTorch 2.8, CUDA 12.8, `diffusers >= 0.32`, and
`transformers >= 4.44`.

## Model Weights

Download the base **CogVideoX-5B-I2V** model from its
[official Hugging Face repository](https://huggingface.co/zai-org/CogVideoX-5b-I2V).

Download the **D2DF-DF** and **D2DF-DG** checkpoints from our
[D2DF Hugging Face repository](https://huggingface.co/BigD233333/D2DF). After
extracting the downloaded archives, pass the corresponding directories to
`--transformer_path`.

The included `prompt_embeds.pt` stores the pre-computed embedding for the prompt
`"background"`, so the T5 encoder does not need to be loaded during normal
inference.

## Input Format

Prepare the input as directories of numbered image files:

```text
inputs/
├── imgs/
│   ├── 00000.jpg
│   ├── 00001.jpg
│   └── ...
├── mask/
│   ├── 00000.png
│   ├── 00001.png
│   └── ...
└── drafts/                 # required only by D2DF-DG
    ├── 00000.jpg
    ├── 00001.jpg
    └── ...
```

- The repository includes this ready-to-run 25-frame example under `inputs/`.
- Frames, masks, and drafts are read in filename order.
- Mask pixels belonging to the removal target should have value `255` by default.
- Inputs are resized to the requested output resolution.
- The default setting uses 25 frames at 480 x 720 resolution.

## Inference

### D2DF-DF: Draft-Free One-Step Removal

No external draft is required:

```bash
python infer.py \
  --mode df \
  --base_model_path <PATH_TO_COGVIDEOX_5B_I2V> \
  --transformer_path <PATH_TO_D2DF_DF> \
  --frames_dir inputs/imgs \
  --mask_dir inputs/mask \
  --output_dir outputs/I-210910_I09032_T04_df \
  --num_frames 25 \
  --height 480 \
  --width 720
```

### D2DF-DG: Draft-Guided One-Step Refinement

Supply a draft produced by a video object removal method such as ProPainter:

```bash
python infer.py \
  --mode dg \
  --base_model_path <PATH_TO_COGVIDEOX_5B_I2V> \
  --transformer_path <PATH_TO_D2DF_DG> \
  --frames_dir inputs/imgs \
  --mask_dir inputs/mask \
  --draft_dir inputs/drafts \
  --output_dir outputs/I-210910_I09032_T04_dg \
  --num_frames 25 \
  --height 480 \
  --width 720
```

The default configuration performs one denoising step with
`guidance_scale=1.0`. The output directory contains an MP4 video and a GIF
preview. Add `--save_frames` to also save individual PNG frames.

## Citation

```bibtex
@misc{chen2026d2df,
  title={From Draft to Draft-Free: One-Step Video Object Removal via Privileged Distillation and Fast Planting},
  author={Zizhao Chen and Ping Wei and Guang Dai and Jingdong Wang and Mengmeng Wang},
  year={2026}
}
```

## Acknowledgements

Our implementation is built upon
[CogVideoX](https://github.com/THUDM/CogVideo),
[Hugging Face Diffusers](https://github.com/huggingface/diffusers), and
[ProPainter](https://github.com/sczhou/ProPainter). We sincerely thank the
contributors of these projects for their excellent work.

## License

This repository is released under the [Apache License 2.0](LICENSE).

## Contact

For questions about this work, please contact
[Zizhao Chen](mailto:bigd_chen@stu.xjtu.edu.cn).