Update README.md
Browse files
README.md
CHANGED
|
@@ -1,3 +1,75 @@
|
|
| 1 |
---
|
| 2 |
license: apache-2.0
|
| 3 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
license: apache-2.0
|
| 3 |
---
|
| 4 |
+
|
| 5 |
+
<div align="center" style="font-family: charter;">
|
| 6 |
+
|
| 7 |
+
<h1>HPSD: Hybrid-Policy Self-Distillation for Text-Image-to-Video Diffusion Models</h1>
|
| 8 |
+
|
| 9 |
+
<div>
|
| 10 |
+
<a href="https://scholar.google.com/citations?user=a8h9Di4AAAAJ" target="_blank">Jiazi Bu*</a><sup></sup> |
|
| 11 |
+
<a href="https://github.com/LPengYang/" target="_blank">Pengyang Ling*<sup>§</sup></a><sup></sup> |
|
| 12 |
+
<a href="https://github.com/YujieOuO" target="_blank">Yujie Zhou*</a><sup></sup> |
|
| 13 |
+
<a href="https://codegoat24.github.io/" target="_blank">Yibin Wang</a><sup></sup> |
|
| 14 |
+
<a href="https://yuhangzang.github.io/" target="_blank">Yuhang Zang</a><sup></sup> |
|
| 15 |
+
<a href="https://scholar.google.com/citations?user=OeeH1HAAAAAJ&hl=en" target="_blank">Xuanlang Dai</a><sup></sup> | <br>
|
| 16 |
+
<a href="https://scholar.google.com/citations?user=iDPJVBsAAAAJ&hl=zh-CN" target="_blank">Shengyuan Ding</a><sup></sup> |
|
| 17 |
+
<a href="https://wtybest.github.io/" target="_blank">Tianyi Wei</a><sup></sup> |
|
| 18 |
+
<a href="https://xiaohangzhan.github.io/" target="_blank">Xiaohang Zhan</a><sup></sup> |
|
| 19 |
+
<a href="https://myownskyw7.github.io/" target="_blank">Jiaqi Wang</a><sup></sup> |
|
| 20 |
+
<a href="https://wutong16.github.io/" target="_blank">Tong Wu</a><sup></sup> |
|
| 21 |
+
<a href="http://dahua.site/" target="_blank">Dahua Lin</a><sup></sup> |
|
| 22 |
+
<a href="https://xingangpan.github.io/" target="_blank">Xingang Pan<sup>†</sup></a><sup></sup>
|
| 23 |
+
</div>
|
| 24 |
+
<br>
|
| 25 |
+
<div>
|
| 26 |
+
<sup></sup>Shanghai Jiao Tong University, Nanyang Technological University, Shanghai AI Laboratory,
|
| 27 |
+
<br> University of Science and Technology of China, Fudan University, Shanghai Innovation Institute
|
| 28 |
+
<br> The Chinese University of Hong Kong, CPII under InnoHK, JD.com, Adobe Research
|
| 29 |
+
</div>
|
| 30 |
+
(*<b>Equal Contribution</b>)(<sup>§</sup><b>Project Leader</b>)(<sup>†</sup><b>Corresponding Author</b>)
|
| 31 |
+
<br><br>
|
| 32 |
+
|
| 33 |
+
</div>
|
| 34 |
+
|
| 35 |
+
This model is presented in the paper [HPSD: Hybrid-Policy Self-Distillation for Text-Image-to-Video Diffusion Models](https://arxiv.org/abs/2608.13205).
|
| 36 |
+
Project page: [https://bujiazi.github.io/hpsd.github.io/](https://bujiazi.github.io/hpsd.github.io/)
|
| 37 |
+
|
| 38 |
+
## Abstract
|
| 39 |
+
|
| 40 |
+
Text-Image-to-Video (TI2V) models are an emerging unified architecture, where a single model simultaneously supports text-to-video (T2V) and image-to-video (I2V) generation. Given a high-quality first frame or a detailed textual prompt, TI2V models unlock substantially better visual quality than their T2V mode, raising a natural question: _can the capability elicited by such privileged conditions be internalized into the model's own base generation ability?_ A common approach toward this goal is model self-distillation. However, the most straightforward solution, supervised fine-tuning, follows an off-policy strategy: its supervision is confined to teacher-generated endpoints from a fixed offline distribution rather than student-visited states, lacking precise correction tailored to the evolving policy. Recent on-policy distillation methods instead suffer from condition-state mismatch, where supervision is steered toward the given first frame instead of the student's actual content, misleading the correction. To achieve self-distillation that absorbs the teacher's privileged prior while retaining precise policy correction, in this work, we propose **H**ybrid-**P**olicy **S**elf-**D**istillation (**HPSD**), a novel self-distillation framework where a single TI2V model acts as both teacher and student under different conditions: the teacher operates in TI2V mode with a high-quality first frame and an enhanced prompt, while the student runs in the base T2V mode with only the vanilla prompt. Specifically, the student inherits off-policy teacher trajectory points as anchors, locally refines them toward its own policy, and finally receives velocity-level supervision on these self-generated roll-outs. Extensive experiments demonstrate that HPSD significantly improves T2V performance while also delivering notable TI2V gains, effectively strengthening the model's base generation ability.
|
| 41 |
+
|
| 42 |
+
## 🌟 Model
|
| 43 |
+
|
| 44 |
+
The `adapter_model.safetensors` is based on [WAN-2.2-TI2V](https://huggingface.co/Wan-AI/Wan2.2-TI2V-5B-Diffusers) with our HPSD training.
|
| 45 |
+
|
| 46 |
+
## 🔧 Github Link
|
| 47 |
+
[https://github.com/Bujiazi/HPSD](https://github.com/Bujiazi/HPSD)
|
| 48 |
+
|
| 49 |
+
## 🚀 Inference
|
| 50 |
+
```
|
| 51 |
+
TBD
|
| 52 |
+
```
|
| 53 |
+
|
| 54 |
+
|
| 55 |
+
## 📎 Citation
|
| 56 |
+
|
| 57 |
+
If you find our work helpful, please consider giving a star ⭐ and citation 📝
|
| 58 |
+
```bibtex
|
| 59 |
+
@article{bu2026hpsd,
|
| 60 |
+
title={HPSD: Hybrid-Policy Self-Distillation for Text-Image-to-Video Diffusion Models},
|
| 61 |
+
author={Bu, Jiazi and Ling, Pengyang and Zhou, Yujie and Wang, Yibin and Zang, Yuhang and Dai, Xuanlang and Ding, Shengyuan and Wei, Tianyi and Zhan, Xiaohang and Wang, Jiaqi and others},
|
| 62 |
+
journal={arXiv preprint arXiv:2608.13205},
|
| 63 |
+
year={2026}
|
| 64 |
+
}
|
| 65 |
+
```
|
| 66 |
+
|
| 67 |
+
## 💞 Acknowledgements
|
| 68 |
+
The code is built upon the below repositories, we thank all the contributors for open-sourcing.
|
| 69 |
+
|
| 70 |
+
* [Diffusers](https://github.com/huggingface/diffusers)
|
| 71 |
+
* [WAN-2.2](https://github.com/Wan-Video/Wan2.2)
|
| 72 |
+
* [LTX-2.3](https://github.com/Lightricks/LTX-2)
|
| 73 |
+
* [Z-Image](https://github.com/Tongyi-MAI/Z-Image)
|
| 74 |
+
* [Qwen3.6-27B](https://github.com/QwenLM/Qwen3.6)
|
| 75 |
+
* [D-OPSD](https://github.com/vvvvvjdy/D-OPSD)
|