| --- |
| license: apache-2.0 |
| --- |
| |
| <div align="center" style="font-family: charter;"> |
|
|
| <h1>HPSD: Hybrid-Policy Self-Distillation for Text-Image-to-Video Diffusion Models</h1> |
|
|
| <div style="font-size: 15px;"> |
| <a href="https://scholar.google.com/citations?user=a8h9Di4AAAAJ" target="_blank">Jiazi Bu*</a><sup></sup> | |
| <a href="https://github.com/LPengYang/" target="_blank">Pengyang Ling*<sup>§</sup></a><sup></sup> | |
| <a href="https://github.com/YujieOuO" target="_blank">Yujie Zhou*</a><sup></sup> | |
| <a href="https://codegoat24.github.io/" target="_blank">Yibin Wang</a><sup></sup> | |
| <a href="https://yuhangzang.github.io/" target="_blank">Yuhang Zang</a><sup></sup> | |
| <a href="https://scholar.google.com/citations?user=OeeH1HAAAAAJ&hl=en" target="_blank">Xuanlang Dai</a><sup></sup> | <br> |
| <a href="https://scholar.google.com/citations?user=iDPJVBsAAAAJ&hl=zh-CN" target="_blank">Shengyuan Ding</a><sup></sup> | |
| <a href="https://wtybest.github.io/" target="_blank">Tianyi Wei</a><sup></sup> | |
| <a href="https://xiaohangzhan.github.io/" target="_blank">Xiaohang Zhan</a><sup></sup> | |
| <a href="https://myownskyw7.github.io/" target="_blank">Jiaqi Wang</a><sup></sup> | |
| <a href="https://wutong16.github.io/" target="_blank">Tong Wu</a><sup></sup> | |
| <a href="http://dahua.site/" target="_blank">Dahua Lin</a><sup></sup> | |
| <a href="https://xingangpan.github.io/" target="_blank">Xingang Pan<sup>†</sup></a><sup></sup> |
| </div> |
| <br> |
| <div> |
| <sup></sup>Shanghai Jiao Tong University, Nanyang Technological University, Shanghai AI Laboratory, |
| <br> University of Science and Technology of China, Fudan University, Shanghai Innovation Institute |
| <br> The Chinese University of Hong Kong, CPII under InnoHK, JD.com, Adobe Research |
| </div> |
| (*<b>Equal Contribution</b>)(<sup>§</sup><b>Project Leader</b>)(<sup>†</sup><b>Corresponding Author</b>) |
| <br><br> |
| |
| </div> |
|
|
| This model is presented in the paper [HPSD: Hybrid-Policy Self-Distillation for Text-Image-to-Video Diffusion Models](https://arxiv.org/abs/2608.13205). |
| Project page: [https://bujiazi.github.io/hpsd.github.io/](https://bujiazi.github.io/hpsd.github.io/) |
|
|
| ## Abstract |
|
|
| Text-Image-to-Video (TI2V) models are an emerging unified architecture, where a single model simultaneously supports text-to-video (T2V) and image-to-video (I2V) generation. Given a high-quality first frame or a detailed textual prompt, TI2V models unlock substantially better visual quality than their T2V mode, raising a natural question: _can the capability elicited by such privileged conditions be internalized into the model's own base generation ability?_ A common approach toward this goal is model self-distillation. However, the most straightforward solution, supervised fine-tuning, follows an off-policy strategy: its supervision is confined to teacher-generated endpoints from a fixed offline distribution rather than student-visited states, lacking precise correction tailored to the evolving policy. Recent on-policy distillation methods instead suffer from condition-state mismatch, where supervision is steered toward the given first frame instead of the student's actual content, misleading the correction. To achieve self-distillation that absorbs the teacher's privileged prior while retaining precise policy correction, in this work, we propose **H**ybrid-**P**olicy **S**elf-**D**istillation (**HPSD**), a novel self-distillation framework where a single TI2V model acts as both teacher and student under different conditions: the teacher operates in TI2V mode with a high-quality first frame and an enhanced prompt, while the student runs in the base T2V mode with only the vanilla prompt. Specifically, the student inherits off-policy teacher trajectory points as anchors, locally refines them toward its own policy, and finally receives velocity-level supervision on these self-generated roll-outs. Extensive experiments demonstrate that HPSD significantly improves T2V performance while also delivering notable TI2V gains, effectively strengthening the model's base generation ability. |
|
|
| ## 🌟 Model |
|
|
| The `adapter_model.safetensors` is based on [WAN-2.2-TI2V](https://huggingface.co/Wan-AI/Wan2.2-TI2V-5B-Diffusers) with our HPSD training. |
|
|
| ## 🔧 Github Link |
| [https://github.com/Bujiazi/HPSD](https://github.com/Bujiazi/HPSD) |
|
|
| ## 🚀 Inference |
| ```python |
| import torch |
| from huggingface_hub import snapshot_download |
| from diffusers import WanPipeline, AutoencoderKLWan |
| from diffusers.utils import export_to_video |
| from peft import PeftModel |
| |
| dtype = torch.bfloat16 |
| device = "cuda" |
| |
| model_id = "Wan-AI/Wan2.2-TI2V-5B-Diffusers" |
| checkpoint = snapshot_download(repo_id="Bujiazi/HPSD", repo_type="model") |
| |
| vae = AutoencoderKLWan.from_pretrained(model_id, subfolder="vae", torch_dtype=torch.float32) |
| pipe = WanPipeline.from_pretrained(model_id, vae=vae, torch_dtype=dtype) |
| pipe.to(device) |
| |
| pipe.transformer = PeftModel.from_pretrained(pipe.transformer, checkpoint, torch_dtype=dtype).to(device) |
| |
| height = 704 |
| width = 1280 |
| num_frames = 81 |
| num_inference_steps = 50 |
| guidance_scale = 5.0 |
| base_seed = 42 |
| |
| prompt = "A sports car speeding along a winding coastal road at sunset. Dynamic low-angle tracking shot, realistic reflections, dramatic clouds, subtle motion blur, cinematic composition." |
| negative_prompt = "色调艳丽，过曝，静态，细节模糊不清，字幕，风格，作品，画作，画面，静止，整体发灰，最差质量，低质量，JPEG压缩残留，丑陋的，残缺的，多余的手指，画得不好的手部，画得不好的脸部，畸形的，毁容的，形态畸形的肢体，手指融合，静止不动的画面，杂乱的背景，三条腿，背景人很多，倒着走" |
| |
| generator = torch.Generator(device=device).manual_seed(base_seed) |
| output = pipe( |
| prompt=prompt, |
| negative_prompt=negative_prompt, |
| height=height, |
| width=width, |
| num_frames=num_frames, |
| guidance_scale=guidance_scale, |
| num_inference_steps=num_inference_steps, |
| generator=generator, |
| ).frames[0] |
| export_to_video(output, "hpsd_test.mp4", fps=16) |
| ``` |
|
|
|
|
| ## 📎 Citation |
|
|
| If you find our work helpful, please consider giving a star ⭐ and citation 📝 |
| ```bibtex |
| @article{bu2026hpsd, |
| title={HPSD: Hybrid-Policy Self-Distillation for Text-Image-to-Video Diffusion Models}, |
| author={Bu, Jiazi and Ling, Pengyang and Zhou, Yujie and Wang, Yibin and Zang, Yuhang and Dai, Xuanlang and Ding, Shengyuan and Wei, Tianyi and Zhan, Xiaohang and Wang, Jiaqi and others}, |
| journal={arXiv preprint arXiv:2608.13205}, |
| year={2026} |
| } |
| ``` |
|
|
| ## 💞 Acknowledgements |
| The code is built upon the below repositories, we thank all the contributors for open-sourcing. |
|
|
| * [Diffusers](https://github.com/huggingface/diffusers) |
| * [WAN-2.2](https://github.com/Wan-Video/Wan2.2) |
| * [LTX-2.3](https://github.com/Lightricks/LTX-2) |
| * [Z-Image](https://github.com/Tongyi-MAI/Z-Image) |
| * [Qwen3.6-27B](https://github.com/QwenLM/Qwen3.6) |
| * [D-OPSD](https://github.com/vvvvvjdy/D-OPSD) |
|
|
