xiaoyu1104/InstanceControl_hed

InstanceControl: Controllable Complex Image Generation without Instance Labeling

This repository contains the Stage 1 model (Sa2va-Instance-4B) for InstanceControl, a multi-instance controllable generation method that eliminates the need for manual instance labeling.

• Paper: InstanceControl: Controllable Complex Image Generation without Instance Labeling
• Project Page: InstanceControl Project Page
• Repository: GitHub - InstanceControl

Method Overview

InstanceControl leverages a Vision-Language Model (VLM) to establish instance-level correspondences between text prompts and visual conditions (such as Canny edge, depth, or HED maps). It automatically parses instance descriptions from the text prompts and simultaneously predicts instance masks based on the visual conditions. An adaptive mask refinement strategy dynamically refines these instance masks during the generation process to achieve superior fidelity and precise instance-level control.

Citation

If you find this work useful, please consider citing:

@article{instancecontrol,
  title   = {InstanceControl: Controllable Complex Image Generation without Instance Labeling},
  author  = {Xiaoyu Liu and Huan Wang and Fan Li and Zhixin Wang and Jiaqi Xu and Ming Liu and Wangmeng Zuo},
  journal = {arXiv preprint arXiv:2606.31924},
  year    = {2026}
}