README.md

---
language: en
license: creativeml-openrail-m
tags:
- stable-diffusion
- stable-diffusion-diffusers
- controlnet
- controlnet-v1-1
- canny
- text-to-image
- image-to-image
- RyzenAI
- Quantization
- ONNX
- Computer Vision
inference: true
---
# 🚀 Stable Diffusion 1.5 ControlNet (Canny) on AMD AI PC NPU

ControlNet v1.1 (Canny) adds conditional control to text-to-image diffusion models using Canny edge maps as guidance. It is designed to be used with Stable Diffusion 1.5. For more information about how ControlNet works, please see the [🧨 Diffusers ControlNet blog](https://huggingface.co/blog/controlnet) and the [ControlNet paper](https://arxiv.org/abs/2302.05543).

More details about this model can be found on the original Hugging Face model card: [lllyasviel/control_v11p_sd15_canny](https://huggingface.co/lllyasviel/control_v11p_sd15_canny).

This model repo contains the optimized ONNX models required to run the Stable Diffusion 1.5 + ControlNet (Canny) image generation pipeline on AMD NPUs.

## Model Details

The folder structure is organized to mirror the main components of the ControlNet diffusion pipeline (controlnet, scheduler, text encoder, tokenizer, UNet, VAE decoder, and supporting modules).

```text
├─ controlnet/
├─ feature_extractor/
├─ safety_checker/
├─ scheduler/
├─ text_encoder/
├─ tokenizer/
├─ unet/
└─ vae_decoder/
```

The [controlnet](https://huggingface.co/amd/sd1.5-controlnet-canny-amdnpu/tree/main/controlnet) folder contains the ControlNet model conditioned on Canny edges. It is exported and structured specifically to leverage the AMD NPU accelerator together with the UNet during denoising.

The [scheduler](https://huggingface.co/amd/sd1.5-controlnet-canny-amdnpu/tree/main/scheduler) folder contains the scheduler configuration (timesteps, betas, alphas, etc.) used during the diffusion sampling process.

The [text_encoder](https://huggingface.co/amd/sd1.5-controlnet-canny-amdnpu/tree/main/text_encoder) folder contains the text encoder model used to convert the input prompt into conditioning embeddings for the diffusion model.

The [tokenizer](https://huggingface.co/amd/sd1.5-controlnet-canny-amdnpu/tree/main/tokenizer) contains the tokenizer configuration and vocabulary files required to preprocess the text prompt before it is fed to the text encoder.

The [unet](https://huggingface.co/amd/sd1.5-controlnet-canny-amdnpu/tree/main/unet) folder contains the UNet model used in the diffusion process. The UNet is exported and structured specifically to leverage the AMD NPU accelerator for the denoising steps.

The [vae_decoder](https://huggingface.co/amd/sd1.5-controlnet-canny-amdnpu/tree/main/vae_decoder) folder contains the VAE decoder model used to map latent representations back to the image space. The VAE decoder is also structured to make use of the NPU accelerator for efficient image reconstruction.

The [feature_extractor](https://huggingface.co/amd/sd1.5-controlnet-canny-amdnpu/tree/main/feature_extractor) and [safety_checker](https://huggingface.co/amd/sd1.5-controlnet-canny-amdnpu/tree/main/safety_checker) folders contain supporting components used with the Stable Diffusion 1.5 pipeline.

> Note: ControlNet, UNet, and VAE decoder models are optimized and structured to run on AMD NPUs. The other components (text encoder, tokenizer, scheduler, feature extractor, and safety checker) are shared between GPU and NPU pipelines, but are provided here for completeness.

| Model Details | Description |
| ----------- | ----------- |
| Person or organization developing model | [Benjamin Consolvo (AMD)](https://huggingface.co/bconsolvo), [makn87amd (AMD)](https://huggingface.co/makn87amd) |
| Original model authors | [Lvmin Zhang](https://huggingface.co/lllyasviel), [Maneesh Agrawala](https://graphics.stanford.edu/~maneesh/) |
| Model date | May 2026 |
| Model version | 1.8.0 |
| Model type | ControlNet-conditioned text-to-image generation (Canny edges) |
| Information about training algorithms, parameters, fairness constraints or other applied approaches, and features | ControlNet v1.1 Canny checkpoint trained on [runwayml/stable-diffusion-v1-5](https://huggingface.co/stable-diffusion-v1-5/stable-diffusion-v1-5). Adds Canny edge conditioning to control image structure during generation. See the [original model card](https://huggingface.co/lllyasviel/control_v11p_sd15_canny) for training and usage details. |
| License | [CreativeML OpenRAIL-M](https://huggingface.co/amd/sd1.5-controlnet-canny-amdnpu/blob/main/LICENSE) — see [lllyasviel/control_v11p_sd15_canny](https://huggingface.co/lllyasviel/control_v11p_sd15_canny) |
| Where to send questions or comments about the model | [Community Tab](https://hf.co/amd/sd1.5-controlnet-canny-amdnpu/discussions) and [AMD Developer Community Discord](https://discord.gg/amd-dev) |

## ⚡ Intended Use

#### Getting Started

To get started with this model, visit [github.com/amd/sd-sandbox](https://github.com/amd/sd-sandbox).

## ⚓ Ethical Considerations

AMD is committed to conducting our business in a fair, ethical and honest manner and in compliance with all applicable laws, rules and regulations. You can find out more at the [AMD Ethics and Compliance](https://www.amd.com/en/corporate/corporate-responsibility/ethics-and-compliance.html) page.

## ⚠️ Caveats and Recommendations

Please visit the original model card for more details on usage, preprocessing (Canny edge extraction), and limitations: [lllyasviel/control_v11p_sd15_canny](https://huggingface.co/lllyasviel/control_v11p_sd15_canny).

## 📌 Citation Details

```bibtex
@misc{zhang2023adding,
      title={Adding Conditional Control to Text-to-Image Diffusion Models},
      author={Lvmin Zhang and Maneesh Agrawala},
      year={2023},
      eprint={2302.05543},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}
```