diffusers/v0.5.1/en/using-diffusers/custom_pipelines.html

<meta charset="utf-8" /><meta http-equiv="content-security-policy" content=""><meta name="hf:doc:metadata" content="{&quot;local&quot;:&quot;custom-pipelines&quot;,&quot;sections&quot;:[{&quot;local&quot;:&quot;loading-custom-pipelines-from-the-hub&quot;,&quot;title&quot;:&quot;Loading custom pipelines from the Hub&quot;},{&quot;local&quot;:&quot;loading-official-community-pipelines&quot;,&quot;title&quot;:&quot;Loading official community pipelines&quot;},{&quot;local&quot;:&quot;adding-custom-pipelines-to-the-hub&quot;,&quot;title&quot;:&quot;Adding custom pipelines to the Hub&quot;}],&quot;title&quot;:&quot;Custom Pipelines&quot;}" data-svelte="svelte-1phssyn">
	<link rel="modulepreload" href="/docs/diffusers/v0.5.1/en/_app/assets/pages/__layout.svelte-hf-doc-builder.css">
	<link rel="modulepreload" href="/docs/diffusers/v0.5.1/en/_app/start-hf-doc-builder.js">
	<link rel="modulepreload" href="/docs/diffusers/v0.5.1/en/_app/chunks/vendor-hf-doc-builder.js">
	<link rel="modulepreload" href="/docs/diffusers/v0.5.1/en/_app/chunks/paths-hf-doc-builder.js">
	<link rel="modulepreload" href="/docs/diffusers/v0.5.1/en/_app/pages/__layout.svelte-hf-doc-builder.js">
	<link rel="modulepreload" href="/docs/diffusers/v0.5.1/en/_app/pages/using-diffusers/custom_pipelines.mdx-hf-doc-builder.js">
	<link rel="modulepreload" href="/docs/diffusers/v0.5.1/en/_app/chunks/Tip-hf-doc-builder.js">
	<link rel="modulepreload" href="/docs/diffusers/v0.5.1/en/_app/chunks/IconCopyLink-hf-doc-builder.js">
	<link rel="modulepreload" href="/docs/diffusers/v0.5.1/en/_app/chunks/CodeBlock-hf-doc-builder.js"> 






<h1 class="relative group"><a id="custom-pipelines" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#custom-pipelines"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a>
	<span>Custom Pipelines
	</span></h1>

<p>Diffusers allows you to conveniently load any custom pipeline from the Hugging Face Hub as well as any <a href="https://github.com/huggingface/diffusers/tree/main/examples/community" rel="nofollow">official community pipeline</a>
via the <a href="/docs/diffusers/v0.5.1/en/api/diffusion_pipeline#diffusers.DiffusionPipeline">DiffusionPipeline</a> class.</p>
<h2 class="relative group"><a id="loading-custom-pipelines-from-the-hub" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#loading-custom-pipelines-from-the-hub"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a>
	<span>Loading custom pipelines from the Hub
	</span></h2>

<p>Custom pipelines can be easily loaded from any model repository on the Hub that defines a diffusion pipeline in a <code>pipeline.py</code> file.
Let’s load a dummy pipeline from <a href="https://huggingface.co/hf-internal-testing/diffusers-dummy-pipeline" rel="nofollow">hf-internal-testing/diffusers-dummy-pipeline</a>.</p>
<p>All you need to do is pass the custom pipeline repo id with the <code>custom_pipeline</code> argument alongside the repo from where you wish to load the pipeline modules.</p>

	<div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5   text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg>
	<div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div>
	Copied</div></button></div>
	<pre><!-- HTML_TAG_START --><span class="hljs-keyword">from</span> diffusers <span class="hljs-keyword">import</span> DiffusionPipeline

pipeline = DiffusionPipeline.from_pretrained(
    <span class="hljs-string">&quot;google/ddpm-cifar10-32&quot;</span>, custom_pipeline=<span class="hljs-string">&quot;hf-internal-testing/diffusers-dummy-pipeline&quot;</span>
)<!-- HTML_TAG_END --></pre></div>
<p>This will load the custom pipeline as defined in the <a href="https://huggingface.co/hf-internal-testing/diffusers-dummy-pipeline/blob/main/pipeline.py" rel="nofollow">model repository</a>.</p>


<div class="course-tip course-tip-orange bg-gradient-to-br dark:bg-gradient-to-r before:border-orange-500 dark:before:border-orange-800 from-orange-50 dark:from-gray-900 to-white dark:to-gray-950 border border-orange-50 text-orange-700 dark:text-gray-400"><p>By loading a custom pipeline from the Hugging Face Hub, you are trusting that the code you are loading
is safe 🔒. Make sure to check out the code online before loading &amp; running it automatically.</p></div>
<h2 class="relative group"><a id="loading-official-community-pipelines" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#loading-official-community-pipelines"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a>
	<span>Loading official community pipelines
	</span></h2>

<p>Community pipelines are summarized in the <a href="https://github.com/huggingface/diffusers/tree/main/examples/community" rel="nofollow">community examples folder</a></p>
<p>Similarly, you need to pass both the <em>repo id</em> from where you wish to load the weights as well as the <code>custom_pipeline</code> argument. Here the <code>custom_pipeline</code> argument should consist simply of the filename of the community pipeline excluding the <code>.py</code> suffix, <em>e.g.</em> <code>clip_guided_stable_diffusion</code>.</p>
<p>Since community pipelines are often more complex, one can mix loading weights from an official <em>repo id</em>
and passing pipeline modules directly.</p>

	<div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5   text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg>
	<div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div>
	Copied</div></button></div>
	<pre><!-- HTML_TAG_START --><span class="hljs-keyword">from</span> diffusers <span class="hljs-keyword">import</span> DiffusionPipeline
<span class="hljs-keyword">from</span> transformers <span class="hljs-keyword">import</span> CLIPFeatureExtractor, CLIPModel

clip_model_id = <span class="hljs-string">&quot;laion/CLIP-ViT-B-32-laion2B-s34B-b79K&quot;</span>

feature_extractor = CLIPFeatureExtractor.from_pretrained(clip_model_id)
clip_model = CLIPModel.from_pretrained(clip_model_id)

pipeline = DiffusionPipeline.from_pretrained(
    <span class="hljs-string">&quot;CompVis/stable-diffusion-v1-4&quot;</span>,
    custom_pipeline=<span class="hljs-string">&quot;clip_guided_stable_diffusion&quot;</span>,
    clip_model=clip_model,
    feature_extractor=feature_extractor,
)<!-- HTML_TAG_END --></pre></div>
<h2 class="relative group"><a id="adding-custom-pipelines-to-the-hub" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#adding-custom-pipelines-to-the-hub"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a>
	<span>Adding custom pipelines to the Hub
	</span></h2>

<p>To add a custom pipeline to the Hub, all you need to do is to define a pipeline class that inherits
from <a href="/docs/diffusers/v0.5.1/en/api/diffusion_pipeline#diffusers.DiffusionPipeline">DiffusionPipeline</a> in a <code>pipeline.py</code> file.
Make sure that the whole pipeline is encapsulated within a single class and that the <code>pipeline.py</code> file
has only one such class.</p>
<p>Let’s quickly define an example pipeline.</p>

	<div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5   text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg>
	<div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div>
	Copied</div></button></div>
	<pre><!-- HTML_TAG_START --><span class="hljs-keyword">import</span> torch
<span class="hljs-keyword">from</span> diffusers <span class="hljs-keyword">import</span> DiffusionPipeline


<span class="hljs-keyword">class</span> <span class="hljs-title class_">MyPipeline</span>(<span class="hljs-title class_ inherited__">DiffusionPipeline</span>):
    <span class="hljs-keyword">def</span> <span class="hljs-title function_">__init__</span>(<span class="hljs-params">self, unet, scheduler</span>):
        <span class="hljs-built_in">super</span>().__init__()

    self.register_modules(unet=unet, scheduler=scheduler)

<span class="hljs-meta">    @torch.no_grad()</span>
    <span class="hljs-keyword">def</span> <span class="hljs-title function_">__call__</span>(<span class="hljs-params">self, batch_size: <span class="hljs-built_in">int</span> = <span class="hljs-number">1</span>, num_inference_steps: <span class="hljs-built_in">int</span> = <span class="hljs-number">50</span></span>):
        <span class="hljs-comment"># Sample gaussian noise to begin loop</span>
        image = torch.randn((batch_size, self.unet.in_channels, self.unet.sample_size, self.unet.sample_size))

        image = image.to(self.device)

        <span class="hljs-comment"># set step values</span>
        self.scheduler.set_timesteps(num_inference_steps)

        <span class="hljs-keyword">for</span> t <span class="hljs-keyword">in</span> self.progress_bar(self.scheduler.timesteps):
            <span class="hljs-comment"># 1. predict noise model_output</span>
            model_output = self.unet(image, t).sample

            <span class="hljs-comment"># 2. predict previous mean of image x_t-1 and add variance depending on eta</span>
            <span class="hljs-comment"># eta corresponds to η in paper and should be between [0, 1]</span>
            <span class="hljs-comment"># do x_t -&gt; x_t-1</span>
            image = self.scheduler.step(model_output, t, image, eta).prev_sample

        image = (image / <span class="hljs-number">2</span> + <span class="hljs-number">0.5</span>).clamp(<span class="hljs-number">0</span>, <span class="hljs-number">1</span>)
        image = image.cpu().permute(<span class="hljs-number">0</span>, <span class="hljs-number">2</span>, <span class="hljs-number">3</span>, <span class="hljs-number">1</span>).numpy()

        <span class="hljs-keyword">return</span> image<!-- HTML_TAG_END --></pre></div>
<p>Now you can upload this short file under the name <code>pipeline.py</code> in your preferred <a href="https://huggingface.co/docs/hub/models-uploading" rel="nofollow">model repository</a>. For Stable Diffusion pipelines, you may also <a href="https://huggingface.co/organizations/sd-diffusers-pipelines-library/share/BUPyDUuHcciGTOKaExlqtfFcyCZsVFdrjr" rel="nofollow">join the community organisation for shared pipelines</a> to upload yours.
Finally, we can load the custom pipeline by passing the model repository name, <em>e.g.</em> <code>sd-diffusers-pipelines-library/my_custom_pipeline</code> alongside the model repository from where we want to load the <code>unet</code> and <code>scheduler</code> components.</p>

	<div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5   text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg>
	<div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div>
	Copied</div></button></div>
	<pre><!-- HTML_TAG_START -->my_pipeline = DiffusionPipeline.from_pretrained(
    <span class="hljs-string">&quot;google/ddpm-cifar10-32&quot;</span>, custom_pipeline=<span class="hljs-string">&quot;patrickvonplaten/my_custom_pipeline&quot;</span>
)<!-- HTML_TAG_END --></pre></div>


		<script type="module" data-hydrate="rw83zj">
		import { start } from "/docs/diffusers/v0.5.1/en/_app/start-hf-doc-builder.js";
		start({
			target: document.querySelector('[data-hydrate="rw83zj"]').parentNode,
			paths: {"base":"/docs/diffusers/v0.5.1/en","assets":"/docs/diffusers/v0.5.1/en"},
			session: {},
			route: false,
			spa: false,
			trailing_slash: "never",
			hydrate: {
				status: 200,
				error: null,
				nodes: [
					import("/docs/diffusers/v0.5.1/en/_app/pages/__layout.svelte-hf-doc-builder.js"),
						import("/docs/diffusers/v0.5.1/en/_app/pages/using-diffusers/custom_pipelines.mdx-hf-doc-builder.js")
				],
				params: {}
			}
		});
	</script>