diffusers/v0.2.4/en/models.html

<meta charset="utf-8" /><meta http-equiv="content-security-policy" content=""><meta name="hf:doc:metadata" content="{&quot;local&quot;:&quot;models&quot;,&quot;sections&quot;:[{&quot;local&quot;:&quot;api&quot;,&quot;title&quot;:&quot;API&quot;},{&quot;local&quot;:&quot;examples&quot;,&quot;title&quot;:&quot;Examples&quot;}],&quot;title&quot;:&quot;Models&quot;}" data-svelte="svelte-1phssyn">
	<link rel="modulepreload" href="/docs/diffusers/v0.2.4/en/_app/assets/pages/__layout.svelte-hf-doc-builder.css">
	<link rel="modulepreload" href="/docs/diffusers/v0.2.4/en/_app/start-hf-doc-builder.js">
	<link rel="modulepreload" href="/docs/diffusers/v0.2.4/en/_app/chunks/vendor-hf-doc-builder.js">
	<link rel="modulepreload" href="/docs/diffusers/v0.2.4/en/_app/chunks/paths-hf-doc-builder.js">
	<link rel="modulepreload" href="/docs/diffusers/v0.2.4/en/_app/pages/__layout.svelte-hf-doc-builder.js">
	<link rel="modulepreload" href="/docs/diffusers/v0.2.4/en/_app/pages/models.mdx-hf-doc-builder.js">
	<link rel="modulepreload" href="/docs/diffusers/v0.2.4/en/_app/chunks/IconCopyLink-hf-doc-builder.js"> 






<h1 class="relative group"><a id="models" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#models"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a>
	<span>Models
	</span></h1>

<p>Diffusers contains pretrained models for popular algorithms and modules for creating the next set of diffusion models.
The primary function of these models is to denoise an input sample, by modeling the distribution $p<em>\theta(\mathbf{x}</em>{t-1}|\mathbf{x}_t)$.
The models are built on the base class [‘ModelMixin’] that is a <code>torch.nn.module</code> with basic functionality for saving and loading models both locally and from the HuggingFace hub.</p>
<h2 class="relative group"><a id="api" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#api"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a>
	<span>API
	</span></h2>

<p>Models should provide the <code>def forward</code> function and initialization of the model.
All saving, loading, and utilities should be in the base [‘ModelMixin’] class.</p>
<h2 class="relative group"><a id="examples" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#examples"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a>
	<span>Examples
	</span></h2>

<ul><li>The [‘UNetModel’] was proposed in <a href="https://arxiv.org/" rel="nofollow">TODO</a> and has been used in paper1, paper2, paper3.</li>
<li>Extensions of the [‘UNetModel’] include the [‘UNetGlideModel’] that uses attention and timestep embeddings for the <a href="https://arxiv.org/abs/2112.10741" rel="nofollow">GLIDE</a> paper, the [‘UNetGradTTS’] model from this <a href="https://arxiv.org/abs/2105.06337" rel="nofollow">paper</a> for text-to-speech, [‘UNetLDMModel’] for latent-diffusion models in this <a href="https://arxiv.org/abs/2112.10752" rel="nofollow">paper</a>, and the [‘TemporalUNet’] used for time-series prediciton in this reinforcement learning <a href="https://arxiv.org/abs/2205.09991" rel="nofollow">paper</a>.</li>
<li>TODO: mention VAE / SDE score estimation</li></ul>


		<script type="module" data-hydrate="1t2owwl">
		import { start } from "/docs/diffusers/v0.2.4/en/_app/start-hf-doc-builder.js";
		start({
			target: document.querySelector('[data-hydrate="1t2owwl"]').parentNode,
			paths: {"base":"/docs/diffusers/v0.2.4/en","assets":"/docs/diffusers/v0.2.4/en"},
			session: {},
			route: false,
			spa: false,
			trailing_slash: "never",
			hydrate: {
				status: 200,
				error: null,
				nodes: [
					import("/docs/diffusers/v0.2.4/en/_app/pages/__layout.svelte-hf-doc-builder.js"),
						import("/docs/diffusers/v0.2.4/en/_app/pages/models.mdx-hf-doc-builder.js")
				],
				params: {}
			}
		});
	</script>