Buckets:
| <meta charset="utf-8" /><meta http-equiv="content-security-policy" content=""><meta name="hf:doc:metadata" content="{"local":"diffedit","sections":[{"local":"generate-source-and-target-embeddings","title":"Generate source and target embeddings"},{"local":"generate-a-caption-for-inversion","title":"Generate a caption for inversion"}],"title":"DiffEdit"}" data-svelte="svelte-1phssyn"> | |
| <link rel="modulepreload" href="/docs/diffusers/v0.21.0/en/_app/assets/pages/__layout.svelte-hf-doc-builder.css"> | |
| <link rel="modulepreload" href="/docs/diffusers/v0.21.0/en/_app/start-hf-doc-builder.js"> | |
| <link rel="modulepreload" href="/docs/diffusers/v0.21.0/en/_app/chunks/vendor-hf-doc-builder.js"> | |
| <link rel="modulepreload" href="/docs/diffusers/v0.21.0/en/_app/chunks/paths-hf-doc-builder.js"> | |
| <link rel="modulepreload" href="/docs/diffusers/v0.21.0/en/_app/pages/__layout.svelte-hf-doc-builder.js"> | |
| <link rel="modulepreload" href="/docs/diffusers/v0.21.0/en/_app/pages/using-diffusers/diffedit.mdx-hf-doc-builder.js"> | |
| <link rel="modulepreload" href="/docs/diffusers/v0.21.0/en/_app/chunks/Tip-hf-doc-builder.js"> | |
| <link rel="modulepreload" href="/docs/diffusers/v0.21.0/en/_app/chunks/IconCopyLink-hf-doc-builder.js"> | |
| <link rel="modulepreload" href="/docs/diffusers/v0.21.0/en/_app/chunks/CodeBlock-hf-doc-builder.js"> | |
| <link rel="modulepreload" href="/docs/diffusers/v0.21.0/en/_app/chunks/DocNotebookDropdown-hf-doc-builder.js"> | |
| <h1 class="relative group"><a id="diffedit" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#diffedit"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> | |
| <span>DiffEdit | |
| </span></h1> | |
| <div class="flex space-x-1 absolute z-10 right-0 top-0"> | |
| <div class="relative colab-dropdown "> | |
| <button class=" " type="button"> | |
| <img alt="Open In Colab" class="!m-0" src="https://colab.research.google.com/assets/colab-badge.svg"> | |
| </button> | |
| </div> | |
| <div class="relative colab-dropdown "> | |
| <button class=" " type="button"> | |
| <img alt="Open In Studio Lab" class="!m-0" src="https://studiolab.sagemaker.aws/studiolab.svg"> | |
| </button> | |
| </div></div> | |
| <p>Image editing typically requires providing a mask of the area to be edited. DiffEdit automatically generates the mask for you based on a text query, making it easier overall to create a mask without image editing software. The DiffEdit algorithm works in three steps:</p> | |
| <ol><li>the diffusion model denoises an image conditioned on some query text and reference text which produces different noise estimates for different areas of the image; the difference is used to infer a mask to identify which area of the image needs to be changed to match the query text</li> | |
| <li>the input image is encoded into latent space with DDIM</li> | |
| <li>the latents are decoded with the diffusion model conditioned on the text query, using the mask as a guide such that pixels outside the mask remain the same as in the input image</li></ol> | |
| <p>This guide will show you how to use DiffEdit to edit images without manually creating a mask.</p> | |
| <p>Before you begin, make sure you have the following libraries installed:</p> | |
| <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> | |
| <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> | |
| Copied</div></button></div> | |
| <pre><!-- HTML_TAG_START --><span class="hljs-comment"># uncomment to install the necessary libraries in Colab</span> | |
| <span class="hljs-comment">#!pip install diffusers transformers accelerate safetensors</span><!-- HTML_TAG_END --></pre></div> | |
| <p>The <a href="/docs/diffusers/v0.21.0/en/api/pipelines/diffedit#diffusers.StableDiffusionDiffEditPipeline">StableDiffusionDiffEditPipeline</a> requires an image mask and a set of partially inverted latents. The image mask is generated from the <a href="/docs/diffusers/v0.21.0/en/api/pipelines/diffedit#diffusers.StableDiffusionDiffEditPipeline.generate_mask">generate_mask()</a> function, and includes two parameters, <code>source_prompt</code> and <code>target_prompt</code>. These parameters determine what to edit in the image. For example, if you want to change a bowl of <em>fruits</em> to a bowl of <em>pears</em>, then:</p> | |
| <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> | |
| <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> | |
| Copied</div></button></div> | |
| <pre><!-- HTML_TAG_START -->source_prompt = <span class="hljs-string">"a bowl of fruits"</span> | |
| target_prompt = <span class="hljs-string">"a bowl of pears"</span><!-- HTML_TAG_END --></pre></div> | |
| <p>The partially inverted latents are generated from the <a href="/docs/diffusers/v0.21.0/en/api/pipelines/diffedit#diffusers.StableDiffusionDiffEditPipeline.invert">invert()</a> function, and it is generally a good idea to include a <code>prompt</code> or <em>caption</em> describing the image to help guide the inverse latent sampling process. The caption can often be your <code>source_prompt</code>, but feel free to experiment with other text descriptions!</p> | |
| <p>Let’s load the pipeline, scheduler, inverse scheduler, and enable some optimizations to reduce memory usage:</p> | |
| <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> | |
| <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> | |
| Copied</div></button></div> | |
| <pre><!-- HTML_TAG_START --><span class="hljs-keyword">import</span> torch | |
| <span class="hljs-keyword">from</span> diffusers <span class="hljs-keyword">import</span> DDIMScheduler, DDIMInverseScheduler, StableDiffusionDiffEditPipeline | |
| pipeline = StableDiffusionDiffEditPipeline.from_pretrained( | |
| <span class="hljs-string">"stabilityai/stable-diffusion-2-1"</span>, | |
| torch_dtype=torch.float16, | |
| safety_checker=<span class="hljs-literal">None</span>, | |
| use_safetensors=<span class="hljs-literal">True</span>, | |
| ) | |
| pipeline.scheduler = DDIMScheduler.from_config(pipeline.scheduler.config) | |
| pipeline.inverse_scheduler = DDIMInverseScheduler.from_config(pipeline.scheduler.config) | |
| pipeline.enable_model_cpu_offload() | |
| pipeline.enable_vae_slicing()<!-- HTML_TAG_END --></pre></div> | |
| <p>Load the image to edit:</p> | |
| <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> | |
| <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> | |
| Copied</div></button></div> | |
| <pre><!-- HTML_TAG_START --><span class="hljs-keyword">from</span> diffusers.utils <span class="hljs-keyword">import</span> load_image | |
| img_url = <span class="hljs-string">"https://github.com/Xiang-cd/DiffEdit-stable-diffusion/raw/main/assets/origin.png"</span> | |
| raw_image = load_image(img_url).convert(<span class="hljs-string">"RGB"</span>).resize((<span class="hljs-number">768</span>, <span class="hljs-number">768</span>))<!-- HTML_TAG_END --></pre></div> | |
| <p>Use the <a href="/docs/diffusers/v0.21.0/en/api/pipelines/diffedit#diffusers.StableDiffusionDiffEditPipeline.generate_mask">generate_mask()</a> function to generate the image mask. You’ll need to pass it the <code>source_prompt</code> and <code>target_prompt</code> to specify what to edit in the image:</p> | |
| <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> | |
| <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> | |
| Copied</div></button></div> | |
| <pre><!-- HTML_TAG_START -->source_prompt = <span class="hljs-string">"a bowl of fruits"</span> | |
| target_prompt = <span class="hljs-string">"a basket of pears"</span> | |
| mask_image = pipeline.generate_mask( | |
| image=raw_image, | |
| source_prompt=source_prompt, | |
| target_prompt=target_prompt, | |
| )<!-- HTML_TAG_END --></pre></div> | |
| <p>Next, create the inverted latents and pass it a caption describing the image:</p> | |
| <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> | |
| <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> | |
| Copied</div></button></div> | |
| <pre><!-- HTML_TAG_START -->inv_latents = pipeline.invert(prompt=source_prompt, image=raw_image).latents<!-- HTML_TAG_END --></pre></div> | |
| <p>Finally, pass the image mask and inverted latents to the pipeline. The <code>target_prompt</code> becomes the <code>prompt</code> now, and the <code>source_prompt</code> is used as the <code>negative_prompt</code>:</p> | |
| <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> | |
| <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> | |
| Copied</div></button></div> | |
| <pre><!-- HTML_TAG_START -->image = pipeline( | |
| prompt=target_prompt, | |
| mask_image=mask_image, | |
| image_latents=inv_latents, | |
| negative_prompt=source_prompt, | |
| ).images[<span class="hljs-number">0</span>] | |
| image.save(<span class="hljs-string">"edited_image.png"</span>)<!-- HTML_TAG_END --></pre></div> | |
| <div class="flex gap-4"><div><img class="rounded-xl" src="https://github.com/Xiang-cd/DiffEdit-stable-diffusion/raw/main/assets/origin.png"> | |
| <figcaption class="mt-2 text-center text-sm text-gray-500">original image</figcaption></div> | |
| <div><img class="rounded-xl" src="https://github.com/Xiang-cd/DiffEdit-stable-diffusion/blob/main/assets/target.png?raw=true"> | |
| <figcaption class="mt-2 text-center text-sm text-gray-500">edited image</figcaption></div></div> | |
| <h2 class="relative group"><a id="generate-source-and-target-embeddings" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#generate-source-and-target-embeddings"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> | |
| <span>Generate source and target embeddings | |
| </span></h2> | |
| <p>The source and target embeddings can be automatically generated with the <a href="https://huggingface.co/docs/transformers/model_doc/flan-t5" rel="nofollow">Flan-T5</a> model instead of creating them manually.</p> | |
| <p>Load the Flan-T5 model and tokenizer from the 🤗 Transformers library:</p> | |
| <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> | |
| <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> | |
| Copied</div></button></div> | |
| <pre><!-- HTML_TAG_START --><span class="hljs-keyword">import</span> torch | |
| <span class="hljs-keyword">from</span> transformers <span class="hljs-keyword">import</span> AutoTokenizer, T5ForConditionalGeneration | |
| tokenizer = AutoTokenizer.from_pretrained(<span class="hljs-string">"google/flan-t5-xl"</span>) | |
| model = T5ForConditionalGeneration.from_pretrained(<span class="hljs-string">"google/flan-t5-xl"</span>, device_map=<span class="hljs-string">"auto"</span>, torch_dtype=torch.float16)<!-- HTML_TAG_END --></pre></div> | |
| <p>Provide some initial text to prompt the model to generate the source and target prompts.</p> | |
| <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> | |
| <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> | |
| Copied</div></button></div> | |
| <pre><!-- HTML_TAG_START -->source_concept = <span class="hljs-string">"bowl"</span> | |
| target_concept = <span class="hljs-string">"basket"</span> | |
| source_text = <span class="hljs-string">f"Provide a caption for images containing a <span class="hljs-subst">{source_concept}</span>. "</span> | |
| <span class="hljs-string">"The captions should be in English and should be no longer than 150 characters."</span> | |
| target_text = <span class="hljs-string">f"Provide a caption for images containing a <span class="hljs-subst">{target_concept}</span>. "</span> | |
| <span class="hljs-string">"The captions should be in English and should be no longer than 150 characters."</span><!-- HTML_TAG_END --></pre></div> | |
| <p>Next, create a utility function to generate the prompts:</p> | |
| <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> | |
| <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> | |
| Copied</div></button></div> | |
| <pre><!-- HTML_TAG_START --><span class="hljs-meta">@torch.no_grad</span> | |
| <span class="hljs-keyword">def</span> <span class="hljs-title function_">generate_prompts</span>(<span class="hljs-params">input_prompt</span>): | |
| input_ids = tokenizer(input_prompt, return_tensors=<span class="hljs-string">"pt"</span>).input_ids.to(<span class="hljs-string">"cuda"</span>) | |
| outputs = model.generate( | |
| input_ids, temperature=<span class="hljs-number">0.8</span>, num_return_sequences=<span class="hljs-number">16</span>, do_sample=<span class="hljs-literal">True</span>, max_new_tokens=<span class="hljs-number">128</span>, top_k=<span class="hljs-number">10</span> | |
| ) | |
| <span class="hljs-keyword">return</span> tokenizer.batch_decode(outputs, skip_special_tokens=<span class="hljs-literal">True</span>) | |
| source_prompts = generate_prompts(source_text) | |
| target_prompts = generate_prompts(target_text) | |
| <span class="hljs-built_in">print</span>(source_prompts) | |
| <span class="hljs-built_in">print</span>(target_prompts)<!-- HTML_TAG_END --></pre></div> | |
| <div class="course-tip bg-gradient-to-br dark:bg-gradient-to-r before:border-green-500 dark:before:border-green-800 from-green-50 dark:from-gray-900 to-white dark:to-gray-950 border border-green-50 text-green-700 dark:text-gray-400"><p>Check out the <a href="https://huggingface.co/docs/transformers/main/en/generation_strategies" rel="nofollow">generation strategy</a> guide if you’re interested in learning more about strategies for generating different quality text.</p></div> | |
| <p>Load the text encoder model used by the <a href="/docs/diffusers/v0.21.0/en/api/pipelines/diffedit#diffusers.StableDiffusionDiffEditPipeline">StableDiffusionDiffEditPipeline</a> to encode the text. You’ll use the text encoder to compute the text embeddings:</p> | |
| <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> | |
| <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> | |
| Copied</div></button></div> | |
| <pre><!-- HTML_TAG_START --><span class="hljs-keyword">import</span> torch | |
| <span class="hljs-keyword">from</span> diffusers <span class="hljs-keyword">import</span> StableDiffusionDiffEditPipeline | |
| pipeline = StableDiffusionDiffEditPipeline.from_pretrained( | |
| <span class="hljs-string">"stabilityai/stable-diffusion-2-1"</span>, torch_dtype=torch.float16, use_safetensors=<span class="hljs-literal">True</span> | |
| ).to(<span class="hljs-string">"cuda"</span>) | |
| pipeline.enable_model_cpu_offload() | |
| pipeline.enable_vae_slicing() | |
| <span class="hljs-meta">@torch.no_grad()</span> | |
| <span class="hljs-keyword">def</span> <span class="hljs-title function_">embed_prompts</span>(<span class="hljs-params">sentences, tokenizer, text_encoder, device=<span class="hljs-string">"cuda"</span></span>): | |
| embeddings = [] | |
| <span class="hljs-keyword">for</span> sent <span class="hljs-keyword">in</span> sentences: | |
| text_inputs = tokenizer( | |
| sent, | |
| padding=<span class="hljs-string">"max_length"</span>, | |
| max_length=tokenizer.model_max_length, | |
| truncation=<span class="hljs-literal">True</span>, | |
| return_tensors=<span class="hljs-string">"pt"</span>, | |
| ) | |
| text_input_ids = text_inputs.input_ids | |
| prompt_embeds = text_encoder(text_input_ids.to(device), attention_mask=<span class="hljs-literal">None</span>)[<span class="hljs-number">0</span>] | |
| embeddings.append(prompt_embeds) | |
| <span class="hljs-keyword">return</span> torch.concatenate(embeddings, dim=<span class="hljs-number">0</span>).mean(dim=<span class="hljs-number">0</span>).unsqueeze(<span class="hljs-number">0</span>) | |
| source_embeds = embed_prompts(source_prompts, pipeline.tokenizer, pipeline.text_encoder) | |
| target_embeds = embed_prompts(target_prompts, pipeline.tokenizer, pipeline.text_encoder)<!-- HTML_TAG_END --></pre></div> | |
| <p>Finally, pass the embeddings to the <a href="/docs/diffusers/v0.21.0/en/api/pipelines/diffedit#diffusers.StableDiffusionDiffEditPipeline.generate_mask">generate_mask()</a> and <a href="/docs/diffusers/v0.21.0/en/api/pipelines/diffedit#diffusers.StableDiffusionDiffEditPipeline.invert">invert()</a> functions, and pipeline to generate the image:</p> | |
| <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> | |
| <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> | |
| Copied</div></button></div> | |
| <pre><!-- HTML_TAG_START --> from diffusers import DDIMInverseScheduler, DDIMScheduler | |
| from diffusers.utils import load_image | |
| pipeline.scheduler = DDIMScheduler.from_config(pipeline.scheduler.config) | |
| pipeline.inverse_scheduler = DDIMInverseScheduler.from_config(pipeline.scheduler.config) | |
| img_url = "https://github.com/Xiang-cd/DiffEdit-stable-diffusion/raw/main/assets/origin.png" | |
| raw_image = load_image(img_url).convert("RGB").resize((768, 768)) | |
| mask_image = pipeline.generate_mask( | |
| image=raw_image, | |
| <span class="hljs-addition">+ source_prompt_embeds=source_embeds,</span> | |
| <span class="hljs-addition">+ target_prompt_embeds=target_embeds,</span> | |
| ) | |
| inv_latents = pipeline.invert( | |
| <span class="hljs-addition">+ prompt_embeds=source_embeds,</span> | |
| image=raw_image, | |
| ).latents | |
| images = pipeline( | |
| mask_image=mask_image, | |
| image_latents=inv_latents, | |
| <span class="hljs-addition">+ prompt_embeds=target_embeds,</span> | |
| <span class="hljs-addition">+ negative_prompt_embeds=source_embeds,</span> | |
| ).images | |
| images[0].save("edited_image.png")<!-- HTML_TAG_END --></pre></div> | |
| <h2 class="relative group"><a id="generate-a-caption-for-inversion" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#generate-a-caption-for-inversion"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> | |
| <span>Generate a caption for inversion | |
| </span></h2> | |
| <p>While you can use the <code>source_prompt</code> as a caption to help generate the partially inverted latents, you can also use the <a href="https://huggingface.co/docs/transformers/model_doc/blip" rel="nofollow">BLIP</a> model to automatically generate a caption.</p> | |
| <p>Load the BLIP model and processor from the 🤗 Transformers library:</p> | |
| <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> | |
| <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> | |
| Copied</div></button></div> | |
| <pre><!-- HTML_TAG_START --><span class="hljs-keyword">import</span> torch | |
| <span class="hljs-keyword">from</span> transformers <span class="hljs-keyword">import</span> BlipForConditionalGeneration, BlipProcessor | |
| processor = BlipProcessor.from_pretrained(<span class="hljs-string">"Salesforce/blip-image-captioning-base"</span>) | |
| model = BlipForConditionalGeneration.from_pretrained(<span class="hljs-string">"Salesforce/blip-image-captioning-base"</span>, torch_dtype=torch.float16, low_cpu_mem_usage=<span class="hljs-literal">True</span>)<!-- HTML_TAG_END --></pre></div> | |
| <p>Create a utility function to generate a caption from the input image:</p> | |
| <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> | |
| <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> | |
| Copied</div></button></div> | |
| <pre><!-- HTML_TAG_START --><span class="hljs-meta">@torch.no_grad()</span> | |
| <span class="hljs-keyword">def</span> <span class="hljs-title function_">generate_caption</span>(<span class="hljs-params">images, caption_generator, caption_processor</span>): | |
| text = <span class="hljs-string">"a photograph of"</span> | |
| inputs = caption_processor(images, text, return_tensors=<span class="hljs-string">"pt"</span>).to(device=<span class="hljs-string">"cuda"</span>, dtype=caption_generator.dtype) | |
| caption_generator.to(<span class="hljs-string">"cuda"</span>) | |
| outputs = caption_generator.generate(**inputs, max_new_tokens=<span class="hljs-number">128</span>) | |
| <span class="hljs-comment"># offload caption generator</span> | |
| caption_generator.to(<span class="hljs-string">"cpu"</span>) | |
| caption = caption_processor.batch_decode(outputs, skip_special_tokens=<span class="hljs-literal">True</span>)[<span class="hljs-number">0</span>] | |
| <span class="hljs-keyword">return</span> caption<!-- HTML_TAG_END --></pre></div> | |
| <p>Load an input image and generate a caption for it using the <code>generate_caption</code> function:</p> | |
| <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> | |
| <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> | |
| Copied</div></button></div> | |
| <pre><!-- HTML_TAG_START --><span class="hljs-keyword">from</span> diffusers.utils <span class="hljs-keyword">import</span> load_image | |
| img_url = <span class="hljs-string">"https://github.com/Xiang-cd/DiffEdit-stable-diffusion/raw/main/assets/origin.png"</span> | |
| raw_image = load_image(img_url).convert(<span class="hljs-string">"RGB"</span>).resize((<span class="hljs-number">768</span>, <span class="hljs-number">768</span>)) | |
| caption = generate_caption(raw_image, model, processor)<!-- HTML_TAG_END --></pre></div> | |
| <div class="flex justify-center"><figure><img class="rounded-xl" src="https://github.com/Xiang-cd/DiffEdit-stable-diffusion/raw/main/assets/origin.png"> | |
| <figcaption class="text-center">generated caption: "a photograph of a bowl of fruit on a table"</figcaption></figure></div> | |
| <p>Now you can drop the caption into the <a href="/docs/diffusers/v0.21.0/en/api/pipelines/diffedit#diffusers.StableDiffusionDiffEditPipeline.invert">invert()</a> function to generate the partially inverted latents!</p> | |
| <script type="module" data-hydrate="1v04361"> | |
| import { start } from "/docs/diffusers/v0.21.0/en/_app/start-hf-doc-builder.js"; | |
| start({ | |
| target: document.querySelector('[data-hydrate="1v04361"]').parentNode, | |
| paths: {"base":"/docs/diffusers/v0.21.0/en","assets":"/docs/diffusers/v0.21.0/en"}, | |
| session: {}, | |
| route: false, | |
| spa: false, | |
| trailing_slash: "never", | |
| hydrate: { | |
| status: 200, | |
| error: null, | |
| nodes: [ | |
| import("/docs/diffusers/v0.21.0/en/_app/pages/__layout.svelte-hf-doc-builder.js"), | |
| import("/docs/diffusers/v0.21.0/en/_app/pages/using-diffusers/diffedit.mdx-hf-doc-builder.js") | |
| ], | |
| params: {} | |
| } | |
| }); | |
| </script> | |
Xet Storage Details
- Size:
- 39.8 kB
- Xet hash:
- 8c96776ae53e9897c0afdc46721094462bfd8842c3a272a77e6edd5a301f866a
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.