Buckets:
| <meta charset="utf-8" /><meta name="hf:doc:metadata" content="{"title":"Semantic segmentation using LoRA","local":"semantic-segmentation-using-lora","sections":[{"title":"Install dependencies","local":"install-dependencies","sections":[],"depth":2},{"title":"Authenticate to share your model","local":"authenticate-to-share-your-model","sections":[],"depth":2},{"title":"Load a dataset","local":"load-a-dataset","sections":[],"depth":2},{"title":"Prepare label maps","local":"prepare-label-maps","sections":[],"depth":2},{"title":"Prepare datasets for training and evaluation","local":"prepare-datasets-for-training-and-evaluation","sections":[],"depth":2},{"title":"Create evaluation function","local":"create-evaluation-function","sections":[],"depth":2},{"title":"Load a base model","local":"load-a-base-model","sections":[],"depth":2},{"title":"Wrap the base model as a PeftModel for LoRA training","local":"wrap-the-base-model-as-a-peftmodel-for-lora-training","sections":[],"depth":2},{"title":"Train the model","local":"train-the-model","sections":[],"depth":2},{"title":"Save the model and run inference","local":"save-the-model-and-run-inference","sections":[],"depth":2}],"depth":1}"> | |
| <link href="/docs/peft/main/en/_app/immutable/assets/0.e3b0c442.css" rel="modulepreload"> | |
| <link rel="modulepreload" href="/docs/peft/main/en/_app/immutable/entry/start.f1bde2f6.js"> | |
| <link rel="modulepreload" href="/docs/peft/main/en/_app/immutable/chunks/scheduler.d627b047.js"> | |
| <link rel="modulepreload" href="/docs/peft/main/en/_app/immutable/chunks/singletons.eea5c746.js"> | |
| <link rel="modulepreload" href="/docs/peft/main/en/_app/immutable/chunks/index.a57a1c33.js"> | |
| <link rel="modulepreload" href="/docs/peft/main/en/_app/immutable/chunks/paths.e43091d4.js"> | |
| <link rel="modulepreload" href="/docs/peft/main/en/_app/immutable/entry/app.182b7f64.js"> | |
| <link rel="modulepreload" href="/docs/peft/main/en/_app/immutable/chunks/index.d48c4817.js"> | |
| <link rel="modulepreload" href="/docs/peft/main/en/_app/immutable/nodes/0.8a16be44.js"> | |
| <link rel="modulepreload" href="/docs/peft/main/en/_app/immutable/chunks/each.e59479a4.js"> | |
| <link rel="modulepreload" href="/docs/peft/main/en/_app/immutable/nodes/39.20ee778b.js"> | |
| <link rel="modulepreload" href="/docs/peft/main/en/_app/immutable/chunks/CodeBlock.5da89496.js"> | |
| <link rel="modulepreload" href="/docs/peft/main/en/_app/immutable/chunks/Heading.47e562a9.js"><!-- HEAD_svelte-u9bgzb_START --><meta name="hf:doc:metadata" content="{"title":"Semantic segmentation using LoRA","local":"semantic-segmentation-using-lora","sections":[{"title":"Install dependencies","local":"install-dependencies","sections":[],"depth":2},{"title":"Authenticate to share your model","local":"authenticate-to-share-your-model","sections":[],"depth":2},{"title":"Load a dataset","local":"load-a-dataset","sections":[],"depth":2},{"title":"Prepare label maps","local":"prepare-label-maps","sections":[],"depth":2},{"title":"Prepare datasets for training and evaluation","local":"prepare-datasets-for-training-and-evaluation","sections":[],"depth":2},{"title":"Create evaluation function","local":"create-evaluation-function","sections":[],"depth":2},{"title":"Load a base model","local":"load-a-base-model","sections":[],"depth":2},{"title":"Wrap the base model as a PeftModel for LoRA training","local":"wrap-the-base-model-as-a-peftmodel-for-lora-training","sections":[],"depth":2},{"title":"Train the model","local":"train-the-model","sections":[],"depth":2},{"title":"Save the model and run inference","local":"save-the-model-and-run-inference","sections":[],"depth":2}],"depth":1}"><!-- HEAD_svelte-u9bgzb_END --> <p></p> <h1 class="relative group"><a id="semantic-segmentation-using-lora" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#semantic-segmentation-using-lora"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Semantic segmentation using LoRA</span></h1> <p data-svelte-h="svelte-ptgpdg">This guide demonstrates how to use LoRA, a low-rank approximation technique, to finetune a SegFormer model variant for semantic segmentation. | |
| By using LoRA from 🤗 PEFT, we can reduce the number of trainable parameters in the SegFormer model to only 14% of the original trainable parameters.</p> <p data-svelte-h="svelte-1q529js">LoRA achieves this reduction by adding low-rank “update matrices” to specific blocks of the model, such as the attention | |
| blocks. During fine-tuning, only these matrices are trained, while the original model parameters are left unchanged. | |
| At inference time, the update matrices are merged with the original model parameters to produce the final classification result.</p> <p data-svelte-h="svelte-e5fkf5">For more information on LoRA, please refer to the <a href="https://arxiv.org/abs/2106.09685" rel="nofollow">original LoRA paper</a>.</p> <h2 class="relative group"><a id="install-dependencies" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#install-dependencies"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Install dependencies</span></h2> <p data-svelte-h="svelte-1hz22x2">Install the libraries required for model training:</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->!pip install transformers accelerate evaluate datasets peft -q<!-- HTML_TAG_END --></pre></div> <h2 class="relative group"><a id="authenticate-to-share-your-model" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#authenticate-to-share-your-model"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Authenticate to share your model</span></h2> <p data-svelte-h="svelte-1t4proi">To share the finetuned model with the community at the end of the training, authenticate using your 🤗 token. | |
| You can obtain your token from your <a href="https://huggingface.co/settings/token" rel="nofollow">account settings</a>.</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-keyword">from</span> huggingface_hub <span class="hljs-keyword">import</span> notebook_login | |
| notebook_login()<!-- HTML_TAG_END --></pre></div> <h2 class="relative group"><a id="load-a-dataset" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#load-a-dataset"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Load a dataset</span></h2> <p data-svelte-h="svelte-1w5yso9">To ensure that this example runs within a reasonable time frame, here we are limiting the number of instances from the training | |
| set of the <a href="https://huggingface.co/datasets/scene_parse_150" rel="nofollow">SceneParse150 dataset</a> to 150.</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-keyword">from</span> datasets <span class="hljs-keyword">import</span> load_dataset | |
| ds = load_dataset(<span class="hljs-string">"scene_parse_150"</span>, split=<span class="hljs-string">"train[:150]"</span>)<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-1d7ffhy">Next, split the dataset into train and test sets.</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->ds = ds.train_test_split(test_size=<span class="hljs-number">0.1</span>) | |
| train_ds = ds[<span class="hljs-string">"train"</span>] | |
| test_ds = ds[<span class="hljs-string">"test"</span>]<!-- HTML_TAG_END --></pre></div> <h2 class="relative group"><a id="prepare-label-maps" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#prepare-label-maps"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Prepare label maps</span></h2> <p data-svelte-h="svelte-getpmi">Create a dictionary that maps a label id to a label class, which will be useful when setting up the model later:</p> <ul data-svelte-h="svelte-phg2cz"><li><code>label2id</code>: maps the semantic classes of the dataset to integer ids.</li> <li><code>id2label</code>: maps integer ids back to the semantic classes.</li></ul> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-keyword">import</span> json | |
| <span class="hljs-keyword">from</span> huggingface_hub <span class="hljs-keyword">import</span> cached_download, hf_hub_url | |
| repo_id = <span class="hljs-string">"huggingface/label-files"</span> | |
| filename = <span class="hljs-string">"ade20k-id2label.json"</span> | |
| id2label = json.load(<span class="hljs-built_in">open</span>(cached_download(hf_hub_url(repo_id, filename, repo_type=<span class="hljs-string">"dataset"</span>)), <span class="hljs-string">"r"</span>)) | |
| id2label = {<span class="hljs-built_in">int</span>(k): v <span class="hljs-keyword">for</span> k, v <span class="hljs-keyword">in</span> id2label.items()} | |
| label2id = {v: k <span class="hljs-keyword">for</span> k, v <span class="hljs-keyword">in</span> id2label.items()} | |
| num_labels = <span class="hljs-built_in">len</span>(id2label)<!-- HTML_TAG_END --></pre></div> <h2 class="relative group"><a id="prepare-datasets-for-training-and-evaluation" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#prepare-datasets-for-training-and-evaluation"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Prepare datasets for training and evaluation</span></h2> <p data-svelte-h="svelte-x8866f">Next, load the SegFormer image processor to prepare the images and annotations for the model. This dataset uses the | |
| zero-index as the background class, so make sure to set <code>do_reduce_labels=True</code> to subtract one from all labels since the | |
| background class is not among the 150 classes.</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-keyword">from</span> transformers <span class="hljs-keyword">import</span> AutoImageProcessor | |
| checkpoint = <span class="hljs-string">"nvidia/mit-b0"</span> | |
| image_processor = AutoImageProcessor.from_pretrained(checkpoint, do_reduce_labels=<span class="hljs-literal">True</span>)<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-fe2rtc">Add a function to apply data augmentation to the images, so that the model is more robust against overfitting. Here we use the | |
| <a href="https://pytorch.org/vision/stable/generated/torchvision.transforms.ColorJitter.html" rel="nofollow">ColorJitter</a> function from | |
| <a href="https://pytorch.org/vision/stable/index.html" rel="nofollow">torchvision</a> to randomly change the color properties of an image.</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-keyword">from</span> torchvision.transforms <span class="hljs-keyword">import</span> ColorJitter | |
| jitter = ColorJitter(brightness=<span class="hljs-number">0.25</span>, contrast=<span class="hljs-number">0.25</span>, saturation=<span class="hljs-number">0.25</span>, hue=<span class="hljs-number">0.1</span>)<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-vm1b9v">Add a function to handle grayscale images and ensure that each input image has three color channels, regardless of | |
| whether it was originally grayscale or RGB. The function converts RGB images to array as is, and for grayscale images | |
| that have only one color channel, the function replicates the same channel three times using <code>np.tile()</code> before converting | |
| the image into an array.</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-keyword">import</span> numpy <span class="hljs-keyword">as</span> np | |
| <span class="hljs-keyword">def</span> <span class="hljs-title function_">handle_grayscale_image</span>(<span class="hljs-params">image</span>): | |
| np_image = np.array(image) | |
| <span class="hljs-keyword">if</span> np_image.ndim == <span class="hljs-number">2</span>: | |
| tiled_image = np.tile(np.expand_dims(np_image, -<span class="hljs-number">1</span>), <span class="hljs-number">3</span>) | |
| <span class="hljs-keyword">return</span> Image.fromarray(tiled_image) | |
| <span class="hljs-keyword">else</span>: | |
| <span class="hljs-keyword">return</span> Image.fromarray(np_image)<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-yxifz1">Finally, combine everything in two functions that you’ll use to transform training and validation data. The two functions | |
| are similar except data augmentation is applied only to the training data.</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-keyword">from</span> PIL <span class="hljs-keyword">import</span> Image | |
| <span class="hljs-keyword">def</span> <span class="hljs-title function_">train_transforms</span>(<span class="hljs-params">example_batch</span>): | |
| images = [jitter(handle_grayscale_image(x)) <span class="hljs-keyword">for</span> x <span class="hljs-keyword">in</span> example_batch[<span class="hljs-string">"image"</span>]] | |
| labels = [x <span class="hljs-keyword">for</span> x <span class="hljs-keyword">in</span> example_batch[<span class="hljs-string">"annotation"</span>]] | |
| inputs = image_processor(images, labels) | |
| <span class="hljs-keyword">return</span> inputs | |
| <span class="hljs-keyword">def</span> <span class="hljs-title function_">val_transforms</span>(<span class="hljs-params">example_batch</span>): | |
| images = [handle_grayscale_image(x) <span class="hljs-keyword">for</span> x <span class="hljs-keyword">in</span> example_batch[<span class="hljs-string">"image"</span>]] | |
| labels = [x <span class="hljs-keyword">for</span> x <span class="hljs-keyword">in</span> example_batch[<span class="hljs-string">"annotation"</span>]] | |
| inputs = image_processor(images, labels) | |
| <span class="hljs-keyword">return</span> inputs<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-sagovx">To apply the preprocessing functions over the entire dataset, use the 🤗 Datasets <code>set_transform</code> function:</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->train_ds.set_transform(train_transforms) | |
| test_ds.set_transform(val_transforms)<!-- HTML_TAG_END --></pre></div> <h2 class="relative group"><a id="create-evaluation-function" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#create-evaluation-function"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Create evaluation function</span></h2> <p data-svelte-h="svelte-1x2616c">Including a metric during training is helpful for evaluating your model’s performance. You can load an evaluation | |
| method with the <a href="https://huggingface.co/docs/evaluate/index" rel="nofollow">🤗 Evaluate</a> library. For this task, use | |
| the <a href="https://huggingface.co/spaces/evaluate-metric/accuracy" rel="nofollow">mean Intersection over Union (IoU)</a> metric (see the 🤗 Evaluate | |
| <a href="https://huggingface.co/docs/evaluate/a_quick_tour" rel="nofollow">quick tour</a> to learn more about how to load and compute a metric):</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-keyword">import</span> torch | |
| <span class="hljs-keyword">from</span> torch <span class="hljs-keyword">import</span> nn | |
| <span class="hljs-keyword">import</span> evaluate | |
| metric = evaluate.load(<span class="hljs-string">"mean_iou"</span>) | |
| <span class="hljs-keyword">def</span> <span class="hljs-title function_">compute_metrics</span>(<span class="hljs-params">eval_pred</span>): | |
| <span class="hljs-keyword">with</span> torch.no_grad(): | |
| logits, labels = eval_pred | |
| logits_tensor = torch.from_numpy(logits) | |
| logits_tensor = nn.functional.interpolate( | |
| logits_tensor, | |
| size=labels.shape[-<span class="hljs-number">2</span>:], | |
| mode=<span class="hljs-string">"bilinear"</span>, | |
| align_corners=<span class="hljs-literal">False</span>, | |
| ).argmax(dim=<span class="hljs-number">1</span>) | |
| pred_labels = logits_tensor.detach().cpu().numpy() | |
| <span class="hljs-comment"># currently using _compute instead of compute</span> | |
| <span class="hljs-comment"># see this issue for more info: https://github.com/huggingface/evaluate/pull/328#issuecomment-1286866576</span> | |
| metrics = metric._compute( | |
| predictions=pred_labels, | |
| references=labels, | |
| num_labels=<span class="hljs-built_in">len</span>(id2label), | |
| ignore_index=<span class="hljs-number">0</span>, | |
| reduce_labels=image_processor.do_reduce_labels, | |
| ) | |
| per_category_accuracy = metrics.pop(<span class="hljs-string">"per_category_accuracy"</span>).tolist() | |
| per_category_iou = metrics.pop(<span class="hljs-string">"per_category_iou"</span>).tolist() | |
| metrics.update({<span class="hljs-string">f"accuracy_<span class="hljs-subst">{id2label[i]}</span>"</span>: v <span class="hljs-keyword">for</span> i, v <span class="hljs-keyword">in</span> <span class="hljs-built_in">enumerate</span>(per_category_accuracy)}) | |
| metrics.update({<span class="hljs-string">f"iou_<span class="hljs-subst">{id2label[i]}</span>"</span>: v <span class="hljs-keyword">for</span> i, v <span class="hljs-keyword">in</span> <span class="hljs-built_in">enumerate</span>(per_category_iou)}) | |
| <span class="hljs-keyword">return</span> metrics<!-- HTML_TAG_END --></pre></div> <h2 class="relative group"><a id="load-a-base-model" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#load-a-base-model"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Load a base model</span></h2> <p data-svelte-h="svelte-z7c6p7">Before loading a base model, let’s define a helper function to check the total number of parameters a model has, as well | |
| as how many of them are trainable.</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-keyword">def</span> <span class="hljs-title function_">print_trainable_parameters</span>(<span class="hljs-params">model</span>): | |
| <span class="hljs-string">""" | |
| Prints the number of trainable parameters in the model. | |
| """</span> | |
| trainable_params = <span class="hljs-number">0</span> | |
| all_param = <span class="hljs-number">0</span> | |
| <span class="hljs-keyword">for</span> _, param <span class="hljs-keyword">in</span> model.named_parameters(): | |
| all_param += param.numel() | |
| <span class="hljs-keyword">if</span> param.requires_grad: | |
| trainable_params += param.numel() | |
| <span class="hljs-built_in">print</span>( | |
| <span class="hljs-string">f"trainable params: <span class="hljs-subst">{trainable_params}</span> || all params: <span class="hljs-subst">{all_param}</span> || trainable%: <span class="hljs-subst">{<span class="hljs-number">100</span> * trainable_params / all_param:<span class="hljs-number">.2</span>f}</span>"</span> | |
| )<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-134egv8">Choose a base model checkpoint. For this example, we use the <a href="https://huggingface.co/nvidia/mit-b0" rel="nofollow">SegFormer B0 variant</a>. | |
| In addition to the checkpoint, pass the <code>label2id</code> and <code>id2label</code> dictionaries to let the <code>AutoModelForSemanticSegmentation</code> class know that we’re | |
| interested in a custom base model where the decoder head should be randomly initialized using the classes from the custom dataset.</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-keyword">from</span> transformers <span class="hljs-keyword">import</span> AutoModelForSemanticSegmentation, TrainingArguments, Trainer | |
| model = AutoModelForSemanticSegmentation.from_pretrained( | |
| checkpoint, id2label=id2label, label2id=label2id, ignore_mismatched_sizes=<span class="hljs-literal">True</span> | |
| ) | |
| print_trainable_parameters(model)<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-1hyd5lx">At this point you can check with the <code>print_trainable_parameters</code> helper function that all 100% parameters in the base | |
| model (aka <code>model</code>) are trainable.</p> <h2 class="relative group"><a id="wrap-the-base-model-as-a-peftmodel-for-lora-training" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#wrap-the-base-model-as-a-peftmodel-for-lora-training"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Wrap the base model as a PeftModel for LoRA training</span></h2> <p data-svelte-h="svelte-h2ilia">To leverage the LoRa method, you need to wrap the base model as a <code>PeftModel</code>. This involves two steps:</p> <ol data-svelte-h="svelte-gdfjac"><li>Defining LoRa configuration with <code>LoraConfig</code></li> <li>Wrapping the original <code>model</code> with <code>get_peft_model()</code> using the config defined in the step above.</li></ol> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-keyword">from</span> peft <span class="hljs-keyword">import</span> LoraConfig, get_peft_model | |
| config = LoraConfig( | |
| r=<span class="hljs-number">32</span>, | |
| lora_alpha=<span class="hljs-number">32</span>, | |
| target_modules=[<span class="hljs-string">"query"</span>, <span class="hljs-string">"value"</span>], | |
| lora_dropout=<span class="hljs-number">0.1</span>, | |
| bias=<span class="hljs-string">"lora_only"</span>, | |
| modules_to_save=[<span class="hljs-string">"decode_head"</span>], | |
| ) | |
| lora_model = get_peft_model(model, config) | |
| print_trainable_parameters(lora_model)<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-1dir0x5">Let’s review the <code>LoraConfig</code>. To enable LoRA technique, we must define the target modules within <code>LoraConfig</code> so that | |
| <code>PeftModel</code> can update the necessary matrices. Specifically, we want to target the <code>query</code> and <code>value</code> matrices in the | |
| attention blocks of the base model. These matrices are identified by their respective names, “query” and “value”. | |
| Therefore, we should specify these names in the <code>target_modules</code> argument of <code>LoraConfig</code>.</p> <p data-svelte-h="svelte-l3bbz">After we wrap our base model <code>model</code> with <code>PeftModel</code> along with the config, we get | |
| a new model where only the LoRA parameters are trainable (so-called “update matrices”) while the pre-trained parameters | |
| are kept frozen. These include the parameters of the randomly initialized classifier parameters too. This is NOT we want | |
| when fine-tuning the base model on our custom dataset. To ensure that the classifier parameters are also trained, we | |
| specify <code>modules_to_save</code>. This also ensures that these modules are serialized alongside the LoRA trainable parameters | |
| when using utilities like <code>save_pretrained()</code> and <code>push_to_hub()</code>.</p> <p data-svelte-h="svelte-1coy915">In addition to specifying the <code>target_modules</code> within <code>LoraConfig</code>, we also need to specify the <code>modules_to_save</code>. When | |
| we wrap our base model with <code>PeftModel</code> and pass the configuration, we obtain a new model in which only the LoRA parameters | |
| are trainable, while the pre-trained parameters and the randomly initialized classifier parameters are kept frozen. | |
| However, we do want to train the classifier parameters. By specifying the <code>modules_to_save</code> argument, we ensure that the | |
| classifier parameters are also trainable, and they will be serialized alongside the LoRA trainable parameters when we | |
| use utility functions like <code>save_pretrained()</code> and <code>push_to_hub()</code>.</p> <p data-svelte-h="svelte-ss5ia">Let’s review the rest of the parameters:</p> <ul data-svelte-h="svelte-1m2rv79"><li><code>r</code>: The dimension used by the LoRA update matrices.</li> <li><code>alpha</code>: Scaling factor.</li> <li><code>bias</code>: Specifies if the <code>bias</code> parameters should be trained. <code>None</code> denotes none of the <code>bias</code> parameters will be trained.</li></ul> <p data-svelte-h="svelte-10wmx8i">When all is configured, and the base model is wrapped, the <code>print_trainable_parameters</code> helper function lets us explore | |
| the number of trainable parameters. Since we’re interested in performing <strong>parameter-efficient fine-tuning</strong>, | |
| we should expect to see a lower number of trainable parameters from the <code>lora_model</code> in comparison to the original <code>model</code> | |
| which is indeed the case here.</p> <p data-svelte-h="svelte-1o171ky">You can also manually verify what modules are trainable in the <code>lora_model</code>.</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-keyword">for</span> name, param <span class="hljs-keyword">in</span> lora_model.named_parameters(): | |
| <span class="hljs-keyword">if</span> param.requires_grad: | |
| <span class="hljs-built_in">print</span>(name, param.shape)<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-1g7xqd2">This confirms that only the LoRA parameters appended to the attention blocks and the <code>decode_head</code> parameters are trainable.</p> <h2 class="relative group"><a id="train-the-model" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#train-the-model"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Train the model</span></h2> <p data-svelte-h="svelte-sicdsw">Start by defining your training hyperparameters in <code>TrainingArguments</code>. You can change the values of most parameters however | |
| you prefer. Make sure to set <code>remove_unused_columns=False</code>, otherwise the image column will be dropped, and it’s required here. | |
| The only other required parameter is <code>output_dir</code> which specifies where to save your model. | |
| At the end of each epoch, the <code>Trainer</code> will evaluate the IoU metric and save the training checkpoint.</p> <p data-svelte-h="svelte-sj51v2">Note that this example is meant to walk you through the workflow when using PEFT for semantic segmentation. We didn’t | |
| perform extensive hyperparameter tuning to achieve optimal results.</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->model_name = checkpoint.split(<span class="hljs-string">"/"</span>)[-<span class="hljs-number">1</span>] | |
| training_args = TrainingArguments( | |
| output_dir=<span class="hljs-string">f"<span class="hljs-subst">{model_name}</span>-scene-parse-150-lora"</span>, | |
| learning_rate=<span class="hljs-number">5e-4</span>, | |
| num_train_epochs=<span class="hljs-number">50</span>, | |
| per_device_train_batch_size=<span class="hljs-number">4</span>, | |
| per_device_eval_batch_size=<span class="hljs-number">2</span>, | |
| save_total_limit=<span class="hljs-number">3</span>, | |
| evaluation_strategy=<span class="hljs-string">"epoch"</span>, | |
| save_strategy=<span class="hljs-string">"epoch"</span>, | |
| logging_steps=<span class="hljs-number">5</span>, | |
| remove_unused_columns=<span class="hljs-literal">False</span>, | |
| push_to_hub=<span class="hljs-literal">True</span>, | |
| label_names=[<span class="hljs-string">"labels"</span>], | |
| )<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-nimjtw">Pass the training arguments to <code>Trainer</code> along with the model, dataset, and <code>compute_metrics</code> function. | |
| Call <code>train()</code> to finetune your model.</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->trainer = Trainer( | |
| model=lora_model, | |
| args=training_args, | |
| train_dataset=train_ds, | |
| eval_dataset=test_ds, | |
| compute_metrics=compute_metrics, | |
| ) | |
| trainer.train()<!-- HTML_TAG_END --></pre></div> <h2 class="relative group"><a id="save-the-model-and-run-inference" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#save-the-model-and-run-inference"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Save the model and run inference</span></h2> <p data-svelte-h="svelte-wzzfy">Use the <code>save_pretrained()</code> method of the <code>lora_model</code> to save the <em>LoRA-only parameters</em> locally. | |
| Alternatively, use the <code>push_to_hub()</code> method to upload these parameters directly to the Hugging Face Hub | |
| (as shown in the <a href="image_classification_lora">Image classification using LoRA</a> task guide).</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->model_id = <span class="hljs-string">"segformer-scene-parse-150-lora"</span> | |
| lora_model.save_pretrained(model_id)<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-8jrf93">We can see that the LoRA-only parameters are just <strong>2.2 MB in size</strong>! This greatly improves the portability when using very large models.</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->!<span class="hljs-built_in">ls</span> -lh {model_id} | |
| total 2.2M | |
| -rw-r--r-- 1 root root 369 Feb 8 03:09 adapter_config.json | |
| -rw-r--r-- 1 root root 2.2M Feb 8 03:09 adapter_model.bin<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-1ws2x6q">Let’s now prepare an <code>inference_model</code> and run inference.</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-keyword">from</span> peft <span class="hljs-keyword">import</span> PeftConfig | |
| config = PeftConfig.from_pretrained(model_id) | |
| model = AutoModelForSemanticSegmentation.from_pretrained( | |
| checkpoint, id2label=id2label, label2id=label2id, ignore_mismatched_sizes=<span class="hljs-literal">True</span> | |
| ) | |
| inference_model = PeftModel.from_pretrained(model, model_id)<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-1k70j7o">Get an image:</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-keyword">import</span> requests | |
| url = <span class="hljs-string">"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/semantic-seg-image.png"</span> | |
| image = Image.<span class="hljs-built_in">open</span>(requests.get(url, stream=<span class="hljs-literal">True</span>).raw) | |
| image<!-- HTML_TAG_END --></pre></div> <div class="flex justify-center" data-svelte-h="svelte-runw56"><img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/semantic-seg-image.png" alt="photo of a room"></div> <p data-svelte-h="svelte-1a29ol4">Preprocess the image to prepare for inference.</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->encoding = image_processor(image.convert(<span class="hljs-string">"RGB"</span>), return_tensors=<span class="hljs-string">"pt"</span>)<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-k2a84">Run inference with the encoded image.</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-keyword">with</span> torch.no_grad(): | |
| outputs = inference_model(pixel_values=encoding.pixel_values) | |
| logits = outputs.logits | |
| upsampled_logits = nn.functional.interpolate( | |
| logits, | |
| size=image.size[::-<span class="hljs-number">1</span>], | |
| mode=<span class="hljs-string">"bilinear"</span>, | |
| align_corners=<span class="hljs-literal">False</span>, | |
| ) | |
| pred_seg = upsampled_logits.argmax(dim=<span class="hljs-number">1</span>)[<span class="hljs-number">0</span>]<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-paeq5">Next, visualize the results. We need a color palette for this. Here, we use ade_palette(). As it is a long array, so | |
| we don’t include it in this guide, please copy it from <a href="https://github.com/tensorflow/models/blob/3f1ca33afe3c1631b733ea7e40c294273b9e406d/research/deeplab/utils/get_dataset_colormap.py#L51" rel="nofollow">the TensorFlow Model Garden repository</a>.</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-keyword">import</span> matplotlib.pyplot <span class="hljs-keyword">as</span> plt | |
| color_seg = np.zeros((pred_seg.shape[<span class="hljs-number">0</span>], pred_seg.shape[<span class="hljs-number">1</span>], <span class="hljs-number">3</span>), dtype=np.uint8) | |
| palette = np.array(ade_palette()) | |
| <span class="hljs-keyword">for</span> label, color <span class="hljs-keyword">in</span> <span class="hljs-built_in">enumerate</span>(palette): | |
| color_seg[pred_seg == label, :] = color | |
| color_seg = color_seg[..., ::-<span class="hljs-number">1</span>] <span class="hljs-comment"># convert to BGR</span> | |
| img = np.array(image) * <span class="hljs-number">0.5</span> + color_seg * <span class="hljs-number">0.5</span> <span class="hljs-comment"># plot the image with the segmentation map</span> | |
| img = img.astype(np.uint8) | |
| plt.figure(figsize=(<span class="hljs-number">15</span>, <span class="hljs-number">10</span>)) | |
| plt.imshow(img) | |
| plt.show()<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-1rbxn15">As you can see, the results are far from perfect, however, this example is designed to illustrate the end-to-end workflow of | |
| fine-tuning a semantic segmentation model with LoRa technique, and is not aiming to achieve state-of-the-art | |
| results. The results you see here are the same as you would get if you performed full fine-tuning on the same setup (same | |
| model variant, same dataset, same training schedule, etc.), except LoRA allows to achieve them with a fraction of total | |
| trainable parameters and in less time.</p> <p data-svelte-h="svelte-1bj2ejn">If you wish to use this example and improve the results, here are some things that you can try:</p> <ul data-svelte-h="svelte-1herdxp"><li>Increase the number of training samples.</li> <li>Try a larger SegFormer model variant (explore available model variants on the <a href="https://huggingface.co/models?search=segformer" rel="nofollow">Hugging Face Hub</a>).</li> <li>Try different values for the arguments available in <code>LoraConfig</code>.</li> <li>Tune the learning rate and batch size.</li></ul> <p></p> | |
| <script> | |
| { | |
| __sveltekit_c8wad6 = { | |
| assets: "/docs/peft/main/en", | |
| base: "/docs/peft/main/en", | |
| env: {} | |
| }; | |
| const element = document.currentScript.parentElement; | |
| const data = [null,null]; | |
| Promise.all([ | |
| import("/docs/peft/main/en/_app/immutable/entry/start.f1bde2f6.js"), | |
| import("/docs/peft/main/en/_app/immutable/entry/app.182b7f64.js") | |
| ]).then(([kit, app]) => { | |
| kit.start(app, element, { | |
| node_ids: [0, 39], | |
| data, | |
| form: null, | |
| error: null | |
| }); | |
| }); | |
| } | |
| </script> | |
Xet Storage Details
- Size:
- 72.3 kB
- Xet hash:
- 50d8b906eb8587c31e4487212526ae1c4548284d33e3bb2ec6fd5671e57de994
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.