Buckets:
| <meta charset="utf-8" /><meta name="hf:doc:metadata" content="{"title":"Adapter injection","local":"adapter-injection","sections":[{"title":"Creating a new PEFT model","local":"creating-a-new-peft-model","sections":[{"title":"Injection based on a state_dict","local":"injection-based-on-a-statedict","sections":[],"depth":3}],"depth":2},{"title":"Saving the model","local":"saving-the-model","sections":[],"depth":2},{"title":"Loading the model","local":"loading-the-model","sections":[],"depth":2},{"title":"Setting and loading base weights","local":"setting-and-loading-base-weights","sections":[],"depth":2}],"depth":1}"/> | |
| <link href="/docs/peft/main/en/_app/immutable/entry/start.H0B-zMV-.js" rel="modulepreload"> | |
| <link href="/docs/peft/main/en/_app/immutable/chunks/B6nluxQH.js" rel="modulepreload"> | |
| <link href="/docs/peft/main/en/_app/immutable/chunks/CCQPhADn.js" rel="modulepreload"> | |
| <link href="/docs/peft/main/en/_app/immutable/entry/app.DFygnjAi.js" rel="modulepreload"> | |
| <link href="/docs/peft/main/en/_app/immutable/chunks/DW4-mrp-.js" rel="modulepreload"> | |
| <link href="/docs/peft/main/en/_app/immutable/chunks/RUwc_6JX.js" rel="modulepreload"> | |
| <link href="/docs/peft/main/en/_app/immutable/chunks/DsnmJJEf.js" rel="modulepreload"> | |
| <link href="/docs/peft/main/en/_app/immutable/chunks/DsPOG0MZ.js" rel="modulepreload"> | |
| <link href="/docs/peft/main/en/_app/immutable/nodes/0.C09delyQ.js" rel="modulepreload"> | |
| <link href="/docs/peft/main/en/_app/immutable/chunks/C5swDPgD.js" rel="modulepreload"> | |
| <link href="/docs/peft/main/en/_app/immutable/nodes/7.CY2vMXrO.js" rel="modulepreload"> | |
| <link href="/docs/peft/main/en/_app/immutable/chunks/C96YZLYZ.js" rel="modulepreload"> | |
| <!--1bnl0ii--><meta name="hf:doc:metadata" content="{"title":"Adapter injection","local":"adapter-injection","sections":[{"title":"Creating a new PEFT model","local":"creating-a-new-peft-model","sections":[{"title":"Injection based on a state_dict","local":"injection-based-on-a-statedict","sections":[],"depth":3}],"depth":2},{"title":"Saving the model","local":"saving-the-model","sections":[],"depth":2},{"title":"Loading the model","local":"loading-the-model","sections":[],"depth":2},{"title":"Setting and loading base weights","local":"setting-and-loading-base-weights","sections":[],"depth":2}],"depth":1}"/><!----> | |
| <link href="/docs/peft/main/en/_app/immutable/assets/0.tn0RQdqM.css" rel="modulepreload"> <!--[--><!--[0--><!--[--><!--[0--><!--[--><p></p> <div class="items-center shrink-0 min-w-[100px] max-sm:min-w-[50px] justify-end ml-auto flex" style="float: right; margin-left: 10px; display: inline-flex; position: relative; z-index: 10;"><div class="inline-flex rounded-md max-sm:rounded-sm"><button class="inline-flex items-center gap-1 h-7 max-sm:h-7 px-2 max-sm:px-1.5 text-sm font-medium text-gray-800 border border-r-0 rounded-l-md max-sm:rounded-l-sm border-gray-200 bg-white hover:shadow-inner dark:border-gray-850 dark:bg-gray-950 dark:text-gray-200 dark:hover:bg-gray-800" aria-live="polite"><span class="inline-flex items-center justify-center rounded-md p-0.5 max-sm:p-0 hover:text-gray-800 dark:hover:text-gray-200"><svg class="sm:size-3.5 size-3" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg><!----></span> <span>Copy page</span></button> <button class="inline-flex items-center justify-center w-6 max-sm:w-5 h-7 max-sm:h-7 disabled:pointer-events-none text-sm text-gray-500 hover:text-gray-700 dark:hover:text-white rounded-r-md max-sm:rounded-r-sm border border-l transition border-gray-200 bg-white hover:shadow-inner dark:border-gray-850 dark:bg-gray-950 dark:text-gray-200 dark:hover:bg-gray-800" aria-haspopup="menu" aria-expanded="false" aria-label="Open copy menu"><svg class="transition-transform text-gray-400 overflow-visible sm:size-3.5 size-3 rotate-0" width="1em" height="1em" viewBox="0 0 12 7" fill="none" xmlns="http://www.w3.org/2000/svg"><path d="M1 1L6 6L11 1" stroke="currentColor"></path></svg><!----></button></div> <!--[-1--><!--]--></div><!----> <!--[0--><h1 class="relative group"><a id="adapter-injection" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#adapter-injection"><span><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg><!----></span></a> <span>Adapter injection</span></h1><!--]--><!----> <p>With PEFT, you can inject trainable adapters into any <code>torch</code> module which allows you to use adapter methods without relying on the modeling classes in PEFT. This works for all adapters except for those based on prompt learning (e.g. prefix tuning or p-tuning).</p> <p>Check the table below to see when you should inject adapters.</p> <table><thead><tr><th>Pros</th><th>Cons</th></tr></thead><tbody><tr><td>the model is modified inplace, keeping all the original attributes and methods</td><td>manually write the <code>from_pretrained</code> and <code>save_pretrained</code> utility functions from Hugging Face to save and load adapters</td></tr><tr><td>works for any <code>torch</code> module and modality</td><td>doesn’t work with any of the utility methods provided by <code>PeftModel</code> such as disabling and merging adapters</td></tr></tbody></table> <!--[1--><h2 class="relative group"><a id="creating-a-new-peft-model" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#creating-a-new-peft-model"><span><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg><!----></span></a> <span>Creating a new PEFT model</span></h2><!--]--><!----> <p>To perform the adapter injection, use the <a href="/docs/peft/pr_3185/en/package_reference/functional#peft.inject_adapter_in_model">inject_adapter_in_model()</a> method. This method takes 3 arguments, the PEFT config, the model, and an optional adapter name. You can also attach multiple adapters to the model if you call <a href="/docs/peft/pr_3185/en/package_reference/functional#peft.inject_adapter_in_model">inject_adapter_in_model()</a> multiple times with different adapter names.</p> <p>For example, to inject LoRA adapters into the <code>linear</code> submodule of the <code>DummyModel</code> module:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg><!----> <div class=" absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0 "><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent;"></div> Copied</div><!----></button><!----></div> <pre class="language-python "><!----><span class="hljs-keyword">import</span> torch | |
| <span class="hljs-keyword">from</span> peft <span class="hljs-keyword">import</span> inject_adapter_in_model, LoraConfig | |
| <span class="hljs-keyword">class</span> <span class="hljs-title class_">DummyModel</span>(torch.nn.Module): | |
| <span class="hljs-keyword">def</span> <span class="hljs-title function_">__init__</span>(<span class="hljs-params">self</span>): | |
| <span class="hljs-built_in">super</span>().__init__() | |
| <span class="hljs-variable language_">self</span>.embedding = torch.nn.Embedding(<span class="hljs-number">10</span>, <span class="hljs-number">10</span>) | |
| <span class="hljs-variable language_">self</span>.linear = torch.nn.Linear(<span class="hljs-number">10</span>, <span class="hljs-number">10</span>) | |
| <span class="hljs-variable language_">self</span>.lm_head = torch.nn.Linear(<span class="hljs-number">10</span>, <span class="hljs-number">10</span>) | |
| <span class="hljs-keyword">def</span> <span class="hljs-title function_">forward</span>(<span class="hljs-params">self, input_ids</span>): | |
| x = <span class="hljs-variable language_">self</span>.embedding(input_ids) | |
| x = <span class="hljs-variable language_">self</span>.linear(x) | |
| x = <span class="hljs-variable language_">self</span>.lm_head(x) | |
| <span class="hljs-keyword">return</span> x | |
| lora_config = LoraConfig( | |
| lora_alpha=<span class="hljs-number">16</span>, | |
| lora_dropout=<span class="hljs-number">0.1</span>, | |
| r=<span class="hljs-number">64</span>, | |
| bias=<span class="hljs-string">"none"</span>, | |
| target_modules=[<span class="hljs-string">"linear"</span>], | |
| ) | |
| model = DummyModel() | |
| model = inject_adapter_in_model(lora_config, model) | |
| dummy_inputs = torch.LongTensor([[<span class="hljs-number">0</span>, <span class="hljs-number">1</span>, <span class="hljs-number">2</span>, <span class="hljs-number">3</span>, <span class="hljs-number">4</span>, <span class="hljs-number">5</span>, <span class="hljs-number">6</span>, <span class="hljs-number">7</span>]]) | |
| dummy_outputs = model(dummy_inputs)<!----></pre></div><!----> <p>Print the model to see that the adapters have been correctly injected.</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg><!----> <div class=" absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0 "><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent;"></div> Copied</div><!----></button><!----></div> <pre class="language-bash "><!---->DummyModel( | |
| (embedding): Embedding(10, 10) | |
| (linear): Linear( | |
| in_features=10, out_features=10, bias=True | |
| (lora_dropout): ModuleDict( | |
| (default): Dropout(p=0.1, inplace=False) | |
| ) | |
| (lora_A): ModuleDict( | |
| (default): Linear(in_features=10, out_features=64, bias=False) | |
| ) | |
| (lora_B): ModuleDict( | |
| (default): Linear(in_features=64, out_features=10, bias=False) | |
| ) | |
| (lora_embedding_A): ParameterDict() | |
| (lora_embedding_B): ParameterDict() | |
| ) | |
| (lm_head): Linear(in_features=10, out_features=10, bias=True) | |
| )<!----></pre></div><!----> <!--[2--><h3 class="relative group"><a id="injection-based-on-a-statedict" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#injection-based-on-a-statedict"><span><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg><!----></span></a> <span>Injection based on a state_dict</span></h3><!--]--><!----> <p>Sometimes, it is possible that there is a PEFT adapter checkpoint but the corresponding PEFT config is not known for whatever reason. To inject the PEFT layers for this checkpoint, you would usually have to reverse-engineer the corresponding PEFT config, most notably the <code>target_modules</code> argument, based on the <code>state_dict</code> from the checkpoint. This can be cumbersome and error prone. To avoid this, it is also possible to call <a href="/docs/peft/pr_3185/en/package_reference/functional#peft.inject_adapter_in_model">inject_adapter_in_model()</a> and pass the loaded <code>state_dict</code> as an argument:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg><!----> <div class=" absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0 "><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent;"></div> Copied</div><!----></button><!----></div> <pre class="language-python "><!----><span class="hljs-keyword">from</span> safetensors.torch <span class="hljs-keyword">import</span> load_file | |
| model = ... | |
| state_dict = load_file(<path-to-safetensors-file>) | |
| lora_config = LoraConfig(...) | |
| model = inject_adapter_in_model(lora_config, model, state_dict=state_dict)<!----></pre></div><!----> <p>In this case, PEFT will use the <code>state_dict</code> as reference for which layers to target instead of using the PEFT config. As a user, you don’t have to set the exact <code>target_modules</code> of the PEFT config for this to work. However, you should still pass a PEFT config of the right type, in this example <code>LoraConfig</code>, you can leave the <code>target_modules</code> as <code>None</code>.</p> <p>Be aware that this still only creates the uninitialized PEFT layers, the values from the <code>state_dict</code> are not used to populate the model weights. To populate the weights, proceed with calling <a href="/docs/peft/pr_3185/en/package_reference/functional#peft.set_peft_model_state_dict">set_peft_model_state_dict()</a> as described below.</p> <p>⚠️ Note that if there is a mismatch between what is configured in the PEFT config and what is found in the <code>state_dict</code>, PEFT will warn you about this. You can ignore the warning if you know that the PEFT config is not correctly specified.</p> <blockquote class="warning"><p>If the original PEFT adapters was using <code>target_parameters</code> instead of <code>target_modules</code>, injecting from a <code>state_dict</code> will not work correctly. In this case, it is mandatory to use the correct PEFT config for injection.</p></blockquote> <!--[1--><h2 class="relative group"><a id="saving-the-model" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#saving-the-model"><span><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg><!----></span></a> <span>Saving the model</span></h2><!--]--><!----> <p>To only save the adapter, use the <a href="/docs/peft/pr_3185/en/package_reference/functional#peft.get_peft_model_state_dict">get_peft_model_state_dict()</a> function:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg><!----> <div class=" absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0 "><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent;"></div> Copied</div><!----></button><!----></div> <pre class="language-python "><!----><span class="hljs-keyword">from</span> peft <span class="hljs-keyword">import</span> get_peft_model_state_dict | |
| peft_state_dict = get_peft_model_state_dict(model) | |
| <span class="hljs-built_in">print</span>(peft_state_dict)<!----></pre></div><!----> <p>Otherwise, <code>model.state_dict()</code> returns the full state dict of the model.</p> <!--[1--><h2 class="relative group"><a id="loading-the-model" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#loading-the-model"><span><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg><!----></span></a> <span>Loading the model</span></h2><!--]--><!----> <p>After loading the saved <code>state_dict</code>, it can be applied using the <a href="/docs/peft/pr_3185/en/package_reference/functional#peft.set_peft_model_state_dict">set_peft_model_state_dict()</a> function:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg><!----> <div class=" absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0 "><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent;"></div> Copied</div><!----></button><!----></div> <pre class="language-python "><!----><span class="hljs-keyword">from</span> peft <span class="hljs-keyword">import</span> set_peft_model_state_dict | |
| model = DummyModel() | |
| model = inject_adapter_in_model(lora_config, model) | |
| outcome = set_peft_model_state_dict(model, peft_state_dict) | |
| <span class="hljs-comment"># check that there were no wrong keys</span> | |
| <span class="hljs-built_in">print</span>(outcome.unexpected_keys)<!----></pre></div><!----> <p>If injecting the adapter is slow or you need to load a large number of adapters, you may use an optimization that allows to create an “empty” adapter on meta device and only fills the weights with real weights when the <a href="/docs/peft/pr_3185/en/package_reference/functional#peft.set_peft_model_state_dict">set_peft_model_state_dict()</a> is called. To do this, pass <code>low_cpu_mem_usage=True</code> to both <a href="/docs/peft/pr_3185/en/package_reference/functional#peft.inject_adapter_in_model">inject_adapter_in_model()</a> and <a href="/docs/peft/pr_3185/en/package_reference/functional#peft.set_peft_model_state_dict">set_peft_model_state_dict()</a>.</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg><!----> <div class=" absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0 "><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent;"></div> Copied</div><!----></button><!----></div> <pre class="language-python "><!---->model = DummyModel() | |
| model = inject_adapter_in_model(lora_config, model, low_cpu_mem_usage=<span class="hljs-literal">True</span>) | |
| <span class="hljs-built_in">print</span>(model.linear.lora_A[<span class="hljs-string">"default"</span>].weight.device.<span class="hljs-built_in">type</span> == <span class="hljs-string">"meta"</span>) <span class="hljs-comment"># should be True</span> | |
| set_peft_model_state_dict(model, peft_state_dict, low_cpu_mem_usage=<span class="hljs-literal">True</span>) | |
| <span class="hljs-built_in">print</span>(model.linear.lora_A[<span class="hljs-string">"default"</span>].weight.device.<span class="hljs-built_in">type</span> == <span class="hljs-string">"cpu"</span>) <span class="hljs-comment"># should be True</span><!----></pre></div><!----> <p>For loading weights from the hub there’s the low-level <code>load_peft_weights()</code> function:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg><!----> <div class=" absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0 "><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent;"></div> Copied</div><!----></button><!----></div> <pre class="language-python "><!---->state_dict = load_peft_weights(<span class="hljs-string">"my-account/my-adapter-repo"</span>)<!----></pre></div><!----> <!--[1--><h2 class="relative group"><a id="setting-and-loading-base-weights" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#setting-and-loading-base-weights"><span><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg><!----></span></a> <span>Setting and loading base weights</span></h2><!--]--><!----> <p>The functions above deal with the state dict of the <em>adapter</em>. There are also situations where the <em>base model</em> weights need to be read or written through the PEFT wrapper. This is not entirely trivial because PEFT renames the parameters of targeted modules (e.g. <code>q_proj.weight</code> becomes <code>q_proj.base_layer.weight</code>) and adds adapter parameters that don’t exist in the base model. Use <code>get_base_model_state_dict()</code> and <code>set_base_model_state_dict()</code> to translate between the two namings:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg><!----> <div class=" absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0 "><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent;"></div> Copied</div><!----></button><!----></div> <pre class="language-python "><!----><span class="hljs-keyword">from</span> peft <span class="hljs-keyword">import</span> LoraConfig, get_peft_model, get_base_model_state_dict, set_base_model_state_dict | |
| <span class="hljs-keyword">from</span> transformers <span class="hljs-keyword">import</span> AutoModelForCausalLM | |
| model = AutoModelForCausalLM.from_pretrained(<span class="hljs-string">"facebook/opt-125m"</span>) | |
| model = get_peft_model(model, LoraConfig(target_modules=<span class="hljs-string">"all-linear"</span>)) | |
| <span class="hljs-comment"># state dict with the original (pre-PEFT) key names, adapter parameters excluded</span> | |
| base_state_dict = get_base_model_state_dict(model) | |
| <span class="hljs-comment"># load a state dict with original key names into the PEFT-wrapped model</span> | |
| outcome = set_base_model_state_dict(model, base_state_dict, strict=<span class="hljs-literal">False</span>) | |
| <span class="hljs-comment"># check that there were no wrong keys</span> | |
| <span class="hljs-built_in">print</span>(outcome.missing_keys, outcome.unexpected_keys)<!----></pre></div><!----> <p>The main use case for this is loading the base weights <em>after</em> the model has already been wrapped by PEFT. For example, FSDP training setups such as TorchTitan initialize the model on the meta device, apply PEFT, and shard the result. Real memory only exists after sharding, so the checkpoint, whose keys use the original names, has to be loaded into the already-wrapped model. Note that <code>get_base_model_state_dict()</code> returns the live tensors of the model, not copies.</p> <a class="!text-gray-400 !no-underline text-sm flex items-center not-prose mt-4" href="https://github.com/huggingface/peft/blob/main/docs/source/developer_guides/low_level_api.md" target="_blank"><svg class="mr-1" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M31,16l-7,7l-1.41-1.41L28.17,16l-5.58-5.59L24,9l7,7z"></path><path d="M1,16l7-7l1.41,1.41L3.83,16l5.58,5.59L8,23l-7-7z"></path><path d="M12.419,25.484L17.639,6.552l1.932,0.518L14.351,26.002z"></path></svg><!----> <span><span class="underline">Update</span> on GitHub</span></a><!----> <p></p><!--]--><!----><!--]--><!--]--><!--]--> <!--[-1--><!--]--><!--]--> | |
| <script> | |
| { | |
| __sveltekit_1jptnp0 = { | |
| base: "/docs/peft/main/en", | |
| assets: "/docs/peft/main/en" | |
| }; | |
| const element = document.currentScript.parentElement; | |
| Promise.all([ | |
| import("/docs/peft/main/en/_app/immutable/entry/start.H0B-zMV-.js"), | |
| import("/docs/peft/main/en/_app/immutable/entry/app.DFygnjAi.js") | |
| ]).then(([kit, app]) => { | |
| kit.start(app, element, { | |
| node_ids: [0, 7], | |
| data: [null,null], | |
| form: null, | |
| error: null | |
| }); | |
| }); | |
| } | |
| </script> | |
Xet Storage Details
- Size:
- 34.4 kB
- Xet hash:
- c188fd33e1309eaa4a310003d264a3ef31bd7014d495db31f435101163089a8a
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.