peft/main/en/package_reference/lora_variant_velora.html

<meta charset="utf-8" /><meta name="hf:doc:metadata" content="{&quot;title&quot;:&quot;VeLoRA&quot;,&quot;local&quot;:&quot;velora&quot;,&quot;sections&quot;:[{&quot;title&quot;:&quot;Caveats&quot;,&quot;local&quot;:&quot;caveats&quot;,&quot;sections&quot;:[],&quot;depth&quot;:4}],&quot;depth&quot;:3}"/>
		<link href="/docs/peft/main/en/_app/immutable/entry/start.CGYl62d0.js" rel="modulepreload">
		<link href="/docs/peft/main/en/_app/immutable/chunks/Cm4lWyK0.js" rel="modulepreload">
		<link href="/docs/peft/main/en/_app/immutable/chunks/CCQPhADn.js" rel="modulepreload">
		<link href="/docs/peft/main/en/_app/immutable/entry/app.DrwbNPIN.js" rel="modulepreload">
		<link href="/docs/peft/main/en/_app/immutable/chunks/DW4-mrp-.js" rel="modulepreload">
		<link href="/docs/peft/main/en/_app/immutable/chunks/RUwc_6JX.js" rel="modulepreload">
		<link href="/docs/peft/main/en/_app/immutable/chunks/DsnmJJEf.js" rel="modulepreload">
		<link href="/docs/peft/main/en/_app/immutable/chunks/DsPOG0MZ.js" rel="modulepreload">
		<link href="/docs/peft/main/en/_app/immutable/nodes/0.3WyUMUPr.js" rel="modulepreload">
		<link href="/docs/peft/main/en/_app/immutable/chunks/C5swDPgD.js" rel="modulepreload">
		<link href="/docs/peft/main/en/_app/immutable/nodes/51.CWNLsqkU.js" rel="modulepreload">
		<link href="/docs/peft/main/en/_app/immutable/chunks/C96YZLYZ.js" rel="modulepreload">
		<!--qgylys--><meta name="hf:doc:metadata" content="{&quot;title&quot;:&quot;VeLoRA&quot;,&quot;local&quot;:&quot;velora&quot;,&quot;sections&quot;:[{&quot;title&quot;:&quot;Caveats&quot;,&quot;local&quot;:&quot;caveats&quot;,&quot;sections&quot;:[],&quot;depth&quot;:4}],&quot;depth&quot;:3}"/><!---->
		<link href="/docs/peft/main/en/_app/immutable/assets/0.tn0RQdqM.css" rel="modulepreload"> <!--[--><!--[0--><!--[--><!--[0--><!--[--><p></p> <!--[2--><h3 class="relative group"><a id="velora" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#velora"><span><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg><!----></span></a> <span>VeLoRA</span></h3><!--]--><!----> <blockquote class="note"><p>This is a variant of LoRA and therefore everything that is possible with LoRA is valid for this method except otherwise stated on this page.</p></blockquote> <p><a href="https://huggingface.co/papers/2405.17991" rel="nofollow">VeLoRA</a> is a LoRA variant that reduces training memory by compressing the activations saved for the LoRA in the forward pass and then reconstructing them in the backwards pass to implement the update rules. In PEFT, VeLoRA is configured as a LoRA variant through the <code>velora_config</code> argument on <a href="/docs/peft/main/en/package_reference/lora#peft.LoraConfig">LoraConfig</a>.</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5   text-gray-600  " title="code excerpt" type="button"><svg xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg><!----> <div class=" absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0 "><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent;"></div> Copied</div><!----></button><!----></div> <pre class="language-py "><!----><span class="hljs-keyword">from</span> peft <span class="hljs-keyword">import</span> LoraConfig, VeloraConfig

config = LoraConfig(
    target_modules=[<span class="hljs-string">&quot;q_proj&quot;</span>, <span class="hljs-string">&quot;v_proj&quot;</span>],
    velora_config=VeloraConfig(
        num_groups=<span class="hljs-number">64</span>,
        scale=<span class="hljs-number">0.2</span>,
        init_type=<span class="hljs-string">&quot;batch_average&quot;</span>,
    ),
)<!----></pre></div><!----> <p>VeLoRA is applied to every LoRA layer selected by <code>target_modules</code>. <code>num_groups</code> controls how the input activation depth is split before compression. If the activation depth is not evenly divisible by <code>num_groups</code>, VeLoRA pads the grouped representation internally and removes the padding after reconstruction. <code>scale</code> rescales the reconstructed activations during the backward pass, and <code>init_type</code> chooses how the projection is initialized.</p> <p>Use <code>batch_average_once</code> to initialize the projection from the first training batch, <code>batch_average</code> to update it from every training forward pass, or <code>random</code> to initialize it immediately from a random normalized vector.</p> <p>Below are some results with the <a href="https://github.com/huggingface/peft/tree/main/method_comparison/MetaMathQA" rel="nofollow">MetaMathQA benchmark</a>.</p> <table><thead><tr><th>Variant</th><th align="right">Training Loss</th><th align="right">Max Memory (GiB)</th><th align="right">Tokens/sec</th></tr></thead><tbody><tr><td>LoRA</td><td align="right">0.5427</td><td align="right">27.69</td><td align="right">2366.2</td></tr><tr><td>LoRA + GC</td><td align="right">0.5426</td><td align="right">13.17</td><td align="right">1671.8</td></tr><tr><td>LoRA+VeLoRA</td><td align="right">0.5427</td><td align="right">19.94</td><td align="right">2057.6</td></tr></tbody></table> <!--[3--><h4 class="relative group"><a id="caveats" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#caveats"><span><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg><!----></span></a> <span>Caveats</span></h4><!--]--><!----> <ul><li>VeLoRA is currently supported on standard LoRA linear layers only.</li></ul> <a class="!text-gray-400 !no-underline text-sm flex items-center not-prose mt-4" href="https://github.com/huggingface/peft/blob/main/docs/source/package_reference/lora_variant_velora.md" target="_blank"><svg class="mr-1" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M31,16l-7,7l-1.41-1.41L28.17,16l-5.58-5.59L24,9l7,7z"></path><path d="M1,16l7-7l1.41,1.41L3.83,16l5.58,5.59L8,23l-7-7z"></path><path d="M12.419,25.484L17.639,6.552l1.932,0.518L14.351,26.002z"></path></svg><!----> <span><span class="underline">Update</span> on GitHub</span></a><!----> <p></p><!--]--><!----><!--]--><!--]--><!--]--> <!--[-1--><!--]--><!--]-->
			
			<script>
				{
					__sveltekit_16y3wl8 = {
						base: "/docs/peft/main/en",
						assets: "/docs/peft/main/en"
					};

					const element = document.currentScript.parentElement;

					Promise.all([
						import("/docs/peft/main/en/_app/immutable/entry/start.CGYl62d0.js"),
						import("/docs/peft/main/en/_app/immutable/entry/app.DrwbNPIN.js")
					]).then(([kit, app]) => {
						kit.start(app, element, {
							node_ids: [0, 51],
							data: [null,null],
							form: null,
							error: null
						});
					});
				}
			</script>