bitsandbytes/v0.43.0/en/integrations.html

<meta charset="utf-8" /><meta name="hf:doc:metadata" content="{&quot;title&quot;:&quot;Transformers&quot;,&quot;local&quot;:&quot;transformers&quot;,&quot;sections&quot;:[],&quot;depth&quot;:1}">
		<link href="/docs/bitsandbytes/v0.43.0/en/_app/immutable/assets/0.e3b0c442.css" rel="modulepreload">
		<link rel="modulepreload" href="/docs/bitsandbytes/v0.43.0/en/_app/immutable/entry/start.eac2c940.js">
		<link rel="modulepreload" href="/docs/bitsandbytes/v0.43.0/en/_app/immutable/chunks/scheduler.0f20f9e5.js">
		<link rel="modulepreload" href="/docs/bitsandbytes/v0.43.0/en/_app/immutable/chunks/singletons.6bcd6fdd.js">
		<link rel="modulepreload" href="/docs/bitsandbytes/v0.43.0/en/_app/immutable/chunks/index.e6666749.js">
		<link rel="modulepreload" href="/docs/bitsandbytes/v0.43.0/en/_app/immutable/chunks/paths.d58d761f.js">
		<link rel="modulepreload" href="/docs/bitsandbytes/v0.43.0/en/_app/immutable/entry/app.24bdf5c5.js">
		<link rel="modulepreload" href="/docs/bitsandbytes/v0.43.0/en/_app/immutable/chunks/index.fbd8b8e3.js">
		<link rel="modulepreload" href="/docs/bitsandbytes/v0.43.0/en/_app/immutable/nodes/0.4860f614.js">
		<link rel="modulepreload" href="/docs/bitsandbytes/v0.43.0/en/_app/immutable/chunks/each.e59479a4.js">
		<link rel="modulepreload" href="/docs/bitsandbytes/v0.43.0/en/_app/immutable/nodes/8.523af57e.js">
		<link rel="modulepreload" href="/docs/bitsandbytes/v0.43.0/en/_app/immutable/chunks/Tip.1ed3d6e3.js">
		<link rel="modulepreload" href="/docs/bitsandbytes/v0.43.0/en/_app/immutable/chunks/CodeBlock.c14486c2.js">
		<link rel="modulepreload" href="/docs/bitsandbytes/v0.43.0/en/_app/immutable/chunks/Heading.2451f3bb.js"><!-- HEAD_svelte-u9bgzb_START --><meta name="hf:doc:metadata" content="{&quot;title&quot;:&quot;Transformers&quot;,&quot;local&quot;:&quot;transformers&quot;,&quot;sections&quot;:[],&quot;depth&quot;:1}"><!-- HEAD_svelte-u9bgzb_END -->      <p></p>  <h1 class="relative group"><a id="transformers" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#transformers"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Transformers</span></h1> <p data-svelte-h="svelte-12fkmk9">With Transformers it’s very easy to load any model in 4 or 8-bit, quantizing them on the fly with <code>bitsandbytes</code> primitives.</p> <p data-svelte-h="svelte-4syxo">Please review the <a href="https://huggingface.co/docs/transformers/main/en/quantization#bitsandbytes" rel="nofollow"><code>bitsandbytes</code> section in the Transformers docs</a>.</p> <p data-svelte-h="svelte-1peqy09">Details about the BitsAndBytesConfig can be found <a href="https://huggingface.co/docs/transformers/v4.37.2/en/main_classes/quantization#transformers.BitsAndBytesConfig" rel="nofollow">here</a>.</p>  <div class="course-tip course-tip-orange bg-gradient-to-br dark:bg-gradient-to-r before:border-orange-500 dark:before:border-orange-800 from-orange-50 dark:from-gray-900 to-white dark:to-gray-950 border border-orange-50 text-orange-700 dark:text-gray-400"><p data-svelte-h="svelte-ugrp4n"><strong>Beware: bf16 is the optimal compute data type!</strong></p> <p data-svelte-h="svelte-nj78a">If your hardware supports it, <code>bf16</code> is the optimal compute dtype. The default is <code>float32</code> for backward compatibility and numerical stability. <code>float16</code> often leads to numerical instabilities, but <code>bfloat16</code> provides the benefits of both worlds: numerical stability equivalent to float32, but combined with the memory footprint and significant computation speedup of a 16-bit data type. Therefore, be sure to check if your hardware supports <code>bf16</code> and configure it using the <code>bnb_4bit_compute_dtype</code> parameter in BitsAndBytesConfig:</p></div> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5   text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-keyword">import</span> torch
<span class="hljs-keyword">from</span> transformers <span class="hljs-keyword">import</span> BitsAndBytesConfig

quantization_config = BitsAndBytesConfig(load_in_4bit=<span class="hljs-literal">True</span>, bnb_4bit_compute_dtype=torch.bfloat16)<!-- HTML_TAG_END --></pre></div>  <h1 class="relative group"><a id="peft" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#peft"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>PEFT</span></h1> <p data-svelte-h="svelte-1pzo32b">With <code>PEFT</code>, you can use QLoRA out of the box with <code>LoraConfig</code> and a 4-bit base model.</p> <p data-svelte-h="svelte-gc66sr">Please review the <a href="https://huggingface.co/docs/peft/developer_guides/quantization#quantize-a-model" rel="nofollow">bitsandbytes section in the PEFT docs</a>.</p>  <h1 class="relative group"><a id="accelerate" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#accelerate"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Accelerate</span></h1> <p data-svelte-h="svelte-nm5i1a">Bitsandbytes is also easily usable from within Accelerate, where you can quantize any PyTorch model simply by passing a quantization config; e.g:</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5   text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-keyword">from</span> accelerate <span class="hljs-keyword">import</span> init_empty_weights
<span class="hljs-keyword">from</span> accelerate.utils <span class="hljs-keyword">import</span> BnbQuantizationConfig, load_and_quantize_model
<span class="hljs-keyword">from</span> mingpt.model <span class="hljs-keyword">import</span> GPT

model_config = GPT.get_default_config()
model_config.model_type = <span class="hljs-string">&#x27;gpt2-xl&#x27;</span>
model_config.vocab_size = <span class="hljs-number">50257</span>
model_config.block_size = <span class="hljs-number">1024</span>

<span class="hljs-keyword">with</span> init_empty_weights():
    empty_model = GPT(model_config)

bnb_quantization_config = BnbQuantizationConfig(
  load_in_4bit=<span class="hljs-literal">True</span>,
  bnb_4bit_compute_dtype=torch.bfloat16,  <span class="hljs-comment"># optional</span>
  bnb_4bit_use_double_quant=<span class="hljs-literal">True</span>,         <span class="hljs-comment"># optional</span>
  bnb_4bit_quant_type=<span class="hljs-string">&quot;nf4&quot;</span>               <span class="hljs-comment"># optional</span>
)

quantized_model = load_and_quantize_model(
  empty_model,
  weights_location=weights_location,
  bnb_quantization_config=bnb_quantization_config,
  device_map = <span class="hljs-string">&quot;auto&quot;</span>
)<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-1hrouw">For further details, e.g. model saving, cpu-offloading andfine-tuning, please review the <a href="https://huggingface.co/docs/accelerate/en/usage_guides/quantization" rel="nofollow"><code>bitsandbytes</code> section in the Accelerate docs</a>.</p>  <h1 class="relative group"><a id="pytorch-lightning-and-lightning-fabric" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#pytorch-lightning-and-lightning-fabric"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>PyTorch Lightning and Lightning Fabric</span></h1> <p data-svelte-h="svelte-fwwq6x">Bitsandbytes is available from within both</p> <ul data-svelte-h="svelte-jsr43i"><li><a href="https://lightning.ai/docs/pytorch/stable/" rel="nofollow">PyTorch Lightning</a>, a deep learning framework for professional AI researchers and machine learning engineers who need maximal flexibility without sacrificing performance at scale;</li> <li>and <a href="https://lightning.ai/docs/fabric/stable/" rel="nofollow">Lightning Fabric</a>, a fast and lightweight way to scale PyTorch models without boilerplate).</li></ul> <p data-svelte-h="svelte-bgq7h4">Please review the <a href="https://lightning.ai/docs/pytorch/stable/common/precision_intermediate.html#quantization-via-bitsandbytes" rel="nofollow">bitsandbytes section in the PyTorch Lightning docs</a>.</p>  <h1 class="relative group"><a id="lit-gpt" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#lit-gpt"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Lit-GPT</span></h1> <p data-svelte-h="svelte-gins1u">Bitsandbytes is integrated into <a href="https://github.com/Lightning-AI/lit-gpt" rel="nofollow">Lit-GPT</a>, a hackable implementation of state-of-the-art open-source large language models, based on Lightning Fabric, where it can be used for quantization during training, finetuning, and inference.</p> <p data-svelte-h="svelte-pd1i6i">Please review the <a href="https://github.com/Lightning-AI/lit-gpt/blob/main/tutorials/quantize.md" rel="nofollow">bitsandbytes section in the Lit-GPT quantization docs</a>.</p>  <h1 class="relative group"><a id="trainer-for-the-optimizers" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#trainer-for-the-optimizers"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Trainer for the optimizers</span></h1> <p data-svelte-h="svelte-1lta7zi">You can use any of the 8-bit and/or paged optimizers by simple passing them to the <code>transformers.Trainer</code> class on initialization.All bnb optimizers are supported by passing the correct string in <code>TrainingArguments</code>’s <code>optim</code> attribute - e.g. (<code>paged_adamw_32bit</code>).</p> <p data-svelte-h="svelte-l324p4">See the <a href="https://huggingface.co/docs/transformers/main_classes/trainer#transformers.Trainer" rel="nofollow">official API docs for reference</a>.</p> <p data-svelte-h="svelte-1auupce">Here we point out to relevant doc sections in transformers / peft / Trainer + very briefly explain how these are integrated:
e.g. for transformers state that you can load any model in 8-bit / 4-bit precision, for PEFT, you can use QLoRA out of the box with <code>LoraConfig</code> + 4-bit base model, for Trainer: all bnb optimizers are supported by passing the correct string in <code>TrainingArguments</code>’s <code>optim</code> attribute - e.g. (<code>paged_adamw_32bit</code>):</p>  <h1 class="relative group"><a id="blog-posts" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#blog-posts"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Blog posts</span></h1> <ul data-svelte-h="svelte-12n4v02"><li><a href="https://huggingface.co/blog/4bit-transformers-bitsandbytes" rel="nofollow">Making LLMs even more accessible with <code>bitsandbytes</code>, 4-bit quantization and QLoRA</a></li> <li><a href="https://huggingface.co/blog/hf-bitsandbytes-integration" rel="nofollow">A Gentle Introduction to 8-bit Matrix Multiplication for transformers at scale using Hugging Face Transformers, Accelerate and <code>bitsandbytes</code></a></li></ul>  <p></p> 
			
			<script>
				{
					__sveltekit_155r5qs = {
						assets: "/docs/bitsandbytes/v0.43.0/en",
						base: "/docs/bitsandbytes/v0.43.0/en",
						env: {}
					};

					const element = document.currentScript.parentElement;

					const data = [null,null];

					Promise.all([
						import("/docs/bitsandbytes/v0.43.0/en/_app/immutable/entry/start.eac2c940.js"),
						import("/docs/bitsandbytes/v0.43.0/en/_app/immutable/entry/app.24bdf5c5.js")
					]).then(([kit, app]) => {
						kit.start(app, element, {
							node_ids: [0, 8],
							data,
							form: null,
							error: null
						});
					});
				}
			</script>