Buckets:
| <meta charset="utf-8" /><meta name="hf:doc:metadata" content="{"title":"DeepSpeed","local":"deepspeed","sections":[{"title":"Compatibility with bitsandbytes quantization + LoRA","local":"compatibility-with-bitsandbytes-quantization--lora","sections":[],"depth":2}],"depth":1}"> | |
| <link href="/docs/peft/pr_3206/en/_app/immutable/assets/0.e3b0c442.css" rel="modulepreload"> | |
| <link rel="modulepreload" href="/docs/peft/pr_3206/en/_app/immutable/entry/start.06e1cdbd.js"> | |
| <link rel="modulepreload" href="/docs/peft/pr_3206/en/_app/immutable/chunks/scheduler.78382b47.js"> | |
| <link rel="modulepreload" href="/docs/peft/pr_3206/en/_app/immutable/chunks/singletons.25ed789f.js"> | |
| <link rel="modulepreload" href="/docs/peft/pr_3206/en/_app/immutable/chunks/index.fadd215c.js"> | |
| <link rel="modulepreload" href="/docs/peft/pr_3206/en/_app/immutable/chunks/paths.d98268af.js"> | |
| <link rel="modulepreload" href="/docs/peft/pr_3206/en/_app/immutable/entry/app.cb2a6689.js"> | |
| <link rel="modulepreload" href="/docs/peft/pr_3206/en/_app/immutable/chunks/preload-helper.1df7c689.js"> | |
| <link rel="modulepreload" href="/docs/peft/pr_3206/en/_app/immutable/chunks/index.6dd35eb6.js"> | |
| <link rel="modulepreload" href="/docs/peft/pr_3206/en/_app/immutable/nodes/0.fa4b1245.js"> | |
| <link rel="modulepreload" href="/docs/peft/pr_3206/en/_app/immutable/chunks/each.e59479a4.js"> | |
| <link rel="modulepreload" href="/docs/peft/pr_3206/en/_app/immutable/nodes/2.89ed1ac6.js"> | |
| <link rel="modulepreload" href="/docs/peft/pr_3206/en/_app/immutable/chunks/MermaidChart.svelte_svelte_type_style_lang.db10b59f.js"> | |
| <link rel="modulepreload" href="/docs/peft/pr_3206/en/_app/immutable/chunks/CodeBlock.e9241c92.js"><!-- HEAD_svelte-u9bgzb_START --><meta name="hf:doc:metadata" content="{"title":"DeepSpeed","local":"deepspeed","sections":[{"title":"Compatibility with bitsandbytes quantization + LoRA","local":"compatibility-with-bitsandbytes-quantization--lora","sections":[],"depth":2}],"depth":1}"><!-- HEAD_svelte-u9bgzb_END --> <p></p> <div class="items-center shrink-0 min-w-[100px] max-sm:min-w-[50px] justify-end ml-auto flex" style="float: right; margin-left: 10px; display: inline-flex; position: relative; z-index: 10;"><div class="inline-flex rounded-md max-sm:rounded-sm"><button class="inline-flex items-center gap-1 h-7 max-sm:h-7 px-2 max-sm:px-1.5 text-sm font-medium text-gray-800 border border-r-0 rounded-l-md max-sm:rounded-l-sm border-gray-200 bg-white hover:shadow-inner dark:border-gray-850 dark:bg-gray-950 dark:text-gray-200 dark:hover:bg-gray-800" aria-live="polite"><span class="inline-flex items-center justify-center rounded-md p-0.5 max-sm:p-0 hover:text-gray-800 dark:hover:text-gray-200"><svg class="sm:size-3.5 size-3" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg></span> <span>Copy page</span></button> <button class="inline-flex items-center justify-center w-6 max-sm:w-5 h-7 max-sm:h-7 disabled:pointer-events-none text-sm text-gray-500 hover:text-gray-700 dark:hover:text-white rounded-r-md max-sm:rounded-r-sm border border-l transition border-gray-200 bg-white hover:shadow-inner dark:border-gray-850 dark:bg-gray-950 dark:text-gray-200 dark:hover:bg-gray-800" aria-haspopup="menu" aria-expanded="false" aria-label="Open copy menu"><svg class="transition-transform text-gray-400 overflow-visible sm:size-3.5 size-3 rotate-0" width="1em" height="1em" viewBox="0 0 12 7" fill="none" xmlns="http://www.w3.org/2000/svg"><path d="M1 1L6 6L11 1" stroke="currentColor"></path></svg></button></div> </div> <h1 class="relative group"><a id="deepspeed" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#deepspeed"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>DeepSpeed</span></h1> <p data-svelte-h="svelte-kjgvfp"><a href="https://www.deepspeed.ai/" rel="nofollow">DeepSpeed</a> is a library designed for speed and scale for distributed training of large models with billions of parameters. At its core is the Zero Redundancy Optimizer (ZeRO) that shards optimizer states (ZeRO-1), gradients (ZeRO-2), and parameters (ZeRO-3) across data parallel processes. This drastically reduces memory usage, allowing you to scale your training to billion parameter models. To unlock even more memory efficiency, ZeRO-Offload reduces GPU compute and memory by leveraging CPU resources during optimization.</p> <p data-svelte-h="svelte-1o8k4c8">Both of these features are supported in 🤗 Accelerate, and you can use them with 🤗 PEFT.</p> <h2 class="relative group"><a id="compatibility-with-bitsandbytes-quantization--lora" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#compatibility-with-bitsandbytes-quantization--lora"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Compatibility with bitsandbytes quantization + LoRA</span></h2> <p data-svelte-h="svelte-1pmd8qt">Below is a table that summarizes the compatibility between PEFT’s LoRA, <a href="https://github.com/TimDettmers/bitsandbytes" rel="nofollow"><code>bitsandbytes</code></a> library and DeepSpeed Zero stages with respect to fine-tuning. DeepSpeed Zero-1 and 2 will have no effect at inference as stage 1 shards the optimizer states and stage 2 shards the optimizer states and gradients:</p> <table data-svelte-h="svelte-uxge3s"><thead><tr><th>DeepSpeed stage</th> <th>Is compatible?</th></tr></thead> <tbody><tr><td>Zero-1</td> <td>🟢</td></tr> <tr><td>Zero-2</td> <td>🟢</td></tr> <tr><td>Zero-3</td> <td>🟢</td></tr></tbody></table> <p data-svelte-h="svelte-b8ifuk">For DeepSpeed Stage 3 + QLoRA, please refer to the section <a href="#use-peft-qlora-and-deepspeed-with-zero3-for-finetuning-large-models-on-multiple-gpus">Use PEFT QLoRA and DeepSpeed with ZeRO3 for finetuning large models on multiple GPUs</a> below.</p> <p data-svelte-h="svelte-1jhakf7">For confirming these observations, we ran the SFT (Supervised Fine-tuning) <a href="https://github.com/huggingface/trl/tree/main/examples" rel="nofollow">offical example scripts</a> of the <a href="https://github.com/huggingface/trl" rel="nofollow">Transformers Reinforcement Learning (TRL) library</a> using QLoRA + PEFT and the accelerate configs available <a href="https://github.com/huggingface/trl/tree/main/examples/accelerate_configs" rel="nofollow">here</a>. We ran these experiments on a 2x NVIDIA T4 GPU.</p> <h1 class="relative group"><a id="use-peft-and-deepspeed-with-zero3-for-finetuning-large-models-on-multiple-devices-and-multiple-nodes" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#use-peft-and-deepspeed-with-zero3-for-finetuning-large-models-on-multiple-devices-and-multiple-nodes"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Use PEFT and DeepSpeed with ZeRO3 for finetuning large models on multiple devices and multiple nodes</span></h1> <p data-svelte-h="svelte-1ubz7wn">This section of guide will help you learn how to use our DeepSpeed <a href="https://github.com/huggingface/peft/blob/main/examples/sft/train.py" rel="nofollow">training script</a> for performing SFT. You’ll configure the script to do SFT (supervised fine-tuning) of Llama-70B model with LoRA and ZeRO-3 on 8xH100 80GB GPUs on a single machine. You can configure it to scale to multiple machines by changing the accelerate config.</p> <h2 class="relative group"><a id="configuration" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#configuration"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Configuration</span></h2> <p data-svelte-h="svelte-ha3yfb">Start by running the following command to <a href="https://huggingface.co/docs/accelerate/quicktour#launching-your-distributed-script" rel="nofollow">create a DeepSpeed configuration file</a> with 🤗 Accelerate. The <code>--config_file</code> flag allows you to save the configuration file to a specific location, otherwise it is saved as a <code>default_config.yaml</code> file in the 🤗 Accelerate cache.</p> <p data-svelte-h="svelte-1oar51z">The configuration file is used to set the default options when you launch the training script.</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->accelerate config --config_file deepspeed_config.yaml<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-u6lpbh">You’ll be asked a few questions about your setup, and configure the following arguments. In this example, you’ll use ZeRO-3 so make sure you pick those options.</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->`zero_stage`: [0] Disabled, [1] optimizer state partitioning, [2] optimizer+gradient state partitioning and [3] optimizer+gradient+parameter partitioning | |
| `gradient_accumulation_steps`: Number of training steps to accumulate gradients before averaging and applying them. Pass the same value as you would pass via cmd argument <span class="hljs-keyword">else</span> you will encounter mismatch error. | |
| `gradient_clipping`: Enable gradient clipping with value. Don<span class="hljs-string">'t set this as you will be passing it via cmd arguments. | |
| `offload_optimizer_device`: [none] Disable optimizer offloading, [cpu] offload optimizer to CPU, [nvme] offload optimizer to NVMe SSD. Only applicable with ZeRO >= Stage-2. Set this as `none` as don'</span>t want to <span class="hljs-built_in">enable</span> offloading. | |
| `offload_param_device`: [none] Disable parameter offloading, [cpu] offload parameters to CPU, [nvme] offload parameters to NVMe SSD. Only applicable with ZeRO Stage-3. Set this as `none` as don<span class="hljs-string">'t want to enable offloading. | |
| `zero3_init_flag`: Decides whether to enable `deepspeed.zero.Init` for constructing massive models. Only applicable with ZeRO Stage-3. Set this to `True`. | |
| `zero3_save_16bit_model`: Decides whether to save 16-bit model weights when using ZeRO Stage-3. Set this to `True`. | |
| `mixed_precision`: `no` for FP32 training, `fp16` for FP16 mixed-precision training and `bf16` for BF16 mixed-precision training. Set this to `True`.</span><!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-1dwo8p1">Once this is done, the corresponding config should look like below and you can find it in config folder at <a href="https://github.com/huggingface/peft/blob/main/examples/sft/configs/deepspeed_config.yaml" rel="nofollow">deepspeed_config.yaml</a>:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-attr">compute_environment:</span> <span class="hljs-string">LOCAL_MACHINE</span> | |
| <span class="hljs-attr">debug:</span> <span class="hljs-literal">false</span> | |
| <span class="hljs-attr">deepspeed_config:</span> | |
| <span class="hljs-attr">deepspeed_multinode_launcher:</span> <span class="hljs-string">standard</span> | |
| <span class="hljs-attr">gradient_accumulation_steps:</span> <span class="hljs-number">4</span> | |
| <span class="hljs-attr">offload_optimizer_device:</span> <span class="hljs-string">none</span> | |
| <span class="hljs-attr">offload_param_device:</span> <span class="hljs-string">none</span> | |
| <span class="hljs-attr">zero3_init_flag:</span> <span class="hljs-literal">true</span> | |
| <span class="hljs-attr">zero3_save_16bit_model:</span> <span class="hljs-literal">true</span> | |
| <span class="hljs-attr">zero_stage:</span> <span class="hljs-number">3</span> | |
| <span class="hljs-attr">distributed_type:</span> <span class="hljs-string">DEEPSPEED</span> | |
| <span class="hljs-attr">downcast_bf16:</span> <span class="hljs-string">'no'</span> | |
| <span class="hljs-attr">machine_rank:</span> <span class="hljs-number">0</span> | |
| <span class="hljs-attr">main_training_function:</span> <span class="hljs-string">main</span> | |
| <span class="hljs-attr">mixed_precision:</span> <span class="hljs-string">bf16</span> | |
| <span class="hljs-attr">num_machines:</span> <span class="hljs-number">1</span> | |
| <span class="hljs-attr">num_processes:</span> <span class="hljs-number">8</span> | |
| <span class="hljs-attr">rdzv_backend:</span> <span class="hljs-string">static</span> | |
| <span class="hljs-attr">same_network:</span> <span class="hljs-literal">true</span> | |
| <span class="hljs-attr">tpu_env:</span> [] | |
| <span class="hljs-attr">tpu_use_cluster:</span> <span class="hljs-literal">false</span> | |
| <span class="hljs-attr">tpu_use_sudo:</span> <span class="hljs-literal">false</span> | |
| <span class="hljs-attr">use_cpu:</span> <span class="hljs-literal">false</span><!-- HTML_TAG_END --></pre></div> <h2 class="relative group"><a id="launch-command" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#launch-command"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Launch command</span></h2> <p data-svelte-h="svelte-if2v7q">The launch command is available at <a href="https://github.com/huggingface/peft/blob/main/examples/sft/run_peft_deepspeed.sh" rel="nofollow">run_peft_deepspeed.sh</a> and it is also shown below:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->accelerate launch --config_file <span class="hljs-string">"configs/deepspeed_config.yaml"</span> train.py \ | |
| --seed 100 \ | |
| --model_name_or_path <span class="hljs-string">"meta-llama/Llama-2-70b-hf"</span> \ | |
| --dataset_name <span class="hljs-string">"smangrul/ultrachat-10k-chatml"</span> \ | |
| --chat_template_format <span class="hljs-string">"chatml"</span> \ | |
| --add_special_tokens False \ | |
| --append_concat_token False \ | |
| --splits <span class="hljs-string">"train,test"</span> \ | |
| --max_seq_len 2048 \ | |
| --num_train_epochs 1 \ | |
| --logging_steps 5 \ | |
| --log_level <span class="hljs-string">"info"</span> \ | |
| --logging_strategy <span class="hljs-string">"steps"</span> \ | |
| --eval_strategy <span class="hljs-string">"epoch"</span> \ | |
| --save_strategy <span class="hljs-string">"epoch"</span> \ | |
| --push_to_hub \ | |
| --hub_private_repo True \ | |
| --hub_strategy <span class="hljs-string">"every_save"</span> \ | |
| --bf16 True \ | |
| --packing True \ | |
| --learning_rate 1e-4 \ | |
| --lr_scheduler_type <span class="hljs-string">"cosine"</span> \ | |
| --weight_decay 1e-4 \ | |
| --warmup_steps 0 \ | |
| --max_grad_norm 1.0 \ | |
| --output_dir <span class="hljs-string">"llama-sft-lora-deepspeed"</span> \ | |
| --per_device_train_batch_size 8 \ | |
| --per_device_eval_batch_size 8 \ | |
| --gradient_accumulation_steps 4 \ | |
| --gradient_checkpointing True \ | |
| --use_reentrant False \ | |
| --dataset_text_field <span class="hljs-string">"content"</span> \ | |
| --use_flash_attn True \ | |
| --use_peft_lora True \ | |
| --lora_r 8 \ | |
| --lora_alpha 16 \ | |
| --lora_dropout 0.1 \ | |
| --lora_target_modules <span class="hljs-string">"all-linear"</span> \ | |
| --use_4bit_quantization False<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-1juuctw">Notice that we are using LoRA with rank=8, alpha=16 and targeting all linear layers. We are passing the deepspeed config file and finetuning 70B Llama model on a subset of the ultrachat dataset.</p> <h2 class="relative group"><a id="the-important-parts" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#the-important-parts"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>The important parts</span></h2> <p data-svelte-h="svelte-y7jq4k">Let’s dive a little deeper into the script so you can see what’s going on, and understand how it works.</p> <p data-svelte-h="svelte-10ugdt7">The first thing to know is that the script uses DeepSpeed for distributed training as the DeepSpeed config has been passed. The <a href="https://huggingface.co/docs/trl/main/en/sft_trainer#trl.SFTTrainer" rel="nofollow">SFTTrainer</a> class handles all the heavy lifting of creating the PEFT model using the peft config that is passed. After that, when you call <code>trainer.train()</code>, <a href="https://huggingface.co/docs/trl/main/en/sft_trainer#trl.SFTTrainer" rel="nofollow">SFTTrainer</a> internally uses 🤗 Accelerate to prepare the model, optimizer and trainer using the DeepSpeed config to create DeepSpeed engine which is then trained. The main code snippet is below:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-comment"># trainer</span> | |
| trainer = SFTTrainer( | |
| model=model, | |
| processing_class=tokenizer, | |
| args=training_args, | |
| train_dataset=train_dataset, | |
| eval_dataset=eval_dataset, | |
| peft_config=peft_config, | |
| ) | |
| trainer.accelerator.<span class="hljs-built_in">print</span>(<span class="hljs-string">f"<span class="hljs-subst">{trainer.model}</span>"</span>) | |
| <span class="hljs-comment"># train</span> | |
| checkpoint = <span class="hljs-literal">None</span> | |
| <span class="hljs-keyword">if</span> training_args.resume_from_checkpoint <span class="hljs-keyword">is</span> <span class="hljs-keyword">not</span> <span class="hljs-literal">None</span>: | |
| checkpoint = training_args.resume_from_checkpoint | |
| trainer.train(resume_from_checkpoint=checkpoint) | |
| <span class="hljs-comment"># saving final model</span> | |
| trainer.save_model()<!-- HTML_TAG_END --></pre></div> <h2 class="relative group"><a id="memory-usage" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#memory-usage"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Memory usage</span></h2> <p data-svelte-h="svelte-1w0ujmv">In the above example, the memory consumed per GPU is 64 GB (80%) as seen in the screenshot below:</p> <div class="flex justify-center" data-svelte-h="svelte-1xcsndp"><img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/peft/peft_deepspeed_mem_usage.png"></div> <small data-svelte-h="svelte-8v78s3">GPU memory usage for the training run</small> <h2 class="relative group"><a id="more-resources" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#more-resources"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>More resources</span></h2> <p data-svelte-h="svelte-d9sf55">You can also refer this blog post <a href="https://medium.com/@sourabmangrulkar/falcon-180b-finetuning-using-peft-and-deepspeed-b92643091d99" rel="nofollow">Falcon 180B Finetuning using 🤗 PEFT and DeepSpeed</a> on how to finetune 180B Falcon model on 16 A100 GPUs on 2 machines.</p> <h1 class="relative group"><a id="use-peft-qlora-and-deepspeed-with-zero3-for-finetuning-large-models-on-multiple-gpus" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#use-peft-qlora-and-deepspeed-with-zero3-for-finetuning-large-models-on-multiple-gpus"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Use PEFT QLoRA and DeepSpeed with ZeRO3 for finetuning large models on multiple GPUs</span></h1> <p data-svelte-h="svelte-1lia0iy">In this section, we will look at how to use QLoRA and DeepSpeed Stage-3 for finetuning 70B llama model on 2X40GB GPUs. | |
| For this, we first need <code>bitsandbytes>=0.43.3</code>, <code>accelerate>=1.0.1</code>, <code>transformers>4.44.2</code>, <code>trl>0.11.4</code> and <code>peft>0.13.0</code>. We need to set <code>zero3_init_flag</code> to true when using Accelerate config. Below is the config which can be found at <a href="https://github.com/huggingface/peft/blob/main/examples/sft/configs/deepspeed_config_z3_qlora.yaml" rel="nofollow">deepspeed_config_z3_qlora.yaml</a>:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-attr">compute_environment:</span> <span class="hljs-string">LOCAL_MACHINE</span> | |
| <span class="hljs-attr">debug:</span> <span class="hljs-literal">false</span> | |
| <span class="hljs-attr">deepspeed_config:</span> | |
| <span class="hljs-attr">deepspeed_multinode_launcher:</span> <span class="hljs-string">standard</span> | |
| <span class="hljs-attr">offload_optimizer_device:</span> <span class="hljs-string">none</span> | |
| <span class="hljs-attr">offload_param_device:</span> <span class="hljs-string">none</span> | |
| <span class="hljs-attr">zero3_init_flag:</span> <span class="hljs-literal">true</span> | |
| <span class="hljs-attr">zero3_save_16bit_model:</span> <span class="hljs-literal">true</span> | |
| <span class="hljs-attr">zero_stage:</span> <span class="hljs-number">3</span> | |
| <span class="hljs-attr">distributed_type:</span> <span class="hljs-string">DEEPSPEED</span> | |
| <span class="hljs-attr">downcast_bf16:</span> <span class="hljs-string">'no'</span> | |
| <span class="hljs-attr">machine_rank:</span> <span class="hljs-number">0</span> | |
| <span class="hljs-attr">main_training_function:</span> <span class="hljs-string">main</span> | |
| <span class="hljs-attr">mixed_precision:</span> <span class="hljs-string">bf16</span> | |
| <span class="hljs-attr">num_machines:</span> <span class="hljs-number">1</span> | |
| <span class="hljs-attr">num_processes:</span> <span class="hljs-number">2</span> | |
| <span class="hljs-attr">rdzv_backend:</span> <span class="hljs-string">static</span> | |
| <span class="hljs-attr">same_network:</span> <span class="hljs-literal">true</span> | |
| <span class="hljs-attr">tpu_env:</span> [] | |
| <span class="hljs-attr">tpu_use_cluster:</span> <span class="hljs-literal">false</span> | |
| <span class="hljs-attr">tpu_use_sudo:</span> <span class="hljs-literal">false</span> | |
| <span class="hljs-attr">use_cpu:</span> <span class="hljs-literal">false</span><!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-1wnm4d7">Launch command is given below which is available at <a href="https://github.com/huggingface/peft/blob/main/examples/sft/run_peft_qlora_deepspeed_stage3.sh" rel="nofollow">run_peft_qlora_deepspeed_stage3.sh</a>:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->accelerate launch <span class="hljs-attr">--config_file</span> <span class="hljs-string">"configs/deepspeed_config_z3_qlora.yaml"</span> train<span class="hljs-selector-class">.py</span> \ | |
| <span class="hljs-attr">--seed</span> <span class="hljs-number">100</span> \ | |
| <span class="hljs-attr">--model_name_or_path</span> <span class="hljs-string">"meta-llama/Llama-2-70b-hf"</span> \ | |
| <span class="hljs-attr">--dataset_name</span> <span class="hljs-string">"smangrul/ultrachat-10k-chatml"</span> \ | |
| <span class="hljs-attr">--chat_template_format</span> <span class="hljs-string">"chatml"</span> \ | |
| <span class="hljs-attr">--add_special_tokens</span> False \ | |
| <span class="hljs-attr">--append_concat_token</span> False \ | |
| <span class="hljs-attr">--splits</span> <span class="hljs-string">"train,test"</span> \ | |
| <span class="hljs-attr">--max_seq_len</span> <span class="hljs-number">2048</span> \ | |
| <span class="hljs-attr">--num_train_epochs</span> <span class="hljs-number">1</span> \ | |
| <span class="hljs-attr">--logging_steps</span> <span class="hljs-number">5</span> \ | |
| <span class="hljs-attr">--log_level</span> <span class="hljs-string">"info"</span> \ | |
| <span class="hljs-attr">--logging_strategy</span> <span class="hljs-string">"steps"</span> \ | |
| <span class="hljs-attr">--eval_strategy</span> <span class="hljs-string">"epoch"</span> \ | |
| <span class="hljs-attr">--save_strategy</span> <span class="hljs-string">"epoch"</span> \ | |
| <span class="hljs-attr">--push_to_hub</span> \ | |
| <span class="hljs-attr">--hub_private_repo</span> True \ | |
| <span class="hljs-attr">--hub_strategy</span> <span class="hljs-string">"every_save"</span> \ | |
| <span class="hljs-attr">--bf16</span> True \ | |
| <span class="hljs-attr">--packing</span> True \ | |
| <span class="hljs-attr">--learning_rate</span> <span class="hljs-number">1</span>e-<span class="hljs-number">4</span> \ | |
| <span class="hljs-attr">--lr_scheduler_type</span> <span class="hljs-string">"cosine"</span> \ | |
| <span class="hljs-attr">--weight_decay</span> <span class="hljs-number">1</span>e-<span class="hljs-number">4</span> \ | |
| <span class="hljs-attr">--warmup_steps</span> <span class="hljs-number">0</span> \ | |
| <span class="hljs-attr">--max_grad_norm</span> <span class="hljs-number">1.0</span> \ | |
| <span class="hljs-attr">--output_dir</span> <span class="hljs-string">"llama-sft-qlora-dsz3"</span> \ | |
| <span class="hljs-attr">--per_device_train_batch_size</span> <span class="hljs-number">2</span> \ | |
| <span class="hljs-attr">--per_device_eval_batch_size</span> <span class="hljs-number">2</span> \ | |
| <span class="hljs-attr">--gradient_accumulation_steps</span> <span class="hljs-number">2</span> \ | |
| <span class="hljs-attr">--gradient_checkpointing</span> True \ | |
| <span class="hljs-attr">--use_reentrant</span> True \ | |
| <span class="hljs-attr">--dataset_text_field</span> <span class="hljs-string">"content"</span> \ | |
| <span class="hljs-attr">--use_flash_attn</span> True \ | |
| <span class="hljs-attr">--use_peft_lora</span> True \ | |
| <span class="hljs-attr">--lora_r</span> <span class="hljs-number">8</span> \ | |
| <span class="hljs-attr">--lora_alpha</span> <span class="hljs-number">16</span> \ | |
| <span class="hljs-attr">--lora_dropout</span> <span class="hljs-number">0.1</span> \ | |
| <span class="hljs-attr">--lora_target_modules</span> <span class="hljs-string">"all-linear"</span> \ | |
| <span class="hljs-attr">--use_4bit_quantization</span> True \ | |
| <span class="hljs-attr">--use_nested_quant</span> True \ | |
| <span class="hljs-attr">--bnb_4bit_compute_dtype</span> <span class="hljs-string">"bfloat16"</span> \ | |
| <span class="hljs-attr">--bnb_4bit_quant_storage_dtype</span> <span class="hljs-string">"bfloat16"</span><!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-1mee3xa">Notice the new argument being passed <code>bnb_4bit_quant_storage_dtype</code> which denotes the data type for packing the 4-bit parameters. For example, when it is set to <code>bfloat16</code>, <strong>32/4 = 8</strong> 4-bit params are packed together post quantization.</p> <p data-svelte-h="svelte-19s9lyl">In terms of training code, the important code changes are:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->... | |
| bnb_config = BitsAndBytesConfig( | |
| load_in_4bit=args.use_4bit_quantization, | |
| bnb_4bit_quant_type=args.bnb_4bit_quant_type, | |
| bnb_4bit_compute_dtype=compute_dtype, | |
| bnb_4bit_use_double_quant=args.use_nested_quant, | |
| <span class="hljs-addition">+ bnb_4bit_quant_storage=quant_storage_dtype,</span> | |
| ) | |
| ... | |
| model = AutoModelForCausalLM.from_pretrained( | |
| args.model_name_or_path, | |
| quantization_config=bnb_config, | |
| trust_remote_code=True, | |
| attn_implementation="flash_attention_2" if args.use_flash_attn else "eager", | |
| <span class="hljs-addition">+ dtype=quant_storage_dtype or torch.float32,</span> | |
| )<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-46tfpq">Notice that <code>dtype</code> for <code>AutoModelForCausalLM</code> is same as the <code>bnb_4bit_quant_storage</code> data type. That’s it. Everything else is handled by Trainer and TRL.</p> <h2 class="relative group"><a id="memory-usage" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#memory-usage"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Memory usage</span></h2> <p data-svelte-h="svelte-1bfnr2r">In the above example, the memory consumed per GPU is <strong>36.6 GB</strong>. Therefore, what took 8X80GB GPUs with DeepSpeed Stage 3+LoRA and a couple of 80GB GPUs with DDP+QLoRA now requires 2X40GB GPUs. This makes finetuning of large models more accessible.</p> <h1 class="relative group"><a id="use-peft-and-deepspeed-with-zero3-and-cpu-offloading-for-finetuning-large-models-on-a-single-gpu" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#use-peft-and-deepspeed-with-zero3-and-cpu-offloading-for-finetuning-large-models-on-a-single-gpu"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Use PEFT and DeepSpeed with ZeRO3 and CPU Offloading for finetuning large models on a single GPU</span></h1> <p data-svelte-h="svelte-jhfwqh">This section of guide will help you learn how to use our DeepSpeed <a href="https://github.com/huggingface/peft/blob/main/examples/conditional_generation/peft_lora_seq2seq_accelerate_ds_zero3_offload.py" rel="nofollow">training script</a>. You’ll configure the script to train a large model for conditional generation with ZeRO-3 and CPU Offload.</p> <blockquote class="tip" data-svelte-h="svelte-1wnq58o"><p>💡 To help you get started, check out our example training scripts for <a href="https://github.com/huggingface/peft/blob/main/examples/causal_language_modeling/peft_lora_clm_accelerate_ds_zero3_offload.py" rel="nofollow">causal language modeling</a> and <a href="https://github.com/huggingface/peft/blob/main/examples/conditional_generation/peft_lora_seq2seq_accelerate_ds_zero3_offload.py" rel="nofollow">conditional generation</a>. You can adapt these scripts for your own applications or even use them out of the box if your task is similar to the one in the scripts.</p></blockquote> <h2 class="relative group"><a id="configuration" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#configuration"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Configuration</span></h2> <p data-svelte-h="svelte-ha3yfb">Start by running the following command to <a href="https://huggingface.co/docs/accelerate/quicktour#launching-your-distributed-script" rel="nofollow">create a DeepSpeed configuration file</a> with 🤗 Accelerate. The <code>--config_file</code> flag allows you to save the configuration file to a specific location, otherwise it is saved as a <code>default_config.yaml</code> file in the 🤗 Accelerate cache.</p> <p data-svelte-h="svelte-1oar51z">The configuration file is used to set the default options when you launch the training script.</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->accelerate config --config_file ds_zero3_cpu.yaml<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-16uruk2">You’ll be asked a few questions about your setup, and configure the following arguments. In this example, you’ll use ZeRO-3 along with CPU-Offload so make sure you pick those options.</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->`zero_stage`: [0] Disabled, [1] optimizer state partitioning, [2] optimizer+gradient state partitioning and [3] optimizer+gradient+parameter partitioning | |
| `gradient_accumulation_steps`: Number of training steps to accumulate gradients before averaging and applying them. | |
| `gradient_clipping`: Enable gradient clipping with value. | |
| `offload_optimizer_device`: [none] Disable optimizer offloading, [cpu] offload optimizer to CPU, [nvme] offload optimizer to NVMe SSD. Only applicable with ZeRO >= Stage-2. | |
| `offload_param_device`: [none] Disable parameter offloading, [cpu] offload parameters to CPU, [nvme] offload parameters to NVMe SSD. Only applicable with ZeRO Stage-3. | |
| `zero3_init_flag`: Decides whether to <span class="hljs-built_in">enable</span> `deepspeed.zero.Init` <span class="hljs-keyword">for</span> constructing massive models. Only applicable with ZeRO Stage-3. | |
| `zero3_save_16bit_model`: Decides whether to save 16-bit model weights when using ZeRO Stage-3. | |
| `mixed_precision`: `no` <span class="hljs-keyword">for</span> FP32 training, `fp16` <span class="hljs-keyword">for</span> FP16 mixed-precision training and `bf16` <span class="hljs-keyword">for</span> BF16 mixed-precision training. <!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-iawfkk">An example <a href="https://github.com/huggingface/peft/blob/main/examples/conditional_generation/accelerate_ds_zero3_cpu_offload_config.yaml" rel="nofollow">configuration file</a> might look like the following. The most important thing to notice is that <code>zero_stage</code> is set to <code>3</code>, and <code>offload_optimizer_device</code> and <code>offload_param_device</code> are set to the <code>cpu</code>.</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-attr">compute_environment:</span> <span class="hljs-string">LOCAL_MACHINE</span> | |
| <span class="hljs-attr">deepspeed_config:</span> | |
| <span class="hljs-attr">gradient_accumulation_steps:</span> <span class="hljs-number">1</span> | |
| <span class="hljs-attr">gradient_clipping:</span> <span class="hljs-number">1.0</span> | |
| <span class="hljs-attr">offload_optimizer_device:</span> <span class="hljs-string">cpu</span> | |
| <span class="hljs-attr">offload_param_device:</span> <span class="hljs-string">cpu</span> | |
| <span class="hljs-attr">zero3_init_flag:</span> <span class="hljs-literal">true</span> | |
| <span class="hljs-attr">zero3_save_16bit_model:</span> <span class="hljs-literal">true</span> | |
| <span class="hljs-attr">zero_stage:</span> <span class="hljs-number">3</span> | |
| <span class="hljs-attr">distributed_type:</span> <span class="hljs-string">DEEPSPEED</span> | |
| <span class="hljs-attr">downcast_bf16:</span> <span class="hljs-string">'no'</span> | |
| <span class="hljs-attr">dynamo_backend:</span> <span class="hljs-string">'NO'</span> | |
| <span class="hljs-attr">fsdp_config:</span> {} | |
| <span class="hljs-attr">machine_rank:</span> <span class="hljs-number">0</span> | |
| <span class="hljs-attr">main_training_function:</span> <span class="hljs-string">main</span> | |
| <span class="hljs-attr">megatron_lm_config:</span> {} | |
| <span class="hljs-attr">mixed_precision:</span> <span class="hljs-string">'no'</span> | |
| <span class="hljs-attr">num_machines:</span> <span class="hljs-number">1</span> | |
| <span class="hljs-attr">num_processes:</span> <span class="hljs-number">1</span> | |
| <span class="hljs-attr">rdzv_backend:</span> <span class="hljs-string">static</span> | |
| <span class="hljs-attr">same_network:</span> <span class="hljs-literal">true</span> | |
| <span class="hljs-attr">use_cpu:</span> <span class="hljs-literal">false</span><!-- HTML_TAG_END --></pre></div> <h2 class="relative group"><a id="the-important-parts" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#the-important-parts"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>The important parts</span></h2> <p data-svelte-h="svelte-y7jq4k">Let’s dive a little deeper into the script so you can see what’s going on, and understand how it works.</p> <p data-svelte-h="svelte-14y4rgi">Within the <a href="https://github.com/huggingface/peft/blob/2822398fbe896f25d4dac5e468624dc5fd65a51b/examples/conditional_generation/peft_lora_seq2seq_accelerate_ds_zero3_offload.py#L103" rel="nofollow"><code>main</code></a> function, the script creates an <a href="https://huggingface.co/docs/accelerate/main/en/package_reference/accelerator#accelerate.Accelerator" rel="nofollow">Accelerator</a> class to initialize all the necessary requirements for distributed training.</p> <blockquote class="tip" data-svelte-h="svelte-198tcjk"><p>💡 Feel free to change the model and dataset inside the <code>main</code> function. If your dataset format is different from the one in the script, you may also need to write your own preprocessing function.</p></blockquote> <p data-svelte-h="svelte-1dgjf2m">The script also creates a configuration for the 🤗 PEFT method you’re using, which in this case, is LoRA. The <a href="/docs/peft/pr_3206/en/package_reference/lora#peft.LoraConfig">LoraConfig</a> specifies the task type and important parameters such as the dimension of the low-rank matrices, the matrices scaling factor, and the dropout probability of the LoRA layers. If you want to use a different 🤗 PEFT method, make sure you replace <code>LoraConfig</code> with the appropriate <a href="../package_reference/tuners">class</a>.</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --> def main(): | |
| <span class="hljs-addition">+ accelerator = Accelerator()</span> | |
| model_name_or_path = "facebook/bart-large" | |
| dataset_name = "twitter_complaints" | |
| <span class="hljs-addition">+ peft_config = LoraConfig(</span> | |
| task_type=TaskType.SEQ_2_SEQ_LM, inference_mode=False, r=8, lora_alpha=32, lora_dropout=0.1 | |
| )<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-1k44gsr">Throughout the script, you’ll see the <a href="https://huggingface.co/docs/accelerate/main/en/package_reference/accelerator#accelerate.Accelerator.main_process_first" rel="nofollow">main_process_first</a> and <a href="https://huggingface.co/docs/accelerate/main/en/package_reference/accelerator#accelerate.Accelerator.wait_for_everyone" rel="nofollow">wait_for_everyone</a> functions which help control and synchronize when processes are executed.</p> <p data-svelte-h="svelte-1vgcnq8">The <a href="/docs/peft/pr_3206/en/package_reference/peft_model#peft.get_peft_model">get_peft_model()</a> function takes a base model and the <code>peft_config</code> you prepared earlier to create a <a href="/docs/peft/pr_3206/en/package_reference/peft_model#peft.PeftModel">PeftModel</a>:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --> model = AutoModelForSeq2SeqLM.from_pretrained(model_name_or_path) | |
| <span class="hljs-addition">+ model = get_peft_model(model, peft_config)</span><!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-yn58ot">Pass all the relevant training objects to 🤗 Accelerate’s <a href="https://huggingface.co/docs/accelerate/main/en/package_reference/accelerator#accelerate.Accelerator.prepare" rel="nofollow">prepare</a> which makes sure everything is ready for training:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->model, train_dataloader, eval_dataloader, test_dataloader, optimizer, lr_scheduler = accelerator.prepare( | |
| model, train_dataloader, eval_dataloader, test_dataloader, optimizer, lr_scheduler | |
| )<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-1lrc4ui">The next bit of code checks whether the DeepSpeed plugin is used in the <code>Accelerator</code>, and if the plugin exists, then we check if we are using ZeRO-3. This conditional flag is used when calling <code>generate</code> function call during inference for syncing GPUs when the model parameters are sharded:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->is_ds_zero_3 = <span class="hljs-literal">False</span> | |
| <span class="hljs-keyword">if</span> <span class="hljs-built_in">getattr</span>(accelerator.state, <span class="hljs-string">"deepspeed_plugin"</span>, <span class="hljs-literal">None</span>): | |
| is_ds_zero_3 = accelerator.state.deepspeed_plugin.zero_stage == <span class="hljs-number">3</span><!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-15nca2e">Inside the training loop, the usual <code>loss.backward()</code> is replaced by 🤗 Accelerate’s <a href="https://huggingface.co/docs/accelerate/main/en/package_reference/accelerator#accelerate.Accelerator.backward" rel="nofollow">backward</a> which uses the correct <code>backward()</code> method based on your configuration:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --> for epoch in range(num_epochs): | |
| with TorchTracemalloc() as tracemalloc: | |
| model.train() | |
| total_loss = 0 | |
| for step, batch in enumerate(tqdm(train_dataloader)): | |
| outputs = model(**batch) | |
| loss = outputs.loss | |
| total_loss += loss.detach().float() | |
| <span class="hljs-addition">+ accelerator.backward(loss)</span> | |
| optimizer.step() | |
| lr_scheduler.step() | |
| optimizer.zero_grad()<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-1tcey7i">That is all! The rest of the script handles the training loop, evaluation, and even pushes it to the Hub for you.</p> <h2 class="relative group"><a id="train" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#train"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Train</span></h2> <p data-svelte-h="svelte-7jdy6n">Run the following command to launch the training script. Earlier, you saved the configuration file to <code>ds_zero3_cpu.yaml</code>, so you’ll need to pass the path to the launcher with the <code>--config_file</code> argument like this:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->accelerate launch --config_file ds_zero3_cpu.yaml examples/peft_lora_seq2seq_accelerate_ds_zero3_offload.py<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-1amszxm">You’ll see some output logs that track memory usage during training, and once it’s completed, the script returns the accuracy and compares the predictions to the labels:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->GPU Memory before entering the train : 1916 | |
| GPU Memory consumed at the end of the train (end-begin): 66 | |
| GPU Peak Memory consumed during the train (max-begin): 7488 | |
| GPU Total Peak Memory consumed during the train (max): 9404 | |
| CPU Memory before entering the train : 19411 | |
| CPU Memory consumed at the end of the train (end-begin): 0 | |
| CPU Peak Memory consumed during the train (max-begin): 0 | |
| CPU Total Peak Memory consumed during the train (max): 19411 | |
| epoch=4: train_ppl=tensor(1.0705, device=<span class="hljs-string">'cuda:0'</span>) train_epoch_loss=tensor(0.0681, device=<span class="hljs-string">'cuda:0'</span>) | |
| 100%|████████████████████████████████████████████████████████████████████████████████████████████| 7/7 [00:27<00:00, 3.92s/it] | |
| GPU Memory before entering the <span class="hljs-built_in">eval</span> : 1982 | |
| GPU Memory consumed at the end of the <span class="hljs-built_in">eval</span> (end-begin): -66 | |
| GPU Peak Memory consumed during the <span class="hljs-built_in">eval</span> (max-begin): 672 | |
| GPU Total Peak Memory consumed during the <span class="hljs-built_in">eval</span> (max): 2654 | |
| CPU Memory before entering the <span class="hljs-built_in">eval</span> : 19411 | |
| CPU Memory consumed at the end of the <span class="hljs-built_in">eval</span> (end-begin): 0 | |
| CPU Peak Memory consumed during the <span class="hljs-built_in">eval</span> (max-begin): 0 | |
| CPU Total Peak Memory consumed during the <span class="hljs-built_in">eval</span> (max): 19411 | |
| accuracy=100.0 | |
| eval_preds[:10]=[<span class="hljs-string">'no complaint'</span>, <span class="hljs-string">'no complaint'</span>, <span class="hljs-string">'complaint'</span>, <span class="hljs-string">'complaint'</span>, <span class="hljs-string">'no complaint'</span>, <span class="hljs-string">'no complaint'</span>, <span class="hljs-string">'no complaint'</span>, <span class="hljs-string">'complaint'</span>, <span class="hljs-string">'complaint'</span>, <span class="hljs-string">'no complaint'</span>] | |
| dataset[<span class="hljs-string">'train'</span>][label_column][:10]=[<span class="hljs-string">'no complaint'</span>, <span class="hljs-string">'no complaint'</span>, <span class="hljs-string">'complaint'</span>, <span class="hljs-string">'complaint'</span>, <span class="hljs-string">'no complaint'</span>, <span class="hljs-string">'no complaint'</span>, <span class="hljs-string">'no complaint'</span>, <span class="hljs-string">'complaint'</span>, <span class="hljs-string">'complaint'</span>, <span class="hljs-string">'no complaint'</span>]<!-- HTML_TAG_END --></pre></div> <h1 class="relative group"><a id="caveats" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#caveats"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Caveats</span></h1> <ol data-svelte-h="svelte-19w5tf1"><li>Merging when using PEFT and DeepSpeed is currently unsupported and will raise error.</li> <li>When using CPU offloading, the major gains from using PEFT to shrink the optimizer states and gradients to that of the adapter weights would be realized on CPU RAM and there won’t be savings with respect to GPU memory.</li> <li>DeepSpeed Stage 3 and qlora when used with CPU offloading leads to more GPU memory usage when compared to disabling CPU offloading.</li></ol> <blockquote class="tip"><p data-svelte-h="svelte-wmqv6d">💡 When you have code that requires merging (and unmerging) of weights, try to manually collect the parameters with DeepSpeed Zero-3 beforehand:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-keyword">import</span> deepspeed | |
| is_ds_zero_3 = ... <span class="hljs-comment"># check if Zero-3</span> | |
| <span class="hljs-keyword">with</span> deepspeed.zero.GatheredParameters(<span class="hljs-built_in">list</span>(model.parameters()), enabled= is_ds_zero_3): | |
| model.merge_adapter() | |
| <span class="hljs-comment"># do whatever is needed, then unmerge in the same context if unmerging is required</span> | |
| ... | |
| model.unmerge_adapter()<!-- HTML_TAG_END --></pre></div></blockquote> <a class="!text-gray-400 !no-underline text-sm flex items-center not-prose mt-4" href="https://github.com/huggingface/peft/blob/main/docs/source/accelerate/deepspeed.md" target="_blank"><svg class="mr-1" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M31,16l-7,7l-1.41-1.41L28.17,16l-5.58-5.59L24,9l7,7z"></path><path d="M1,16l7-7l1.41,1.41L3.83,16l5.58,5.59L8,23l-7-7z"></path><path d="M12.419,25.484L17.639,6.552l1.932,0.518L14.351,26.002z"></path></svg> <span data-svelte-h="svelte-zjs2n5"><span class="underline">Update</span> on GitHub</span></a> <p></p> | |
| <script> | |
| { | |
| __sveltekit_1pyekj7 = { | |
| assets: "/docs/peft/pr_3206/en", | |
| base: "/docs/peft/pr_3206/en", | |
| env: {} | |
| }; | |
| const element = document.currentScript.parentElement; | |
| const data = [null,null]; | |
| Promise.all([ | |
| import("/docs/peft/pr_3206/en/_app/immutable/entry/start.06e1cdbd.js"), | |
| import("/docs/peft/pr_3206/en/_app/immutable/entry/app.cb2a6689.js") | |
| ]).then(([kit, app]) => { | |
| kit.start(app, element, { | |
| node_ids: [0, 2], | |
| data, | |
| form: null, | |
| error: null | |
| }); | |
| }); | |
| } | |
| </script> | |
Xet Storage Details
- Size:
- 81 kB
- Xet hash:
- e10e58137df29335b23d7ec76138b7e768ad9d1e6feffa3d650ae3c6da980aa6
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.